Analytics in Game Development: Data-Driven What?

Back in the old days, at the beginning of the current decade, I would have started this article with, “Analytics: You’re doing it wrong!”

These days, not so much. I still get a few approving nods, the ones with the lower lip slightly flexed and tense eyebrows denoting sincere approval with a hint of surprise, as if to say, “Hmmm, that’s interesting. I had never thought of that.” Because of that, when asked to write a post about analytics in the world of game development, I decided to go back to the basics.

And just like everything else I write, there is a story behind it…

It Was Just Another Busy Morning in September

Almost a lifetime ago. We had just released a major update of my company’s most important free-to-play game. I received a call from one of the stakeholders, who wanted to congratulate the team for the successful rollout.

But there was something else on his mind. Something that would completely shift my mindset about analytics and data science in games. He asked me, “Do we know how many gems were bought last year?” Well, we had no decent instrumentation in place that would allow me to calculate that.

But after some soul searching, it finally struck me: There is no point in gathering data just for the sake of it. That is something you did in the past because no one actually knew what to do with all that information. It’s how buzzwords like “Big Data” started to emerge, becoming part of the tech zeitgeist, like it was the year 2000 all over again.

Mobile Caused a Global Mindset Shift in Game Development Analytics

I strongly believe that mobile is the culprit. The transactional data volume in modern mobile top grossing games greatly surpass many AAA (a term used for games with high production values, many of which surpass many Hollywood films) online titles. I still remember reading, somewhere in 2006, how EVE Online was leading the technological innovation in multiplayer games. A couple of years ago, in a talk with a tech lead from CCP Games, I found that I had launched a game that, back then, had almost four times more concurrent users at peak time than they had. I guess it is all about perspective.

Games started to become more and more complex in terms of design. Player economy ecosystems need to be maintained, and we no longer create games from a fire-and-forget point of view.

OK, But What Does That Mean Exactly?

Business models change. Many companies adopted the SaaS (Software as a Service) approach to GaaS (Games as a Service). Free-to-Play (F2P) is the trendiest kid on the block and we all want to sell our companies to Asian tech giants on a billion dollar evaluation.

According to research firm EEDAR, revenues from MOBA (Multiplayer Online Battle Arena, a revamped esports approach to the classic Real-Time Strategy genre) games have grown from little over a million dollars in 2010 to over $500M last year.

If we want to succeed in a F2P approach, besides having a good game you also need to know how to handle your player data. And if you do it right you only need to know one thing.

Knowing nothing is the key.

Keep It Simple: The Core KPIs

Yes, you heard it right, knowing nothing is the key to understanding your player demographics. Did you know that by assuming you know a lot about your games you are only paying Bay Area fees to top analytics solutions providers?

Granted, there are a few things we can start with as foundations for the whole learning process, so let’s take it from the beginning.

What are the key performance indicators (KPI) that we need to know when we launch a game?

• How many players installed our game (number of installs);
• How many times did our players actually play the game (number of sessions);
• How long did they play the game (session length);
• Did they play the game over multiple sessions (player retention);
• How much money is the game making (revenue).

You could probably argue that one or two KPIs are missing. No problem, just do whatever works best for you. In my experience these are the key indicators of any game’s performance upon launch. Forget anything else. Want to know why? Because you (and me) are just like Jon Snow when we launch a game. Provided that we know that we did our best in terms of design (not even talking about tech or art), the game should be successful, or at least we hope so.

I never met a person who launched a game deliberately knowing it would fail miserably out of sheer incompetence.

So What Do I Do with These KPIs?

After you launch the game you need to monitor these KPIs on a regular basis. No need to do it every hour—for that you have other instrumentation approaches that can greatly benefit operations and prevent some pitfalls as I mentioned in my previous article.

And then please give your product some time to get significant critical mass. There is no point in taking notes from the actions of ten people who “went left when you expected them to go right.” Having a significant sample of your core demographic is crucial. Otherwise you are making the same mistake all over: guessing.

Let me give you some examples, and keep in mind that I am talking post-launch and before the first update. Some games may have a huge install base but not many sessions and incredibly low session length when compared to the pre-launch estimates. This means that your game is not that good, or at the very least, that players don’t get it.

On the other hand, a game that has a modest install base, expected average session length, and high retention, now that is something you can work with. Theoretically you will only need to acquire more users to get those financial KPIs going up.

But what if you have a huge install base, moderate to high retention, and less than expected revenue? Congratulations: you created Candy Crush! I am (somewhat) joking but the thing is that many top-grossing games are not top performers in terms of monetization (i.e. the process by which a video game product returns money for those involved in its creation or copyright ownership). Why? Because they must spend huge amounts of money on user acquisition (i.e. the process of persuading a consumer to purchase a company’s goods or services).

These examples should give you a very rough idea of how and why keeping up with your KPIs, and knowing what counts, is not only a good policy but also a critical step when launching a game. Many game developers, especially stakeholders, rant a lot about poor revenue performance at start. I understand their worries, but their focus should not be on money alone. Instead, retention should be the one thing to worry about. Remember the movie Field of Dreams? “If you build it they will come” kind of thing? Well, here it’s more like, “If they keep playing it, money will come.” It’s not a 100% infallible rule, but, from my experience, top performing products are measured by retention. You can always get new users, provided you have the tools and enough money. But if those new users don’t like your game, it’s just money being thrown out the window. They will not stay long enough to convert to paying users.

That Sounds Pretty Simple. What Else Can I Do?

It is never simple, trust me!

If your game is over-performing and you have a new Ferrari, then I think it is fair to assume that you did a good job and you can take some time off, maybe two or three days. Otherwise, congratulations, you now have the job of overseeing your game’s performance.

And guess what? Now you can start asking questions!

Questions like “Why am I getting churn on day three?” or “Why are my sessions so short in length?” are good ones to start with because they are based in real data and not random guesswork about where you “think” you should improve your game. Keep in mind that I am pro-guesses and hunches, but only when there is data to support them. Data driven should be a constant buzzword in everyone’s analysis. Write it on a Post-It and put it on your standing desk or fancy ergonomic display unit. All your decisions should be data driven. All the time.

Oh, and why not complement analytics with your live log monitoring data that is lying out there in the cloud? That same data can be used as data points for all the questions you ask in your analytics system. In my experience they go hand-in-hand. Analysis should always be made with all relevant information you are able to gather. Logs may not give you the straight answers you want, but they can give you context for sure.

Finally, never take your player’s rants to the letter. People will always complain. I know a very popular game, which shall remain nameless, that illustrates this idea very well. One time a whole facelift was made to the original game. Players ranted a lot. Forums, social media, etc, were filled with complaints. Players even emailed death threats. What really happened? An increase in DAU and retention. Another time, the game’s shop had a massive visual overhaul. Same tough love from the players, all over again. What really happened? I will let you guess, but that shop was put to a much better use than before.

OK, I’m Ready. Now What?

Now comes the magic moment. If you think of junior high, when you learned all about the scientific method, you know that the next elementary step is to formulate an hypothesis.

Consider this example: A game has a decent retention rate up until day three. The question to ask is: “Why am I getting churn at day three?” Immediately after asking yourself that, you should probably look into what an average player is doing on day three, considering average session length and your game’s average progression rate. “Oh, on day three players should have reached level seven. What is happening on level seven? “Oh there’s this boss that is so OP [gamer lingo for overpowered]! Maybe I should do something about it…”

Then comes the hypothesis: “If I lower the boss’ stats, then players will overcome the challenge with less difficulty and maybe continue playing.”

Noticed the “maybe” part? This is all in your head and a product of your tacit knowledge. Nothing is a fact until you prove it. So how can we make it so? Many people tend to just make an update and wait for the results. This is wrong.

What if after the update, you had some undetected server issues and a lot more players churned than usual? Then your “solution,” even if it was the best approach to the original problem, would not deliver the desired results as the churn rate could be greater and you would be back to the drawing board.

The only way to actually prove that you are right is by means of A/B Testing, again a by-product of the scientific method. You create two different test groups for new users: A and B, and show different variations to each one. The control group should stay untouched, meaning that new users will have the same layouts or stats as every other user prior to the test, while the experimental group will have your hypothesis implemented. By doing so you can compare the KPI outcome of both groups. (Remember that your hypothesis should be measured by an already-existing KPI—in this case, retention.)

Got It! But… Will It Ever End?

No! This is the loop you will always have to endure throughout your product’s life. Pretty much like in the Lean method: Build -> Measure -> Learn.

If you look at the jobs page of a big publisher like Riot, you will find that there are probably more openings on Data Science and Analytics than in Engineering. This is mainly because as you scale your operation, you need to look at what your players are doing and, where the real money is, predict what is to come solely based on your data. This is called predictive analytics and involves a lot of different areas, like data mining and machine learning. It’s the oh-so-hot kind of offer in analytics.

And from an investment viewpoint, even if you are not making much money, if you are able to retain your users and grow your ecosystem, you are much sexier than a one-hit-wonder. You wouldn’t believe how many deals are made with companies just because some other entity will get their hands on third-party data. Data is indeed power.

To sum up, by actively monitoring and reacting to your product’s data, you are able to make decisions based on real data.Your users should validate your decisions, not your ego.

So have fun, and I will look at this post’s metrics to see if you actually had any fun!

---

Read More: Confessions of a Logging Fanboy »

 

The Loggly and SolarWinds trademarks, service marks, and logos are the exclusive property of SolarWinds Worldwide, LLC or its affiliates. All other trademarks are the property of their respective owners.