How do you decide whether a metric should change your game design?

All good questions with no good answer.

You need good information to make informed decisions. Then you need more good data to understand the effect of those decisions. Repeat again and again as you iterate. What works for one may not work for another, and you're unlikely to ever be able to paint a complete picture of what the current customers and the potential customers all think. They're also subject to culture and tastes that are in constant flux.

Thanks, that makes sense.

I like the way you framed it as iteration rather than one perfect metric. For small teams, I guess the hard part is not only collecting data, but deciding which signal is strong enough to act on.

For example, if wishlists, retention, or ad revenue changes, the first question is probably whether the change is large enough and consistent enough to mean something, or just noise.

Do you think small teams should focus on a few repeated review habits, like weekly trend checks, instead of trying to build a full analytics process?

I think small teams should focus on finishing anything, then iterating.

I also think this advice still applies , that was a set of articles from 25 years ago formerly hosted on the site. Also this and this fit your questions. The author of those articles, Steve Pavlina, went by the name "Dexterity" on the forums, founding Dexterity Games, and IIRC for a while was president of the Association of Shareware Professionals.

Thanks, that is a useful way to think about it.

I like the idea of finishing one piece of work, learning from the result, and then iterating, instead of trying to design a perfect analytics process upfront.

I’ll read through those articles. Older practical advice often ages better than trend-based advice, especially for small teams.

ecpm_insights wrote:

tutorial drop-off

I'd work on the tutorial somehow. Tutorials are not gameplay, so the risks of making a wrong change here, are a lot lower. That's a triage you could generally do with all these concerns. Is the issue core to the game?

That distinction makes sense. If the problem is in the tutorial, the risk of changing it is lower because it is mostly about helping players reach the real game, not changing the core loop itself.

I like the “is this core to the game?” framing. It makes prioritization clearer: fix low-risk access/friction problems first, then be more careful with changes that affect the actual gameplay promise.

For tutorial drop-off, I’d probably look for the first point where players stop understanding what to do, rather than redesigning the whole early game.

ecpm_insights wrote:

For tutorial drop-off, I’d probably look for the first point where players stop understanding what to do, rather than redesigning the whole early game.

I agree that the design of the game is an option, but not the 1st thing you'd consider doing. I do not see a tutorial in terms of "whether they understand". I see it in terms of whether they get bored. People typically want to get right into a game without a lot of studying and waiting. Making the tutorial more engaging, more like playing a game, less like learning something, might be a way to go. An increase in attention to its content quality. That said, the clarity of a tutorial could be a problem. But finally, there may be some absolute amount of time that most people will spend, after which you're toast.

So: clarity, content quality, length.

That’s a good distinction. “Not understanding” is only one possible cause; boredom or waiting too long before the real game starts may be just as important.

I like the three-part framing: clarity, content quality, and length.

For tutorial drop-off, I’d probably try to identify whether players are leaving because they are confused, because the tutorial is not engaging enough, or because it takes too long to reach the core loop. Those would lead to very different fixes.

Ah what a loaded and thought-provoking question. In my opinion I wouldn't jump the gun one way or the other unless if have a truly solid set of data to go on. So many factors to consider - some of which could very well be time sensitive and may require you wait it out. For example, setting up a metric based on metrics during the academic school year will be significantly different from summer months. On the point of 'not understanding', i suppose that's why it is critical to have scheduled and predictable gaming test sessions to get actual opinions from from people not directly involved in your project that way there is no bias.

That makes sense. The seasonality point is easy to underestimate, especially if the audience changes between school months and summer.

I also agree that tutorial drop-off should not be judged from numbers alone. Metrics can show where players leave, but outside playtesters can explain why: confusion, boredom, pacing, or something else.

So the safer approach is probably: use metrics to find the suspicious point, then use playtesting to understand the reason before changing the core design.