A/B Testing HTML5 Games

Building games requires optimizing those games for engagement, enjoyment, and retention. You want users to play games longer and more often.

However, knowing how to build a game with the best engagement metrics is challenging, and it's rare that any game developer will get it right the first time.

I've written this tutorial with that in mind - with a system for how to properly a/b test features within Web-based games.

When building my first site, FreeCell Challenge, I wanted to figure out how to encourage users to play another round of games. After winning a game, I present users with a “congratulations modal,” that tells them they won. In that modal, I have a button, “Play next game.”

What I wanted to know was, should the next game be a game that I knew was winnable (thus building confidence in the user, encouraging them to play yet another game), or should the deck shuffle be random (which seems to be the preference of the more advanced players)?

My A/B test was simple. The “Control” cell was the current format - the button led to a new randomly-shuffled game. The Experimental cell would change the button's link to go to a guaranteed-winnable game.

Thus, the variable that I changed was, what “shuffle type" should I send the user to. For your games, that flag could be something else - what level to send a user to, how hard the big boss is on the second level, how many enemies are there, etc.

I also didn't want to (ahem) pay for costly software or infrastructure.

I decided to choose Google Optimize. Google Optimize works hand in hand with Google Analytics, thus providing free data collection and a nice UI to boot.

However, the default use case for Google Optimize is “client-side” - the scripts modify the front-end of the website. I didn't want to do this, because my game variables were generated server-side, and I didn't want to have Google Optimize show an unsightly “flash” as it changed the variables on the fly on the frontend.

To make this work, you need the following:

1. An A/B testing library
2. Google Optimize and Analytics
3. A “flag” sent to your Google Analytics account that identifies which bucket a user is in.

A/B testing library

We use node & Javascript on our site, so I made an extremely simple library (adapt to the language of your choice):

exports.assignToTest = (req, testName) => {
  //if no tests are set, or different test is set, assign to current test.
  if (!req.session.test || req.session.test.name !== testName) {
    req.session.test = {};
    req.session.test.name = testName;

    //gives a 10% split
    let bucket = Date.now().toString().slice(-1);
    if (bucket <= 1) {
      req.session.test.run = true;
      if (bucket  == 0) {
        req.session.test.bucket = 0;
      } else if (bucket == 1) {
        req.session.test.bucket = 1;
      }
    } else {
      req.session.test.run = false;
    }
  }
  return req.session.test;
};

This middleware does the following: Get the last digit of the current time. If it's a zero, assign it to the control bucket (bucket 0). If it's a 1, give it to bucket 1. Otherwise, ignore this user. This gives me two buckets, each at 10% of our visitors, and then leaves the 80% alone. Why not just do 50/50? Well, we didn't want to expose half of our users to the new experiment; but you certainly could modify the code above to do that.

Google Optimize and Analytics

Sign up to Google Analytics and Google Optimize.

Create a new experiment as described here. Make sure to copy the Experiment ID.

Send site data to Google Optimize

On your site, push the user to the A/B testing bucket if they are in one:

  <script>
    let abTest = !{JSON.stringify(abTest)}; //inject your server-side a/b test code
    if (abTest.name && abTest.run) {
      gtagPayload['experiments'] = [ { id: abTest.name, variant: abTest.bucket } ];
    }
  </script>
  <script async src="https://www.googletagmanager.com/gtag/js?id=UA-129059877-7">
​  <script>
    window.dataLayer = window.dataLayer || [];
    function gtag(){dataLayer.push(arguments);}
    gtag('js', new Date());
    gtag('config', 'UA-129059877-7', gtagPayload);
  </script>

For the abTest variable, you must pass this from your server. The code above assumes the session.test variable is sent into the views as “abTest."

Once you have this done, deploy your code, and “Start” the Optimize experiment. You should begin to see data being collected, and in a few days you'll have enough metrics to make a proper decision.

So what did we learn in our case? It turned out that users that played random games played 15% more games than the other cell - it was counterintuitive to us, but that's why we ran the test. Happy testing!