A/B Testing Your Game with PubNub and Optimizely

A/B testing is an important tool for modern game designers and product managers.  Conducting A/B tests provide a robust framework to facilitate experimentation with many aspects of the game in a controlled way and to confidently make decisions about positive changes to the game’s design.  There are many services that offer A/B Testing like Optimizely, Google Analytics, and open source.

In today’s tutorial, we are going to take a look at how easy it was to integrate Optimizely A/B testing into our Multiplayer Ninja Platformer PubNub game.

Integration

The goal of this experiment was to see which level in the multiplayer game provided the highest amount of engagement.  Additionally, we wanted to test how easy each level was to complete by taking the percentage of people who actually went through the door to move on to the next stage.  In order to measure this information, we need to setup three different variation variables in the Optimizely portal, and an additional completedLevel metric which tracks when the player enters the door and changes scenes.

Below are some examples of what or A/B test looks like in the Optimizely portal.  Each variation is divided into even increments of 33.3% where accuracy improves as more users visit the webpage.

The first step to implement the Optimizely SDK is to simply require it in your index.html document:

<script type="text/javascript" src="https://cdn.optimizely.com/optimizely-2.1.2.min.js"></script>

Next, grab your Optimizely ID from the portal, and place it as the optimizelyID variable.  The following request grabs your Optimizely data file which downloads your settings and A/B testing variables.

var optimizelyID = 'YOUR-ID-HERE' // ID of full stack project
// Function to request JSON datafile to initalize Optimizely SDK
var getDatafile = () => {
  return new Promise( (resolve, reject) => {
    var request = new XMLHttpRequest();
    request.open('GET', `https://cdn.optimizely.com/json/${optimizelyID}.json`);
    request.onload = () => {
      if (request.status >= 200 && request.status < 400) {
        var resp = request.responseText;
        resolve(resp);
      } else {
        reject('Unable to capture datafile');
      }
    };
    request.onerror = function() {
      reject('Unable to capture datafile');
    };
    request.send();
  });
}

Once you have received the data file, you can initialize the Optimizely A/B test.  In this code, we check to see if Optimizely is enabled, make the current level either 0,1,2 depending upon the A/B testing probabilities.

// Initializing Optimizely SDK & resetting default level based on bucket
getDatafile().then((datafile) => {
  // Initalize the client after getting the datafile
  window.optlyClient = window.optimizelyClient.createInstance({datafile: datafile});
  // Bucket user and send impression to Optimizely
  var enabled = window.optlyClient.isFeatureEnabled('ninjaMultiplayer', window.UniqueID);
  // set a new level to start at 
  window.startingLevel = window.optlyClient.getFeatureVariableInteger('ninjaMultiplayer', 'startingLevel', window.UniqueID);
  if(enabled){
    window.globalCurrentLevel = startingLevel;
  };
});

Additionally, we can track when a user completes a level by inserting this code when the player enters the door, which runs the _goToNextLevel() function.

window.optlyClient.track('completedLevel', window.UniqueID);

Once the code is implemented and pushed live, you can simply change user interactions through the Optimizely portal.

Conclusion

Optimizely is a well integrated solution for A/B testing that implements well with PubNub’s Realtime API’s.  If you want to learn more about how you can use PubNub to create this realtime game yourself, check out this tutorial.