At Kijiji, we’ve been using Optimizely extensively for traditional A/B testing, building different UI variations and measuring which one performs better.Recently, we started using this tool in a different way, in order to gather qualitative feedback. The idea is extremely simple: ask directly to our users what they think of something on our site, or something that we could be adding to it.

Here is an example. We wanted to know whether our users would appreciate receiving real-time feedback about the quality of their ad, as they type it. We could have sent out a survey by email, or include this question into our user testing sessions. But what better way than asking people directly, precisely where and when it matters to them?

Title Quality Satisfaction Survey

All we had to do was build a little tooltip with our question and measure the interest with custom events.

We wanted to use smileys for users to click and share their satisfaction (happy, neutral or angry) but Optimizely doesn’t currently support uploading static assets. An easy work-around was to encode our pictures into base64 and insert them with javascript.

Here are our 3 little icons:

var angry = "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABgAAAAYCAYAAADgdz34AAAABmJLR0QA/wD/AP+gvaeTAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAB3RJTUUH4AMDBAUtsCRk8AAAA45JREFUSMe9VktsG1UUPfP85mN7Jo4zjFynLiU0rZxYpCqEgFAAkaJ2wwKJRRewYIFUiMSKtiuyouzKCqQmUoS6Q0gVYgEsIqSIECGIiJBCnVhq68ROmjZMHMeZaWb8xn7TTWM5ieNgqXCXM+eec+7cufc9oEkYktST0tTR7nAoq4sio4LgU0HwdVFk3eFQNqWpo4Yk9TTjEA4i1iXxxqpbHng7HsNgRxRJTYUhSQAAkzFkLBvTG0V8f38NnYo8U2De+yZjC4cK9Grqp7fth599/Owz+OD4MUiENDMIxjnGc8v4MruEk2p4ZN6yrx4okNLUUafKL15LJdEXaUMrMVfawqV0BsEAGUtb9oc7zwP1zt0qv3zj+T50q2G0GjFFxllDx83VB/1xRa6ajE3VKjAkqWfT8+a/7T/TsvNGlVz48y+0i2KvydhCAACeDio/vnfs6NG3jsRqQM45fN+HIAhNCSucg9RhYooMIgi457ovrjNvXDAkqafM+fzvr71Sa+gW83DX2QYAnAiG0CaJDYmXHRd5x8EpNYy4ouxq/MtTv0EmpDfQFQpeHTL0/iHjqZrzCbOAa3eymFwvQJcknAyH9lWS23bw+e07+GblPircx0A0UqskIAhYK5exValQUub83GBHtE7dxy3LQt5xkHcc3LIsMO7vc593HKS3bBQYw2ypBJfzXZjBjijKnJ+jReYlkppae6HQAM4bBv4plwEA5w0DCg3sSqaE4JQaxjvxI5gtlfBuohOhPfOS1FQUmZcQqCD4f7/xKsQ9ALdSrQk2a7DLOUKEgOzJ9zjHc5O/gh6U3Iy4vhL1kEknEUo9kzE86TAZQ4RSj0QlcSVj2U9cIGPZiEriCpUJmZjeKF7c+U33RpF5+KVQwM9mAbObJWx4HjpEES+0R/CmoeN1XUe0wZxMbxQhEzLRcNB24vpiDl/cXTzU7ScnuvBR1/GGg0ZNxhaSanhmPLc8MFwHur6Yw+R6AV+f6UNKUxGhFLTOQIVzlCoVpC0bX2WXAKAmMp5bRqciz2TshwsHLrs/ipt4Kdr+r7/5Dr7hstuuVteTmlr9ac0cOmvo0ChFIqi01NREUMGq62J4Lo2YIo/kHOe7XeeBydhUpyLHb64+6D/dpiGmyC2v6eG5NIIBMjZv2Vf2HTiPRX6IK3J1bCk/RAQBpyNtCByyrhnnGFvK43I6g5gij9ST/y+HPv7ra8sjaKyrgYZAl9UAAAAASUVORK5CYII=";
var happy = "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABgAAAAYCAYAAADgdz34AAAABmJLR0QA/wD/AP+gvaeTAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAB3RJTUUH4AMDBAYaI7SSPAAAA0FJREFUSMe9VkFsG0UUfTPzZxw7i7dO4jhtamRFosjlEok2N5DIoYeeeuDADTjRKwcqDviAWimcCr0FiQO9VUJEIMGlQo0UVRwqIUVCag4RVtQWhSQL9obdrD27s8PBrRsntiML6Lvun//+n/f+nwWGQBREVc2pZVmWde5yDQELActdrmVZ1tWcWhYFUR2Wgw1KzF3+tdkzC+NvjSM7n4WqKIiCAACYhoHe0ojWI4SrIURRPEj99D3TMBsnEqg59Yl+pK+feucU3CsumGTDCoSNLfzvfDTvNKFeVjVd1zcGEqg5tWzb9oOpD6eQeSWDUdDebMP73APLsC91XV/t3sbhym3bflT6tARVVhgVNEnILeQQ/BRcoCIZ0zBr3Q5EQVTN3+bh6c9Oj1x5v062P96GeEmcNw2zIQCAZujH/OX8rPOGg38LmiSAA8lucjFtpl8xURBVq+3D8u1yj6DW2I5I4gSRjT0WY2OLx+8+BlPsvJBn5I3sQvZC7mKuG5DsJgjvh9C/aQhXgI/zvsnj32NEv0RgikHkxXPnCIbkrwRpkBK3sb2Unc/2VNTebCO4FyC4F6C92e5201NlYhGsBWh+00SwFsAmvTHZ+SxsbC+R2TdnVeWQazhAM4TMuY7YNNO501GhKgpm35ylNEjlswkFAMYYZFnCveJ2+FwOxo7rwIjBedOBLElkXs2AETu6DZAGqaR+7FxxsCnWJRzomDMEKtGhaeqTizs8Ng1zvELGhibvxlD/ONMw4A6PuciLJ3pL47+G3tIQefGEM8nuRutR78dHoxMePROtR2CS3eWmYW6FqyFs/Nxm3k0PR0mHIVqP4N30egYtXA1hGuaWsC3r0SxdTg/S2bHXxgAAsiyxU9tB+HMImiZwh3fumrPurKRBitavLewu7WL/+30UrxVB0x3P+Cs+kj+TB8kfydLAZdfaaGGnttPTWV+hJUPpeglj1bHBy862rKcqyhzcP1jMLeTAxzmoSHDfdpF7PQeRF7DaIg1TMMGgKgrOooOJ9ycwcXUCclp2Vsxegr2lPdAk1ZLtZOXFPjhPvfsDFcn43/qL4EDmXObkbRpb+Cs+vC880CTVdF1fe6GPPv7v35Z/ABSWjcljcLndAAAAAElFTkSuQmCC";
var neutral = "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABgAAAAYCAYAAADgdz34AAAABmJLR0QA/wD/AP+gvaeTAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAB3RJTUUH4AMDBAQQ8VcZoAAAA05JREFUSMe9Vl9oU1cc/u655yQ3aarJRZPWZKtV0aWNkZUueymlyNb5NIYi7mGg+Afrg+xNECKDrS+DDTrr0MIeNtjDhmwPY4PRp1I2xsKoGNNGpdKtJpp7vU0j/ZOTe0/u9UEWSdMGdHUfXO7L7/d93zn3O797JDRBKEijbSH6Ia84g8WiiCyWqgwAAn7ZUlWaU9zSeEETX2i6yG7EIW1EvG2b/HUubyXeP+rHQH8LYt0KQkEKANB0gcw0x8TkCr67XkIkzFKGUT3RTKiGeExJMgYneXG7o81HncVCV9NHm486yYvbHcbgxGNKsukKDuxXrq2W7bPXRsPoed2D58HUjTKGzufh9ZCxm7f4UINAPKYky9z+5MfvOxAJM7wIcnkLh4/9A49CLqUzfLgmEArSaHFRzPz6U+dzO19vJYfenYMaoF2aLrIyAOzsYL+cPqmGj7y3ta5YCBuESE0J19a0tzMQGZi/b73x6FH1KykUpFFesWfu3toHl+tZ4f28iXSaIx5X8ErYtS55/oGJ7O0KOne60PEqA6UEAGCaDvbuvwPFTbrkPbtcw4cGW3vfebu11miaNj76WMfnIwYWFqp462ALZFlqcH75ywV8+pmB6RmO3h4v1MDTGMuyhIImUCpVKeEVZ3Cgv6WumXMbv/+xgqXlp2/O7Qb3nAN/plZhLAjcTFcw97cJIZ7VDfS3gFecQVosikisW6lr9vkIhs6o+ObbEo5/4IfPRxoEvF7g1AkVQBFvJryIvuaubREAxLoVFIsiIlEK58FcFIzVb4Ft21hetuHzERBC1v0Gtm1jdRVQFNSRA4BlOdjRmQXdKB2EEGzZQpomiBACn695bEnAL1uaLrDZ0HSBgF+2iKrSXGaab7pAZppDVWmOKG5pfGJyZdMFJiZXoLil8Q0P2sOChes/PMZfU2Xcu2eioAssLVVhWQBjQGurjLYgxe7dLvT2eHD0yFa0t7GGg0Y0XWQjYZYavWpsmvvRqwYiYZbSdJF96cOuYVynftvznwQSfbN147oW9HSGD3s9ZCzRN4upG+UXcp7om4XXQ8b+JQcAeU12f47sYNWRK8ZBIgO9Pd6GIbcWpulg5IqBc+fzaG9jl9IZfuF//elLL/va8gSgW6qo2uTKKAAAAABJRU5ErkJggg==";

Using these icons to track votes is done by adding an onclick binding that records the custom event, like this:

window.satisfactionVote = function(e) {
  e.preventDefault();
  var eventName = 'titleQuality_' + e.target.id;
  window.optimizely.push(["trackEvent", eventName]);
};
var happyIcon = "<img id='happy' onclick='satisfactionVote(event)' style='cursor:pointer;' src='" + happy + "'/>";

While we’re not looking for scientific precision, it’s probably better to get each person’s vote only once. Optimizely shows results for unique visitors by default, but another consideration is to provide the best possible user experience and get the survey out of the way as soon as a user replied.

In order to do that, we can simply store a flag in the browser’s local storage. When a user votes, we set the flag to 1 and hide the survey. Afterward, we don’t show the survey again if the flag is already set.

Putting everything together, the code of our Optimizely variation looks something like this:

window.satisfactionVote = function(e) {
  e.preventDefault();
  var eventName = 'titleQuality_' + e.target.id;
  window.optimizely.push(["trackEvent", eventName]);
  localStorage.setItem('titleQuality', 1);
  // hide the survey
  document.getElementById("satisfaction").style.display = "none";
};

function buildSurvey() {
  var angry = "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABgAAAAYCAYAAADgdz34AAAABmJLR0QA/wD/AP+gvaeTAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAB3RJTUUH4AMDBAUtsCRk8AAAA45JREFUSMe9VktsG1UUPfP85mN7Jo4zjFynLiU0rZxYpCqEgFAAkaJ2wwKJRRewYIFUiMSKtiuyouzKCqQmUoS6Q0gVYgEsIqSIECGIiJBCnVhq68ROmjZMHMeZaWb8xn7TTWM5ieNgqXCXM+eec+7cufc9oEkYktST0tTR7nAoq4sio4LgU0HwdVFk3eFQNqWpo4Yk9TTjEA4i1iXxxqpbHng7HsNgRxRJTYUhSQAAkzFkLBvTG0V8f38NnYo8U2De+yZjC4cK9Grqp7fth599/Owz+OD4MUiENDMIxjnGc8v4MruEk2p4ZN6yrx4okNLUUafKL15LJdEXaUMrMVfawqV0BsEAGUtb9oc7zwP1zt0qv3zj+T50q2G0GjFFxllDx83VB/1xRa6ajE3VKjAkqWfT8+a/7T/TsvNGlVz48y+0i2KvydhCAACeDio/vnfs6NG3jsRqQM45fN+HIAhNCSucg9RhYooMIgi457ovrjNvXDAkqafM+fzvr71Sa+gW83DX2QYAnAiG0CaJDYmXHRd5x8EpNYy4ouxq/MtTv0EmpDfQFQpeHTL0/iHjqZrzCbOAa3eymFwvQJcknAyH9lWS23bw+e07+GblPircx0A0UqskIAhYK5exValQUub83GBHtE7dxy3LQt5xkHcc3LIsMO7vc593HKS3bBQYw2ypBJfzXZjBjijKnJ+jReYlkppae6HQAM4bBv4plwEA5w0DCg3sSqaE4JQaxjvxI5gtlfBuohOhPfOS1FQUmZcQqCD4f7/xKsQ9ALdSrQk2a7DLOUKEgOzJ9zjHc5O/gh6U3Iy4vhL1kEknEUo9kzE86TAZQ4RSj0QlcSVj2U9cIGPZiEriCpUJmZjeKF7c+U33RpF5+KVQwM9mAbObJWx4HjpEES+0R/CmoeN1XUe0wZxMbxQhEzLRcNB24vpiDl/cXTzU7ScnuvBR1/GGg0ZNxhaSanhmPLc8MFwHur6Yw+R6AV+f6UNKUxGhFLTOQIVzlCoVpC0bX2WXAKAmMp5bRqciz2TshwsHLrs/ipt4Kdr+r7/5Dr7hstuuVteTmlr9ac0cOmvo0ChFIqi01NREUMGq62J4Lo2YIo/kHOe7XeeBydhUpyLHb64+6D/dpiGmyC2v6eG5NIIBMjZv2Vf2HTiPRX6IK3J1bCk/RAQBpyNtCByyrhnnGFvK43I6g5gij9ST/y+HPv7ra8sjaKyrgYZAl9UAAAAASUVORK5CYII=";
  var happy = "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABgAAAAYCAYAAADgdz34AAAABmJLR0QA/wD/AP+gvaeTAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAB3RJTUUH4AMDBAYaI7SSPAAAA0FJREFUSMe9VkFsG0UUfTPzZxw7i7dO4jhtamRFosjlEok2N5DIoYeeeuDADTjRKwcqDviAWimcCr0FiQO9VUJEIMGlQo0UVRwqIUVCag4RVtQWhSQL9obdrD27s8PBrRsntiML6Lvun//+n/f+nwWGQBREVc2pZVmWde5yDQELActdrmVZ1tWcWhYFUR2Wgw1KzF3+tdkzC+NvjSM7n4WqKIiCAACYhoHe0ojWI4SrIURRPEj99D3TMBsnEqg59Yl+pK+feucU3CsumGTDCoSNLfzvfDTvNKFeVjVd1zcGEqg5tWzb9oOpD6eQeSWDUdDebMP73APLsC91XV/t3sbhym3bflT6tARVVhgVNEnILeQQ/BRcoCIZ0zBr3Q5EQVTN3+bh6c9Oj1x5v062P96GeEmcNw2zIQCAZujH/OX8rPOGg38LmiSAA8lucjFtpl8xURBVq+3D8u1yj6DW2I5I4gSRjT0WY2OLx+8+BlPsvJBn5I3sQvZC7mKuG5DsJgjvh9C/aQhXgI/zvsnj32NEv0RgikHkxXPnCIbkrwRpkBK3sb2Unc/2VNTebCO4FyC4F6C92e5201NlYhGsBWh+00SwFsAmvTHZ+SxsbC+R2TdnVeWQazhAM4TMuY7YNNO501GhKgpm35ylNEjlswkFAMYYZFnCveJ2+FwOxo7rwIjBedOBLElkXs2AETu6DZAGqaR+7FxxsCnWJRzomDMEKtGhaeqTizs8Ng1zvELGhibvxlD/ONMw4A6PuciLJ3pL47+G3tIQefGEM8nuRutR78dHoxMePROtR2CS3eWmYW6FqyFs/Nxm3k0PR0mHIVqP4N30egYtXA1hGuaWsC3r0SxdTg/S2bHXxgAAsiyxU9tB+HMImiZwh3fumrPurKRBitavLewu7WL/+30UrxVB0x3P+Cs+kj+TB8kfydLAZdfaaGGnttPTWV+hJUPpeglj1bHBy862rKcqyhzcP1jMLeTAxzmoSHDfdpF7PQeRF7DaIg1TMMGgKgrOooOJ9ycwcXUCclp2Vsxegr2lPdAk1ZLtZOXFPjhPvfsDFcn43/qL4EDmXObkbRpb+Cs+vC880CTVdF1fe6GPPv7v35Z/ABSWjcljcLndAAAAAElFTkSuQmCC";
  var neutral = "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABgAAAAYCAYAAADgdz34AAAABmJLR0QA/wD/AP+gvaeTAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAB3RJTUUH4AMDBAQQ8VcZoAAAA05JREFUSMe9Vl9oU1cc/u655yQ3aarJRZPWZKtV0aWNkZUueymlyNb5NIYi7mGg+Afrg+xNECKDrS+DDTrr0MIeNtjDhmwPY4PRp1I2xsKoGNNGpdKtJpp7vU0j/ZOTe0/u9UEWSdMGdHUfXO7L7/d93zn3O797JDRBKEijbSH6Ia84g8WiiCyWqgwAAn7ZUlWaU9zSeEETX2i6yG7EIW1EvG2b/HUubyXeP+rHQH8LYt0KQkEKANB0gcw0x8TkCr67XkIkzFKGUT3RTKiGeExJMgYneXG7o81HncVCV9NHm486yYvbHcbgxGNKsukKDuxXrq2W7bPXRsPoed2D58HUjTKGzufh9ZCxm7f4UINAPKYky9z+5MfvOxAJM7wIcnkLh4/9A49CLqUzfLgmEArSaHFRzPz6U+dzO19vJYfenYMaoF2aLrIyAOzsYL+cPqmGj7y3ta5YCBuESE0J19a0tzMQGZi/b73x6FH1KykUpFFesWfu3toHl+tZ4f28iXSaIx5X8ErYtS55/oGJ7O0KOne60PEqA6UEAGCaDvbuvwPFTbrkPbtcw4cGW3vfebu11miaNj76WMfnIwYWFqp462ALZFlqcH75ywV8+pmB6RmO3h4v1MDTGMuyhIImUCpVKeEVZ3Cgv6WumXMbv/+xgqXlp2/O7Qb3nAN/plZhLAjcTFcw97cJIZ7VDfS3gFecQVosikisW6lr9vkIhs6o+ObbEo5/4IfPRxoEvF7g1AkVQBFvJryIvuaubREAxLoVFIsiIlEK58FcFIzVb4Ft21hetuHzERBC1v0Gtm1jdRVQFNSRA4BlOdjRmQXdKB2EEGzZQpomiBACn695bEnAL1uaLrDZ0HSBgF+2iKrSXGaab7pAZppDVWmOKG5pfGJyZdMFJiZXoLil8Q0P2sOChes/PMZfU2Xcu2eioAssLVVhWQBjQGurjLYgxe7dLvT2eHD0yFa0t7GGg0Y0XWQjYZYavWpsmvvRqwYiYZbSdJF96cOuYVynftvznwQSfbN147oW9HSGD3s9ZCzRN4upG+UXcp7om4XXQ8b+JQcAeU12f47sYNWRK8ZBIgO9Pd6GIbcWpulg5IqBc+fzaG9jl9IZfuF//elLL/va8gSgW6qo2uTKKAAAAABJRU5ErkJggg==";

  var survey = "<div id='satisfaction'>";
  survey += "<strong>We're working on providing feedback on the quality of your title.</strong><br/><br/>";
  survey += "Would you like this?";
  survey += "<img id='happy' onclick='satisfactionVote(event)' style='cursor:pointer;' src='" + happy + "'/>";
  survey += "<img id='neutral' onclick='satisfactionVote(event)' style='cursor:pointer;' src='" + neutral + "'/>";
  survey += "<img id='angry' onclick='satisfactionVote(event)' style='cursor:pointer;' src='" + angry + "'/>";
  survey += "</div>";
  return survey;
}

function initTitleQuality() {
  if(typeof(Storage) === "undefined") {
    // local storage unsupported, stop there
    return;
  }
  if (localStorage.getItem('titleQuality')) {
    // user already voted
    return;
  }
  // insert the survey on the page
  $('#title-field').before(buildSurvey());
}

initTitleQuality();

After this, we just have to run the experiment and the results look something like that:

Title Quality Satisfaction Survey

We could stop there, but leaving some clutter in our users browser doesn’t seem like a good habit. So, when we’re done with the experiment, we can reuse it to clean-up after ourselves and remove the flag from the local storage. Since any returning user who had voted will still be targeted, we can just change the code to remove the flag and keep the experiment running for a few more days. We can even track the progress of this clean-up process with an additional goal:

if (localStorage.getItem('titleQuality')) {
  localStorage.removeItem('titleQuality');
  window.optimizely.push(["trackEvent", 'titleQuality_remove']);
}

Happy discovery!