Optimizing Your Testing/CRO Program

Tweet about this on TwitterShare on FacebookShare on LinkedInGoogle+Email to someone

This year I went to the Conversion Conference in Las Vegas where I saw many amazing speakers, one of which was Clair Vo from experimentengine.com who spoke about how to determine if your testing program is successful and what metrics to look at. For those who were unable to attend I will discuss some of the main points.

Success is driven by testing inputs not outputs.

Most teams would determine success of a program by testing outputs, that is to say the results of testing. Did a test increase conversion rates? Yes = success; No = failure. This however is a very limited evaluation because we can learn as much, if not more about our audience and what they respond to with results from tests that don’t increase conversion rates. The key is asking the question “why did the test fail?”

image1-23

A better approach is to determine the success of a program based on the testing inputs, for example: testing efficiency, quality and profitability. So we can improve the success of a program by potentially influencing three areas:

  1. Increase the number of tests run
  2. Increase the number of effective tests (e.g. tests with insights/learnings)
  3. Increase the profitability of tests

The key here is tracking success and improving these metrics over time, not just based on the last test that was concluded. There will always be fluctuation in how long a test will take to conclude based on what is being tested as well as the conversion improvement from those tests but we can look year-over-year or quarter-over-quarter to see how the program is improving as a whole.

1. Increase the number of tests run

Metrics for this can be determined by asking the following questions:

  • How many tests can we run?
  • How many are we actually running?
  • If we are not running as many as we can, why?
  • What percentage of days are we running?
  • What percentage of traffic are we using to test?

2. Increase the number of effective tests

Metrics for this can be determined by asking the following questions:

  • Are we running effective tests (tests that have the potential to really improve results)?
  • Are we effectively running tests (testing the right concepts on the right pages/audience)?
  • Are we getting better at running tests (increasing win rate and average lift over time)?

3. Increase the profitability of tests

Metrics for this can be determined by asking the following questions:

  • Are we accurately measuring the value of a lead, order, etc.?
  • Are we improving that value by increasing the volume and/or quality of the lead, order, etc.?
  • Is the value of the leads, orders etc. higher than the cost of the testing and implementation?

Comments are closed.