This year I had the pleasure of speaking at the annual WhichTestWon The Live Event (TLE). It was a great event where I learned a lot of new ideas and some key refreshers. Here are some of the key takeaways that I came away with from the event and can also help you as well if you were unable to attend this year.
This is the idea that you run a test of the exact same page against itself. Why would you do this you may ask?
As testers we always want statistically relevant results and want hard numbers to prove it. While this approach has its obvious merits and should not be ignored, it doesn’t need to exclude qualitative data such as usability tests, surveys, heatmap data, and more. This qualitative is often a faster and more efficient way to get valuable information for improving your visitor’s experience. Once you have the information you can then take that information and test as necessary to prove/disprove the value of the change. Its more important to know what to test than running the test itself and that’s where Qualitative data can really help
By not having a default option but still making the form required, you can make the visitor think about the related question before answering. It also will be less irritating for visitors on an opt-in field for example when you don’t presume to know what they will want to check, and check it for them.
We can sometimes fall into a trap of testing the latest idea that has come up when a previous test concludes. Rather, create and nurture a list of ideas to pull from and then strategically order those in relation to your goals and needs to have the biggest impact and build upon previous tests.
A test that you ran a year ago or even more recently may no longer be true for your visitors today. The way visitors interact and expect your site to look, along with the content that will resonate with them is constantly changing and evolving. Retest your old tests periodically to ensure that your visitor’s experience remains relevant. The frequency of which you can re-test old tests will depend on your unique situation and length that tests run/conclude. Some of the largest companies stated that they re-test on a monthly basis.
Ensure that your testing conclusions look not just at the immediate conversion but also how it is affecting conversions further down the funnel. Also, consider looking at multiple factors when determining the success of a test and weigh all of them in your final decision. For example, not just a shopping cart purchase but also, amount of purchase, repeat purchases, customer satisfaction, number of returned/refunded products, etc.
Executives often don’t care if you got a 50% lift in conversions, what they care about is what was their lift in revenue, profit etc. Even if you need to estimate the numbers, do so conservatively and with consensus as starting point. This will help show the value and importance of your results and get buy-in to continue and grow your testing program.