When it comes to testing you can think of any test falling into two categories or “buckets”. There are what we will call innovative testing and iterative testing.
We call this testing innovative because it is focused on the big picture and often involves radically different designs. It involves testing large themes like overall page layout, design, functionality etc.
Generally these types of tests are run as an A/B split tests with version A being your control version and version B being the new overall page you want to test.
There are many changes that we are making to the landing page all at once in order to try and come up with the best page that we can overall. With this type of testing, we are not interested in how each part of the page impacts the overall conversion rate; we simply want to improve the overall experience and thus increase the overall conversion rate.
We call this testing iterative because it is focused on the individual elements of a page and multiple versions (iterations) of those elements. It involves testing elements such as call to action titles, heroshots, form buttons, form size, body text, etc.
These types of tests can be run as an A/B split tests or Multivariate tests (MVT). Usually this decision comes down to the amount of traffic you have to start with and if you want to test how different combinations of elements work together.
While there may be several changes we make to the page through multiple versions, we want to understand the impact of each variation. With this type of testing, we are interested how each part of the page impacts the overall conversion rate; so we must keep everything the same except for the specific element or combination of elements we are testing. This allows us to say that a increase or decrease in conversion rates was directly related to the element/elemnts being tested.
using the above innovative testing example above, version B won so we then tested multiple versions of the background CTA image.
While these two types of tests are very different, one is not always better than the other. It is important to figure out what your purpose for testing is and what you want to achieve and then pick the type of test that will best acheive that. A combination of both of these types of tests over the course of your testing timeline will yield the greatest results in the long run.