Why run experiments?
- According to a Harvard Business Review (HBR) study, an advertiser that ran 15 experiments (versus none) in a given year sees about a 30% higher ad performance that year; those that ran 15 experiments in the prior year see about a 45% increase in performance.
- According to Bain & Company, experimentation has helped several global companies spur business growth with 20% or more ROI increases.
- Allianz’s experiments resulted in a 25% uplift in marketing effectiveness.
- Coca-Cola’s experiments contributed to the objective of a 20% improvement in marketing effectiveness globally.
- Uber runs over 1,000 experiments on average in a month
- Amazon runs more than 2,000 per year
- Netflix runs over 1,000 marketing tests per year
- Booking.com runs over 1,000 experiments at any given moment
- Coca-Cola now runs 50 to 100 experiments per quarter
- Microsoft…
- Google…
- Airbnb…
Not every company needs to run 100s or 1000s of experiments, but running experiments will improve business results and give you a competitive advantage.
When done right, experimenting can empower companies with the most challenging marketing decisions: how to allocate marketing budgets, balance brand portfolios, which segments to target with what products and content, and via which channels.
Choosing a statistical test for marketing experiments
Marketers learn most of these techniques in MBA. You can use the following tests to validate the results of marketing experiments related to campaigns, conversion optimization, or the findings of any research.
Enterprises generally have dedicated data/research specialists to plan and run complex experiments frequently. If you are a marketer working for a startup, you may need to run the basic experiments and validate the results yourself, depending on your role.
Data specialists use better alternatives to traditional tests like t-tests. This flowchart gives you only a basic understanding of the different tests available.
The following image is large. To zoom in on the image, right-click and select “Open image in new tab” or click here to view the high-resolution image.
Google Ads experiment solutions
The following image is large. To zoom in on the image, right-click and select “Open image in new tab” or click here to view the high-resolution image.
Meta Ads experiment solutions
The following image is large. To zoom in on the image, right-click and select “Open image in new tab” or click here to view the high-resolution image.
