Testing Methodology
Selecting the right methodology is key to running an effective incrementality test. In this guide, we walk through the three most common testing methods Rockerbox supports—explaining how they work, when to use them, and how we leverage the results.
🗺 Method 1: Geo-Experimental Testing
Best for: Channels where you can control media geographically
- Rockerbox selects test geos (e.g. specific U.S. states, cities, metros) to either start or pause media spend
- Separate control geos are also selected and weighted to estimate the counterfactual: what would have happened without a change in spend
- The difference between actual vs. expected conversions in test geos = incremental lift
- We use lift + spend to calculate incremental ROAS or incremental CPA
📉 Method 2: Time-Based Testing
Best for: Channels that can't be controlled geographically (e.g. print, influencer, podcast, affiliate, linear TV)
- Identify a channel to test and change its spend level
- Use univerariate modeling to create a counterfactual based on historical conversion patterns
- The gap between actual and predicted conversions = the media impact
🧪 Method 3: Bespoke Testing
Best for: Platforms or publishers with complicated testing needs
- For some channels we will design fully bespoke tests or analysis to evaluate performance:
- Affiliate marketing, where different product categories are treated differently and can be analyzed against each other
- Paid search tests, where non-brand search attributed performance can be used as a counterfactual for brand search attributed performance
Note: Rockerbox does not run randomized controlled trials but can help independently verify results from RCTs, and incorporate publisher-run trials into your MMM or other broader analysis.
🧭 Choosing the Right Approach
- Use GeoTests where you can control geographic spend
- Use time-based tests where geographic control isn’t possible
- Use fully bespoke tests if working with unique marketing situations that require special treatment