Testing Methodology
Selecting the right methodology is key to running an effective incrementality test. In this guide, we walk through the three most common testing methods Rockerbox supports—explaining how they work, when to use them, and how we leverage the results.
🗺 Method 1: Geo-Experimental Testing
Best for: Channels where you can control media geographically
- Rockerbox selects test geos (e.g. specific U.S. states, cities, DMAs) to either start or pause media spend
- Separate control geos are also selected and weighted to estimate the counterfactual: what would have happened without a change in spend
- The difference between actual vs. expected conversions in test geos = incremental lift
- We use lift + spend to calculate incremental ROAS or incremental CPA
📉 Method 2: Difference-in-Differences
Best for: Channels that can't be controlled geographically (e.g. print, influencer, podcast)
- Identify two highly correlated channels (e.g. A and B)
- Change spend in Channel A and observe the drop in conversions
- Use Channel B's conversion patterns to predict what would’ve happened in Channel A without the change
- The gap between actual and predicted conversions in Channel A = the media impact
🧪 Method 3: Randomized Control Trials
Best for: Platforms or publishers that support audience-level randomization
- Media audiences are randomly split into:
- Test group (exposed to ads)
- Control group (not exposed or shown PSAs)
- Conversion rate differences between the two groups indicate media impact
- Rockerbox uses these results to calibrate Media Mix Models (MMM) and inform broader measurement strategy
Note: Rockerbox does not run RCTs directly but can help incorporate publisher-run trials into your broader analysis.
🧭 Choosing the Right Approach
- Use GeoTests where you can control geographic spend
- Use DiD where geographic control isn’t possible
- Use RCTs if working with publishers that support randomized testing