Guide to Incrementality Testing with Rockerbox
This guide combines five key modules to help you understand and execute incrementality testing with Rockerbox. Each section includes an overview of concepts covered in our training videos, along with supporting context to help you confidently apply them to your business.
Module 1: Understanding Incrementality Testing
Incrementality testing helps measure the true impact of media by stripping away conversions that would have happened organically. It ensures that marketing decisions are based on what’s genuinely driving new demand.
Rockerbox supports incrementality in three ways:
- Ingest and calibrate your existing tests
- Consult on your test plans (run by you or your partners)
- Fully manage tests end-to-end, including design, analysis, and strategic insights
6-Step Testing Process:
- Prioritize tests based on spend and potential lift
- Select test methodology (Geo or DiD)
- Design the test (dates, budget, geo)
- Support media execution by your team or agency
- Analyze for incremental ROAS or CPA
- Contextualize results within your broader measurement strategy
Module 2: What You Need to Get Started
For GeoTests, you’ll need:
- KPI data by date and geography (state, DMA, zip)
- Store-level or retail partner data if testing offline or Amazon sales
- Ability to buy media at the same geographic level used in your test
If geo-based testing isn't possible:
- Use Rockerbox attribution data to run Difference-in-Differences (DiD) tests by channel type, product category, or tactic
Module 3: Building a Learning Agenda & Testing Calendar
Planning when and what to test ensures you maximize insights while minimizing risk.
How to Prioritize Tests:
- High-spend channels: Maximize ROI by validating large investments
- Low incrementality channels: Identify true value of remarketing, brand search, and affiliate
- Channels lacking other measurement: e.g., print or influencer
- Channels with conflicting MMM/MTA results
When to Avoid Testing:
- Peak promo periods (e.g. BFCM, product launches, sponsorships)
- Major external events (Prime Day, competitor promos)
When to Retest:
- Significant changes to brand strength, media tactics, or competitive landscape
Module 4: Choosing a KPI
Selecting the right KPI is essential for meaningful test results.
Good KPIs should be:
- Responsive to media in a short timeframe (e.g., conversions, site visits)
- Measurable at a geo level or within Rockerbox
- Tied to long-term success, with stable conversion paths (e.g., lead → purchase)
Tip: You can track both a primary KPI (like leads) and secondary KPIs (like purchases) for richer analysis.
Module 5: Methodology Selection
Rockerbox supports three core testing methodologies:
1. Geo-Experimental Testing
- Spend on or exclude specific geos
- Compare to weighted control geos
- Measure lift and calculate incremental ROAS or CPA
2. Difference-in-Differences (DiD)
- Compare two correlated channels (e.g., brand vs. non-brand)
- Adjust spend in one and use the other as a control
3. Randomized Control Trials (RCTs)
- Publisher-run audience-level split
- Test group vs. control group
- Rockerbox uses results to calibrate MMM and inform measurement
Choosing the right method depends on:
- Your media control capabilities
- Type of media
- Your ability to target/test by geography or tactic