Guide to Incrementality Testing with Rockerbox

Updated by Will Burghes

This guide combines five key modules to help you understand and execute incrementality testing with Rockerbox. Each section includes an overview of concepts covered in our training videos, along with supporting context to help you confidently apply them to your business.

Module 1: Understanding Incrementality Testing

🎥 Watch the video

Incrementality testing helps measure the true impact of media by stripping away conversions that would have happened organically. It ensures that marketing decisions are based on what’s genuinely driving new demand.

Rockerbox supports incrementality in three ways:

  1. Ingest and calibrate MTA and MMM using your existing test results
  2. Consult on your test plans (run by you or your partners)
  3. Fully manage tests end-to-end, including design, analysis, and strategic insights

6-Step Testing Process:

  1. Prioritize tests based on spend, the availability of results from MMM and MTA, potential lift etc.
  2. Select the test methodology (geo-based, time-based, randomized controlled trial or fully custom)
  3. Design the test (KPIs, platform, dates, budget, test geographies)
  4. Support and provide instructions for test execution by your media team or media agency
  5. Analyze test results by building a model to estimate lift in each KPI, along with efficiency (incremental ROAS or CPA)
  6. Contextualize results within your broader measurement strategy

Module 2: What You Need to Get Started

🎥 Watch the video

For geo tests, you will need:

  • One year of daily data for each of the KPIs to be measured in the test, by date and geography (state, metro, zip, postcode etc.)
    • The data should be available for each point of sale that you want to examine (i.e. ecomm, bricks and mortar, third party retail etc.)
    • If you want to break out customer acquisition, first time purchases should also be included as a dimension in the data set
    • If you are a Rockerbox attribution customer then this data is most likely already available in your Rockerbox data foundation
    • If you are a Rockerbox testing only client but use Shopify for your point of sale then Rockerbox can automatically collect the required conversion data for you
  • Ability to buy media at the same geographic level used in your test design - Rockerbox will design your tests at the correct level based on the buying options available to the publisher / platform

If geo-based testing isn't possible you will need:

  • One year of daily data for each of the KPIs to be measured in the test, by date (state, metro, zip, postcode etc.)

Module 3: Building a Learning Agenda & Testing Calendar

🎥 Watch the video

Planning when and what to test ensures you maximize insights while minimizing risk.

How to Prioritize Tests:

  • High-spend channels: Maximize ROI by validating large investments
  • Suspected Low Incrementality channels: Identify true incremental value of remarketing, brand search, and affiliate
  • Channels lacking other measurement: e.g., print or influencer
  • Channels with conflicting MMM/MTA results

When to Avoid Testing:

  • Peak promo periods (e.g. BFCM, product launches, sponsorships)
  • Major external events (Prime Day, competitor promos)

When to Retest:

  • When there have been significant changes to:
    • Brand strength
    • Media buying tactics
    • Budget allocated to publisher / platform
    • Competitive landscape
  • When you want to get deeper insights into the channel, such as the impact of increased (instead of business-as-usual) spend

    Module 4: Choosing a KPI

🎥 Watch the video

Selecting the right KPI is essential for meaningful test results.

Good KPIs should be:

  • Responsive to media in a short timeframe (e.g., conversions, leads, site visits)
  • Measurable at a geo level or within Rockerbox
  • Tied to long-term success, with stable conversion paths (e.g., sessions → leads → revenue)

Tip: You can track both a primary KPI (like revenue) and multiple secondary KPIs (like orders, leads and site sessions) for richer analysis that exposes the performance of a channel at different levels of the funnel.

Module 5: Methodology Selection

🎥 Watch the video

Rockerbox supports three core testing methodologies:

1. Geo-Based Testing

  • Increase spend in or exclude specific geos
  • Compare to a counterfactual modeled using KPI performance in control geos
  • Measure lift for all KPIs and calculate incremental ROAS or CPA

2. Time-Based Testing

  • Increase or cut spend in the channel being tested for a specified period of time
  • Compare the test time period vs. control time period using univariate analysis techniques

3. Bespoke Testing

  • Fully custom tests designed for channels that cannot easily be measured using geo-based or time-based tests

Choosing the right method depends on:

  • The targeting capabilities of the media channel being tested
  • The availability of conversion data with geographic identifiers

Tip: Rockerbox can also help analyze the results of publisher-run randomized controlled trials using data in the Rockerbox data foundation

Need help applying these concepts? Your Rockerbox team is here to guide you through every step—from test planning to interpreting results and acting on insights.

How did we do?