Incrementality Testing: Frequently Asked Questions
Incrementality Testing: Frequently Asked Questions
š Test Design & Setup
Do you need our attribution data to design or analyze a test? No. Incrementality tests are designed and analyzed using your conversion or revenue data directly, not attribution outputs. Attribution tells you which touchpoints received credit; a geo test compares what actually happened in test markets against a modeled counterfactual. The two methodologies are independent, which is part of what makes incrementality a valuable check on attribution. That said, attribution data can be useful context when prioritizing which channels to test or interpreting results alongside a broader measurement picture.
Why do you use media markets instead of states? Media markets give us more units to work with, which is critical for building a well-matched synthetic control and achieving adequate statistical power. The U.S. has over 200 media markets but only 50 states, meaning market-level designs allow for more precise matching, more flexibility in holdout construction, and smaller holdout footprints relative to total spend. States also vary enormously in size, making them harder to balance across test and control groups. Media markets are the standard unit for media buying in the U.S., which means holdouts can be cleanly executed with most channel partners. See How Rockerbox Minimizes Business Disruption During a Test for more on how market selection works in practice.
How many markets do we need to run a test? There is no fixed minimum, but the practical floor is determined by the power analysis. You need enough conversion volume across test markets that a real incremental effect, if it exists, produces a signal the model can detect against normal variance. The simulation-based MDE calculation Rockerbox runs before launch will tell you whether a proposed market set is adequate. In practice, tests with fewer than 20 media markets in each cell are harder to power, particularly for lower-volume KPIs. See How Rockerbox Minimizes Business Disruption During a Test for how market selection is balanced against statistical requirements.
What channels can be geo-tested? Any channel where spend can be adjusted at the market level can be tested with a geo-based holdout or heavy-up design. This includes paid social, paid search, connected TV, streaming audio, display, direct mail, and out-of-home. Channels where geo-level control is difficult, such as linear TV, influencer, some podcast buys, or organic social, may require alternative methodologies like difference-in-differences designs. If you are unsure whether your channel is testable, your Rockerbox Professional Services contact can advise on design options.
What KPIs can be measured in a test? Any metric you can observe at the market level on a daily or weekly basis. Common KPIs include purchases, revenue, new customer revenue, site visits, leads, sign-ups, and app installs. The quality of the geographic signal in your KPI data matters significantly. KPIs with a hard address, such as a shipping address for an ecommerce purchase or a physical store visit, are the most reliable for geo-based testing because they tie conversions to a precise location. IP-based location signals, which underpin many site visit and lead metrics, are less accurate and more prone to misattribution across markets, which can introduce noise into the counterfactual and widen confidence intervals. If your primary business KPI is something other than purchases or store conversions, it is worth discussing with your Rockerbox team how the location signal quality affects test design before launch.
Can we run multiple tests at the same time? Yes, but only if the tests use completely separate market pools. Running two tests that overlap in the same media markets will contaminate both results, because you cannot isolate the effect of each channel independently. If your market footprint is large enough to divide into non-overlapping pools with sufficient conversion volume in each, concurrent tests are possible. For most brands, the more practical approach is to sequence tests and prioritize the most budget-impactful channels first.
š Business Impact
How much revenue are we actually putting at risk by running a test? Much less than it typically feels like. For a holdout test on a channel representing 20% of your budget, run in 20% of markets, for one month during a normal non-promotional period, the absolute maximum revenue affected works out to approximately 0.1% of annual revenue. That calculation is 20% (channel spend share) Ć 20% (holdout market fraction) Ć 50% (media-driven share of total revenue) Ć 5% (share of annual revenue in the test window). Even that ceiling overstates the real impact, because organic demand continues in holdout markets regardless of whether the tested media is running. See How Rockerbox Minimizes Business Disruption During a Test for the full breakdown.
Can we avoid running the holdout in our most important markets? Yes, and this is standard practice. Rockerbox selects holdout markets to balance statistical validity against commercial sensitivity. High-revenue markets, markets with major physical store footprints, and markets that are disproportionately important to near-term targets are typically excluded from holdout treatment. The holdout is usually run in mid-tier media markets that are representative of national patterns without carrying outsized commercial risk.
What if we can't pause spend at all? A heavy-up test, which increases spend above normal levels in test markets rather than pausing it, involves no revenue foregone and no holdout. The tradeoff is that heavy-up tests measure the incremental return on additional spend rather than the baseline contribution of the channel. There is still an opportunity cost to consider: the incremental media deployed in heavy-up markets is likely to be less efficient than baseline spend, since you are pushing further up the marginal cost curve. This is a real cost, even if it does not show up as lost revenue. For brands where any holdout is commercially difficult, a heavy-up can often answer the most pressing planning question ("should we scale this channel?") at a lower disruption cost than a holdout, but it is not entirely free. Strategy tests, which test changes in how media is deployed rather than how much is spent, are also an option where disruption is not a factor at all. See How Rockerbox Minimizes Business Disruption During a Test.
Does the timing of a test matter for business impact? Yes, but the timing decision is about more than just minimizing impact. For most evergreen questions about whether a channel is working, stable non-promotional periods produce cleaner results in shorter windows and avoid the noise introduced by peak demand. However, if your question is specifically about how media contributes to a promotional period, the test should run during that promotion. A result from a standard period cannot answer a question about promotional performance. See How Rockerbox Minimizes Business Disruption During a Test for guidance on timing by question type.
š Results & Interpretation
What does it mean if our result isn't statistically significant? It means the confidence interval on your result includes zero, so the data cannot rule out the possibility of no effect at the specified confidence level. It does not mean the channel is not working. A non-significant result with a point estimate of 2.0x iROAS looks very different from a non-significant result with a point estimate near zero, and those two outcomes call for very different decisions. See Understanding Confidence Intervals & Statistical Significance for a full explanation of how to read non-significant results and what questions to ask before acting on them.
Does statistical significance tell us whether the point estimate is the true effect? No. Statistical significance is a property of the confidence interval. It tells you whether the interval is entirely above zero. It does not validate the point estimate, and the point estimate is the same number regardless of whether the result is significant or not. The point estimate is your best guess at the true effect; the confidence interval tells you how much uncertainty surrounds that guess. See Understanding Confidence Intervals & Statistical Significance for a detailed explanation.
How long do we need to run a test to get a result? Rockerbox typically recommends four to eight weeks for geo-based holdout tests, but the right duration depends on your KPI's conversion velocity, the variance in your data, and how quickly the channel itself drives a response. A channel like paid search or retargeting tends to produce conversions within hours or days of exposure, meaning its effect shows up quickly in the data. Upper-funnel channels like connected TV or streaming audio work on longer consideration cycles, where a consumer might be exposed today and convert weeks later. Tests on slower-acting channels need to run long enough to capture that lagged effect, otherwise you are measuring an incomplete picture of the channel's true contribution. The power analysis Rockerbox runs before launch accounts for this and will specify a recommended minimum flight length for your configuration. Running longer than necessary adds disruption without improving the result, so tests are not extended beyond what the data requires.
Can test results change over time? Test results accumulate as more data is collected during the flight. Early in the test, the confidence interval will be wide and the result may not be significant even if the true effect is real. As cumulative data accumulates, the interval actually widens in absolute terms, because the uncertainty around the cumulative counterfactual grows over time. However, the point estimate grows alongside it, and if a real effect exists, the signal grows faster than the noise, which is what drives the result toward significance. Results should be evaluated at the end of the planned test window, not mid-flight, to avoid drawing conclusions from an underpowered sample.
What happens to results after the test window ends? Test results are point-in-time estimates. They reflect the incremental impact of the channel at the specific spend level, market conditions, creative, and timing of the test. They should not be treated as permanent truths about the channel. Market conditions, creative quality, competitive dynamics, and audience saturation all shift over time, which is why an ongoing testing program produces more reliable measurement than a single test.
š Connecting Tests to the Rest of Your Measurement
How do incrementality test results connect to our attribution data? Attribution and incrementality measure different things. Attribution distributes credit across touchpoints based on observed signals; incrementality measures the causal effect of a channel on outcomes. When the two agree, that is a useful cross-validation. When they diverge, attribution is typically overcounting channels that capture demand rather than create it. The test result can be used to apply a calibration multiplier to attribution outputs, bringing attributed performance closer to true incremental performance. See the Incrementality Testing Glossary for definitions of calibration and unified KPI.
How do test results connect to MMM? Incrementality test results can be used as inputs to calibrate Rockerbox's MMM. Rather than relying purely on the model's observed correlations, calibrated priors from a geo test constrain the MMM's estimates to be consistent with measured causal effects. This produces more accurate MMM outputs and reduces the risk of the model overcrediting channels that are correlated with conversions but not causing them.
Do we need to run tests on every channel? Not necessarily. Testing is most valuable for channels where attribution is structurally unreliable (upper funnel, view-through, brand keywords), channels that represent a large share of spend, and channels where you are actively considering scaling or cutting. Channels with high attribution reliability, small budget footprints, or stable historical performance may not be the highest priority for a formal test. Your Rockerbox Professional Services team can help you build a learning agenda that sequences tests by expected value.
š§® Methodology
How does Rockerbox build the counterfactual?Rockerbox uses Bayesian modeling to estimate what would have happened in test markets if no treatment had occurred. The model uses data from control markets that closely tracked the test markets before the test began to construct a synthetic counterfactual. The difference between observed outcomes in test markets and the modeled counterfactual represents the incremental effect. See the Incrementality Testing Glossary for definitions of BSTS, synthetic control, and posterior distribution.
Why Bayesian instead of a simpler approach?Bayesian methods produce a full probability distribution over the true effect rather than a single point estimate with a binary significance verdict. This means you get a more informative result with less data, which in practice means shorter tests or smaller holdouts for the same level of confidence. BSTS also handles the structural complexity of geo-level time series data, including seasonality, trends, and cross-market correlations, better than simpler regression or difference-in-differences based approaches. See How Rockerbox Minimizes Business Disruption During a Test for more on how methodology choice affects test sensitivity and disruption.
What is the MDE and why does it matter? The Minimum Detectable Effect (MDE) is the smallest true lift that a test is statistically powered to detect. If a test's MDE is higher than the lift that would change your business decision, the test will not give you a reliable answer even if you run it. Rockerbox calculates the MDE using simulation on your actual historical data before a test launches, which produces a more accurate estimate than simple formula-based approaches. See the Incrementality Testing Glossary for a full definition, and How Rockerbox Minimizes Business Disruption During a Test for how MDE is used to right-size test designs.
For questions not covered here, reach out to your Rockerbox Professional Services contact.