How Rockerbox Minimizes Business Disruption During a Test
How Rockerbox Minimizes Business Disruption During a Test
A common concern before running an incrementality test is the cost of running it, specifically the revenue or conversions that might be foregone while media is paused or adjusted in test markets. Rockerbox's test design process is built to make that cost as small as possible while still producing a result you can act on. This guide explains how.
π― The Core Principle: Right-Size the Test to the Question
Every test involves a tradeoff. Larger holdouts and longer flights produce more precise results, but they also mean more disruption. Smaller, shorter tests are less disruptive, but they may not be sensitive enough to detect a real effect.
The goal is not to minimize disruption at any cost. The goal is to design a test that is just powerful enough to answer the question you're asking, and no larger than that.
That requires answering one question before the test launches: what is the smallest lift you would actually care to detect?
π Aligning the MDE with Spend
The Minimum Detectable Effect (MDE) is the smallest true lift a test is statistically powered to find. It is determined by the variance in your KPI data, the size of your test markets, and the duration of the test.
When Rockerbox designs a test, we calculate the MDE before a test launches and compare it to the lift threshold that would change your business decision. This alignment matters because:
- If your MDE is lower than a meaningful lift, the test is appropriately powered and you will be able to detect a real effect if one exists
- If your MDE is higher than the lift you care about (for example, if the test can only detect a 30% lift but a 10% lift would change your decision), the test is underpowered and the disruption cost is not justified by the information you'll receive
In practice, we use your KPI's historical variance and the conversion volume in candidate test markets to model the expected MDE across a range of holdout sizes and test durations. This lets us find the smallest test configuration that still answers the question, rather than defaulting to a larger test than necessary.
Calculating the MDE requires careful simulation. A naive MDE estimate, for example one derived from a simple formula based on sample size alone, will typically understate the true MDE of a geo test. Geo-level data introduces complexities that simple calculations do not account for: KPI variance is correlated across time and markets, the synthetic control introduces its own estimation uncertainty, and the number of geographic units (typically metro areas) in a test is almost always small relative to what a standard power formula assumes. Rockerbox uses simulation-based power analysis, drawing on your actual historical KPI data, to produce MDE estimates that reflect the true sensitivity of the test design rather than a theoretical approximation.
This distinction matters practically. A simulated MDE may be meaningfully higher than a formula-based estimate for the same test configuration, which could change whether a proposed holdout size is actually sufficient, or whether the test is worth running at all at a given spend level.
The choice of modeling methodology also directly affects the achievable MDE. More statistically efficient methods, such as Bayesian Structural Time Series, are able to extract more signal from the same data, producing lower MDEs for equivalent holdout sizes and flight lengths. This is one of the most concrete ways that investing in advanced methodology reduces disruption: it lowers the bar for how much the data needs to move before the model can detect it.
The key implication: a more sensitive test design means you need less disruption to get a reliable answer, and that sensitivity depends on both careful MDE simulation and the quality of the underlying methodology.
πΊ Market Selection: Avoiding High-Stakes Geographies
Not all markets are created equal, and where a holdout runs matters as much as how large it is. Rockerbox selects test and control markets to balance two goals:
- Statistical validity: Test markets should collectively represent a meaningful share of conversions, enough that a real effect will produce a detectable signal
- Business protection: High-revenue markets, markets with active promotions, markets with major store footprints, or markets that are disproportionately critical to near-term business goals should generally be excluded from holdout treatment
In practice, this often means running holdouts in mid-tier metros that are representative of national conversion patterns without being the markets where a temporary pause would create the most commercial pressure.
π Choosing the Right Test Type
The choice between a holdout test and a heavy-up test has a direct impact on how disruptive a test is:
Holdout tests pause or reduce spend in test markets. They are the most statistically powerful test design, because pausing spend creates a larger observable gap between test and control, but they involve foregoing some conversions in those markets during the test window.
Heavy-up tests increase spend above normal levels in test markets. Because you are adding media rather than removing it, there is no direct revenue foregone. The tradeoff is that heavy-up tests measure the marginal return on additional spend, not the baseline incrementality of the channel. They are well suited to questions like "should we scale this channel?" rather than "is this channel working at all?"
For brands where even a small holdout is commercially difficult, a heavy-up test can often answer the most pressing planning questions with essentially no disruption cost.
Multi-cell tests include both a holdout cell and a heavy-up cell alongside a business-as-usual control, offering a middle path. Because the holdout and heavy-up cells partially offset each other in terms of overall spend impact, the net business disruption is lower than a pure holdout of equivalent statistical power. The holdout establishes baseline incrementality while the heavy-up measures marginal returns, delivering both answers from a single test.
The tradeoff is that multi-cell designs require splitting markets across three or more cells rather than two, which means each cell has fewer markets and less conversion volume. This raises the MDE for each cell, making the test harder to power adequately. In practice, multi-cell tests are only viable for brands with high overall conversion volume and channels with significant spend, where there are enough markets and enough KPI signal to distribute across cells without sacrificing sensitivity. For smaller brands or lower-volume channels, a well-designed single-cell test will typically produce a more reliable result with less complexity.
Strategy tests are a different category entirely, and for these the business disruption question largely disappears. Rather than pausing or adjusting spend levels, these tests change how media is deployed (for example, reducing frequency caps in one set of markets, shifting bid strategies, or reallocating budget across placements) while holding overall spend roughly constant. Because total investment is unchanged, there is no direct revenue foregone and no opportunity cost in the traditional sense. Tests of this kind are typically designed not to measure whether a channel drives lift, but to identify whether a tactical change can deliver the same or better results at lower cost. A frequency reduction test, for instance, might find that cutting impressions per user by 30% has no measurable effect on conversions, which would be a finding worth acting on immediately. Because the media change itself is modest and spend is preserved, these tests can often be run with minimal internal approval friction and at any point in the calendar without the timing constraints that govern holdout or heavy-up designs. However, they are less likely to reach statistical significance because the test impact is often smaller.
π¬ Why Bayesian Methods Increase Sensitivity
Rockerbox uses Bayesian Structural Time Series (BSTS) modeling to estimate the counterfactual during a geo test. This methodology has meaningful advantages over simpler frequentist approaches, particularly for minimizing the disruption required to get a reliable result.
Tighter counterfactual estimation BSTS uses all available control market data, including seasonality patterns, trend components, and cross-market correlations, to produce the most precise possible estimate of what would have happened in test markets. The more accurate the counterfactual, the smaller the signal the model needs to detect a real effect.
Bayesian updating The Bayesian framework naturally incorporates uncertainty. Rather than requiring a predetermined fixed sample size before drawing conclusions (as frequentist tests do), BSTS produces a posterior distribution over the true effect that reflects all available data. This means the model can often detect a meaningful effect with a shorter flight or a smaller holdout than a frequentist equivalent.
Posterior credible intervals over binary p-values Because BSTS produces a full distribution of plausible effects rather than just a single estimate, it provides richer information with fewer observations. You get the full picture of the likely range of the true effect, not just a binary pass/fail verdict.
In short, the more statistically efficient the modeling methodology, the less disruption is required to produce a result with the same level of confidence.
π Pre-Period Correlation: The Foundation of Test Sensitivity
The precision of a BSTS model depends directly on how closely the control markets track the test markets before the test begins. Rockerbox targets a pre-period correlation coefficient of >0.8 between the synthetic control and the test markets.
A tight pre-period correlation means:
- The model has a precise baseline to compare against during the test
- Small deviations from the counterfactual are detectable, which lowers the effective MDE
- The confidence interval is narrower, meaning a shorter test or smaller holdout can still produce an actionable result
This is why geographic market selection is not an afterthought. The work done to build a well-matched synthetic control directly reduces the disruption needed to run a conclusive test.
β± Test Duration: Running Long Enough, but Not Longer
Test duration is one of the most direct levers on disruption. Rockerbox calibrates recommended test lengths based on:
- The expected MDE at a given holdout size: longer tests accumulate more data and lower the MDE
- Diminishing returns: beyond the point where the model has sufficient data, extending the flight adds disruption without meaningfully improving precision
- Conversion lag: tests need to run long enough for media-driven conversions to actually occur and be recorded, particularly for channels with long consideration cycles
Rockerbox typically recommends test windows of four to eight weeks for geo-based holdout tests, with the specific duration informed by your KPI's conversion velocity and historical variance. Tests are not extended beyond what the power analysis indicates is necessary.
π Timing: Choosing the Right Window for the Right Question
Timing a test well is not just about avoiding noise. It is about making sure the conditions during the test actually reflect the question you are trying to answer.
As a default, avoid periods of unusually high organic demand. Peak periods (major sales events, holiday windows, brand campaign launches) tend to be driven heavily by existing brand awareness and returning customers acting on a promotion. The incremental lift from paid media, which typically shows up as new customer acquisition driven by advertising exposure, can be harder to detect against a backdrop of elevated organic demand. A test run during a peak period may understate media effectiveness, produce noisier results that require a longer flight to resolve, or both.
For most evergreen incrementality questions ("is this channel working?" or "what is the iROAS of this spend level?"), a stable, representative period with normal conversion velocity will produce a cleaner result in a shorter window.
However, if your question is specifically about how media performs during a promotion, then you should test during the promotion. A test run in a standard period cannot tell you how media contributes to a sale-period lift in demand. If understanding media's role during BFCM, a product launch, or a seasonal push is the business question, the test should be designed around that window, with the expectation that results will reflect promotional conditions and may not generalize to always-on performance.
The key is to match the test window to the decision you are trying to make, and to sequence tests so they do not overlap with each other in the same markets.
π’ Putting the Impact in Perspective
Even when a test feels commercially significant, such as pausing a major channel for a full month, the actual revenue at risk is typically much smaller than it appears. A simple back-of-envelope calculation makes this concrete.
Suppose you are running a holdout test on a channel that represents 20% of your total media budget. The holdout covers 20% of markets, a typical test footprint, while the remaining 80% of markets continue running at normal spend levels. And assume the test runs for one month during a normal, non-promotional period, which for most brands represents roughly 5% of annual revenue.
The final factor is the share of revenue that is actually driven by paid media, as opposed to what would come in anyway through organic demand, brand awareness, and promotions. This is not the same as a channel's share of spend. A channel that accounts for 20% of your media budget might only be responsible for around 10% of your total revenue, because a meaningful portion of all conversions, across all channels, would have happened without any advertising at all. If we assume media as a whole drives around 50% of revenue (with the other 50% coming in organically), then any individual channel's true revenue contribution is roughly half of what its spend share might suggest.
Putting it together, the absolute maximum revenue affected by the test is:
20% (channel share of spend) Γ 20% (holdout market fraction) Γ 50% (media-driven share of revenue) Γ 5% (revenue share of test window) = 0.1% of annual revenue
And this is still a ceiling, not an expectation. In reality the impact is lower still, because:
- Revenue in holdout markets does not drop to zero: organic demand continues, and some customers will find the brand through other channels even without the tested media
- The 50% media contribution assumption is generous for many brands; those with strong organic demand or high brand awareness will see an even smaller real impact
The point is not that tests have no cost. It is that the cost, when calculated honestly, is almost always a fraction of a percent of annual revenue. For a business considering a test on a channel it has never validated, that is a very small price for a meaningful answer about whether the spend is working.
π‘ Key Takeaways
- Business disruption is a function of test design, not an inherent cost of testing: the goal is to design the smallest test that answers the question
- Aligning the MDE with your decision threshold ensures you are not running a larger test than the question requires
- Market selection protects high-value geographies from holdout treatment while preserving statistical validity
- Bayesian Structural Time Series modeling is more data-efficient than frequentist alternatives, requiring less disruption to reach an equivalent level of confidence
- Tight pre-period correlation between test and control markets lowers the effective MDE, meaning shorter flights and smaller holdouts can still produce actionable results
- For brands where revenue risk is a primary concern, heavy-up tests can often answer the most pressing planning questions with no holdout cost at all
Reach out to your Rockerbox Professional Services contact to discuss test design options for your specific business and channel mix.