Incrementality Testing Glossary
Incrementality Testing Glossary
A reference guide to the key terms you'll encounter when planning, running, and interpreting incrementality tests with Rockerbox.
๐ฌ Core Concepts
Incrementality Testing: A measurement methodology that uses controlled experiments to determine how many conversions were caused by your marketing, not just associated with it. It separates advertising-driven conversions from those that would have happened organically, regardless of spend.
Incremental Lift: The additional conversions, revenue, or other KPI outcomes that can be directly attributed to a marketing activity. Lift = what happened in the test minus what would have happened without it (the counterfactual).
Counterfactual: The estimated outcome that would have occurred in the absence of the media being tested. In geo tests, Rockerbox models the counterfactual using control market data. The gap between the observed outcome and the counterfactual is the incremental effect.
Baseline or Organic Conversions: Conversions that would have occurred even without advertising (driven by brand awareness, direct demand, word of mouth, or other organic factors). Incrementality testing exists to separate these from media-driven conversions.
Test Group (Treatment Cell): The markets, users, or audiences that receive the media treatment during the test period. Results in the test group are compared against the control group to estimate lift.
Control Group (Holdout): The markets, users, or audiences that do not receive the media treatment. Their behavior during the test period provides the baseline against which lift is measured.
Test Pollution / Spillover: When media targeting in test markets bleeds into control markets (or vice versa) contaminating the results. Metro-level geo tests are preferred in part because they are more resilient to spillover than zip code, state or city-level designs.
๐ Test Design
Geo-Based Test (Geo-Lift Test): An incrementality test that uses geographic regions (typically U.S. metropolitan markets, states, or countries) as the unit of randomization. Media spend is adjusted (increased, decreased, or paused) in selected test markets, while control markets maintain business-as-usual spend. The difference in outcomes between test and control markets (after accounting for baseline differences) represents the incremental effect. There are different options available for geo-regions including media markets, metro areas, states / provinces, countries and postcodes.ย
Holdout Test: A test design in which media spend is paused entirely in test markets. Measures the full baseline contribution of a channel at its current spend level, the most powerful test type for establishing true incrementality.
Heavy-Up Test: A test design in which media spend is increased above normal levels in test markets. Measures the incremental impact of additional spend, useful for understanding marginal returns. Cannot by itself establish whether baseline spend is incremental.
Multi-Cell Test: A test design that includes more than two treatment conditions (for example, a holdout, a standard spend level, and a heavy-up) across separate market groups. Enables spend response curve analysis but is more complex to power adequately.
Synthetic Control: A statistical method that creates a weighted combination of control markets to best match the pre-test behavior of the test markets. Rockerbox uses synthetic control to identify which markets should serve as the counterfactual for a given set of test markets.
Difference-in-Differences (DiD): A testing methodology that measures the change in outcomes in a test group relative to a control group, using pre-period data to account for baseline differences. Used when geographic targeting is not possible. For example, testing channels like print, influencer, or podcast by comparing two correlated internal channels.
Randomized Controlled Trial (RCT): A test in which audiences are randomly assigned to exposed or holdout groups at the individual user or device level. Rockerbox can incorporate publisher-run RCT results into MMM and calibration, though Rockerbox does not administer RCTs directly.
Ghost Ads: An RCT methodology in which the holdout group is served a placeholder "ghost" ad (typically a PSA or unrelated creative) instead of the actual campaign, ensuring equivalent exposure opportunities between groups. This minimizes selection bias in the holdout.
Pre-Period / Pre-Test Window: The time period before a test begins, used to validate the correlation between test and control markets and to establish baseline KPI patterns. A strong pre-period correlation (typically >0.9) is a prerequisite for a well-designed geo test.
Post-Window / Post-Test Window: A period of time after the test completes that can be incorporated into the test analysis to see if there is any impact from the media that was tested which extends past the end of the test. This can be particularly important for upper funnel channels that take longer to have an impact.
Test Window / Flight: The active period during which the media treatment is live and data is being collected. Results are measured against the counterfactual during this window.
Minimum Detectable Effect (MDE): The smallest true lift that a test is statistically powered to detect, given the KPI variance, sample size, and test duration. Tests should be designed so that the MDE is smaller than the lift you'd actually care to observe. A test that cannot detect a 5% lift will not tell you whether a 3% lift exists.
Statistical Power: The probability that a test will correctly detect a real effect when one exists. Power is a function of sample size, variance, and the size of the effect being tested for. Rockerbox targets 80% power as a standard threshold for test design.
Test Overlap: Running multiple tests simultaneously across the same markets or audiences. This can confound results and should be avoided unless tests are carefully isolated with separate market pools.
๐ Results & Metrics
Point Estimate: The single most likely value for the true incremental effect, as estimated by the model. The center of the confidence interval. Represents the best guess (not a guaranteed truth) about what the channel is doing.
Confidence Interval (CI): A range of plausible values for the true incremental effect. The interval is shaped like a bell curve: the point estimate at the center is the most probable value, with probability tapering off toward the edges. A 90% CI means that, if the test were repeated many times, 90% of those intervals would contain the true effect. See also: Understanding Confidence Intervals & Statistical Significance.
Statistical Significance: A binary property of the confidence interval. A result is statistically significant when the entire confidence interval is above zero, meaning the data is inconsistent with no effect at the specified confidence level. A non-significant result does not mean there is no lift; it means the data cannot rule it out. See also: Understanding Confidence Intervals & Statistical Significance.
Confidence Level (e.g., 80%, 90%): The threshold used to define the confidence interval. Rockerbox typically reports at a 90% confidence level. A higher confidence level produces a wider interval and a harder-to-meet bar for statistical significance.
p-value: A statistical measure that reflects how unlikely the observed result would be if there were truly no effect. A p-value below 0.10 is equivalent to reaching 90% statistical significance. The p-value alone should not be used to make binary decisions about channel effectiveness. The full confidence interval carries more information.
iROAS (Incremental Return on Ad Spend): The revenue generated per dollar of ad spend, calculated using only the incremental lift, not total attributed revenue. Formula: Incremental Revenue รท Ad Spend. A channel with a high attributed ROAS but a low iROAS is capturing demand rather than creating it.
iCPA (Incremental Cost Per Acquisition): The cost to generate one incremental conversion. Formula: Ad Spend รท Incremental Conversions. A higher than attributed CPA suggests that a portion of attributed conversions are organic.
Lift %: The percentage by which the tested metric (e.g., conversions, revenue) increased in test markets compared to the counterfactual. Directionally useful, but iROAS and iCPA are generally more actionable for budget decisions.
Material Significance: Whether the magnitude of the measured effect is large enough to matter for business decisions, independent of whether it reaches statistical significance. A statistically significant iROAS of 0.2 may not be materially significant for a brand with a target of 2.0.
๐งฎ Modeling & Methods
Bayesian Structural Time Series (BSTS): The primary modeling framework Rockerbox uses to estimate the counterfactual in geo-lift tests. BSTS uses control market data to model what would have happened in test markets absent the media, producing a posterior distribution of the incremental effect (expressed as a confidence interval).
Posterior Distribution: In Bayesian modeling, the probability distribution over possible values of the true effect, given the observed data. The confidence interval reported in Rockerbox test results is derived from this distribution.
Correlation Coefficient: A measure of how closely two time series move together, ranging from -1 to +1. Rockerbox targets a pre-period correlation of >0.8 between test markets and the synthetic control to confirm that the control is a valid counterfactual.
Priors (Bayesian): In Bayesian modeling, prior beliefs about the likely range of an effect before observing test data. Incrementality test results can be used to set or update priors in Rockerbox's MMM, improving model calibration over time.
Point-in-Time Results: Test results are inherently bound to the test window. They reflect the incremental impact of the tested channel at a specific spend level, flight date, and set of market conditions, not a permanent or universal truth about the channel.
๐ Planning & Strategy
Learning Agenda: A prioritized roadmap of the questions your testing program is designed to answer, tied to specific channels, tactics, or hypotheses. A well-structured learning agenda ensures tests are sequenced logically and that results feed into each other.
Testing Calendar: A schedule of planned tests, including proposed channels, methodology, timing, and expected KPIs. Good calendars avoid overlapping tests and steer clear of promotional periods that would distort baseline behavior.
Opportunity Cost: The revenue or conversions foregone in test markets during a holdout test, where media is paused. A necessary cost of a well-controlled test, typically small relative to the value of the insight.
Incremental Fraction: The share of total attributed conversions that are truly incremental i.e., would not have happened without the media. A channel with a high attributed conversion count but a low incremental fraction is largely taking credit for organic demand.
Calibration: The process of using incrementality test results to adjust attribution model outputs. If a test shows a channel's true iCPA is 2x higher than its attributed CPA, a calibration multiplier can be applied to produce a more accurate unified KPI.
Unified KPI: A marketing performance metric that blends attribution data with calibration inputs (from incrementality tests, MMM, surveys, or other analyses) to better reflect true causal impact across channels.
For clarifying questions about incrementality testing concepts, reach out to your Rockerbox Professional Services contact.