How to Test a New Ad Channel Without Geo Sales Data

A brand already spends on Meta, TikTok, and YouTube. Now it wants to know whether Snapchat deserves a place in the media mix.

There is one problem:

The company cannot see sales by geography.

It has 18 months of daily sales history, but only as one national revenue number.

That rules out a traditional geo holdout. There are no regions that can be separated into treatment and control based on observed regional sales.

But it does not necessarily rule out an incrementality test.

If Snapchat can be switched on and off nationally, and its important sales effects happen quickly enough, the company can instead run a time-based switchback experiment.

Research on switchback experiments describes exactly this general structure: one system is repeatedly exposed to treatment and control across different time intervals rather than splitting customers or regions into separate groups. The design becomes particularly important when outcomes are aggregated over time and when carryover between treatment periods is possible. (Bojinov, Simchi-Levi & Zhao, 2023; Xiong, Chin & Taylor, 2024.)

Here is the example we will work through:

The goal is not simply to ask whether Snapchat drives some incremental revenue.

The real business question is:

Does Snapchat create enough incremental revenue to justify the investment?

Start with the business threshold, not the statistical test

Before designing the experiment, translate the business requirement into the sales effect that actually matters.

The company plans to spend:

$50,000 on Snapchat

The required incremental ROAS, or iROAS, is:

1.5

So the campaign needs to create at least:

$50,000 × 1.5 = $75,000

in incremental revenue to meet the business threshold.

Now translate that dollar amount into a percentage sales lift.

During the 12-week experiment, Snapchat will be ON for six weeks and OFF for six weeks.

Normal national revenue during the six ON weeks is approximately:

6 × $250,000 = $1,500,000

The $75,000 required incremental revenue therefore represents:

$75,000 / $1,500,000 = 5%

So the most important planning number is:

Snapchat needs to generate approximately a 5% sales lift during the ON periods to correspond to an iROAS of 1.5.

That 5% was not chosen because it is statistically convenient.

It comes directly from the economics of the decision.

Snap budget × required iROAS
────────────────────────────
Revenue during treatment periods

In this case:

$50,000 × 1.5
────────────── = 5%
 $1,500,000

This is the first major principle:

Design an incrementality test around the smallest effect that would actually change the business decision.

Why a simple before-and-after test is weak

The easiest test would be:

Weeks 1–6:  Snapchat OFF
Weeks 7–12: Snapchat ON

Then compare sales.

But that creates a major problem.

If revenue is higher during weeks 7–12, was Snapchat responsible?

Or did sales rise because of:

• seasonality
• underlying growth
• a promotion
• a holiday
• changing consumer demand
• activity in Meta, TikTok, or YouTube
• another event that simply happened later in time?

A single before/after split makes time itself highly correlated with treatment.

Instead, a switchback test repeatedly changes the treatment over time.

For example:

Total Snapchat spend is approximately $50,000.

Meta, TikTok, and YouTube continue running throughout the experiment and should remain as stable as practically possible. Material changes in those channels, promotions, pricing, or other important business events should also be recorded.

The experimental question becomes:

How do national sales behave when Snapchat is ON compared with when it is OFF, after accounting for the normal ways sales change over time?

In a production experiment, the sequence does not have to mechanically alternate exactly as shown above. Treatment assignment can be randomized while maintaining an appropriate balance between ON and OFF periods.

What matters is that the schedule is determined before the outcome is observed, rather than changing Snapchat exposure because early sales happen to look strong or weak.

Randomization and the construction of treatment intervals are central components of formal switchback designs. (Bojinov, Simchi-Levi & Zhao, 2023.)

Why the length of each block matters

Why switch weekly?

Because we are assuming the commercially important response to Snapchat happens quickly enough.

Imagine most Snapchat-driven purchases happen:

• on the day of exposure
• one day later
• or within another day or two

Then a one-week treatment block may give enough time for most of the effect to occur before the next treatment state begins.

But now imagine Snapchat continues influencing purchases for three weeks.

Then this happens:

Week 1: Snapchat ON
Week 2: Snapchat OFF

Some Week 2 sales may still have been caused by advertising from Week 1.

The OFF week is no longer a clean period without Snapchat influence.

This is called carryover or a temporal spillover effect.

Research on switchback design specifically identifies carryover as one of the factors that determines how frequently treatment should switch. Longer periods can reduce contamination from carryover, but fewer switches also mean fewer independent treatment-control comparisons and can reduce precision. (Bojinov, Simchi-Levi & Zhao, 2023; Xiong, Chin & Taylor, 2024.)

So weekly switching is not a universal rule.

Block length should be chosen based on the expected conversion lag and carryover of the intervention being tested.

Before spending $50,000, rehearse the experiment

We now know two things:

• the proposed test lasts 12 weeks
• the commercially relevant effect is a 5% sales lift

But we still do not know whether a 12-week test using national sales is precise enough to see a 5% effect.

That question can be investigated before Snapchat goes live.

The company has 18 months of historical daily national revenue.

That history tells us how noisy the business normally is.

It contains real patterns such as:

• weekdays versus weekends
• seasonal demand
• trends
• promotions
• unusually strong or weak weeks
• recurring sales patterns
• correlation between nearby days
• random fluctuations

Instead of assuming how noisy the future test will be, we can use this historical behavior to simulate the experiment.

Data-driven research on switchback design similarly uses prior observations to inform experimental design because factors such as periodicity, temporal correlation, and carryover affect the precision of the treatment estimate. (Xiong, Chin & Taylor, 2024.)

First, run placebo experiments where the true effect is zero

Take historical periods when Snapchat was not running.

Select a 12-week window and pretend the future test schedule had already been used:

Week 1: OFF
Week 2: ON
Week 3: OFF
Week 4: ON
...

But do not alter the revenue.

Because no Snapchat treatment actually occurred, the true Snapchat effect in this historical period is:

0%

Now analyze the fake experiment using the same statistical process planned for the real test.

Ideally, the model should estimate an effect reasonably close to zero.

If placebo experiments repeatedly return apparent effects such as:

+9%
-8%
+11%

when nothing actually happened, that is a warning.

Either the proposed design is extremely noisy, the statistical method is poorly specified, or both.

Placebo testing therefore asks:

When we know there is no treatment effect, does our planned experiment behave as if there were one?

This is a useful validation step before adding any artificial treatment effect.

Now insert effects where we know the correct answer

Next, repeat the historical experiments but deliberately modify revenue during the fake Snapchat-ON periods.

For example, simulate true lifts of:

+1%
+2%
+3%
+4%
+5%
+6%
+8%

If we inject exactly a 5% increase into every treatment period, we know the simulated truth:

The true treatment effect in that simulation is 5%.

Now run the exact analysis that would be used after the real experiment.

The question is no longer whether the model produces an estimate.

The question is whether the experiment can reliably distinguish the inserted effect from normal sales variation.

What does it mean for the test to "detect" a lift?

This rule needs to be defined before looking at the simulation results.

One possible decision rule is:

Count the effect as detected when the estimated incremental effect is positive and the 95% confidence interval excludes zero.

Suppose we inserted a true 5% lift.

One simulation might produce:

True inserted lift:   +5.0%
Estimated lift:       +5.4%

95% confidence interval:
+1.1% to +9.7%

The entire interval is above zero.

Detected: YES

But another 12-week period might be much noisier:

True inserted lift:   +5.0%
Estimated lift:       +4.1%

95% confidence interval:
-0.8% to +9.0%

The interval includes zero.

Detected: NO

Nothing changed about the true effect.

We inserted 5% in both cases.

The second experiment simply did not contain enough information to distinguish that effect clearly from normal variation.

Repeat the experiment hundreds or thousands of times

One simulation tells us very little.

So we repeat the process across many historical windows and eligible treatment schedules.

The result might look like this:

These numbers are illustrative. They have not been calculated from a real client's sales data.

Suppose we ran 1,000 simulations with a true 5% lift:

1,000 simulations contained a true 5% lift

810 detected it
190 did not

810 / 1,000 = 81%

That approximately 81% detection rate is the statistical power of this test at a true 5% effect, given the simulation assumptions and chosen detection rule.

Statistical power is the probability that a study will detect an effect of a specified size when that effect actually exists, under the assumptions used for planning. Power calculations therefore require a target effect size rather than answering whether the treatment is likely to work in the first place. (See NCBI Bookshelf, "Study Size Planning" and its definition of statistical power.)

What the 81% does and does not tell us

We previously calculated:

Required iROAS:                1.5
Required incremental revenue: $75,000
Required sales lift:           5%

Suppose historical simulation now gives:

Power to detect a true 5% lift:
approximately 81%

We can then say:

If Snapchat truly creates the 5% incremental lift associated with a 1.5 iROAS in this test, the proposed experiment would detect a positive effect in roughly 81% of comparable simulated tests under our assumptions.

We cannot say:

Snapchat has an 81% chance of achieving a 1.5 iROAS.

That is an entirely different statement.

The simulation evaluates the experiment.

It does not predict Snapchat's performance.

The Minimum Detectable Effect is also a property of the test

Suppose our illustrative simulations show:

If we choose roughly 80% power as the design target, the Minimum Detectable Effect, or MDE, is approximately:

5%

That means:

A true effect of approximately 5% would be detected in about 80% of comparable experiments under the planning assumptions.

It does not mean:

We expect Snapchat to generate a 5% lift.

The MDE describes experimental sensitivity, not expected channel performance.

In this example, the alignment is useful:

Commercially relevant lift:    5%
Approx. MDE at 80% power:      5%

The proposed design therefore appears capable of detecting an effect of the size that matters to the business.

If the MDE had instead been 10%, the experiment would be poorly aligned with the commercial question.

A channel could genuinely create the 5% lift required to hit the target and the test would still frequently fail to detect it.

In that situation, it may be better to redesign the experiment before spending the budget.

Now run the real 12-week test

Only after the proposed design passes the pre-test checks does Snapchat go live.

During the 12 weeks, collect at least:

• date
• national revenue
• Snapchat ON/OFF status
• Snapchat spend
• Meta spend
• TikTok spend
• YouTube spend
• major promotions
• holidays
• pricing changes
• other important business events

The treatment sequence should remain fixed according to the pre-agreed experimental plan.

Do not switch Snapchat on because yesterday's sales looked disappointing.

Do not extend an ON period because sales suddenly look strong.

The treatment schedule itself is part of the experiment.

Changing it in response to observed results can bias the analysis.

What happens statistically at the end?

The objective is to estimate:

How much additional national revenue occurred because Snapchat was ON?

A simple comparison of average ON-week revenue against average OFF-week revenue may be misleading if sales contain strong time patterns.

The analysis may therefore need to account for factors such as:

• day-of-week effects
• trends
• recurring seasonality
• holidays
• promotions
• known changes in other media
• correlation between nearby observations

The exact estimation method depends on the data and experimental design.

Formal switchback literature includes randomization-based inference as well as model-based approaches, and more recent research explicitly addresses serial correlation, periodicity, and carryover in time-indexed experiments. (Bojinov, Simchi-Levi & Zhao, 2023; Xiong, Chin & Taylor, 2024.)

For the business, however, the output should remain understandable:

Estimated incremental revenue
+
uncertainty around that estimate

Suppose the result is an estimated $90,000 of incremental revenue

After 12 weeks, imagine the analysis estimates:

Incremental revenue:
$90,000

Snapchat spend was:

$50,000

The estimated iROAS is therefore:

$90,000 / $50,000 = 1.8

The company's threshold was:

1.5

At first glance, Snapchat appears to have passed.

But this is only the point estimate.

The experiment cannot determine the true effect with perfect precision.

Suppose the estimated incremental revenue comes with this uncertainty:

Best estimate:
$90,000

95% confidence interval:
$40,000 to $140,000

Divide each value by the $50,000 Snapchat investment:

Estimated iROAS:
1.8

95% confidence interval:
0.8 to 2.8

Now the decision is less obvious.

What a 95% confidence interval actually means

A common interpretation would be:

There is a 95% probability that Snapchat's true iROAS is between 0.8 and 2.8.

That is not the strict frequentist interpretation of a classical 95% confidence interval.

NIST describes the confidence level in terms of the long-run behavior of the interval procedure: if the same sampling and interval-construction process were repeated many times, approximately 95% of those intervals would contain the true parameter.

For a business audience, a safer interpretation of the observed interval is:

Our best estimate is an iROAS of 1.8, but the data are not precise enough to rule out values substantially below or above that estimate. Values in roughly the 0.8 to 2.8 range remain compatible with the observed data under the model at the chosen confidence level.

The exact technical interpretation depends on the inferential method used, but one principle remains simple:

A wide interval means we know the effect less precisely.

Compare the interval with the business threshold

Our hypothetical result is:

Estimated iROAS:      1.8
95% interval:         0.8 to 2.8
Business threshold:   1.5

The point estimate is above 1.5.

But the uncertainty interval contains values on both sides of 1.5.

So we should not conclude:

Snapchat definitely clears the target.

A more accurate conclusion is:

Snapchat appears to generate incremental revenue and the best estimate is above our 1.5 target, but this experiment was not precise enough to establish confidently that the true incremental return exceeds 1.5.

That is an inconclusive commercial result, even though the point estimate looks attractive.

Detecting a positive effect is not the same as proving iROAS is above 1.5

This distinction is critical.

During planning, we asked whether a 5% effect could be detected against:

No incremental effect

or:

0%

If simulations show 81% power at a true 5% lift, that tells us the design is reasonably capable of answering:

Does Snapchat create a positive incremental effect?

But the stricter business question is:

Can we establish that Snapchat's effect exceeds the amount required for iROAS 1.5?

Those are not the same hypothesis.

To conclude confidently that the channel clears the commercial threshold, the uncertainty around the final estimate must also be evaluated relative to the 1.5 iROAS boundary, not merely relative to zero.

So this statement is wrong:

The test has 80% power at the 5% lift associated with iROAS 1.5, therefore it has an 80% chance of proving iROAS is above 1.5.

It does not.

Power against zero and precision around a commercial threshold are different design questions.

If proving the threshold itself is the primary decision criterion, that stricter requirement should be considered during pre-test simulation as well.

Three possible outcomes

The final decision becomes easier when the uncertainty range is compared with both zero and the business threshold.

Outcome 1: Strong evidence the channel clears the target

Estimated iROAS: 2.2
95% interval:    1.7 to 2.7
Threshold:       1.5

The entire interval is above 1.5.

Interpretation:

The experiment provides evidence that Snapchat's incremental return clears the required business threshold.

Possible decision:

Scale or continue.

Outcome 2: Positive, but commercially inconclusive

Estimated iROAS: 1.8
95% interval:    0.8 to 2.8
Threshold:       1.5

The best estimate is above the target, but the interval spans both sides.

Interpretation:

Snapchat appears incremental, but the experiment does not determine precisely enough whether the true return clears 1.5.

Possible decision:

Inconclusive. Consider additional evidence or another test.

Outcome 3: Even positive incrementality remains uncertain

Estimated iROAS: 0.9
95% interval:   -0.3 to 2.1
Threshold:       1.5

The interval includes zero.

Interpretation:

The experiment did not provide sufficiently strong evidence that Snapchat generated positive incremental revenue.

This is still not the same as proving:

Snapchat has no effect.

A weak effect and an imprecise experiment can produce similar observed results.

Possible decision:

Stop, retest, or classify as inconclusive depending on the economics and wider evidence.

The mistakes that make these tests look more certain than they are

Mistake 1: Looking only at the point estimate

Wrong:

Estimated iROAS is 1.8, so Snapchat passed our 1.5 target.

Better:

1.8 is the best estimate. The uncertainty interval determines how confidently we can compare it with 1.5.

Mistake 2: Treating an inconclusive result as proof the channel does not work

Suppose:

Estimated iROAS: 0.9
95% interval:   -0.3 to 2.1

Wrong:

Snapchat doesn't work.

Better:

The experiment did not estimate Snapchat's effect precisely enough to establish positive incrementality.

Mistake 3: Treating statistical power as the probability the channel works

Suppose:

Power at a true 5% lift:
81%

Wrong:

Snapchat has an 81% chance of producing a 5% lift.

Correct:

If the true effect were 5%, this experimental design would detect a positive effect in approximately 81% of comparable tests under the simulation assumptions.

Power describes the experiment, not the channel.

Mistake 4: Treating the MDE as a forecast

Suppose:

MDE ≈ 5%

Wrong:

We expect Snapchat to increase sales by 5%.

Correct:

A true effect of roughly 5% is around the size this experiment can detect with the selected planning power.

Mistake 5: Confusing "detectable at iROAS 1.5" with "proven above iROAS 1.5"

A design that can often distinguish a 5% lift from zero is not automatically able to show that the effect is confidently greater than 5%.

That stricter claim requires more precision.

Mistake 6: Using one long before/after split

First six weeks:  OFF
Second six weeks: ON

This makes treatment difficult to distinguish from anything else changing over time.

Repeated treatment switching creates much more useful experimental variation.

Mistake 7: Ignoring carryover

If Snapchat continues affecting purchases well into the following OFF period, the treatment and control states become contaminated.

Block length is therefore part of the causal design, not an arbitrary reporting choice.

The entire process in one view

BUSINESS QUESTION

Is Snapchat worth using if we require iROAS ≥ 1.5?

        ↓

TEST ECONOMICS

Snap budget = $50,000
Required incremental revenue = $75,000

        ↓

TRANSLATE INTO SALES LIFT

6 treatment weeks × $250,000
= $1,500,000 baseline revenue

$75,000 / $1,500,000
= 5% required lift

        ↓

DESIGN THE SWITCHBACK

12 weeks
6 ON periods
6 OFF periods
weekly switching if carryover allows

        ↓

USE 18 MONTHS OF HISTORY

Measure normal sales patterns and noise

        ↓

PLACEBO TESTS

Does the analysis find fake effects
when the true effect is zero?

        ↓

INJECT KNOWN EFFECTS

1%, 2%, 3%, 4%, 5%, 6%, ...

        ↓

SIMULATE MANY EXPERIMENTS

How often is each true effect detected?

        ↓

CHECK POWER AND MDE

Can the design reliably detect
the 5% effect that matters?

        ↓

RUN THE REAL TEST

Spend $50,000 according to
the predefined ON/OFF schedule

        ↓

ESTIMATE INCREMENTAL REVENUE

Point estimate
+
uncertainty interval

        ↓

CALCULATE iROAS

Incremental revenue / Snap spend

        ↓

COMPARE WITH

0 = positive incrementality threshold
1.5 = business threshold

        ↓

DECIDE

Scale
Stop
or
Inconclusive

The bigger lesson

A national switchback test is not:

Turn Snapchat on and see whether revenue goes up.

A defensible process is:

Start with the economic threshold, translate it into the incremental sales effect that matters, use historical data to check whether the proposed experiment is capable of detecting an effect of that size, create treatment variation deliberately over time, and interpret the final result together with its uncertainty.

The pre-test simulation cannot tell you whether Snapchat will work.

That is not its job.

It answers a different and extremely valuable question before the budget is committed:

If Snapchat works well enough to matter, is this experiment actually capable of showing us?

That is what turns an incrementality test from a campaign comparison into a decision framework.

Methodology references

The numerical results in this article are illustrative worked examples, not empirical results from a real Snapchat experiment. The following sources support the underlying experimental and statistical concepts:

• Bojinov, I., Simchi-Levi, D. & Zhao, J. — "Design and Analysis of Switchback Experiments," Management Science 69(7), 3759–3777. DOI: 10.1287/mnsc.2022.4583. Supports the use and formal design of switchback experiments, treatment assignment over time, carryover effects, statistical power, randomization-based inference, and confidence intervals.
• Xiong, R., Chin, A. & Taylor, S. J. — "Data-Driven Switchback Experiments: Theoretical Tradeoffs and Empirical Bayes Designs." arXiv: 2406.06768. Supports the importance of carryover, periodicity, serial correlation, and using prior data to inform switchback design.
• NIST/SEMATECH e-Handbook of Statistical Methods — "Confidence Limits for the Mean" and "What are confidence intervals?" Supports the frequentist interpretation of a 95% confidence interval and the relationship between interval width and precision.
• NCBI Bookshelf — "Study Size Planning" in Developing a Protocol for Observational Comparative Effectiveness Research: A User's Guide. Supports the concepts of statistical power and specifying a minimum treatment effect of interest during study planning.