How Much Credit Did One Video Take? Counterfactual Mismatch in Short-Form Measurement

ONCE Social Short-Form Video Marketing Research · Social Short-Form Video Marketing Research 08/10

Research article / literature comparison and creative decision analysis. Research cutoff: October 5, 2026.

One short video is followed by 100 purchases and another by 70. Is the first the winner?

Not yet. The first may have reached more people who had already searched for the brand, while the second reached more unfamiliar viewers. Some of those 100 buyers may have purchased without seeing the video. Other viewers may have searched, visited a dealer or bought elsewhere, allowing another channel to receive the credit. If the two versions did not even have the same opportunity to be seen by comparable people, a direct conversion-rate comparison compresses creative, audience selection, media delivery and pre-existing demand into one score.

Those numbers are hypothetical and are not ONCE client data.

Our central argument is that the most common error in short-form measurement is not failing to add one more metric. It is asking a metric from one layer to answer a causal question from another. Watch time can diagnose attention but cannot establish comprehension on its own. A click can describe a recorded path but cannot establish incremental purchases on its own. Platform ROAS can support daily operations but does not automatically answer how many outcomes would have disappeared without the campaign. We call this error counterfactual mismatch.

One score cannot answer five different questions

Short-form video may pass through at least five result layers: exposure, attention, comprehension, brand and action. They are not a guaranteed sequence, and they should not share one universal metric.

Episode 01 showed that recommendation changes who enters the observed sample. Episode 07 then examined how short-form influence can migrate into off-platform search and purchase. “Exposure to action” is therefore better treated as a set of interacting observation windows than as a complete funnel in one platform dashboard.

This also explains why completion and sales can rise together without the former causing the latter. Both may be affected by more selective delivery, a stronger offer, seasonal demand or a larger proportion of existing customers. Correlation can generate a mechanism hypothesis. A causal answer still needs a treatment, a comparison, an outcome window and a defined contrast.

Attribution allocates credit; incrementality estimates a world without the campaign

Attribution and incrementality both matter, but they do different work.

Attribution distributes completed conversions among visible touchpoints. Last-click, first-touch and multi-touch models answer how to assign credit within a recorded path. Incrementality compares two potential outcomes: what would happen to the target population with the campaign and without it. Attribution mainly supports reporting, optimisation and channel coordination. Incrementality supports the decision about whether the budget created additional outcomes.

Brett R. Gordon, Robert Moakler and Florian Zettelmeyer’s 2023 Marketing Science article, “Close Enough?”, examines 663 large Facebook advertising experiments. Even with more than 5,000 user- and experiment-level features, double/debiased machine learning and stratified propensity score matching did not reliably recover the randomised results. The article reports median RCT lifts of 29%, 18% and 5% for upper-, middle- and lower-funnel outcomes. The corresponding double-machine-learning median lifts were 83%, 58% and 24%. This does not make observational data useless. It shows that complex modelling does not guarantee removal of selection created by platform delivery. The evidence comes from Facebook advertising experiments rather than every contemporary short-form platform, and the article discloses the authors’ working relationships with Meta/Facebook.

A May 2026 working paper posted to SSRN in July, Attributed, But Not Incremental: Cannibalization-Corrected Attribution for Large-Scale Advertising, uses incrementality experiments as causal anchors to calibrate daily attribution signals. Several authors are ByteDance researchers. It is a short, non-peer-reviewed working paper, and we read only the public abstract; its deployment claims are not independent validation. Its limited but useful implication is that attribution need not be discarded, but it should be calibrated by experiments rather than promoted into a causal fact.

Last-click is especially prone to taking too much credit. Someone may watch a short video, actively search and then buy after clicking a branded-search ad. The order may be awarded entirely to search. Conversely, the absence of an in-platform click does not prove the video had no effect. Measurement maturity is not a more elaborate allocation rule. It is knowing when an allocation rule cannot answer incrementality.

Recommendation can make the “same experiment” stop being the same

Random assignment matters, but the word random does not repair every design.

First, assignment is not exposure. Being assigned to a group eligible to see an ad creates an opportunity; recommendation, auction, frequency controls and audience choice still determine who actually sees it. Comparing only people who were exposed with people who were not reintroduces selection. A more defensible primary estimate often begins with the result of assignment to an eligible group versus assignment to an ineligible group—the intention-to-treat effect. Estimating the effect among people actually exposed requires additional assumptions and methods.

Second, recommendation learns. Jennifer Brennan and colleagues’ WWW 2025 paper studies “symbiosis bias” in dynamic recommender experiments. When two algorithms share training data generated under both conditions, behaviour in one arm changes the learning environment of the other. The paper compares cluster-randomised, data-diverted and user-corpus co-diverted designs and validates the existence of the bias with data from a large A/B test on a global recommender. This is a study of recommender experiments, not brand advertising. Its lesson for advertisers is narrower: version A and version B inside automated optimisation may not remain two static, independent worlds.

Third, users affect one another. Sharing, direct messages, following a creator and later search can transport treatment into the control group. Xu Min and colleagues’ February 2026 Kuaishou preprint uses clusters built from social-interaction graphs to contain spillover in experiments on sharing strategies. It reports large-scale online experiments but remains an unreviewed preprint and is not a universal platform design. It supports one design question: if content travels through relationships, user-level randomisation may underestimate the full social effect because the “untreated” group may still receive treatment indirectly.

Fourth, outcomes can occur elsewhere. Yitian (Sky) Liang and colleagues’ 2025 Marketing Sciencerandomised ByteDance field experiment allowed only treatment users the opportunity to receive one automobile brand’s short-form ads and matched platform users to the brand’s lead records. The formal abstract reports that most of the effect occurred beyond ByteDance and that exposed users were more than eight times as likely to convert off-platform as on-platform. This was one automobile advertising environment, and conversion meant providing contact details rather than buying a vehicle. It is not a multiplier for other sectors. It shows that missing off-platform outcomes can change cost and targeting conclusions.

Holdouts, geo experiments and quasi-experiments estimate different things

No design wins across every budget, platform and data environment.

User-level randomised holdouts are useful when a platform or advertising system can reliably prevent the control group from receiving the treatment and can observe the same outcome window for both groups. They directly estimate the average incremental effect of assignment to an advertising opportunity, but must still address non-exposure, cross-device activity, sharing contamination and missing event matches. Organic content rarely creates a genuinely ineligible control group in the same way as paid advertising.

Geo experiments are more useful when person-level matching is unavailable, online and offline results must be combined, or several channels need to be observed together. Jon Vaver and Jim Koehler’s 2011 Google paper randomly assigns non-overlapping regions to treatment and control. Google’s current documentation also identifies contamination as a risk: someone can see the ad in a treatment region and convert in a control region, reducing the estimated difference. Geo experiments also require enough comparable regions, a stable pre-period relationship and sufficient budget. Two very different cities do not become a reliable experiment merely because a line separates them.

Quasi-experiments are used when genuine randomisation is unavailable—for example, a natural campaign pause, abrupt budget change, channel launch or policy shift. They require stronger assumptions: absent the intervention, the treated unit and the constructed comparison would have followed a predictable relationship, and no simultaneous shock affected only the treated side. A September 2026 preprint, Pseudo-Incrementality Testing, proposes detecting naturally occurring marketing interventions and then constructing a structural time-series counterfactual. We read only its abstract and it has not been peer reviewed. It is a methodological candidate, not permission to call every budget change a natural experiment.

Before choosing a design, write the estimand in one complete sentence: for which eligible population, over what time window, are we comparing which advertising opportunity with which no-treatment condition, on what outcome? If this sentence is unclear, a more sophisticated method usually creates more room for interpretation.

Peeking and then picking a winner can invalidate significance

Short-form programmes often test many openings, captions, people and offers at once, checking the dashboard every day and stopping as soon as one version leads. That process continually increases the chance of finding an accidental winner.

Ramesh Johari, Pete Koomen, Leonid Pekelis and David Walsh’s Operations Research article explains why conventional p-values and confidence intervals become unreliable when users choose sample size through continuous monitoring. It develops always-valid p-values and confidence intervals and addresses sequential and multiple testing. Brands need not use one identical statistical system, but they must treat the stopping rule as part of the experiment.

In practice, pre-specify one primary business outcome—such as incremental qualified leads within 14 days—then treat viewing, comprehension, brand linkage and search as mechanism or secondary outcomes. If many versions are compared, control the multiple comparisons or explicitly label the first round exploratory and confirm it with fresh data. An increase selected from dozens of metrics cannot be rewritten as the original objective.

Signal, path and incrementality: keep three ledgers

ONCE proposes a limited three-layer measurement framework. It is an organisation of the evidence and a decision tool, not a validated universal model.

The signal ledger records exposure, staying, comprehension, brand linkage and negative reactions. It diagnoses where the creative mechanism changed and answers what occurred between image, information and audience.

The path ledger records clicks, search, landing pages, leads, store visits and purchases, together with the rules that allocated credit. It answers which decision paths were visible.

The incrementality ledger uses randomised holdouts, geo experiments or sufficiently credible quasi-experiments to estimate outcomes that would not have occurred without the campaign. It answers what the investment added.

The ledgers can explain one another, but they cannot impersonate one another. Better signals with no detected incrementality may mean the creative improvement failed to cross a commercial threshold, or that the experiment lacked power. More attributed conversions with no incrementality may mean credit was reallocated or organic demand was cannibalised. Real incrementality with little platform attribution may indicate off-platform or delayed outcomes. Diagnosis begins with these competing explanations rather than ending with one attractive score.

Use a published case to write a measurement plan

ONCE’s Xpeng G6 limited-time down-payment short-form case describes a sequence of promotion, vehicle presentation, cabin detail, road context and an action prompt. We read the case text and visible page information; we did not view a verifiable complete film and have no public budget, audience or conversion data. The case does not prove that this order increased sales.

It can, however, demonstrate how one creative problem becomes several measurement tasks.

• If the question is whether the offer was understood, the primary result could be correct recall of its conditions, with misunderstanding as a guardrail. Completion is secondary.
• If the question is whether the person-and-cabin sequence increases model comprehension, compare two information orders under comparable exposure while keeping the offer, duration, destination and audience as consistent as possible.
• If the question is whether the entire campaign creates additional bookings, use a user holdout, a geo comparison or another defensible counterfactual rather than comparing people who clicked “book” with those who did not.
• If the question is how short-form video, search and dealerships work together, record the path but use incremental bookings or visits as the causal outcome. The final touchpoint cannot own all the credit.

A randomised A-versus-B creative test answers which version performs better under the present media conditions. An incrementality experiment comparing the campaign with no campaign answers how much the advertising itself added. Both are often called A/B tests, but they are not the same question.

A minimum measurement agreement that can fail

Before launch, put these seven items on one page:

1. Eligible population: who can enter, and whether existing customers, employees or prior leads are included.
2. Treatment and comparison: A versus B, advertising versus no advertising, or high versus low budget.
3. Primary outcome: one result that determines the budget decision, with an explicit observation window.
4. Mechanism metrics and guardrails: the roles of viewing, comprehension, brand linkage, complaints and unsubscribes.
5. Randomisation unit and contamination paths: user, region, account or community, and how sharing, travel and cross-device use connect groups.
6. Stopping and multiple-testing rules: when results are read, how many versions are compared and how exploration will be confirmed.
7. Missing data and failure interpretation: how off-platform purchases, store visits, failed matching and small samples limit the claim.

A null result can still be useful. It may rule out a commercially meaningful effect, or it may simply reflect a wide interval, contamination or an outcome too distant from the mechanism. Report the estimate, interval, sample and anomalies together. “Not significant” is not the same as “definitely no effect.”

Conclusion: measurement should not award one total score

Short-form performance is not a single story in which plays automatically become purchases. Exposure determines who enters the sample. Attention determines whether processing continues. Comprehension and brand linkage determine what remains. Search and clicks capture parts of the path. Leads and purchases sit closer to the business result. Even then, incrementality needs a credible counterfactual.

Ask which ledger a metric belongs to, then ask which counterfactual it lacks. Attribution can run continuously while experiments calibrate it periodically. Creative tests can select versions while incrementality tests evaluate budget. Geo and quasi-experimental methods can support settings where user randomisation is unavailable, but their assumptions must be visible. A mature measurement system does not stack more curves onto one dashboard. It lets each number answer only the question it is qualified to answer.

If you are planning social assets that compare openings, product actions, captions and calls to action, review ONCE’s short-form filming and multi-version production approach. A brief can define the communication task, shared shots, primary outcome and variables that must not change at the same time. This creates interpretable creative assets; it is not a claim that ONCE provides platform attribution infrastructure or guarantees campaign results.

Sources and reading scope

• Brett R. Gordon, Robert Moakler and Florian Zettelmeyer, “Close Enough? A Large-Scale Exploration of Non-Experimental Approaches to Advertising Measurement,”Marketing Science 42(4), 2023, DOI: 10.1287/mksc.2022.1413. We read the formal page, abstract, study scope, results and conflict disclosure.
• Yitian (Sky) Liang, Xinlei Chen, Shengnan Han, Jinglong Zhang and Yubo Chen, “Is the Money Spent on Short-Form Video Social Platforms Worth It?”Marketing Science 44(5), 2025, DOI: 10.1287/mksc.2023.0575. We read the formal record, abstract and publicly accessible method material and keep the interpretation to its automobile lead outcome.
• Jennifer Brennan et al., “Reducing Symbiosis Bias through Better A/B Tests of Recommendation Algorithms,” WWW 2025. We read the Google Research record and abstract; its estimand concerns recommender algorithms rather than brand advertising.
• Xu Min et al., “Towards Reliable Social A/B Testing: Spillover-Contained Clustering with Robust Post-Experiment Analysis,” arXiv:2602.08569, submitted February 9, 2026. We read the public full text; it is not peer reviewed.
• Donghui Li et al., Attributed, But Not Incremental: Cannibalization-Corrected Attribution for Large-Scale Advertising, SSRN 6992318, 2026. We read only the public abstract; it is not peer reviewed. The SSRN landing page did not pass the final availability check, so it is cited here without a hyperlink.
• Jon Vaver and Jim Koehler, “Measuring Ad Effectiveness Using Geo Experiments,” Google, 2011; Ramesh Johari et al., “Always Valid Inference: Continuous Monitoring of A/B Tests,”Operations Research 70(3), 2022. They establish the method boundaries for geo and sequential or multiple testing.
• ONCE’s short-form solution, published case and completed series articles were rechecked on October 5, 2026. The case supports creative and measurement design analysis only; no undisclosed result is claimed.