A polished AI product-video demo is not evidence that the same supplier can handle your catalog. Before a brand expands from one showcase asset to dozens of SKUs, the vendor should prove that its workflow can preserve an unfamiliar product, visualize the intended selling point, and survive multi-shot storytelling without turning QA into manual rescue work.
That is the operational lesson behind a new research signal: product advertising now needs its own evaluation criteria. Generic video quality is no longer enough.
What AdSpark changes about the evaluation conversation
On October 7, 2026, researchers released the preprint “AdSpark: A Large-Scale Dataset and Benchmark for Product-Centric Advertisement Video Generation”. The paper describes AdSpark-300K, a dataset of roughly 300,000 reference-image, prompt and video triplets: 100,000 real-world ads and 200,000 synthetic ads, spanning 40 product categories, more than 3,000 subcategories and 651 hours of video.
Its more important contribution for buyers is the evaluation design. AdSpark-Bench uses 220 held-out product cases that do not overlap with the training SKUs. It evaluates six areas: visual quality, product fidelity, instruction adherence, temporal coherence, audio alignment and advertising effectiveness. Of those cases, 175 are multi-shot, making it possible to inspect whether the product remains the same across cuts and whether selling points form a coherent ad rather than a sequence of attractive but disconnected scenes.
The authors report persistent problems under complex motion: shape distortion, appearance drift and damaged packaging text can reduce product fidelity even when the footage looks dynamic. They also report larger weaknesses among the tested open-source models in instruction adherence, multi-shot consistency and advertising effectiveness.
Those findings do not pick a winning vendor for a brand. They do support a practical conclusion: approving AI product video on beauty alone is an incomplete procurement test.
What the paper does not prove
AdSpark is a preprint, not an adopted industry standard. The paper says the source assets came through authorized internal interfaces at a major ecommerce platform, but it does not name that platform in the body. The authors say the dataset will be released after acceptance, so buyers should not treat the study as a fully reproducible commercial benchmark today.
Its coverage is also primarily ecommerce. Product selection was guided by platform click-through statistics from April 15, 2026, which the authors acknowledge may underrepresent long-tail items, other markets and different advertising styles. “Advertising effectiveness” includes multimodal-model judging; it is not a substitute for human consumer research, incrementality testing or sales results.
The right move is therefore not to copy the paper’s scores into a request for proposal. It is to borrow the principle of held-out product testing and build a brand-owned evaluation set.
The ONCE SKU stress test
1. Hold back products the vendor has not optimized for
Ask the vendor to work on a small set of real future-production SKUs that were not used to create the pitch demo. Cover three levels of difficulty:
- Low risk: clean silhouettes, simple packaging and few fine details. These reveal baseline subject stability and visual quality.
- Medium risk: reflective metal, transparent materials, deformable objects, accessories or human-product interaction. These test physical relationships and selling-point demonstration.
- High risk: small packaging text, ports, scales, replaceable parts, regulated claims or visually similar variants. These expose footage that looks plausible while depicting the wrong item.
For every test SKU, freeze the approved reference images, exact variant, non-negotiable identity regions, selling point to be visualized and prohibited claims. If the inputs keep changing, the brand cannot tell whether a failure came from the model, the source data or a moving brief.
2. Separate hard gates from soft scores
A single weighted average can hide the most expensive errors. Product truth should not be offset by cinematic polish.
Use hard gates for release eligibility. The product must retain its defining shape and components. Packaging and brand marks must not mutate. The ad must not demonstrate a feature that does not exist. The selling-point action must actually happen. Color, accessories and construction must remain consistent across shots. Narration and imagery must describe the same version of the product.
If an error could change a purchase decision, the asset fails the gate. It should not receive a passing average because the lighting is beautiful. The US Federal Trade Commission’s advertising guidance says advertising claims must be truthful, non-deceptive and evidence-based; its small-business FAQ explains that regulators consider words, pictures, express claims and implied claims in context. Generative production does not remove the advertiser’s responsibility for what the finished ad communicates.
Apply soft scores only after the hard gates are met: attention value, pacing, clarity of the selling point, brand fit, format adaptability and required cleanup. These scores help distinguish good options without allowing a wrong SKU to become “good enough.”
3. Compare risk-adjusted deliverable cost
Do not compare vendors on generation price alone. Use a closer approximation of what the brand actually buys:
Risk-adjusted cost per deliverable = testing and generation + manual repair + product review + rework and discarded outputs ÷ assets that pass every hard gate
This is a decision model, not an industry benchmark. A workflow can be inexpensive per render and still be costly per approved asset if packaging, ports, claims or voice-over must be rebuilt repeatedly.
Record first-pass acceptance, post-repair acceptance and final rejection by SKU. The result becomes a brand-specific “generation suitability map”: which products can be fully generated, which need real-product footage within generated environments, and which should remain live-action or 3D-led.
Let the failure pattern choose the production route
Fidelity fails across several products
If unrelated SKUs all show shape drift, packaging substitution or cross-shot identity changes, the problem is likely systemic. Pause the rollout. Change the model or workflow, simplify the shot task, or move the product layer to live action, 3D or conventional compositing. Scaling manual repair defeats the economics of automation.
The product is accurate, but the selling point is invisible
A video can preserve the item perfectly and still do nothing beyond rotating it in an attractive scene. Reframe the benefit as an observable action, outcome and usage condition, then redesign the shot plan. For hybrid routes, ONCE’s analysis of keeping real products grounded in generated environments explains why lighting, contact and spatial logic must be planned together.
Text, logos or critical interfaces remain unstable
This is often a case for hybrid production, not total rejection. Let generation handle atmosphere, environment and selected motion; bring verified packshots, packaging, screens and identity-critical details into the final through live action, 3D or compositing. The goal is not the highest AI percentage. It is the lowest-risk route to a truthful, usable ad.
Keep the benchmark as a reusable brand asset
At the end of the pilot, retain more than the finished videos. Save approved references, SKU versions, selling-point statements, prompts and shot plans, rejected outputs, repair logs, gate results and actual human review time. A new vendor or model should face the same held-out products, so the brand can compare systems instead of reacting to a fresh demo reel.
For brands sourcing in China, this also improves the handoff between product, ecommerce and production teams. Fact ownership stays with the brand; visual execution stays with the production partner; release eligibility is decided against a shared test pack. If you are weighing generative, live-action and 3D routes, ONCE’s video production team can begin with a representative SKU stress test before a catalog-wide commitment.
The research signal is not that AI product video has reached guaranteed scale. It is that scale now requires a product-specific benchmark. A vendor that can explain where its workflow fails—and route those failures intelligently—is often more valuable than one that only shows where it looks best.
