An overseas brand receives an AI creative pilot report containing a hundred tested drafts. Before expanding the engagement with its China-based production partner, it should ask how many projects those files represent. Short edits, translations and revisions of one brief can create volume without providing equally broad evidence.
ONCE recommends keeping related project material together when separating tuning from acceptance. The question is whether the workflow handles tasks that did not shape its configuration, not whether it can improve its score on familiar examples.
A current use for established evaluation methods
Anthropic’s Demystifying evals for AI agents, published January 9, 2026, distinguishes capability evaluation from regression checks and describes the need to calibrate model graders against human judgment. These are agent-evaluation practices, not a certification for advertising creative.
The official scikit-learn cross-validation guide explains that repeatedly tuning against test results can leak test information into model selection. For dependent groups, it describes keeping the evaluated groups out of the corresponding training groups. Applying that principle to creative-service pilots is ONCE’s proposed adaptation; the documentation does not demonstrate advertising outcomes.
Both sources were checked November 1, 2026. This is an evergreen procurement method supported by current material, rather than a newly announced product feature.
Count project families before counting files
Imagine ten fictional product briefs, each producing twelve related scripts. A random file split could put one brief’s full version into tuning and its cutdowns into acceptance. The factual constraints, voice and revision pattern may already be familiar. This is a hypothetical design example; no experiment was run.
No model retraining is necessary for that problem to arise. Prompt examples, retrieved cases and human rule changes can all carry information from work described as “unseen.”
| Relationship | Proposed treatment | Reason |
|---|---|---|
| Long, short and vertical scripts from one brief | Keep in one project group | They share the underlying assignment |
| Translations and revisions of one parent script | Keep with the parent | Answers and failure patterns may carry across |
| Different projects using a near-identical template | Record the template relationship too | A new identifier does not guarantee a different task |
| A project unused during tuning | Candidate acceptance group | It still needs to represent the intended workload |
Choose the grouping level for the decision. New-project work within one brand may justify a project split. A service intended for unfamiliar brands or product categories needs a broader grouping strategy. Making every test maximally unfamiliar would answer a different purchasing question.
ONCE’s three-box pilot
The reference box holds material that may inform examples and rules. The tuning box holds tasks used to compare prompts, models and workflow changes. The acceptance box holds project groups not used for those adjustments, with expected judgments controlled by a designated reviewer.
Withhold tuning access and answer keys, not necessary task inputs. During the actual acceptance run, provide the complete brief, approved product facts and required references. Missing information should not become an artificial failure trap. Use customer materials only within their authorized purpose and environment.
Once a result informs changes, relabel the task
A failed acceptance task can become useful improvement material. Once the team adjusts the workflow around it, its next score should no longer be presented as independent evidence on an untouched task. Record the move to tuning or regression use and reserve fresh groups for the next acceptance decision.
Regression tasks remain valuable: they ask whether a fix broke previously working behavior. Keep that result separate from the ability to handle new projects.
Report small projects as well as large ones
A long project may supply many easy variants while a short one contains a difficult product restriction. File-level totals can conceal that difference. Report results by project and issue category alongside the underlying file counts.
For fact-checking work, track confirmed misses, reviewer-confirmed false alarms, unresolved judgments and human review time. Keep subjective creative appeal separate from factual correctness. A valid new finding should not become a false alarm merely because the original answer key omitted it.
A pilot with few projects should be described that way. A hundred related drafts do not establish a hundred independent demonstrations, and a limited pilot cannot establish a universal accuracy rate across brands, languages and categories.
What to request before an annual commitment
- A map of projects, derivatives and the grouping rationale.
- The tuning and acceptance assignments, including any later changes of role.
- Versions of the model, prompt, retrieved reference material and grading rules, plus responsibility for disputed judgments.
- Project-level results and human review workload, rather than only an aggregate score.
The supplier need not disclose another client’s confidential files. Authorized or clearly identified fictional tasks can support a bounded pilot, with their limits stated. A demonstration should remain labeled as a demonstration; it is insufficient grounds for removing human approval.
Separate the cost of preparing cases and confirming criteria from recurring execution and review costs. Expansion depends on the delivered cost and scope of acceptable work, not just a low inference price.
Why the relationships become a durable asset
For ONCE’s move toward managed content and workflow knowledge, the useful record connects briefs, derivatives, confirmed judgments and usage history. It shows which examples shaped the process and where fresh validation remains necessary. This is a service-design proposal, not a claim that ONCE has deployed the system or measured a return.
The next action is small: map the project families and reserve whole projects before tuning the prompt. Explore related methods in ONCE’s AIGC insights when defining the broader production scope.
Sources and limits
Anthropic, Demystifying evals for AI agents, January 9, 2026; scikit-learn, Cross-validation: evaluating estimator performance, especially test-set tuning and grouped-data sections, no clear publication date displayed. Checked November 1, 2026.
Project grouping, the three-box pilot and purchasing checklist are ONCE’s adaptations. No model was trained and no client experiment or performance result is claimed. Workflow evaluation does not measure actual consumer response to the creative.
