Crawling Is Not the Same as Use: A Four-Lane AI Access Policy for Brand Websites

Brand websites should not choose between “allow every AI bot” and “block every AI bot.” Discovery crawlers, answer engines, shopping agents, and model-training crawlers create different value—and different risk.

A timely signal arrived on September 30, 2026, when Cloudflare launched the beta of Pay Per Use. An AI company can define a paid use, offer a price, identify its crawler, and report each use; a publisher chooses whether to participate. Cloudflare distinguishes this from Pay Per Crawl: one prices access, while the other prices what happens after access.

That announcement proves only that Cloudflare has introduced this beta. It does not prove that usage-based AI licensing is an industry default. For most commercial brand sites, charging crawlers may not be the best first move at all. The structural lesson is more useful: a crawl, a citation, a recommendation, and a training use are not the same economic event.

What the official sources establish

The evidence supports purpose-specific controls, with important limits.

• Cloudflare says Pay Per Use can distinguish a cited answer, a quoted passage, and a product review that shapes a shopping recommendation. The buyer defines the use and self-reports events; the publisher decides whether to accept the offer.
• IAB Tech Lab's AI Bot Management Guidance, released May 27, 2026, frames bot management as a business decision with different strategies, costs, and trade-offs—not a universal block list.
• Google's current crawler documentation says Google-Extended is separate from Google Search. Its setting does not affect inclusion or ranking in Google Search.
• OpenAI's crawler documentation separates OAI-SearchBot, GPTBot, and user-triggered ChatGPT-User. A site can allow search visibility while disallowing model-training crawling.

The same sources also mark a boundary. Robots directives and content signals are expressions of preference, not guaranteed access control. Cloudflare explicitly recommends combining them with technical measures when enforcement matters.

The ONCE D-A-A-T framework

Use four lanes to decide what each class of content should permit. These are business purposes, not vendor names.

D — Discover

Discovery lets search and answer systems find a page, understand its subject, and route a prospect toward the brand. Public product pages, service pages, specifications, and help content usually benefit from this lane.

The acceptance test is not merely a 200 response. Product names, markets, dates, image descriptions, claims, and structured data must agree. Wider access amplifies ambiguity as efficiently as it amplifies good information.

A — Answer

Answer access allows page facts to support real-time responses, grounding, excerpts, or citations. It can influence a buying decision even when the user never visits the site.

Prioritize verifiable, updateable facts that remain accurate outside a long narrative: materials, dimensions, compatibility, warranty conditions, delivery markets, and product limitations. Treat full methods, unreleased products, client data, complete creative assets, and context-sensitive comparisons more cautiously.

A — Act

Action means checking inventory, generating a quote, building a cart, making a booking, submitting a form, or calling an API. It touches price, personal data, availability, and promises made in the brand's name.

Require verified agent identity, explicit read and write scopes, freshness windows, human confirmation for consequential actions, complete logs, and revocation. Our related framework, Do Not Let an AI Buy for the Customer Yet, covers how to stage shopping-agent permissions. Here, the key point is simpler: permission to read a page is not permission to act for a customer.

T — Train

Training is the hardest benefit to attribute and may have the longest-lasting effects. Allowing search or real-time answers does not require allowing training; Google and OpenAI both document independent controls for these purposes.

Evaluate training access separately. Does the brand control all relevant rights? Do client, creator, talent, or stock-media terms impose limits? Would broad reuse erode the value of an exclusive method or visual library? Marketing, legal, technical, and content owners should make this decision deliberately instead of inheriting a web-platform default.

Give different page types different passes

An executable policy maps page types to lanes.

Public product and service pages: Allow Discover by default. Allow Answer when facts are clear and maintained. Permit Act only through controlled endpoints or explicit confirmation. Decide Train according to the brand's content and rights strategy.

Case summaries and insight articles: Use Discover and selective Answer to communicate capability. Keep source files, full production assets, confidential client material, and reusable proprietary templates behind real authentication. Robots rules are not a security boundary.

Price, inventory, and regional availability: These can support Discover and Answer only when currency, geography, timestamps, and expiry conditions travel with the data. Any write action should use a verified interface and require confirmation before a quote or order is submitted.

Client files, bids, masters, and internal libraries: Deny all four lanes by default. Protect them with identity, authorization, network, and server-side controls.

Calculate net access value

Do not decide with a generic “open” or “closed” posture. Evaluate each page class with variables the business can actually observe:

Net access value = qualified discovery + citation or recommendation value + attributable conversion + support savings − substitution loss − infrastructure cost − rights and compliance risk − maintenance cost

• Qualified discovery means target-market visibility and useful landing-page sessions, not raw crawl volume.
• Citation value asks whether the brand is represented accurately, attributed, and present in high-intent questions.
• Conversion value counts trackable outcomes such as quote requests, trials, carts, or orders.
• Substitution loss covers zero-click answers, decontextualized reuse, and competitors learning from proprietary methods.
• Infrastructure cost includes bandwidth, logging, bot management, WAF rules, and abusive-traffic handling.
• Rights risk includes third-party media, creators, clients, contracts, privacy, and local regulation.
• Maintenance cost covers vendor changes, rule updates, false blocks, testing, and cross-functional review.

If discovery, citation, and conversion value outweigh the risks for a public sales page, open Discover and Answer first. If the content itself is the product—a research archive, premium review corpus, or specialist publication—licensing or usage-based payment may deserve more attention. A normal brand site often gains more from being found and quoted correctly than from charging for every machine visit.

A two-week implementation sprint

Days 1–3: Export 30 days of non-human traffic. Group requests by verified identity, user agent, IP range, paths, frequency, and response codes. Do not trust a crawler name alone.

Days 4–6: Classify pages as public sales content, answerable knowledge, dynamic transaction data, or restricted assets. Assign default D-A-A-T permissions and leave genuinely unknown decisions unresolved.

Days 7–9: Configure robots directives, vendor-specific tokens, or content signals. Use authentication, WAF rules, rate limits, and server-side authorization for sensitive paths. Document policy signals separately from enforcement controls.

Days 10–12: Test that allowed discovery crawlers can reach important pages, training crawlers receive the intended response, dynamic facts carry freshness metadata, and restricted paths remain inaccessible without credentials.

Days 13–14: Establish a baseline for crawling, citations, qualified sessions, inquiries, and false blocks. Review vendor documentation and logs monthly, with a rollback plan.

The decision that changes

Cloudflare's Pay Per Use is not a signal that every brand should start charging AI companies. It is a signal that access is becoming more granular: discovery, answers, recommendations, and training can represent different exchanges of value.

The next question for a brand website is no longer “Should we block AI?” It is: who may discover, who may answer, who may act, and who may train? A four-lane policy protects restricted assets without making public sales content invisible to the systems increasingly mediating customer decisions.

This is a marketing and operations framework, not legal advice. Copyright, contract, privacy, and regulatory questions require review by qualified counsel for the relevant content and markets.