Cloudflare announced the launch of a private beta of AI Index , marketed as a "compensable web index." Once enabled, the platform automatically generates an AI-optimized index for websites and provides standardized APIs (including MCP servers, LLMs.txt, and a search API). Site owners own and control how the index is used, with AI access billed through Pay per crawl and upcoming x402 billing integration. AI builders can also access licensed, up-to-date data through AI Index, avoiding "blind crawls."
This project aligns with Cloudflare's "default blocking of AI crawlers" and "pay-per-use crawling" policies, which have been in place for the past year: It's license-based and provides clear usage and billing mechanisms. An application form for private testing is now open, with separate entry points for site owners and AI builders. Detailed pricing and coverage will be announced as testing progresses.
Frequently Asked Questions
Q: What does AI Index provide for webmasters?
A: Automatically generate AI-optimized indexes that can be used for on-site search and external licensing, and bill AI access through Pay-per-Crawl/x402.
Q: What can the AI team gain?
A: You can obtain high-quality, timely updated data sources licensed by the site for training or retrieval, reducing copyright and compliance risks.
Q: What is the relationship with “default blocking of AI crawlers”?
A: Based on the default blocking, AI Index + Pay-per-Crawl provides a compliant channel for "visits that are willing to pay and state the purpose".
Q: Is it fully open?
A: This is currently in Private Beta and an application is required. Subsequent features and prices are subject to official announcements.
Q: What components are included technically?
A: Standardized interfaces such as MCP server, LLMs.txt, and search API facilitate integration and measurement between both parties.
