Media corporations have gotten fairly good at blocking AI bots. They’ve been much less good at deciding which of them ought to be allowed in. Reports say most main publications virtually universally block crawlers from main AI corporations like OpenAI and Anthropic, and plenty of small to midsize publishers mirror that, configuring their robots exclusion protocol settings, or robots.txt, to maintain them out.
It’s an comprehensible stance, and it would even be applicable. But it surely additionally crudely compresses a fancy situation right into a binary: Block or don’t block. And the reasoning is usually simply as easy, pointing to the truth that most AI platforms don’t give again vital human visitors. On the identical time, the variety of bot crawls might be excessive, excessive sufficient that value turns into an element. From an ROI standpoint, it’s a simple name.
There are exceptions, after all. Google is the apparent sophisticated one, since AI is more and more constructed into Search itself, and blocking Googlebot means opting out of much more than AI answers. Google says AI Overviews and AI Mode are part of Search, and that website house owners management inclusion by the identical Search crawling and preview controls they already use. And positively, in case your publication has a licensing take care of a selected platform, you’d let its bots by.
{“blockType”:”mv-promo-block”,”information”:{“imageDesktopUrl”:”https://pictures.fastcompany.com/picture/add/f_webp,q_auto,c_fit/wp-cms-2/2025/03/media-copilot.png”,”imageMobileUrl”:”https://pictures.fastcompany.com/picture/add/f_webp,q_auto,c_fit/wp-cms-2/2025/03/fe289316-bc4f-44ef-96bf-148b3d8578c1_1440x1440.png”,”eyebrow”:””,”headline”:”u003Cstrongu003ESubscribe to u003Cemu003EThe Media Copilotu003C/emu003Eu003C/strongu003E”,”dek”:”Need extra about how AI is altering media? By no means miss an replace from Pete Pachal by signing up for u003Cemu003EThe Media Copilotu003C/emu003E. To be taught extra, go to u003Ca href=u0022https://mediacopilot.substack.com/u0022u003Emediacopilot.substack.comu003C/au003E”,”subhed”:””,”description”:””,”ctaText”:”SIGN UP”,”ctaUrl”:”https://mediacopilot.substack.com/”,”theme”:{“bg”:”#f5f5f5″,”textual content”:”#000000″,”eyebrow”:”#9aa2aa”,”subhed”:”#ffffff”,”buttonBg”:”#000000″,”buttonHoverBg”:”#3b3f46″,”buttonText”:”#ffffff”},”imageDesktopId”:91453847,”imageMobileId”:91453848,”shareable”:false,”slug”:””,”wpCssClasses”:””}}
But when direct viewers or income is the one criterion for permitting bots in, that’s too slender a window to essentially reap the benefits of the chance AI presents. Finally, blocking is only one element of an AI-forward technique that understands the various kinds of bots, how they join your content material to audiences, the strategic significance of treating bots as part of the audience, and how you can current content material in AI experiences so that you keep in management.
Block the best bots
Let’s begin with the reality that totally different bots do various things. There are a number of varieties, however broadly three matter most to publishers: coaching bots, search bots, and retrieval bots. A minimum of these are the kinds which are a part of the “reputable” info ecosystem, in that they’re normally declared and documented, and customarily respect robots.txt preferences. Let’s put unauthorized scrapers apart for the second.
It’s the retrieval bots, which usually have “-user” as a part of their title, that publishers ought to pay shut consideration to and take into account giving the inexperienced gentle. These are the bots that go and get info on the fly when a person varieties a query into an AI chatbot. Their exercise is modest and predictable, normally hitting solely a small variety of pages, they usually can usually be rate-limited or served cached content material. Cloudflare, actually, has made it simpler to differentiate between makes use of like search indexing, real-time AI enter, and AI coaching, and its AI Crawl Control tools now embrace choices to permit, cost, or block particular AI crawlers.
Permitting retrieval bots, and probably a rate-limited variety of search bots, can have a significant impact on presence in AI summaries.
“So what?” is the query many publishers retort—you’ll be able to’t take AI presence to the financial institution. Which is true, nevertheless it misses the strategic level: When you’re not within the reply, another person is. That cedes authority on a topic to a competitor, as a result of the extra audiences see them cited, the extra they’ll develop thoughts share, and what little referral visitors there may be will go to them. Profitable in AI means understanding that authority is the prize, not visitors.
Present the worth, don’t give it away
Profitable that authority, nonetheless, doesn’t imply successful in any respect prices. Your most precious belongings—whether or not they be content material, neighborhood, or experiences—want to stay yours, not given freely to AI techniques. Strategic content material specifically requires a cautious steadiness between what you permit AI to see and what readers want to return on to you to find. When you carefully handle metadata, snippets, entry controls, and clear directions for LLMs, you’ll be able to assist bots perceive that beneficial materials exists with out giving it away totally.
For instance, a website specialised within the agricultural trade may need an in depth, data-rich report on the results of pesticides on soybean demand. A snippet may describe precisely what sort of information is inside and why the info is related to sure forms of analysis whereas not revealing the info or conclusions. The report itself may clarify what is out there, what’s restricted, and the place licensed customers can entry the total work.
Ideally, the end result could be that the existence of the report is thought to AI engines, however anybody who wished to truly see it will want to go to the writer and have interaction deeper, with a subscription or at the very least an electronic mail gate. The purpose isn’t to cover totally from AI reply engines. It’s to make them conscious of the writer’s worth with out letting them reproduce that worth.
Your archive is your alpha
There’s an analogy right here with what Palantir cofounder Alex Karp stated in a widely shared CNBC interview, the place he suggested enterprise AI prospects to cease utilizing fashions from the foremost AI labs, because it was successfully giving them their “alpha”—the corporate information that provides them an edge over rivals. That is, in a way, what media corporations have been pressured to do from the start of the AI revolution, since publishers’ information is essentially public and was a giant a part of what the fashions initially educated on.
Now the creators of these fashions are successfully competing with publishers by providing reply engines to the general public, which preserve customers on the portal as an alternative of sending them to websites. That is, after all, the foundational thought behind the various lawsuits and licensing offers between the AI corporations and the media.
To provide customers the most effective and most modern info, although, it’s essential get it for the time being of the question, which is why the market has shifted from coaching information to retrieval. Rob Kelly has a useful analysis of this shift, exhibiting that coaching rights are not a given in publicly introduced AI licensing offers. In his database, solely about 4 in 10 public 2026 offers embrace coaching rights, an indication that the market is transferring from “purchase content material to construct higher fashions” towards “license content material to ship higher solutions.”
The unhappy fact is that right this moment the overwhelming majority of publishers don’t have the leverage to safe their very own licensing offers with OpenAI, et al. However that doesn’t imply there aren’t methods to extract the worth of your content material in an AI market. It begins with making your content material machine-readable, a separate idea from bot blocking.
The variety of locations publishers can market their content material to AI experiences is rising, together with Factiva and Microsoft’s Writer Content material Market, which Microsoft describes as a technique to help licensed entry to premium content material whereas preserving writer management, independence, and sustainable income. Lastly, constructing your individual agentic layer that reputable prospects or companions might entry through MCP (mannequin context protocol) is essentially the most forward-looking possibility.
That takes some technical sophistication, usually requiring a associate, nevertheless it offers the writer management over the expertise. Reasonably than counting on some third-party bot to scrape and interpret your content material, you’ll have accomplished that work already, so licensed exterior bots will get what your inner bot has already processed, like a relay. That’s the entire thought behind AI media initiatives like Reuters’ new MCP server, which permits prospects to look, retrieve, and use the Reuters content material they subscribe to inside AI workflows.
As I’ve argued before about publisher-built agents and AI-ready archives, when you’ve accomplished the onerous work of formatting, ingesting, and processing your archive for AI, the writer turns into one thing extra like a instrument vendor, not only a content material provider. Reuters clearly acknowledges that publishers who don’t construct some sort of managed retrieval layer over their archives are letting third-party crawlers set the foundations. Ultimately, that may result in negotiating from weak spot.
I haven’t stated a lot concerning the dangerous actors thus far—the unauthorized scrapers that hoover up big elements of what publishers produce after which promote that information in numerous black and grey markets. Actually, that is the place blocking ought to be inspired, and publishers want each dependable instruments for doing so in addition to broader ecosystem help, reminiscent of what Cloudflare has accomplished to higher establish bots and probably monetize their exercise. As I wrote earlier this year, unauthorized AI crawling is rampant, and publishers want greater than wishful pondering and a robots.txt file to take care of it.
From protection to leverage
However blocking is simply the defensive half of the technique. The reply is to not throw the doorways open or nail them shut. It’s to construct a gate with guidelines. Let the retrieval bots see sufficient to determine your authority. Preserve the coaching crawlers and unauthorized scrapers away from something they haven’t any proper to soak up. Use snippets, metadata, paywalls, and price limits to separate discovery from entry. After which do the more durable work: Make your archive clear, structured, machine-readable, and out there by channels you management.
Most publishers will not be going to get the large AI licensing deal. Which is okay. That was by no means going to be a marketing strategy anyway. The higher play is to make your work legible to AI techniques with out making it free, so when the market does come in search of trusted solutions, you’re not begging to be included. You’re already the gate.
{“blockType”:”mv-promo-block”,”information”:{“imageDesktopUrl”:”https://pictures.fastcompany.com/picture/add/f_webp,q_auto,c_fit/wp-cms-2/2025/03/media-copilot.png”,”imageMobileUrl”:”https://pictures.fastcompany.com/picture/add/f_webp,q_auto,c_fit/wp-cms-2/2025/03/fe289316-bc4f-44ef-96bf-148b3d8578c1_1440x1440.png”,”eyebrow”:””,”headline”:”u003Cstrongu003ESubscribe to u003Cemu003EThe Media Copilotu003C/emu003Eu003C/strongu003E”,”dek”:”Need extra about how AI is altering media? By no means miss an replace from Pete Pachal by signing up for u003Cemu003EThe Media Copilotu003C/emu003E. To be taught extra, go to u003Ca href=u0022https://mediacopilot.substack.com/u0022u003Emediacopilot.substack.comu003C/au003E”,”subhed”:””,”description”:””,”ctaText”:”SIGN UP”,”ctaUrl”:”https://mediacopilot.substack.com/”,”theme”:{“bg”:”#f5f5f5″,”textual content”:”#000000″,”eyebrow”:”#9aa2aa”,”subhed”:”#ffffff”,”buttonBg”:”#000000″,”buttonHoverBg”:”#3b3f46″,”buttonText”:”#ffffff”},”imageDesktopId”:91453847,”imageMobileId”:91453848,”shareable”:false,”slug”:””,”wpCssClasses”:””}}
