NEW YORK, Dec. 17 — OpenAI on Wednesday expanded the rollout of its SearchGPT crawler, OAI-SearchBot, to all eligible web publishers, a move that several major news organizations said they would block by default, arguing the company has not established acceptable terms for the use of their content in AI-generated answers.

The expansion, which OpenAI announced in a brief blog post, makes SearchGPT — the AI-search product the company has tested with select users since 2024 — broadly available to ChatGPT subscribers, with web results drawn from the OAI-SearchBot index rather than from a third-party search partner. OpenAI said publishers can opt out via a User-agent: OAI-SearchBot directive in their robots.txt file but framed the default state as inclusion.

“Publishers retain full control over whether their content appears in SearchGPT results, and we have built straightforward tools for both opt-out and revenue-share participation,” an OpenAI spokesperson said in a written statement. “The expanded rollout is designed to give SearchGPT users meaningfully better source coverage.”

Publisher response

The News/Media Alliance, the trade group that represents most major U.S. news publishers, issued a statement Wednesday afternoon calling the rollout “structurally premature.” Citing unresolved litigation between several of its member organizations and OpenAI, the Alliance said its members would, by default, configure their robots.txt files to block OAI-SearchBot pending negotiated terms.

“The position of our membership is that opt-out is not consent,” said Rosalind Boateng-Velasquez, the Alliance’s executive vice president, in an interview with Consumer Tech Wire. “Compensation and provenance need to be settled before a crawler is in market at this scale, not after.”

The New York Times, which sued OpenAI in late 2023 over training-data and answer-attribution practices, confirmed it would block OAI-SearchBot. A Times spokesperson said the company’s position is “consistent with our existing litigation posture.” Condé Nast, which has a paid licensing arrangement with OpenAI, said its content would continue to appear in SearchGPT results under the existing agreement.

The economics question

Cloudflare, which provides the network-infrastructure layer for a substantial fraction of the open web, said in a research note this week that AI-related crawler traffic now accounts for roughly 1.8 percent of all bot traffic across its network, up from 0.3 percent in early 2024. The company said OAI-SearchBot is the second-most-active AI crawler on its network, behind Google’s GoogleBot but ahead of Anthropic’s ClaudeBot.

“The economic question that has been deferred for two years is now in front of every publisher in the world,” said Cassius Yamamoto-Reilly, an analyst at the media research firm Toolkit. “The OpenAI rollout makes that question concrete in a way that the more limited SearchGPT pilot did not.”

OpenAI said it has signed publisher revenue-share agreements with more than 30 outlets, including The Atlantic, Axel Springer, and the Financial Times. The company declined to disclose aggregate revenue-share payouts to date.

A spokesperson said OpenAI plans to publish a publisher-payments transparency report in the first quarter of 2026.

Reginald Rosenberg-Vance is editor in chief of Consumer Tech Wire.