Google now displays handy, artificial intelligence-based answers at the top of its search results pages — meaning users may never click through to the sites whose data is used to produce those results. But many website owners say they can’t afford to stop Google’s AI from summarizing their content.
That’s because the Google tool that crawls web content to find its AI answers is the same one that tracks web pages for search results, the publishers say. Blocking Google, just as websites have blocked some of their AI competitors, would also hurt a site’s ability to be found online.
Google’s dominance in search – which a U.S. federal court ruled last week was an illegal monopoly – gives the company a decisive advantage in the burgeoning artificial intelligence war, which search engine startups and publishers say is unfair as the industry takes shape.
The dilemma is particularly acute for publishers, who are faced with the choice of either making their content available to AI models that could render their websites obsolete or disappearing from Google search, one of the most important sources of traffic.
“It will be like an existential crisis for these companies,” said Joe Ragazzo, editor of the news site “Those are two bad options. You get out and die immediately, or you partner with them and probably die slowly because at some point they won’t need you anymore either.”
According to Google, AI Overviews – the summaries that appear at the top of Google Search – are part of the company’s long-standing commitment to providing higher quality information and improving opportunities for publishers and other companies.
Since its inception, Google has used software called Googlebot to visit, or “crawl,” millions of websites, creating a detailed index of the global Internet. Over the years, this index has represented a formidable barrier to entry for companies trying to build competing search engines – even those with deep pockets like Microsoft.
The rise of generative AI has sparked a new wave of startups seeking to offer search products that use AI models to provide succinct answers to users’ questions. But before these startups can pose a real threat to the search giant, they need to scour the internet—and that’s no easy task.
Crawling costs website owners money, processing power and storage space, so many publishers include a file that sets rules for bots visiting their sites. The companies that have the most leeway are usually Google and Microsoft’s Bing, which can drive traffic to websites through their search engines.
The artificial intelligence boom has seen a number of deals between media companies and startups, but Google has been a notable outsider so far. With the exception of a reported $60 million deal with Reddit, Google has signaled to publishers behind closed doors that it is not interested in negotiating, according to two people familiar with the matter who asked not to be identified because the information is confidential.
Google’s agreement with Reddit provides the company with a treasure trove of information for AI models. The deal coincided with changes Google made to increase the presence of results from forums like Reddit in search results, leading to a huge increase in traffic to the social media site.
Media companies have little say in the conversations with these tech giants. Earlier this year, Google introduced AI Overviews, where the company uses AI to provide succinct answers to some of users’ questions at the top of the search page. Publishers were immediately concerned about the impact the answers could have on their traffic, but had no clear way to allay those fears.
Google has stated that publishers can block certain pages or parts of pages from appearing in the AI summaries of search results, but doing so would also likely prevent those snippets from appearing in all other Google search features, including web link listings.
Many publishers, who often rely on search engines for at least half of their traffic, are unwilling to take the risk of reducing their reach.
Google’s position “underestimates the significant risk this poses to content creators, particularly those who rely on visibility in search results for their livelihoods,” said Marc McCollum, head of innovation at Raptive, a company that represents publishers and influencers.
“By taking down content, creators may inadvertently reduce their overall search presence, which could impact their ability to reach their audience and generate revenue.”
Kyle Wiens, chief executive of iFixit, a website that publishes free online repair guides for consumer electronics, said the site’s relationship with Google is “much tenuous” than with other AI companies.
“I can block ClaudeBot from indexing us without hurting our business,” Mr. Wiens wrote in an email, referring to the bot from generative AI startup Anthropic. “But if I block Googlebot, we lose traffic and customers.”
Regardless of the outcome of the antitrust case against the tech giant, the changes currently taking place in the search landscape underscore how important it is for publishers to take control of their own destiny and not rely too heavily on any single technology platform – including Google, Ragazzo said.
“We believe in building a real relationship with readers,” says Ragazzo. “This creates a publication that can transcend different eras.”
