The web’s top user? Bots

In June, bot traffic surpassed human traffic online for the first time ever.  Cloudflare CEO Matthew Prince posted the news on X June 3, citing data from the company’s Radar “Bot vs. Human” real-time tracker tool, which currently shows that bots are responsible for 57.6% of HTTP requests, compared to 42.4% for humans.

“Welp, that happened faster than I predicted,” Prince wrote. “Thought it would be end of 2027, then early 2027.” But maybe don’t whip out the Cylon Detector on your online friends quite yet. According to the panel on the latest Mixture of Experts podcast, these numbers may not mean what you think they mean.

“There’s a lot of asterisks on that [ratio],” said  Rynne Whitnah, IBM AI Ecosystem Technical Lead, on the podcast. She said that, in a manner of speaking, “bots have outnumbered humans on the internet for quite a long time,” in the form of web scrapers, AI indexing, search engine indexing and direct requests to websites. “I think that’s a lot of what [Cloudflare Radar] is capturing here,” she said. According to Whitnah, AMP, the Google feature that caches web pages and serves them to the user, is likely driving down the human traffic stats too. “[The high bot user percentage] is a bit of a shift, but not a seismic one.”

Sandi Besen, AI Research Engineer and Ecosystem Lead at IBM Research, suggested that the alarmingly low human HTTP request figures are less about humans being less represented on the web and more about “how the way UX is shifting, and the way [humans] interact with UX is shifting. If you search something through your research agent, it’s hitting, like, 20 searches,” she said. “It doesn’t mean the agent is interacting with them all in the same way.”

Furthermore, the Cloudflare Radar data doesn’t necessarily account for the fact that information is formatted differently now from in the past. For example, “a lot of doc sites now have an llm.txt file, so the LLM doesn’t have to parse the HTML,” she said. Just because an llm.txt makes the text easier for a bot to read for the purpose of AI indexing, it doesn’t follow that it was a bot rather than a human who made the request, she explained.

Which is not to say that the bots will not become more dominant one day, the panelists noted, particularly given the unknown future of agentic commerce. “Right now, if I shop on websites, I have to browse through 10 different shoes, look at the pricing, get some offers, put something in the cart, ship it,” Ambhi Ganesan, AI Practice Lead at IBM Consulting, said on the podcast. “The holy grail promise is there is a world in which agents can do that.” When it does, he said, that may change the bot-to-human ratio, though he also expects that the future of the online advertising monetization model will have “a profound impact”—and more immediately than agentic commerce. “How you serve up ads is a very, very tricky and very fast-evolving situation.”

Whitnah expanded on that, saying that the increased involvement of bots in HTTP requests raises the question of who and/or what pays, and gets paid, for internet content. “Ads matter because a lot of the content on the internet we care about, like news, is funded by advertising,” she said. But if AI agents become the primary consumers of news content, who are businesses advertising to? “The model for that has to change,” she said, which will likely include the expansion of paywalled premium content, which is for human users. “I don’t know where that will land,” Whitnah said. “We have not yet figured out a modern funding model for how an agent [pays] back places where it’s getting content from.”