How AI Agents Actually Browse the Web (And Why Most Sites Are Hostile to Them)

Right now, somewhere, an AI agent is opening a webpage. Maybe it is pulling raw HTML to answer a question someone typed into a chat window. Maybe it is driving a real Chrome tab, scrolling, clicking a date picker and filling in a form with someone's saved login. Either way it belongs to a shift Cloudflare put in plain numbers in July 2026: more than half of the traffic on the internet is now non-human, and 52% of crawler requests in June 2026 were for AI training, up from 22% in spring 2025.

The web was designed for humans. Nearly every assumption baked into how sites are built (layout, authentication, anti-bot defenses) assumes the thing on the other end has eyes, hands and patience. Agents break those assumptions, and the friction has become one of the more interesting problems in web infrastructure. Lately it is a business negotiation too.

What an AI agent actually does when it browses

Ask Claude, ChatGPT or Gemini something that needs current information and the model can't answer from training data alone. It has to go get the page. In 2026 three distinct kinds of machine visitors do that work, and they behave very differently.

Fetchers grab a page on demand when a user asks a question. OpenAI's is ChatGPT-User; Anthropic's is Claude-User. They request the HTML, reduce it to text and hand it to the model. They are fast and cheap, and they mostly don't run JavaScript. A late-2024 Vercel analysis of crawler traffic found that OpenAI's and Anthropic's bots download script files without executing them, while Gemini (riding on Googlebot's infrastructure) and Applebot do render pages.

Crawlers work ahead of time, building training sets and search indexes: GPTBot and OAI-SearchBot for OpenAI, ClaudeBot and Claude-SearchBot for Anthropic. Google and Apple handle AI training through separate robots.txt tokens, Google-Extended and Applebot-Extended, rather than separate crawlers.

Browser agents are the category that arrived in force this year. They run a real browser, see the rendered page and act on it. Anthropic opened computer use to developers as an API beta in October 2024, and on August 26, 2026 made Claude in Chrome generally available on every paid plan. It reads pages, clicks, navigates and fills forms using your existing logins, and it no longer needs approval for each action; a classifier checks every step against what you actually asked for.

Google's auto browse in Chrome handles multi-step chores for AI Pro and Ultra subscribers, first on desktop in the US and, since August 2026, on Android. OpenAI launched ChatGPT agent in July 2025 and the Atlas browser that October, then shut Atlas down on August 9, 2026 and moved browsing into the ChatGPT desktop app, where its ChatGPT Work agent uses a built-in browser. Perplexity's Comet has been free on desktop since October 2025 and now runs on Android and iOS as well.

Whatever the type, the loop looks like a human researcher on fast-forward: take a task, search, pick a result, fetch it, extract what matters, and try another source if it doesn't answer the question. A paywall or a blank page is just a signal to move on. The intelligence is in those decisions, not in the fetching.

How agents identify themselves

For most of the web's history a bot announced itself with its user-agent string, and that string is self-declared. Anyone can put "ClaudeBot" in a header. Operators back the claim with published IP ranges (Anthropic lists its crawler addresses in a public JSON file), but IP lists are clumsy for agents running on shared cloud infrastructure or inside a user's own browser.

The fix gaining ground is cryptographic. Under Web Bot Auth, a bot signs its requests with HTTP Message Signatures (RFC 9421) and publishes its public keys at a well-known URL, so a site can verify the sender instead of trusting a string. Cloudflare built a signed agents category on it in August 2025 for bots acting on behalf of individual users, with OpenAI's ChatGPT agent in the first cohort alongside Block's Goose, Browserbase and Anchor Browser. The IETF chartered a Web Bot Auth working group to standardize the approach; as of September 2026 its documents are still Internet-Drafts.

Identity also decides which rules apply. Anthropic says its bots honor robots.txt, the user-triggered Claude-User included. OpenAI's documentation says robots.txt rules may not apply to ChatGPT-User, because a person initiated the request. And getting caught dodging the rules now has a price. In August 2025 Cloudflare said Perplexity was using undeclared crawlers with rotating IPs to get around no-crawl directives, and removed it from its verified bot list.

Why most websites are still hostile to AI agents

Some of the friction is accidental, a side effect of how modern sites get built. A growing share of it is deliberate.

JavaScript-rendered content. React, Vue and Svelte apps often ship a nearly empty HTML document and build the page in the browser. People never notice. A fetcher that doesn't execute scripts sees a shell with nothing in it. Browser agents do render the page, but they are slower and costlier to run, so a lot of agent traffic still arrives as plain fetches.

Anti-bot infrastructure. CAPTCHAs, fingerprinting and behavioral scoring exist to separate humans from scripts, and a browser agent is a script with very good manners. From the agent's side the page simply never loads, and the task fails or detours to another source.

Default blocks at the CDN. Cloudflare has blocked AI training crawlers by default on new domains since July 2025. On September 15, 2026 it moved to per-purpose defaults built on three classes, Search, Agent and Training: for newly onboarded domains, Training and Agent bots are blocked on pages that display ads, while Search stays allowed. Crawlers that mix purposes get the strictest applicable rule, so Cloudflare says multi-purpose crawlers such as Googlebot, Applebot and Bingbot will be blocked by customers who choose to block Training. Plenty of sites now turn agents away without anyone there deciding to.

Payment walls. Login walls matter less than they used to, because tools like Claude in Chrome work inside the user's own signed-in browser. Payment is a different story. Cloudflare's pay per crawl, still in closed beta, answers AI crawlers with HTTP 402 Payment Required and a price, and in July 2026 the company opened a waitlist for a Monetization Gateway that charges for pages, datasets, APIs or MCP tools over the x402 protocol. For an agent with no way to pay, a 402 is as final as a CAPTCHA.

Cookie banners. Humans dismiss them on reflex. Fetchers get the banner text mixed into the content; browser agents have to find the right button on every site.

Unstructured HTML. Content buried in nested divs with generated class names, tangled up with ads and navigation, renders fine for people and parses as soup. A button with no accessible name or an unlabeled input confuses a browser agent as much as a screen reader.

What AI-friendly websites look like

Server-rendered primary content. Next.js, Nuxt, SvelteKit and Astro can all render on the server, so the first HTML response carries the actual text. Fetchers can read it and humans get a faster first paint. TerminalFeed's live dashboard is a client-side app, so the homepage ships a static text summary in its initial HTML, and reference sections like the glossary and the crypto pages are plain static HTML.

Markup an agent can operate. Real headings, real buttons, labeled inputs and a layout that doesn't jump around while it loads. Google made this measurable: Lighthouse now has an experimental Agentic Browsing category that checks accessible names and labels, layout shift, llms.txt and WebMCP support, scored as a plain ratio of checks passed.

An llms.txt file, with modest expectations. Jeremy Howard proposed the format in September 2024: a Markdown file at the site root with a title, a short summary and lists of links to the pages that matter. Adoption raced ahead of use. An Ahrefs study of 137,210 domains, published in June 2026, found that 28% publish one and that 97% of those files got zero requests in May 2026. Google says the file will neither help nor harm visibility in Google Search. It helps agents that are pointed at it directly, but crawlers don't go looking for it. TerminalFeed publishes one anyway, because it costs almost nothing to keep current.

Markdown on request. In February 2026 Cloudflare shipped Markdown for Agents for Pro plans and up. When a client sends Accept: text/markdown, Cloudflare converts the HTML to Markdown at the edge and reports an estimated token count in an x-markdown-tokens header. Less markup means fewer tokens and less for a model to misread.

Tools instead of scraping. The cleanest way to serve an agent is to skip the page entirely. The Model Context Protocol, released by Anthropic in November 2024 and donated to the Linux Foundation's new Agentic AI Foundation in December 2025, lets a site expose typed tools that an agent calls directly. ChatGPT, Claude, Gemini, Cursor and VS Code all support it as clients. The in-browser counterpart is WebMCP, a proposed standard for exposing JavaScript functions and forms to browser agents, which Chrome put into an origin trial starting with Chrome 149. TerminalFeed runs a hosted MCP server: an agent asks for the Bitcoin price or the latest earthquakes and gets structured data back with no HTML parsing involved. Setup lives on the MCP page.

A deliberate bot policy. Blanket blocking and blanket allowing are both ways of not deciding. Search crawlers that send visitors, agents acting for a real person and training crawlers that take content for model development are three different trades. Verify identity where you can (signed requests, published IP ranges) and rate-limit before you ban.

The traffic question

The old pitch for being AI-friendly was a flywheel: agents cite you, readers click through, more citations follow. Part of that still holds. An agent that can read your page can recommend it, and one that can't, won't.

The numbers around it are sobering, though. Cloudflare says Google still accounts for approximately 88% of referral traffic, and that some of the most heavily crawled categories have seen human traffic fall as much as 40% in less than a year. That is why the conversation moved from "let the bots in" to terms: pay per crawl, content signals in robots.txt, per-purpose defaults. Being readable by agents is still a good bet for most sites. Now it is one you make on purpose.

What you can do this week

Check what your CDN already decided. On Cloudflare, open AI Crawl Control and see which crawlers are blocked.

Write a robots.txt that names bots by purpose. GPTBot, ClaudeBot, Google-Extended and Applebot-Extended cover training. OAI-SearchBot and Claude-SearchBot cover AI search. ChatGPT-User and Claude-User are user-triggered fetches. Allow or disallow each one deliberately.

Server-render the pages that matter. Fetch one of your pages without JavaScript. If you get an empty shell, so do the fetchers.

Run Lighthouse's Agentic Browsing audit. It needs Chrome 150 or later. Fix unlabeled buttons and inputs first; screen reader users benefit from the same work.

Add an llms.txt if it is cheap for you. Keep it short and honest, and don't expect it to move traffic on its own.

If agents want your data or your actions, give them tools. A small MCP server puts you in control of exactly what gets exposed.

Mobile-friendliness went from a nice extra to table stakes once phones took over browsing. Agent-friendliness is on a similar path, with one difference: this time the site owner also decides who gets in and on what terms.