Your Website Has Two Audiences Now: Humans and AI Agents

There is a new kind of visitor on your website, in two forms. One never opens a browser: it requests your HTML, strips it to text and hands it to a language model. The other drives a real browser, reads the rendered page, clicks your buttons and fills in your forms for someone who asked it to. Both are AI agents, and if your site isn't ready for them, a growing share of your traffic leaves with nothing useful.

This is not a forecast. Cloudflare reported in July 2026 that more than half of internet traffic is now non-human. Claude in Chrome went generally available on Anthropic's paid plans in August 2026, and Google's auto browse runs inside Chrome for AI Pro and Ultra subscribers in the US. The question is no longer whether agents will visit. It is whether your site gives them anything they can use, and on what terms.

What AI Agents Actually Do

An AI agent is software that pursues a goal instead of waiting for the next prompt. It searches, compares sources, fills in forms and calls tools. Some book travel. Some watch data feeds and alert a human when something changes.

From your server's side they arrive in three shapes. Training crawlers such as GPTBot and ClaudeBot collect content for model development. Search crawlers such as OAI-SearchBot and Claude-SearchBot build the indexes behind AI search answers. User-directed agents (ChatGPT-User, Claude-User, and browser agents running in a person's own Chrome) fetch a page because a human asked something right now. The TerminalFeed Agent Tracker follows more than 50 of these agents across seven categories.

Decide Who Gets In

First decide which agents you want. robots.txt is still the main lever, and the big operators now split their bots by purpose. Here is one site's choice, keeping content out of model training while welcoming AI search and live fetches:

# Training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# AI search and user-triggered fetches
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-User
Allow: /

Two caveats. OpenAI says robots.txt rules may not apply to ChatGPT-User, because a person triggered the request, while Anthropic says its bots, Claude-User included, honor robots.txt. And Google-Extended and Applebot-Extended are control tokens, not crawlers: Google states that blocking Google-Extended does not affect inclusion or ranking in Google Search.

To say more than yes or no, Cloudflare's Content Signals Policy adds a robots.txt line such as Content-Signal: search=yes, ai-train=no, with a third signal, ai-input, for real-time use in AI answers. Cloudflare is explicit that these are preferences, not enforcement. The IETF's AI Preferences working group is standardizing a vocabulary for the same idea; its documents are still drafts. Then check your CDN: since September 15, 2026, Cloudflare's defaults for newly onboarded domains block Training and Agent bots on pages that display ads.

The llms.txt Standard

llms.txt, proposed by Jeremy Howard in September 2024, is a Markdown file at your site root that tells a language model what the site is and where the important content lives. The format is short: an H1 with the site name, a blockquote summary, then H2 sections listing links. A minimal one looks like this:

# Example Recipes
> Tested weeknight recipes with ingredient lists, timings and nutrition data.

## Recipes
- [Recipe index](https://example.com/recipes.md): every recipe, grouped by cuisine
- [Pantry guide](https://example.com/pantry.md): substitutions and storage

Be realistic about what it does. An Ahrefs study of 137,210 domains found that 28% publish an llms.txt and that 97% of those files got zero requests in May 2026. Google's guidance says you don't need new machine-readable files, AI text files or Markdown to appear in Google Search, and that llms.txt will neither help nor harm your rankings. The file is cheap, it helps agents that are pointed at it directly, and it won't move your search traffic. TerminalFeed keeps one at the root describing the site, its data categories and its MCP server.

Structured Data Still Matters

JSON-LD has been a search recommendation for years. Google says structured data isn't required for its generative AI features and that no special schema.org markup exists for them; it still powers rich results in regular search. The case for it with agents is simpler. A JSON-LD block tells any parser who wrote a page, when it was published and updated, and what kind of thing it is, with no guessing from the layout.

Every TerminalFeed article carries Article and breadcrumb markup with author, publish date and modified date. Keep it honest: bump dateModified only when the content actually changed.

Clean HTML Is Agent UX

Many fetchers still don't run JavaScript, so a client-rendered page can look empty to them. Server-render what matters. For browser agents the page also has to be operable: real headings, buttons that are buttons, inputs with labels, and a layout that stays put while it loads. Lighthouse's experimental Agentic Browsing category (Chrome 150 or later) turns that into a checklist, testing accessible names and labels, layout shift, llms.txt and WebMCP support. Screen reader users benefit from the same fixes.

On Cloudflare's Pro plan or above, Markdown for Agents serves a Markdown version of any page, converted at the edge, to clients that send Accept: text/markdown.

Give Agents Tools, Not Just Pages

For a site with live data or actions, the strongest move is to skip scraping entirely. The Model Context Protocol lets you publish typed tools an agent can call: get the current price, search the catalog, book the slot. Anthropic released MCP in November 2024 and donated it to the Linux Foundation's Agentic AI Foundation in December 2025; ChatGPT, Claude, Gemini, Cursor and VS Code all support it as clients. Chrome is also testing WebMCP, a proposed standard for exposing a page's JavaScript functions and forms to browser agents, in an origin trial that began with Chrome 149.

That is the approach TerminalFeed takes. People get the live dashboard. Agents get llms.txt, a robots.txt that explicitly welcomes the major AI crawlers, and a hosted MCP server, listed on the official MCP Registry, whose free tools return the Bitcoin price, the Fear & Greed Index, prediction market odds, recent earthquakes and service status. TerminalFeed no longer promotes a public API; for agents, the tools are the interface. Setup is on the MCP page, and the MCP quickstart covers building your own.

Practical Steps You Can Take Today

• Audit your bot settings. Check your CDN's AI crawler controls, then write robots.txt rules per purpose: training, AI search and user-triggered fetches.
• Server-render primary content. Load a key page with JavaScript off. If it comes back empty, that is what the fetchers get.
• Fix names and labels. Every button, link and input needs an accessible name. The Lighthouse Agentic Browsing audit will show the gaps.
• Keep JSON-LD accurate. Article, Organization and WebSite schemas cover most content sites.
• Add llms.txt if it's cheap. Short, honest, pointing at your best pages.
• Offer tools if you have data or actions. An MCP server, or an OpenAPI spec for an API you already run, beats hoping agents scrape you correctly.

The Web is Becoming Bilingual

The web spent three decades optimizing for people using browsers. Now it needs two languages. Human-readable pages for people who browse, click and scroll. Machine-readable structure and tools for agents that parse, query and act.

What changed this year is that the second audience stopped being one crowd. Some agents sign their requests, some are asked to pay, some ignore your rules, and your CDN may already be sorting them for you. Sites that speak both languages, and set clear terms for who gets what, will be the ones agents can actually use.