The page you see in a browser and the page an AI crawler sees can be two different documents. Browsers run your JavaScript, wait for your API calls and assemble the result; most AI fetchers read the raw HTML response and leave. If your content lives in the second render, to them it does not live at all.
Here is who the crawlers are, the four checks that show exactly what they see, and the fixes when the answer is "nothing".
Meet the cast
| User agent | Who | It fetches for |
|---|---|---|
| GPTBot | OpenAI | Model training |
| OAI-SearchBot | OpenAI | ChatGPT's search index |
| ChatGPT-User | OpenAI | Live fetches during user chats |
| PerplexityBot | Perplexity | Its answer engine |
| ClaudeBot | Anthropic | Training and fetching |
| Google-Extended | A control for AI training use |
The distinction that matters: training bots read for the memory door, search and user fetchers read for the retrieval door, the split from the GEO explainer. Different bots, different consequences, which is why blocking decisions deserve per-bot thought.
The JavaScript gap
Google's crawler renders JavaScript at scale; most AI fetchers do not, or do so far less reliably. A client-rendered site can rank in classic search and still be a blank page to the systems writing answers.
The four checks
Check 1: curl the truth. Fetch your page from a terminal and read what comes back: is your main content, your headings, your table actually in the HTML? That raw response approximates the crawler's whole experience. Add the crawler's user agent string to also catch agent-specific behavior from your stack.
Check 2: the no-JS view. Disable JavaScript in a browser tab and reload your key pages. Whatever survives is your machine-readable site; whatever vanished is invisible to most of the cast above.
Check 3: read your logs. Search access logs for the user agents in the table. You learn three things fast: whether AI bots visit at all, which pages they like, and whether anything is silently blocking them, a 403 pattern that often traces to CDN bot rules nobody configured on purpose.
Check 4: ask the assistant. In a search-grounded chat, ask "what does this page say" with your URL. The answer shows you the live-fetch door working or failing in real time, and the failure message usually names the reason.
The fixes, in priority order
Server-render the content that matters: the article text, the specs, the prices, the FAQ, present in the HTML response itself, with client-side flourish layered on top rather than underneath.
Keep the quotable elements as real HTML: text as text, tables as table markup, headings as headings, dates visible. Content drawn into images, canvases or JS-only widgets forfeits its extractability, which was the whole prize.
And audit the gatekeepers: CDN and firewall bot protections that challenge unknown agents catch AI fetchers constantly. Verify the bots you want are allowed through, because a security default can quietly cost the channel.
The quarterly crawler check
Fifteen minutes: curl the top five pages, one no-JS reload, one log grep for the cast, one live ask. New templates, new widgets and new CDN rules all reintroduce the gap, so the check earns its recurrence, and the referral segment confirms downstream that fetched pages are becoming cited ones.
The one-line takeaway: AI crawlers read your raw HTML, not your rendered app. Curl the page, reload without JavaScript, grep the logs for GPTBot and friends, and ask the assistant directly; then server-render what you want quoted and let the bots you want through the firewall.