AI Crawlers and JavaScript Rendering: The Invisible Site
The invisible content problem, in one sentence
Your page can look perfect in a browser and still be an empty shell to the bots that feed AI answers, because most AI crawlers download your HTML and stop — no JavaScript execution, no rendering, no waiting for your framework to hydrate.
That's the whole problem. Everything else in this article is detail. But the detail matters, because if your site is a React, Vue, or Angular single-page app that builds its content in the browser, GPTBot and its peers are reading a page that says little more than "loading…" and a bundle reference. You can have flawless content, generous crawl permissions, immaculate schema markup — and none of it exists as far as ChatGPT is concerned, because the words were never in the HTML it fetched.
Why AI crawlers don't render JavaScript
Rendering is expensive. Running a headless browser for every fetched URL multiplies compute cost by an order of magnitude or more, and AI crawlers operate at a scale where that math is brutal. Google spent roughly two decades building its rendering pipeline into something that works; the AI labs, crawling billions of pages on much younger infrastructure, mostly skipped it.
As of August 2026, the practical picture looks like this. OpenAI's crawlers (GPTBot, OAI-SearchBot, ChatGPT-User) fetch raw HTML and do not execute JavaScript. Anthropic's ClaudeBot behaves the same way. PerplexityBot, per Perplexity's own documentation, does not run scripts either. The one real exception is Google: Googlebot renders JavaScript through its Web Rendering Service, and both AI Overviews and Gemini's grounding draw on that rendered index. So a client-side app isn't equally invisible everywhere — which is exactly why partial visibility fools people. A brand sees itself in AI Overviews, assumes everything is fine, and never learns that ChatGPT and Perplexity are reading blank pages.
One honesty note: crawler behavior is among the fastest-moving claims in this field. Some AI companies have experimented with rendering pipelines, and this paragraph may age badly. Treat any "bot X doesn't render JS" claim — including ours — as dated the moment you read it, and verify against your own pages rather than trusting a blog post's snapshot.
What this means for SPAs and React sites
Not every JavaScript-heavy site has this problem, and this distinction gets muddled constantly. The question is not "does my site use React?" It's "where does the HTML get assembled?"
If your framework renders on the server or at build time — Next.js with SSR or static generation, Nuxt in universal mode, Astro, SvelteKit with prerendering, or a classic CMS like WordPress or Shopify's storefront — the content arrives as complete HTML and AI crawlers read it fine. React itself is innocent.
The trouble starts with pure client-side rendering: Create React App builds, Vite SPAs, Angular without Angular Universal, or any setup where the server ships a near-empty <div id="root"> and the browser does the rest. It also hides in half-migrated sites, which are sneakier. The marketing pages are server-rendered and test clean, while the pricing table is fetched client-side from an API, the product specs live in a JS-driven tab component, and the FAQ accordion pulls its answers on click. The page is 80% visible, and the 20% that's missing is the part an AI engine would actually quote when someone asks what you cost.
Pricing, comparison tables, and FAQs are precisely the content AI assistants get asked about, and they're precisely the components teams most often build client-side. That overlap is the invisible-content problem in its most expensive form.
How to check what AI crawlers see — curl vs browser
The test costs nothing and takes two minutes. Fetch your page the way a non-rendering bot does:
curl -A "GPTBot" https://yoursite.com/pricing
Then search the output for a sentence you know is on the page — a price, a product name, an FAQ answer. If it's in the raw HTML, non-rendering crawlers can read it. If it's not, they can't, full stop.
Two lower-effort variants: in your browser, use "View page source" (Ctrl+U / Cmd+U) — not DevTools' Elements panel, which shows the DOM after JavaScript ran and will happily lie to you about this. Or disable JavaScript in DevTools and reload; whatever survives is what most AI bots get.
While you're at it, watch the HTTP status codes. A page that returns 200 with empty content and a page that returns 403 to bot user-agents are different failures with different fixes — the second is an access problem, and we've covered that side in our guide to robots.txt for AI crawlers. Check both before touching your architecture, because rendering work is wasted if a firewall rule is blocking the bots anyway.
Fixing JavaScript rendering for AI crawlers, ranked by effort
There are four realistic responses, and the right one depends on how much of your content is affected and how much engineering appetite you have. In ascending order of pain:
1. Verify you even have the problem — zero effort. Run the curl test on your five most important pages before planning anything. A surprising number of "we need SSR" projects die here, pleasantly, because the content was in the HTML all along.
2. Static generation for content pages — low effort, high return. Your blog, docs, pricing, and comparison pages rarely need to be dynamic. Rendering them to static HTML at build time — whether via your existing framework's static export or by moving those routes to Astro or Eleventy — fixes the visibility problem where it matters most, without touching the application itself. For most SaaS sites, this is the move: keep the app behind login as an SPA (crawlers can't log in anyway) and make everything public static.
3. A prerendering layer — medium effort, real caveats. Services and middleware in the Prerender.io mold detect bot user-agents and serve them a pre-rendered snapshot while humans get the SPA. It works, and for a large legacy SPA it's often the only affordable option. But you're now maintaining a user-agent list in a period when new AI bots appear monthly, your snapshots can go stale, and Google itself walked back its old "dynamic rendering" advice, calling it a workaround rather than a long-term solution. We'd treat prerendering as a bridge, not a destination.
4. Full server-side rendering — high effort, permanent fix. Migrating a client-side app to Next.js, Nuxt, or SvelteKit with SSR solves the problem at the root for every current and future bot, renderer or not. It's also weeks or months of engineering and a real hosting-complexity increase. Worth it if your public content and your app are deeply entangled; overkill if option 2 gets you 90% of the value in a week.
The counter-argument deserves a fair hearing: if nearly all your traffic and revenue come from logged-in product usage or paid channels, AI-crawler visibility may genuinely not justify a rewrite. Not every site needs this fix. But make that call with the curl output in front of you, not on the assumption that bots see what you see.
A made-up example of how this plays out
A hypothetical illustration — no real company behind it. A B2B analytics startup ships its site as a Vite SPA. Google indexes it fine (Googlebot renders), so organic traffic looks healthy and nobody suspects a thing. Meanwhile every ChatGPT and Perplexity answer about their category recommends competitors — including one with objectively thinner content that happens to run WordPress. The founder assumes the competitors are doing sophisticated GEO. The actual difference: their HTML contains their content, and the startup's contains a script tag. The fix was moving six marketing pages to static generation; nothing sophisticated, roughly a week of work. The pattern generalizes: when a "worse" competitor consistently outperforms you in AI answers, suspect plumbing before strategy.
Test before you build anything
Ordering matters here: measure, then fix, then measure again. The whole failure mode of this problem is that it's invisible from a browser, so the only trustworthy view is the bot's view. You can get it in ten seconds with our crawler simulator — it fetches any URL as GPTBot, ClaudeBot, or PerplexityBot and shows you exactly what came back: status code, raw HTML, and whether your key content is actually in it. If your pricing page passes that test, JavaScript rendering is not your problem. If it fails, you now know precisely which pages to fix, and everything above tells you how.
This is one piece of the broader technical GEO checklist.