AI search and visibility

How AI Crawlers Read Your Site: Prerendering, llms.txt and Markdown

Most AI crawlers fetch HTML and do not run your JavaScript. If your content only appears after a script runs, they see an empty page. Prerendering fixes that, and a few small files make the site easier to read still.

By Cloud Technopix · Published 5 June 2026 · 11 min read

What a crawler actually receives

When a crawler requests your page, it receives whatever your server returns. For a traditional website, that is complete HTML with the text in it. For a single-page application built with a framework such as React, the server often returns a nearly empty HTML shell and a script. A browser runs the script, which builds the page. Many crawlers, including a number used by AI products, do not run that script, so they see the shell and nothing else.

This is the most common reason a good site is invisible to AI search: the content exists for people, but not in the response a bot reads. The fixes are well understood, and most are inexpensive.

How to see what a crawler sees

Request your page without running scripts and read the result. A simple way is to fetch it with a command-line tool, or to use your browser's view-source option, which shows the HTML the server sent rather than the page after scripts ran. If your headings and body text are missing from that source, a non-rendering crawler cannot read them either.

Fix 1: serve real HTML

There are three ways to get content into the first response.

Three ways to put real content in the first response.
ApproachHow it worksGood for
Server-side renderingThe server builds the HTML for each requestHighly dynamic, personalised pages
Static generationPages are built ahead of time into HTML filesContent that changes at deploy time
PrerenderingA built single-page app is visited by a headless browser and each page is saved as static HTMLExisting single-page apps with mostly public content

Our own marketing site is a React application, and we prerender it at build time: a headless browser visits each public route and saves the finished HTML. Visitors still get the interactive app, while crawlers receive the full page. It also lets the build fail if a page is missing its heading, title, description or required structure, so a regression is caught before deployment.

Fix 2: return correct status codes and canonicals

A crawler decides what exists from status codes. A missing page should return a real 404, not a 200 with a "not found" message, which teaches crawlers that your site has endless duplicate pages. Each page should declare one canonical address, matching the address it is served on. And your www and non-www versions, and http and https, should redirect to one version.

Fix 3: robots rules that say what you mean

A robots file tells crawlers what they may fetch. Be explicit about the crawlers you want and keep private areas out. Some sites also publish content-use signals, which state whether content may be used for search, for AI answers or for training, in a machine-readable line. This is a policy choice, and the point is to make it deliberate rather than accidental. Keep in mind that robots rules are requests that well-behaved crawlers follow, not access control, so anything truly private needs authentication.

Fix 4: describe the site with structured data

Structured data in JSON-LD states facts in a form machines read directly: that you are an organisation with a name, address and contact point; that a page is an article with an author and a date; that a page is part of a site. Keep it consistent with the visible content, link the nodes together by identifier, and validate it. We check this in our build, failing it if the structured data does not parse or references something that is not defined.

Fix 5: offer a plain-text and markdown version

A page full of navigation, scripts and layout is noisy for a model. Many sites now offer a stripped-down version of each page in markdown, and a single file listing the important pages with short descriptions, commonly called llms.txt. These are conveniences, not a ranking factor, and not every crawler uses them. They cost little to produce when generated at build time from the same content as the pages, so they cannot drift out of date.

  • A markdown twin of each page, linked from the page's head with an alternate link
  • A short llms.txt describing who you are and listing your key pages
  • A larger text file with the content of the whole site for tools that want everything in one request

Fix 6: a sitemap that is honest

A sitemap lists the pages you want indexed and when they last changed. Generate it from the same route list as your build, set last-modified dates from real change history rather than the build time, and include only pages that return 200 and are indexable. An accurate sitemap helps crawlers prioritise, and a sloppy one trains them to ignore it.

Fix 7: keep it fast and stable

Crawlers have budgets. Pages that respond slowly, time out or change address often get fetched less. Serve static files from a content delivery network, compress assets, and avoid chains of redirects.

A short checklist

  1. View source on a key page. Is the main content there?
  2. Does an unknown address return 404, and does each page have a single canonical?
  3. Does robots.txt allow the crawlers you want and exclude private areas, with your policy stated?
  4. Is structured data valid, consistent and linked?
  5. Is there a markdown version and a guide file for AI readers, generated from the same source as the pages?
  6. Is the sitemap accurate and referenced in robots.txt?

Test it continuously

These properties regress silently. A new page forgets its title, a deploy breaks a redirect, a template change removes a heading. Put checks in the build and a short smoke test against the deployed site, so problems are caught the same day. That is how we keep our own site honest, and it is part of the work we do in our web application development and digital marketing and SEO services. For the strategy behind it, read what GEO is, or reach out through the contact page.

Do all AI crawlers ignore JavaScript?

Behaviour varies by crawler and changes over time. The safe approach is to serve complete HTML so you do not depend on a crawler running your scripts.

Will llms.txt improve my rankings?

There is no guarantee any engine uses it. Treat it as a low-cost aid to machine readers, not a ranking lever.