Skip to the lesson
0%

Machine-Readable Pages

Intermediate 10 min read Lesson 6 of 12

Many AI crawlers do not run JavaScript the way a browser does. Content that only appears after scripts run, or that is trapped in images and PDFs, may never be read.

Serve the content in HTML

Use server-side rendering or static generation so the main text is in the HTML response. Test by viewing the page source (not the inspector): if your key answer is not in the source, a non-rendering crawler cannot see it.

Clean structure helps extraction

  • One H1, question-style H2s, short paragraphs.
  • Real lists and tables for steps and comparisons.
  • Facts stated as text as well as in any image or chart.
  • Stable URLs that do not change with tracking parameters.

About llms.txt

llms.txt is a proposed file at the root of a site listing the pages a language model should read, in Markdown. It is a community proposal, not a standard, and the major search providers have not said they use it. It is cheap to add and harmless, but it is no substitute for crawlable, indexed pages.

What a non-rendering crawler sees
Client-side onlyServer-renderedHTML source is an empty shellHTML source contains the full textText appears after scripts runText is there on first responseMany AI crawlers see nothingEvery crawler sees the answerRisky for retrievalSafe for retrieval

Pro tips

  • Check view-source for your top pages after every redesign.
  • Put key numbers in text next to any chart that shows them.
  • Add llms.txt if you like, but treat it as optional.

Common mistakes

  • Shipping a single-page app with no server rendering for marketing pages.
  • Publishing price lists only as images or PDFs.
  • Believing llms.txt alone makes a site visible to AI.
Summary

Serve key content in the HTML, structure it cleanly, keep facts as text, and treat llms.txt as an optional extra rather than a fix.

Quick quiz

1. How can you check what a non-rendering crawler sees?
2. What is llms.txt?
3. Why avoid putting prices only in images?

Stuck on something in this lesson? Ask our team on WhatsApp. We answer course questions for free.

Ask on WhatsApp