Machine-Readable Pages
Many AI crawlers do not run JavaScript the way a browser does. Content that only appears after scripts run, or that is trapped in images and PDFs, may never be read.
Serve the content in HTML
Use server-side rendering or static generation so the main text is in the HTML response. Test by viewing the page source (not the inspector): if your key answer is not in the source, a non-rendering crawler cannot see it.
Clean structure helps extraction
- One H1, question-style H2s, short paragraphs.
- Real lists and tables for steps and comparisons.
- Facts stated as text as well as in any image or chart.
- Stable URLs that do not change with tracking parameters.
About llms.txt
llms.txt is a proposed file at the root of a site listing the pages a language model should read, in Markdown. It is a community proposal, not a standard, and the major search providers have not said they use it. It is cheap to add and harmless, but it is no substitute for crawlable, indexed pages.
Pro tips
- Check view-source for your top pages after every redesign.
- Put key numbers in text next to any chart that shows them.
- Add llms.txt if you like, but treat it as optional.
Common mistakes
- Shipping a single-page app with no server rendering for marketing pages.
- Publishing price lists only as images or PDFs.
- Believing llms.txt alone makes a site visible to AI.
Serve key content in the HTML, structure it cleanly, keep facts as text, and treat llms.txt as an optional extra rather than a fix.