llms.txt
A proposed standard file placed at the root of a domain (e.g. example.com/llms.txt) that tells AI language models which pages and content on a site are most relevant to read and index.
Also known as: LLMs.txt, llms-txt, AI crawler instruction file
What is llms.txt?
llms.txt is a proposed standard for websites to communicate directly with AI language models and crawlers. Similar in concept to robots.txt (which tells search engine crawlers which pages to crawl) and sitemap.xml (which lists pages for indexing), llms.txt is intended to help AI systems identify which content on a site is most authoritative, most relevant, and most appropriate for training, indexing, and citation. The file is placed at the root of a domain and written in a structured Markdown format.
What goes in an llms.txt file
The llms.txt specification, proposed by Jeremy Howard, defines a structured format that includes: a brief description of the site, links to key documentation pages, links to the most important content, and optional instructions about what should not be used for training. The goal is to reduce the noise that AI crawlers encounter when indexing a large site, directing their attention to the most valuable and authoritative content first.
llms.txt vs robots.txt
- robots.txt: Tells search engine crawlers which pages to crawl or not crawl. Primarily affects indexing. Supported by all major search engines.
- llms.txt: Tells AI language model crawlers which content is most relevant and authoritative. Affects AI citation and training. Support varies by AI engine.
- sitemap.xml: Lists all pages on a site for search engine indexing. Complementary to both robots.txt and llms.txt.
Should you add llms.txt to your site?
As of 2026, llms.txt is a proposed standard rather than a universally adopted one. Some AI crawlers support it; others do not yet. However, adding an llms.txt file is a low-effort, zero-risk action that may improve how AI systems prioritise your content as adoption increases. For sites with large amounts of content or complex information architectures, llms.txt provides a clear signal about which pages should carry the most weight for AI citation purposes. Minineo's LLM.txt Generator tool creates and validates your llms.txt file automatically.
FAQ
Common Questions
No. llms.txt is a community-proposed standard, not an official Google or W3C specification. It is not the same as robots.txt and is not required for Google Search. Its adoption is growing among AI-focused companies and tools.
No. llms.txt is not read or processed by Google Search's indexing system. It is specifically for AI language model crawlers. It has no positive or negative effect on traditional Google rankings.
llms.txt uses a structured Markdown format. It typically includes a title (H1), a brief description paragraph, and H2 sections listing the most important pages as Markdown links, optionally with a brief description of each page.
llms.txt can signal which content should not be used for training, but there is no enforcement mechanism. Compliance depends on whether the AI crawler respects the instructions, which is voluntary.
Glossary