What llms.txt is
llms.txt is a proposed convention for placing a Markdown file at the root of a website with a concise description and links to important resources. The proposal is intended to help language-model workflows find a curated set of content. It is not an established replacement for web crawling, search indexing, robots.txt, or XML sitemaps.
The proposal can be useful as a readable directory for humans and tools that deliberately request it. That does not prove that a major answer engine reads the file, uses it to select sources, or rewards the domain that publishes it.
What the current evidence says
| Evidence | Supported conclusion | Not supported |
|---|---|---|
| SE Ranking analysis of nearly 300,000 domains | No observed correlation between llms.txt and domain citation frequency in its dataset. | It does not prove the file can never be used by any tool or future system. |
| OpenAI publisher guidance | Allowing OAI-SearchBot is an access condition for content to be included in ChatGPT search summaries and snippets. | The guidance does not name llms.txt as a citation or recommendation signal. |
| llms.txt proposal | Defines the intended file format and use as a curated content map. | A proposal does not establish adoption, ranking impact, or commercial outcome. |
llms.txt, robots.txt, sitemap, and noindex are different
robots.txt
Controls whether named crawlers may request paths. Blocking a crawler can prevent it from reading page-level directives.
XML sitemap
Lists canonical public URLs a site wants search engines to discover. Inclusion does not guarantee indexing.
noindex
A page-level or response directive requesting exclusion from an index. A crawler must be able to read it.
llms.txt
A proposed curated Markdown directory. Current evidence does not establish citation or recommendation impact.
Step 1
Fix stronger prerequisites first
Before spending time on llms.txt, verify the foundations that current platform guidance does support:
- Important public pages return successful responses and are not hidden behind authentication.
- OAI-SearchBot and the search crawlers you want are not blocked from those pages.
- Canonical URLs and the sitemap agree.
- Important pages have descriptive internal links in crawlable HTML.
- Each page answers a distinct buyer question and supports product claims with current evidence.
Step 2
Define the job before creating the file
A defensible job is “publish a maintained directory of our most important public documentation for tools that intentionally read it.” “Make ChatGPT rank us” is not a defensible job because current evidence does not support that mechanism.
Decide who owns updates, which URLs are safe and canonical, and what happens when product facts change. Do not put private, staging, account, or duplicate URLs in the file.
Step 3
Run it as a controlled experiment
- Freeze a small set of buyer prompts and record mentions, recommendations, citations, date, model, and sources.
- Save relevant server-log evidence before publication if available and permitted.
- Publish only the llms.txt change; avoid mixing it with page rewrites, link campaigns, or broad technical changes.
- Wait for the defined observation window, then rerun like-for-like samples.
- Record validated, failed, or inconclusive. A crawler request alone does not prove citation impact.
Decision rule for a lean SaaS
If crawl access, indexing, internal links, product facts, comparison pages, and measurement are incomplete, work on those first. If they are already sound and llms.txt takes little effort to maintain, publish it only as an explicitly optional experiment with no promised lift.
Sources and limits
SE Ranking’s study is observational at the domain level and states that results are context-dependent. The llms.txt site describes the proposed format. OpenAI documents its own crawler access requirements. None of these sources can guarantee how every model or future product will behave.
Frequently asked questions
Does OpenAI require an llms.txt file?
OpenAI’s publisher guidance tells sites that want to appear in ChatGPT search not to block OAI-SearchBot. It does not state that llms.txt is required or that the file improves ranking, citations, or recommendations.
Can llms.txt replace robots.txt or an XML sitemap?
No. robots.txt communicates crawl permissions, while an XML sitemap lists canonical URLs you want search engines to discover. llms.txt is a separate proposed convention for presenting a curated content list to language-model workflows.
Will adding llms.txt hurt SEO?
A small, accurate public file is unlikely to change ordinary SEO by itself. The practical risks are publishing stale or conflicting claims, exposing URLs that should not be promoted, and spending time on the file before fixing crawl, index, internal-link, and content problems.
When should a SaaS test llms.txt?
Test it only after important public pages are crawlable, internally linked, canonical, and supported by verifiable product facts. Define the baseline, preserve server-log and visibility evidence, and treat the result as inconclusive unless a controlled comparison supports a change.