About the LLMs.txt Directory

What is a llms.txt file?

A growing number of websites now publish a short, plain-text file — conventionally at /llms.txt — written specifically for AI systems rather than human visitors or traditional search crawlers. Where a normal web page is full of navigation, styling and scripts, a llms.txt file is meant to be a clean, curated pointer: a short summary of what the site is, followed by links to the pages the site itself considers most worth an AI system reading. Think of it as a site's own reading list for language models, in a format those systems can parse without wading through the rest of the page.

What does the ISEEYOU Finder search?

The Finder searches ISEEYOU's own catalog of discovered llms.txt records — not the live web directly, and not any other site's directory. Every record you can find here was independently confirmed at some point: the file was requested and something real came back at that URL. You can search by domain, by site name, or by pasting the exact llms.txt URL. The catalog is large (hundreds of thousands of records) but it is not exhaustive — a site not appearing here may still have a real llms.txt file that ISEEYOU simply hasn't discovered yet.

How are files discovered and verified?

A separate collection process — not this website — continuously visits sites across the public web looking for a llms.txt file. When it finds one that actually loads, the record is marked found. Only found records are published in this directory; everything else (a site with no file, an empty file, a broken link, a page robots.txt disallows checking) stays out of the public catalog entirely. Not every record was found the same way, and each domain page shows which method applied:

• Declared at site root — the site was checked directly at its conventional llms.txt location.
• Imported from a curated directory — the site was already listed in an existing curated llms.txt directory.
• Found via an on-page link — a link to the file was found referenced on the page itself.
• Found via sitemap — the file was located via the site's own sitemap.
• Found via web-scale crawl — found through a broad, web-scale crawl pass rather than a manual, per-site check.

Is llms.txt the same as robots.txt?

No, and they do opposite jobs. robots.txt tells automated crawlers what they are not allowed to access — it's a permissions file, decades old, aimed at search-engine crawlers in general. llms.txt is the reverse: a site opting in, actively pointing AI systems toward the content it considers most worth reading. A site can have both, one, or neither — they're unrelated files with unrelated purposes, and ISEEYOU's catalog only tracks the second one.

Is llms.txt the same as sitemap.xml?

No. A sitemap.xml is a comprehensive, machine-formatted list of a site's URLs, built for search-engine crawlers to find and prioritize pages for indexing — it's usually large and exhaustive. A llms.txt file is the opposite in spirit: a short, human-curated, plain-text summary meant to be small enough for an AI system to read in full, pointing to a deliberately narrow set of pages the site itself picked out — not every page it has.

Does publishing llms.txt guarantee AI visibility?

No. Publishing a llms.txt file does not guarantee search rankings, AI citations, inclusion in any model's training data, or that any particular AI system will read or use the file at all. It's an emerging, voluntary convention — some AI systems may consult it, others may not, and that can change over time. ISEEYOU records whether a file exists and loads; it makes no claim about what effect, if any, that has on how an AI system treats the site.

Why can one website have multiple llms.txt files?

Large sites are often organized into products, docs sections, or subdomains that each publish their own llms.txt — for example a marketing site at the root domain and a separate one under a /docs path or a docs. subdomain. ISEEYOU's discovery process can find several distinct, independently-hosted files under the same domain; a domain's page in this directory shows one primary record plus any others found, rather than hiding them.

What does "last verified" mean?

"Verified" and "last checked" on each page refer to when ISEEYOU's collection process last confirmed the file was still there and loading — not when the site last edited its content. ISEEYOU re-checks records over time, so a domain can appear more than once in the "recently verified" list without anything about the underlying file actually changing.

Will ISEEYOU support llms-full.txt and Markdown resources?

Some sites publish a companion file, often called llms-full.txt, that bundles fuller page content into one larger document rather than just linking out to it, and some declare individual clean-Markdown page alternates via HTML <link rel="alternate"> tags or HTTP headers. ISEEYOU's catalog today tracks discovered llms.txt records only; distinguishing llms-full.txt and Markdown page alternates as their own tracked types is on the roadmap, not yet live — a record on this site currently means a working llms.txt (or a file found at that conventional path) was found and confirmed, nothing more specific than that.

A note on scale and reuse

ISEEYOU's catalog structure, verification history and presentation are its own work, built up over continuous discovery passes. The individual llms.txt files themselves remain the property of the sites that publish them — this directory links out to the original source for every record rather than hosting or rewriting file contents. Automated bulk copying or republishing of the directory itself isn't permitted; browsing, searching and following individual links through to the original sites always is.