Do all AI engines support llms.txt?

Support is growing but not universal. As of early 2026, several major AI engines and crawlers recognise llms.txt, and adoption is increasing. Even engines that do not yet parse the file directly benefit from the structured signals it provides when they crawl your site through other means.

How long should an llms.txt file be?

Keep it concise — typically 50-200 lines. The file should be comprehensive enough to cover your key pages and brand information but brief enough that AI crawlers can parse it quickly. Focus on your highest-priority content rather than listing every page.

Is llms.txt a replacement for robots.txt or sitemap.xml?

No. llms.txt complements robots.txt and sitemap.xml. Robots.txt controls crawler access, sitemap.xml helps with page discovery, and llms.txt provides context and prioritisation signals specifically for AI language models. You should maintain all three.

Resource

Template

llms.txt Template and Guide

Create and optimise your llms.txt file to help AI engines understand your site's purpose and structure.

Quick answer

The llms.txt file is a machine-readable document placed at your site's root that tells AI language models what your site is about, how it is structured, and which pages contain your most important content. Think of it as a combination of robots.txt and a site brief — written specifically for AI consumption.

Overview

About this template

This template provides the exact format, required fields, and optional enhancements you need to create an effective llms.txt file. It covers everything from the basic specification to advanced techniques for guiding AI engines toward your highest-value pages.

A well-crafted llms.txt file can meaningfully improve how AI engines index and cite your content. Sites with llms.txt give AI crawlers explicit signals about content hierarchy, reducing the guesswork that leads to missed or misrepresented content.

Basic File Structure

1
Place the file at your domain root: /llms.txt
AI crawlers look for this file at the root URL, similar to robots.txt or sitemap.xml.
2
Start with a one-line site description
The first line should concisely state what your site or business does in plain language.
3
Include your primary site URL
Ensure the canonical URL of your site is clearly stated for AI engines to associate content.
4
List your main content categories
Group your site content into logical categories that help AI engines understand your expertise areas.
5
Add last-updated date
Include a date so AI crawlers know how current your llms.txt file is.

Priority Pages Section

1
List your top 10-20 pages by importance
These are the pages you most want AI engines to index, cite, and recommend.
2
Include a one-line description per page
Help AI engines understand what each page covers without needing to crawl it first.
3
Order pages by priority (highest first)
AI crawlers may not process the entire file — frontload your most important content.
4
Use full absolute URLs
Relative URLs can cause resolution errors in AI crawlers. Always use the complete URL.
5
Tag pages with content type labels
Indicate whether each page is a guide, product page, comparison, pricing, etc.

Brand & Entity Information

1
State your brand name and any common variations
AI engines need to know how your brand is referenced in different contexts.
2
List your primary product or service categories
Clarify which industry categories your brand belongs to for accurate classification.
3
Include competitive positioning context
State what makes your offering different — AI engines use this for comparison queries.
4
Add key facts: founding year, headquarters, team size
Factual data helps AI engines construct accurate entity profiles for your brand.

Content Guidance Section

1
Specify your preferred citation format
Tell AI engines how you want your content attributed when cited in responses.
2
Indicate content freshness expectations
Note which content is evergreen vs time-sensitive so AI engines weight recency appropriately.
3
Flag deprecated or outdated pages
Prevent AI engines from citing old content that no longer reflects your current offering.
4
List content that should not be summarised
Some content, such as legal terms or precise specifications, should be linked rather than paraphrased.

Validation & Maintenance

1
Validate file encoding is UTF-8
Non-standard encodings can cause parsing errors in AI crawlers.
2
Test that the file is accessible at the root URL
Verify with a direct HTTP request that the file returns a 200 status code.
3
Update whenever you add or remove key pages
Stale llms.txt files can direct AI engines to pages that no longer exist.
4
Review monthly against your content calendar
Align your llms.txt updates with major content publishes to ensure new content is included.
5
Monitor AI engine responses for improved citations
Track whether pages listed in llms.txt show increased citation rates over time.

FAQ