With AI now using more web content to generate answers, businesses face a new challenge: should you try to guide what AI takes from your website, or keep letting AI crawl your site freely?
The llms.txt protocol is a proposed way to guide AI crawlers to the most important pages on your site. However, few companies use it so far, and leading AI firms haven’t agreed to follow its rules. This puts business leaders in a tricky spot – should you spend time and resources on a tool that might not be widely adopted?
This article looks at whether llms.txt is worth implementing from a business perspective – focusing on risk, effort, and realistic outcomes rather than deep technical details.
What is llms.txt and why does it matter?
llms.txt is a special file you can add to your website to guide AI tools to your top content. Unlike robots.txt, which tells bots what to avoid, llms.txt curates your key pages and resources for AI to focus on.
The file is written in Markdown so AI systems can easily understand it. With many sites having content hidden behind scripts or having far too many pages, AI can struggle to find the best information. llms.txt lets you highlight important sections yourself – such as company overviews, product pages, and crucial documentation – giving you more control over how your brand is shown in AI answers.
By using llms.txt, you reduce the risk of AI pulling outdated or incorrect info from your website, which could impact customer choices or your reputation. Giving AI a clear path to your preferred content helps keep your message accurate and up-to-date.
What are the risks of ignoring AI content governance?
If you don’t guide AI, several risks appear. The most obvious is misinformation. AI might use old or irrelevant pages from your site to generate answers – leading to customers getting wrong details about your products or services. This can harm your brand and even cause support or legal headaches if, for example, AI shares confidential or outdated info.
Worse, once wrong information enters an AI system, it can be copied and repeated by many other AI tools, making it very hard to correct. Taking control early helps you avoid these issues before they snowball.
How does llms.txt work in practice?
Using llms.txt is technically simple – you create a text file (in Markdown) and put it in your website’s main folder (so it’s at yourdomain.com/llms.txt). The file organises and links to your most important pages with clear sections and short descriptions.
Here is a mock example:
# Title
> Optional description goes here
Optional details go here
## Section name
- [Link title](https://link_url): Optional link details
## Optional
- [Link title](https://link_url)The main effort isn’t technical; it’s deciding which content matters most for your brand and keeping the file updated. If your highlighted pages change (new products, new priorities), you’ll need to update the file. If you don’t, you risk sending AI to outdated info, doing more harm than good.
Who is using llms.txt so far?
As of mid-2025, llms.txt is mainly used by a handful of mostly tech-focused companies like Hugging Face, Vercel, and Zapier. Fewer than 1,000 websites have an llms.txt file – so it’s still an emerging idea and not the norm.
Major AI players like OpenAI, Google, and Anthropic haven’t committed to following these files, so their current impact is uncertain. Some companies use very detailed files; others keep it simple. There’s still no standard best practice.
How should you decide if llms.txt is right for you?
Don’t just use llms.txt to follow the crowd – consider your situation first. If you have lots of important technical content or documentation that you want AI to show correctly, you may see more value. If incorrect AI answers could cause big problems for you, it’s worth considering.
But if you mainly have marketing content or don’t get much AI exposure, it may be less urgent. Also, you’ll need resources to keep the file up to date – an outdated llms.txt can be worse than none at all. If you’re still getting your general web content in order, start there before adding extra protocols.
The key question: is the future benefit worth the time spent now, knowing AI providers might not ever support this? Sometimes, a “wait and see” approach makes sense.
llms.txt limitations and what to expect
The biggest limitation is that leading AI companies don’t officially use llms.txt yet. Even if they do, there’s nothing stopping them from ignoring parts of your file. llms.txt helps guide AI, but it can’t force good behaviour, fix misunderstandings, or act as legal protection if AI misuses your content.
Also, the entire process only works if your underlying content is high-quality and current. Highlighting out-of-date pages does more harm than good.
Is it worth the effort?
Making a llms.txt file isn’t hard and doesn’t cost much. The real work is maintaining it over time – keeping it accurate as your business changes. This takes ongoing effort.
If AI companies start using llms.txt, the benefits could include better AI answers about your brand, happier customers, and fewer mistakes. But if adoption stays low, you’re just adding another file to manage. Right now, investing time has uncertain payoff, but may make future adjustments easier if llms.txt catches on.
Final recommendations
If you’re thinking about llms.txt, start small. Try it out on your most important content first, and figure out how to keep it up to date. Only roll it out more broadly if you see AI companies supporting it, or if your risk from AI “getting it wrong” is high.
Make sure you have good systems for content quality and regular updates. If you lack those, focus there first – llms.txt is no fix for deeper content issues.
Keep an eye on what major AI providers are doing. If they begin supporting llms.txt, you’ll be ahead. If not, reconsider whether to keep investing effort. For help with this or broader content governance, reach out to digital strategy experts like Wilson Cooke.
