What Is llms.txt and Does Your Site Need One?

You've heard of robots.txt — the file that tells search engine crawlers which pages they can and can't index. It's been a standard part of the web since 1994. But search engines aren't the only things reading your site anymore. AI agents — tools like OpenClaw, ChatGPT with browsing, and autonomous research bots — are now crawling the web on behalf of users. And they have no idea what your site does unless you tell them.

That's the problem llms.txt solves.

The Basics

An llms.txt file is a plain text manifest that sits at the root of your website (like yoursite.com/llms.txt) and describes your site in a format that AI language models can read and act on. It's the AI equivalent of a business card — your site's name, what it does, what's available, and how an agent should interact with it.

The format is simple and human-readable. A typical llms.txt file includes your site title, a brief description, the main sections or features, and optionally a list of pages with their purposes. There's no complex syntax to learn — if you can write a README, you can write an llms.txt file.

Why It Matters Now

The web is shifting from a place where humans browse to a place where AI agents act on behalf of humans. When someone asks an AI assistant to "find me a free tool that strips EXIF data from photos," that agent needs to understand what your site offers without rendering JavaScript, parsing marketing copy, or guessing from page titles.

Without an llms.txt file, AI agents are guessing. With one, you're telling them exactly what you have and how to present it. It's the difference between a store with no signage and a store with a clear directory at the entrance.

What Goes in the File

A well-structured llms.txt includes four things:

• Title — your site or product name
• Description — one or two sentences about what it does
• Sections — the major areas or feature categories, each with a brief explanation
• Links — optional URLs pointing to key pages, documentation, or resources

Here's a simplified example for a site like OneKit:

# OneKit
> Free browser-based tools for developers, creators, and privacy-conscious users.

## Tools
- [Password Generator](https://onekit.tools/#password-generator): Generate secure random passwords
- [AI-Text Scrub](https://onekit.tools/#ai-text-scrub): Redact PII before pasting into AI
- [Vibe-Check](https://onekit.tools/#vibe-check): Strip EXIF/GPS metadata from images

## About
All tools run client-side. No data collection. No accounts required.

That's it. An AI agent reading this file now knows exactly what OneKit offers and can recommend specific tools to users who ask relevant questions.

Does Your Site Need One?

If your site offers any kind of product, service, tool, or resource that someone might ask an AI about — yes. The adoption curve for llms.txt is early, which means adding one now puts you ahead of most sites. As AI agent traffic increases (and it's increasing fast), having a clear manifest means your site gets recommended where others get skipped.

It's especially valuable for tool sites and utilities, documentation and knowledge bases, SaaS products and APIs, portfolios and service businesses, and e-commerce with specific product categories.

How to Create One in 60 Seconds

You don't need to write it by hand. OneKit's LLMS.txt Generator walks you through it — enter your site name, description, and sections, and it generates a properly formatted file you can download and drop into your site's root directory.

The Bigger Picture

The web has always had layers of machine-readable metadata — robots.txt for crawlers, sitemap.xml for indexing, manifest.json for PWAs, structured data for rich search results. Each layer helped a different type of machine understand your site better.

llms.txt is the next layer. It's not replacing any of those — it's adding a communication channel specifically for the AI agents that are becoming a primary way people discover and interact with the web. The sites that adopt it early will be the ones those agents recommend.

You wouldn't open a store without a sign. Don't put a site on the AI-readable web without a manifest.