Robots.txt

A sign on the door, not a lock

The file lives at one predictable spot, your domain followed by /robots.txt. Anyone can read it, including your competitors. Inside it are rules addressed to crawlers by name: this one may not request that folder, this other one may not request anything.

Well behaved crawlers read the file before they fetch anything and follow what it says. Badly behaved ones skip it entirely. That is the whole nature of the thing. It is a posted sign, not a lock, and anything that genuinely has to stay private needs a login rather than a line in a text file.

The contents are simpler than people expect. Each block names a crawler, then lists the paths that crawler should skip. A crawler that finds no block written for it by name follows the catch-all block instead, so the catch-all is the rule that governs almost everything arriving at your site. The rules apply to that one site, on that one domain, which is why a subdomain needs its own file rather than inheriting yours.

One misunderstanding causes most of the trouble. Blocking a page here does not reliably pull it out of search results. It tells the crawler not to fetch the page, and the address can still be known from links elsewhere. If you want a page kept out of results, the instruction goes on the page itself, and the crawler has to be allowed to fetch the page in order to read it. Block the page and you have hidden the very instruction that would have removed it.

Two ways this small file quietly costs a local business

The first is the accident. A site gets built on a staging address with every crawler shut out, then goes live, and the block travels with it. The site looks finished and works perfectly for anyone you send there, while search engines are still standing outside. It is worth checking after every launch and every redesign, because nothing on the page itself looks wrong.

The second is deliberate. This same file is where you allow or refuse the crawlers that gather text for AI answer systems. Refuse them and the crawlers that cooperate stop collecting your pages, which some owners want. The tradeoff is that your business tends to stay out of the answers those systems write, and that answer can be what a customer reads instead of a list of links.

Deciding which of those crawlers you want reading your pages is a real decision, and it belongs with the rest of your plan for AI search visibility rather than being left to whatever the file said on day one.

What robots.txt is not

It is not crawling. It is the rulebook a crawler reads first. It is not a sitemap either, though it normally names one: a line pointing at your XML sitemap is how many crawlers find that file at all.

On a large site it does shape crawl budget, by keeping crawlers out of endless filter and sort addresses that produce nothing worth filing. And the crawlers it governs now include both ordinary search crawlers and the AI crawlers run by AI companies, each of which announces itself by name, such as GPTBot.