ClaudeBot

What it actually does when it visits

A crawler is a program that requests your pages the way a browser would, except nobody is reading in real time. It takes the text and moves on. ClaudeBot is the one Anthropic sends out.

Say you run a garage door company. ClaudeBot asks for your services page, your service area page, and your FAQ, gets the HTML back, and keeps the words. It does not fill out your quote form, it does not call you, and there is no person on the other end of the visit.

The part worth knowing is that it identifies itself. Every crawler sends a name along with its request, which is why your hosting logs can tell Googlebot apart from everything else. Because ClaudeBot gives a name, you can write rules aimed at that name: let it read everything, let it read part of the site, or tell it to stay out entirely.

That last one is a request, not a lock. Mainstream crawlers honor it. If a page genuinely must stay private, it needs a password or a server rule, not a line in a text file.

Should you let it in?

Blocking ClaudeBot does not protect anything, because everything it would read is already published for anyone to see. What blocking does is remove one path by which an assistant learns that your business exists and what it does.

Before you decide, separate the two errands. Reading pages to help build a model and fetching a page in the middle of somebody's question are different jobs, and the companies doing them generally run them under different names. A rule addressed to one name says nothing about the other. ClaudeBot is the general reader, so a rule aimed at it is mostly a decision about your writing, and much less a decision about whether you turn up in an answer somebody gets this afternoon.

For a business whose content is the product, a publisher, a course seller, a photographer, there is a real argument for shutting the door. For a contractor whose product is the work, the calculation usually runs the other way. You want your own words to be what an assistant has on hand when a homeowner asks it for a garage door company in your town, and a door held shut leaves it working from directories, listings, and whoever else wrote about your trade.

Either way, allowing it is not a deal you are signing. Nothing comes back the other way: no notice, no credit, no traffic. You are deciding whether a reader that will never buy anything gets to see the same pages a customer sees.

The more useful move is deciding what you want crawlers to read first, rather than only deciding whether to let them read at all, and our free llms.txt generator writes the file that points them at your best pages. It is one of more than 50 free tools we publish, with no email wall.

One name on a longer list

ClaudeBot sits alongside GPTBot from OpenAI and PerplexityBot, and the companies behind them tend to run several crawlers under different names for different jobs. Those names come and go. Since the mechanics do not change, the durable approach is to set one policy for every AI crawler as a class and keep your robots.txt file saying what you mean, instead of chasing individual bot names as they appear.

If you want to know which ones actually come by, the answer is in your own server logs rather than in anybody's advice about a single bot. The names arrive with the requests. The pattern across a month tells you more than the argument about any one of them.