In one sentence: An AI crawler is an automated program that visits and downloads web pages on behalf of an artificial intelligence company, either to collect text for training a model or to fetch current pages when someone asks a question.
A crawler is a program that requests pages the way a browser does and keeps a copy. Search engines have sent them for decades. The AI versions are newer, and they run two errands that are easy to confuse.
One errand is training. A crawler works its way across the web collecting text, and that text becomes part of the material a model learns from. The visit may happen long before any customer asks about you.
The other errand is fetching. Someone asks an assistant a question right now, the system decides it needs live pages, and a crawler goes and gets a few, possibly including yours. That visit happens in the middle of a customer's question.
Each bot announces itself by name in the request, the way a delivery driver has a logo on the truck. That name is how your site tells them apart, and how you can let one in while turning another away.
Worth saying plainly: a crawler is not looking at your design. It takes text, links, and markup. A page whose main claim lives inside a photo is close to a blank page as far as it is concerned.
None of them ask permission in advance, either. The well-behaved ones read whatever rules your site publishes and follow them, which puts the decision on your side of the wire rather than theirs.
Because the errands are separate, the decision is not a simple yes or no to AI. Blocking the training crawler keeps your writing out of the next model. Blocking the fetcher keeps you out of answers being written today, for customers who are asking right now.
Here is the trap. Some hosting companies and security services block these bots by default, as a reasonable protection against scraping. A local business can sit outside AI answers for months without anyone having made that choice on purpose, and nothing tells you it happened. It is worth checking rather than assuming.
The second thing to know is that permission and appearance are different questions. Letting a crawler in does not mean an answer will mention you. It means you are eligible. Getting named still comes down to clear, specific pages that answer what the customer actually asked.
The honest version of the tradeoff: if the writing itself is what you sell, blocking has a real case. If you sell drain cleaning across three counties and want to be the name an assistant gives out, being readable is the entire point, and a plain llms.txt file, an emerging convention for exactly this, is one way to point them at the pages worth reading.
Most of this traffic comes from a short list of named agents. GPTBot and Google-Extended are the training-side controls people ask about most, while agents such as PerplexityBot turn up around live questions. The file that tells them what they may take is still robots.txt, the same plain text file that has governed crawling since long before any of this existed.
Allowing or refusing any of them is a line in a text file, not a firewall project. That is exactly why the setting tends to get made once, by whoever set up the hosting, and then forgotten by everybody.
There is no single right answer, and anyone handing you one is selling something. A publisher whose product is the writing has a real case for blocking the training crawlers. A local service business that wants to be recommended usually does not, because the same door decides whether an assistant can read your service page when a customer asks. Decide the two errands separately.
Server logs. Every request carries the agent's name, so a hosting dashboard or log viewer will show which ones came, how often, and what they took. If the log shows nothing from any of them, that is usually a blocking rule rather than a lack of interest.
llms.txt generator · GPTBot · Google-Extended · Robots.txt · Crawling · All glossary terms · Plain-English answers · AI search optimization services
Free consultation, plain-English advice. If you don't need us, we'll say so.
Book a free consultation → Or call/text directly: (407) 694-2055Tell us a little about the business and we will come back with an honest read: what we would fix first, what it costs, and whether you need us at all. Prefer to see work before you talk numbers? Get a free homepage mockup, built for your business, yours to keep either way.
Brandon reads every one of these himself. You will hear back shortly with an honest read on what we would do first, what it costs, and whether it is worth it for you.