In one sentence: Google-Extended is a robots.txt control name rather than a separate crawler, used by a site owner to say whether the pages Googlebot already crawls may be used to train and ground Google's generative AI products outside of Search itself.
Most names in a server log belong to something that showed up and asked for a page. Google-Extended is not one of those. No request ever arrives from it, because it does not fetch anything. Googlebot does the crawling, the same way it always has, and Google-Extended is the label a site owner uses to say what may happen to that crawled text afterward.
The setting lives in robots.txt, the plain text file at the root of your domain, and it obeys the ordinary logic of that file: a rule applies to the name it addresses and to nothing else. Telling Google-Extended to stay out does not tell Googlebot to stay out. Splitting the use off into its own name is the entire point. It is also narrower than most owners assume: it covers Google's separate generative products, not the AI features that appear inside Search itself, which run off the ordinary search crawl.
The everyday version: a photographer hands over the prints you ordered, and separately you decide whether those prints may appear in her portfolio. One visit, two different permissions, and answering the second one does not cancel the first.
The confusion this causes is predictable. Owners go looking for the name in their traffic reports, find nothing at all, and conclude the setting never took. It took. There was never going to be a visit attached to it.
Because the crawl and the AI use are separated, a business can stay in ordinary search results while declining the generative uses this name covers, or accept one and refuse the other. That is genuinely useful control. It also means the decision has to stand on its own merits, because there is no ranking penalty waiting on either side. A robots.txt preference is a preference, not a demerit.
For a roofing company or a single office dental practice, the case for staying included is plain enough: when someone asks a generative product about your trade in your city, you would rather the system have read your own pages than assembled something out of directories and a competitor's blog. The case for opting out is about ownership of writing you paid for, and it gets stronger the more original that writing is. Two shops on the same street can make opposite calls here, and neither call is what decides how they rank.
One piece of the mechanic is easy to miss: the name only governs text Google already has. If a page is blocked from Googlebot, or sits behind a login, or was never linked from anywhere, the AI question never comes up for it at all. Permissions apply to what is reachable in the first place, which is why crawl problems and AI visibility problems usually turn out to be the same problem wearing different clothes.
What neither choice buys is a guarantee. Permission is a floor, not a lever, and nobody can promise a given system will mention you. The setting decides what is allowed, and allowed sits a long way from chosen. What closes that gap is the same unfashionable work it has always been: pages that say something worth repeating. We keep 361 in-depth guides in the learning library at kellywm.com/blog, and the one that walks through how these answers get assembled is our plain English guide to AI search.
The file itself is robots.txt, and what Googlebot does on arrival is crawling. Part of what this name governs is grounding, where a system checks real sources before it writes an answer. OpenAI splits the same question across names of its own, starting with GPTBot.
No. Search crawling and indexing run under Googlebot. Google-Extended addresses a different name and therefore a different use, so a rule for one leaves the other alone.
No. It is a user agent line in your robots.txt file, which means whoever controls your site files controls the setting, not whoever has console access.
What AI search optimization is · Robots.txt · Crawling · Grounding · GPTBot · All glossary terms · Plain-English answers · AI search optimization services
Free consultation, plain-English advice. If you don't need us, we'll say so.
Book a free consultation → Or call/text directly: (407) 694-2055Tell us a little about the business and we will come back with an honest read: what we would fix first, what it costs, and whether you need us at all. Prefer to see work before you talk numbers? Get a free homepage mockup, built for your business, yours to keep either way.
Brandon reads every one of these himself. You will hear back shortly with an honest read on what we would do first, what it costs, and whether it is worth it for you.