Crawling

What a crawler does when it visits

A crawler is a program that asks your web server for a page the same way a browser does, minus the person. It takes the HTML that comes back, reads it, writes down every link on the page, and adds those addresses to a list to fetch later. Then it does that again, across the whole web, without stopping.

It does not click, scroll or fill in forms, and it may never run the scripts on your page at all. That last part is where plenty of sites come unstuck. If your service list only appears once a script has run, or your locations sit behind a search box, a crawler can see an empty page where you see a full one.

Ordinary things that stop a crawl: a rule in robots.txt telling bots to stay out, a page that lives behind a login, a server that times out under load, a link that is really just a button with no address behind it. An XML sitemap helps by handing over a list of your URLs, but it is a suggestion, not a guarantee that anything gets fetched.

If it was never fetched, nothing else matters

Everything else you might work on sits downstream of this. A page that was never crawled cannot rank, cannot be quoted in an AI answer, and cannot be found by anyone who did not already have the link. It is not competing badly. It is not competing.

There is a newer wrinkle worth knowing about. Search engines are no longer the only things fetching your pages, because AI systems run their own crawlers as well. Separately, hosting companies and content delivery networks often block unfamiliar bots by default as a security measure. Put those together and you get a site that reads perfectly to one crawler and does not exist to another, with nothing on the site itself looking wrong to you. Choosing which bots you want, then confirming the ones you want can actually get through, is a build and hosting decision, so it is web development work rather than writing.

Crawled, indexed, ranked: three different things

People use these three words as if they mean the same thing. They do not, and knowing which one you are stuck at tells you what to fix.

Crawled means a bot fetched the page. Indexing means the search engine decided to store that page and consider it for results, which it can decline to do if it judges the page thin or duplicated. Ranked means it chose to show your page for a particular search. A page can be crawled and never indexed. A page can be indexed and never rank for anything worth having. Each step fails for its own reasons, so find out where yours stops before you start changing things. A crawl problem is usually fixed in the plumbing of the site. An indexing problem is usually fixed on the page, by giving it something worth storing.