Robots.txt is a plain text file at the root of a domain, always at /robots.txt, that tells automated crawlers which parts of the site they may request. It is the first thing most well-behaved crawlers fetch, and a single careless line in it can remove a site from search results entirely.
The most important thing to understand about it is also the most commonly misunderstood: robots.txt controls crawling, not indexing. It asks crawlers not to fetch a URL. It does not remove that URL from search results, and in some cases it actively prevents removal.
Crawling versus indexing, and why the distinction bites
If you disallow a page in robots.txt, Google will not fetch it. But if other pages link to that URL, Google can still index it, listing the bare URL with no description because it was never allowed to read the content. The page appears in search looking broken and says nothing useful.
Worse, if you block a page that carries a noindex tag, Google cannot see the tag, because seeing it requires fetching the page. The block prevents the very instruction you wanted followed. To keep a page out of search results, allow crawling and use a noindex directive. Use robots.txt to manage crawl effort, not visibility. Our entry on indexability covers the other side of this.
The syntax
The file is a series of groups. Each names one or more user agents and the rules that apply to them:
User-agent: which crawler the following rules address. * means any crawler without its own group.
Disallow: a path prefix the crawler should not request. Disallow: / blocks everything. An empty Disallow: blocks nothing.
Allow: an exception carving something back out of a broader disallow.
Sitemap: the full URL of your XML sitemap. This one is independent of user agent and belongs in every robots.txt.
Google supports * as a wildcard within paths and $ to anchor the end of a URL. Where rules conflict, the most specific matching rule wins rather than the first one listed, so an Allow for a longer path overrides a broader Disallow.
A crawler reads only the group that best matches its own name. If you write a group for Googlebot, Googlebot follows that group and ignores the * group entirely, including any rules you assumed still applied. This catches people out constantly.
Directives that do not do what people think
Noindex: in robots.txt does nothing. Google stopped supporting it in September 2019. If your file still contains it, those pages are not protected.
Crawl-delay: is ignored by Google. Bing and some others honor it. If Googlebot is genuinely straining your server, the crawl rate setting in Search Console and better hosting are the real levers.
Robots.txt is not access control. The file is public, readable by anyone at yourdomain.com/robots.txt. Listing a private directory there advertises it to exactly the people you did not want finding it. Anything genuinely sensitive needs authentication, not a polite request.
What to block, and what never to
Reasonable candidates: internal search result pages, faceted filter combinations that generate near-infinite URLs, cart and checkout paths, admin areas, and duplicate URL patterns created by parameters.
Never block CSS and JavaScript. Google renders pages to understand them, and a page it cannot render properly may be judged broken or mobile-unfriendly. Blocking /wp-content/ or /wp-includes/ wholesale, which older WordPress advice recommended, does exactly this. It is one of the most damaging pieces of outdated SEO guidance still circulating.
Also never block: pages carrying noindex tags, pages you want removed from the index, or your sitemap.
AI crawlers
Robots.txt has become the main control surface for AI systems, and each operates its own user agents for different purposes. The distinction that matters is between crawling for model training and crawling to answer a user’s question right now, because you may well feel differently about the two.
OpenAI: GPTBot gathers content that may be used for training, OAI-SearchBot indexes for ChatGPT Search, and ChatGPT-User fetches a page when a user’s request sends it there.
Perplexity: PerplexityBot indexes for answers, Perplexity-User fetches on demand. See Perplexity.
Anthropic: ClaudeBot.
Google: Google-Extended is the one most often misread. It does not affect Googlebot, your search rankings, or whether you appear in AI Overviews. It controls whether your content is used for Gemini and Vertex AI generative training and grounding. Blocking it does not remove you from Google Search.
Common Crawl: CCBot, an open dataset many models have trained on.
A blanket block on everything AI-shaped is the decision most sites make by accident, through an over-broad rule or a security plugin default, and it quietly removes the business from AI answers. If AI visibility matters to you, the useful posture for most service businesses is to allow the search and user-fetch agents, and to decide about training agents separately on your own terms. Our guide to llms.txt for websites covers the related emerging convention.
Robots.txt on WordPress
WordPress generates a virtual robots.txt if no physical file exists, and SEO plugins including Yoast let you edit it from the admin. A real file placed at the web root overrides the virtual one entirely, which is a frequent source of confusion when edits in the plugin appear to have no effect.
Two WordPress-specific traps worth knowing. The Reading setting “Discourage search engines from indexing this site” is a staging safeguard that adds a sitewide block, and it travels with a database copy to production more often than anyone would like. And security or firewall plugins can rate-limit or block legitimate crawlers in ways robots.txt never mentions, so a clean robots file does not by itself prove crawlers are getting through.
Checking it works
Fetch the file itself at yourdomain.com/robots.txt and read what is actually served, not what the plugin shows you.
Use the robots.txt report in Google Search Console to see the version Google last fetched and when.
Use URL Inspection to confirm whether a specific page is blocked.
Re-check after any migration, launch, or security plugin change.
One lesson from our own site belongs here. We once found the correct sitemap sitting in WordPress while the host’s edge cache served an outdated copy at the bare URL that Googlebot actually requests. Every check with a query string appended looked fine. The same failure mode applies to robots.txt: verify what is served at the exact URL a crawler requests, with no parameters, and re-check after anything that touches caching.
Where it fits
Robots.txt is among the first things we read in any audit, because a single line in it can suppress everything else. It is part of our technical SEO services and a standing item in our crawlability checks. If pages are missing from Google and you are not sure why, book a discovery call.
