The robots.txt file sits at the root of a website (yourdomain.com/robots.txt) and provides directives to search engine crawlers about which areas of the site they should and should not access. It can block crawlers from staging environments, duplicate pages, or resource files that don’t need to be indexed. Robots.txt is a blunt instrument — it prevents crawling but does not guarantee a page won’t be indexed if it’s linked from elsewhere.
