Screenshot Generator
Capture website screenshots
What is Robots.txt?
Robots.txt is a plain text file that tells search engine crawlers which URLs on your site they're allowed to access. Every time Googlebot, Bingbot, or any other crawler visits your site, the first thing it checks is yourdomain.com/robots.txt. If the file exists, the crawler reads the rules before crawling anything else. For server-level access control (redirects, password protection, IP blocking), you'd use an .htaccess file instead — robots.txt only handles crawler instructions.
The file always lives at the root of your domain. Not in a subfolder, not with a different name. Google will only look at https://example.com/robots.txt — nothing else counts.
Core Directives
| Directive | What It Does | Example |
|---|---|---|
User-agent
|
Specifies which crawler the rules apply to. Use * for all crawlers.
|
User-agent: *
|
Disallow
|
Blocks crawling of a path. An empty value means nothing is blocked. |
Disallow: /admin/
|
Allow
|
Overrides a Disallow for a specific path. Useful for allowing a subfolder inside a blocked folder. |
Allow: /admin/public/
|
Sitemap
|
Points crawlers to your XML sitemap. Not technically part of the robots protocol, but universally supported. |
Sitemap: https://example.com/sitemap.xml
|
Robots.txt vs Noindex — They're Not the Same
This trips people up constantly. Robots.txt blocks crawling — it stops the bot from visiting the page. A noindex meta tag blocks indexing — it tells the search engine not to show the page in results. Here's the catch: if you block a page via robots.txt, Google can't see the noindex tag on that page (because it never crawls it). The page can still appear in search results with a "No information is available for this page" snippet if other sites link to it. If you want a page out of Google entirely, use noindex and allow crawling so the bot can actually read the tag.
Common Robots.txt Mistakes
-
Blocking CSS and JS files. Google needs to render your page to understand it. If you block
/wp-content/themes/or/wp-includes/, Googlebot sees a blank page. That kills your rankings. -
Blocking the entire site by accident. A single
Disallow: /underUser-agent: *blocks everything. This happens more often than you'd think, especially during staging-to-production migrations. - Forgetting the Sitemap directive. Robots.txt is the first file crawlers check. Including your sitemap URL here is the fastest way to make sure crawlers find all your pages, especially on new sites.
- Using robots.txt to hide pages from search results. As covered above, blocking crawling doesn't prevent indexing. Use noindex instead.
- Not testing after changes. Always verify your robots.txt in Google Search Console's URL Inspection tool or the robots.txt report. A typo in a path can block your highest-traffic pages.
-
Ignoring AI crawlers. Bots like GPTBot, ClaudeBot, and PerplexityBot now crawl sites to train models or power AI answers. If you want to control AI bot access separately from search engines, you can add specific
User-agentrules — or use our AI Crawler Inspector to see which AI bots are already hitting your site.