Free Robots.txt Generator Tool

What is Robots.txt?

Robots.txt is a plain text file that tells search engine crawlers which URLs on your site they're allowed to access. Every time Googlebot, Bingbot, or any other crawler visits your site, the first thing it checks is yourdomain.com/robots.txt. If the file exists, the crawler reads the rules before crawling anything else. For server-level access control (redirects, password protection, IP blocking), you'd use an .htaccess file instead — robots.txt only handles crawler instructions.

The file always lives at the root of your domain. Not in a subfolder, not with a different name. Google will only look at https://example.com/robots.txt — nothing else counts.

Core Directives

Robots.txt vs Noindex — They're Not the Same

This trips people up constantly. Robots.txt blocks crawling — it stops the bot from visiting the page. A noindex meta tag blocks indexing — it tells the search engine not to show the page in results. Here's the catch: if you block a page via robots.txt, Google can't see the noindex tag on that page (because it never crawls it). The page can still appear in search results with a "No information is available for this page" snippet if other sites link to it. If you want a page out of Google entirely, use noindex and allow crawling so the bot can actually read the tag.

Common Robots.txt Mistakes

• Blocking CSS and JS files. Google needs to render your page to understand it. If you block /wp-content/themes/ or /wp-includes/, Googlebot sees a blank page. That kills your rankings.
• Blocking the entire site by accident. A single Disallow: / under User-agent: * blocks everything. This happens more often than you'd think, especially during staging-to-production migrations.
• Forgetting the Sitemap directive. Robots.txt is the first file crawlers check. Including your sitemap URL here is the fastest way to make sure crawlers find all your pages, especially on new sites.
• Using robots.txt to hide pages from search results. As covered above, blocking crawling doesn't prevent indexing. Use noindex instead.
• Not testing after changes. Always verify your robots.txt in Google Search Console's URL Inspection tool or the robots.txt report. A typo in a path can block your highest-traffic pages.
• Ignoring AI crawlers. Bots like GPTBot, ClaudeBot, and PerplexityBot now crawl sites to train models or power AI answers. If you want to control AI bot access separately from search engines, you can add specific User-agent rules — or use our AI Crawler Inspector to see which AI bots are already hitting your site.

Robots.txt Examples by Platform

Every platform has different URL structures and admin paths. Here are working robots.txt examples you can use as a starting point for the most common setups.

WordPress

WordPress is the most common case. You want to block the admin area and internal search results while keeping everything else open. Don't block /wp-content/ — Google needs access to your theme CSS and JS to render pages properly.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Sitemap: https://example.com/sitemap_index.xml

Shopify

Shopify auto-generates a robots.txt that covers most cases. Since mid-2021, you can customize it via a robots.txt.liquid template in your theme. The default blocks checkout, cart, and internal search — which is usually what you want.

User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /checkout
Disallow: /search
Sitemap: https://example.com/sitemap.xml

Next.js / Static Sites

Static sites and Next.js apps usually have minimal paths to block. If you're using Next.js, place the file in your public/ directory. For most static sites, an open robots.txt with just a sitemap reference is enough.

User-agent: *
Disallow: /api/
Disallow: /_next/
Allow: /_next/static/
Sitemap: https://example.com/sitemap.xml

These are starting points. Use the generator above to create a robots.txt tailored to your specific platform and crawling preferences.