robots.txt Generator

Create a robots.txt file to control search engine crawler access.

Presets:

Rule 1

User-agent(s) — one per lineAllow — one path per lineDisallow — one path per line

Sitemap URLCrawl-delay (sec)

Generated robots.txt

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

About This Tool

The robots.txt Generator creates a well-formatted robots.txt file that you can upload to the root of your website. The robots.txt protocol (Robots Exclusion Standard) tells compliant web crawlers which pages or sections of your site they are allowed or not allowed to crawl and index.

Use presets for common configurations or build custom rules for each crawler. The Sitemap URL field adds a pointer so crawlers can discover your XML sitemap automatically.

How to Use

Choose a preset or configure User-agent, Allow, and Disallow fields manually.
Add multiple rules with the + Add Rule button for different bots.
Enter your Sitemap URL and optionally set a Crawl-delay.
Copy the output or download robots.txt and upload it to your site root.

Use Cases

SEO professionals create robots.txt files to prevent crawl budget waste by blocking duplicate pages, admin panels, and search result pages. Developers use it to hide staging environments from search engines. Site owners block aggressive scrapers and SEO tools that consume bandwidth without benefit.

FAQ

Does robots.txt prevent pages from appearing in search?— No. It prevents crawling but not indexing. Use a noindex meta tag or X-Robots-Tag header to prevent indexing of pages you can't disallow.
Does every crawler respect robots.txt? — Reputable crawlers (Googlebot, Bingbot) respect it. Malicious scrapers may ignore it.
Where should robots.txt be placed? — Always at the root: https://example.com/robots.txt