Robots.txt Generator

Create a robots.txt file in seconds with this robots.txt generator. Choose which paths crawlers may visit, point them to your sitemap, block AI crawlers if you want to, and get warnings about the mistakes that most often hurt rankings.

What robots.txt does

A robots.txt file is a plain text file at the root of your site (https://example.com/robots.txt). It tells well-behaved crawlers which parts of the site they may request. Search engines read it before they crawl.

It controls crawling, not indexing. A page blocked in robots.txt can still appear in search results if other sites link to it, usually with no description. To keep a page out of results, leave it crawlable and add a noindex tag instead.

The rules in plain words

• User-agent names the crawler a group of rules applies to. * means all of them.
• Disallow blocks a path. Disallow: /admin/ blocks everything under that folder. An empty Disallow: blocks nothing.
• Allow makes an exception inside a blocked folder. When rules overlap, the most specific one wins.
• Sitemap gives the full URL of your XML sitemap.
• * matches any text and $ marks the end of a URL, for example Disallow: /*.pdf$.

Mistakes to avoid

• Do not block CSS or JavaScript. Google needs them to render the page like a visitor sees it.
• Do not use robots.txt for secrets. The file is public, and it lists exactly what you want hidden. Protect private pages with a login.
• Check Disallow: /. On its own it blocks the whole site. It belongs on staging servers only.
• Remember it is case sensitive. /Admin/ and /admin/ are different paths.

Blocking AI crawlers

You can list the user-agents of AI training and answer-engine crawlers and disallow them. Well-behaved crawlers respect the file, but scrapers that ignore robots.txt will not, and blocking does not remove content that has already been collected.