Runs locally
Robots.txt Builder
Allow and disallow rules for one user agent.
Ctrl+Enter runs. Ctrl+Shift+Backspace clears.
Export and import are available after you sign in. Sign in
Allow and disallow rules for one user agent. Runs in the browser. Nothing is uploaded.
About this tool
Tell search engines which parts of your site they may crawl. List the paths to allow and the paths to block, one per line, and copy the generated file.
How to use it
- 01
List the paths to allow, one per line.
- 02
List the paths to disallow.
- 03
Copy the file and upload it as /robots.txt.
FAQ
Frequently asked questions
Which user agent do the rules apply to?
All crawlers: the file starts with User-agent: *.
Does Disallow hide a page from search results?
Not reliably. It stops crawling, but a page linked elsewhere can still appear; use a noindex tag to keep it out.