Create a Robots.txt File
Choose a crawler, add paths to allow or block, and generate a copy-ready robots.txt file.
Important Robots.txt Warning
A wrong rule can hide large parts of a website from crawlers. Robots.txt is not a security system and blocked URLs may still be visible elsewhere.
Related Website & SEO Tools
Robots.txt Generator FAQ
Where should robots.txt be uploaded?
Place it at the root of the domain, such as example.com/robots.txt.
Does Disallow remove pages from Google?
Not reliably. It prevents crawling, but a URL can still be indexed from external links. Use noindex where appropriate.
Can robots.txt protect private files?
No. Use authentication and proper server permissions for private content.