What robots.txt does
Robots.txt is a plain text file at the root of your site, at /robots.txt. It tells well-behaved crawlers, such as search engine bots, which parts of the site they may visit. It is also the usual place to list your sitemap. This tool builds the file from simple choices, shows warnings for risky rules and lets you download it.
How to use it
- Start from a preset: allow everything, block everything, WordPress or custom.
- Add paths to disallow and, if needed, paths to allow, one per line.
- Add your sitemap URL or URLs. Use full addresses that start with https://.
- Optionally add a crawl delay, and block common AI training crawlers.
- Copy the output or download robots.txt and upload it to your site root.
Example
A typical file:
User-agent: *
Disallow: /admin/
Disallow: /api/
Allow: /api/public/
Sitemap: https://www.example.com/sitemap.xmlMistakes that hurt SEO
- Disallow: / blocks the entire site. It is correct only for private or staging sites.
- Blocking CSS or JavaScript files can stop search engines rendering your pages properly.
- Robots.txt does not remove a page from search. A blocked page can still be listed if other sites link to it. To keep a page out of results, use a noindex tag and let it be crawled.
- Robots.txt is not a security tool. Anyone can read it, and bad bots ignore it. Protect private areas with login and permissions.
- A typo in a path can block pages you meant to keep open. Test with the robots.txt report in Search Console.
About AI crawlers and crawl delay
The AI crawler option adds a block for common AI training bots. Only crawlers that follow the standard will obey it. Google ignores the Crawl-delay line, while some other crawlers respect it, so use it only if a specific bot is overloading your server.
Related tools
- Meta Tag & Open Graph Generator: Generate title, description, canonical, Open Graph and Twitter card tags for the head of your page.
- Slug Generator: Turn titles into clean URL slugs. Handles accents and symbols, with bulk lists and a max length.
- Favicon Generator: Upload one image and generate every standard favicon size plus a web manifest and copy-paste HTML tags, all bundled in a downloadable ZIP.
- URL Encoder / Decoder: Encode or decode URLs and query strings using percent-encoding, right in your browser.