Loading...
Generate valid robots.txt files with presets and custom rules
Robots.txt is a plain text file at the root of a website that instructs web crawlers (Googlebot, Bingbot, etc.) which pages to crawl or avoid. It follows the Robots Exclusion Protocol (REP) and supports User-agent, Allow, Disallow, Crawl-delay, and Sitemap directives.
User-agent: * Allow: / Disallow: /admin/ Sitemap: https://example.com/sitemap.xml
| Bot | Crawler |
|---|---|
| * | All crawlers |
| Googlebot | Google Search |
| Bingbot | Microsoft Bing |
| GPTBot | OpenAI (ChatGPT) |
| Google-Extended | Google AI (Gemini) |
| Twitterbot | Twitter/X previews |
Formula
User-agent: {bot} Disallow: {path} Allow: {path} Sitemap: {url}User-agent = Which crawler the rules apply to (* = all)
Disallow = Path or directory the crawler should not access
Allow = Path or directory explicitly permitted (overrides Disallow)
Sitemap = Full URL to XML sitemap for crawler discovery
Worked Example
Standard robots.txt for a blog with admin area
Did you know? The Robots Exclusion Protocol was first proposed by Martijn Koster in 1994 on the www-talk mailing list. Google formally adopted it as an internet standard (RFC 9309) in September 2022, making it the first official IETF specification for robots.txt (source: IETF RFC 9309).
Sources
Convert text to ASCII codes and back. Supports decimal, hex, binary, and octal formats.
Convert width/height to aspect ratios, scale dimensions, and generate responsive embed CSS for any layout.
Calculate required internet bandwidth based on devices and activities in your household.
Encode and decode Base64 strings. Convert text for data URIs, APIs, and safe data transmission.
Create CSS border-radius with visual controls for each corner. Generate circles, pills, and custom shapes.
Create CSS box shadows with multiple layers, live preview, and copy-ready code.