Robots.txt Generator & Builder
Build clean, search-engine-compliant robots.txt crawl directives with allow/disallow rules and sitemap links.
Crawler Directives & Access Controls
User-agent: * rules.
Generated robots.txt Output
Valid SyntaxAbout the Robots.txt Generator & Builder
A robots.txt file is a plain text file placed in the root directory of a website that tells search engine crawlers and web spiders which pages they can or cannot request.
It serves as a first line of defense to protect private administration panels, prevent duplicate content indexing, and manage search bot crawling bandwidth.
This free builder creates fully validated robots.txt syntax with support for Googlebot, Bingbot, custom crawl delays, AI scraper blocks, and XML sitemaps.
How the Calculations Work
Converts your selected crawler permissions and disallowed directory paths into standard Robots Exclusion Protocol (REP) directives.
robots.txt = User-agent: [bot] + (Allow | Disallow): [path] + Sitemap: [url]
Variables & Definitions
- User-agent: *: Directives that apply to all search engines and web crawlers
- Disallow: /path/: Explicitly forbids compliant crawlers from requesting pages under that directory
- Allow: /: Permits crawling of pages under the designated path
- Sitemap:: Provides the absolute URL of the XML sitemap index
To protect WordPress administrative screens while allowing public search engines: "User-agent: * \n Disallow: /wp-admin/ \n Allow: /wp-admin/admin-ajax.php".
Key Terms Explained
Robots Exclusion Protocol
The standard convention adopted by webmasters and search engines since 1994 to regulate web robot behavior.
Crawl Budget
The number of pages a search engine bot will crawl and index on your website within a given timeframe.
GPTBot / CCBot
Web scrapers operated by OpenAI and Common Crawl used for training artificial intelligence language models.
Practical Tips & Pitfalls to Avoid
Never disallow CSS or JavaScript assets
Googlebot needs access to CSS and JS files to render pages properly and verify mobile responsiveness.
Robots.txt is not an access control mechanism
Sensitive directories (passwords, private databases) must be protected by server authentication, not just robots.txt.
Frequently Asked Questions
The robots.txt file must always be placed directly in the top-level root directory of your website (e.g. https://example.com/robots.txt).
Yes! Check the GPTBot and CCBot options in this tool to instruct compliant AI scrapers to bypass your content.
Not necessarily. If other websites link to the disallowed URL, Google may still show the URL in search results without a description. To fully de-index, use a "noindex" meta tag instead.