Free robots.txt checker
Fetch any site's robots.txt and test whether Googlebot, Bingbot, and AI crawlers like GPTBot, ClaudeBot, and PerplexityBot can crawl the site root or any path. Matching follows Google's robots.txt rules.
CalmSEO's server fetches https://your-domain/robots.txt, follows up to 5 redirects to public domains, and stops after 5 seconds. It reads the first 500 KiB, the same limit Google uses. Path matching runs in your browser. You can run 5 checks per minute.
What robots.txt does
robots.txt is a plain text file at a site's root, such as https://example.com/robots.txt. It tells crawlers which URLs they may request. Its rules cover only the host, protocol, and port where the file lives, so https://shop.example.com needs its own file.
The file is made of groups. Each group opens with one or more user-agent lines that name crawlers, followed by allow and disallow rules for URL paths. Sitemap lines can go anywhere and apply to every crawler.
User-agent: * Disallow: /cart/ Allow: /cart/help Sitemap: https://example.com/sitemap.xml
Google supports four fields: user-agent, allow, disallow, and sitemap. It ignores all others, including crawl-delay. The checker lists those lines under Lines Google ignores.
How to read the results
The status at the top shows the file's HTTP response and how Google handles it.
- 2xx: Google reads the file and follows its rules.
- 4xx other than 429: Google treats the site as having no robots.txt file and may crawl every URL.
- 5xx, 429, and network errors: Google stops crawling the site for 12 hours, then uses its last good copy of the file for up to 30 days.
The crawler table applies Google's matching rules. Each crawler follows the group that names it, matched case-insensitively. It uses the * group only when no group names it. Groups that name the same crawler are merged. Within a group, the rule with the longest matching path wins. When an allow and a disallow rule are the same length, allow wins.
In paths, * matches any run of characters and $ marks the end of the URL, so Disallow: /*.pdf$ blocks every URL ending in .pdf. Paths are case-sensitive, so /Blog and /blog are different rules.
Two crawlers fall back to another group when the file does not name them. Apple documents that Applebot follows Googlebot rules. Bing's webmaster blog says Bingbot follows msnbot rules.
How to allow or block AI crawlers
AI companies use separate user-agent tokens for model training, AI search, and pages fetched on a user's request. You can target each token with its own user-agent line.
- Model training: GPTBot (OpenAI) and ClaudeBot (Anthropic) crawl pages that may be used to train models. Google-Extended and Applebot-Extended do not crawl. They control whether pages crawled by Googlebot and Applebot can be used for Gemini and Apple model training.
- AI search: OAI-SearchBot, Claude-SearchBot, and PerplexityBot crawl pages for search in ChatGPT, Claude, and Perplexity.
- User requests: ChatGPT-User, Claude-User, and Perplexity-User fetch a page when a person asks about it. OpenAI says robots.txt rules may not apply to ChatGPT-User. Perplexity says Perplexity-User generally ignores robots.txt.
This group opts out of model training for OpenAI, Anthropic, Google, and Apple, and leaves their search crawlers on your * rules:
User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended Disallow: /
To block a crawler from part of the site, add a path such as Disallow: /drafts/ to its group. Google says blocking Google-Extended does not change how your site appears in Google Search. CCBot, which collects pages for Common Crawl's open web dataset, has its own token too.
Crawling and indexing
robots.txt controls which URLs a crawler may request. A blocked URL can still appear in Google results if other pages link to it. It usually shows without a description because Google cannot read the page.
To keep a page out of Google results, add a noindex meta tag or an X-Robots-Tag HTTP header. Google has to crawl the page to see noindex, so leave the URL open to crawling in robots.txt.
Check sitemaps and pages with your AI assistant
This checker reads one robots.txt file at a time. CalmSEO is an MCP server that connects Claude, ChatGPT, or Cursor to your Google Search Console data and to SEO tools your assistant can call.
Your assistant can list the sitemaps you submitted to Search Console with each one's last download date, errors, and warnings. It can also audit a page and report its status code, canonical tag, and meta robots tag. Search Console tools are free on every plan. Page audits cost 5 credits each.
See how the Google Search Console MCP works, or read about the SEO tools available to AI agents.
Frequently asked.
Is the robots.txt checker free?+
Yes, and it needs no account. A quick security check blocks automated use, and each visitor can run 5 checks per minute.
Is robots.txt case-sensitive?+
Field names such as user-agent and disallow are case-insensitive. So are crawler names such as Googlebot. Rule paths are case-sensitive, so Disallow: /Private does not block /private.
How long does Google take to notice robots.txt changes?+
Google usually caches robots.txt for up to 24 hours. The Cache-Control max-age header can make that period longer or shorter.
Does Google support crawl-delay?+
No. Google ignores crawl-delay and supports only user-agent, allow, disallow, and sitemap.
Does blocking GPTBot remove my site from ChatGPT search?+
No. GPTBot covers model training. ChatGPT search uses OAI-SearchBot, so your OAI-SearchBot rules decide search visibility. OpenAI notes that sites that block OAI-SearchBot can still appear as navigational links.
Does blocking Google-Extended affect Google Search rankings?+
No. Google says Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal. It controls whether content can be used for Gemini training and grounding.
Will blocking a page in robots.txt remove it from Google?+
Not always. A blocked URL can stay in Google results if other pages link to it. To remove a page, add a noindex meta tag or X-Robots-Tag header and let Google crawl the page so it sees the tag.
Review crawling and indexing across your whole site
20 free credits + free Google Search Console tools. No card required.