Paste a robots.txt file, a URL and a user agent, and see straight away whether that crawler may fetch the URL and which rule decided it. The check follows Google's documented rules: how a crawler picks its group, the longest match, and the * and $ wildcards.
Free, no sign-up. Your robots.txt never leaves your browser.
The box starts with an example file and a URL, so the result below is already filled in. Replace them with your own robots.txt (open yoursite.com/robots.txt in a new tab and copy it), the URL you want to check and the crawler's user agent. The result updates as you type.
The path and query string are tested; the host and anything after # are left out.
Try with the example file
Every rule that matches this path
Paths are case-sensitive. Characters outside ASCII are percent-encoded before matching, the way a crawler sends them. Content after 500 KiB is ignored, as Google does.
The rules come from Google's own documentation on how Google interprets robots.txt, which also says that Google reads at most 500 KiB of the file and that path values are case-sensitive.
The robots.txt Tester that used to sit in Search Console has been retired. In its place, Search Console has a robots.txt report, which shows the robots.txt files Google found for the top 20 hosts on your site, when each was last crawled, and any warnings or errors it ran into.
The report does not test a URL against your rules. For one URL, Google points to the URL Inspection tool, which tells you whether the last crawl was blocked by robots.txt, and to its open-source robots.txt library. This page fills the gap between the two: a quick test of a URL against any version of the file, including a draft you have not published yet. The guide to robots.txt in Google Search Console covers the report itself.
Testing one URL at a time is fine for a draft. To see which pages of a live site are blocked, the Full On-Page Check in GSC Wizard reads your robots.txt before every request and lists each page with Search Console impressions that robots.txt blocks for its crawler, GSCWizard-Bot, as a critical issue, with the rule that matched. The on-page SEO checker page shows an example report, and the GSCWizard-Bot page explains how to allow or block that crawler.
Yes. It runs in your browser, needs no sign-up and has no limit. Nothing you paste is sent anywhere.
No. It tests the text you paste, so you can try a change before you publish it. To test the live file, open /robots.txt on your site, copy everything and paste it in.
Use the token the crawler names in its documentation, such as Googlebot for Google Search, Googlebot-Image for Google Images or Bingbot for Bing. A group applies when its User-agent line names that token, and a crawler without a group of its own follows the User-agent: * group.
When a group names Googlebot, Google follows only that group and ignores the rules under User-agent: *. A Disallow you wrote for all crawlers therefore does not reach Googlebot unless you repeat it in the Googlebot group.
No. Disallow stops crawling, not indexing. Google can still index a blocked URL without its content when other pages link to it. To keep a page out of Google, let it be crawled and use a noindex meta tag or header instead.
No. Google ignores Crawl-delay, as it ignores every field other than User-agent, Allow, Disallow and Sitemap. Other crawlers, such as Bingbot, may honour it, and the tester points it out when your file has one.
The robots.txt report, which lists the robots.txt files Google found for the top 20 hosts on your site, when each was last crawled and any warnings or errors. To check a single URL, use the URL Inspection tool in Search Console or a tester like this one.
Crawl your site for technical and on-page issues.
Check whether Google has indexed your pages.
Find pages whose canonical tag points elsewhere.
See how your title and description look in Google.
Word and phrase frequencies for any text.
Every free Search Console and SEO tool on one page.
Practical GSC and GA4 workflows, product updates and SEO experiments. No spam, unsubscribe anytime.