What is GSCWizard-Bot?
GSCWizard-Bot is the crawler that powers the On-Page SEO report inside GSC Wizard. It runs only when a signed-in user opens the report for a Google Search Console property they own - it does not spider the open web, follow links, or build any kind of index.
If you see this user agent in your access logs, it means a verified owner of your site (or someone with access to your Search Console property) opened the On-Page SEO Checker in GSC Wizard and asked it to analyse your pages.
The same user agent is used when a signed-in owner asks GSC Wizard (or the GSC Wizard MCP server) to read one of your XML sitemaps - for example in the Sitemap Performance report, which joins the URLs in a sitemap with their Search Console clicks and impressions. That is a single request for the sitemap file itself (plus its child sitemaps for a sitemap index), never a crawl of the URLs it lists.
You may therefore also see GSCWizard-Bot request files like
robots.txt and
sitemap.xml. The bot reads
robots.txt to discover the
Sitemap: directives it contains
(falling back to /sitemap.xml when
none are listed) so it can check which of your pages are included in your XML sitemap.
What the crawler checks
For every page it fetches, GSCWizard-Bot extracts three HTML elements and compares them against the highest-impression query that page already ranks for in Google Search Console over the last 28 days.
1
Page title - <title>
Reads the contents of the <title>
tag and checks whether every significant word (3+ characters) from the top GSC query appears in it.
Surfaces the current title verbatim alongside a pass/fail flag.
2
Meta description - <meta name="description">
Reads the content attribute of the
meta description and checks for the same keyword coverage. Also records the current character length.
3
H1 headings - <h1>
Extracts every <h1> on the page and
checks whether any of them contain the target keywords.
That's it. GSCWizard-Bot does not execute JavaScript, does not render pages, does not download stylesheets or scripts, does not follow links inside your pages, and does not collect any other data. For page checks it reads the raw HTML response only. The one exception is the feed audit: when a merchant connects their own Google Merchant Center account, the bot fetches the product images referenced in that merchant's feed to measure their pixel dimensions. Each image request asks for the first 64 KB only (a ranged request), the bot keeps the width and height, and it discards the bytes. It never fetches images it finds inside a page.
Crawl behaviour
Within a single run each URL is fetched once, with a hard 10-second timeout. Redirects are followed (up to five hops, each one re-checked). The same URL can be fetched again by a later run, for example when a scheduled audit repeats or a site owner re-runs a report, so you may see repeat requests for the same URL over time.
Successfully crawled URLs are stored and reused for up to 7 days by scheduled background runs. A user re-running the report always triggers a fresh crawl, so edits to a title, meta description, or H1 show up immediately - which means you can see repeat requests for the same URL when the site owner is actively working on it.
A page crawl (on-page checks and the internal link crawl) covers at most 1,000 pages per run, always the property's own pages, selected from the URLs Search Console reports for it. A feed audit covers every distinct product URL in the merchant's own Merchant Center feed and the images those feed entries reference; it has no fixed page cap and is bounded by politeness instead: a small number of parallel requests per host, backing off whenever your server answers 429 or 503, and the robots.txt rules described below.
Before any live page fetch, the crawler downloads your
robots.txt (once per site per run)
and evaluates its Allow /
Disallow rules for
GSCWizard-Bot (falling back to
*), including
* wildcards and
$ anchors. Disallowed URLs are never
fetched; they are reported back to the user in the report instead. A missing or unreachable
robots.txt is treated as "allow all".
For sitemap coverage checks the crawler requests
/robots.txt to find
Sitemap: directives, falls back to
/sitemap.xml, and follows a sitemap
index one level deep (at most 50 sitemap files / 50,000 URLs). Sitemap URLs are used for
comparison only - they are never queued for crawling.
Sends Accept: text/html and parses the
response body directly. No headless browser, no script execution, no asset fetching.
URLs that resolve to private, internal, or loopback IP ranges are blocked before any network request is made.
How to allow GSCWizard-Bot
Most sites need no changes. If you run a WAF, bot-management layer, or restrictive firewall and the On-Page SEO report comes back empty (or with HTTP 403 / 429 errors), whitelist the user agent.
robots.txt (explicit allow)
User-agent: GSCWizard-Bot
Allow: /Cloudflare WAF / Bot Fight Mode
Add a WAF custom rule under Security → WAF → Custom rules:
(http.user_agent contains "GSCWizard-Bot")
Action: Skip → All remaining custom rules, Bot Fight Mode, Super Bot Fight ModeNginx
if ($http_user_agent ~* "GSCWizard-Bot") {
set $allow_bot 1;
}
# use $allow_bot to bypass rate limits / challengesApache
SetEnvIfNoCase User-Agent "GSCWizard-Bot" allow_bot
<RequireAll>
Require all granted
Require env allow_bot
</RequireAll>How to block GSCWizard-Bot
GSCWizard-Bot checks your robots.txt
before every page fetch and honors its rules. Add the following rule to stop GSCWizard-Bot from
fetching any page on your site - blocked URLs are skipped and flagged as
"Blocked by robots.txt" in the report instead of being crawled:
User-agent: GSCWizard-Bot
Disallow: /
If you prefer a hard block at the edge, any rule that rejects requests where the
User-Agent header matches
GSCWizard-Bot will work - invert the
allow-rules above.
Verifying the user agent
Our crawler sends one of two user-agent strings. The On-Page SEO Checker (title / meta description / H1 checks and sitemap reads) uses the first; the extended technical audit (which also performs the robots.txt and sitemap-discovery requests) uses the second:
GSCWizard-Bot/1.0 (On-Page SEO Checker)
GSCWizard-Bot/1.0 (Full On-Page Checker)
Both share the GSCWizard-Bot prefix, so a
single substring rule covers them. Page requests carry
Accept: text/html and originate from our
hosted workers. If someone else is sending a user-agent string that contains "GSCWizard" but does not
match either string above, it is not us - treat it accordingly.
Questions or abuse reports
If you believe GSCWizard-Bot is misbehaving, crawling more than described above, or you'd like us to stop crawling your site entirely, please get in touch: