What is GSCWizard-Bot?
GSCWizard-Bot is the crawler that powers the On-Page SEO report inside GSC Wizard. It runs only when a signed-in user opens the report for a Google Search Console property they own - it does not spider the open web, follow links, or build any kind of index.
If you see this user agent in your access logs, it means a verified owner of your site (or someone with access to your Search Console property) opened the On-Page SEO Checker in GSC Wizard and asked it to analyse your pages.
The same user agent is used when a signed-in owner asks GSC Wizard (or the GSC Wizard MCP server) to read one of your XML sitemaps - for example in the Sitemap Performance report, which joins the URLs in a sitemap with their Search Console clicks and impressions. That is a single request for the sitemap file itself (plus its child sitemaps for a sitemap index), never a crawl of the URLs it lists.
You may therefore also see GSCWizard-Bot request files like
robots.txt and
sitemap.xml. The bot reads
robots.txt to discover the
Sitemap: directives it contains
(falling back to /sitemap.xml when
none are listed) so it can check which of your pages are included in your XML sitemap.
What the crawler checks
For every page it fetches, GSCWizard-Bot extracts three HTML elements and compares them against the highest-impression query that page already ranks for in Google Search Console over the last 28 days.
1
Page title - <title>
Reads the contents of the <title>
tag and checks whether every significant word (3+ characters) from the top GSC query appears in it.
Surfaces the current title verbatim alongside a pass/fail flag.
2
Meta description - <meta name="description">
Reads the content attribute of the
meta description and checks for the same keyword coverage. Also records the current character length.
3
H1 headings - <h1>
Extracts every <h1> on the page and
checks whether any of them contain the target keywords.
That's it. GSCWizard-Bot does not execute JavaScript, does not render pages, does not download stylesheets or scripts, does not follow links inside your pages, and does not collect any other data. For page checks it reads the raw HTML response only. The one exception is the feed audit: when a merchant connects their own Google Merchant Center account, the bot fetches the product images referenced in that merchant's feed to measure their pixel dimensions. Each image request asks for the first 64 KB only (a ranged request), the bot keeps the width and height, and it discards the bytes. It never fetches images it finds inside a page.
Crawl behaviour
Within a single run each URL is fetched once, with a hard 10-second timeout. Redirects are followed (up to five hops, each one re-checked). The same URL can be fetched again by a later run, for example when a scheduled audit repeats or a site owner re-runs a report, so you may see repeat requests for the same URL over time.
Successfully crawled URLs are stored and reused for up to 7 days by scheduled background runs. A user re-running the report always triggers a fresh crawl, so edits to a title, meta description, or H1 show up immediately - which means you can see repeat requests for the same URL when the site owner is actively working on it.
A page crawl (on-page checks and the internal link crawl) covers at most 1,000 pages per run, always the property's own pages, selected from the URLs Search Console reports for it. A feed audit covers every distinct product URL in the merchant's own Merchant Center feed and the images those feed entries reference; it has no fixed page cap and is bounded by politeness instead: a small number of parallel requests per host, backing off whenever your server answers 429 or 503, and the robots.txt rules described below.
Before any live page fetch, the crawler downloads your
robots.txt (once per site per run)
and evaluates its Allow /
Disallow rules for
GSCWizard-Bot (falling back to
*), including
* wildcards and
$ anchors. Disallowed URLs are never
fetched; they are reported back to the user in the report instead. A missing or unreachable
robots.txt is treated as "allow all".
For sitemap coverage checks the crawler requests
/robots.txt to find
Sitemap: directives, falls back to
/sitemap.xml, and follows a sitemap
index one level deep (at most 50 sitemap files / 50,000 URLs). Sitemap URLs are used for
comparison only - they are never queued for crawling.
Sends Accept: text/html and parses the
response body directly. No headless browser, no script execution, no asset fetching.
URLs that resolve to private, internal, or loopback IP ranges are blocked before any network request is made.
How to allow GSCWizard-Bot
Most sites need no changes. If you run a WAF, bot-management layer, or restrictive firewall and the On-Page SEO report comes back empty (or with HTTP 403 / 429 errors), whitelist the user agent.
robots.txt (explicit allow)
User-agent: GSCWizard-Bot
Allow: /Cloudflare WAF and Bot Fight Mode
For your own custom rules (country blocks, ASN rules and the like), add a rule under Security → WAF → Custom rules and place it above the rules that block:
(http.user_agent contains "GSCWizard-Bot")
Action: Skip → All remaining custom rules
(Pro plans and above: also All Super Bot Fight Mode rules)This does not work for Bot Fight Mode, the bot protection on Cloudflare's Free plan. Cloudflare runs it outside the WAF, so Skip and Allow actions and Page Rules have no effect on it. Short of turning it off, the only exception it honours is an IP Access Rule that matches the request first; email us for our current server address. An IP Access Rule also lets that address past your other security checks, so treat it as temporary: once Cloudflare lists GSCWizard-Bot as a verified bot (see Verifying the user agent), you will not need it.
Nginx
if ($http_user_agent ~* "GSCWizard-Bot") {
set $allow_bot 1;
}
# use $allow_bot to bypass rate limits / challengesApache
SetEnvIfNoCase User-Agent "GSCWizard-Bot" allow_bot
<RequireAll>
Require all granted
Require env allow_bot
</RequireAll>How to block GSCWizard-Bot
GSCWizard-Bot checks your robots.txt
before every page fetch and honors its rules. Add the following rule to stop GSCWizard-Bot from
fetching any page on your site - blocked URLs are skipped and flagged as
"Blocked by robots.txt" in the report instead of being crawled:
User-agent: GSCWizard-Bot
Disallow: /
If you prefer a hard block at the edge, any rule that rejects requests where the
User-Agent header matches
GSCWizard-Bot will work - invert the
allow-rules above.
Verifying the user agent
Our crawler sends one of three user-agent strings. The On-Page SEO Checker (title / meta description / H1 checks and sitemap reads) uses the first; the extended technical audit (which also performs the robots.txt and sitemap-discovery requests) uses the second; the Merchant Center feed audit uses the third when it measures the dimensions of the product images a merchant's own feed references:
GSCWizard-Bot/1.0 (On-Page SEO Checker)
GSCWizard-Bot/1.0 (Full On-Page Checker)
GSCWizard-Bot/1.0 (Feed Audit image probe)
All three share the GSCWizard-Bot prefix, so a
single substring rule covers them. Page requests carry
Accept: text/html and originate from our
application servers. A user-agent string is not proof of anything on its own, though - anyone can copy
one. If you want certainty, check the signature described below.
Cryptographic verification (Web Bot Auth)
Every request GSCWizard-Bot makes is signed with an Ed25519 key using HTTP Message Signatures (RFC 9421), the same Web Bot Auth scheme Cloudflare uses to identify verified bots. A signed request carries three extra headers:
Signature-Agent: "https://tool.gscwizard.com/.well-known/http-message-signatures-directory"
Signature-Input: sig1=("@authority" "signature-agent");created=...;keyid="...";alg="ed25519";expires=...;nonce="...";tag="web-bot-auth"
Signature: sig1=:...:Our public key is published as a JSON Web Key Set at tool.gscwizard.com/.well-known/http-message-signatures-directory. Verify the signature against that key and you know the request came from us and was not replayed against a different host: the signature covers the host it was sent to and expires after five minutes.
We have applied to Cloudflare's verified bots directory with this key, and the application is still
under review. Until it is approved, Cloudflare does not know our key and treats GSCWizard-Bot like any
other unverified bot, so Bot Fight Mode can challenge it. Once it is approved, Cloudflare checks the
signature for you: verified bots are left alone by Cloudflare's default bot protection, and you can
match them in custom rules with the Known Bots field,
cf.client.bot.
If you see a request claiming to be GSCWizard-Bot with no valid signature, it is not us; treat it accordingly, and please tell us about it.
Questions or abuse reports
If you believe GSCWizard-Bot is misbehaving, crawling more than described above, or you'd like us to stop crawling your site entirely, please get in touch: