BuyerCiteBot/1.0 — an honest GEO audit crawler that takes “no” for an answer.
BuyerCiteBot fetches public pages so BuyerCite customers can see how AI answer engines read their own sites. It always identifies itself, always honors robots.txt, and never attempts to defeat a CAPTCHA, challenge or WAF.
Mozilla/5.0 (compatible; BuyerCiteBot/1.0; +https://buyercite.com/bot)
When BuyerCiteBot visits a site
A BuyerCite customer audits pages on their own workspace domain to check structured data, answer-first extractability and AI crawler readiness.
A visitor runs the zero-signup free GEO audit on a single URL. Each visitor gets one free URL audit; further audits need an account.
Before any page fetch it reads /robots.txt, then sitemaps and /llms.txt when present. It never crawls a whole site unprompted.
What BuyerCiteBot/1.0 + AI Onboarding Prepares Automatically
When a customer adds their own domain, BuyerCiteBot crawls a handful of its public pages — robots.txt-compliant, zero CAPTCHA solving, and paused by the 3-block / 24h circuit breaker — then combines what it read with BuyerCite Search Intelligence and BuyerCite AI to pre-populate Brand DNA, Target Keywords, Guardrail Rules and Source-Linked Brand Facts. Nothing goes live until a human reviews and confirms it.
-
robots.txt & 24 AI crawler policy check
Reads your robots.txt first and records what it allows for BuyerCiteBot and for each of the 24 AI crawlers we track.
-
sitemap.xml & /llms.txt discovery
Finds your sitemap(s) and /llms.txt so the crawl plan covers your real key pages, not guesses.
-
BuyerCiteBot/1.0 multi-page content harvest
Fetches a small plan of public pages (homepage, about, product, pricing…) with the honest user-agent; a robots.txt Disallow is never fetched.
-
BuyerCite Search Intelligence: Google keywords, search volumes & category peers
Looks up the Google keywords your domain ranks for, high-intent category keywords it does not rank for yet, their search volumes, and the real competitors in your category (navigational and login noise filtered out).
-
BuyerCite AI Brand DNA, Persona & Selling Pointers synthesis
BuyerCite AI reads only the harvested pages and drafts your brand name and aliases, category, buyer persona, markets, summary and key selling points.
-
BuyerCite AI Topics-to-Avoid & Hard-Fact Guardrail drafting
Proposes off-niche topics to avoid and editorial guardrails, then compiles them into enforceable brand rules.
-
BuyerCite AI Verified Brand Facts extraction (with source URLs)
Extracts concrete facts stated on your own pages, each linked to the exact crawled URL it came from — never an invented source.
-
Topic relevance check & brand guardrails readiness
Builds the topic relevance check and writing guardrails that keep prompts, keywords and content on-brand, ready for your review.
Human approval, always
The AI only drafts. The customer reviews every pre-filled field on the Brand DNA step, edits or deletes any fact, and only the confirmed version is used for AI-answer tracking and content. AI-extracted facts may cite only pages BuyerCiteBot actually fetched from the customer's own domain.
What BuyerCiteBot will — and will never — do
Always
- Identifies itself with the exact user-agent above, on plain HTTP fetches and on headless renders alike.
- Honors
robots.txt(RFC 9309), includingCrawl-delay. An unreachablerobots.txt(5xx or network failure) is treated as a complete disallow. - Fetches only public hosts: every hop is checked so private, loopback and internal addresses are never contacted.
- Reports a block honestly as “not measurable” — it never pretends a blocked page passed.
Never
- Solves, outsources or bypasses CAPTCHAs, JavaScript challenges or bot checks. Every challenge response is final.
- Uses stealth plugins, browser fingerprint spoofing or rotating IPs to get around a block. A
403is never retried from a different IP. - Impersonates another crawler to crawl. The per-AI-bot access check sends each AI bot's user-agent only to the customer's own verified domain, once per bot per run, and labels every result “simulated”.
- Overrides a
Disallow— even on a customer's own domain, a disallow is reported as a finding, not ignored.
The 3-strike, 24-hour circuit breaker
If a host answers with 3 consecutive 403 or challenge responses, BuyerCiteBot marks it WAF_BLOCKED and sends no further requests to that host for 24 hours. The customer is asked to use an owner-authorized channel instead — the WordPress plugin, an allowlist rule they control, or pasted content.
Allowlisting BuyerCiteBot on your WAF
If you use BuyerCite to audit your own site and your WAF blocks the audit, you can let BuyerCiteBot through. Allow it only on hosts you own, and keep the rule as narrow as you can.
We do not publish a fixed IP list
BuyerCiteBot's outbound traffic may go through an egress proxy provider, so its source IPs are not stable enough to publish. Allowlist by user-agent instead, scope the rule to the paths you want audited, and email [email protected] if you need to confirm a specific request came from us. User-agent strings can be copied by others, so never use this rule to grant access to anything private.
Cloudflare (WAF custom rule)
Security → WAF → Custom rules → Create rule. Set the action to Skip and choose the managed features to skip (for example Bot Fight Mode / Super Bot Fight Mode and managed rules).
(http.user_agent contains "BuyerCiteBot/1.0")
(http.user_agent contains "BuyerCiteBot/1.0" and http.host eq "www.example.com" and starts_with(http.request.uri.path, "/products/"))
Akamai, Fastly, AWS WAF & others
- Create a match condition on the
User-Agentheader containingBuyerCiteBot/1.0. - Combine it with a host (and ideally path) condition for the site you are auditing.
- Exempt that match from bot-management or challenge actions only — keep rate limits and other protections on.
- Remove the rule when you no longer need audits.
After adding a rule, re-run the audit in BuyerCite. If the host was already paused by the circuit breaker, the audit can fetch again once the 24-hour pause ends.
Blocking BuyerCiteBot with robots.txt
BuyerCiteBot matches the BuyerCiteBot product token (case-insensitive) in your robots.txt. A disallowed URL is never fetched.
Block everything
User-agent: BuyerCiteBot Disallow: /
Block some paths
User-agent: BuyerCiteBot Disallow: /checkout/ Disallow: /account/ Allow: /
Slow it down
User-agent: BuyerCiteBot Crawl-delay: 10 Allow: /
See BuyerCite's own robots.txt and /llms.txt for examples.
The 24 AI crawlers an audit evaluates
For each of these user agents, an audit reports what your robots.txt policy says. Live access checks for them run only on a customer's own verified domain and are labeled “simulated”.
- GPTBot
- OAI-SearchBot
- ChatGPT-User
- Google-Extended
- ClaudeBot
- Claude-User
- Claude-SearchBot
- Meta-ExternalAgent
- PerplexityBot
- Applebot-Extended
- GoogleOther
- GrokBot
- anthropic-ai
- Bytespider
- CCBot
- cohere-ai
- Diffbot
- FacebookBot
- ImagesiftBot
- Meta-ExternalFetcher
- Omgilibot
- Bingbot
- Googlebot
- Perplexity-User
Seeing unexpected BuyerCiteBot traffic?
Tell us the host, timestamps and request paths. We investigate every report and can stop audits of your domain. Email [email protected].