Skip to content
Crawler Policy · Official Documentation

BuyerCiteBot/1.0 — an honest GEO audit crawler that takes “no” for an answer.

BuyerCiteBot fetches public pages so BuyerCite customers can see how AI answer engines read their own sites. It always identifies itself, always honors robots.txt, and never attempts to defeat a CAPTCHA, challenge or WAF.

User-Agent
Mozilla/5.0 (compatible; BuyerCiteBot/1.0; +https://buyercite.com/bot)
Honors robots.txt & Crawl-delay Zero CAPTCHA solving 3-strike / 24h circuit breaker
What it does

When BuyerCiteBot visits a site

01 · On-demand GEO audits

A BuyerCite customer audits pages on their own workspace domain to check structured data, answer-first extractability and AI crawler readiness.

02 · Free instant audits

A visitor runs the zero-signup free GEO audit on a single URL. Each visitor gets one free URL audit; further audits need an account.

03 · Preflight files

Before any page fetch it reads /robots.txt, then sitemaps and /llms.txt when present. It never crawls a whole site unprompted.

Onboarding workflow

What BuyerCiteBot/1.0 + AI Onboarding Prepares Automatically

When a customer adds their own domain, BuyerCiteBot crawls a handful of its public pages — robots.txt-compliant, zero CAPTCHA solving, and paused by the 3-block / 24h circuit breaker — then combines what it read with BuyerCite Search Intelligence and BuyerCite AI to pre-populate Brand DNA, Target Keywords, Guardrail Rules and Source-Linked Brand Facts. Nothing goes live until a human reviews and confirms it.

  1. robots.txt & 24 AI crawler policy check

    Reads your robots.txt first and records what it allows for BuyerCiteBot and for each of the 24 AI crawlers we track.

  2. sitemap.xml & /llms.txt discovery

    Finds your sitemap(s) and /llms.txt so the crawl plan covers your real key pages, not guesses.

  3. BuyerCiteBot/1.0 multi-page content harvest

    Fetches a small plan of public pages (homepage, about, product, pricing…) with the honest user-agent; a robots.txt Disallow is never fetched.

  4. BuyerCite Search Intelligence: Google keywords, search volumes & category peers

    Looks up the Google keywords your domain ranks for, high-intent category keywords it does not rank for yet, their search volumes, and the real competitors in your category (navigational and login noise filtered out).

  5. BuyerCite AI Brand DNA, Persona & Selling Pointers synthesis

    BuyerCite AI reads only the harvested pages and drafts your brand name and aliases, category, buyer persona, markets, summary and key selling points.

  6. BuyerCite AI Topics-to-Avoid & Hard-Fact Guardrail drafting

    Proposes off-niche topics to avoid and editorial guardrails, then compiles them into enforceable brand rules.

  7. BuyerCite AI Verified Brand Facts extraction (with source URLs)

    Extracts concrete facts stated on your own pages, each linked to the exact crawled URL it came from — never an invented source.

  8. Topic relevance check & brand guardrails readiness

    Builds the topic relevance check and writing guardrails that keep prompts, keywords and content on-brand, ready for your review.

Human approval, always

The AI only drafts. The customer reviews every pre-filled field on the Brand DNA step, edits or deletes any fact, and only the confirmed version is used for AI-answer tracking and content. AI-extracted facts may cite only pages BuyerCiteBot actually fetched from the customer's own domain.

Ethical crawling rules

What BuyerCiteBot will — and will never — do

Always

  • Identifies itself with the exact user-agent above, on plain HTTP fetches and on headless renders alike.
  • Honors robots.txt (RFC 9309), including Crawl-delay. An unreachable robots.txt (5xx or network failure) is treated as a complete disallow.
  • Fetches only public hosts: every hop is checked so private, loopback and internal addresses are never contacted.
  • Reports a block honestly as “not measurable” — it never pretends a blocked page passed.

Never

  • Solves, outsources or bypasses CAPTCHAs, JavaScript challenges or bot checks. Every challenge response is final.
  • Uses stealth plugins, browser fingerprint spoofing or rotating IPs to get around a block. A 403 is never retried from a different IP.
  • Impersonates another crawler to crawl. The per-AI-bot access check sends each AI bot's user-agent only to the customer's own verified domain, once per bot per run, and labels every result “simulated”.
  • Overrides a Disallow — even on a customer's own domain, a disallow is reported as a finding, not ignored.

The 3-strike, 24-hour circuit breaker

If a host answers with 3 consecutive 403 or challenge responses, BuyerCiteBot marks it WAF_BLOCKED and sends no further requests to that host for 24 hours. The customer is asked to use an owner-authorized channel instead — the WordPress plugin, an allowlist rule they control, or pasted content.

For site owners

Allowlisting BuyerCiteBot on your WAF

If you use BuyerCite to audit your own site and your WAF blocks the audit, you can let BuyerCiteBot through. Allow it only on hosts you own, and keep the rule as narrow as you can.

We do not publish a fixed IP list

BuyerCiteBot's outbound traffic may go through an egress proxy provider, so its source IPs are not stable enough to publish. Allowlist by user-agent instead, scope the rule to the paths you want audited, and email [email protected] if you need to confirm a specific request came from us. User-agent strings can be copied by others, so never use this rule to grant access to anything private.

Cloudflare (WAF custom rule)

Security → WAF → Custom rules → Create rule. Set the action to Skip and choose the managed features to skip (for example Bot Fight Mode / Super Bot Fight Mode and managed rules).

Expression
(http.user_agent contains "BuyerCiteBot/1.0")
Narrower: product pages only
(http.user_agent contains "BuyerCiteBot/1.0"
 and http.host eq "www.example.com"
 and starts_with(http.request.uri.path, "/products/"))

Akamai, Fastly, AWS WAF & others

  • Create a match condition on the User-Agent header containing BuyerCiteBot/1.0.
  • Combine it with a host (and ideally path) condition for the site you are auditing.
  • Exempt that match from bot-management or challenge actions only — keep rate limits and other protections on.
  • Remove the rule when you no longer need audits.

After adding a rule, re-run the audit in BuyerCite. If the host was already paused by the circuit breaker, the audit can fetch again once the 24-hour pause ends.

Opting out

Blocking BuyerCiteBot with robots.txt

BuyerCiteBot matches the BuyerCiteBot product token (case-insensitive) in your robots.txt. A disallowed URL is never fetched.

Block everything

User-agent: BuyerCiteBot
Disallow: /

Block some paths

User-agent: BuyerCiteBot
Disallow: /checkout/
Disallow: /account/
Allow: /

Slow it down

User-agent: BuyerCiteBot
Crawl-delay: 10
Allow: /

See BuyerCite's own robots.txt and /llms.txt for examples.

AI crawler readiness

The 24 AI crawlers an audit evaluates

For each of these user agents, an audit reports what your robots.txt policy says. Live access checks for them run only on a customer's own verified domain and are labeled “simulated”.

  • GPTBot
  • OAI-SearchBot
  • ChatGPT-User
  • Google-Extended
  • ClaudeBot
  • Claude-User
  • Claude-SearchBot
  • Meta-ExternalAgent
  • PerplexityBot
  • Applebot-Extended
  • GoogleOther
  • GrokBot
  • anthropic-ai
  • Bytespider
  • CCBot
  • cohere-ai
  • Diffbot
  • FacebookBot
  • ImagesiftBot
  • Meta-ExternalFetcher
  • Omgilibot
  • Bingbot
  • Googlebot
  • Perplexity-User

Seeing unexpected BuyerCiteBot traffic?

Tell us the host, timestamps and request paths. We investigate every report and can stop audits of your domain. Email [email protected].