canaicrawl Get early access

Who blocks AI · #373 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does bbb.org block AI crawlers?

No. bbb.org’s robots.txt lets all 19 AI crawlers we track crawl the site, though it keeps OAI-SearchBot, PerplexityBot and 9 more out of some paths. Search engines such as Googlebot and Bingbot may crawl it.

GPTBotPartly blockedOAI-SearchBotPartly blockedClaudeBotPartly blockedPerplexityBotPartly blockedGoogle-ExtendedPartly blockedCCBotPartly blocked
robots.txt
www.bbb.org/robots.txt · 84 lines
llms.txt
found
Checked
9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

BotOperatorrobots.txtDeciding line
GooglebotGoogleAllowedno rule matches the home page
BingbotMicrosoftAllowedno rule matches the home page
DuckDuckBotDuckDuckGoAllowedno rule matches the home page
ApplebotAppleAllowedno rule matches the home page
PetalBotHuaweiAllowedno rule matches the home page
YandexBotYandexAllowedno rule matches the home page
BaiduspiderBaiduAllowedno rule matches the home page

AI search

BotOperatorrobots.txtDeciding line
OAI-SearchBotOpenAIPartly blockedline 18 · Disallow: /cdn-cgi/ line 19 · Disallow: /util/ and 18 more
PerplexityBotPerplexityPartly blockedline 18 · Disallow: /cdn-cgi/ line 19 · Disallow: /util/ and 18 more
Claude-SearchBotAnthropicPartly blockedline 18 · Disallow: /cdn-cgi/ line 19 · Disallow: /util/ and 18 more
DuckAssistBotDuckDuckGoAllowedno rule matches the home page
AmazonbotAmazonAllowedno rule matches the home page
YouBotYou.comAllowedno rule matches the home page

AI training

BotOperatorrobots.txtDeciding line
GPTBotOpenAIPartly blockedline 18 · Disallow: /cdn-cgi/ line 19 · Disallow: /util/ and 18 more
ClaudeBotAnthropicPartly blockedline 18 · Disallow: /cdn-cgi/ line 19 · Disallow: /util/ and 18 more
Google-ExtendedGooglePartly blockedline 18 · Disallow: /cdn-cgi/ line 19 · Disallow: /util/ and 18 more
CCBotCommon CrawlPartly blockedline 18 · Disallow: /cdn-cgi/ line 19 · Disallow: /util/ and 18 more
Applebot-ExtendedApplePartly blockedline 18 · Disallow: /cdn-cgi/ line 19 · Disallow: /util/ and 18 more
GoogleOtherGoogleAllowedno rule matches the home page
Meta-ExternalAgentMetaAllowedno rule matches the home page
BytespiderByteDanceAllowedno rule matches the home page

User-triggered fetchers

BotOperatorrobots.txtDeciding line
ChatGPT-UserOpenAIPartly blockedline 18 · Disallow: /cdn-cgi/ line 19 · Disallow: /util/ and 18 more
Claude-UserAnthropicPartly blockedline 18 · Disallow: /cdn-cgi/ line 19 · Disallow: /util/ and 18 more
Perplexity-UserPerplexityPartly blockedline 18 · Disallow: /cdn-cgi/ line 19 · Disallow: /util/ and 18 more
MistralAI-UserMistralAllowedno rule matches the home page
Meta-ExternalFetcherMetaAllowedno rule matches the home page

SEO tools

BotOperatorrobots.txtDeciding line
AhrefsBotAhrefsAllowedno rule matches the home page
SemrushBotSemrushAllowedno rule matches the home page
MJ12botMajesticAllowedno rule matches the home page
DotBotMozAllowedno rule matches the home page
View www.bbb.org/robots.txt · 84 lines
1#fb external hits.
2User-agent: facebookexternalhit
3Allow: /*/public/insights/share/anniversary*?
4
5User-agent: GPTBot
6User-agent: OAI-SearchBot
7User-agent: ChatGPT-User
8User-agent: ClaudeBot
9User-agent: Claude-User
10User-agent: Claude-SearchBot
11User-agent: anthropic-ai
12User-agent: PerplexityBot
13User-agent: Perplexity-User
14User-agent: Google-Extended
15User-agent: Applebot-Extended
16User-agent: CCBot
17Content-signal: search=yes, ai-input=yes, ai-train=yes
18Disallow: /cdn-cgi/
19Disallow: /util/
20Disallow: /authentication/*
21Disallow: /business-reviews/*
22Disallow: /get-listed/success
23Disallow: /apply/thank-you
24Disallow: /bbb-reports-on
25Disallow: /contact-form
26Disallow: /verify-sms
27Disallow: /submit-review-expiration
28Disallow: /submit-review-thank-you
29Disallow: /dynamic-seal
30Disallow: /leave-a-review/no-bid
31Disallow: /leave-a-review/no-bid/thank-you
32Disallow: /leave-a-review/no-bid/expiration
33Disallow: /manage-location
34Disallow: /file-a-complaint/*
35Disallow: /accredited-business-directory/*?
36Disallow: /test-all-page
37Disallow: /*?
38Allow: /us/*/*/profile/*/*?
39Allow: /ca/*/*/profile/*/*?
40
41#block all other user agents from the following
42User-agent: *
43Content-signal: search=yes, ai-input=yes, ai-train=yes
44Disallow: /cdn-cgi/
45Disallow: /util/
46Disallow: /authentication/*
47Disallow: /business-reviews/*
48Disallow: /get-listed/success
49Disallow: /apply/thank-you
50Disallow: /bbb-reports-on
51Disallow: /contact-form
52Disallow: /verify-sms
53Disallow: /submit-review-expiration
54Disallow: /submit-review-thank-you
55Disallow: /dynamic-seal
56Disallow: /leave-a-review/no-bid
57Disallow: /leave-a-review/no-bid/thank-you
58Disallow: /leave-a-review/no-bid/expiration
59Disallow: /manage-location
60Disallow: /file-a-complaint/*
61Disallow: /accredited-business-directory/*?
62Disallow: /test-all-page
63Disallow: /*?
64Allow: /us/*/*/profile/*/*?
65Allow: /ca/*/*/profile/*/*?
66
67sitemap: https://www.bbb.org/sitemap-business-profiles-index.xml
68sitemap: https://www.bbb.org/sitemap-accredited-business-profiles-index.xml
69sitemap: https://www.bbb.org/sitemap-articles-index.xml
70sitemap: https://www.bbb.org/sitemap-nearme-articles-index.xml
71sitemap: https://www.bbb.org/sitemap-events-index.xml
72sitemap: https://www.bbb.org/sitemap-ca-categories-index.xml
73sitemap: https://www.bbb.org/sitemap-accredited-ca-categories-index.xml
74sitemap: https://www.bbb.org/sitemap-us-categories-index.xml
75sitemap: https://www.bbb.org/sitemap-accredited-us-categories-index.xml
76sitemap: https://www.bbb.org/sitemap-ca-province-categories-index.xml
77sitemap: https://www.bbb.org/sitemap-accredited-ca-province-categories-index.xml
78sitemap: https://www.bbb.org/sitemap-us-state-categories-index.xml
79sitemap: https://www.bbb.org/sitemap-accredited-us-state-categories-index.xml
80sitemap: https://www.bbb.org/sitemap-ca-city-categories-index.xml
81sitemap: https://www.bbb.org/sitemap-accredited-ca-city-categories-index.xml
82sitemap: https://www.bbb.org/sitemap-us-city-categories-index.xml
83sitemap: https://www.bbb.org/sitemap-accredited-us-city-categories-index.xml
84

What do the bots actually get?

This page reads robots.txt, which states what bbb.org asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on bbb.org Test a path in the robots.txt tester

History

Questions

Does bbb.org block GPTBot?

Partly. bbb.org’s robots.txt keeps GPTBot out of some paths, such as /cdn-cgi/, and lets it crawl the rest. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does bbb.org block ClaudeBot?

Partly. bbb.org’s robots.txt keeps ClaudeBot out of some paths, such as /cdn-cgi/, and lets it crawl the rest. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does bbb.org block Google-Extended?

Partly. bbb.org’s robots.txt keeps Google-Extended out of some paths, such as /cdn-cgi/, and lets it crawl the rest. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does bbb.org block PerplexityBot?

Partly. bbb.org’s robots.txt keeps PerplexityBot out of some paths, such as /cdn-cgi/, and lets it crawl the rest. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find bbb.org?

Partly. bbb.org’s robots.txt keeps OAI-SearchBot out of some paths, such as /cdn-cgi/, and lets it crawl the rest. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.