Who blocks AI · #802 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does weather.com block AI crawlers?
Yes. weather.com’s robots.txt blocks 5 of the 19 AI crawlers we track, including CCBot, Bytespider, Amazonbot and 2 more. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotBlocked- robots.txt
- weather.com/robots.txt · 191 lines
- llms.txt
- found
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Allowed | no rule matches the home page |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Blocked | line 116 · Disallow: / under User-agent: Amazonbot |
YouBot | You.com | Blocked | line 178 · Disallow: / under User-agent: YouBot |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | no rule matches the home page |
ClaudeBot | Anthropic | Allowed | no rule matches the home page |
Google-Extended | Allowed | no rule matches the home page | |
CCBot | Common Crawl | Blocked | line 126 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Allowed | no rule matches the home page |
GoogleOther | Blocked | line 144 · Disallow: / under User-agent: GoogleOther | |
Meta-ExternalAgent | Meta | Allowed | no rule matches the home page |
Bytespider | ByteDance | Blocked | line 123 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | line 20 · Allow: / under User-agent: ChatGPT-User |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View weather.com/robots.txt · 191 lines
1#
2# /robots.txt
3#
4
5#
6# Last updated by nick.hester 10/26/2026
7#
8
9#
10# AI / AEO bot policy (CW-9994)
11# We explicitly allow AI bots that fetch pages on behalf of a user's direct
12# request (answer engines like ChatGPT-User, Brightbot) so our content can
13# be cited in AI-generated answers. Bulk AI-training/indexing crawlers
14# remain blocked below under "Block Bots" (e.g. YouBot, Google-CloudVertexBot).
15#
16
17# Explicit Allows
18
19User-agent: ChatGPT-User
20Allow: /
21
22User-agent: AmazonAdBot
23Allow: /
24
25User-agent: Brightbot 1.0
26Allow: /
27
28User-agent: CriteoBot/0.1
29Allow: /
30
31# Disallowed for PhantomJS
32
33User-agent: *
34Crawl-delay: 10
35# Directories
36Disallow: /includes/
37Disallow: /life/
38Disallow: /misc/
39Disallow: /modules/
40Disallow: /profiles/
41Disallow: /scripts/
42Disallow: /themes/
43Disallow: /api/payload
44Allow: /api/payload/v1/sitemaps
45Disallow: /taxonomy/
46# Files
47Disallow: /appspromo
48Disallow: /CHANGELOG.txt
49Disallow: /cron.php
50Disallow: /INSTALL.mysql.txt
51Disallow: /INSTALL.pgsql.txt
52Disallow: /INSTALL.sqlite.txt
53Disallow: /install.php
54Disallow: /INSTALL.txt
55Disallow: /LICENSE.txt
56Disallow: /MAINTAINERS.txt
57Disallow: /update.php
58Disallow: /UPGRADE.txt
59Disallow: /xmlrpc.php
60# Paths (clean URLs)
61Disallow: /migration/
62Disallow: /admin/
63Disallow: /comment/reply/
64Disallow: /filter/tips/
65Disallow: /node/add/
66Disallow: /search/
67Disallow: /user/register/
68Disallow: /user/password/
69Disallow: /user/login/
70Disallow: /user/logout/
71# Paths (no clean URLs)
72Disallow: /*?apiKey=
73Disallow: /*&apiKey=
74Disallow: /?q=admin/
75Disallow: /?q=comment/reply/
76Disallow: /?q=filter/tips/
77Disallow: /?q=node/add/
78Disallow: /?q=search/
79Disallow: /?q=user/password/
80Disallow: /?q=user/register/
81Disallow: /?q=user/login/
82Disallow: /?q=user/logout/
83
84Disallow: /sponsored
85Disallow: /ugc
86Disallow: /sponsored-content
87
88Disallow: /api/v1/p/redux-dal
89# International weather language URL paths (404s)
90Disallow: /tiempo/
91Disallow: /clima/
92Disallow: /temps/
93Disallow: /vreme/
94Disallow: /idojaras/
95Disallow: /pogoda/
96Disallow: /wetter/
97Disallow: /vejret/
98Disallow: /pocasi/
99Disallow: /cuaca/
100Disallow: /weer/
101# Duplicate forecast URLs that canonicalize to the param-less page (CW-12397)
102Disallow: /*/hourbyhour/day-
103Disallow: /*?t=
104Disallow: /*&t=
105Disallow: /*?metric=
106Disallow: /*&metric=
107# Scoped to /localalerts so legacy /l/{id}?phenomena= 301s stay crawlable (CW-10292)
108Disallow: /*/localalerts?phenomena=
109Disallow: /*/localalerts?*&phenomena=
110
111#
112# Block Bots
113#
114
115User-agent: Amazonbot
116Disallow: /
117
118User-agent: AwarioRssBot
119User-agent: AwarioSmartBot
120Disallow: /
121
122User-agent: Bytespider
123Disallow: /
124
125User-agent: CCBot
126Disallow: /
127
128User-agent: DataForSeoBot
129Disallow: /
130
131User-agent: Diffbot
132Disallow: /
133
134User-agent: FacebookBot
135Disallow: /
136
137User-agent: FriendlyCrawler
138Disallow: /
139
140User-agent: Google-CloudVertexBot
141Disallow: /
142
143User-agent: GoogleOther
144Disallow: /
145
146User-agent: ImagesiftBot
147Disallow: /
148
149User-agent: magpie-crawler
150Disallow: /
151
152User-agent: NewsNow
153Disallow: /
154
155User-agent: news-please
156Disallow: /
157
158User-agent: omgili
159Disallow: /
160
161User-agent: omgilibot
162Disallow: /
163
164User-agent: peer39_crawler
165User-agent: peer39_crawler/1.0
166Disallow: /
167
168User-agent: Quora-Bot
169Disallow: /
170
171User-agent: Scrapy
172Disallow: /
173
174User-agent: TurnitinBot
175Disallow: /
176
177User-agent: YouBot
178Disallow: /
179
180#
181# Sitemaps
182#
183
184Sitemap: https://weather.com/en-US/sitemaps/bios-sitemap.xml
185Sitemap: https://weather.com/en-US/sitemaps/collections-manual.xml
186Sitemap: https://weather.com/api/payload/v1/sitemaps/en-US/news.xml
187Sitemap: https://weather.com/de-DE/sitemaps/sitemap.xml
188Sitemap: https://weather.com/es-US/sitemaps/sitemap.xml
189Sitemap: https://weather.com/api/payload/v1/sitemaps/en-US/sitemap.xml
190Sitemap: https://weather.com/api/location/sitemaps/en-US/sitemap.xml
191
What do the bots actually get?
This page reads robots.txt, which states what weather.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on weather.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does weather.com block GPTBot?
No. weather.com’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does weather.com block ClaudeBot?
No. weather.com’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does weather.com block Google-Extended?
No. weather.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does weather.com block PerplexityBot?
No. weather.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find weather.com?
Yes. weather.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.