canaicrawl Get early access

Who blocks AI · #283 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does latimes.com block AI crawlers?

Yes. latimes.com’s robots.txt blocks 3 of the 19 AI crawlers we track, including GPTBot, ClaudeBot and Bytespider. Search engines such as Googlebot and Bingbot may crawl it.

GPTBotBlockedOAI-SearchBotAllowedClaudeBotBlockedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed
robots.txt
www.latimes.com/robots.txt · 93 lines
llms.txt
none
Checked
9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

BotOperatorrobots.txtDeciding line
GooglebotGoogleAllowedno rule matches the home page
BingbotMicrosoftAllowedno rule matches the home page
DuckDuckBotDuckDuckGoAllowedno rule matches the home page
ApplebotAppleAllowedno rule matches the home page
PetalBotHuaweiAllowedno rule matches the home page
YandexBotYandexAllowedno rule matches the home page
BaiduspiderBaiduAllowedno rule matches the home page

AI search

BotOperatorrobots.txtDeciding line
OAI-SearchBotOpenAIAllowedno rule matches the home page
PerplexityBotPerplexityAllowedno rule matches the home page
Claude-SearchBotAnthropicAllowedno rule matches the home page
DuckAssistBotDuckDuckGoAllowedno rule matches the home page
AmazonbotAmazonAllowedno rule matches the home page
YouBotYou.comAllowedno rule matches the home page

AI training

BotOperatorrobots.txtDeciding line
GPTBotOpenAIBlockedline 81 · Disallow: / under User-agent: GPTBot
ClaudeBotAnthropicBlockedline 72 · Disallow: / under User-agent: ClaudeBot
Google-ExtendedGoogleAllowedno rule matches the home page
CCBotCommon CrawlAllowedno rule matches the home page
Applebot-ExtendedAppleAllowedno rule matches the home page
GoogleOtherGoogleAllowedno rule matches the home page
Meta-ExternalAgentMetaAllowedno rule matches the home page
BytespiderByteDanceBlockedline 69 · Disallow: / under User-agent: Bytespider

User-triggered fetchers

BotOperatorrobots.txtDeciding line
ChatGPT-UserOpenAIAllowedno rule matches the home page
Claude-UserAnthropicAllowedno rule matches the home page
Perplexity-UserPerplexityAllowedno rule matches the home page
MistralAI-UserMistralAllowedno rule matches the home page
Meta-ExternalFetcherMetaAllowedno rule matches the home page

SEO tools

BotOperatorrobots.txtDeciding line
AhrefsBotAhrefsAllowedno rule matches the home page
SemrushBotSemrushAllowedno rule matches the home page
MJ12botMajesticAllowedno rule matches the home page
DotBotMozAllowedno rule matches the home page
View www.latimes.com/robots.txt · 93 lines
1User-agent: *
2Disallow: /search
3Disallow: /recipes/search
4Disallow: /changebrowser
5Disallow: /thirdpartyservice
6Disallow: /*/thirdpartyservice
7Disallow: /config/
8Disallow: /*/config
9Disallow: /dzcfg
10Disallow: /*/dzcfg
11Disallow: /*/test-template
12Disallow: /test/
13Disallow: /includes/
14Disallow: /_comments/
15Disallow: /deeplinkid/
16Disallow: /searchsuggest
17Disallow: /*/searchsuggest
18Disallow: /svgimageproc
19Disallow: /hive/
20Disallow: /get-galleryfragment*
21Disallow: /gs-*
22
23User-agent: Googlebot-News
24Disallow: /*-ugc-*.html
25Disallow: /about/
26Disallow: /cbp-*.html
27Disallow: /*/cbp-*.html
28Disallow: /bp/*
29Disallow: /*-adstory.html
30Disallow: /*sns-ap
31Disallow: /*photogallery
32Disallow: /brandpublishing/
33Disallow: /*/brandpublishing/
34Disallow: /shopping/
35Disallow: /*/shopping/
36Disallow: /paid-posts/
37Disallow: /*/paid-posts/
38Disallow: /search
39Disallow: /*/search
40Disallow: /changebrowser
41Disallow: /thirdpartyservice
42Disallow: /*/thirdpartyservice
43Disallow: /config/
44Disallow: /*/config
45Disallow: /dzcfg
46Disallow: /*/dzcfg
47Disallow: /*/test-template
48Disallow: /test/
49Disallow: /_comments/
50Disallow: /deeplinkid/
51Disallow: /searchsuggest
52Disallow: /*/searchsuggest
53Disallow: /svgimageproc
54Disallow: /hive/
55Disallow: /get-galleryfragment*
56Disallow: /sponsored/
57Disallow: /*/sponsored/
58
59User-agent: Twitterbot
60Allow: /
61
62User-agent: Mediapartners-Google
63Allow: /
64
65User-agent: AmazonAdBot
66Allow: /
67
68User-agent: Bytespider
69Disallow: /
70
71User-agent: ClaudeBot
72Disallow: /
73
74User-agent: Diffbot
75Disallow: /
76
77User-agent: discobot
78Disallow: /
79
80User-agent: GPTBot
81Disallow: /
82
83User-agent: ImagesiftBot
84Disallow: /
85
86User-agent: omgili
87Disallow: /
88
89User-agent: omgilibot
90Disallow: /
91
92Sitemap: https://www.latimes.com/sitemaps/sitemap.xml
93Sitemap: https://www.latimes.com/news-sitemap.xml

What do the bots actually get?

This page reads robots.txt, which states what latimes.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on latimes.com Test a path in the robots.txt tester

History

Questions

Does latimes.com block GPTBot?

Yes. latimes.com’s robots.txt disallows GPTBot from the whole site (line 81: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does latimes.com block ClaudeBot?

Yes. latimes.com’s robots.txt disallows ClaudeBot from the whole site (line 72: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does latimes.com block Google-Extended?

No. latimes.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does latimes.com block PerplexityBot?

No. latimes.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find latimes.com?

Yes. latimes.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.