Who blocks AI · #1139 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does thehindu.com block AI crawlers?
Yes. thehindu.com’s robots.txt blocks 10 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, PerplexityBot and 7 more. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotBlockedOAI-SearchBotBlockedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedAllowedCCBotBlocked- robots.txt
- www.thehindu.com/robots.txt · 166 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Blocked | line 151 · Disallow: / under User-agent: OAI-SearchBot |
PerplexityBot | Perplexity | Blocked | line 151 · Disallow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Blocked | line 151 · Disallow: / under User-agent: Amazonbot |
YouBot | You.com | Blocked | line 151 · Disallow: / under User-agent: YouBot |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 151 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Blocked | line 151 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Allowed | no rule matches the home page | |
CCBot | Common Crawl | Blocked | line 151 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Allowed | no rule matches the home page |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Blocked | line 151 · Disallow: / under User-agent: Meta-ExternalAgent |
Bytespider | ByteDance | Blocked | line 151 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Blocked | line 151 · Disallow: / under User-agent: ChatGPT-User |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.thehindu.com/robots.txt · 166 lines
1User-agent: *
2Disallow: /cgi-bin/
3Disallow: /cdn-cgi/*
4Disallow: /config/
5Disallow: */analysis-logger/*
6Disallow: /todays-paper/*article*.ece*
7Disallow: /migration_catalog/
8Disallow: */getapplink*
9Disallow: /summercamp/
10Disallow: */wf.fragment/*
11Disallow: /poll/vote.do
12Disallow: /photo/10000/
13Disallow: */photo/10000/
14#Disallow: /newsletter/*
15Disallow: /news-service/*
16#Disallow: /ebooks/
17Disallow: /search/
18Disallow: /SEARCH/
19Disallow: /Search/
20Disallow: /22390678/
21Disallow: /walkathon
22Disallow: /heelsonwheels
23Disallow: /archive/print/
24#Disallow: /todays-paper/
25#Disallow: /static/content/
26#Disallow: /static/
27Disallow: /todayspaper/
28Disallow: /today-paper/
29Disallow: /thseweekend
30Disallow: /ywsummercamp
31Disallow: /frnf2014
32Disallow: /summermusic
33Disallow: /summercamp
34Disallow: /yw25
35Disallow: /thischess
36Disallow: /ywquiz
37Disallow: /mumbaiedition
38Disallow: /marathon
39Disallow: /flndrmk-spl
40Disallow: /system/
41Disallow: /sachin
42Disallow: /supermom2016
43Disallow: /ywq2015
44Disallow: /fooddirectory
45Disallow: /chennaicoastalcleanup
46Disallow: /onam-1
47Disallow: /onam-2
48Disallow: /hohkidscarnival
49Disallow: /HOW2014
50Disallow: /how2014
51Disallow: /cookingcontest
52Disallow: /youngworld
53Disallow: /impulse2014
54Disallow: /mssaward2015
55Disallow: /mssaward2016
56Disallow: /wow2015
57Disallow: /wow2017/chennai/
58Disallow: /wow2017/bangalore/
59Disallow: /tickets2016/
60Disallow: /tickets2017/
61Disallow: /tickets2018/
62Disallow: /ticket2018/
63Disallow: /ywsummercamp/
64Disallow: /tags/TH category/
65Disallow: /tags/TopicRoot_TH/
66Disallow: /thehindu/
67Disallow: /200*
68Disallow: /201*
69Disallow: */http:*
70Disallow: */https:*
71Disallow: */mailto:*
72Disallow: *.ecehttp*
73Disallow: *.ece1http*
74Disallow: *.ece2http*
75Disallow: /brandhub/sponsored-content/
76Disallow: /coupons/
77Disallow: *?homepage*
78Disallow: *?service*
79Disallow: *?type*
80Disallow: */1000*
81Disallow: */edge/widgets/*
82
83# Blocking ptid and gl parameter URLs that are causing crawl budget wastage
84#Disallow: /*?*_ptid=*
85Disallow: /*?*_gl=*
86# Block picture sitemap because urls are noindex therein
87Disallow: /sitemap/archive/picture/
88# Block all space or %20 suffixed with http or https protocols
89Disallow: * http:*
90Disallow: * https:*
91Disallow: *;http:*
92Disallow: *;https:*
93Disallow: *%20http:*
94Disallow: *%20https:*
95Disallow: */couponRedirect
96Disallow: *?redirect=
97Disallow: *?store=
98Disallow: /tag/*/*/
99# Blocked until duplicate profile bylines fixed
100#Disallow: /profile/
101# profile articles allowed
102Allow: /profile/*article*.ece
103# profile exceptions allowed for pages with page=1 for redirects
104Allow: /profile/contributor/?
105Allow: /profile/photographers/?
106Allow: /profile/author/?
107Allow: /profile/contributor/$
108Allow: /profile/photographers/$
109Allow: /profile/author/$
110Allow: /profile/$
111Disallow: /sitemap/archive*
112Disallow: */fragment/*
113Disallow: /sitemap/
114Allow: /sitemap/$
115Allow: /sitemap/update/all.xml$
116Allow: /sitemap/update/section.xml$
117Allow: /sitemap/liveblog.xml$
118Allow: /sitemap/googlenews/all/all.xml$
119Disallow: /*click_url*
120Disallow: /*theme/images/th-online/th-circle-white.svg
121#Blocking pagination
122#Disallow: /*?page=*
123#Disallow URLs ending with hyphen
124Disallow: /*-$
125Disallow: *//*
126Disallow: */fontawesome-*
127Disallow: */edge/widgets/*
128
129
130#Disallow Wayback machine which is a problem crawler
131User-Agent: ia_archiver
132Disallow: /
133
134
135
136
137# Blocking AI crawlers
138User-agent: GPTBot
139User-agent: ChatGPT-User
140User-agent: OAI-SearchBot
141User-agent: PerplexityBot
142User-agent: Amazonbot
143User-agent: Bytespider
144User-agent: ClaudeBot
145User-agent: Claude-Web
146User-agent: Meta-ExternalAgent
147User-agent: FacebookBot
148User-agent: CCBot
149User-agent: Anthropic-ai
150User-agent: YouBot
151Disallow: /
152
153
154
155
156# Sitemaps
157Sitemap: https://www.thehindu.com/sitemap/googlenews/all/all.xml
158Sitemap: https://www.thehindu.com/sitemap/update/all.xml
159Sitemap: https://www.thehindu.com/sitemap/update/section.xml
160Sitemap: https://www.thehindu.com/feeder/default.rss
161Sitemap: https://www.thehindu.com/?service=videositemap
162Sitemap: https://www.thehindu.com/entertainment/feeder/default.rss
163Sitemap: https://www.thehindu.com/shorts/feeder/default.rss
164Sitemap: https://www.thehindu.com/sport/feeder/default.rss
165Sitemap: https://www.thehindu.com/business/feeder/default.rss
166
What do the bots actually get?
This page reads robots.txt, which states what thehindu.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on thehindu.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does thehindu.com block GPTBot?
Yes. thehindu.com’s robots.txt disallows GPTBot from the whole site (line 151: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does thehindu.com block ClaudeBot?
Yes. thehindu.com’s robots.txt disallows ClaudeBot from the whole site (line 151: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does thehindu.com block Google-Extended?
No. thehindu.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does thehindu.com block PerplexityBot?
Yes. thehindu.com’s robots.txt disallows PerplexityBot from the whole site (line 151: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find thehindu.com?
Not through its own crawler. thehindu.com’s robots.txt disallows OAI-SearchBot from the whole site (line 151: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.