Who blocks AI · #308 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does indiatimes.com block AI crawlers?

Yes. indiatimes.com’s robots.txt blocks 1 of the 19 AI crawlers we track, including Meta-ExternalAgent. Search engines such as Googlebot and Bingbot may crawl it.

GPTBotAllowed OAI-SearchBotAllowed ClaudeBotAllowed PerplexityBotAllowed Google-ExtendedAllowed CCBotAllowed

robots.txt: www.indiatimes.com/robots.txt · 59 lines
llms.txt: none
Checked: 9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

Bot	Operator	robots.txt	Deciding line
`Googlebot`	Google	Allowed	line 13 · `Allow: /` under `User-agent: *`
`Bingbot`	Microsoft	Allowed	line 13 · `Allow: /` under `User-agent: *`
`DuckDuckBot`	DuckDuckGo	Allowed	line 13 · `Allow: /` under `User-agent: *`
`Applebot`	Apple	Allowed	line 13 · `Allow: /` under `User-agent: *`
`PetalBot`	Huawei	Allowed	line 13 · `Allow: /` under `User-agent: *`
`YandexBot`	Yandex	Allowed	line 13 · `Allow: /` under `User-agent: *`
`Baiduspider`	Baidu	Allowed	line 13 · `Allow: /` under `User-agent: *`

AI search

Bot	Operator	robots.txt	Deciding line
`OAI-SearchBot`	OpenAI	Allowed	line 13 · `Allow: /` under `User-agent: *`
`PerplexityBot`	Perplexity	Allowed	line 13 · `Allow: /` under `User-agent: *`
`Claude-SearchBot`	Anthropic	Allowed	line 13 · `Allow: /` under `User-agent: *`
`DuckAssistBot`	DuckDuckGo	Allowed	line 13 · `Allow: /` under `User-agent: *`
`Amazonbot`	Amazon	Allowed	line 13 · `Allow: /` under `User-agent: *`
`YouBot`	You.com	Allowed	line 13 · `Allow: /` under `User-agent: *`

AI training

Bot	Operator	robots.txt	Deciding line
`GPTBot`	OpenAI	Allowed	line 13 · `Allow: /` under `User-agent: *`
`ClaudeBot`	Anthropic	Allowed	line 13 · `Allow: /` under `User-agent: *`
`Google-Extended`	Google	Allowed	line 13 · `Allow: /` under `User-agent: *`
`CCBot`	Common Crawl	Allowed	line 13 · `Allow: /` under `User-agent: *`
`Applebot-Extended`	Apple	Allowed	line 13 · `Allow: /` under `User-agent: *`
`GoogleOther`	Google	Allowed	line 13 · `Allow: /` under `User-agent: *`
`Meta-ExternalAgent`	Meta	Blocked	line 10 · `Disallow: /` under `User-agent: meta-externalagent`
`Bytespider`	ByteDance	Allowed	line 13 · `Allow: /` under `User-agent: *`

User-triggered fetchers

Bot	Operator	robots.txt	Deciding line
`ChatGPT-User`	OpenAI	Allowed	line 13 · `Allow: /` under `User-agent: *`
`Claude-User`	Anthropic	Allowed	line 13 · `Allow: /` under `User-agent: *`
`Perplexity-User`	Perplexity	Allowed	line 13 · `Allow: /` under `User-agent: *`
`MistralAI-User`	Mistral	Allowed	line 13 · `Allow: /` under `User-agent: *`
`Meta-ExternalFetcher`	Meta	Allowed	line 13 · `Allow: /` under `User-agent: *`

SEO tools

Bot	Operator	robots.txt	Deciding line
`AhrefsBot`	Ahrefs	Allowed	line 13 · `Allow: /` under `User-agent: *`
`SemrushBot`	Semrush	Allowed	line 13 · `Allow: /` under `User-agent: *`
`MJ12bot`	Majestic	Allowed	line 13 · `Allow: /` under `User-agent: *`
`DotBot`	Moz	Allowed	line 13 · `Allow: /` under `User-agent: *`

View www.indiatimes.com/robots.txt · 59 lines

1#robots.txt PS

4User-agent: AmazonAdBot

5Allow: /

6Allow: /*

9User-agent: meta-externalagent

10Disallow: /

12User-agent: *

13Allow: /

14Disallow: *.php*

15Disallow: */7176/*

16Disallow: */whatshot/*

17Disallow: */wattpad/*

18Disallow: */uc_feed/*

19Disallow: */brief/*

20Disallow: */search/*

21Disallow: */user/*

22Disallow: */idtoin/*

23Disallow: */video_player/*

24Disallow: */blog/*

25Disallow: */content/*

26Disallow: */seoarchive/*

27Disallow: */trends/audios/*

28Disallow: */trends/videos/*

29Disallow: *{*

30Disallow: *?ref=*

31Disallow: *?v=*

32Disallow: *?no_fullscreen_*

33Disallow: *?symbol=*

34Disallow: */?date=*

35Disallow: *?fb_comment_id=*

36Disallow: *?_twitter*

37Disallow: */2011/*

38Disallow: */2012/*

39Disallow: */2013/*

40Disallow: */2014/*

41Disallow: */2015/*

42Disallow: */2016/*

43Disallow: */2017/*

44Disallow: */2018/*

45Disallow: */2019/*

46Disallow: */2020/*

47Disallow: */2021/*

48Disallow: */2022/*

49Disallow: */2023/*

50Disallow: */2024/*

51Disallow: *.ttf*

52Disallow: *.woff*

53Disallow: *.png*

54Disallow: *.jpg*

55Disallow: /partner

57#Sitemaps

58Sitemap:https://www.indiatimes.com/staticsitemap/id-454/sitemap-index.xml

59Sitemap:https://www.indiatimes.com/staticsitemap/id-454/sitemap-48hours.xml

What do the bots actually get?

This page reads robots.txt, which states what indiatimes.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on indiatimes.com Test a path in the robots.txt tester

History

19 October 2026: first checked.

Questions

Does indiatimes.com block GPTBot?

No. indiatimes.com’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does indiatimes.com block ClaudeBot?

No. indiatimes.com’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does indiatimes.com block Google-Extended?

No. indiatimes.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does indiatimes.com block PerplexityBot?

No. indiatimes.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find indiatimes.com?

Yes. indiatimes.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.