Who blocks AI · #68 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does flickr.com block AI crawlers?

Yes. flickr.com’s robots.txt blocks 14 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and 11 more. It also blocks YandexBot among search engines.

GPTBotBlocked OAI-SearchBotPartly blocked ClaudeBotBlocked PerplexityBotPartly blocked Google-ExtendedBlocked CCBotBlocked

robots.txt: flickr.com/robots.txt · 77 lines
llms.txt: none
Checked: 9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

Bot	Operator	robots.txt	Deciding line
`Googlebot`	Google	Partly blocked	line 46 · `Disallow: /gp/` line 47 · `Disallow: /report_abuse.gne` and 17 more
`Bingbot`	Microsoft	Partly blocked	line 46 · `Disallow: /gp/` line 47 · `Disallow: /report_abuse.gne` and 17 more
`DuckDuckBot`	DuckDuckGo	Partly blocked	line 46 · `Disallow: /gp/` line 47 · `Disallow: /report_abuse.gne` and 17 more
`Applebot`	Apple	Partly blocked	line 46 · `Disallow: /gp/` line 47 · `Disallow: /report_abuse.gne` and 17 more
`PetalBot`	Huawei	Partly blocked	line 46 · `Disallow: /gp/` line 47 · `Disallow: /report_abuse.gne` and 17 more
`YandexBot`	Yandex	Blocked	line 70 · `Disallow: /` under `User-agent: *`
`Baiduspider`	Baidu	Partly blocked	line 46 · `Disallow: /gp/` line 47 · `Disallow: /report_abuse.gne` and 17 more

AI search

Bot	Operator	robots.txt	Deciding line
`OAI-SearchBot`	OpenAI	Partly blocked	line 46 · `Disallow: /gp/` line 47 · `Disallow: /report_abuse.gne` and 17 more
`PerplexityBot`	Perplexity	Partly blocked	line 46 · `Disallow: /gp/` line 47 · `Disallow: /report_abuse.gne` and 17 more
`Claude-SearchBot`	Anthropic	Partly blocked	line 46 · `Disallow: /gp/` line 47 · `Disallow: /report_abuse.gne` and 17 more
`DuckAssistBot`	DuckDuckGo	Blocked	line 70 · `Disallow: /` under `User-agent: *`
`Amazonbot`	Amazon	Blocked	line 70 · `Disallow: /` under `User-agent: *`
`YouBot`	You.com	Blocked	line 70 · `Disallow: /` under `User-agent: *`

AI training

Bot	Operator	robots.txt	Deciding line
`GPTBot`	OpenAI	Blocked	line 70 · `Disallow: /` under `User-agent: *`
`ClaudeBot`	Anthropic	Blocked	line 70 · `Disallow: /` under `User-agent: *`
`Google-Extended`	Google	Blocked	line 70 · `Disallow: /` under `User-agent: *`
`CCBot`	Common Crawl	Blocked	line 70 · `Disallow: /` under `User-agent: *`
`Applebot-Extended`	Apple	Blocked	line 70 · `Disallow: /` under `User-agent: *`
`GoogleOther`	Google	Blocked	line 70 · `Disallow: /` under `User-agent: *`
`Meta-ExternalAgent`	Meta	Blocked	line 70 · `Disallow: /` under `User-agent: *`
`Bytespider`	ByteDance	Blocked	line 70 · `Disallow: /` under `User-agent: *`

User-triggered fetchers

Bot	Operator	robots.txt	Deciding line
`ChatGPT-User`	OpenAI	Partly blocked	line 46 · `Disallow: /gp/` line 47 · `Disallow: /report_abuse.gne` and 17 more
`Claude-User`	Anthropic	Partly blocked	line 46 · `Disallow: /gp/` line 47 · `Disallow: /report_abuse.gne` and 17 more
`Perplexity-User`	Perplexity	Blocked	line 70 · `Disallow: /` under `User-agent: *`
`MistralAI-User`	Mistral	Blocked	line 70 · `Disallow: /` under `User-agent: *`
`Meta-ExternalFetcher`	Meta	Blocked	line 70 · `Disallow: /` under `User-agent: *`

SEO tools

Bot	Operator	robots.txt	Deciding line
`AhrefsBot`	Ahrefs	Blocked	line 70 · `Disallow: /` under `User-agent: *`
`SemrushBot`	Semrush	Blocked	line 70 · `Disallow: /` under `User-agent: *`
`MJ12bot`	Majestic	Blocked	line 70 · `Disallow: /` under `User-agent: *`
`DotBot`	Moz	Blocked	line 70 · `Disallow: /` under `User-agent: *`

View flickr.com/robots.txt · 77 lines

1User-agent: Adsbot-Google

2User-agent: Alexabot

3User-agent: AppleBot

4User-agent: archive.org_bot

5User-agent: Assaybot

6User-agent: baiduspider

7User-agent: bitlybot

8User-agent: bingbot

9User-agent: BingPreview

10User-agent: ChatGPT-User

11User-agent: Claude-SearchBot

12User-agent: Claude-User

13User-agent: coccocbot

14User-agent: Discordbot

15User-agent: DuckDuckBot

16User-agent: facebookexternalhit

17User-agent: Google-Site-Verification

18User-agent: Google-Sitemaps

19User-agent: Googlebot

20User-agent: Googlebot-Image

21User-agent: Googlebot-Mobile

22User-agent: Googlebot-News

23User-agent: Googlebot-Video

24User-agent: gsa-crawler

25User-Agent: HatenaBlog-bot

26User-agent: ia_archiver

27User-agent: Mediapartners-Google

28User-agent: msnbot

29User-agent: NAVER

30User-agent: OAI-SearchBot

31User-agent: PerplexityBot

32User-agent: PetalBot

33User-agent: Pingdom

34User-agent: Pinterest

35User-agent: redditbot

36User-agent: Slackbot

37User-agent: Slurp

38User-agent: snapchat

39User-agent: Twitterbot

40User-agent: wp.com

41User-agent: Yandex

42User-agent: YandexImages

43User-agent: YandexVideoParser

44User-agent: Yeti

45User-agent: ZoomBot

46Disallow: /gp/

47Disallow: /report_abuse.gne

48Disallow: /abuse

49Disallow: /images/*

50Disallow: /apps/*

51Disallow: /tools/demos/*

52Disallow: /search

53Disallow: /services/oauth

54Disallow: /groups/10millionphotos/

55Disallow: /photos/youpy/

56Disallow: /photos/i_love_u_get_away_from_me/

57Disallow: /faves-i_love_u_get_away_from_me/

58Disallow: /photos/gbachelie/

59Disallow: /photos/archivesact/6011019532/nearby/

60Disallow: /yss_fragment.gne

61Disallow: /photos/tags/*/page*

62Disallow: /photos/*/lightbox/

63Disallow: /groups/*/lightbox/

64Disallow: /explore/*/lightbox/

66User-agent: magpie-crawler

67Disallow: /

69User-agent: *

70Disallow: /

72Sitemap: https://www.flickr.com/sitemap/index/users/sitemap-index-users-00000000.xml.gz

73Sitemap: https://www.flickr.com/sitemap/index/tags/sitemap-index-tags-00000000.xml.gz

74Sitemap: https://www.flickr.com/sitemap/index/sets/sitemap-index-sets-00000000.xml.gz

75Sitemap: https://www.flickr.com/sitemap/index/photos/sitemap-index-photos-00000000.xml.gz

76Sitemap: https://www.flickr.com/sitemap/index/groups/sitemap-index-groups-00000000.xml.gz

77Sitemap: https://www.flickr.com/sitemap/cameras/sitemap-cameras-00000000.xml.gz

What do the bots actually get?

This page reads robots.txt, which states what flickr.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on flickr.com Test a path in the robots.txt tester

History

19 October 2026: first checked.

Questions

Does flickr.com block GPTBot?

Yes. flickr.com’s robots.txt disallows GPTBot from the whole site (line 70: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does flickr.com block ClaudeBot?

Yes. flickr.com’s robots.txt disallows ClaudeBot from the whole site (line 70: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does flickr.com block Google-Extended?

Yes. flickr.com’s robots.txt disallows Google-Extended from the whole site (line 70: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does flickr.com block PerplexityBot?

Partly. flickr.com’s robots.txt keeps PerplexityBot out of some paths, such as /gp/, and lets it crawl the rest. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find flickr.com?

Partly. flickr.com’s robots.txt keeps OAI-SearchBot out of some paths, such as /gp/, and lets it crawl the rest. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.