Who blocks AI · #1161 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does smugmug.com block AI crawlers?

Yes. smugmug.com’s robots.txt blocks all 19 AI crawlers we track, from GPTBot and ClaudeBot to PerplexityBot. It also blocks DuckDuckBot, PetalBot and YandexBot among search engines.

GPTBotBlocked OAI-SearchBotBlocked ClaudeBotBlocked PerplexityBotBlocked Google-ExtendedBlocked CCBotBlocked

robots.txt: www.smugmug.com/robots.txt · 78 lines
llms.txt: none
Checked: 9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

Bot	Operator	robots.txt	Deciding line
`Googlebot`	Google	Partly blocked	line 39 · `Disallow: /404` line 40 · `Disallow: /access-denied` and 17 more
`Bingbot`	Microsoft	Partly blocked	line 39 · `Disallow: /404` line 40 · `Disallow: /access-denied` and 17 more
`DuckDuckBot`	DuckDuckGo	Blocked	line 77 · `Disallow: /` under `User-agent: *`
`Applebot`	Apple	Partly blocked	line 39 · `Disallow: /404` line 40 · `Disallow: /access-denied` and 17 more
`PetalBot`	Huawei	Blocked	line 77 · `Disallow: /` under `User-agent: *`
`YandexBot`	Yandex	Blocked	line 77 · `Disallow: /` under `User-agent: *`
`Baiduspider`	Baidu	Partly blocked	line 39 · `Disallow: /404` line 40 · `Disallow: /access-denied` and 17 more

AI search

Bot	Operator	robots.txt	Deciding line
`OAI-SearchBot`	OpenAI	Blocked	line 77 · `Disallow: /` under `User-agent: *`
`PerplexityBot`	Perplexity	Blocked	line 77 · `Disallow: /` under `User-agent: *`
`Claude-SearchBot`	Anthropic	Blocked	line 77 · `Disallow: /` under `User-agent: *`
`DuckAssistBot`	DuckDuckGo	Blocked	line 77 · `Disallow: /` under `User-agent: *`
`Amazonbot`	Amazon	Blocked	line 77 · `Disallow: /` under `User-agent: *`
`YouBot`	You.com	Blocked	line 77 · `Disallow: /` under `User-agent: *`

AI training

Bot	Operator	robots.txt	Deciding line
`GPTBot`	OpenAI	Blocked	line 77 · `Disallow: /` under `User-agent: *`
`ClaudeBot`	Anthropic	Blocked	line 77 · `Disallow: /` under `User-agent: *`
`Google-Extended`	Google	Blocked	line 77 · `Disallow: /` under `User-agent: *`
`CCBot`	Common Crawl	Blocked	line 77 · `Disallow: /` under `User-agent: *`
`Applebot-Extended`	Apple	Blocked	line 77 · `Disallow: /` under `User-agent: *`
`GoogleOther`	Google	Blocked	line 77 · `Disallow: /` under `User-agent: *`
`Meta-ExternalAgent`	Meta	Blocked	line 77 · `Disallow: /` under `User-agent: *`
`Bytespider`	ByteDance	Blocked	line 77 · `Disallow: /` under `User-agent: *`

User-triggered fetchers

Bot	Operator	robots.txt	Deciding line
`ChatGPT-User`	OpenAI	Blocked	line 77 · `Disallow: /` under `User-agent: *`
`Claude-User`	Anthropic	Blocked	line 77 · `Disallow: /` under `User-agent: *`
`Perplexity-User`	Perplexity	Blocked	line 77 · `Disallow: /` under `User-agent: *`
`MistralAI-User`	Mistral	Blocked	line 77 · `Disallow: /` under `User-agent: *`
`Meta-ExternalFetcher`	Meta	Blocked	line 77 · `Disallow: /` under `User-agent: *`

SEO tools

Bot	Operator	robots.txt	Deciding line
`AhrefsBot`	Ahrefs	Blocked	line 77 · `Disallow: /` under `User-agent: *`
`SemrushBot`	Semrush	Blocked	line 77 · `Disallow: /` under `User-agent: *`
`MJ12bot`	Majestic	Blocked	line 77 · `Disallow: /` under `User-agent: *`
`DotBot`	Moz	Blocked	line 77 · `Disallow: /` under `User-agent: *`

View www.smugmug.com/robots.txt · 78 lines

1# See https://secure.smugmug.com/help/contact if you'd like to apply to be allowlisted for crawling SmugMug.

3user-agent: Adsbot-Google

4user-agent: Alexabot

5user-agent: BingPreview

6user-agent: CloudflarePrefetch

7user-agent: FriendFeedBot

8user-agent: FunnelBack

9user-agent: Google

10user-agent: Google Favicon

11user-agent: Google-Site-Verification

12user-agent: Google-Sitemaps

13user-agent: Googlebot

14user-agent: Googlebot-Image

15user-agent: Googlebot-Mobile

16user-agent: Googlebot-News

17user-agent: Googlebot-Video

18user-agent: Mediapartners-Google

19user-agent: Pingdom

20user-agent: Pinterest

21user-agent: ScoutJet

22user-agent: Slurp

23user-agent: Spinn3r

24user-agent: Teoma

25user-agent: Twitterbot

26user-agent: Yandex

27user-agent: YandexImages

28user-agent: YandexVideoParser

29user-agent: Yeti

30user-agent: archive.org_bot

31user-agent: baiduspider

32user-agent: bingbot

33user-agent: facebookexternalhit

34user-agent: gsa-crawler

35user-agent: houzzbot

36user-agent: ia_archiver

37user-agent: msnbot

38user-agent: rogerbot

39disallow: /404

40disallow: /access-denied

41disallow: /admin

42disallow: /api

43disallow: /cart

44disallow: /checkout

45disallow: /date

46disallow: /downloads

47disallow: /go

48disallow: /hack

49disallow: /keyword

50disallow: /order

51disallow: /password

52disallow: /popular

53disallow: /search

54disallow: /services/api/php

55disallow: /services/api/rest

56disallow: /services/api/xmlrpc

57disallow: /test

58allow: /api/developer

59allow: /api/doc

60allow: /api/v2

62user-agent: Google

63user-agent: Twitterbot

64user-agent: facebookexternalhit

65user-agent: houzzbot

66allow: /date

67allow: /keyword

68allow: /popular

70user-agent: zombies

71disallow: /brains

73user-agent: werewolves

74disallow: /villagers

76user-agent: *

77disallow: /

What do the bots actually get?

This page reads robots.txt, which states what smugmug.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on smugmug.com Test a path in the robots.txt tester

History

19 October 2026: first checked.

Questions

Does smugmug.com block GPTBot?

Yes. smugmug.com’s robots.txt disallows GPTBot from the whole site (line 77: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does smugmug.com block ClaudeBot?

Yes. smugmug.com’s robots.txt disallows ClaudeBot from the whole site (line 77: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does smugmug.com block Google-Extended?

Yes. smugmug.com’s robots.txt disallows Google-Extended from the whole site (line 77: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does smugmug.com block PerplexityBot?

Yes. smugmug.com’s robots.txt disallows PerplexityBot from the whole site (line 77: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find smugmug.com?

Not through its own crawler. smugmug.com’s robots.txt disallows OAI-SearchBot from the whole site (line 77: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.