Who blocks AI · #800 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does dhs.gov block AI crawlers?

No. dhs.gov’s robots.txt lets all 19 AI crawlers we track crawl the site. It also blocks PetalBot among search engines.

GPTBotAllowed OAI-SearchBotAllowed ClaudeBotAllowed PerplexityBotAllowed Google-ExtendedAllowed CCBotAllowed

robots.txt: www.dhs.gov/robots.txt · 88 lines
llms.txt: none
Checked: 9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

Bot	Operator	robots.txt	Deciding line
`Googlebot`	Google	Allowed	no rule matches the home page
`Bingbot`	Microsoft	Allowed	no rule matches the home page
`DuckDuckBot`	DuckDuckGo	Allowed	no rule matches the home page
`Applebot`	Apple	Allowed	no rule matches the home page
`PetalBot`	Huawei	Blocked	line 84 · `Disallow: /` under `User-agent: Petalbot`
`YandexBot`	Yandex	Allowed	no rule matches the home page
`Baiduspider`	Baidu	Allowed	no rule matches the home page

AI search

Bot	Operator	robots.txt	Deciding line
`OAI-SearchBot`	OpenAI	Allowed	no rule matches the home page
`PerplexityBot`	Perplexity	Allowed	no rule matches the home page
`Claude-SearchBot`	Anthropic	Allowed	no rule matches the home page
`DuckAssistBot`	DuckDuckGo	Allowed	no rule matches the home page
`Amazonbot`	Amazon	Allowed	no rule matches the home page
`YouBot`	You.com	Allowed	no rule matches the home page

AI training

Bot	Operator	robots.txt	Deciding line
`GPTBot`	OpenAI	Allowed	no rule matches the home page
`ClaudeBot`	Anthropic	Allowed	no rule matches the home page
`Google-Extended`	Google	Allowed	no rule matches the home page
`CCBot`	Common Crawl	Allowed	no rule matches the home page
`Applebot-Extended`	Apple	Allowed	no rule matches the home page
`GoogleOther`	Google	Allowed	no rule matches the home page
`Meta-ExternalAgent`	Meta	Allowed	no rule matches the home page
`Bytespider`	ByteDance	Allowed	no rule matches the home page

User-triggered fetchers

Bot	Operator	robots.txt	Deciding line
`ChatGPT-User`	OpenAI	Allowed	no rule matches the home page
`Claude-User`	Anthropic	Allowed	no rule matches the home page
`Perplexity-User`	Perplexity	Allowed	no rule matches the home page
`MistralAI-User`	Mistral	Allowed	no rule matches the home page
`Meta-ExternalFetcher`	Meta	Allowed	no rule matches the home page

SEO tools

Bot	Operator	robots.txt	Deciding line
`AhrefsBot`	Ahrefs	Allowed	no rule matches the home page
`SemrushBot`	Semrush	Allowed	no rule matches the home page
`MJ12bot`	Majestic	Allowed	no rule matches the home page
`DotBot`	Moz	Allowed	no rule matches the home page

View www.dhs.gov/robots.txt · 88 lines

2# robots.txt

4# This file is to prevent the crawling and indexing of certain parts

5# of your site by web crawlers and spiders run by sites like Yahoo!

6# and Google. By telling these "robots" where not to go on your site,

7# you save bandwidth and server resources.

9# This file will be ignored unless it is at the root of your host:

10# Used: http://example.com/robots.txt

11# Ignored: http://example.com/site/robots.txt

12#

13# For more information about the robots.txt standard, see:

14# http://www.robotstxt.org/robotstxt.html

16User-agent: *

17# CSS, JS, Images

18Allow: /core/*.css$

19Allow: /core/*.css?

20Allow: /core/*.js$

21Allow: /core/*.js?

22Allow: /core/*.avif

23Allow: /core/*.gif

24Allow: /core/*.jpg

25Allow: /core/*.jpeg

26Allow: /core/*.png

27Allow: /core/*.svg

28Allow: /core/*.webp

29Allow: /profiles/*.css$

30Allow: /profiles/*.css?

31Allow: /profiles/*.js$

32Allow: /profiles/*.js?

33Allow: /profiles/*.avif

34Allow: /profiles/*.gif

35Allow: /profiles/*.jpg

36Allow: /profiles/*.jpeg

37Allow: /profiles/*.png

38Allow: /profiles/*.svg

39Allow: /profiles/*.webp

40# Directories

41Disallow: /core/

42Disallow: /profiles/

43# Files

44Disallow: /README.md

45Disallow: /composer/Metapackage/README.txt

46Disallow: /composer/Plugin/ProjectMessage/README.md

47Disallow: /composer/Plugin/Scaffold/README.md

48Disallow: /composer/Plugin/VendorHardening/README.txt

49Disallow: /composer/Template/README.txt

50Disallow: /modules/README.txt

51Disallow: /sites/README.txt

52Disallow: /themes/README.txt

53# Paths (clean URLs)

54Disallow: /admin/

55Disallow: /comment/reply/

56Disallow: /filter/tips

57Disallow: /node/add/

58Disallow: /search/

59Disallow: /search?

60Disallow: /user/register

61Disallow: /user/password

62Disallow: /user/login

63Disallow: /user/logout

64Disallow: /media/oembed

65Disallow: /*/media/oembed

66# Paths (no clean URLs)

67Disallow: /index.php/admin/

68Disallow: /index.php/comment/reply/

69Disallow: /index.php/filter/tips

70Disallow: /index.php/node/add/

71Disallow: /index.php/search/

72Disallow: /index.php/search?

73Disallow: /index.php/user/password

74Disallow: /index.php/user/register

75Disallow: /index.php/user/login

76Disallow: /index.php/user/logout

77Disallow: /index.php/media/oembed

78Disallow: /index.php/*/media/oembed

81# dhs.gov customizations

82Sitemap: https://www.dhs.gov/sitemap.xml

83User-agent: Petalbot

84Disallow: /

86User-agent: *

87Disallow: /archive/

What do the bots actually get?

This page reads robots.txt, which states what dhs.gov asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on dhs.gov Test a path in the robots.txt tester

History

19 October 2026: first checked.

Questions

Does dhs.gov block GPTBot?

No. dhs.gov’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does dhs.gov block ClaudeBot?

No. dhs.gov’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does dhs.gov block Google-Extended?

No. dhs.gov’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does dhs.gov block PerplexityBot?

No. dhs.gov’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find dhs.gov?

Yes. dhs.gov’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.