Who blocks AI · #928 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does news.com.au block AI crawlers?
Yes. news.com.au’s robots.txt blocks 18 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and 15 more. It also blocks PetalBot, YandexBot and Baiduspider among search engines.
GPTBotBlockedOAI-SearchBotBlockedClaudeBotBlockedPerplexityBotBlockedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- www.news.com.au/robots.txt · 131 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Partly blocked | line 76 · Disallow: /*?expand_comments=true line 77 · Disallow: /*&expand_comments=true and 33 more | |
Bingbot | Microsoft | Partly blocked | line 76 · Disallow: /*?expand_comments=true line 77 · Disallow: /*&expand_comments=true and 33 more |
DuckDuckBot | DuckDuckGo | Partly blocked | line 76 · Disallow: /*?expand_comments=true line 77 · Disallow: /*&expand_comments=true and 33 more |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Blocked | line 120 · Disallow: / under User-agent: * |
YandexBot | Yandex | Blocked | line 120 · Disallow: / under User-agent: * |
Baiduspider | Baidu | Blocked | line 120 · Disallow: / under User-agent: * |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Blocked | line 120 · Disallow: / under User-agent: * |
PerplexityBot | Perplexity | Blocked | line 120 · Disallow: / under User-agent: * |
Claude-SearchBot | Anthropic | Blocked | line 120 · Disallow: / under User-agent: * |
DuckAssistBot | DuckDuckGo | Blocked | line 120 · Disallow: / under User-agent: * |
Amazonbot | Amazon | Blocked | line 120 · Disallow: / under User-agent: * |
YouBot | You.com | Blocked | line 120 · Disallow: / under User-agent: * |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 120 · Disallow: / under User-agent: * |
ClaudeBot | Anthropic | Blocked | line 120 · Disallow: / under User-agent: * |
Google-Extended | Blocked | line 113 · Disallow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Blocked | line 120 · Disallow: / under User-agent: * |
Applebot-Extended | Apple | Blocked | line 120 · Disallow: / under User-agent: * |
GoogleOther | Partly blocked | line 76 · Disallow: /*?expand_comments=true line 77 · Disallow: /*&expand_comments=true and 33 more | |
Meta-ExternalAgent | Meta | Blocked | line 120 · Disallow: / under User-agent: * |
Bytespider | ByteDance | Blocked | line 120 · Disallow: / under User-agent: * |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Blocked | line 120 · Disallow: / under User-agent: * |
Claude-User | Anthropic | Blocked | line 120 · Disallow: / under User-agent: * |
Perplexity-User | Perplexity | Blocked | line 120 · Disallow: / under User-agent: * |
MistralAI-User | Mistral | Blocked | line 120 · Disallow: / under User-agent: * |
Meta-ExternalFetcher | Meta | Blocked | line 120 · Disallow: / under User-agent: * |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Blocked | line 120 · Disallow: / under User-agent: * |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Blocked | line 120 · Disallow: / under User-agent: * |
DotBot | Moz | Blocked | line 120 · Disallow: / under User-agent: * |
View www.news.com.au/robots.txt · 131 lines
1# NOTICE: Collection of content and other data on www.news.com.au through automated means is prohibited unless you have express written permission from the publisher of the website and may only be conducted for the limited purpose contained in said permission.
2
3# Website Terms of Use may be found at https://www.news.com.au/terms-and-conditions
4
5#Agent Specific Allowed Section
6# ==========================
7User-agent: ScreamingFrogSEOSpider
8User-agent: proximic
9User-agent: Outbrain
10User-agent: Yahoo Ad monitoring
11User-agent: SemrushBot
12User-agent: facebookexternalhit
13User-agent: ias_crawler
14User-agent: identrics
15User-agent: MeltwaterNews
16User-agent: links.streem.com.au
17User-agent: LinkedInBot
18User-agent: redditbot
19User-agent: Mediapartners-Google
20User-agent: Google-InspectionTool
21User-agent: Adsbot-Google
22User-agent: Storebot-Google
23User-agent: AdsBot-Google-Mobile
24User-agent: AdsBot-Google-Mobile-Apps
25User-agent: APIs-Google
26User-agent: Google-Ads-Creatives-Assistant
27User-agent: Google-adwords
28User-agent: Google-amphtml
29User-agent: Google-cloudvertexbot
30User-agent: Google-play-newsstand
31User-agent: Google-read-aloud
32User-agent: Google-Safety
33User-agent: Google-site-verification
34User-agent: Google-Structured-Data-Testing-Tool
35User-agent: Googleimageproxy
36User-agent: Applebot
37User-agent: AppleNewsBot
38User-agent: adidxbot
39User-agent: admantx
40User-agent: admantx-usbatch/3.1
41User-agent: AmazonAdBot
42User-agent: Chrome-Lighthouse
43User-agent: Verity
44User-agent: Leikibot
45User-agent: parselybot
46User-agent: peer39_crawler/1.0
47User-agent: pingdom
48User-agent: Pinterestbot
49User-agent: proximic
50User-agent: PubMatic Crawler Bot
51User-agent: Mobian
52User-agent: Slack-ImgProxy
53User-agent: Slackbot
54User-agent: Slackbot-LinkExpanding
55User-agent: thetradedesk
56User-agent: Uptimebot
57User-agent: yahoo-linkpreview
58Disallow:
59
60# Web Search Engines
61==========================
62User-agent: Googlebot
63User-agent: Googlebot-Image
64User-agent: Googlebot-News
65User-agent: Googlebot-Video
66User-agent: GoogleOther
67User-agent: GoogleOther-Image
68User-agent: GoogleOther-Video
69User-agent: Bingbot
70User-agent: bingpreview
71User-agent: MicrosoftPreview
72User-agent: DuckDuckBot
73User-agent: Slurp
74User-agent: yahoo
75Disallow:
76Disallow: /*?expand_comments=true
77Disallow: /*&expand_comments=true
78Disallow: /*/comments-*
79Disallow: /*/print/
80Disallow: /*SIT*
81Disallow: /*.swf
82Disallow: /printpage/
83Disallow: /double-rainbow
84Disallow: /double-rainbow/
85Disallow: /*-simplify/*
86Disallow: /it-test-only/
87Disallow: /enewsletters/
88Disallow: /search?q=*
89Disallow: */rss$
90Disallow: */rss2/$
91Disallow: */feed$
92Disallow: /*/coupons/visit/*
93Disallow: /app-route/
94Disallow: /app/
95Disallow: /finance/business/stockhead/news/*/news-story/*
96Disallow: /akam/
97Disallow: /*?utm_medium=
98Disallow: /*?utm_campaign=
99Disallow: /*?arena_mid=
100Disallow: /*/akam/
101Disallow: /*?from=
102Disallow: /*?campaignType=
103Disallow: /*?campaignSource=
104Disallow: /*?s=
105Disallow: /*?pg=
106Disallow: /*?dRegion=
107Disallow: /*?sv=
108Disallow: /*?gclid=
109Disallow: /search-results?q=
110Disallow: /checkout/search-results?q=
111
112User-agent: Google-Extended
113Disallow: /
114
115# All remaining visitors
116# ==========================
117User-agent: *
118Allow: /ads.txt
119Allow: /app-ads.txt
120Disallow: /
121
122
123# ==========================
124Sitemap: https://www.news.com.au/sitemap.xml
125Sitemap: https://www.news.com.au/news-sitemap.xml
126Sitemap: https://www.news.com.au/coupons/sitemap.xml
127Sitemap: https://www.news.com.au/video-sitemap.xml
128Sitemap: https://www.news.com.au/video-vertical-sitemap.xml
129Sitemap: https://www.news.com.au/video-horizontal-sitemap.xml
130Sitemap: https://www.news.com.au/best-of-sitemap.xml
131
What do the bots actually get?
This page reads robots.txt, which states what news.com.au asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on news.com.au Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does news.com.au block GPTBot?
Yes. news.com.au’s robots.txt disallows GPTBot from the whole site (line 120: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does news.com.au block ClaudeBot?
Yes. news.com.au’s robots.txt disallows ClaudeBot from the whole site (line 120: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does news.com.au block Google-Extended?
Yes. news.com.au’s robots.txt disallows Google-Extended from the whole site (line 113: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does news.com.au block PerplexityBot?
Yes. news.com.au’s robots.txt disallows PerplexityBot from the whole site (line 120: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find news.com.au?
Not through its own crawler. news.com.au’s robots.txt disallows OAI-SearchBot from the whole site (line 120: Disallow: /). OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.