Who blocks AI · #616 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does slate.com block AI crawlers?
Yes. slate.com’s robots.txt blocks 10 of the 19 AI crawlers we track, including GPTBot, ClaudeBot, Google-Extended and 7 more. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotBlockedOAI-SearchBotAllowedClaudeBotBlockedPerplexityBotAllowedGoogle-ExtendedBlockedCCBotBlocked- robots.txt
- slate.com/robots.txt · 461 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Allowed | no rule matches the home page |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Blocked | line 115 · Disallow: / under User-agent: Amazonbot |
YouBot | You.com | Blocked | line 115 · Disallow: / under User-agent: YouBot |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Blocked | line 115 · Disallow: / under User-agent: GPTBot |
ClaudeBot | Anthropic | Blocked | line 115 · Disallow: / under User-agent: ClaudeBot |
Google-Extended | Blocked | line 115 · Disallow: / under User-agent: Google-Extended | |
CCBot | Common Crawl | Blocked | line 115 · Disallow: / under User-agent: CCBot |
Applebot-Extended | Apple | Blocked | line 115 · Disallow: / under User-agent: Applebot-Extended |
GoogleOther | Blocked | line 115 · Disallow: / under User-agent: GoogleOther | |
Meta-ExternalAgent | Meta | Blocked | line 115 · Disallow: / under User-agent: meta-externalagent |
Bytespider | ByteDance | Blocked | line 115 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Partly blocked | line 452 · Disallow: /search line 453 · Disallow: /comments/ and 6 more |
SemrushBot | Semrush | Partly blocked | line 452 · Disallow: /search line 453 · Disallow: /comments/ and 6 more |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Partly blocked | line 452 · Disallow: /search line 453 · Disallow: /comments/ and 6 more |
View slate.com/robots.txt · 461 lines
1User-agent: *
2Disallow: /search
3Disallow: /comments/
4Disallow: /_
5
6Sitemap: https://slate.com/sitemaps/sitemap-index.xml
7
8User-agent: AmazonAdBot
9User-agent: Amazon-Advertising-ad-standards-bot
10Allow: /
11
12# Known agents
13
14# AI Data Scraper, Scraper
15User-agent: 008
16User-agent: AdsTxtCrawler
17User-agent: AgentDataBot
18User-agent: AI2Bot
19User-agent: Ai2Bot-Dolma
20User-agent: Amazonbot
21User-agent: ApifyBot
22User-agent: ApifyWebsiteContentCrawler
23User-agent: Applebot-Extended
24User-agent: artsdata-crawler
25User-agent: BeansLLM-CorpusBot
26User-agent: bedrockbot
27User-agent: BixelBot
28User-agent: Bravebot
29User-agent: Brightbot
30User-agent: Bytespider
31User-agent: CCBot
32User-agent: ChatGLM-Spider
33User-agent: ClaudeBot
34User-agent: CloudflareBrowserRenderingCrawler
35User-agent: CloudVertexBot
36User-agent: cohere-training-data-crawler
37User-agent: CohereBot
38User-agent: Cotoyogi
39User-agent: CragCrawler
40User-agent: Crawlspace
41User-agent: Datenbank Crawler
42User-agent: dcrawl
43User-agent: DeepSeekBot
44User-agent: DesearchBot
45User-agent: Diffbot
46User-agent: Doubaobot
47User-agent: ElevenlabsCrawler
48User-agent: ERNIEBot
49User-agent: ExaBot
50User-agent: FacebookBot
51User-agent: FirecrawlAgent
52User-agent: Google-Extended
53User-agent: GoogleOther
54User-agent: GPTBot
55User-agent: HelloworkJobPostingBot
56User-agent: HenkBot
57User-agent: HTTrack
58User-agent: HTTrack 3.0
59User-agent: Hunyuan
60User-agent: ICC-Crawler
61User-agent: ImageBot
62User-agent: imageSpider
63User-agent: IndeedJobBot
64User-agent: Kangaroo Bot
65User-agent: Keenable-User
66User-agent: KeenableBot
67User-agent: KimiBot
68User-agent: KimiCrawler
69User-agent: KrawlerBot
70User-agent: laion-huggingface-processor
71User-agent: LCC
72User-agent: meta-externalagent
73User-agent: MetaInspector
74User-agent: MetaJobBot
75User-agent: micro-crawl
76User-agent: MistralBot
77User-agent: MoonshotBot
78User-agent: MoonSpider
79User-agent: Mozilla-Tabstack
80User-agent: netEstate Imprint Crawler
81User-agent: newspaper
82User-agent: Nutch
83User-agent: nyt_scraping
84User-agent: Offline Explorer
85User-agent: omgili
86User-agent: OpenindexSpider
87User-agent: PanguBot
88User-agent: Potions
89User-agent: Quartr Crawler
90User-agent: Querit-SearchBot
91User-agent: QueritBot
92User-agent: QwenBot
93User-agent: recraft-image-dataset
94User-agent: ReflectionBot
95User-agent: SalamandraVLM
96User-agent: SBIntuitionsBot
97User-agent: Scrapy
98User-agent: ScryBot
99User-agent: ServerHunterSpider
100User-agent: ShapBot
101User-agent: SofyaBot
102User-agent: Spider
103User-agent: StatsDroneBot
104User-agent: TalarionFrontierCrawler
105User-agent: TavilyBot
106User-agent: Terra Cotta
107User-agent: TerraCotta
108User-agent: Thinkbot
109User-agent: Timpibot
110User-agent: VelenPublicWebCrawler
111User-agent: web-image-collection-research
112User-agent: webzio-extended
113User-agent: YiBot
114User-agent: YouBot
115Disallow: /
116
117# Intelligence Gatherer, SEO Crawler
118User-agent: A360-Search
119User-agent: aa
120User-agent: ActiveComply
121User-agent: Adagiobot
122User-agent: adbeat_bot
123User-agent: adidxbot
124User-agent: AdkernelTopicCrawler
125User-agent: admantx
126User-agent: AdmixerBot
127User-agent: AdpResearchBot
128User-agent: AdsBot-Google
129User-agent: AdsBot-Google-Mobile
130User-agent: AffsignalCrawler
131User-agent: Ahrefs Site Audit
132User-agent: AhrefsBot
133User-agent: AhrefsSiteAudit
134User-agent: aiHitBot
135User-agent: Amazing-SearchBot
136User-agent: AmazingTest-SearchBot
137User-agent: Amazon AdBot
138User-agent: AmazonProductDiscovery
139User-agent: AndersPinkBot
140User-agent: ArchiveBot
141User-agent: Assaybot
142User-agent: Attracta
143User-agent: AudigentAdBot
144User-agent: Audisto Crawler
145User-agent: Automattic Analytics Crawler
146User-agent: AutotouchCompanyCrawler
147User-agent: Awario
148User-agent: AwarioBot
149User-agent: AwarioSmartBot
150User-agent: Barkrowler
151User-agent: BigUpDataBot
152User-agent: Bing Ads
153User-agent: BitSightBot
154User-agent: Blackboard
155User-agent: BLEXBot
156User-agent: BomboraBot
157User-agent: botify
158User-agent: BrandVerity
159User-agent: BrightEdge Bot
160User-agent: BrightEdge Crawler
161User-agent: Buck
162User-agent: BuiltWith
163User-agent: CaliberBot
164User-agent: Caliperbot
165User-agent: CandidOrgWebCheck
166User-agent: captify-crawler
167User-agent: centryx-domain-mill
168User-agent: channable
169User-agent: CheckMarkNetwork
170User-agent: Cincraw
171User-agent: ClarityBot
172User-agent: Clearscopebot
173User-agent: Clickagy Intelligence Bot
174User-agent: Clickagy Intelligence Bot v2
175User-agent: Cloudflare Custom Hostname Verification
176User-agent: Cloudflare Stream Webhook
177User-agent: Cloudflare-Healthchecks
178User-agent: CloudFlare-Prefetch
179User-agent: Cloudflare-SSLDetector
180User-agent: Cloudflare-Traffic-Manager
181User-agent: Cloudtrellis
182User-agent: cludo.com
183User-agent: Cocolyzebot
184User-agent: cognitiveSEO Crawler
185User-agent: CompanyMapBot
186User-agent: contentking
187User-agent: ContextualBot
188User-agent: ContextWebCom-AdsTxt
189User-agent: contxbot
190User-agent: Convermax
191User-agent: CriteoBot
192User-agent: Cxense
193User-agent: cXensebot
194User-agent: CypressGrowthDiligence
195User-agent: DataForSEO
196User-agent: DataForSEO Bot
197User-agent: DataForSeoBot
198User-agent: Dataprovider
199User-agent: Dataprovider.com
200User-agent: deepcrawl
201User-agent: deepcrawl.com
202User-agent: DomainDrift
203User-agent: DomainStatsBot
204User-agent: DotBot
205User-agent: Dragonbot
206User-agent: Dragonfly
207User-agent: ds9
208User-agent: DVbot
209User-agent: dwelltime-bot
210User-agent: Echobot Bot
211User-agent: EcoVadisSustainabilityBot
212User-agent: EpivozCrawler
213User-agent: ev-crawler
214User-agent: EzLynx
215User-agent: EzoicBot
216User-agent: EzoicBot-Nicheiq
217User-agent: factset_spyderbot
218User-agent: FDL Stats Bot
219User-agent: Fedicabot
220User-agent: FirmCredBot
221User-agent: FlowIQLabsBot
222User-agent: FlyingPress
223User-agent: ForestEngine
224User-agent: getdomaindata
225User-agent: GlobRadarBot
226User-agent: GnowitNewsbot
227User-agent: Google AdsBot
228User-agent: Google AdSense
229User-agent: Google Digital Asset Links
230User-agent: Google StoreBot
231User-agent: Google-adstxt
232User-agent: Google-Adwords-Instant
233User-agent: Google-Display-Ads-Bot
234User-agent: Google-Structured-Data-Testing
235User-agent: GovAPIEnricher
236User-agent: GrapeshotCrawler
237User-agent: GroovinaAdsbot
238User-agent: GrowthXBot
239User-agent: HubSpot
240User-agent: Huckabot
241User-agent: Huckabuy Bot
242User-agent: hypestat
243User-agent: IAS crawler
244User-agent: ias-ie
245User-agent: IDG/IT
246User-agent: ImagesiftBot
247User-agent: IncumbINTBot
248User-agent: Innguma
249User-agent: IntentBeamBot
250User-agent: InteractiveAdvertisingBureau
251User-agent: InterviewOS
252User-agent: IonCrawl
253User-agent: James BOT
254User-agent: Jugendschutzprogramm-Crawler
255User-agent: KargoBot
256User-agent: KargoBot-Artemis-Mobile
257User-agent: KimeBot
258User-agent: Klaviyo
259User-agent: Konturbot
260User-agent: KStandBot
261User-agent: LeadFinderBot
262User-agent: LeadForge
263User-agent: Leikibot
264User-agent: LibreCrawl
265User-agent: LightspeedSystemsCrawler
266User-agent: LinkCheck by Siteimprove
267User-agent: LinkChecker Bot
268User-agent: linkchecker.pro
269User-agent: LinkCheckerBot
270User-agent: linkdexbot
271User-agent: linkfluence
272User-agent: LinksIndexerBot
273User-agent: LinkWalker
274User-agent: ListSignalBot
275User-agent: Macrobondbot
276User-agent: magpie-crawler
277User-agent: Mantisbot
278User-agent: MapTheNetBot
279User-agent: MarketGoo
280User-agent: MBCrawler
281User-agent: McontextualBot
282User-agent: mediaboardbot
283User-agent: MedialogiaBot
284User-agent: MediaMonitoringBot
285User-agent: Mediapartners-Google
286User-agent: Mediatoolkitbot
287User-agent: Mediavine Medatada Parser
288User-agent: MediavineMetadataParser
289User-agent: MegaIndex.ru
290User-agent: meta-externalads
291User-agent: MgidBot
292User-agent: Missinglettr Bot
293User-agent: MixrankBot
294User-agent: MJ12bot
295User-agent: ModernizeBot
296User-agent: Moz dotbot
297User-agent: Moz rogerbot
298User-agent: MRLeadsBot
299User-agent: MuckRack
300User-agent: NanoInteractive
301User-agent: Netcraft
302User-agent: NetcraftSurveyAgent
303User-agent: netEstate NE Crawler
304User-agent: Neticle Crawler
305User-agent: NetSeer crawler
306User-agent: Netvibes
307User-agent: NeuwoBot
308User-agent: NewsRoom.BI
309User-agent: Nitro-
310User-agent: NitroBot
311User-agent: NoahBot
312User-agent: noorobot
313User-agent: NostoCrawlerBot
314User-agent: OAI-AdsBot
315User-agent: OmvionLeadLake
316User-agent: OnCrawl
317User-agent: OnCrawl Mobile
318User-agent: online-webceo-bot
319User-agent: osentix-crawler
320User-agent: outboundbuddy-enrichment
321User-agent: outbrain
322User-agent: Owler
323User-agent: page-preview-tool
324User-agent: Pandalytics
325User-agent: panscient.com
326User-agent: parse.ly scraper
327User-agent: PartnerOptimizer-bot
328User-agent: peer39_crawler
329User-agent: PipericBot
330User-agent: PostGooBot
331User-agent: PostitLeadDiscovery
332User-agent: PoweredByBot
333User-agent: Pro Sitemaps
334User-agent: Pro Sitemaps Generator
335User-agent: productsup.io/crawler
336User-agent: PrognostiXonThinMiner
337User-agent: proximic
338User-agent: QlyzeBot
339User-agent: Quantcastbot
340User-agent: QuuuBot
341User-agent: Qwairy-Bot
342User-agent: Qwarrybot
343User-agent: RankMath Link Checker
344User-agent: RankshiftFetcher
345User-agent: RankurBot
346User-agent: RavenCrawler
347User-agent: Readable
348User-agent: RetroListeCOM
349User-agent: revops-nuvo-detector
350User-agent: RevvimGort
351User-agent: rogerbot-crawler
352User-agent: RSiteAuditor
353User-agent: RyteBot
354User-agent: SalesViewerBot
355User-agent: ScalinthBot
356User-agent: scoop.it
357User-agent: Scope3
358User-agent: scraping@nytimes.com
359User-agent: Screaming Frog SEO Spider
360User-agent: SearchAtlas Bot
361User-agent: SEBot-WA
362User-agent: semantic-visions-analytica
363User-agent: semantic-visions-infoseeker
364User-agent: SemanticVisionsBot
365User-agent: SemrushBot
366User-agent: SemrushBot-BA
367User-agent: SemrushBot-CT
368User-agent: SemrushBot-OCOB
369User-agent: SemrushBot-SA
370User-agent: SemrushBot-SI
371User-agent: SemrushBotBacklinks
372User-agent: SenderBot
373User-agent: SentiBot
374User-agent: Senutobot
375User-agent: seo-audit-check-bot
376User-agent: seo4ajax
377User-agent: seo4ajax.com
378User-agent: SeobilityBot
379User-agent: SEOJuice-SearchBot
380User-agent: SEOkicks
381User-agent: SEOlizer
382User-agent: SEOToolsOutreach
383User-agent: SERankingBacklinksBot
384User-agent: SerpMinerBot
385User-agent: serpstatbot
386User-agent: SirdataBot
387User-agent: SiteAuditBot
388User-agent: Sitebulb
389User-agent: SiteCheck-sitecrawl
390User-agent: SiteCheckerBotCrawler
391User-agent: SiteGuruCrawler
392User-agent: Siteimprove Crawl
393User-agent: Siteliner
394User-agent: SiteStatsBot
395User-agent: SlickBot
396User-agent: slickstream
397User-agent: SmartologyBot
398User-agent: SMBsBot
399User-agent: SMTBot
400User-agent: Sottopop
401User-agent: SpamCipherBot
402User-agent: Speroll-AdsTxt-Crawler
403User-agent: SpyglassesBot
404User-agent: StatistikAustria
405User-agent: StatusNestBacklinkSpider
406User-agent: Storebot-Google
407User-agent: StormIntelCrawler
408User-agent: Stripebot
409User-agent: Taboola
410User-agent: Taboolabot
411User-agent: TechSpyBot
412User-agent: TheWebReport
413User-agent: TitanBot
414User-agent: topicsearch-crawler
415User-agent: tphotobot
416User-agent: trendictionbot
417User-agent: TrendsmapResolver
418User-agent: TTD-Content
419User-agent: TurnitinBot
420User-agent: TweetmemeBot
421User-agent: Twingly
422User-agent: Twingly Recon-Sjostrom
423User-agent: uipbot
424User-agent: um-FC
425User-agent: um-IC
426User-agent: um-LN
427User-agent: UpContent
428User-agent: UpContent-QA
429User-agent: UpContent-Test
430User-agent: Verity
431User-agent: virustotal
432User-agent: W3 Validator Services
433User-agent: WappalyzerBot
434User-agent: weborama-fetcher
435User-agent: webspidermount
436User-agent: WebTrackrCrawler
437User-agent: webzio
438User-agent: WhiteCircleTrustBot
439User-agent: WhyNoIPv6Bot
440User-agent: wknd-bot
441User-agent: woorankreview
442User-agent: XoviOnpageCrawler
443User-agent: YaDirectFetcher
444User-agent: Yahoo Ad monitoring
445User-agent: Yahoo Japan
446User-agent: YaK
447User-agent: Yext Inc
448User-agent: YextBot
449User-agent: Zoombot
450User-agent: ZoominfoBot
451User-agent: Zvelo
452Disallow: /search
453Disallow: /comments/
454Disallow: /_
455Disallow: /css
456Disallow: /fonts
457Disallow: /media
458Disallow: /piano
459Disallow: /static
460Crawl-delay: 10
461
What do the bots actually get?
This page reads robots.txt, which states what slate.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on slate.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does slate.com block GPTBot?
Yes. slate.com’s robots.txt disallows GPTBot from the whole site (line 115: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does slate.com block ClaudeBot?
Yes. slate.com’s robots.txt disallows ClaudeBot from the whole site (line 115: Disallow: /). ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does slate.com block Google-Extended?
Yes. slate.com’s robots.txt disallows Google-Extended from the whole site (line 115: Disallow: /). Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does slate.com block PerplexityBot?
No. slate.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find slate.com?
Yes. slate.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.