Applebot-Extended
Applebot-Extended isn’t a crawler. It is a name you put in robots.txt to tell Apple whether pages its crawler, Applebot, fetched may be used to train Apple’s foundation models. Disallowing it doesn’t remove your site from Siri, Spotlight or Safari.
- Operator
- Apple
- Purpose
- Model training. Crawlers that collect pages to train models. Blocking them does not affect search or AI answers.
- robots.txt token
Applebot-Extended- We test with
- None. This is a robots.txt token, not a crawler: the operator’s normal crawler fetches the page and this name only governs how it may be used.
- Documentation
- support.apple.com/en-us/119829
What Applebot-Extended does
Applebot-Extended does not crawl webpages. Webpages that disallow Applebot-Extended can still be included in search results. Applebot-Extended is only used to determine how to use the data crawled by the Applebot user agent.
support.apple.com
The data crawled by Applebot may also be used to help train Apple foundation models powering generative AI features across Apple products, including Apple Intelligence, Services, and Developer Tools. Web publishers can opt-out from having their content used to train generative foundation models by disallowing Applebot-Extended in the robots.txt file.
support.apple.com
Apple also says rules for Applebot-Extended aren’t considered in ranking for search.
The Applebot-Extended user agent
None. No request ever arrives as Applebot-Extended: Applebot fetches the pages, and this token only governs what Apple may do with them. You won’t see it in your logs.
How Applebot-Extended reads robots.txt
Add a group naming it:
User-agent: Applebot-Extended
Disallow: /
- It is read from Applebot’s fetch of robots.txt, so keep Applebot allowed to read
/robots.txt. - Narrower rules work too.
Disallow: /blog/opts only that section out of training.
Applebot-Extended IP addresses
Applebot-Extended has no addresses of its own, because it sends no requests. Applebot’s addresses are on the Applebot page, with a verifier.
Can you block Applebot-Extended at a firewall?
No. A firewall sees requests, and none carry this name. Blocking Applebot at the firewall would also remove you from Siri, Spotlight and Safari. Use robots.txt.
Block Applebot-Extended
Add a group naming it to /robots.txt. Nothing stops fetching; the operator’s crawler still reads the page for its normal purpose, but may not use it this way. Applebot keeps crawling for Siri, Spotlight and Safari; Apple just won’t use the pages to train its models.
User-agent: Applebot-Extended
Disallow: /
Allow Applebot-Extended when everything else is blocked
A group naming the bot wins over User-agent: *, so an explicit allow lets it in while your catch-all rules stay in place.
User-agent: Applebot-Extended
Allow: /
User-agent: *
Disallow: /
Does Applebot-Extended reach your site?
The checker applies your robots.txt the way Applebot-Extended does, alongside 29 other bots, and opens the report on Applebot-Extended.
Questions about Applebot-Extended
What is Applebot-Extended?
A robots.txt token Apple uses to decide whether pages Applebot crawled may be used to train its generative foundation models. Apple says Applebot-Extended doesn’t crawl webpages itself.
Does disallowing Applebot-Extended remove my site from Siri or Spotlight?
No. Apple says pages that disallow Applebot-Extended can still be included in search results, and its rules aren’t considered in search ranking.
Can I block Applebot-Extended at my firewall?
No requests use that name, so there is nothing to block. The only control is a robots.txt group naming it.
Also run by Apple: Applebot.
Other ai training: GPTBot, ClaudeBot, Google-Extended, CCBot, GoogleOther, Meta-ExternalAgent, Bytespider.
Build a complete file with the robots.txt generator.