Search engines · Apple

Applebot

Apple’s crawler for Siri and Spotlight suggestions. When robots.txt doesn’t name it, it follows the Googlebot rules.

Operator: Apple
Purpose: Search index. The crawlers behind Google, Bing and the other search engines. Blocking one drops you out of its results.
robots.txt token: Applebot
User agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)
Documentation: support.apple.com/en-us/119829

Block Applebot

Add a group naming it to /robots.txt. Compliant crawlers stop before requesting any page. A firewall rule on the user agent is the only way to enforce it against crawlers that ignore robots.txt.

User-agent: Applebot
Disallow: /

Allow Applebot when everything else is blocked

A group naming the bot wins over User-agent: *, so an explicit allow lets it in while your catch-all rules stay in place.

User-agent: Applebot
Allow: /

User-agent: *
Disallow: /

Does Applebot reach your site?

The checker applies your robots.txt the way Applebot does, then requests your page with the user agent above, alongside 29 other bots.

Also run by Apple: Applebot-Extended.

Other search engines: Googlebot, Bingbot, DuckDuckBot, PetalBot, YandexBot, Baiduspider.

Build a complete file with the robots.txt generator.