SEO tools · Ahrefs

AhrefsBot

Builds Ahrefs’ backlink index. Blocking it hides your site from Ahrefs users’ reports, not from search engines.

Operator: Ahrefs
Purpose: Backlink & SEO index. Crawlers behind backlink and SEO tools. Many sites block them to save bandwidth; it has no effect on search.
robots.txt token: AhrefsBot
User agent: Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)
Documentation: ahrefs.com/robot

Block AhrefsBot

Add a group naming it to /robots.txt. Compliant crawlers stop before requesting any page. A firewall rule on the user agent is the only way to enforce it against crawlers that ignore robots.txt.

User-agent: AhrefsBot
Disallow: /

Allow AhrefsBot when everything else is blocked

A group naming the bot wins over User-agent: *, so an explicit allow lets it in while your catch-all rules stay in place.

User-agent: AhrefsBot
Allow: /

User-agent: *
Disallow: /

Does AhrefsBot reach your site?

The checker applies your robots.txt the way AhrefsBot does, then requests your page with the user agent above, alongside 29 other bots.

Other seo tools: SemrushBot, MJ12bot, DotBot.

Build a complete file with the robots.txt generator.