SEO tools · Ahrefs
AhrefsBot
Builds Ahrefs’ backlink index. Blocking it hides your site from Ahrefs users’ reports, not from search engines.
- Operator
- Ahrefs
- Purpose
- Backlink & SEO index. Crawlers behind backlink and SEO tools. Many sites block them to save bandwidth; it has no effect on search.
- robots.txt token
AhrefsBot- User agent
Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)- Documentation
- ahrefs.com/robot
Block AhrefsBot
Add a group naming it to /robots.txt. Compliant crawlers stop before requesting any page. A firewall rule on the user agent is the only way to enforce it against crawlers that ignore robots.txt.
User-agent: AhrefsBot
Disallow: /
Allow AhrefsBot when everything else is blocked
A group naming the bot wins over User-agent: *, so an explicit allow lets it in while your catch-all rules stay in place.
User-agent: AhrefsBot
Allow: /
User-agent: *
Disallow: /
Does AhrefsBot reach your site?
The checker applies your robots.txt the way AhrefsBot does, then requests your page with the user agent above, alongside 29 other bots, and opens the report on AhrefsBot.
Other seo tools: SemrushBot, MJ12bot, DotBot.
Build a complete file with the robots.txt generator.