Bot directory · SEO tools

SEO tools · Majestic

MJ12bot

Majestic’s distributed crawler for its link index. Blocking it does not affect search engines.

Operator
Majestic
Purpose
Backlink & SEO index. Crawlers behind backlink and SEO tools. Many sites block them to save bandwidth; it has no effect on search.
robots.txt token
MJ12bot
User agent
Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)
Documentation
mj12bot.com/

Block MJ12bot

Add a group naming it to /robots.txt. Compliant crawlers stop before requesting any page. A firewall rule on the user agent is the only way to enforce it against crawlers that ignore robots.txt.

User-agent: MJ12bot
Disallow: /

Allow MJ12bot when everything else is blocked

A group naming the bot wins over User-agent: *, so an explicit allow lets it in while your catch-all rules stay in place.

User-agent: MJ12bot
Allow: /

User-agent: *
Disallow: /

Does MJ12bot reach your site?

The checker applies your robots.txt the way MJ12bot does, then requests your page with the user agent above, alongside 29 other bots, and opens the report on MJ12bot.

Other seo tools: AhrefsBot, SemrushBot, DotBot.

Build a complete file with the robots.txt generator.