robots.txt generator for AI crawlers, search engines and SEO bots

Search engines · Search index

The crawlers behind Google, Bing and the other search engines. Blocking one drops you out of its results.

GooglebotGoogle Allow Block

BingbotMicrosoft Allow Block

DuckDuckBotDuckDuckGo Allow Block

ApplebotApple Allow Block

PetalBotHuawei Allow Block

YandexBotYandex Allow Block

BaiduspiderBaidu Allow Block

Crawlers that build the indexes AI assistants answer from and cite. Blocking one keeps you out of its answers.

Crawlers that collect pages to train models. Blocking them does not affect search or AI answers.

Fetchers that load a page because a person asked an assistant about it. Blocking one means the assistant can’t open your pages on request.

Crawlers behind backlink and SEO tools. Many sites block them to save bandwidth; it has no effect on search.

AhrefsBotAhrefs Allow Block

SemrushBotSemrush Allow Block

MJ12botMajestic Allow Block

DotBotMoz Allow Block

Bots not named above User-agent: * Allow Block

Sitemap URL (optional)