Who blocks AI · #509 in the Majestic Million

robots.txt · checked 9 Nov 2026

Does meetup.com block AI crawlers?

Yes. meetup.com’s robots.txt blocks 1 of the 19 AI crawlers we track, including GPTBot. Search engines such as Googlebot and Bingbot may crawl it.

GPTBotBlocked OAI-SearchBotAllowed ClaudeBotAllowed PerplexityBotAllowed Google-ExtendedAllowed CCBotAllowed

robots.txt: www.meetup.com/robots.txt · 154 lines
llms.txt: found
Checked: 9 November 2026, as canaicrawl. We check again every week.

Every bot, and the line that decides

Search engines

Bot	Operator	robots.txt	Deciding line
`Googlebot`	Google	Allowed	no rule matches the home page
`Bingbot`	Microsoft	Allowed	no rule matches the home page
`DuckDuckBot`	DuckDuckGo	Allowed	no rule matches the home page
`Applebot`	Apple	Allowed	no rule matches the home page
`PetalBot`	Huawei	Allowed	no rule matches the home page
`YandexBot`	Yandex	Allowed	no rule matches the home page
`Baiduspider`	Baidu	Allowed	no rule matches the home page

AI search

Bot	Operator	robots.txt	Deciding line
`OAI-SearchBot`	OpenAI	Allowed	no rule matches the home page
`PerplexityBot`	Perplexity	Allowed	no rule matches the home page
`Claude-SearchBot`	Anthropic	Allowed	no rule matches the home page
`DuckAssistBot`	DuckDuckGo	Allowed	no rule matches the home page
`Amazonbot`	Amazon	Allowed	no rule matches the home page
`YouBot`	You.com	Allowed	no rule matches the home page

AI training

Bot	Operator	robots.txt	Deciding line
`GPTBot`	OpenAI	Blocked	line 153 · `Disallow: /` under `User-agent: GPTBot`
`ClaudeBot`	Anthropic	Allowed	no rule matches the home page
`Google-Extended`	Google	Allowed	no rule matches the home page
`CCBot`	Common Crawl	Allowed	no rule matches the home page
`Applebot-Extended`	Apple	Allowed	no rule matches the home page
`GoogleOther`	Google	Allowed	no rule matches the home page
`Meta-ExternalAgent`	Meta	Allowed	no rule matches the home page
`Bytespider`	ByteDance	Allowed	no rule matches the home page

User-triggered fetchers

Bot	Operator	robots.txt	Deciding line
`ChatGPT-User`	OpenAI	Allowed	no rule matches the home page
`Claude-User`	Anthropic	Allowed	no rule matches the home page
`Perplexity-User`	Perplexity	Allowed	no rule matches the home page
`MistralAI-User`	Mistral	Allowed	no rule matches the home page
`Meta-ExternalFetcher`	Meta	Allowed	no rule matches the home page

SEO tools

Bot	Operator	robots.txt	Deciding line
`AhrefsBot`	Ahrefs	Allowed	no rule matches the home page
`SemrushBot`	Semrush	Allowed	no rule matches the home page
`MJ12bot`	Majestic	Allowed	no rule matches the home page
`DotBot`	Moz	Allowed	no rule matches the home page

View www.meetup.com/robots.txt · 154 lines

1Sitemap: https://www.meetup.com/pro-index-sitemap.xml

2Sitemap: https://www.meetup.com/topics-index-sitemap.xml

3Sitemap: https://www.meetup.com/topics-localized-index-sitemap.xml

4Sitemap: https://www.meetup.com/topics-country-index-sitemap.xml

5Sitemap: https://www.meetup.com/groups-index-sitemap.xml

6Sitemap: https://www.meetup.com/events-index-sitemap.xml

7Sitemap: https://www.meetup.com/international-cities-static-ga-index-sitemap.xml

8Sitemap: https://www.meetup.com/usa-cities-static-ga-index-sitemap.xml

9Sitemap: https://www.meetup.com/find-international-index-sitemap.xml

10Sitemap: https://www.meetup.com/find-usa-index-sitemap.xml

11Sitemap: https://www.meetup.com/blog/sitemap_index.xml

12Sitemap: https://www.meetup.com/cities-index-sitemap.xml

14Sitemap: https://www.meetup.com/cities_sitemap_index.xml

15Sitemap: https://www.meetup.com/interests_sitemap_index.xml

17User-agent: *

18Disallow: /files/

19Disallow: /fb/

20Disallow: /preview/

21Disallow: /n/*

23User-agent: *

24Disallow: */calendar/*atom*

25Disallow: */calendar/*rss*

26Disallow: */calendar/*xml*

28User-agent: *

29Disallow: */events/atom/*

30Disallow: */events/rss/*

31Disallow: */events/xml/*

33User-agent: *

34Disallow: */rsvps/*atom*

35Disallow: */rsvps/*rss*

36Disallow: */rsvps/*xml*

38User-agent: *

39Disallow: */newest/*atom*

40Disallow: */newest/*rss*

41Disallow: */newest/*xml*

43User-agent: *

44Disallow: /meetup_api/?

45Disallow: /meetup_api?

46Disallow: /*/meetup_api/

47Disallow: /*/meetup_api?

48Disallow: /*/meetup_api$

50User-agent: *

51Disallow: /api/?

52Disallow: /api?

53Disallow: /*/api/

54Disallow: /*/api?

55Disallow: /*/api$

57User-agent: *

58Disallow: */report_abuse/*

60User-agent: *

61Disallow: /*?keywords=*

62Disallow: /*&keywords=*

63Disallow: /*?source=*

64Disallow: /*&source=*

65Disallow: /*?distance=*

66Disallow: /*&distance=*

67Disallow: /*?dateRange=*

68Disallow: /*&dateRange=*

69Disallow: /*?eventType=*

70Disallow: /*&eventType=*

71Disallow: /*?sortField=*

72Disallow: /*&sortField=*

73Disallow: /*?categoryId=*

74Disallow: /*&categoryId=*

75Disallow: /*?s=*

76Disallow: /*?utf8=*

77Disallow: /*&s=*

78Disallow: /*&utf8=*

79Disallow: /*?_af_cid=*

80Disallow: /*?_cookie-check=*

81Disallow: /*?_locale=*

82Disallow: /*?albumsOffset=*

83Disallow: /*?chapter_analytics_code=*

84Disallow: /*?delete_id=*

85Disallow: /*?edit_id=*

86Disallow: /*?err=*

87Disallow: /*?errors=*

88Disallow: /*?fbclid=*

89Disallow: /*?forumId=*

90Disallow: /*?mv=*

91Disallow: /*?op=*

92Disallow: /*?pager.offset=*

93Disallow: /*?photoAlbumId=*

94Disallow: /*?post=*

95Disallow: /*?returnPage=*

96Disallow: /*?search_forum=*

97Disallow: /*?set_mobile=*

98Disallow: /*?showAllGroups=*

99Disallow: /*?sort=*

100Disallow: /*?sortBy=*

101Disallow: /*?success=*

102Disallow: /*?_locale=*

103Disallow: /*?zipstatecity=*

104Disallow: /*?state=*

105Disallow: /*?radius=*

106Disallow: /*?pageToken=*

107Disallow: /*?offset=*

108Disallow: /*?hidePromoBar=*

109Disallow: /*?country=*

110Disallow: /*?city=*

111Disallow: /*&_af_cid=*

112Disallow: /*&_cookie-check=*

113Disallow: /*&_locale=*

114Disallow: /*&albumsOffset=*

115Disallow: /*&chapter_analytics_code=*

116Disallow: /*&delete_id=*

117Disallow: /*&edit_id=*

118Disallow: /*&err=*

119Disallow: /*&errors=*

120Disallow: /*&fbclid=*

121Disallow: /*&forumId=*

122Disallow: /*&mv=*

123Disallow: /*&op=*

124Disallow: /*&pager.offset=*

125Disallow: /*&photoAlbumId=*

126Disallow: /*&post=*

127Disallow: /*&returnPage=*

128Disallow: /*&returnUri=*

129Disallow: /*&search_forum=*

130Disallow: /*&set_mobile=*

131Disallow: /*&showAllGroups=*

132Disallow: /*&sort=*

133Disallow: /*&sortBy=*

134Disallow: /*&success=*

135Disallow: /*&zipstatecity=*

136Disallow: /*&state=*

137Disallow: /*&radius=*

138Disallow: /*&pageToken=*

139Disallow: /*&offset=*

140Disallow: /*&hidePromoBar=*

141Disallow: /*&country=*

142Disallow: /*&city=*

143Disallow: /*&location=*

144Disallow: /*?location=*

145Disallow: /mu_api/

146Disallow: /gql*

147Disallow: /intl/*

148Disallow: /_next/data/*

149Allow: */login/*

150Allow: /*?_locale=

151

152User-agent: GPTBot

153Disallow: /

154

What do the bots actually get?

This page reads robots.txt, which states what meetup.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.

Run a live check on meetup.com Test a path in the robots.txt tester

History

19 October 2026: first checked.

Questions

Does meetup.com block GPTBot?

Yes. meetup.com’s robots.txt disallows GPTBot from the whole site (line 153: Disallow: /). GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.

Does meetup.com block ClaudeBot?

No. meetup.com’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.

Does meetup.com block Google-Extended?

No. meetup.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.

Does meetup.com block PerplexityBot?

No. meetup.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.

Can ChatGPT search find meetup.com?

Yes. meetup.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.