Who blocks AI · #408 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does gofundme.com block AI crawlers?
No. gofundme.com’s robots.txt lets all 19 AI crawlers we track crawl the site. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotAllowedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- www.gofundme.com/robots.txt · 113 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Allowed | no rule matches the home page |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Allowed | no rule matches the home page |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | no rule matches the home page |
ClaudeBot | Anthropic | Allowed | no rule matches the home page |
Google-Extended | Allowed | no rule matches the home page | |
CCBot | Common Crawl | Allowed | no rule matches the home page |
Applebot-Extended | Apple | Allowed | no rule matches the home page |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Allowed | no rule matches the home page |
Bytespider | ByteDance | Allowed | no rule matches the home page |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Allowed | no rule matches the home page |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Allowed | no rule matches the home page |
View www.gofundme.com/robots.txt · 113 lines
1User-Agent: *
2Disallow: /mvc.php*
3Disallow: /*contact?t=donation_page_report
4Disallow: /*campaign/gallery/*
5Disallow: /f/*/widget/*
6Disallow: /f/*/donate/sign-in
7Disallow: /track
8Disallow: /track/exposure
9Disallow: /auth
10
11# Disallow all poster images
12# https://gofundme.atlassian.net/browse/YO-3839
13Disallow: /f/*/fb/* # Facebook
14Disallow: /f/*/x/* # Twitter
15Disallow: /f/*/ig/* # Instagram
16Disallow: /f/*/wa/* # WhatsApp
17Disallow: /f/*/li/* # LinkedIn
18Disallow: /f/*/e/* # Email
19Disallow: /f/*/sms/* # SMS
20Disallow: /f/*/fm/* # Messenger
21Disallow: /f/*/cl/* # Copy Link
22Disallow: /f/*/qr/* # QR Code
23Disallow: /f/*/nd/* # Nextdoor
24Disallow: /f/*/igs/* # Instagram Story
25
26# Disallow all profile poster images
27Disallow: /u/*/fb/* # Facebook
28Disallow: /u/*/x/* # Twitter
29Disallow: /u/*/ig/* # Instagram
30Disallow: /u/*/wa/* # WhatsApp
31Disallow: /u/*/li/* # LinkedIn
32Disallow: /u/*/e/* # Email
33Disallow: /u/*/sms/* # SMS
34Disallow: /u/*/fm/* # Messenger
35Disallow: /u/*/cl/* # Copy Link
36Disallow: /u/*/qr/* # QR Code
37Disallow: /u/*/nd/* # Nextdoor
38Disallow: /u/*/igs/* # Instagram Story
39
40# Block Next.js internal routes from crawlers
41Disallow: /_next/
42Allow: /_next/static/
43Allow: /_next/image
44
45# Disallow modal urls
46# https://gofundme.atlassian.net/browse/YO-3840
47Disallow: /f/*?modal
48Disallow: /f/*&modal=
49
50Disallow: /f/*/donate
51Disallow: /f/*/donations
52Disallow: /f/*/share
53Disallow: /f/*/leaderboard
54Disallow: /sign-in?redirect*
55Disallow: /s?*
56
57# Allow LinkedIn bot to access /f/*/li/*
58User-agent: LinkedInBot
59Allow: /f/*/li/*
60Allow: /u/*/li/*
61
62# Allow Twitter bot to access /f/*/x/*
63User-agent: Twitterbot
64Allow: /f/*/x/*
65Allow: /u/*/x/*
66
67# Allow Facebook bots to access /f/*/fb/* and /u/*/fb/* so share links unfurl
68User-agent: facebookexternalhit
69Allow: /f/*/fb/*
70Allow: /u/*/fb/*
71
72User-agent: Facebot
73Allow: /f/*/fb/*
74Allow: /u/*/fb/*
75
76Sitemap: https://www.gofundme.com/sitemap.xml
77Sitemap: https://www.gofundme.com/sitemaps/i18n/sitemap.xml
78Sitemap: https://www.gofundme.com/sitemaps/de-at/sitemap.xml
79Sitemap: https://www.gofundme.com/sitemaps/de-ch/sitemap.xml
80Sitemap: https://www.gofundme.com/sitemaps/de-de/sitemap.xml
81Sitemap: https://www.gofundme.com/sitemaps/en-au/sitemap.xml
82Sitemap: https://www.gofundme.com/sitemaps/en-ca/sitemap.xml
83Sitemap: https://www.gofundme.com/sitemaps/en-dk/sitemap.xml
84Sitemap: https://www.gofundme.com/sitemaps/en-fi/sitemap.xml
85Sitemap: https://www.gofundme.com/sitemaps/en-gb/sitemap.xml
86Sitemap: https://www.gofundme.com/sitemaps/en-ie/sitemap.xml
87Sitemap: https://www.gofundme.com/sitemaps/en-no/sitemap.xml
88Sitemap: https://www.gofundme.com/sitemaps/en-se/sitemap.xml
89Sitemap: https://www.gofundme.com/sitemaps/es-es/sitemap.xml
90Sitemap: https://www.gofundme.com/sitemaps/es-us/sitemap.xml
91Sitemap: https://www.gofundme.com/sitemaps/fr-be/sitemap.xml
92Sitemap: https://www.gofundme.com/sitemaps/fr-ca/sitemap.xml
93Sitemap: https://www.gofundme.com/sitemaps/fr-ch/sitemap.xml
94Sitemap: https://www.gofundme.com/sitemaps/fr-fr/sitemap.xml
95Sitemap: https://www.gofundme.com/sitemaps/fr-lu/sitemap.xml
96Sitemap: https://www.gofundme.com/sitemaps/it-it/sitemap.xml
97Sitemap: https://www.gofundme.com/sitemaps/nl-be/sitemap.xml
98Sitemap: https://www.gofundme.com/sitemaps/nl-nl/sitemap.xml
99Sitemap: https://www.gofundme.com/sitemaps/pt-pt/sitemap.xml
100Sitemap: https://www.gofundme.com/c/sitemap_index.xml
101Sitemap: https://www.gofundme.com/de-de/c/sitemap_index.xml
102Sitemap: https://www.gofundme.com/en-au/c/sitemap_index.xml
103Sitemap: https://www.gofundme.com/en-ca/c/sitemap_index.xml
104Sitemap: https://www.gofundme.com/en-gb/c/sitemap_index.xml
105Sitemap: https://www.gofundme.com/en-ie/c/sitemap_index.xml
106Sitemap: https://www.gofundme.com/es-es/c/sitemap_index.xml
107Sitemap: https://www.gofundme.com/es-mx/c/sitemap_index.xml
108Sitemap: https://www.gofundme.com/es-us/c/sitemap_index.xml
109Sitemap: https://www.gofundme.com/fr-fr/c/sitemap_index.xml
110Sitemap: https://www.gofundme.com/it-it/c/sitemap_index.xml
111Sitemap: https://www.gofundme.com/nl-nl/c/sitemap_index.xml
112Sitemap: https://www.gofundme.com/pt-pt/c/sitemap_index.xml
113
What do the bots actually get?
This page reads robots.txt, which states what gofundme.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on gofundme.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does gofundme.com block GPTBot?
No. gofundme.com’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does gofundme.com block ClaudeBot?
No. gofundme.com’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does gofundme.com block Google-Extended?
No. gofundme.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does gofundme.com block PerplexityBot?
No. gofundme.com’s robots.txt lets PerplexityBot crawl the site. PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find gofundme.com?
Yes. gofundme.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.