Updated weekly · last checked 9 Nov 2026 · 1,000 websites
Who blocks AI crawlers?
Every week we read the robots.txt of 1,000 of the web’s most-linked websites and record which of 30 crawlers each one shuts out: AI training and search crawlers, the fetchers assistants use when someone asks about a page, search engines and SEO tools.
32%block at least one AI crawler
22%block GPTBot, OpenAI’s training crawler
18%opt out of Gemini training (Google-Extended)
14%publish an llms.txt
Out of 891 sites whose robots.txt we could read on 9 November 2026. 109 more refused our request or answered with something other than a robots.txt; they are left out rather than counted as allowing everything.
Which bots are blocked most
More sites block OpenAI’s training crawler, GPTBot (22%), than its search crawler, OAI-SearchBot (10%): blocking training while staying in ChatGPT search is a common choice. A bot outside this list, or one that ignores robots.txt, isn’t counted. Blocking in robots.txt is a request; whether a site also blocks a bot at its firewall can only be seen with a live check.
Changes since 2 November 2026
- github.com changed its rules for Claude-SearchBot, changed its rules for ChatGPT-User, changed its rules for Claude-User, changed its rules for Perplexity-User
- forbes.com started blocking PerplexityBot, started blocking Amazonbot, started blocking GPTBot, started blocking ClaudeBot, started blocking CCBot, started blocking Applebot-Extended, started blocking Meta-ExternalAgent, started blocking Bytespider
- researchgate.net started blocking GoogleOther
- aliyun.com changed its rules for GPTBot, changed its rules for ClaudeBot, changed its rules for CCBot
- hugedomains.com changed its rules for Googlebot, changed its rules for Applebot, changed its rules for PerplexityBot, changed its rules for GPTBot, changed its rules for ClaudeBot, changed its rules for Google-Extended
- apnews.com started blocking Perplexity-User
- fortune.com stopped blocking Google-Extended
- dw.com stopped blocking SemrushBot
- justice.gov started blocking ChatGPT-User, started blocking Claude-User, started blocking Perplexity-User, started blocking MistralAI-User, started blocking Meta-ExternalFetcher
- mail.ru changed its rules for Googlebot, stopped blocking Bingbot, stopped blocking DuckDuckBot, changed its rules for Applebot, stopped blocking PetalBot, stopped blocking YandexBot, stopped blocking Baiduspider, stopped blocking OAI-SearchBot, stopped blocking PerplexityBot, stopped blocking Claude-SearchBot, stopped blocking DuckAssistBot, stopped blocking Amazonbot, stopped blocking YouBot, stopped blocking GPTBot, stopped blocking ClaudeBot, stopped blocking Google-Extended, stopped blocking CCBot, stopped blocking Applebot-Extended, stopped blocking GoogleOther, stopped blocking Meta-ExternalAgent, stopped blocking Bytespider, stopped blocking ChatGPT-User, stopped blocking Claude-User, stopped blocking Perplexity-User, stopped blocking MistralAI-User, stopped blocking Meta-ExternalFetcher, stopped blocking AhrefsBot, stopped blocking SemrushBot, stopped blocking MJ12bot, stopped blocking DotBot
- hostinger.com stopped blocking CCBot
- digitaltrends.com changed its rules for Claude-SearchBot, started blocking GPTBot, started blocking ClaudeBot, started blocking Google-Extended, started blocking CCBot, started blocking Applebot-Extended, started blocking Bytespider, changed its rules for ChatGPT-User, changed its rules for Claude-User
- iheart.com changed its rules for Applebot, changed its rules for OAI-SearchBot, changed its rules for PerplexityBot, changed its rules for Claude-SearchBot, changed its rules for Amazonbot, started blocking GPTBot, started blocking ClaudeBot, started blocking Google-Extended, started blocking CCBot, started blocking Applebot-Extended, started blocking Meta-ExternalAgent, started blocking Bytespider, changed its rules for ChatGPT-User, changed its rules for Claude-User, changed its rules for Perplexity-User, changed its rules for Meta-ExternalFetcher
- zeit.de changed its rules for Googlebot, changed its rules for Bingbot, changed its rules for DuckDuckBot, changed its rules for Applebot, changed its rules for PetalBot, changed its rules for YandexBot, started blocking OAI-SearchBot, started blocking Claude-SearchBot, started blocking DuckAssistBot, started blocking Amazonbot, started blocking YouBot, changed its rules for GoogleOther, started blocking ChatGPT-User, started blocking Claude-User, started blocking MistralAI-User, started blocking Meta-ExternalFetcher, started blocking AhrefsBot, started blocking SemrushBot, started blocking MJ12bot, started blocking DotBot
- pcworld.com stopped blocking Google-Extended
- computerworld.com stopped blocking Google-Extended
- mapquest.com changed its rules for Googlebot, changed its rules for Bingbot, changed its rules for Applebot, started blocking YandexBot, started blocking Baiduspider, started blocking Claude-SearchBot, started blocking DuckAssistBot, started blocking GoogleOther, started blocking Claude-User, started blocking Perplexity-User, started blocking MistralAI-User, started blocking AhrefsBot, started blocking SemrushBot, started blocking MJ12bot, started blocking DotBot
- corriere.it started blocking OAI-SearchBot, started blocking PerplexityBot, started blocking Claude-SearchBot, started blocking DuckAssistBot, started blocking Claude-User, started blocking Perplexity-User, started blocking MistralAI-User
All 1,000 sites
Can each bot crawl the site? No: robots.txt blocks it from the whole site. Part: a group naming it blocks some paths. Each site links to its report, with the lines that decide.
| # | Site | AI bots blocked | GPTBot | OAI-SearchBot | ClaudeBot | PerplexityBot | Google-Extended | CCBot | Search engines blocked | llms.txt |
|---|---|---|---|---|---|---|---|---|---|---|
| 1 | google.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 2 | facebook.com | 19 | No | No | No | No | No | No | 3 | |
| 3 | youtube.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 4 | instagram.com | 19 | No | No | No | No | No | No | 3 | |
| 5 | twitter.com | 19 | No | No | No | No | No | No | 4 | |
| 6 | linkedin.com | 17 | No | Part | No | No | No | No | 2 | |
| 7 | apple.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 1 | |
| 9 | microsoft.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 10 | github.com | 1 | Part | Part | Part | Part | Yes | Yes | 0 | ✓ |
| 12 | wikipedia.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 16 | wordpress.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 17 | x.com | 19 | No | No | No | No | No | No | 4 | |
| 22 | pinterest.com | 18 | No | No | No | No | No | No | 2 | |
| 23 | tiktok.com | 15 | No | No | No | No | No | No | 2 | |
| 26 | nginx.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 27 | whatsapp.com | 6 | No | Yes | No | No | No | Yes | 1 | |
| 28 | vimeo.com | 5 | No | Yes | No | Yes | No | No | 1 | |
| 29 | f5.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 31 | amazon.com | 17 | No | No | No | No | No | No | 1 | |
| 35 | mozilla.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 37 | europa.eu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 39 | adobe.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 40 | wordpress.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 44 | apache.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 48 | reddit.com | 19 | No | No | No | No | No | No | 7 | |
| 50 | nih.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 51 | yahoo.com | 8 | No | Part | No | No | No | No | 1 | |
| 52 | spotify.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 53 | qq.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 58 | godaddy.com | couldn’t read robots.txt (403) | ||||||||
| 60 | tumblr.com | 7 | Yes | Yes | No | Yes | No | No | 0 | |
| 61 | archive.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 62 | macromedia.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 63 | creativecommons.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 67 | w3.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 68 | flickr.com | 14 | No | Part | No | Part | No | No | 1 | |
| 69 | nytimes.com | 16 | No | No | No | No | No | No | 0 | |
| 70 | baidu.com | couldn’t read robots.txt | ||||||||
| 71 | doi.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 73 | medium.com | 7 | No | Yes | No | Yes | Yes | Yes | 0 | |
| 76 | namecheap.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 77 | soundcloud.com | 13 | No | No | No | No | No | No | 0 | |
| 78 | forbes.com | 8 | No | Yes | No | No | Yes | No | 0 | |
| 81 | sciencedirect.com | couldn’t read robots.txt (403) | ||||||||
| 82 | theguardian.com | 11 | Yes | Yes | No | No | Yes | No | 1 | |
| 83 | zoom.us | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 85 | sourceforge.net | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 87 | paypal.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 88 | cpanel.net | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 89 | who.int | 1 | Yes | Yes | Yes | Yes | Yes | No | 0 | |
| 90 | shopify.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 91 | oracle.com | couldn’t read robots.txt (403) | ||||||||
| 93 | opera.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 94 | dropbox.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 95 | bbc.com | 15 | No | No | No | No | No | No | 1 | |
| 97 | vk.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 98 | cnn.com | couldn’t read robots.txt (403) | ||||||||
| 99 | researchgate.net | 2 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 101 | reuters.com | 16 | No | Part | No | No | No | No | 4 | |
| 102 | nature.com | 12 | No | Yes | No | No | No | No | 0 | |
| 103 | discord.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 104 | launchpad.net | 15 | No | No | No | No | No | No | 1 | |
| 105 | mit.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 106 | wikimedia.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 108 | ibm.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 109 | bbc.co.uk | 15 | No | No | No | No | No | No | 1 | |
| 110 | springer.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 111 | bing.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 113 | php.net | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 115 | harvard.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 116 | wiley.com | 9 | No | Yes | No | Yes | No | No | 2 | |
| 117 | weibo.com | 19 | No | No | No | No | No | No | 3 | |
| 119 | gnu.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 121 | cdc.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 122 | un.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 123 | debian.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 124 | linktr.ee | 4 | Part | Part | Part | Part | Yes | Part | 3 | ✓ |
| 125 | issuu.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 1 | |
| 127 | imdb.com | 17 | No | No | No | No | No | No | 2 | |
| 128 | plesk.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 129 | weebly.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 130 | expireddomains.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 131 | stanford.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 133 | openai.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 135 | www.gov.uk | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 136 | miit.gov.cn | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 138 | bloomberg.com | 16 | No | No | No | No | No | No | 1 | ✓ |
| 139 | dropcatch.com | couldn’t read robots.txt (200) | ||||||||
| 142 | washingtonpost.com | 8 | Yes | Yes | No | No | Yes | No | 1 | |
| 144 | stripe.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 145 | calendly.com | 1 | Yes | Yes | Yes | Yes | Yes | No | 0 | ✓ |
| 146 | nasa.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 147 | wsj.com | 15 | Part | Part | No | No | No | No | 2 | |
| 148 | msn.com | 19 | No | No | No | No | No | No | 1 | |
| 149 | arxiv.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 150 | unsplash.com | 3 | Yes | Yes | No | Yes | Yes | Yes | 0 | |
| 153 | trustpilot.com | 8 | Part | Part | Part | Part | Part | Part | 2 | |
| 154 | statista.com | 14 | No | Yes | No | No | No | No | 0 | |
| 155 | businessinsider.com | 3 | Yes | Yes | No | Yes | Yes | No | 0 | ✓ |
| 156 | gitlab.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 157 | ietf.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 158 | line.me | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 161 | bsky.app | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 163 | substack.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 164 | canva.com | 6 | No | Yes | No | Yes | Yes | No | 0 | |
| 165 | mysql.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 166 | twitch.tv | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 168 | cloudflare.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 169 | cnbc.com | 14 | No | No | No | No | No | No | 0 | |
| 171 | eventbrite.com | 1 | Yes | Yes | Yes | Yes | Yes | No | 0 | |
| 172 | oup.com | couldn’t read robots.txt (202) | ||||||||
| 174 | etsy.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 176 | wired.com | 16 | Yes | Yes | No | No | No | No | 1 | |
| 177 | openstreetmap.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 179 | time.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 180 | npr.org | couldn’t read robots.txt | ||||||||
| 182 | samsung.com | 0 | Part | Part | Yes | Part | Part | Yes | 0 | ✓ |
| 183 | stackoverflow.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 184 | behance.net | 7 | No | Yes | No | Yes | No | No | 0 | |
| 186 | slideshare.net | couldn’t read robots.txt (200) | ||||||||
| 187 | tandfonline.com | 1 | No | Yes | Yes | Yes | Yes | Yes | 0 | |
| 188 | hp.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 189 | ebay.com | 8 | No | Part | No | No | Yes | No | 0 | |
| 191 | unesco.org | 7 | No | Yes | No | Yes | Yes | No | 0 | ✓ |
| 192 | chatgpt.com | 19 | No | No | No | No | No | No | 7 | |
| 193 | techcrunch.com | 7 | No | Yes | No | Yes | No | No | 0 | |
| 195 | hubspot.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 198 | iso.org | couldn’t read robots.txt (403) | ||||||||
| 199 | ft.com | 10 | Yes | Yes | No | No | No | No | 0 | |
| 200 | sagepub.com | 4 | Yes | Yes | Yes | Yes | Yes | No | 0 | |
| 201 | mdpi.com | couldn’t read robots.txt (403) | ||||||||
| 204 | loc.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 1 | |
| 205 | go.com | 2 | No | Yes | Yes | Yes | No | Yes | 0 | |
| 206 | berkeley.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 211 | cornell.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 213 | ca.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 214 | usatoday.com | 19 | No | No | No | No | No | No | 4 | |
| 215 | telegraph.co.uk | 17 | No | No | No | No | Part | No | 1 | |
| 216 | booking.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 217 | bilibili.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 218 | jotform.com | 0 | Part | Yes | Part | Part | Yes | Part | 0 | |
| 219 | fda.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 220 | zendesk.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 221 | tripadvisor.com | 9 | No | Part | No | Part | No | No | 1 | |
| 222 | squarespace.com | 0 | Part | Yes | Part | Yes | Part | Part | 0 | ✓ |
| 223 | yandex.ru | 0 | Yes | Part | Yes | Yes | Yes | Yes | 0 | |
| 224 | aliyun.com | 0 | Part | Yes | Part | Yes | Yes | Part | 0 | |
| 225 | rfc-editor.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 227 | cnet.com | 17 | No | No | No | No | Yes | No | 2 | |
| 229 | bandcamp.com | 7 | No | Yes | No | Yes | No | No | 0 | |
| 232 | blogger.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 235 | britannica.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 236 | sina.com.cn | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 238 | cbsnews.com | 1 | No | Yes | Yes | Yes | Yes | Yes | 0 | |
| 239 | surveymonkey.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 240 | mckinsey.com | couldn’t read robots.txt | ||||||||
| 241 | ted.com | 8 | No | Part | No | Part | No | No | 0 | |
| 243 | free.fr | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 1 | |
| 244 | pixabay.com | 6 | No | Yes | No | Yes | No | No | 0 | |
| 245 | redhat.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 246 | sedo.com | couldn’t read robots.txt (503) | ||||||||
| 247 | cambridge.org | 1 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 249 | wix.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 250 | prnewswire.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 251 | hugedomains.com | 0 | Part | Yes | Part | Part | Part | Yes | 0 | ✓ |
| 253 | ieee.org | couldn’t read robots.txt (202) | ||||||||
| 254 | independent.co.uk | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 256 | mailchimp.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 257 | 163.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 258 | nist.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 259 | dailymotion.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 260 | theverge.com | 15 | Yes | Yes | No | No | No | No | 2 | |
| 261 | hbr.org | 3 | No | Yes | Yes | Yes | No | No | 0 | |
| 262 | frontiersin.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 265 | steampowered.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 266 | princeton.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 267 | docker.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 268 | worldbank.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 269 | yelp.com | 19 | No | No | No | No | No | No | 4 | |
| 270 | huawei.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 271 | epa.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 274 | taobao.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 275 | usda.gov | couldn’t read robots.txt (403) | ||||||||
| 276 | python.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 277 | myspace.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 278 | telegram.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 281 | netflix.com | 11 | Part | Part | Part | Part | Part | No | 2 | |
| 282 | imgur.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 283 | latimes.com | 3 | No | Yes | No | Yes | Yes | Yes | 0 | |
| 284 | goodreads.com | 2 | No | Yes | Yes | Yes | Yes | No | 1 | |
| 285 | ubuntu.com | 0 | Part | Part | Part | Part | Part | Yes | 0 | ✓ |
| 286 | oecd.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 287 | nvidia.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 288 | weforum.org | couldn’t read robots.txt (403) | ||||||||
| 290 | mps.gov.cn | couldn’t read robots.txt (521) | ||||||||
| 291 | whitehouse.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 292 | washington.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 293 | nationalgeographic.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 294 | apnews.com | 9 | No | Yes | No | No | Yes | No | 0 | |
| 295 | investopedia.com | 15 | Part | Part | No | No | No | No | 2 | |
| 298 | deloitte.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 299 | scribd.com | couldn’t read robots.txt (200) | ||||||||
| 300 | academia.edu | 11 | No | Part | No | Part | No | No | 2 | |
| 301 | youku.com | couldn’t read robots.txt (200) | ||||||||
| 303 | safety.google | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 304 | ovhcloud.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 305 | pexels.com | 5 | No | Yes | No | Yes | Yes | No | 0 | |
| 306 | themeforest.net | 1 | Yes | Yes | Yes | Yes | Yes | No | 0 | |
| 307 | fandom.com | couldn’t read robots.txt (403) | ||||||||
| 308 | indiatimes.com | 1 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 309 | noaa.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 310 | mayoclinic.org | couldn’t read robots.txt (403) | ||||||||
| 311 | buydomains.com | couldn’t read robots.txt (403) | ||||||||
| 312 | ox.ac.uk | couldn’t read robots.txt (403) | ||||||||
| 313 | theatlantic.com | 13 | Part | Part | No | No | No | No | 1 | |
| 314 | state.gov | couldn’t read robots.txt (403) | ||||||||
| 315 | marriott.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 1 | |
| 316 | plos.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 318 | nbcnews.com | 17 | No | No | No | No | No | No | 1 | |
| 319 | cisco.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 320 | tencent.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 324 | huffpost.com | 16 | No | No | No | No | No | No | 0 | |
| 325 | columbia.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 326 | aol.com | 6 | No | Yes | No | Part | No | No | 1 | |
| 327 | sohu.com | 18 | No | No | No | No | No | No | 4 | |
| 329 | apa.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 330 | livejournal.com | couldn’t read robots.txt | ||||||||
| 331 | xing.com | 0 | Part | Yes | Part | Part | Yes | Yes | 0 | ✓ |
| 332 | arcgis.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 333 | intel.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 337 | indeed.com | 1 | Part | Part | Part | Part | Part | Part | 0 | |
| 338 | odoo.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 1 | |
| 339 | webmd.com | 3 | No | Yes | No | Yes | Yes | No | 0 | |
| 340 | patreon.com | 1 | Yes | Yes | Yes | No | Yes | Yes | 0 | |
| 341 | yale.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 342 | salesforce.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 345 | theconversation.com | 19 | No | No | No | No | No | No | 2 | |
| 346 | umich.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 347 | kickstarter.com | couldn’t read robots.txt (403) | ||||||||
| 348 | ftc.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 349 | pbs.org | 4 | No | Yes | Yes | No | Yes | No | 0 | |
| 351 | quora.com | 17 | No | No | No | No | Part | No | 5 | |
| 352 | biomedcentral.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 353 | ico.org.uk | couldn’t read robots.txt (403) | ||||||||
| 354 | economist.com | 12 | No | No | No | No | No | No | 1 | |
| 355 | dribbble.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 356 | healthline.com | 7 | No | Yes | No | Yes | Yes | No | 0 | |
| 357 | change.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 359 | canada.ca | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 360 | android.com | 0 | Part | Part | Part | Part | Part | Part | 0 | |
| 362 | businesswire.com | couldn’t read robots.txt (403) | ||||||||
| 366 | postgresql.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 367 | eff.org | 0 | Part | Yes | Part | Yes | Part | Part | 0 | ✓ |
| 368 | alibaba.com | 2 | No | Yes | Yes | Yes | No | Yes | 0 | |
| 369 | bmj.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 373 | bbb.org | 0 | Part | Part | Part | Part | Part | Part | 0 | ✓ |
| 374 | fortune.com | 1 | Yes | Yes | Yes | Yes | Yes | No | 0 | |
| 376 | upenn.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 377 | typeform.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 379 | psu.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 380 | admin.ch | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 381 | ovh.com | 3 | No | Yes | Yes | Yes | Yes | No | 0 | |
| 383 | gartner.com | 5 | No | No | Part | Yes | No | No | 0 | |
| 384 | jimdo.com | 0 | Part | Part | Part | Part | Part | Part | 0 | ✓ |
| 386 | pewresearch.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 387 | cbc.ca | couldn’t read robots.txt | ||||||||
| 388 | psychologytoday.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 389 | foxnews.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 390 | fao.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 391 | zhihu.com | 19 | No | No | No | No | No | No | 3 | |
| 392 | deviantart.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 393 | wpastra.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 394 | bund.de | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 395 | cam.ac.uk | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 396 | jstor.org | 4 | No | Yes | No | Yes | No | No | 0 | |
| 397 | moodle.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 398 | firefox.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 399 | usnews.com | 15 | No | No | No | No | No | No | 0 | |
| 401 | letsencrypt.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 402 | uci.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 404 | zdnet.com | 17 | No | No | No | No | Yes | No | 3 | |
| 405 | akismet.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 407 | abc.net.au | 9 | No | Yes | No | No | No | No | 0 | |
| 408 | gofundme.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 409 | sciencedaily.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 410 | coursera.org | 1 | Part | Part | Part | Part | Part | Part | 0 | ✓ |
| 411 | nypost.com | 11 | Yes | Yes | No | No | No | No | 1 | |
| 413 | umn.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 414 | elsevier.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 416 | dailymail.com | 18 | No | No | No | No | No | No | 1 | ✓ |
| 417 | acm.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 418 | xinhuanet.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 419 | elegantthemes.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 420 | newyorker.com | 16 | Yes | Yes | No | No | No | No | 1 | |
| 423 | vice.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 425 | newsweek.com | 8 | No | Yes | No | No | Yes | No | 0 | |
| 426 | thelancet.com | 11 | No | Yes | No | No | No | No | 0 | |
| 428 | scientificamerican.com | 9 | No | No | No | No | No | No | 0 | |
| 429 | kernel.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 430 | arstechnica.com | 16 | Yes | Yes | No | No | No | No | 1 | |
| 431 | oreilly.com | 5 | No | Part | Part | Part | Part | No | 3 | ✓ |
| 432 | irs.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 433 | dw.com | 14 | No | No | No | No | No | No | 2 | |
| 434 | threads.com | 19 | No | No | No | No | No | No | 3 | |
| 435 | orcid.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 436 | themeisle.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 437 | nps.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 439 | uchicago.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 441 | jamanetwork.com | 8 | No | Yes | No | Yes | No | No | 0 | |
| 442 | dell.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 1 | ✓ |
| 444 | git-scm.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 447 | stackexchange.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 449 | mitre.org | couldn’t read robots.txt (403) | ||||||||
| 450 | naver.com | 19 | No | No | No | No | No | No | 7 | |
| 451 | cmu.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 452 | blog.google | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 453 | fastcompany.com | 10 | No | Part | No | No | Part | No | 0 | |
| 454 | wisc.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 455 | slack.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 458 | ucla.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 459 | acs.org | couldn’t read robots.txt (200) | ||||||||
| 460 | pnas.org | 1 | No | Yes | Yes | Yes | Yes | Yes | 0 | |
| 461 | anthropic.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 463 | hilton.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 464 | wiktionary.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 466 | jd.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 467 | gizmodo.com | 3 | Yes | Yes | Yes | Yes | Yes | No | 0 | |
| 469 | bls.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 470 | getbootstrap.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 471 | wikihow.com | 16 | No | No | No | No | Yes | No | 1 | |
| 472 | si.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 473 | globenewswire.com | couldn’t read robots.txt (403) | ||||||||
| 474 | jhu.edu | couldn’t read robots.txt (200) | ||||||||
| 475 | people.com.cn | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 476 | techtarget.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 1 | |
| 477 | merriam-webster.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 478 | usgs.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 480 | elpais.com | 10 | Yes | Yes | No | No | Yes | No | 2 | |
| 481 | huggingface.co | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 482 | science.org | 5 | No | Yes | Yes | Yes | No | No | 0 | |
| 483 | aljazeera.com | 5 | No | Yes | No | No | Yes | Yes | 0 | |
| 484 | openssl.org | 19 | No | No | No | No | No | No | 7 | |
| 485 | unicef.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 486 | itu.int | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 487 | euronews.com | 2 | No | Part | Yes | Part | Yes | No | 0 | |
| 488 | www.nhs.uk | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 489 | utexas.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 490 | magnific.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 491 | intuit.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 1 | ✓ |
| 492 | walmart.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 493 | ssrn.com | couldn’t read robots.txt (403) | ||||||||
| 494 | ed.gov | couldn’t read robots.txt (403) | ||||||||
| 495 | iana.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 496 | giphy.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 1 | |
| 499 | atlassian.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 500 | sec.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 503 | hhs.gov | couldn’t read robots.txt (403) | ||||||||
| 504 | mashable.com | 17 | No | No | No | No | Yes | No | 2 | |
| 506 | engadget.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 507 | aliexpress.com | 6 | No | Yes | No | Yes | No | No | 0 | |
| 508 | w3schools.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 509 | meetup.com | 1 | No | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 510 | lemonde.fr | 12 | Yes | Yes | No | Yes | No | No | 1 | |
| 511 | matterport.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 513 | ikea.com | 1 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 517 | justice.gov | 5 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 518 | www.gov.cn | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 519 | automattic.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 520 | teamviewer.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 521 | service.gov.uk | couldn’t read robots.txt (200) | ||||||||
| 522 | house.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 524 | spiegel.de | 12 | No | No | No | Yes | Yes | No | 2 | |
| 525 | freebsd.org | 0 | Part | Yes | Yes | Yes | Yes | Part | 0 | |
| 526 | pwc.com | couldn’t read robots.txt (403) | ||||||||
| 527 | laravel.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 531 | fedoraproject.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 532 | snapchat.com | 11 | No | Part | No | Part | No | No | 1 | |
| 533 | airbnb.com | 3 | No | Part | No | Part | Yes | Yes | 0 | |
| 534 | telegra.ph | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 535 | clevelandclinic.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 538 | npmjs.com | couldn’t read robots.txt (403) | ||||||||
| 539 | generatepress.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 541 | nodejs.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 542 | census.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 543 | web.dev | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 544 | siemens.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 546 | energy.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 547 | buzzfeed.com | 12 | No | Yes | No | No | No | No | 0 | |
| 548 | opensource.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 549 | vox.com | 15 | Yes | Yes | No | No | No | No | 2 | |
| 551 | box.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 552 | moodle.com | couldn’t read robots.txt (520) | ||||||||
| 554 | ok.ru | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 556 | disqus.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 557 | nyu.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 558 | cctv.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 559 | ionos.de | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 1 | |
| 561 | strato.de | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 562 | jquery.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 565 | marketwatch.com | 15 | Part | Part | No | No | No | No | 1 | |
| 566 | bigcartel.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 568 | inc.com | 10 | No | Part | No | No | Part | No | 0 | |
| 569 | kde.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 570 | variety.com | 12 | No | No | No | No | No | No | 0 | |
| 571 | douyin.com | 0 | Part | Part | Yes | Part | Part | Yes | 0 | |
| 572 | senate.gov | couldn’t read robots.txt (200) | ||||||||
| 573 | imf.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 575 | gallup.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 1 | |
| 576 | congress.gov | 18 | No | No | No | No | No | No | 1 | |
| 577 | cnil.fr | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 579 | ucsd.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 580 | medicalnewstoday.com | 7 | No | Yes | No | Yes | Yes | No | 0 | |
| 581 | nejm.org | 5 | No | Yes | Yes | No | No | No | 0 | |
| 582 | digg.com | 0 | Yes | Part | Part | Part | Part | Yes | 0 | ✓ |
| 583 | purdue.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 587 | constantcontact.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 588 | barnesandnoble.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 590 | steamcommunity.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 591 | hatena.ne.jp | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 593 | techradar.com | 5 | Yes | Yes | Yes | Yes | Yes | Yes | 1 | |
| 594 | perl.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 596 | java.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 597 | rakuten.co.jp | couldn’t read robots.txt (200) | ||||||||
| 598 | technologyreview.com | 14 | No | No | No | No | No | No | 0 | |
| 599 | fliphtml5.com | 5 | Yes | Yes | Yes | Yes | Yes | No | 2 | ✓ |
| 600 | photobucket.com | 1 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 601 | trello.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 602 | namepros.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 603 | politico.com | couldn’t read robots.txt (403) | ||||||||
| 604 | apachefriends.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 605 | amd.com | couldn’t read robots.txt | ||||||||
| 606 | cve.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 607 | matomo.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 609 | entrepreneur.com | 2 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 610 | qz.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 611 | scmp.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 612 | zend.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 613 | ufl.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 614 | so.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 615 | claude.ai | 4 | No | No | Yes | Yes | No | Yes | 0 | |
| 616 | slate.com | 10 | No | Yes | No | Yes | No | No | 0 | |
| 617 | drupal.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 618 | chinadaily.com.cn | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 620 | earthlink.net | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 621 | duckduckgo.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 622 | lww.com | couldn’t read robots.txt (200) | ||||||||
| 623 | udemy.com | couldn’t read robots.txt (403) | ||||||||
| 625 | duke.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 628 | usc.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 629 | vmware.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 630 | sap.com | 6 | Part | Part | Part | Part | Part | Part | 2 | |
| 631 | meta.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 1 | |
| 632 | rollingstone.com | 12 | No | No | No | No | No | No | 0 | |
| 633 | coe.int | couldn’t read robots.txt (403) | ||||||||
| 635 | pcmag.com | 17 | No | No | No | No | Yes | No | 2 | |
| 637 | shutterstock.com | 4 | No | Part | No | Part | Yes | No | 0 | |
| 638 | ifeng.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 639 | mail.ru | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 640 | semrush.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 641 | umd.edu | 2 | No | Yes | Yes | No | Yes | Yes | 0 | |
| 642 | ccc.de | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 643 | gutenberg.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 644 | qualtrics.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 646 | utoronto.ca | couldn’t read robots.txt (403) | ||||||||
| 647 | ny.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 648 | timeanddate.com | 5 | Yes | Yes | Yes | No | Yes | Yes | 0 | |
| 649 | ycombinator.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 651 | autodesk.com | couldn’t read robots.txt (403) | ||||||||
| 652 | ucl.ac.uk | couldn’t read robots.txt (403) | ||||||||
| 653 | hetzner.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 654 | fifa.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 656 | mixcloud.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 657 | zoho.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 658 | slashdot.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 659 | axios.com | 3 | Yes | Yes | Yes | Yes | Part | No | 0 | |
| 660 | fc2.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 661 | parallels.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 662 | woocommerce.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 664 | ey.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 665 | semanticscholar.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 667 | qcloud.com | 19 | No | No | No | No | No | No | 7 | |
| 668 | icann.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 669 | digitalocean.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 670 | newscientist.com | 19 | No | No | No | No | No | No | 1 | |
| 671 | ilo.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 672 | ucdavis.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 673 | cell.com | 11 | No | Yes | No | No | No | No | 0 | |
| 676 | unc.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 677 | indiegogo.com | couldn’t read robots.txt (403) | ||||||||
| 678 | livescience.com | 5 | Yes | Yes | Yes | Yes | Yes | Yes | 1 | |
| 680 | broadcom.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 681 | sfgate.com | 9 | Yes | Yes | No | No | Yes | No | 0 | |
| 683 | nikkei.com | 17 | No | No | No | No | No | No | 0 | |
| 684 | mediafire.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 686 | illinois.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 687 | getpocket.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 688 | espn.com | 5 | No | Yes | Yes | Yes | No | No | 0 | |
| 689 | venturebeat.com | 4 | Part | Yes | Part | Part | Part | No | 1 | |
| 690 | sky.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 691 | thesun.co.uk | 15 | Part | Part | No | No | No | No | 3 | |
| 694 | mirror.co.uk | 8 | No | No | No | No | Yes | No | 1 | |
| 696 | howstuffworks.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 697 | playstation.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 1 | ✓ |
| 699 | msu.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 700 | altervista.org | couldn’t read robots.txt (200) | ||||||||
| 701 | yandex.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 702 | lenovo.com | couldn’t read robots.txt (403) | ||||||||
| 703 | smh.com.au | 14 | No | No | No | No | No | No | 0 | |
| 704 | curl.se | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 705 | mastodon.social | 7 | No | Yes | No | Yes | No | No | 0 | |
| 706 | itch.io | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 707 | archives.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 708 | iea.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 709 | undp.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 710 | ubc.ca | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 714 | claude.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 715 | ethz.ch | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 716 | smithsonianmag.com | couldn’t read robots.txt (200) | ||||||||
| 717 | hollywoodreporter.com | 12 | No | No | No | No | No | No | 0 | |
| 718 | pypi.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 719 | iop.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 720 | caniuse.com | couldn’t read robots.txt (200) | ||||||||
| 721 | phys.org | 0 | Part | Yes | Part | Yes | Part | Part | 0 | ✓ |
| 722 | mi.com | couldn’t read robots.txt (403) | ||||||||
| 723 | proofpoint.com | 1 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 725 | hostinger.com | 1 | Part | Part | Part | Part | Part | Part | 0 | ✓ |
| 726 | medlineplus.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 727 | fcc.gov | couldn’t read robots.txt (403) | ||||||||
| 728 | ign.com | 17 | No | No | No | No | Yes | No | 2 | |
| 730 | wipo.int | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 731 | cpan.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 732 | news.cn | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 733 | colorado.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 734 | northwestern.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 735 | asu.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 736 | thehill.com | 9 | No | No | No | No | Yes | No | 0 | |
| 737 | chicagotribune.com | 18 | No | No | No | No | No | No | 4 | |
| 738 | history.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 739 | csdn.net | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 741 | tmall.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 744 | heise.de | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 745 | laracasts.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 746 | imagemagick.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 747 | haxx.se | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 749 | ohchr.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 751 | theglobeandmail.com | 18 | No | No | No | No | No | No | 2 | |
| 752 | calameo.com | 1 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 754 | thetimes.co.uk | 15 | Part | Part | No | No | No | No | 3 | |
| 755 | nike.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 1 | |
| 756 | garmin.com | 1 | Yes | Yes | Yes | No | Yes | Yes | 0 | |
| 757 | bizjournals.com | 9 | No | Yes | No | Yes | No | No | 2 | |
| 758 | abcnews.com | 5 | No | Yes | Yes | Yes | No | No | 0 | |
| 760 | treasury.gov | couldn’t read robots.txt (200) | ||||||||
| 762 | openldap.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 764 | arizona.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 765 | fontawesome.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 767 | kpmg.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 769 | accor.com | couldn’t read robots.txt (200) | ||||||||
| 770 | joomla.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 771 | man7.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 772 | ameblo.jp | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 1 | |
| 773 | fbi.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 774 | podbean.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 775 | virginia.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 777 | binance.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 778 | notion.so | 1 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 780 | france24.com | 13 | No | No | No | No | No | No | 0 | |
| 781 | deezer.com | 0 | Part | Part | Part | Part | Part | Part | 0 | |
| 782 | khanacademy.org | couldn’t read robots.txt (200) | ||||||||
| 784 | owasp.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 785 | lefigaro.fr | 11 | No | No | No | Yes | No | No | 0 | |
| 787 | heart.org | couldn’t read robots.txt (403) | ||||||||
| 788 | ihg.com | couldn’t read robots.txt (403) | ||||||||
| 789 | asus.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 790 | moz.com | 0 | Part | Yes | Yes | Yes | Yes | Yes | 0 | |
| 791 | about.me | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 792 | cygwin.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 794 | jetbrains.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 796 | uspto.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 798 | uber.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 799 | mastercard.com | couldn’t read robots.txt (403) | ||||||||
| 800 | dhs.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 1 | |
| 801 | uol.com.br | 15 | Yes | Yes | No | No | No | No | 2 | |
| 802 | weather.com | 5 | Yes | Yes | Yes | Yes | Yes | No | 0 | ✓ |
| 804 | dot.gov | couldn’t read robots.txt (403) | ||||||||
| 805 | evernote.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 806 | verisign.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 807 | cancer.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 809 | va.gov | 1 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 812 | kaspersky.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 813 | brookings.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 815 | ed.ac.uk | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 816 | accenture.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 817 | webex.com | couldn’t read robots.txt (403) | ||||||||
| 818 | wa.gov | 19 | No | No | No | No | No | No | 7 | |
| 819 | mpg.de | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 822 | people.com | 15 | Part | Part | No | No | No | No | 2 | |
| 823 | bitbucket.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 824 | globo.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 825 | tamu.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 826 | redis.io | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 827 | texas.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 828 | instructables.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 830 | esa.int | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 831 | nobelprize.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 832 | figma.com | 10 | No | No | No | No | No | No | 0 | |
| 834 | freedesktop.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 835 | gmail.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 836 | rutgers.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 837 | novell.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 838 | alipay.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 839 | usembassy.gov | couldn’t read robots.txt (403) | ||||||||
| 840 | unep.org | 9 | No | Yes | No | Yes | No | No | 2 | |
| 841 | perplexity.ai | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 842 | ahrefs.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 844 | olympics.com | couldn’t read robots.txt | ||||||||
| 845 | wireshark.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 846 | wpengine.com | couldn’t read robots.txt (502) | ||||||||
| 847 | wikisource.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 849 | gesetze-im-internet.de | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 850 | bitnami.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 851 | osu.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 853 | bu.edu | 1 | Part | Yes | Part | Yes | Yes | Part | 0 | |
| 854 | envato.com | 1 | Yes | Yes | Yes | Yes | Yes | No | 0 | |
| 855 | crunchbase.com | 7 | No | Yes | No | Yes | No | No | 0 | |
| 856 | speedtest.net | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 857 | usa.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 859 | 1688.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 1 | |
| 860 | vatican.va | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 862 | army.mil | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 863 | www.gov.br | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 864 | centos.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 866 | dol.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 867 | cancer.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 868 | nasdaq.com | couldn’t read robots.txt | ||||||||
| 869 | greenpeace.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 870 | express.co.uk | 7 | No | No | No | No | Yes | No | 0 | |
| 871 | ipcc.ch | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 873 | utah.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 874 | fiverr.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 875 | imageshack.us | 10 | No | Part | No | Part | No | No | 1 | |
| 877 | dreamhost.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 878 | xbox.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 880 | ea.com | 2 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 881 | federalregister.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 882 | rutube.ru | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 883 | gumroad.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 885 | douban.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 886 | nintendo.com | 7 | No | Yes | No | Yes | No | No | 1 | |
| 887 | lua.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 888 | toutiao.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 889 | go.dev | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 890 | nhk.or.jp | 10 | No | No | No | No | No | No | 2 | |
| 891 | mediawiki.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 893 | investing.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 894 | hrw.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 895 | waze.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 896 | last.fm | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 897 | hyatt.com | couldn’t read robots.txt (403) | ||||||||
| 898 | nyc.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 899 | prezi.com | 6 | No | Part | No | Yes | Yes | No | 0 | ✓ |
| 900 | gimp.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 901 | panasonic.com | couldn’t read robots.txt (403) | ||||||||
| 902 | heyzine.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 904 | cisa.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 1 | |
| 905 | libsyn.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 906 | osha.gov | couldn’t read robots.txt (403) | ||||||||
| 907 | ncsu.edu | couldn’t read robots.txt (202) | ||||||||
| 908 | lego.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 910 | isc.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 911 | welt.de | 9 | Yes | Yes | Yes | No | Yes | No | 2 | |
| 912 | navy.mil | couldn’t read robots.txt (403) | ||||||||
| 913 | amap.com | 9 | Part | Part | No | Part | Part | Part | 3 | ✓ |
| 914 | sfu.ca | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 915 | lifehacker.com | 17 | No | No | No | No | Yes | No | 2 | |
| 917 | ucsb.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 918 | colorlib.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 919 | vkontakte.ru | 19 | No | No | No | No | No | No | 7 | |
| 920 | nba.com | 8 | No | Yes | No | No | No | No | 0 | |
| 922 | tesla.com | couldn’t read robots.txt (403) | ||||||||
| 923 | zimbra.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 924 | theregister.com | 14 | No | Yes | No | Yes | No | No | 1 | |
| 925 | caltech.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 926 | zapier.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 927 | domaintools.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 928 | news.com.au | 18 | No | No | No | No | No | No | 3 | |
| 930 | mlb.com | couldn’t read robots.txt (403) | ||||||||
| 931 | americanexpress.com | couldn’t read robots.txt | ||||||||
| 932 | hootsuite.com | 0 | Part | Part | Part | Part | Part | Yes | 0 | ✓ |
| 933 | target.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 934 | fool.com | 4 | No | Yes | No | Yes | Yes | No | 0 | ✓ |
| 935 | weather.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 936 | iqiyi.com | 4 | No | Yes | No | No | Yes | Yes | 1 | |
| 937 | ispconfig.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 938 | securityfocus.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 939 | max.ru | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 940 | wikibooks.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 941 | tableau.com | couldn’t read robots.txt (403) | ||||||||
| 943 | ec-lyon.fr | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 944 | nymag.com | 15 | Yes | Yes | No | No | No | No | 2 | |
| 945 | perl.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 946 | narod.ru | couldn’t read robots.txt | ||||||||
| 947 | digitaltrends.com | 6 | No | Part | No | Part | No | No | 2 | |
| 948 | themegrill.com | couldn’t read robots.txt (520) | ||||||||
| 950 | amnesty.org | 2 | Yes | Yes | Yes | No | Yes | No | 1 | |
| 951 | ko-fi.com | 9 | No | Yes | No | Yes | No | No | 2 | |
| 952 | vogue.com | 16 | Yes | Yes | No | No | No | No | 1 | |
| 954 | ourworldindata.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 956 | georgetown.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 958 | consumerreports.org | 5 | No | Yes | Yes | Yes | No | No | 0 | |
| 959 | tradingview.com | 0 | Yes | Part | Part | Part | Part | Yes | 0 | |
| 960 | aepd.es | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 961 | metmuseum.org | couldn’t read robots.txt (429) | ||||||||
| 962 | yumpu.com | 11 | No | Yes | No | No | No | No | 1 | |
| 963 | iheart.com | 7 | No | Part | No | Part | No | No | 0 | |
| 964 | searchengineland.com | couldn’t read robots.txt (403) | ||||||||
| 965 | hopkinsmedicine.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 967 | legislation.gov.uk | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 968 | upwork.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 1 | ✓ |
| 969 | att.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 970 | metro.co.uk | 19 | No | No | No | No | No | No | 1 | |
| 971 | nokia.com | couldn’t read robots.txt (403) | ||||||||
| 972 | today.com | 17 | No | No | No | No | No | No | 1 | |
| 973 | medscape.com | 6 | No | Yes | No | No | No | No | 0 | |
| 974 | bell-labs.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 975 | justia.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 976 | flaticon.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 977 | clinicaltrials.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 978 | xkcd.com | 1 | No | Yes | Yes | Yes | Yes | Yes | 0 | |
| 979 | ning.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 980 | mcgill.ca | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 982 | tomshardware.com | 5 | Yes | Yes | Yes | Yes | Yes | Yes | 1 | |
| 984 | xiaohongshu.com | 19 | No | No | No | No | No | No | 7 | |
| 985 | pitt.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 986 | zeit.de | 18 | No | No | No | No | No | No | 1 | |
| 987 | codeberg.org | 12 | No | No | No | No | No | No | 1 | |
| 988 | feedly.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 989 | boston.com | 8 | No | Yes | No | Yes | No | No | 0 | |
| 990 | salon.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 991 | brandbucket.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 992 | nydailynews.com | 18 | No | No | No | No | No | No | 4 | |
| 993 | cia.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 1 | |
| 994 | mariadb.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 995 | hbs.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 996 | chron.com | 10 | Yes | Yes | No | No | No | No | 0 | |
| 997 | trendmicro.com | 0 | Yes | Part | Part | Part | Part | Part | 0 | ✓ |
| 998 | eurekalert.org | 1 | Yes | Yes | Yes | Yes | Yes | Yes | 2 | |
| 999 | istockphoto.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1000 | sueddeutsche.de | 5 | No | Yes | No | No | No | Yes | 0 | |
| 1001 | ndtv.com | couldn’t read robots.txt (403) | ||||||||
| 1002 | federalreserve.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1004 | zillow.com | 4 | Yes | Yes | Part | Yes | Yes | Yes | 0 | |
| 1005 | routledge.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1006 | padlet.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 1007 | mhlw.go.jp | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1008 | fedex.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1011 | thedailybeast.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1012 | lonelyplanet.com | 1 | No | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1013 | foodnetwork.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1014 | rsc.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1015 | pcworld.com | 6 | No | Yes | No | Yes | Yes | No | 0 | |
| 1016 | pastebin.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1017 | videolan.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1018 | verizon.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1020 | lse.ac.uk | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1021 | urbandictionary.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1023 | hindustantimes.com | 1 | Yes | Yes | Yes | Yes | Part | Yes | 0 | ✓ |
| 1024 | dzen.ru | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1025 | dev.to | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 1026 | 500px.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1027 | sba.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1028 | zenodo.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1029 | thefreedictionary.com | 15 | No | No | No | No | No | No | 1 | |
| 1030 | brown.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1031 | mega.nz | 19 | No | No | No | No | No | No | 7 | |
| 1032 | lg.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 1033 | thenextweb.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1034 | tufts.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1035 | houzz.com | 0 | Part | Part | Part | Part | Part | Yes | 0 | |
| 1036 | usps.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1037 | gamespot.com | 9 | No | Yes | No | Yes | No | No | 2 | |
| 1038 | hawaii.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1039 | wunderground.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1040 | sony.com | couldn’t read robots.txt (403) | ||||||||
| 1042 | hpe.com | couldn’t read robots.txt | ||||||||
| 1044 | squareup.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 1046 | prestashop.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1047 | deepseek.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1048 | space.com | 5 | Yes | Yes | Yes | Yes | Yes | Yes | 1 | |
| 1050 | ups.com | couldn’t read robots.txt (200) | ||||||||
| 1051 | 360.cn | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1053 | zlib.net | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1054 | discogs.com | couldn’t read robots.txt (403) | ||||||||
| 1055 | bitly.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1056 | deadline.com | 12 | No | No | No | No | No | No | 0 | |
| 1057 | cwi.nl | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1058 | gatech.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1059 | ntp.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1060 | apachelounge.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1061 | squid-cache.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1062 | billboard.com | 12 | No | No | No | No | No | No | 0 | |
| 1063 | voanews.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1064 | vt.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1066 | ssa.gov | couldn’t read robots.txt (403) | ||||||||
| 1067 | airtable.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1069 | grandviewresearch.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 1070 | dhl.com | couldn’t read robots.txt | ||||||||
| 1071 | dictionary.com | 3 | No | Yes | Yes | Yes | Yes | No | 0 | |
| 1072 | over-blog.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1073 | epicgames.com | couldn’t read robots.txt (403) | ||||||||
| 1074 | computerworld.com | 6 | No | Yes | No | Yes | Yes | No | 0 | |
| 1075 | unity.com | 1 | Yes | Yes | Yes | Yes | Yes | Yes | 1 | ✓ |
| 1077 | gitea.com | 6 | No | No | No | No | Yes | No | 0 | |
| 1078 | faz.net | 16 | No | No | No | No | No | No | 1 | |
| 1079 | spglobal.com | couldn’t read robots.txt (403) | ||||||||
| 1080 | sketchfab.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1083 | unl.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1085 | jpost.com | 0 | Part | Part | Yes | Yes | Yes | Yes | 0 | |
| 1086 | bostonglobe.com | 9 | No | Yes | No | Yes | No | No | 0 | |
| 1087 | unfccc.int | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1088 | aarp.org | 2 | Yes | Yes | Yes | Yes | Yes | No | 1 | |
| 1091 | edx.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 1092 | nber.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1093 | thestar.com | 19 | No | No | No | No | No | No | 1 | |
| 1094 | one.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 1095 | vanityfair.com | 16 | Yes | Yes | No | No | No | No | 1 | |
| 1097 | fraunhofer.de | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1098 | sophos.com | 1 | Yes | Yes | Yes | Yes | Yes | Yes | 1 | ✓ |
| 1100 | irishtimes.com | 2 | No | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1103 | worldcat.org | 4 | No | Yes | Yes | Yes | No | No | 0 | |
| 1104 | timeout.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1105 | repubblica.it | 9 | No | Yes | Yes | No | No | No | 2 | ✓ |
| 1106 | boutell.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1107 | thawte.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 1108 | site.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1109 | asana.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1111 | ghost.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1113 | bluehost.com | couldn’t read robots.txt (403) | ||||||||
| 1115 | strava.com | 4 | No | Yes | No | Yes | No | Yes | 0 | |
| 1116 | eia.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1117 | asahi.com | 15 | No | No | No | No | No | No | 0 | |
| 1118 | libpng.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1120 | penguinrandomhouse.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1122 | grammarly.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1123 | archlinux.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1124 | foursquare.com | 2 | No | Yes | Yes | Yes | Yes | Part | 0 | |
| 1125 | phpbb.com | couldn’t read robots.txt (403) | ||||||||
| 1126 | indianexpress.com | couldn’t read robots.txt (403) | ||||||||
| 1128 | hotjar.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1129 | note.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1131 | fema.gov | 1 | Yes | Yes | Yes | Yes | Yes | Yes | 1 | |
| 1132 | thomsonreuters.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1133 | readthedocs.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1134 | gohugo.io | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1136 | wampserver.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1137 | japantimes.co.jp | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1138 | gitee.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 1139 | thehindu.com | 10 | No | No | No | No | Yes | No | 0 | |
| 1141 | unhcr.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1143 | gnome.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1145 | uga.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1146 | t-online.de | 7 | No | Yes | No | No | Part | No | 0 | |
| 1147 | memcached.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1148 | nationalacademies.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1149 | buzzsprout.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1150 | accuweather.com | couldn’t read robots.txt (403) | ||||||||
| 1152 | timesofisrael.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 1 | |
| 1153 | wetransfer.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1154 | mapquest.com | 19 | No | No | No | No | No | No | 4 | |
| 1156 | uwaterloo.ca | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1157 | glassdoor.com | 9 | No | Yes | No | Yes | No | No | 1 | |
| 1158 | pcre.org | couldn’t read robots.txt | ||||||||
| 1159 | globalnews.ca | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1160 | umass.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1161 | smugmug.com | 19 | No | No | No | No | No | No | 3 | |
| 1162 | duolingo.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1163 | jst.go.jp | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1164 | arduino.cc | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 1165 | oregonstate.edu | 9 | No | Yes | No | Yes | No | No | 2 | |
| 1166 | directadmin.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1168 | redbull.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 1170 | bcg.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1171 | wufoo.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1172 | snopes.com | 10 | No | Yes | No | No | Yes | No | 1 | |
| 1174 | rochester.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1176 | klarna.com | couldn’t read robots.txt (202) | ||||||||
| 1177 | coindesk.com | 9 | No | Yes | No | No | No | No | 0 | |
| 1178 | notion.com | 2 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 1180 | buymeacoffee.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1181 | findlaw.com | couldn’t read robots.txt (403) | ||||||||
| 1182 | gzip.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1184 | boe.es | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1185 | elmundo.es | 18 | No | No | No | No | No | No | 0 | |
| 1187 | deepl.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1188 | logitech.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 1192 | kadencewp.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 1 | |
| 1193 | searchenginejournal.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 1194 | proquest.com | 2 | No | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1195 | seattletimes.com | couldn’t read robots.txt (202) | ||||||||
| 1196 | britishcouncil.org | couldn’t read robots.txt | ||||||||
| 1197 | brevo.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1199 | freetype.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1200 | rbc.ru | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1201 | openoffice.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1202 | biblegateway.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1203 | ria.ru | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1204 | namebright.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1205 | faa.gov | couldn’t read robots.txt (403) | ||||||||
| 1206 | cuny.edu | 19 | No | No | No | No | No | No | 3 | |
| 1207 | discovery.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1208 | sqlite.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1209 | cern.ch | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1210 | rottentomatoes.com | 0 | Yes | Yes | Yes | Yes | Part | Yes | 0 | |
| 1212 | uefa.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1213 | unam.mx | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1214 | virustotal.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1215 | iastate.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1216 | raspberrypi.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1217 | ctvnews.ca | 5 | No | No | Yes | No | Yes | No | 0 | |
| 1218 | habr.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1219 | china.com.cn | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1220 | oxfordjournals.org | 19 | No | No | No | No | No | No | 7 | |
| 1221 | chromium.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1222 | codepen.io | 9 | No | Yes | No | Yes | No | No | 2 | |
| 1223 | disney.com | 2 | No | Yes | Yes | Yes | No | Yes | 0 | |
| 1224 | redcross.org | couldn’t read robots.txt (403) | ||||||||
| 1226 | answers.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1227 | unicode.org | 1 | Yes | Yes | Yes | Yes | Yes | No | 2 | |
| 1229 | buffer.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 1230 | blender.org | 9 | No | Yes | No | Yes | No | No | 2 | |
| 1231 | macrumors.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1232 | modsecurity.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1233 | hardened-php.net | couldn’t read robots.txt | ||||||||
| 1235 | enlightenment.org | 19 | No | No | No | No | No | No | 1 | |
| 1236 | cnrs.fr | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1237 | defense.gov | couldn’t read robots.txt (403) | ||||||||
| 1238 | uiowa.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1240 | bild.de | 9 | Yes | Yes | Yes | No | Yes | No | 2 | |
| 1241 | filezilla-project.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1243 | eclipse.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1244 | standard.co.uk | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1245 | gettyimages.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1246 | corriere.it | 17 | No | No | No | No | No | No | 3 | |
| 1247 | philips.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1248 | foreignpolicy.com | 13 | No | Yes | No | No | No | No | 1 | |
| 1250 | kakao.com | 19 | No | No | No | No | No | No | 7 | |
| 1254 | thebalancemoney.com | 15 | Part | Part | No | No | No | No | 2 | |
| 1256 | af.mil | couldn’t read robots.txt (403) | ||||||||
| 1257 | uscourts.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1258 | livechat.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 1259 | ahajournals.org | 8 | No | Yes | No | Yes | No | No | 0 | |
| 1261 | nsf.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1262 | biglobe.ne.jp | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1263 | se.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1265 | howtogeek.com | couldn’t read robots.txt | ||||||||
| 1266 | govinfo.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1267 | messenger.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1268 | gwu.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1269 | anu.edu.au | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1271 | ap.org | 9 | No | Yes | No | Yes | No | No | 2 | |
| 1272 | torproject.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1273 | homedepot.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1274 | bayern.de | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1276 | service-public.fr | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1278 | producthunt.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 1279 | goethe.de | 1 | Yes | Yes | Yes | Yes | Yes | Yes | 1 | |
| 1280 | bleepingcomputer.com | 3 | Yes | Yes | No | Yes | Yes | No | 0 | |
| 1285 | craigslist.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1287 | nationalarchives.gov.uk | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1288 | foxbusiness.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1289 | cms.gov | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1291 | ijg.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1295 | visa.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1296 | unsw.edu.au | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1298 | udel.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1299 | serverwatch.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 1301 | html5up.net | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1302 | wikidot.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1303 | icloud.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1304 | uu.nl | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1305 | business-standard.com | couldn’t read robots.txt (403) | ||||||||
| 1306 | wassenaar.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1307 | monday.com | 0 | Yes | Part | Part | Part | Part | Yes | 0 | ✓ |
| 1309 | verywellmind.com | 15 | Part | Part | No | No | No | No | 2 | |
| 1312 | prweb.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1313 | gandi.net | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 1 | |
| 1316 | scopus.com | couldn’t read robots.txt (403) | ||||||||
| 1317 | coinmarketcap.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
| 1318 | github.blog | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1320 | ama-assn.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1321 | zdf.de | 0 | Part | Yes | Yes | Yes | Part | Part | 0 | |
| 1324 | adweek.com | 1 | Yes | Yes | Yes | Yes | Yes | Yes | 1 | |
| 1325 | colostate.edu | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1328 | freetds.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1329 | hwg.org | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1330 | pdflib.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | |
| 1331 | straitstimes.com | 1 | Yes | Yes | Yes | Yes | No | Yes | 0 | |
| 1332 | rfi.fr | 13 | No | No | No | No | No | No | 0 | |
| 1333 | tagesschau.de | 7 | No | Yes | No | Yes | No | No | 0 | |
| 1334 | coinbase.com | 0 | Yes | Yes | Yes | Yes | Yes | Yes | 0 | ✓ |
How this is measured
- The sites. The 1,000 highest-ranked websites in the Majestic Million (by Majestic, CC BY 3.0), which ranks domains by how many networks link to them. We keep one domain per website and leave out CDNs, APIs, link shorteners, hosting platforms’ customer domains, country copies of the same brand and domains that redirect elsewhere.
- The check. Once a week we request
https://<site>/robots.txt(falling back towww.when the bare domain doesn’t serve) and/llms.txt, ascanaicrawl/1.0with a contact address. No other page is fetched. - Blocked means the group of rules a bot follows disallows the whole site (
Disallow: /), applied the way crawlers do: the group naming the bot wins, thenUser-agent: *. Part means a group naming the bot disallows some paths. A site with no robots.txt (a 404) blocks nothing. - Unknown. When our request is refused, challenged, answered with a web page or fails, the site is left out of the figures: bot protection often turns our checker away while real crawlers get the file.
The data is free to use; please link to this page. Download it as JSON, or open any site’s report for the exact lines.