Who blocks AI · #648 in the Majestic Million
robots.txt · checked 9 Nov 2026
Does timeanddate.com block AI crawlers?
Yes. timeanddate.com’s robots.txt blocks 5 of the 19 AI crawlers we track, including PerplexityBot, Meta-ExternalAgent, Applebot-Extended and 2 more. Search engines such as Googlebot and Bingbot may crawl it.
GPTBotAllowedOAI-SearchBotAllowedClaudeBotAllowedPerplexityBotBlockedGoogle-ExtendedAllowedCCBotAllowed- robots.txt
- www.timeanddate.com/robots.txt · 253 lines
- llms.txt
- none
- Checked
- 9 November 2026, as canaicrawl. We check again every week.
Every bot, and the line that decides
Search engines
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
Googlebot | Allowed | no rule matches the home page | |
Bingbot | Microsoft | Allowed | no rule matches the home page |
DuckDuckBot | DuckDuckGo | Allowed | no rule matches the home page |
Applebot | Apple | Allowed | no rule matches the home page |
PetalBot | Huawei | Allowed | no rule matches the home page |
YandexBot | Yandex | Allowed | no rule matches the home page |
Baiduspider | Baidu | Allowed | no rule matches the home page |
AI search
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
OAI-SearchBot | OpenAI | Allowed | no rule matches the home page |
PerplexityBot | Perplexity | Blocked | line 21 · Disallow: / under User-agent: PerplexityBot |
Claude-SearchBot | Anthropic | Allowed | no rule matches the home page |
DuckAssistBot | DuckDuckGo | Allowed | no rule matches the home page |
Amazonbot | Amazon | Blocked | line 27 · Disallow: / under User-agent: Amazonbot |
YouBot | You.com | Allowed | no rule matches the home page |
AI training
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
GPTBot | OpenAI | Allowed | no rule matches the home page |
ClaudeBot | Anthropic | Allowed | no rule matches the home page |
Google-Extended | Allowed | no rule matches the home page | |
CCBot | Common Crawl | Allowed | no rule matches the home page |
Applebot-Extended | Apple | Blocked | line 30 · Disallow: / under User-agent: Applebot-Extended |
GoogleOther | Allowed | no rule matches the home page | |
Meta-ExternalAgent | Meta | Blocked | line 33 · Disallow: / under User-agent: Meta-ExternalAgent |
Bytespider | ByteDance | Blocked | line 24 · Disallow: / under User-agent: Bytespider |
User-triggered fetchers
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
ChatGPT-User | OpenAI | Allowed | no rule matches the home page |
Claude-User | Anthropic | Allowed | no rule matches the home page |
Perplexity-User | Perplexity | Allowed | no rule matches the home page |
MistralAI-User | Mistral | Allowed | no rule matches the home page |
Meta-ExternalFetcher | Meta | Allowed | no rule matches the home page |
SEO tools
| Bot | Operator | robots.txt | Deciding line |
|---|---|---|---|
AhrefsBot | Ahrefs | Partly blocked | line 41 · Disallow: /*? line 42 · Disallow: /information/privacy.html and 1 more |
SemrushBot | Semrush | Allowed | no rule matches the home page |
MJ12bot | Majestic | Allowed | no rule matches the home page |
DotBot | Moz | Partly blocked | line 46 · Disallow: /*? line 47 · Disallow: /information/privacy.html and 1 more |
View www.timeanddate.com/robots.txt · 253 lines
1# http://web.nexor.co.uk/mak/doc/robots/norobots.html
2#
3# internal note, this file is in git now!
4
5User-agent: MSIECrawler
6Disallow: /
7
8User-agent: PortalBSpider
9Disallow: /
10
11User-agent: Mediapartners-Google*
12Disallow:
13
14User-agent: AmazonAdBot
15Disallow:
16
17User-agent: Yahoo Pipes 1.0
18Disallow: /
19
20User-agent: PerplexityBot
21Disallow: /
22
23User-agent: Bytespider
24Disallow: /
25
26User-agent: Amazonbot
27Disallow: /
28
29User-agent: Applebot-Extended
30Disallow: /
31
32User-agent: Meta-ExternalAgent
33Disallow: /
34
35User-agent: KeenableBot
36Disallow: /
37# SITE-SPECIFIC STARTS HERE
38
39# disallow any urls with ? in
40User-Agent: AhrefsBot
41Disallow: /*?
42Disallow: /information/privacy.html
43Disallow: /information/terms-conditions.html
44
45User-Agent: dotbot
46Disallow: /*?
47Disallow: /information/privacy.html
48Disallow: /information/terms-conditions.html
49
50User-agent: tadcrawler
51Disallow: /.pbs/
52Disallow: /calendar/*?year*&
53Disallow: /eclipse/globe/*?
54Disallow: /eclipse/map/*?
55Allow: /information/privacy.html
56Allow: /information/terms-conditions.html
57Allow: /eclipse/globe/202
58Allow: /eclipse/map/202
59Allow: /calendar/*?year=2026
60Allow: /calendar/*?year=2027
61Allow: /calendar/*?year=2028
62Allow: /calendar/*?year=2029
63Allow: /calendar/*?year=2030
64Allow: /astronomy/moon/distance.html?year=2026
65Allow: /astronomy/moon/distance.html?year=2027
66Allow: /astronomy/moon/distance.html?year=2028
67Allow: /worldclock/clockchange.html?*year=2026
68Allow: /worldclock/clockchange.html?*year=2027
69Allow: /worldclock/clockchange.html?*year=2028
70Disallow: /*y1=
71Disallow: /*nojs=
72Disallow: /worldclock/fixedform.html?
73Disallow: /worldclock/fixedtime.html?
74Disallow: /worldclock/sunearth.html?
75Disallow: /worldclock/sunearth.html?iso
76Disallow: /worldclock/sunearth.html?earth
77Disallow: /astronomy/moon/light.html?
78Disallow: /worldclock/timezone.html?*syear
79Disallow: /scripts/goweather.php
80Disallow: /scripts/ics.php
81Disallow: /worldclock/dialing.html?
82Disallow: /worldclock/custom.html?nurlid=
83Disallow: /worldclock/custom.html?cities=
84Disallow: /calendar/*?year
85Disallow: /calendar/seasons.html?
86Disallow: /clocks/freecountdown.html?n=
87Disallow: /clocks/countdown.html?n=
88Disallow: /countdown/create?
89Disallow: /countdown/eclipse?
90Disallow: /countdown/generic?
91Disallow: /countdown/to?
92Disallow: /astronomy/moon/distance.html?year
93Disallow: /eclipse/globe/
94Disallow: /eclipse/map/
95Disallow: /moon/*?
96Disallow: /sun/*?
97Disallow: /time/change/@*
98Disallow: /time/change/*?
99Disallow: /time/zone/@*
100Disallow: /time/zone/*?
101Disallow: /worldclock/astronomy.html?*year=
102Disallow: /worldclock/clockchange.html?
103Disallow: /worldclock/converter.html?
104Disallow: /worldclock/results.html
105Disallow: /scripts/tzq.php
106
107User-agent: ScoutJet
108Disallow: /worldclock/distanceresult.html
109Disallow: /worldclock/distanceresult.html*
110Disallow: /information/privacy.html
111Disallow: /information/terms-conditions.html
112
113User-agent: *
114Disallow: /.pbs/
115Allow: /calendar/create.html?*year=2026
116Allow: /calendar/create.html?*year=2027
117Allow: /calendar/create.html?*year=2028
118Disallow: /adverts/
119Disallow: /advgifs/
120Disallow: /astronomy/moon/light.html?iso
121Disallow: /calendar/basic.html?year
122Disallow: /calendar/create.html?*df=
123Disallow: /calendar/create.html?*year
124Disallow: /calendar/custom.html?year
125Disallow: /calendar/custommenu.html?year
126Disallow: /calendar//index.html
127Disallow: /calendar/monthly.html?year=1
128Disallow: /calendar/monthly.html?year=21
129Disallow: /calendar/moonphases.html?year=&
130Disallow: /calendar/moonphases.html?year=10
131Disallow: /calendar/moonphases.html?year=11
132Disallow: /calendar/moonphases.html?year=12
133Disallow: /calendar/moonphases.html?year=13
134Disallow: /calendar/moonphases.html?year=14
135Disallow: /calendar/moonphases.html?year=15
136Disallow: /calendar/moonphases.html?year=16
137Disallow: /calendar/moonphases.html?year=17
138Disallow: /calendar/moonphases.html?year=18
139Disallow: /calendar/moonphases.html?year=2006&
140Disallow: /calendar/moonphases.html?year=2007&
141Disallow: /calendar/moonphases.html?year=2008&
142Disallow: /calendar/moonphases.html?year=21
143Disallow: /calendar/moonphases.html?year=22
144Disallow: /calendar/moonphases.html?year=23
145Disallow: /calendar/moonphases.html?year=24
146Disallow: /calendar/moonphases.html?year=25
147Disallow: /calendar/moonphases.html?year=26
148Disallow: /calendar/moonphases.html?year=27
149Disallow: /calendar/moonphases.html?year=28
150Disallow: /calendar/moonphases.html?year=29
151Disallow: /calendar/moonphases.html?year=3
152Disallow: /calendar/moonphases.html?year=4
153Disallow: /calendar/moonphases.html?year=5
154Disallow: /calendar/moonphases.html?year=6
155Disallow: /calendar/moonphases.html?year=7
156Disallow: /calendar/moonphases.html?year=8
157Disallow: /calendar/moonphases.html?year=9
158Disallow: /calendar/print.html
159Disallow: /calendar/repeating.html?year=1
160Disallow: /calendar/repeating.html?year=21
161Disallow: /calendar/repeating.html?year=22
162Disallow: /calendar/repeating.html?year=23
163Disallow: /calendar/repeating.html?year=23
164Disallow: /calendar/repeating.html?year=24
165Disallow: /calendar/repeating.html?year=25
166Disallow: /calendar/repeating.html?year=3
167Disallow: /calendar/?year=10
168Disallow: /calendar/?year=11
169Disallow: /calendar/?year=12
170Disallow: /calendar/?year=13
171Disallow: /calendar/?year=14
172Disallow: /calendar/?year=15
173Disallow: /calendar/?year=16
174Disallow: /calendar/?year=17
175Disallow: /calendar/?year=18
176Disallow: /calendar/?year=21
177Disallow: /calendar/?year=22
178Disallow: /calendar/?year=23
179Disallow: /calendar/?year=24
180Disallow: /calendar/?year=25
181Disallow: /calendar/?year=26
182Disallow: /calendar/?year=27
183Disallow: /calendar/?year=28
184Disallow: /calendar/?year=29
185Disallow: /calendar/?year=3
186Disallow: /clocks/free.html?n
187Disallow: /clocks/onlyforusebyconfiguration.php
188Disallow: /counters/customcounter.html?day
189Disallow: /counters/fullscreen.html?
190Disallow: /counters/millennium.html?p0
191Disallow: /counters/year2000a.html?p0
192Disallow: /counters/year2000.html?p0
193Disallow: /createshort.html
194Disallow: /custom/created.html
195Disallow: /custom/activate.html?
196Disallow: /custom/forgot.html?
197Disallow: /date/dateadded.html?
198Disallow: /date/dateadd.html?
199Disallow: /date/durationed.html?
200Disallow: /date/durationresult.html?
201Disallow: /date/timeadd.html?
202Disallow: /date/timeduration.html?
203Disallow: /date/weekdayadd.html?
204Disallow: /date/workdays.html?
205Disallow: /eclipse/in/*?starty=
206Disallow: /eclipse/in/*?iso
207Allow: /eclipse/in/*?iso=202
208Disallow: /gfx/stock/
209Disallow: /information/feedback.html?
210Disallow: /information/privacy.html
211Disallow: /information/terms-conditions.html
212Disallow: /moon/*year=
213Disallow: /scripts/accountextlogin.php
214Disallow: /scripts/distanceserver.php
215Disallow: /scripts/moon.php
216Disallow: /scripts/savecustom.php
217Disallow: /scripts/savepersonal.php
218Disallow: /scripts/wquery.php
219Disallow: /search/results.html
220Disallow: /sun/*year=
221Disallow: /time/map/*_escaped_fragment_
222Disallow: /weather/*?hd=*
223Disallow: /weather/*/historic?hd=*
224Disallow: /weather/*/hourly?hd=*
225Disallow: /weather/*sort*
226Disallow: /worldclock/astronomy.html?*year=
227Disallow: /worldclock/custom.html?*n=
228Disallow: /worldclock/custom.html?*sort*
229Disallow: /worldclock/custom.html?n=
230Disallow: /worldclock/custom.html?query=
231Disallow: /worldclock/difference.html?*continent
232Disallow: /worldclock/difference.html?*sort*
233Disallow: /worldclock/distance.html?p1
234Disallow: /worldclock/distanceresult.html?
235Disallow: /worldclock/distances.html?
236Disallow: /worldclock/fixedform.html?p1
237Disallow: /worldclock/fixedtime.html?*continent*
238Disallow: /worldclock/fixedtime.html?*sort*
239Disallow: /worldclock/fixedtime.html?day
240Disallow: /worldclock/fixedtime.html?iso=200
241Disallow: /worldclock/fixedtime.html?iso=201
242Disallow: /worldclock/fixedtime.html?month
243Disallow: /worldclock/full.html/city.html
244Disallow: /worldclock/fullscreen.html?
245Disallow: /worldclock/meeting.html?p1
246Disallow: /worldclock/meetingdetails.html?
247Disallow: /worldclock/meetingtime.html?
248Disallow: /worldclock/personal.html?*continent*
249Disallow: /worldclock/personal.html?*sort*
250Disallow: /worldclock/setpersonal.cgi
251Disallow: /worldclock/sunearth.html?
252Disallow: /worldclock/sunearth.html?iso
253
What do the bots actually get?
This page reads robots.txt, which states what timeanddate.com asks of crawlers. A firewall or CDN can still refuse a bot that robots.txt allows. The live check requests the page as each bot and shows the status it gets.
Run a live check on timeanddate.com Test a path in the robots.txt tester
History
- 19 October 2026: first checked.
Questions
Does timeanddate.com block GPTBot?
No. timeanddate.com’s robots.txt lets GPTBot crawl the site. GPTBot is OpenAI’s crawler for training data; blocking it doesn’t affect ChatGPT search, which uses OAI-SearchBot. Checked 9 November 2026.
Does timeanddate.com block ClaudeBot?
No. timeanddate.com’s robots.txt lets ClaudeBot crawl the site. ClaudeBot is Anthropic’s crawler for training data. Checked 9 November 2026.
Does timeanddate.com block Google-Extended?
No. timeanddate.com’s robots.txt lets Google-Extended crawl the site. Google-Extended is the token that controls whether Google may use pages for Gemini training and grounding; it doesn’t affect Google Search. Checked 9 November 2026.
Does timeanddate.com block PerplexityBot?
Yes. timeanddate.com’s robots.txt disallows PerplexityBot from the whole site (line 21: Disallow: /). PerplexityBot is the crawler that indexes pages for Perplexity’s answers. Checked 9 November 2026.
Can ChatGPT search find timeanddate.com?
Yes. timeanddate.com’s robots.txt lets OAI-SearchBot crawl the site. OAI-SearchBot is the crawler behind ChatGPT search; blocking it keeps the site out of ChatGPT’s search answers. Checked 9 November 2026.