crawlex 1.0.6

Stealth crawler with Chrome-perfect TLS/H2 fingerprint, render pool, hooks, persistent queue
Documentation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
# crawlex baseline ad/tracker domain list.
#
# Source: hand-curated set of well-known ad networks, trackers, beacon
# endpoints and tag managers. One domain per line. Lines starting with
# `#` are comments. The runtime parser strips comments / blanks and
# stores the rest in a HashSet — subdomain match is suffix-based, so
# putting `doubleclick.net` here blocks `pagead.l.doubleclick.net` too.
#
# Expand at runtime with `crawlex update-blocklist` (writes an override
# file from EasyList that wins over this baseline).
#
# Google / DoubleClick / Adsense
2mdn.net
adservice.google.com
adservice.google.de
adservice.google.co.uk
ad.doubleclick.net
adclick.g.doubleclick.net
ade.googlesyndication.com
afs.googlesyndication.com
cm.g.doubleclick.net
doubleclick.net
google-analytics.com
googleads.g.doubleclick.net
googleadservices.com
googlesyndication.com
googletagmanager.com
googletagservices.com
googletraveladservices.com
imasdk.googleapis.com
pagead2.googleadservices.com
pagead2.googlesyndication.com
partner.googleadservices.com
partnerad.l.doubleclick.net
pubads.g.doubleclick.net
securepubads.g.doubleclick.net
stats.g.doubleclick.net
tpc.googlesyndication.com
www.googleadservices.com
www.googletagmanager.com
www.googletagservices.com

# Facebook / Meta
analytics.facebook.com
business.facebook.com
connect.facebook.net
fbcdn-profile-a.akamaihd.net
graph.facebook.com
pixel.facebook.com

# Amazon ads
aax.amazon-adsystem.com
amazon-adsystem.com
assoc-amazon.com
fls-na.amazon.com
fls-eu.amazon.com
unagi.amazon.com
unagi-na.amazon.com
unagi-eu.amazon.com

# Microsoft / Bing / LinkedIn
ads.linkedin.com
ads.microsoft.com
bat.bing.com
clarity.ms
clkstat.china.cn
www.bing.com/aclick

# Apple
iad.apple.com
iad-apple.com
iad.apple.com.akadns.net

# Twitter / X
ads-api.twitter.com
ads-twitter.com
analytics.twitter.com
syndication.twitter.com
t.co
static.ads-twitter.com

# TikTok / Bytedance
ads.tiktok.com
analytics.tiktok.com
business-api.tiktok.com
log.byteoversea.com
log16-normal-useast5.tiktokv.com
mssdk.bytedance.com
mssdk-va.tiktokv.com

# Snap
sc-static.net
tr.snapchat.com
tr6.snapchat.com

# Pinterest
analytics.pinterest.com
ct.pinterest.com
tr.pinterest.com

# Reddit
events.redditmedia.com

# Yandex
an.yandex.ru
mc.yandex.ru
mc.yandex.com
yandex-metrika.com
metrika-informer.com

# Baidu
hm.baidu.com
pos.baidu.com
cbjs.baidu.com
cpro.baidu.com
eclick.baidu.com

# Generic analytics platforms
adobedtm.com
adobe.tt.omtrdc.net
amplitude.com
api.amplitude.com
api.mixpanel.com
api.segment.com
api.segment.io
api2.amplitude.com
app-measurement.com
appboycdn.com
appsflyer.com
braze.com
chartbeat.com
crazyegg.com
fullstory.com
heapanalytics.com
hotjar.com
hotjar.io
hs-analytics.net
hs-banner.com
hs-scripts.com
hubspot.com
intercom.io
js.intercomcdn.com
kissmetrics.com
liveperson.net
loggly.com
logrocket.com
matomo.cloud
metricsapi.heap.io
mixpanel.com
mouseflow.com
ms.appboy.com
newrelic.com
nr-data.net
optimizely.com
parsely.com
piwik.pro
plausible.io
quantserve.com
rudderstack.com
script.crazyegg.com
script.hotjar.com
scorecardresearch.com
segment.io
segmentapi.com
sentry.io
service.bly.appboy.com
smartlook.com
splitmetrics.com
static.cloudflareinsights.com
static.hotjar.com
stats.wp.com
tealiumiq.com
typekit.net
vars.hotjar.com
woopra.com
wt-eu02.net
yieldlab.net

# Ad networks / SSPs / DSPs
3lift.com
33across.com
acuityplatform.com
adblade.com
adcolony.com
addthis.com
addtoany.com
adform.net
adfox.ru
adgear.com
adgrx.com
adition.com
adkernel.com
adlightning.com
adnxs.com
adnxs-simple.com
adpushup.com
adriver.ru
adsafeprotected.com
adscale.de
adsco.re
adsnative.com
adspirit.de
adsrvr.org
adtech.com
adtechjp.com
adtechus.com
adtelligent.com
adthrive.com
advangelists.com
adverline.com
adverticum.net
advertising.com
adzerk.net
amazon-adsystem.com
amobee.com
aniview.com
appier.net
appnexus.com
audiencemanager.de
beachfront.com
betweendigital.com
bidr.io
bidswitch.net
bidtheatre.com
bidvertiser.com
bizographics.com
bluekai.com
brightcove.com
brightcove.net
btrll.com
buysellads.com
buzzfeed-partners.com
casalemedia.com
cedexis.com
celtra.com
chango.com
chartboost.com
chitika.net
cj.com
cl.qq.com
clickonometrics.pl
clickpoint.io
clicktale.net
cloudfilt.com
cnzz.com
collective-media.net
connatix.com
contextweb.com
conversantmedia.com
coull.com
criteo.com
criteo.net
crwdcntrl.net
cxense.com
demdex.net
deployads.com
displaymarketplace.com
distilnetworks.com
dotomi.com
doubleverify.com
e-planning.net
emxdgt.com
engagebdr.com
ensighten.com
ero-advertising.com
evidon.com
exoclick.com
exosrv.com
extole.com
exponential.com
eyeota.net
eyereturn.com
eyeviewads.com
fastclick.net
flashtalking.com
freewheel.tv
fwmrm.net
gemini.yahoo.com
geniee.co.jp
go-mpulse.net
googleadapis.com
gravity.com
gumgum.com
heias.com
hi-pi.com
imrworldwide.com
indexww.com
infolinks.com
inmobi.com
innovid.com
insightexpressai.com
instinctiveads.com
ip-api.com
ipredictive.com
juicyads.com
kontera.com
krxd.net
lijit.com
lockerdome.com
loopme.com
ltassrv.com
lucidpath.io
m6r.eu
magnite.com
mantisadnetwork.com
mathtag.com
media.net
media-net.com
medianet.com
mediavine.com
meetrics.net
mfadsrvr.com
mgid.com
microad.jp
mintegral.com
mngads.com
moatads.com
moatpixel.com
mookie1.com
mpxads.com
mxptint.net
nativeads.com
nativendo.de
neodatagroup.com
netmera.com
nextroll.com
nexac.com
ninthdecimal.com
nrelate.com
nuggad.net
omg-ssp.com
omnitagjs.com
onaudience.com
onesignal.com
openx.com
openx.net
optimatic.com
optnmstr.com
ourcdn.com
outbrain.com
ozone.tech
parsec.media
pagefair.com
permutive.com
piano.io
podsights.com
prebid.org
pressboard.com
pubmatic.com
pulsepoint.com
quantcast.com
quora.com/_/ad
rakuten.com/adv
rakutenadvertising.com
redintelligence.net
revcontent.com
revjet.com
rfihub.com
rfihub.net
richaudience.com
risingstack.com
rkdms.com
rlcdn.com
rocketfuel.com
rqtrk.eu
rubiconproject.com
rundsp.com
scarabresearch.com
sekindo.com
servebom.com
servedbyopenx.com
sharethrough.com
simpli.fi
sitescout.com
skimresources.com
smaato.com
smaato.net
smartadserver.com
smartclip.net
smartstream.tv
snigelweb.com
sojern.com
sonobi.com
sovrn.com
spotxchange.com
spotx.tv
ssl-images-amazon.com/ads
steelhousemedia.com
stickyadstv.com
strkbid.com
sumome.com
sundaysky.com
supersonicads.com
sxp.smartclip.net
synapsegroupinc.com
taboola.com
tapad.com
tapclicks.com
tcs.teliad.com
teads.tv
tealium.com
the-ozone-project.com
trafficjunky.com
trafficjunky.net
trafmag.com
travelaudience.com
tremorhub.com
tribalfusion.com
triplelift.com
trustarc.com
truste.com
tubemogul.com
turn.com
tvsquared.com
twiago.com
ubidca.com
uidapi.com
unbxd.com
underdog.media
undertone.com
unrulymedia.com
upravel.com
usabilla.com
usemessages.com
videohub.tv
videologygroup.com
vidible.tv
viralize.com
vitamine.networldmedia.net
vivamio.com
vungle.com
w55c.net
wmads.com
xad.com
xaxis.com
xg4ken.com
yashi.com
yieldmo.com
yieldoptimizer.com
yieldpartners.com
yieldselect.com
zedo.com
zemanta.com
zergnet.com
zergscape.com
zucks.net

# CDN / tag delivery
cdn.optimizely.com
cdn.segment.com
cdn.taboola.com
cdn4.forter.com
ct.api.bing.com
tags.tiqcdn.com

# Misc beacons / pixels
b.aol.com
b.collective-media.net
b.scorecardresearch.com
beacon.gu-web.net
beacon.krxd.net
beacon.walmart.com
log.outbrain.com
log.spotxchange.com
log.taboola.com
pixel.adsafeprotected.com
pixel.advertising.com
pixel.facebook.com
pixel.mathtag.com
pixel.quantserve.com
pixel.rubiconproject.com
pixel.servebom.com
pixel.sitescout.com
pixel.tapad.com
px.ads.linkedin.com
px.moatads.com
trk.adserver.com
trk.atemda.com
trk.pinterest.com
www.googleadservices.com/pagead