crawlrs 0.2.0

crawlrs - Enterprise-grade web scraping and crawling platform built with Rust
Documentation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1254
1255
1256
1257
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
1270
1271
1272
1273
1274
1275
1276
1277
1278
1279
1280
1281
1282
1283
1284
1285
1286
1287
1288
1289
1290
1291
1292
1293
1294
1295
1296
1297
1298
1299
1300
1301
1302
1303
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313
1314
1315
1316
1317
1318
1319
1320
1321
1322
1323
1324
1325
1326
1327
1328
1329
1330
1331
1332
1333
1334
1335
1336
1337
1338
1339
1340
1341
1342
1343
1344
1345
1346
1347
1348
1349
1350
1351
1352
1353
1354
1355
1356
1357
1358
1359
1360
1361
1362
1363
1364
1365
1366
1367
1368
1369
1370
1371
1372
1373
1374
1375
1376
1377
1378
1379
1380
1381
1382
1383
1384
1385
1386
1387
1388
1389
1390
1391
1392
1393
1394
1395
1396
1397
1398
1399
1400
1401
1402
1403
1404
1405
1406
1407
1408
1409
1410
1411
1412
1413
1414
1415
1416
1417
1418
1419
1420
1421
1422
1423
1424
1425
1426
1427
1428
1429
1430
1431
1432
1433
1434
1435
1436
1437
1438
1439
1440
1441
1442
1443
1444
1445
1446
1447
1448
1449
1450
1451
1452
1453
1454
1455
1456
1457
1458
1459
1460
1461
1462
1463
1464
1465
1466
1467
1468
1469
1470
1471
1472
1473
1474
1475
1476
1477
1478
1479
1480
1481
1482
1483
1484
1485
1486
1487
1488
1489
1490
1491
1492
1493
1494
1495
1496
1497
1498
1499
1500
1501
1502
1503
1504
1505
1506
1507
1508
1509
1510
1511
1512
1513
1514
1515
1516
1517
1518
1519
1520
1521
1522
1523
1524
1525
1526
1527
1528
1529
1530
1531
1532
1533
1534
1535
1536
1537
1538
1539
1540
1541
1542
1543
1544
1545
1546
1547
1548
1549
1550
1551
1552
1553
1554
1555
1556
1557
1558
1559
1560
1561
1562
1563
1564
1565
1566
1567
1568
1569
1570
1571
1572
1573
1574
1575
1576
1577
1578
1579
1580
1581
1582
1583
1584
1585
1586
1587
1588
1589
1590
1591
1592
1593
1594
1595
1596
1597
1598
1599
1600
1601
1602
1603
1604
1605
1606
1607
1608
1609
1610
1611
1612
1613
1614
1615
1616
1617
1618
1619
1620
1621
1622
1623
1624
1625
1626
1627
1628
1629
1630
1631
1632
1633
1634
1635
1636
1637
1638
1639
1640
1641
1642
1643
1644
1645
1646
1647
1648
1649
1650
1651
1652
1653
1654
1655
1656
1657
1658
1659
1660
1661
1662
1663
1664
1665
1666
1667
1668
1669
1670
1671
1672
1673
1674
1675
1676
1677
1678
1679
1680
1681
1682
1683
1684
1685
1686
1687
1688
1689
1690
1691
1692
1693
1694
1695
1696
1697
1698
1699
1700
1701
1702
1703
1704
1705
1706
1707
1708
1709
1710
1711
1712
1713
1714
1715
1716
1717
1718
1719
1720
1721
1722
1723
1724
1725
1726
1727
1728
1729
1730
1731
1732
1733
1734
1735
1736
1737
1738
1739
1740
1741
1742
1743
1744
1745
1746
1747
1748
1749
1750
1751
1752
1753
1754
1755
1756
1757
1758
1759
1760
1761
1762
1763
1764
1765
1766
1767
1768
1769
1770
1771
1772
1773
1774
1775
1776
1777
1778
1779
1780
1781
1782
1783
1784
1785
1786
1787
1788
1789
1790
1791
1792
1793
1794
1795
1796
1797
1798
1799
1800
1801
1802
1803
1804
1805
1806
1807
1808
1809
1810
1811
1812
1813
1814
1815
1816
1817
1818
1819
1820
1821
1822
1823
1824
1825
1826
1827
1828
1829
1830
1831
1832
1833
1834
1835
1836
1837
1838
1839
1840
1841
1842
1843
1844
1845
1846
1847
1848
1849
1850
1851
1852
1853
1854
1855
1856
1857
1858
1859
1860
1861
1862
1863
1864
1865
1866
1867
1868
1869
1870
1871
1872
1873
1874
1875
1876
1877
1878
1879
1880
1881
1882
1883
1884
1885
1886
1887
1888
1889
1890
1891
1892
1893
1894
1895
1896
1897
1898
1899
1900
1901
1902
1903
1904
1905
1906
1907
1908
1909
1910
1911
1912
1913
1914
1915
1916
1917
1918
1919
1920
1921
1922
1923
1924
1925
1926
1927
1928
1929
1930
1931
1932
1933
1934
1935
1936
1937
1938
1939
1940
1941
1942
1943
1944
1945
1946
1947
1948
1949
1950
1951
1952
1953
1954
1955
1956
1957
1958
1959
1960
1961
1962
1963
1964
1965
1966
1967
1968
1969
1970
1971
1972
1973
1974
1975
1976
1977
1978
1979
1980
1981
1982
1983
1984
1985
1986
1987
1988
1989
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
2025
2026
2027
2028
2029
2030
2031
2032
2033
2034
2035
2036
2037
2038
2039
2040
2041
2042
2043
2044
2045
2046
2047
2048
2049
2050
2051
2052
2053
2054
2055
2056
2057
2058
2059
2060
2061
2062
2063
2064
2065
2066
2067
2068
2069
2070
2071
2072
2073
2074
2075
2076
2077
2078
2079
2080
2081
2082
2083
2084
2085
2086
2087
2088
2089
2090
2091
2092
2093
2094
2095
2096
2097
2098
2099
2100
2101
2102
2103
2104
2105
2106
2107
2108
2109
2110
2111
2112
2113
2114
2115
2116
2117
2118
2119
2120
2121
2122
2123
2124
2125
2126
2127
2128
2129
2130
2131
2132
2133
2134
2135
2136
2137
2138
2139
2140
2141
2142
2143
2144
2145
2146
2147
2148
2149
2150
2151
2152
2153
2154
2155
2156
2157
2158
2159
2160
2161
2162
2163
2164
2165
2166
2167
2168
2169
2170
2171
2172
2173
2174
2175
2176
2177
2178
2179
2180
2181
2182
2183
2184
2185
2186
# 🏗️ Crawlrs Architecture

<div align="center">

![Documentation](https://img.shields.io/badge/type-architecture-purple?style=for-the-badge)
![Version](https://img.shields.io/github/v/release/Kirky-X/crawlrs?label=version&style=for-the-badge)
![License](https://img.shields.io/github/license/Kirky-X/crawlrs?label=license&style=for-the-badge)

**Version:** 0.2.0 | **Last Updated:** 2026-11-15 | **Author:** Kirky.X

</div>

---

## 📋 目录

- [🧭 Overview](#-overview)
- [📐 Architectural Principles](#-architectural-principles)
- [🖥️ System Architecture](#️-system-architecture)
- [🧱 Layer Architecture](#-layer-architecture)
- [🧩 Core Components](#-core-components)
  - [Feature Gate Architecture](#feature-gate-architecture)
- [🔄 Data Flow](#-data-flow)
- [🕷️ Crawling Engines](#️-crawling-engines)
- [🚀 Crawl Capability Enhancement Modules](#-crawl-capability-enhancement-modules)
- [📬 Queue System](#-queue-system)
- [💾 Caching Strategy](#-caching-strategy)
- [⏱️ Rate Limiting](#️-rate-limiting)
- [🔁 自研库反向需求](#-自研库反向需求)
- [🔒 Security Model](#-security-model)
- [🚢 Deployment Architecture](#-deployment-architecture)
- [📈 Scalability Considerations](#-scalability-considerations)
- [🔮 Future Enhancements](#-future-enhancements)
- [📚 Documentation](#-documentation)
---

## 🧭 Overview

crawlrs is built using **Domain-Driven Design (DDD)** principles with a clean, layered architecture. The system is designed for high performance, scalability, and maintainability.

### Key Design Goals

1. **Performance** - 3-5x higher throughput than Node.js alternatives(设计目标,未经权威测量验证,详见 [PERFORMANCE.md](PERFORMANCE.md))
2. **Scalability** - Horizontal scaling capabilities via stateless workers
3. **Type Safety** - Leverage Rust's type system for compile-time safety
4. **Flexibility** - Trait-based architecture for engines and extensions
5. **Observability** - Built-in metrics and inklog 0.3 structured logging via the `log::` facade

### Technology Stack

| Component | Technology | Version |
|-----------|-----------|---------|
| Async Runtime | Tokio | 1.54 |
| Web Framework | Axum | 0.8 |
| ORM | Sea-ORM (via dbnexus) | 2.0.4 |
| HTTP Client | Reqwest | 0.13 |
| Database | PostgreSQL | 16+ |
| Cache | oxcache (in-memory) | 0.5 |
| Rate Limiting | limiteron | 0.3 |
| DI Framework | trait-kit | 0.5 |
| Config | confers | 0.6 |
| SDK Generator | sdforge | 0.5 |
| Logging | inklog | 0.3 |
| Browser Engine | chromiumoxide | 0.9 |
| HTML Parser | scraper | 0.29 |

---

## 📐 Architectural Principles

### 1. Separation of Concerns

Each layer has a specific responsibility:
- **Presentation** - HTTP request/response handling, middleware
- **Application** - Use cases and business workflows, DTO mapping
- **Domain** - Core business logic and entities, repository interfaces
- **Infrastructure** - External integrations, database, cache, security

### 2. Dependency Inversion

High-level modules depend on abstractions (traits), not concrete implementations:

```rust
// Domain defines interface
trait TaskRepository: Send + Sync {
    async fn create(&self, task: &Task) -> Result<Task>;
    async fn find_by_id(&self, id: crate::domain::id::Id) -> Result<Option<Task>>;
    async fn acquire_next(&self, worker_id: crate::domain::id::Id) -> Result<Option<Task>>;
}

// Infrastructure provides implementation
struct TaskRepoImpl {
    db: Arc<DbPool>,
}

impl TaskRepository for TaskRepoImpl { ... }
```

### 3. Single Responsibility

Each component has one reason to change:
- `scrape_worker` - Only handles scrape task execution
- `webhook_worker` - Only handles webhook delivery
- `backlog_worker` - Only handles expired task cleanup
- `LimiteronService` - Only manages rate limits

### 4. Open/Closed Principle

System is open for extension, closed for modification:
- Add new scraping engines by implementing `ScraperEngine` trait
- Add new search engines by implementing `SearchEngine` trait
- Add new middleware by implementing `tower::Layer`

---

## 🖥️ System Architecture

```mermaid
flowchart TD
    subgraph Clients [Client Applications]
        C1[Web Apps]
        C2[Mobile Apps]
        C3[SDKs - sdforge]
        C4[CLI Tools]
    end

    subgraph API [API Gateway - Axum 0.8]
        subgraph Middleware [Middleware Stack]
            M1[Authentication - API Key / Bearer]
            M2[Rate Limiting - single middleware, limiteron-backed]
            M3[Team Concurrency - Semaphore]
            M4[Security Headers, CORS, Logging]
        end

        subgraph Handlers [HTTP Handlers & Routes]
            H1[Scrape Handler]
            H2[Crawl Handler]
            H3[Search Handler]
            H4[Extract Handler]
            H5[Task Handler]
            H6[Webhook Handler]
            H7[Team Handler]
            H8[Audit Handler]
            H9[Metrics Handler]
        end
    end

    subgraph App [Application Layer]
        A1[CreateScrape Use Case]
        A2[Crawl Use Case]
        A3[DTOs]
    end

    subgraph Workers [Worker Pool]
        W1[Scrape Worker]
        W2[Webhook Worker]
        W3[Backlog Worker]
        W4[Expiration Worker]
        W6[Manager]
    end

    subgraph Domain [Domain Layer]
        D1[Models / Entities]
        D2[Services]
        D3[Repository Interfaces]
    end

    subgraph Engines [Engine Layer]
        E1[EngineClient - Public API]
        E2[EngineRouter]
        E3[ReqwestEngine]
        E4[PlaywrightEngine]
        E5[FlareSolverrEngine]
    end

    subgraph Infra [Infrastructure Layer]
        DB[(Postgres Database<br/>dbnexus 0.6.1)]
        RC[(oxcache 0.5<br/>in-memory)]
        EXT[External APIs]
        MON[inklog / Prometheus]
        SEC[SSRF / Geolocation / DNS]
    end

    subgraph Queue [Queue Layer]
        Q1[TaskQueue]
        Q2[PostgresTaskQueue]
        Q3[MessageQueue]
    end

    subgraph Search [Search Subsystem]
        S1[SearchClient]
        S2[SmartSearchEngine]
        S3[Baidu/Bing/Sogou]
        S4[Exa/Tavily]
    end

    subgraph Common [Common Utilities]
        CU1[CacheMode]
        CU2[Error Types]
        CU3[Constants]
    end

    subgraph Utils [Utils]
        U1[Retry/Backoff]
        U2[Coalesce]
        U3[UA Pool]
        U4[Hedge]
        U5[Dedup]
    end

    subgraph AgentLib [Agent Library]
        AL1[Fetch]
        AL2[Search]
    end

    subgraph I18n [Internationalization]
        I1[I18nBundle]
        I2[Locale]
    end

    subgraph Config [Configuration]
        CF1[Settings]
        CF2[Config Loader]
    end

    subgraph Bootstrap [Bootstrap]
        B1[Route Assembly]
        B2[Service Init]
        B3[Worker Init]
    end

    Clients -->|"HTTPS/REST API"| API
    API --> App
    API --> Workers
    App --> Domain
    Workers --> Domain
    Domain --> DB
    Domain --> RC
    Domain --> SEC
    Domain --> EXT
    Domain --> MON
    App --> Engines
    Workers --> Engines
    Engines --> EXT
    Workers --> Queue
    Queue --> DB
    App --> Search
    Search --> EXT
    Domain --> Common
    Domain --> Utils
    Domain --> I18n
    Config --> API
    Bootstrap --> API
    Bootstrap --> Workers
    Bootstrap --> Engines
```

---

## 🧱 Layer Architecture

### Presentation Layer

**Location:** `src/presentation/`

**Responsibilities:**
- HTTP request/response handling
- Request validation
- Response formatting
- Middleware implementation (auth, rate limiting, security)

**Components:**

```
presentation/
├── handlers/              # HTTP endpoint handlers
│   ├── scrape_handler.rs
│   ├── scrape_commands.rs
│   ├── scrape_queries.rs
│   ├── crawl_handler.rs
│   ├── crawl_commands.rs
│   ├── crawl_queries.rs
│   ├── search_handler.rs
│   ├── extract_handler.rs
│   ├── task_handler.rs
│   ├── task_commands.rs
│   ├── task_queries.rs
│   ├── webhook_handler.rs       #[cfg(feature = "webhook")]
│   ├── team_handler.rs          #[cfg(feature = "teams")]
│   ├── audit_handler.rs
│   ├── metrics_handler.rs
│   ├── api_key_handler.rs       #[cfg(feature = "auth")]
│   ├── map_handler.rs           #[cfg(feature = "sitemap")]
│   ├── response_builder.rs
│   └── mod.rs
├── forge_api/             # sdforge inventory 注册面(始终编译)
│   ├── mod.rs
│   ├── crawl.rs
│   ├── management.rs
│   ├── map.rs
│   ├── scrape.rs
│   ├── search.rs
│   └── task.rs
├── routes/                # Handler organization and unit-test assembly; route assembly in `src/bootstrap/routes.rs`
│   ├── mod.rs             # health, version, base routes
│   ├── handlers.rs        # Handler route registration
│   ├── scrape.rs
│   ├── crawl.rs
│   ├── extract.rs
│   └── task.rs
├── middleware/             # HTTP middleware
│   ├── auth_middleware.rs
│   ├── auth_bridge.rs
│   ├── auth_types.rs
│   ├── rate_limit_middleware.rs          # 唯一限流中间件(经 RateLimitingService trait 走 limiteron)
│   ├── security_headers_middleware.rs
│   ├── team_semaphore_middleware.rs
│   ├── scope_validation.rs
│   └── mod.rs
├── sdk/                   # sdforge SDK interface
│   ├── mod.rs
│   ├── mocks.rs
│   └── tests.rs
├── helpers/               # Shared helpers
│   ├── mod.rs
│   └── rate_limit_helper.rs
├── extractors/            # Request extractors
│   ├── mod.rs
│   └── app_deps.rs
├── errors.rs
├── state.rs
└── mod.rs
```

SSRF 校验实现不在 presentation 层:`src/infrastructure/security/ssrf/{mod,error,redirect,static_validator,types}.rs`。

**SDK Interface Layer (sdforge 0.5):**

The presentation layer includes an SDK interface built on **sdforge 0.5**, which wraps domain services as HTTP endpoints via sdforge's `#[forge]` macro. All SDK handlers extract authentication context from `AuthState` (populated by `auth_middleware`), never from the request body.

**Handler Flow:**

生产 handler 的依赖提取有两种真实形态。`scrape_commands::create_scrape` 经聚合提取器 `CrawlRsDeps`(`src/presentation/extractors/app_deps.rs:43`)一次性取常用依赖,地理限制与团队服务以 `Extension<Option<..>>` 注入(teams-off 时为 `None`),另需 `ConnectInfo` 取客户端 IP 做 SSRF 审计:

```rust
pub async fn create_scrape(
    CrawlRsDeps {
        queue,
        settings: _settings,
        task_repo: task_repository,
        rate_limiting_service,
        auth_state,
    }: CrawlRsDeps,
    // teams-off(Extension 未装配)时为 None,跳过检查
    Extension(geo_restriction_repo): Extension<Option<Arc<dyn GeoRestrictionRepository>>>,
    Extension(team_service): Extension<Option<Arc<TeamService>>>,
    ConnectInfo(addr): ConnectInfo<SocketAddr>,
    payload: Result<Json<ScrapeRequestDto>, axum::extract::rejection::JsonRejection>,
) -> impl IntoResponse
```

`extract_handler::extract`(`src/presentation/handlers/extract_handler.rs:54`)是少数直列 `Extension<...>` 的 handler,且带 `GR` 泛型由路由装配决定(teams-on 装配 `DatabaseGeoRestrictionRepository`,teams-off 用无泛型形式):

```rust
pub async fn extract<GR>(
    Extension(queue): Extension<Arc<dyn TaskQueue>>,
    Extension(_settings): Extension<Arc<Settings>>,
    Extension(task_repository): Extension<Arc<dyn TaskRepository>>,
    Extension(rate_limiting_service): Extension<Arc<dyn RateLimitingService>>,
    Extension(geo_restriction_repo): Extension<Arc<GR>>,
    Extension(team_service): Extension<Arc<TeamService>>,
    Extension(auth_state): Extension<AuthState>,
    ConnectInfo(addr): ConnectInfo<SocketAddr>,
    // ...
) -> impl IntoResponse
```

**Middleware Stack:**

1. **Authentication Middleware** - API key validation, Bearer token, team ID extraction
2. **Security Headers Middleware** - CSP, X-Content-Type-Options, X-Frame-Options
3. **CORS** - Configurable origins via `tower-http`
4. **Rate Limiting** - 单一中间件 `rate_limit_middleware.rs`:实现本身只做 429 + `Retry-After` 判定与头部设置,实际速率判断与配额扣减经 `RateLimitingService` trait 委托给 `LimiteronService`(limiteron 0.3,PostgreSQL 后端)。全仓无 Redis 依赖
5. **Team Semaphore Middleware** - Per-team concurrency control
6. **Scope Validation** - API key permission scope checking(未挂载生产路由)

---

### Application Layer

**Location:** `src/application/`

**Responsibilities:**
- Use case orchestration
- DTO definitions and transformations
- Business workflow coordination

**Components:**

```
application/
├── use_cases/             # Business use cases
│   ├── mod.rs
│   ├── create_scrape.rs
│   ├── crawl_use_case.rs
│   ├── crawl_link_processor.rs
│   ├── crawl_state_machine.rs
│   └── map_use_case.rs
└── dto/                   # Data Transfer Objects
    ├── mod.rs
    ├── scrape_request.rs
    ├── scrape_response.rs
    ├── crawl_request.rs
    ├── extract_request.rs
    ├── search_request.rs
    ├── task_query_request.rs
    ├── webhook_request.rs
    ├── geo_restriction_request.rs
    └── map_request.rs
```

**Use Case Pattern:**

```rust
pub struct CreateScrapeUseCase<R, Q, C> {
    task_repo: Arc<R>,
    queue: Arc<Q>,
    cache: Arc<C>,
}

impl<R, Q, C> CreateScrapeUseCase<R, Q, C>
where
    R: TaskRepository + Send + Sync,
    Q: TaskQueue + Send + Sync,
    C: CacheClient + Send + Sync,
{
    pub async fn execute(&self, request: ScrapeRequestDto) -> Result<ScrapeResponseDto> {
        // 1. Validate request
        // 2. Check rate limits
        // 3. Check cache
        // 4. Create task
        // 5. Enqueue
        // 6. Return response
    }
}
```

---

### Domain Layer

**Location:** `src/domain/`

**Responsibilities:**
- Core business entities
- Business rules and validation
- Repository interfaces (contracts)
- Domain services

**Components:**

```
domain/
├── models/                 # Domain entities
│   ├── mod.rs
│   ├── builders.rs
│   ├── task_domain.rs
│   ├── task_model.rs
│   ├── crawl_model.rs
│   ├── scrape_result_model.rs
│   ├── search_result.rs
│   ├── team_model.rs
│   ├── webhook_model.rs
│   ├── credits_model.rs
│   └── validations.rs
├── repositories/           # Repository interfaces
│   ├── mod.rs
│   ├── task_repository.rs
│   ├── crawl_repository.rs
│   ├── scrape_result_repository.rs
│   ├── webhook_repository.rs
│   ├── webhook_event_repository.rs
│   ├── team_repository.rs
│   ├── credits_repository.rs
│   ├── geo_restriction_repository.rs
│   ├── audit_log_repository.rs
│   └── tasks_backlog_repository.rs
├── services/               # Domain services
│   ├── mod.rs
│   ├── rate_limiting_service.rs
│   ├── extraction_service.rs
│   ├── extraction_utils.rs
│   ├── team_service.rs
│   ├── team_service_test.rs          # 内联测试(与实现同文件)
│   ├── team_semaphore.rs
│   ├── search_service.rs
│   ├── webhook_service.rs
│   ├── webhook_sender.rs
│   ├── webhook_event_builder.rs
│   ├── noop_webhook_service.rs
│   ├── audit_service.rs
│   ├── audit_log_builder.rs
│   ├── geo_location.rs
│   ├── id_service.rs                 # NebulaId ID 生成服务
│   ├── rag_strategy.rs               # RAG 语义分块 + 向量检索提取
│   ├── rerank.rs                     # 重排策略
│   ├── llm/                       # LLM 边界(mod: trait + ExplicitErrorCarrier;
│   │                              #  prompt_builder, chunker, provenance;
│   │                              #  vision_adapter 由 engine-mllm 门控)
│   ├── content_extractor/         # 正文提取(trafilatura/dom_smoothie/css_rule/quality_gate/facade)
│   ├── markdown/                 # 转换器装配(alchemize_converter 默认 + htmd 诊断臂)
│   ├── markdown_service.rs
│   ├── relevance_scorer.rs
│   ├── retry_handler.rs
│   ├── webhook/                  # Webhook 管理/领域逻辑(management, mod)
│   └── tests/                    # 领域服务独立测试目录
├── id/                     # NebulaId 域 ID 类型(Id / IdGenerator)
├── auth/                   # Authentication models
│   ├── mod.rs
│   └── scope.rs
├── errors.rs
├── use_cases/              # Domain use cases
│   ├── mod.rs
│   └── create_webhook.rs
├── retention_policy.rs     # Data retention policy definitions
```

**Core Domain Entities:**

**Task Entity:**
```rust
pub struct Task {
    pub id: crate::domain::id::Id,
    pub team_id: crate::domain::id::Id,
    pub api_key_id: crate::domain::id::Id,
    pub task_type: TaskType,
    pub status: TaskStatus,
    pub priority: i32,
    pub url: String,
    pub payload: Value,
    pub retry_count: i32,
    pub attempt_count: i32,
    pub max_retries: i32,
    pub scheduled_at: Option<DateTime<Utc>>,
    pub expires_at: Option<DateTime<Utc>>,
    pub created_at: DateTime<Utc>,
    pub started_at: Option<DateTime<Utc>>,
    pub completed_at: Option<DateTime<Utc>>,
    pub crawl_id: Option<crate::domain::id::Id>,
    pub updated_at: DateTime<Utc>,
    pub lock_token: Option<crate::domain::id::Id>,
    pub lock_expires_at: Option<DateTime<Utc>>,
}
```

所有 ID 字段均为 `crate::domain::id::Id`(NebulaId 域 ID 类型,`src/domain/id/`),不是旧式 UUID 值。

**Task Types:**
- `Scrape` - Single page scrape
- `Crawl` - Multi-page crawl
- `Extract` - Data extraction

**Task Statuses** (`src/domain/models/task_domain.rs:67-79`,五态):
- `queued` - Created, waiting for worker
- `active` - Being processed by worker(不是 `running`)
- `completed` - Successfully completed
- `failed` - Failed with error
- `cancelled` - Cancelled by user

注意:不存在 `Running` 与 `Expired` 变体。任务过期不改变状态取值,而是由 `expires_at` 字段 + `TaskRepository::expire_tasks()` 周期性清理完成;`Expired` 只作为 `tasks_backlog` 的 `TasksBacklogStatus` 存在(回灌队列的状态机,与 `tasks.status` 是两套枚举)。

---

### Infrastructure Layer

**Location:** `src/infrastructure/`

**Responsibilities:**
- External service implementations
- Database access (via dbnexus + Sea-ORM)
- Cache management (oxcache)
- Security (SSRF validation, API key hashing)
- DNS resolution and caching
- Geolocation
- Metrics and observability

**Components:**

```
infrastructure/
├── database/               # Database layer
│   ├── mod.rs
│   ├── dbnexus_connection.rs
│   ├── id_sea_orm.rs
│   ├── query_monitor.rs
│   ├── transaction.rs
│   ├── entities/           # Sea-ORM entity models
│   │   ├── mod.rs
│   │   ├── api_key.rs
│   │   ├── crawl.rs
│   │   ├── credits.rs
│   │   ├── credits_transactions.rs
│   │   ├── geo_restriction_log.rs
│   │   ├── scrape_result.rs
│   │   ├── sea_orm_active_enums.rs
│   │   ├── task.rs
│   │   ├── tasks_backlog.rs
│   │   ├── team.rs
│   │   ├── webhook.rs
│   │   ├── webhook_event.rs
│   │   └── auth/
│   │       ├── mod.rs
│   │       ├── audit_log.rs
│   │       └── scope.rs
│   └── repositories/       # Repository implementations
│       ├── mod.rs
│       ├── macros.rs
│       ├── task_repo_impl.rs
│       ├── crawl_repo_impl.rs
│       ├── scrape_result_repo_impl.rs
│       ├── credits_repo_impl.rs
│       ├── webhook_repo_impl.rs
│       ├── webhook_event_repo_impl.rs
│       ├── database_geo_restriction_repo.rs
│       ├── audit_log_repo_impl.rs
│       ├── tasks_backlog_repo_impl.rs
│       └── tests/               # Repository tests
│           ├── geo_restriction_cleanup_test.rs
│           ├── scrape_result_cleanup_test.rs
│           ├── task_repo_test.rs
│           └── webhook_event_cleanup_test.rs
├── oxcache/                # Unified cache (oxcache-backed)
│   ├── mod.rs
│   └── cache_service.rs
├── security/               # Security implementations
│   ├── mod.rs
│   ├── api_key_hash.rs
│   ├── constant_time_compare.rs
│   ├── env_var_security.rs
│   ├── env_injection.rs
│   ├── env_validation.rs
│   ├── secure_ip.rs
│   └── ssrf/               # SSRF 校验(mod, error, redirect, static_validator, types)
├── auth/                   # garrison 认证引擎 DAO 集成
│   ├── mod.rs              # garrison_config / garrison_dao / garrison_interface / garrison_listener
├── events/                 # 进程内事件总线
│   ├── mod.rs
│   ├── event_bus.rs        # EventBus trait
│   └── broadcast.rs        # BroadcastEventBus(tokio::sync::broadcast 实现)
├── persistence/            # Domain <-> DB entity mappers
│   ├── mod.rs
│   └── mappers/
│       ├── mod.rs
│       ├── task_mapper.rs
│       ├── crawl_mapper.rs
│       ├── credits_mapper.rs
│       └── webhook_mapper.rs
├── services/               # Infrastructure services
│   ├── mod.rs
│   ├── config_service.rs
│   ├── limiteron_service.rs
│   ├── noop_rate_limiting_service.rs
│   ├── http_rerank_provider.rs     # 远端 Rerank HTTP 适配(仅 reqwest,无需 feature)
│   ├── fluxweir_llm_carrier.rs     # 宿主 LLM carrier 注入(T011 退役 rag-remote/genai)
│   ├── fluxweir_providers.rs       # 嵌入/重排远端兜底(fluxweir 统一出口)
│   ├── vecboost_provider.rs        # rag-local:本地嵌入/重排(vecboost)
│   └── webhook_sender_impl.rs
├── dns/                    # DNS resolution + caching
│   ├── mod.rs
│   ├── dns_cache.rs
│   └── ipv4_resolver.rs
├── observability/          # Observability + 失败快照
│   ├── mod.rs
│   ├── metrics.rs
│   └── failure_snapshot.rs
├── geolocation.rs
├── metrics.rs
├── errors.rs
└── mod.rs
```

**Database Layer:**

**Technology:** dbnexus 0.6.1 (builds on Sea-ORM 2.0.4)

dbnexus provides connection pooling, permission control, migration framework, metrics monitoring, and audit logging on top of Sea-ORM's type-safe database access. Three database backends are supported: PostgreSQL (default), SQLite, and MySQL — selectable via `db-postgres` / `db-sqlite` / `db-mysql` features.

**Key Tables:**

| Table | Purpose |
|-------|---------|
| `tasks` | Task records with locking for worker coordination |
| `tasks_backlog` | Backlog of expired/pending tasks for reprocessing |
| `crawls` | Crawl configurations (depth, patterns) |
| `scrape_results` | Scrape results (raw HTML, markdown, metadata) |
| `api_keys` | ⚠️ PARTIALLY DEPRECATED (0.2.0): `key_hash`/`scopes` 列由 garrison 接管,仅保留 `id`/`team_id`/`key` 列供 `api_key_id→team_id` 反查映射(`deprecated_at` 标记) |
| `webhooks` | Webhook configurations (URL, events, retry) |
| `webhook_events` | Webhook event logs and delivery status |
| `audit_logs` | API access and event audit logs (garrison `audit-log` 也写入此表) |
| `team` | Team accounts and settings |
| `credits` | Team credit balances |
| `credits_transactions` | Credit usage history |
| `geo_restriction_logs` | Geographic restriction check logs |
| `auth_scopes` | ⚠️ DEPRECATED (0.2.0): garrison RBAC 接管权限映射,本表不再写入(`deprecated_at` 标记,仅作历史审计只读) |

**Cache Layer:**

**Technology:** oxcache 0.5 (in-memory, no Redis backend)

```rust
// Cache types
pub type SearchCache = Cache<String, Vec<SearchResult>>;
pub type DnsCache = Cache<String, DnsCacheEntry>;
pub type RegexCacheType = Cache<String, String>;
```

oxcache features activated(`Cargo.toml:316`): `["memory", "serialization", "metrics", "bloom"]`。Type-specific TTL:
- Search cache: configurable (default 60s)
- DNS cache: configurable (default 300s)
- Regex cache: configurable (default 600s)

---

### Additional Top-Level Modules

**Location:** `src/queue/`, `src/search/`, `src/common/`, `src/utils/`, `src/agent_lib/`, `src/i18n/`, `src/config/`, `src/bootstrap/`

```
queue/                     # Task queue abstraction
├── mod.rs
├── message_queue.rs       # Message queue trait + implementations
├── postgres_message_queue.rs  # PostgreSQL-backed message queue
└── task_queue.rs          # TaskQueue trait + PostgresTaskQueue

search/                    # Search subsystem
├── mod.rs
├── client.rs              # SearchClient - multi-engine aggregation
├── client/
│   ├── baidu.rs           # Baidu search engine
│   ├── bing.rs            # Bing search engine
│   ├── exa.rs             # Exa search engine (API)
│   ├── sogou.rs           # Sogou search engine
│   ├── tavily.rs          # Tavily search engine (API)
│   ├── searxng.rs         # SearXNG adapter (public instance JSON API)
│   ├── html_parser.rs     # Shared Google-style HTML parser
│   └── shared_utils.rs    # Shared search utilities
├── dedup.rs               # ResultDeduplicator (SimHash)
├── engine_trait.rs        # SearchEngine trait
├── error.rs               # SearchError types
├── response.rs            # SearchResponse types
├── rrf.rs                 # RRFFuser (k=60)
├── smart/
│   └── mod.rs             # SmartSearchEngine (concurrent aggregation)
└── types.rs               # Search types

common/                    # Shared utilities
├── mod.rs
├── cache_mode.rs          # CacheMode enum + CacheContext
├── constants.rs           # Global constants
├── error.rs               # CrawlRsError types
├── http_method.rs         # HTTP method types
├── masking.rs             # Data masking utilities
├── metrics_shim.rs        # Metrics shim
├── test_fixtures.rs       # Test fixtures
├── test_helpers.rs        # Test helpers
├── text_slice.rs          # Text slice utilities
└── time_utils.rs          # Time utilities

utils/                     # Utility modules
├── mod.rs
├── adaptive_concurrency.rs # AIMD adaptive concurrency
├── backoff.rs              # Full-jitter backoff
├── coalesce.rs            # RequestCoalescer
├── crawl_text_integration.rs # Crawl text integration
├── dedup/
│   ├── mod.rs
│   ├── bloom.rs           # Bloom filter dedup
│   ├── deduplicator.rs    # Deduplicator trait
│   └── interner.rs        # String interner
├── error_helpers.rs       # Error helper utilities
├── glob.rs                # Glob pattern matching
├── hedge.rs               # Hedge request controller (EMA/P84)
├── host_discipline.rs     # Per-host politeness discipline
├── http_client.rs         # HTTP client utilities
├── port_sniffer.rs        # Port detection
├── proxy.rs               # Proxy utilities
├── regex_cache.rs         # Regex cache
├── retry/
│   ├── mod.rs
│   ├── directive.rs       # RetryDirective
│   └── tracker.rs         # RetryTracker
├── retry_after.rs         # Retry-After header parsing
├── retry_policy.rs        # Retry policy
├── robots.rs              # Robots.txt checker
├── search_test.rs         # Search test utilities
├── telemetry.rs            # Telemetry utilities
├── text_processing/
│   ├── mod.rs
│   ├── encoding.rs        # Text encoding
│   └── processor.rs       # Text processor
├── ua_pool.rs             # UA profile pool
├── url.rs                 # URL utilities
└── url_normalizer.rs      # URL normalization

agent_lib/                 # Agent library
├── mod.rs
├── error.rs               # Agent error types
├── fetch.rs               # Agent fetch utilities
└── search.rs              # Agent search utilities

i18n/                      # Internationalization
├── mod.rs
├── bundle.rs              # I18nBundle
├── locale.rs              # Locale handling
├── middleware.rs          # i18n middleware
└── translate.rs           # Translation utilities

config/                    # Configuration
├── mod.rs
├── app.rs                 # App config
├── engines.rs             # Engine config
├── llm.rs                 # LLM config
├── logging.rs             # Logging config
├── rag.rs                 # RAG config
├── runtime.rs             # Runtime config
├── search.rs              # Search config (incl. BingSearchSettings)
├── security.rs            # Security config
├── settings.rs            # Settings struct
└── validation.rs          # Config validation

bootstrap/                 # Application bootstrap
├── mod.rs
├── config/
│   ├── mod.rs
│   ├── config_loader.rs   # Config loading
│   └── config_validator.rs # Config validation
├── engines.rs             # Engine initialization
├── error.rs               # Bootstrap errors
├── infrastructure.rs      # Infrastructure initialization
├── routes.rs              # Route assembly (authoritative)
├── route_groups/
│   ├── mod.rs
│   ├── crawl_routes.rs    # Crawl route group
│   ├── management_routes.rs # Management route group
│   ├── map_routes.rs      # Map route group
│   ├── scrape_routes.rs   # Scrape route group
│   └── search_routes.rs   # Search route group
├── services.rs            # Service initialization
├── telemetry.rs           # Telemetry initialization
└── workers.rs             # Worker initialization
```

---

## 🧩 Core Components

### Dependency Injection (trait-kit 0.5)

DI uses **trait-kit 0.5** with async module builders. `src/main.rs:389-405` registers **9 个 Module** 到 `AsyncKit`:

| # | Module | 提供 |
|---|--------|------|
| 1 | `SettingsModule` | `Arc<Settings>`(含 i18n 解析后的配置) |
| 2 | `DatabaseModule` | dbnexus 连接池、迁移、权限 |
| 3 | `HttpModule` | `Arc<reqwest::Client>` |
| 4 | `CacheModule` | oxcache 缓存组件(search/dns/regex/generic) |
| 5 | `RepositoryModule` | 仓储组件(依赖 DatabaseModule) |
| 6 | `EngineModule` | 引擎向量(默认 5 个;`engine-moli` 启用时 +MoliEngine)+ `EngineRouter` + `EngineClient`(依赖 HttpModule、SettingsModule) |
| 7 | `InfrastructureModule` | DB pool + 缓存服务等基础设施组件 |
| 8 | `ServiceModule` | 限流/搜索/webhook/团队服务 + 队列 + worker |
| 9 | `IdModule` | `IdService`(NebulaId ID 生成) |

`EngineModule` 提供 5 个引擎实现:`ReqwestEngine` / `PlaywrightEngine` / `WreqEngine`(`engine-tls-fingerprint`)/ `FlareSolverrEngine`(`engine-flaresolverr`)/ `MllmEngine`(`engine-mllm`)。

**CrawlRsState** 是运行时状态,由 `CrawlRsState::from_kit`(`src/di/axum_state.rs:207`)从构建完成的 kit 提取;它 require 5 个模块:`InfrastructureModule`、`EngineModule`、`ServiceModule`、`SettingsModule`、`IdModule`。自 v0.2.0(`feature-gate-optional-modules` change)起,有 7 个字段被 `#[cfg(feature = "...")]` 门控,仅在对应对业务特性启用时编译:

```rust
#[derive(Clone)]
pub struct CrawlRsState {
    pub db_pool: Arc<DbPool>,
    pub id_service: Arc<IdService>,
    pub task_repo: Arc<dyn TaskRepository>,
    pub credits_repo: Arc<dyn CreditsRepository>,
    pub crawl_repo: Arc<dyn CrawlRepository>,
    pub result_repo: Arc<dyn ScrapeResultRepository>,
    #[cfg(feature = "webhook")]
    pub webhook_repo: Arc<dyn WebhookRepository>,
    #[cfg(feature = "webhook")]
    pub webhook_event_repo: Arc<dyn WebhookEventRepository>,
    pub tasks_backlog_repo: Arc<dyn TasksBacklogRepository>,
    pub task_queue: Arc<dyn TaskQueue>,
    pub rate_limiting_service: Arc<dyn RateLimitingService>,
    #[cfg(feature = "teams")]
    pub team_service: Arc<TeamService>,
    // webhook_service 始终编译(trait),webhook-off 时装配 NoopWebhookService
    pub webhook_service: Arc<dyn WebhookService>,
    pub robots_checker: Arc<dyn RobotsCheckerTrait>,
    pub team_semaphore: Arc<TeamSemaphore>,
    pub request_coalescer: Arc<RequestCoalescer>,
    pub engine_router: Arc<EngineRouter>,
    pub engine_client: Arc<EngineClient>,
    pub create_scrape_use_case: Arc<dyn CreateScrapeUseCaseTrait>,
    pub search_client: Arc<SearchClient>,
    pub search_service: Arc<dyn SearchServiceTrait>,
    // 默认 ExplicitErrorCarrier;宿主经 pub 字段/工厂/from_kit 注入 carrier
    pub llm_service: Arc<dyn LLMServiceTrait>,
    pub extraction_service: Arc<dyn ExtractionServiceTrait>,
    // RAG:rag.enabled=false 或未配置时为 None(Option,非 cfg 门控)
    pub embedding_provider: Option<Arc<dyn EmbeddingProvider>>,
    pub rerank_provider: Option<Arc<dyn RerankProvider>>,
    pub content_extractor: Arc<ContentExtractionFacade>,
    pub cache_service: Arc<dyn CacheService>,
    pub audit_service: Arc<dyn AuditServiceTrait>,
    #[cfg(feature = "webhook")]
    pub webhook_worker: Arc<WebhookWorker>,
    pub backlog_worker: Arc<BacklogWorker>,
    pub expiration_worker: Arc<ExpirationWorker>,
    #[cfg(feature = "teams")]
    pub geo_location_service: Arc<dyn GeoLocationService>,
    #[cfg(feature = "teams")]
    pub geo_restriction_repo: Arc<dyn GeoRestrictionRepository>,
    pub i18n_bundle: Arc<I18nBundle>,
    pub event_bus: Arc<dyn EventBus>,
    #[cfg(feature = "sitemap")]
    pub map_use_case: Arc<MapUseCase>,
}
```

`regex_cache` 已不在 `CrawlRsState` 上(无此字段)。默认构建(platform,含 teams/sitemap)下共 **36 个字段**:29 个恒定编译 + 7 个 cfg 门控(`webhook_repo`/`webhook_event_repo`/`webhook_worker` 受 `webhook`,`team_service`/`geo_location_service`/`geo_restriction_repo` 受 `teams`,`map_use_case` 受 `sitemap`)。字段数以 `src/di/axum_state.rs:70-193` 为准。

| Field | Gated By | Off-mode Behavior |
|-------|----------|-------------------|
| `webhook_repo` / `webhook_event_repo` / `webhook_worker` | `webhook` | Field not compiled; `/v1/webhooks/*` routes not registered; `webhook_worker` spawn block skipped |
| `team_service` / `geo_location_service` / `geo_restriction_repo` | `teams` | Field not compiled; `/v1/teams/*` routes not registered; `extract_handler` signature loses GR generic |
| `map_use_case` | `sitemap` | Field not compiled; `/v1/map` route not registered (sitemap 流式站点图) |
| `webhook_service` | (none, always compiled) | webhook-off: assembled with `NoopWebhookService` (no-op trait impl) |
| `rate_limiting_service` | (none, always compiled) | rate-limit-off: assembled with `NoopRateLimitingService` (always-allow trait impl) |
| `embedding_provider` / `rerank_provider` | (none, `Option`) | rag 未启用或未配置时为 `None`,非 cfg 门控 |

See [Feature Gate Architecture](#feature-gate-architecture) for the full gating matrix.

State is injected into Axum handlers via `Extension<CrawlRsState>`.

**Module dependency graph**(`src/main.rs:389-405` 的注册顺序即依赖顺序;`CrawlRsState::from_kit` require 其中 5 个):

```text
SettingsModule (Arc<Settings> + i18n 配置)
  ├── DatabaseModule → dbnexus pool(迁移/权限)
  ├── HttpModule → Arc<reqwest::Client>
  └── CacheModule → oxcache 缓存组件
         ├── RepositoryModule → Repositories(依赖 DatabaseModule)
         ├── InfrastructureModule → 基础设施组件(DB pool、CacheService 等)
         │     └── ServiceModule → ServicesComponents(依赖 RepositoryModule、EngineModule)
         ├── EngineModule → EngineComponents(依赖 HttpModule、SettingsModule)
         │     └── ServiceModule
         └── IdModule → Arc<IdService>(NebulaId ID 生成)
```

`CrawlRsState::from_kit` 提取 `InfrastructureModule`、`EngineModule`、`ServiceModule`、`SettingsModule`、`IdModule` 五个模块的组件并拍平为扁平字段。

### Feature Gate Architecture

Since v0.2.0 (`feature-gate-optional-modules` change), crawlrs exposes **6 feature axes** that allow operators to build stripped-down binaries for single-tenant / no-auth / no-webhook / no-rate-limit deployments, plus auth-off web assembly and SDK inventory registration. The gating strategy follows two complementary patterns:

1. **Field/module gating** (`#[cfg(feature = "...")]` on `pub mod`/struct field/fn) — gated symbols are **not compiled** when the feature is off, eliminating dead code and reducing binary size.
2. **Noop trait injection** — for traits that business logic calls unconditionally (`WebhookService`, `RateLimitingService`), the trait is always compiled but the DI container assembles a no-op implementation when the feature is off, preserving call-site compatibility.

### Feature Matrix

| Feature | Default | Depends On | Off-mode Behavior |
|---------|---------|------------|-------------------|
| `teams` | on | `auth` | `/v1/teams/*` routes not registered; `extract_handler` loses GR generic + geo-restriction block; `CrawlRsState.{team_service, geo_location_service, geo_restriction_repo}` not compiled; `team_id` falls back to `DEFAULT_TEAM_ID` |
| `auth` | on | `dep:garrison, dep:inventory` | **0.2.0 起 garrison v0.9 接管认证**:`auth_middleware_inner` 调用 `GarrisonUtil::check_api_key` + `bridge_to_auth_state` 注入 `AuthState`;提供 RBAC + JWT + firewall-bruteforce(5 次/60 秒/300 秒锁定)+ audit-log。关闭时改为 `default_identity_middleware` 注入固定 `AuthState{team_id=DEFAULT_TEAM_ID, api_key_id=DEFAULT_API_KEY_ID, scope=ApiKeyScope::denied()}`,无 DB 查询、无暴力破解防护;受保护路由默认 401,放行须经 `allow_unauthenticated_protected()` 显式 opt-in(放行后仍是 denied 受限身份) |
| `rate-limit` | on | —(空数组) | `LimiteronService` replaced by `NoopRateLimitingService` (check_rate_limit→Allowed, check_and_deduct_quota→Ok, get_quota_balance→Ok(i64::MAX), process_backlog_tasks→Ok(0)); `rate-limit` 只门控装配选择,`limiteron` crate 因 `utils/backoff` 无条件依赖其 retry 原语而始终编译,`rate_limit_middleware` 始终挂载(内部 trait 调用 no-op 实现) |
| `webhook` | on | — | `WebhookServiceImpl`/`WebhookManagementServiceImpl`/`webhook_sender`/`webhook_handler`/`webhook_worker` modules not compiled; `/v1/webhooks/*` routes not registered; `webhook_worker` spawn blocks skipped; `WebhookService` trait preserved and assembled with `NoopWebhookService` (all ops return `Ok(())`) |
| `web-axum` | off | `rate-limit`, `webhook`, `metrics`, `content`, `dbnexus`, `axum` | auth-off Web 中间件构建面(platform 隐含 auth 导致原矩阵中该路径不可构建);`--no-default-features --features web-axum,db-postgres --lib` 构建/测试 auth-off 语义 |
| `forge-full` | off | `http` | 激活 `presentation/forge_api` 的 sdforge inventory 注册面;与手写 protected/v2 路由组互斥(并存即启动 panic) |

### Default feature set

```toml
[features]
default = ["platform", "db-postgres"]
# platform:完整平台服务能力面。sitemap 是 platform 的组成部分(0.2.0 起)
platform = ["teams", "auth", "rate-limit", "webhook", "metrics", "content", "sitemap",
            "dep:dbnexus", "dep:axum", "http"]
teams   = ["auth"]
auth    = ["dep:garrison", "dep:inventory"]   # 0.2.0: garrison v0.9 接管认证引擎
rate-limit = []            # 空数组:只门控装配选择,limiteron crate 始终编译
webhook = []
full    = ["standard", "engine-flaresolverr", "extractors", "llm"]
llm     = []                # trait + 编排 + ExplicitErrorCarrier,零外部依赖
engine-mllm = ["engine-playwright", "llm", "gateway"]  # 视觉面经 fluxweir chat 承载(genai 已退役)

# sitemap(0.2.0 起并入 platform):
sitemap = ["dep:quick-xml"]
```

`rate-limit = []` 的原因:`utils/backoff` 的重试原语无条件依赖 `limiteron` crate,故该依赖不随 feature 关闭而摘除;feature 关闭只把 `RateLimitingService` 的装配从 `LimiteronService` 换成 `NoopRateLimitingService`。同理 `teams` 隐含 `auth`,`auth` 隐含 `dep:garrison` + `dep:inventory`。

### Gating pattern examples

**Field gating** (`src/di/axum_state.rs`):

```rust
#[cfg(feature = "webhook")]
pub webhook_repo: Arc<dyn WebhookRepository>,
```

**Noop injection** (`src/bootstrap/services.rs::init_rate_limiting_service`):

```rust
#[cfg(feature = "rate-limit")]
{ /* assemble LimiteronService */ }
#[cfg(not(feature = "rate-limit"))]
{
    log::warn!("rate-limit feature disabled, using NoopRateLimitingService");
    Arc::new(NoopRateLimitingService::new())
}
```

**Route gating with shadowing** (`src/bootstrap/routes.rs`):

```rust
#[cfg(feature = "webhook")]
let app = app.route("/v1/webhooks", post(webhook_handler::create_webhook::<WebhookRepoImpl>));
// webhook-off: route not registered, returns 404
```

**Middleware layer gating** (`src/bootstrap/routes.rs`):

```rust
#[cfg(feature = "auth")]
let app = app.layer(axum::middleware::from_fn(auth_middleware::auth_middleware()));
#[cfg(not(feature = "auth"))]
let app = {
    let template = build_default_identity_template(state);
    app.layer(axum::middleware::from_fn_with_state(
        template,
        auth_middleware::default_identity_middleware,
    ))
};
```

### CI verification

The `feature-matrix` job in `.github/workflows/ci.yml` runs `cargo check` against 7 feature combinations to ensure no broken cfg paths slip in:

| Combination | Flags |
|-------------|-------|
| no-default | `--no-default-features` |
| teams-only | `--no-default-features --features teams` |
| auth-only | `--no-default-features --features auth` |
| rate-limit-only | `--no-default-features --features rate-limit` |
| webhook-only | `--no-default-features --features webhook` |
| default | `--features default` |
| full | `--features full` |

### Migration notes

When a feature is off, the corresponding endpoints return **404** (not 401/403) because routes are not registered at startup. Operators switching from full to stripped-down builds should:

- For `auth`-off: pre-provision credits for `DEFAULT_TEAM_ID` via `add_credits` CLI (see `docs/USER_GUIDE.md` → Single-Tenant / No-Auth Deployment)
- For `rate-limit`-off: ensure upstream gateways enforce their own rate limiting
- For `webhook`-off: notify clients that completion callbacks will not be delivered
- For `teams`-off: only `DEFAULT_TEAM_ID` exists; multi-tenant data isolation is enforced at the application layer (all rows owned by `DEFAULT_TEAM_ID`)

### Configuration Hierarchy

**Location:** `config/default.toml` (confers 0.6)

```toml
[database]              # Connection pool, timeouts, prepare-cache
[server]                # Host, port, enable_port_detection
[cors]                  # Allowed origins
[rate_limiting]         # Enabled, default_rpm, burst_size, storage_backend
[cache]                 # Enabled, memory capacity/TTL
[cache.memory]          # Moka L1 capacity + TTL
[cache.types.search]    # Search result cache TTL + max_size
[cache.types.dns]       # DNS cache TTL + max_size
[cache.types.regex]     # Regex cache TTL + max_size
[concurrency]           # Team limits, memory pressure, AIMD
[politeness]            # Per-host min/max interval, AutoThrottle, backoff, jitter
[retention]             # Data retention (scrape_results_days)
[search]                # Timeout, rate limiting, max_results, default_engine
[search.engines]        # Per-engine enable switches (bing/baidu/sogou)
[search.fallback]       # Fallback engine order + API keys
[search.fallback.exa]   # Exa API key + endpoint
[search.fallback.parallel] # Parallel API key + endpoint
[search.fallback.tavily]   # Tavily API key + endpoint
[search.exa]            # Exa single-engine credentials
[search.tavily]         # Tavily single-engine credentials
[search.searxng]        # SearXNG opt-in: enabled/pool_target/max_failover_attempts/seed_urls/user_agent
[search.searxng.collector] # Dataset url/timeouts/min_uptime_day/retire_miss_cycles
[search.searxng.health]    # Probe interval/timeout/query/candidate_batch/failure_retire_threshold
[search.searxng.breaker]   # Per-instance breaker threshold/recovery_timeout_seconds
[engines]               # max_response_body_bytes
[engines.playwright]     # XHR/Fetch capture settings
[engines.flaresolverr]  # FlareSolverr Full mode
[engines.flaresolverr_cdp] # FlareSolverr CDP mode
[engines.flaresolverr_tls] # FlareSolverr TLS mode
[engines.tls_fingerprint]  # Wreq TLS fingerprint (feature-gated)
[engines.mllm]          # MLLM autonomous navigation (feature-gated)
[webhook]               # Secret, max_retries, batch_size
[bing_search]           # Bing Search API key (via env var)
[llm]                   # model, timeout, provenance(供应商访问已移交宿主 carrier)
[llm.chunking]          # Token window chunking settings
[content_extraction]    # Min word count, replacement ratio, prose gates
[rag]                   # Embedding + rerank providers, local/remote settings
[proxy]                 # URLs, strategy, sticky TTL, cooldown
[session]               # Session identity pool (proxy + UA/TLS + cookie)
[workers]               # Count, graceful shutdown, dedup warmup
[sitemap]               # Enabled, max_urls, concurrency, channel capacity
[crawl]                 # Soft 404 fingerprint learning
[timeouts.workers]      # Webhook/backlog intervals
[timeouts.engines]      # Default/playwright/flaresolverr timeouts + MRT
[timeouts.retry]        # Initial/max backoff
[timeouts.cache]        # Default/memory TTL
[logging]               # Console + file output settings
[observability]         # Failure snapshot directory + limits
[trusted_proxies]       # Enabled, proxy IP list (CIDR)
[auth]                  # JWT secret (garrison)
[robots]                # User-agent, scrape respect, fail-closed, cache TTL
[i18n]                  # Default locale, supported locales, locales dir
[id]                    # NebulaId algorithm, workspace, group
```

---

## 🔄 Data Flow

### Scrape Request Flow

```mermaid
flowchart TD
    A[1. Client Request] --> B[2. API Gateway<br/>Authentication<br/>Rate Limit<br/>SSRF Check]
    B --> C[3. Handler<br/>Validation<br/>DTO Mapping]
    C --> D[4. Use Case<br/>Business Logic]
    D --> E[5. Repository<br/>Task Creation]
    E --> F[6. Task Queue<br/>enqueue]
    F --> G[7. Scrape Worker<br/>dequeue + process]
    G --> H{8. Engine Selection}
    H --> H1[ReqwestEngine]
    H --> H2[PlaywrightEngine<br/>(if needs_js)]
    H --> H3[FlareSolverrEngine<br/>(if anti-bot)]
    H1 --> I[9. Result Storage<br/>oxcache + DB]
    H2 --> I
    H3 --> I
    I --> J[10. Task Update<br/>Status = Completed]
    J --> K[11. Webhook Notification<br/>Async]
    K --> L[12. Client Response<br/>Poll/Webhook]
```

### Crawl Request Flow

```mermaid
flowchart TD
    A[1. Client creates crawl task<br/>with URL and depth] --> B[2. Handler validates request<br/>and creates crawl record]
    B --> C[3. Crawl worker picks up task]
    C --> D[4. Worker fetches initial page]
    D --> E[5. Extracts links matching patterns]
    E --> F[6. Filters URLs<br/>include/exclude patterns]
    F --> G{7. For each URL}
    G --> G1[Create scrape subtask]
    G1 --> G2[Enqueue for processing]
    G2 --> G
    G --> H[8. Wait for all subtasks<br/>to complete]
    H --> I[9. Aggregate results]
    I --> J[10. Update crawl status<br/>to completed]
    J --> K[11. Trigger webhook notification]
```

---

## 🕷️ Crawling Engines

### Engine Architecture

The engine layer is organized around three core abstractions: `ScraperEngine` trait, `EngineRouter` with load balancing, and `EngineClient` as the public API.

**EngineClient** (`src/engines/engine_client.rs`) is the single public entry point for all scraping operations. It encapsulates UA rotation, circuit breaker state, engine selection, and retry logic.

**ScraperEngine Trait:**

```rust
#[async_trait]
pub trait ScraperEngine: Send + Sync {
    async fn scrape(
        &self,
        request: &InternalScrapeRequest,
    ) -> Result<InternalScrapeResponse, EngineError>;

    fn support_score(&self, request: &InternalScrapeRequest) -> u8;

    fn name(&self) -> &'static str;

    fn supports_tls_fingerprint(&self) -> bool {
        false
    }

    fn max_response_time(&self) -> Duration {
        Duration::from_secs(30)
    }
}
```

### Engine Router

The `EngineRouter` (`src/engines/router/mod.rs`,目录实现) holds a `Vec<Arc<dyn ScraperEngine>>` and selects engines. The internal strategy field 在 `EngineRouter::new` 中是**硬编码默认** `LoadBalancingStrategy::SmartHybrid`——settings 侧没有可配置入口,生产装配不调用 setter(`set_strategy` 等 setter 存在但仅在测试中使用):

```rust
// src/engines/router/mod.rs:79,86-186
pub enum LoadBalancingStrategy {
    RoundRobin,
    WeightedRoundRobin,
    LeastConnections,
    FastestResponse,
    Random,
    SmartHybrid,  // hard-coded default — not settable via config
}
```

其余内部参数同样为构造期常量:`max_engine_attempts = 3`、`max_retries = 5`、`race_mode_enabled = false`(顺序回退为默认模式,竞速模式为显式启用)、`feature_filter_enabled = true`、`dynamic_threshold_factor = 1.0`。

`src/engines/router/` 目录内含四个文件:`mod.rs`(路由器主体)、`engine_selector.rs`(同步候选引擎过滤)、`route_race.rs`(并发竞速)、`route_sequential.rs`(顺序回退)。

The router:
1. Filters engines by `support_score` (feature-based filtering,同步过滤在 `engine_selector.rs`)
2. Applies the load balancing strategy(`SmartHybrid`,构造期硬编码)
3. Falls back to sequential engines on failure (`route_sequential.rs`)
4. Supports race mode (concurrent execution, return first success, `route_race.rs`)
5. race mode 胜出后调用 `hedge_controller.record_latency(response_time)` 更新 EMA/方差,用于后续 P84 阈值估算(详见 [Hedge 请求副本控制器](#hedge-请求副本控制器))
6. (`engine-moli` 特性)评分后经 `apply_memory_policy` 做内存水位前置修正(见下节)

### 引擎矩阵与内存水位调度(engine-moli)

<a id="引擎矩阵与内存水位调度"></a>

引擎能力矩阵(`support_score` 口径,均为 GET-only;actions=交互动作支持):

| 引擎 | feature | needs_js | needs_screenshot | actions | TLS 指纹 | 备注 |
|------|---------|----------|------------------|---------|----------|------|
| Reqwest | — | 0 | 0 | 0 | 0 | 静态抓取基线 |
| Playwright | `engine-playwright` | 100 | 100 | ✅ | 0 | 质量基准;BrowserPool max_instances=5 |
| FlareSolverr (Full/Cdp/Tls) | `engine-flaresolverr` | 60-100(按模式) | 0-80 | 90 | Tls 模式 | 反爬/挑战页 |
| Wreq | `engine-tls-fingerprint` | 0 | 0 | 0 | ✅ | 真实 JA3/JA4 |
| MLLM | `engine-mllm` | 视觉导航 | ✅ | ✅ | 0 | 自主导航 |
| Moli | `engine-moli` | 90 | 80 | 0(v1 未实现,诚实出局) | 0 | CDP 控制外部 `moli serve --layout`(端点模式,进程归部署方);响应级等待变体返回 `Unsupported`(Network 事件面不可达) |

内存水位调度器(`src/engines/memory_watermark.rs`,`[engines.memory_scheduler]` 配置):sysinfo 周期采样(默认 2000ms)系统可用内存 → 三档水位(AtomicU8 镜像无锁读):

| 水位 | 判定(默认阈值) | EngineRouter 行为 | BrowserPool 行为 |
|------|------------------|-------------------|------------------|
| Abundant | ≥ 2048 MiB 可用 | 原策略不变(Playwright 优先,moli 顺序回退) | 无限制 |
| Tight | < 2048 MiB | moli `support_score` +30(封顶 255);其余不改分 | 复用在运实例;总量不增(冷池放行首实例) |
| Critical | < 512 MiB | 浏览器系引擎归 0(仅 moli 保留承担 JS);无 moli 时按 `fallback_to_static` 回退静态抓取(响应标记 `js_degraded=true`)或快速失败 `MemoryConstrained` | 拒绝新 acquire + 一次性回收空闲实例(水位回落复位) |

分层契约:router 是精确策略(needs_js 维度修正分数),池是粗粒度安全阀(Critical 全拒 + Tight 控容量)——两个控制点作用面刻意不同。非 `needs_js` 请求不受任何水位影响。metrics:`crawlrs_memory_level`(gauge 0/1/2)、`crawlrs_engine_policy_overrides_total{engine,level}`。

### Hedge 请求副本控制器

<a id="hedge-请求副本控制器"></a>

**Location:** `src/utils/hedge.rs`

**背景(design.md §17):** 移植 spider `hedge.rs`,基于 EMA(指数移动平均)+ 方差估算 P84 延迟阈值,超阈值时建议发送副本请求降尾延迟。

**核心算法:**

- EMA:`EMA_new = α·x + (1-α)·EMA_old`
- 方差:`Var_new = (1-α)·Var_old + α·(x-EMA_new)·(x-EMA_old)`(指数加权移动方差,递推式)
- P84 阈值:`P84 = EMA + σ_multiplier · sqrt(Var)`(标准正态分布 P84 ≈ μ+σ)

**并发模型:** 使用 `parking_lot::Mutex<HedgeState>` 保护 `(ema, var, sample_count)` 三元组原子更新。非热路径(race 胜出后单次记录),Mutex 开销相对 race 100ms+ 网络耗时占比 < 0.0001%。

**接入路径:**

```rust
// EngineRouter::route_race_mode 胜出后记录延迟
Ok((engine_name, response, response_time)) => {
    self.hedge_controller.record_latency(response_time);
    Ok(response)
}

// 顺序路径未来可调用 should_hedge 决策副本触发
if router.hedge_controller().should_hedge(elapsed) {
    // 发送副本请求...
}
```

**样本来源限制(M-2):** 当前 `record_latency` 仅在 race 胜出后调用,记录 `min(各引擎延迟)` 分布。**不可直接用于顺序路径 P84 估算**(顺序路径是单引擎全延迟,分布完全不同,复用会导致阈值系统性偏低)。顺序路径接入时需独立的 HedgeController 实例。

### WaitFor 策略

<a id="waitfor-策略"></a>

**Location:** 枚举定义在 `src/engines/engine_client.rs`,实现在 `src/engines/wait.rs`(`engine-playwright` feature 门控)

**背景(design.md §17):** 替代原有 `sync_wait_ms` 固定 sleep,提供条件式等待:满足条件立即返回,超时返回错误,避免无谓阻塞。

```rust
pub enum WaitFor {
    NetworkIdle,           // 等待网络空闲(无新请求持续 500ms)
    Selector(String),      // 等待指定 CSS selector 出现在 DOM 中
    DomStable(Duration),   // 等待 DOM 稳定(无变化持续指定时长,上限 60s)
}
```

**接入路径:** `InternalScrapeRequest.wait_for: Option<WaitFor>` 字段,Playwright 引擎在 `goto` 后调用 `request.wait_for.unwrap_or_default().wait(&page, timeout)`。

**响应级变体(wait-signals,吸收 moli `--wait-response-*` 语义):** `ResponseUrl(String)`(响应 URL 匹配 regex)与 `ResponseBody(String)`(响应体包含子串,单响应体匹配检查上限 10 MiB)。regex 在 `ScrapeRequest::to_internal` 桥接层与 `create_scrape` use case 双层前置校验,非法 regex 请求不入队。引擎支持面:Playwright 监听 CDP `Network.responseReceived` 事件流;moli 的 Network 事件面实测不可达(`Network.enable` 被接受但 0 事件),对其显式返回 `EngineError::Unsupported`——不静默吞变体。

**桥接实现:** API DTO 层 `ScrapeRequestOptions.wait_for: Option<u64>`(毫秒)在 `create_scrape` use case 中映射为 `WaitFor::DomStable(Duration::from_millis(ms))`,并钳制到 10s(引擎侧 `WAIT_TIMEOUT_CAP`),防合法输入触发 DomStable 超预算确定性失败(`src/application/use_cases/create_scrape.rs:126-130`)。

### TabPool 浏览器 Tab 复用

<a id="tabpool-浏览器-tab-复用"></a>

**Location:** `src/engines/client/tab_pool.rs`

**背景(design.md §17):** Playwright 引擎每次抓取创建新 Tab 开销大(CDP `Target.createTarget` + `Page.goto`,约 50-200ms),TabPool 在 BrowserPool 之上进一步复用 Page,消除 tab 创建开销。

**实现:** 基于 `DashMap<usize, Page>` + `AtomicUsize` 栈顶指针的 LIFO 无锁栈:

```rust
pub struct TabPool {
    slots: DashMap<usize, Page>,  // slot 索引 → Page
    head: AtomicUsize,           // 单调递增的栈顶指针
    max_size: usize,             // 最大池容量
}
```

**acquire/release 流程:**

- `acquire(&browser)`: 原子递减 `head` 取栈顶 Page;池空时调用 `browser.new_page("about:blank")` 新建。
- `release(page)`: 先将 Page 导航到 `about:blank` 清理状态(5s 超时,超时则 drop 关闭 tab),再压回栈;池满(`head >= max_size`)则直接 drop。

**多 Browser 安全:** TabPool 不绑定特定 Browser,`acquire` 接受 `&Browser` 参数。池空时用传入的 Browser 新建 Page;池非空时弹出的 Page 可能属于其他 Browser。调用方应保证 TabPool 生命周期与单个 Browser 一致(per-Browser 实例化),或在多 Browser 场景下由上层 `BrowserPool` 按 instance_id 路由。

### Page Action Types

```rust
pub enum PageAction {
    Wait { milliseconds: u64 },
    Click { selector: String },
    Scroll { direction: ScrollDirection },
    Input { selector: String, text: String },
}
```

### Engine Types

#### 1. Reqwest Engine

**Use Cases:**
- Static HTML pages
- API responses
- JSON/XML data
- Fast scraping without JS

**Features:**
- HTTP/2 support via rustls
- TLS 1.2/1.3
- Cookie handling
- Custom headers and proxy support
- Gzip/brotli decompression

**Pros:**
- Fastest performance
- Lowest resource usage
- No browser overhead

**Cons:**
- No JavaScript execution
- Limited dynamic content support

#### 2. Playwright Engine (via chromiumoxide 0.9)

**Use Cases:**
- Single Page Applications (SPAs)
- JavaScript-heavy sites
- Sites requiring interactions
- Screenshots

**Features:**
- Full JavaScript execution via Chrome DevTools Protocol
- Page interactions (click, scroll, input)
- Screenshots
- Network interception

**Pros:**
- Full browser capabilities
- Renders dynamic content
- Can handle complex interactions

**Cons:**
- Higher resource usage
- Slower than HTTP client
- Requires Chromium installation

#### 3. FlareSolverr Engine

FlareSolverr merges the original three fire engines (FlareSolverrEngine, FireEngineCdp, FireEngineTls) into a single engine with mode selection:

```rust
pub enum FlareSolverrMode {
    Full,  // Full mode: JS rendering, session mgmt, CAPTCHA detection, screenshots
    Cdp,   // CDP mode: browser automation, TLS fingerprinting
    Tls,   // TLS mode: TLS fingerprinting only, no screenshots
}
```

All three modes share the same FlareSolverr API client implementation, differing only in `support_score` and `name`:

| Mode | `name()` | `supports_tls_fingerprint()` | Screenshots |
|------|----------|------------------------------|-------------|
| Full | `flaresolverr` | No | Yes |
| Cdp | `flaresolverr_cdp` | Yes | Yes |
| Tls | `flaresolverr_tls` | Yes | Rejected |

**Use Cases:**
- Cloudflare-protected sites
- Anti-bot protected sites
- High-anonymity requirements
- Search engine CAPTCHA bypass

**Features (by mode):**
- Full: JS rendering, session management, CAPTCHA detection
- Cdp: TLS fingerprinting, browser automation, CDP protocol
- Tls: TLS fingerprint adversarial, fast execution, no screenshot

---

### Search Subsystem

**Location:** `src/search/`

**职责分层:**

- **smart 层** (`src/search/smart/mod.rs`):`SmartSearchEngine` 并发聚合——`tokio::join!` 同时查询 Baidu/Bing/Sogou 三引擎,经 SimHash 去重(`ResultDeduplicator`)→ RRF 融合(`RRFFuser`,k=60)→ `RelevanceScorer`(TF-IDF + 新鲜度)评分排序;单引擎 fan-out 超时按失败降级(部分结果容忍),出站 per-engine 最小间隔 gate 防止触发搜索引擎封禁
- **client 层** (`src/search/client/{baidu,bing,exa,sogou,tavily,searxng}.rs`):各引擎实现(HTML 解析、URL 提取、XSS 防护、CSS 选择器管理),`html_parser.rs` 提供共享的 Google 风格 `HtmlParser`(`once_cell::sync::Lazy` 全局单例,避免重复编译 CSS 选择器);`searxng.rs` 是公共实例 JSON API 适配器(实例从实例服务端口获取,不直连 DB)

#### SearXNG 实例生命周期

`search.searxng.enabled = true` 时启用(默认关闭)。登记簿 `searxng_instances` 是唯一共享真相,进程内热池只是缓存:

```mermaid
flowchart LR
    DS[searx.space<br/>instances.json] -->|采集轮<br/>advisory lock 防重| C[准入谓词<br/>HTTPS / 非内网 / normal / uptime]
    C -->|upsert 幂等| REG[(searxng_instances)]
    SEED[seed_urls] -->|upsert_seed| REG
    REG -->|候选/冷却| PROBE[探活<br/>GET /search?format=json]
    PROBE -->|200 + results[]| POOL[热池<br/>target=3]
    PROBE -->|失败| CD[cooldown]
    CD -->|连续失败达阈值| RET[retired]
    POOL -->|搜索请求失败| CB[熔断 searxng:url]
    CB -->|逐出 + 异步晋升| PROBE
    POOL --> SE[引擎/搜索路径]
```

- **采集**:周期拉取 `collector.data_url` → 宽容 serde(全字段可缺)→ 准入谓词 → 幂等 upsert;缺席数据集的 `searx_space` 实例累计 `dataset_miss_count`,超 `retire_miss_cycles` 转 `retired`(`seed` 来源永不因缺席退役)
- **探活**:热池全体 + 冷却期满的 `cooldown` + 前 `candidate_batch` 个候选,按状态机 `candidate → active → cooldown → retired` 迁移(`retired` 重新出现在数据集并探活通过可回 `active`)
- **熔断与故障切换**:专属 `CircuitBreaker` 实例(键 `searxng:<url>`),失败达阈值/熔断开 → 逐出 + 转 `cooldown` → 异步探测替补回池;单次搜索最多换 `max_failover_attempts` 个实例
- **降级口径**:池满 `Healthy`、非空不足 `Degraded`(+warn)、空 `Unhealthy`;「3 实例」是控制回路目标,非对外部第三方的硬保证
- **管理 API**:`GET /v1/searxng/instances` / `GET /v1/searxng/pool` / `POST /v1/searxng/instances/collect`(auth 保护组,见 `src/presentation/handlers/searxng_handler.rs`)

---

## 🚀 Crawl Capability Enhancement Modules

> 以下模块由 `crawler-capability-absorption` 变更引入(0.2.0)。

### Anti-bot Detection (`src/engines/antibot/`)

三层分类器移植自 crawl4ai `antibot_detector.py`,gated `antibot` 特性:

- `patterns.rs`:Tier1(20+ WAF 结构标记:Cloudflare/Akamai/PerimeterX/DataDome 等)+ Tier2(通用词字面量集,构建 `aho_corasick::AhoCorasick`)+ Tier3 结构正则
- `classifier.rs`:`AntiBotTech` 枚举 + `Detection{tech,reason,needs_browser}` + `classify(status,body,headers,url)->Option<Detection>`
- 路由集成:`EngineRouter::route_internal` 在引擎成功分支调用 `classify`,命中 `needs_browser` 时强制后续 attempt `needs_js=true` 改派浏览器引擎

### HTTP→Chrome Upgrade Probe (`src/engines/upgrade_probe.rs`)

`JsUpgradeProbe` 通过强/弱信号评分探测 SPA 空壳:

- 强信号(score+=10):`__NUXT_DATA__`/`__NEXT_DATA__`/`window.__INITIAL_STATE__`/空 root+hydration
- 弱信号(score+=1):非追踪 `<script src>`(排除 ga/gtm/analytics/pixel)
- `score>=threshold` 时 `EngineRouter` 以 `needs_js=true` 重新改派 Playwright
- 升级死端降级:`needs_js=true` 重选引擎为空(未注册任何 JS-capable 引擎,如仅 reqwest 的嵌入形态)时,降级返回已抓取的原始 HTTP 响应并 `warn!`(含 probe score/reason/URL),按成功语义记账(stats/熔断/host 纪律/session Cookie 捕获)——不再以 `No suitable engines available` 硬失败并丢弃响应

### Memory-aware Scheduler (`src/workers/scheduler/`)

- `memory_scheduler.rs`:`MemoryState{Normal,Pressure,Critical}` + `MemoryScheduler` 复用 `SystemMonitorTrait`,`admit()` 返回 `Admission::{Proceed,Defer,Reschedule}`
- `priority_queue.rs`:`BinaryHeap<ScheduledTask>` 按 `effective_priority = base + waited_secs/aging_factor` 排序,防饿死
- 接入 `scrape_worker::process_task`:获取并发许可前调用 `admit()`

### UA Pool (`src/utils/ua_pool.rs`)

`UaProfile{ua,accept_language,sec_ch_ua,platform,viewport,mobile}` + `UaPool{desktop,mobile}`:

- 内置 20+ 桌面 + 移动真实 profile,每 profile 绑定一致的 UA/Accept-Language/sec-ch-ua/viewport
- `pick(mobile)`/`pick_seeded(seed,mobile)`:同 seed 稳定返回
- 已集成到 `ReqwestEngine`(header 设置)和 `PlaywrightEngine`(UA + viewport 设置)

### Smart Retry (`src/utils/retry/` + `src/utils/backoff.rs`)

- `tracker.rs`:`RetryReason{Transient,FeatureToggle,AntiBot}` + `RetryTracker` 各 reason 独立计数与上限
- `directive.rs`:`RetryDirective{rotate_proxy,rotate_ua,change_viewport,enable_stealth,force_browser}` + `for_attempt(reason,attempt)` 递增升级
- `backoff.rs`:full-jitter 退避(`cap = min(base*2^attempt, max)`,均匀 `[0, cap]`)
- 已集成到 `scrape_worker` 错误路径:分类错误 → 计算指令 → reason-specific 限制检查

### JS Injection (`src/engines/js_inject/`)

gated `engine-playwright`:

- `scripts/`:`flatten_shadow_dom.js`/`navigator_overrider.js`/`remove_consent_popups.js`/`remove_overlay_elements.js`(源自 crawl4ai,`include_str!` 嵌入)
- `injector.rs`:`InjectPhase{BeforeLoad,AfterLoad}` + `JsInjector::stealth()`/`cleanup()`/`apply(page,phase)`
- Playwright 引擎导航前注入 stealth,加载后注入 cleanup

### Request Interception (`src/engines/intercept.rs`)

gated `engine-playwright`,R-jsrender-003:

- 广告/追踪域名黑名单(`AD_DOMAIN_BLACKLIST`):命中请求经 CDP `Fetch.failRequest` 中止
- 可选媒体资源拦截(`ResourceType::{Image, Media, Font}`):由 `ScrapeOptions.block_media` 开关
- 拦截计数可观测
- `ScrapeOptions.block_ads` / `ScrapeOptions.block_media` 控制开关,经 `InternalScrapeRequest` 桥接至 Playwright 引擎

### Request Coalescing (`src/utils/coalesce.rs`)

移植 spider `coalesce.rs`:`RequestCoalescer{ in_flight: Arc<DashMap<CoalesceKey, InFlightEntry>> }`:

- `CoalesceKey` = 租户(`Task::team_id`)+ 规范化 URL(scheme/host 小写、去 fragment、去默认端口)+ `ScrapeOptions` 全字段结构化指纹(headers 名小写后按名值排序,`DefaultHasher` 64 位,同一进程内稳定)
- `try_start(&CoalesceKey) -> Proceed(CoalesceGuard) | Wait(broadcast::Receiver, leader_task_id)`
- 跨租户或 headers/proxy 不同一律不合并(M-2:防止等待方按 leader task_id 收养另一租户的结果)
- `STALE_TIMEOUT=120s` + `purge_stale()`
- 经 `CoalesceCoordinator` 接入 `scrape_worker`,同合并键并发仅 1 次实际 fetch

### AIMD Adaptive Concurrency (`src/utils/adaptive_concurrency.rs`)

移植 spider `adaptive_concurrency.rs`:

- `AIMDController`:`AtomicUsize` 无锁,连续 N 次成功 +1,失败减半 clamp min
- `AdaptiveSemaphore`:桥接 `tokio::sync::Semaphore`,`set_target` 经 `add_permits`/`forget_permits`
- 集成 `TeamSemaphore::with_adaptive(...)`,默认关闭(`concurrency.adaptive_enabled=false`)

### Markdown Conversion (`src/domain/services/markdown_service.rs` + `markdown/`)

gated `content` 特性。**三层管线**(html2md-converter-chain 变更后):

```text
HTML(静态抓取或浏览器渲染产物)
  → ① 提取链(`content_extractor/facade.rs`):Trafilatura → DomSmoothie → CSS 规则
       (only_main_content=true 时先裁正文;零改动)
  → ② 转换器(`markdown/`):alchemize html 引擎默认(htmd 基础链 + 自研深链,
       内建空输出防呆/诚实降级/math $TeX$ 保真;[markdown] converter = "alchemize"
       默认;"htmd" 直通诊断臂)
  → ③ 后处理(`workers/markdown_post_processor.rs`):only_main_content 三路 +
       链接绝对化(零改动)
```

- 抽象:`MarkdownConverter{name,convert}` 窄接口(装配面 `markdown::build_converter`
  按 `[markdown].converter` 选择 alchemize 默认臂或 htmd 诊断臂;旧
  `MarkdownServiceTrait` 已删除——一步到位,不留兼容层)
- 兜底语义:alchemize 内建(htmd 基础链 + 自研深链;空输出防呆、双失败诚实报错、
  math `$TeX$` 保真路由);旧 `ConverterChain`(moli 主 + htmd 兜底)与其
  `crawlrs_md_converter_fallback_total` 指标已随 moli-html2md 退役移除
- 依赖:`alchemize`(自研 html 引擎仓,`html` 特性)为 crates.io registry 依赖
  (0.1 稳定版,minor 精度);同仓 `audio` 特性才需的 vecboost 由本仓按需独立引入
  (`rag-local`),干净 checkout 即可解析,无需 sibling 检出
- `ScrapeResponse.markdown: Option<String>`;`scrape_worker` 经
  `MarkdownPostProcessor` 在 formats 含 `"markdown"` 时生成

**诚实边界(computed CSS 钩子未接):** `Dom::control_value` / `has_block_layout` /
`has_text_boundary` 保留 trait 默认实现(None/false)——静态 HTML 无计算样式,
接了就是假数据。`has_block_layout` 缺失意味着「样式化链接(如按钮式导航)」在
纯结构视角下可能少断块、inline 边界(tab/badge)折叠处理弱于全渲染视角;
`control_value` 缺失意味着受 JS 控制过 value 的表单控件输出源默认值。
重开条件:MoliEngine 全渲染视角落地后经实测评估是否提供 computed style 注入面。

### Content Extraction (`src/domain/services/content_extractor/`)

gated `trafilatura`/`dom-smoothie`/`extractors`:

- `traits.rs`:`ContentExtractor` trait + `ExtractedContent{text,title,author,confidence,page_type}`
- `trafilatura_extractor.rs`(主路径)→ `dom_smoothie_extractor.rs`(回退)→ `css_rule_extractor.rs`(兆底)
- `facade.rs`:`ContentExtractionFacade` 按优先级路由,`confidence<0.7` 触发 LLM 回退

### LLM Carrier Boundary (`src/domain/services/llm/`)

- `mod.rs` 的 `LLMServiceTrait` 是 crawlrs 唯一 LLM 边界:`extract_data` / `extract_data_chunked` / `extract_data_with_provenance`,接口签名对宿主稳定。
- 供应商访问不自持:`LLMService` / `provider_adapter` / `llm_provider_strategy` 已删除;视觉面经 fluxweir `FluxweirClient::chat` 承载(`vision_adapter` 由 `engine-mllm` 门控,genai 已退役)。
- 默认装配 `ExplicitErrorCarrier`:未注入 carrier 时三入口均返回 i18n 双语结构化错误(`domain-error-llm-carrier-not-injected`),不 panic、不静默降级。
- 宿主注入三路径:`Services.llm_service` pub 字段、`init_llm_service` 工厂替换、`CrawlRsState::from_kit`。
- 编排工具保留:`prompt_builder` / `chunker`(F8)/ `provenance`(F9)以公开模块形式供注入 carrier 复用;`[llm]` 只保留 `model` / `request_timeout_secs` / `chunking` / `provenance_*`。

### URL Normalization & Layered Dedup (`src/utils/url_normalizer.rs` + `src/utils/dedup/`)

- `url_normalizer.rs`:小写 host/去 fragment/统一 trailing slash/query 排序/可选去 query/`permutations`
- `dedup/bloom.rs` + `dedup/interner.rs`:Bloom⊕Interner 分层去重
- 接入 `scrape_worker::extract_and_queue_links`:Bloom 阴性直接入队,阳性回落 DB 精确校验

### Proxy Pool (`src/engines/proxy_pool.rs`)

- `ProxyEntry{url,healthy,cooldown,category}` + `ProxyPool`(RR `next(category)` + `sticky(session_id)`+TTL + `mark_failure/success`)
- 实现 `ProxyProvider` trait,`ReqwestEngine` 依赖抽象 trait
- `ProxySettings` 扩展 `urls: Vec<String>` + `strategy`(保留单 `url` 向后兼容)

### Advanced Cache Modes (`src/common/cache_mode.rs`)

- `CacheMode{Enabled,Disabled,ReadOnly,Bypass}` + `CacheContext{url,method,mode}`(原 WriteOnly 语义已并入 Bypass)
- `should_read()`/`should_write()`/`is_cacheable()`(data:/blob:/非幂等→false)
- `ScrapeOptions.cache_mode: Option<CacheMode>` + DTO `cache_mode`/`bypass_cache` 桥接
- `scrape_worker` 读写缓存前经 `CacheContext` 门控

### Waterfall/MRT Timeout

- `ScraperEngine::max_response_time() -> Duration`:各引擎覆写(fetch:5s/playwright:30s/cdp:30s/tls:15s)
- `EngineRouter::route_internal` 用 `min(remaining, engine.mrt())` 包裹单引擎调用,超 MRT 切下一引擎
- `EngineError::EngineMrtExceeded{engine,mrt}` 变体,`is_retryable()=true`

### Deep Crawling (`src/workers/crawl/`)

- `filters.rs`:`UrlFilter` trait + `FilterChain` + `DomainFilter`/`ContentTypeFilter`/`UrlPatternFilter`
- `scorers.rs`:`UrlScorer` trait + `CompositeScorer` + `KeywordRelevanceScorer`/`PathDepthScorer`
- `frontier.rs`:`ScoredURL` + `BinaryHeap` 优先级 + 域名 round-robin
- `adaptive.rs`:`AdaptiveStrategy`(BM25/覆盖率/饱和度)+ `StopCondition`(最大页数/置信度/饱和度/无待处理链接/KG 覆盖率)+ `DrlConfig`
- `knowledge_graph.rs`:`KnowledgeGraphAccumulator` 知识图谱累积 + Chao1 覆盖率估计 + 结构空洞检测
- `drl_policy.rs`:`DrlPolicy` ONNX 推理 + `HeuristicPolicy` 启发式退化
- `scrape_worker::handle_crawl_success` 集成 `StopCondition` 检查,命中时提前终止 crawl

### TLS Fingerprint Engine (`src/engines/client/wreq_engine.rs`)

`WreqEngine` 基于 `wreq` crate 实现 TLS 指纹伪装,通过自定义 TLS ClientHello 的 JA3/JA4 指纹模拟真实浏览器。

- **Feature gate**: `engine-tls-fingerprint`
- **核心能力**: 自定义 TLS 扩展顺序、密码套件、曲线列表,生成与 Chrome/Firefox/Safari 一致的 JA3 指纹
- **集成路径**: 注册为 `ScraperEngine` 实现,`support_score` 在 `needs_tls_fingerprint=true` 时返回 100
- **配置**: `[engines.tls_fingerprint]` 控制默认指纹 profile 和启用状态

### MLLM Engine (`src/engines/client/mllm.rs` + `src/engines/mllm/`)

`MllmEngine` 实现多模态 LLM 自主导航爬取,通过视觉模型理解页面内容并决定下一步动作。引擎本体在 `src/engines/client/mllm.rs`(与其他引擎同一层,注册为 `ScraperEngine` 实现);其内部模块独立放在 `src/engines/mllm/` 目录下:

- `mod.rs` — 模块装配
- `ax_tree.rs` — 无障碍树抽取
- `decision.rs` — 动作决策
- `action_executor.rs` — 点击/滚动/输入执行
- `config.rs` — 配置

- **Feature gate**: `engine-mllm`
- **架构**: Vision Adapter(截图→视觉模型)→ Decision(动作决策)→ Action Executor(执行点击/滚动/输入)
- **Agentic Loop**: 最大 N 轮迭代,每轮截图→分析→动作,直到提取目标内容或达到终止条件

```mermaid
flowchart LR
    A[截图] --> B[Vision Adapter]
    B --> C[MLLM Decision]
    C --> D[Action Executor]
    D --> E{目标达成?}
    E -->|否| A
    E -->|是| F[返回结果]
```

### RAG 增强提取 (`src/domain/services/rag_strategy.rs`)

将 HTML 文档按 DOM 语义边界分块,通过向量嵌入 + 相似度检索找到最相关片段,再送 LLM 精确提取。

- **SemanticChunker**: 按 `<article>`, `<section>`, `<div>`, `<table>` 等 DOM 边界分块,表格/列表不截断
- **EmbeddingProvider**: trait 抽象(HTTP API / 本地模型),支持 oxcache 持久化
- **VectorStore**: 内存向量存储 + 余弦相似度 top-K 检索
- **集成**: `ExtractionServiceTrait::extract_with_rag` 作为第三种提取模式(与 CSS/XPath、LLM 直接提取并列)

### 可观测性增强 (Prometheus Metrics)

新增 5 个 Prometheus 指标(`src/infrastructure/observability/metrics.rs`):

| 指标 | 类型 | 标签 | 说明 |
|------|------|------|------|
| `crawlrs_queue_depth` | Gauge | `queue_type` | 任务队列深度(AtomicU64 近似追踪) |
| `crawlrs_engine_success_total` | Counter | `engine`, `result` | 引擎成功/失败计数 |
| `crawlrs_engine_duration_seconds` | Histogram | `engine` | 引擎请求耗时分布 |
| `crawlrs_cache_hit_total` | Counter | `cache_type`, `result` | 缓存命中/未命中(search/dns/regex/generic) |
| `crawlrs_webhook_delivery_total` | Counter | `result`, `status_code` | Webhook 投递成功/失败 |

---

## 📬 Queue System

### Architecture

The queue system uses a **trait-based** approach with a PostgreSQL-backed implementation:

```rust
#[async_trait]
pub trait TaskQueue: Send + Sync {
    async fn enqueue(&self, task: Task) -> Result<Task, QueueError>;
    async fn dequeue(&self, worker_id: crate::domain::id::Id) -> Result<Option<Task>, QueueError>;
    async fn complete(&self, task_id: crate::domain::id::Id) -> Result<(), QueueError>;
    async fn fail(&self, task_id: crate::domain::id::Id) -> Result<(), QueueError>;
    async fn cancel(&self, task_id: crate::domain::id::Id) -> Result<(), QueueError>;
}
```

所有 ID 为 `crate::domain::id::Id`(NebulaId 域 ID),不是旧式 UUID 值。

**PostgresTaskQueue** wraps the `TaskRepository` trait:

```rust
pub struct PostgresTaskQueue {
    pub repository: Arc<dyn TaskRepository>,
}
```

The `TaskRepository` provides `acquire_next(worker_id)` which uses `FOR UPDATE SKIP LOCKED` semantics for safe concurrent worker access. Ordering is `priority ASC, created_at ASC`(数值越小越优先),由 `migrations/003` 的两个 partial index(`idx_tasks_acquire_queued` / `idx_tasks_acquire_stale`)支撑有序取任务。详见 [Queue Architecture](#-queue-system)。

### Worker Types

实际运行 **4 类 worker**:

| Worker | Purpose | 启动方 | 驱动方式 |
|--------|---------|--------|----------|
| `scrape_worker` | Process scrape tasks via EngineClient | `manager`(N 个副本) | 自有 `run`(不经 AbstractWorker) |
| `webhook_worker` | Deliver webhook events to configured URLs | `spawn_common_workers` | `AbstractWorker` 模板 |
| `backlog_worker` | Reprocess expired/pending tasks | `spawn_common_workers` | `AbstractWorker` 模板 |
| `expiration_worker` | Expire stale tasks past their TTL | `spawn_common_workers` | `AbstractWorker` 模板 |

后三者(webhook/backlog/expiration)由 `src/bootstrap/workers.rs:20-57` 的 `spawn_common_workers` 启动,API 模式与 worker 模式共享,**不经 manager**;scrape_worker 由 manager 启动。`src/workers/task_state_machine.rs` 全仓零调用方(死代码,无运行时接线)。

**Worker 模板**:`AbstractWorker<P>` 只驱动通用 worker(webhook/backlog/expiration);scrape_worker 用自己的 `run` 循环服务于任务队列:

```rust
pub struct AbstractWorker<P>
where
    P: WorkerProcess + Send + Sync,
{
    processor: Arc<P>,
    interval: Duration,
}
```

Worker lifecycle:
1. API/worker 进程启动时 `spawn_common_workers` 拉起 webhook/backlog/expiration;worker 模式下 manager 另起 N 个 scrape_worker
2. Each worker runs its `process()` on a configurable interval
3. Workers share the same TaskRepository for task coordination via `FOR UPDATE SKIP LOCKED`
4. **Graceful shutdown** 由 `ShutdownCoordinator`(`src/workers/shutdown.rs:35-44`)协调:内嵌 trait-kit 的 `AsyncShutdownCoordinator` 做分阶段停机 hook,`listen_unix_signals` 捕获 SIGTERM/SIGINT 置位关闭 flag;宽限期(默认 30s)内 worker 排空活跃任务,`ShutdownPhase::DrainQueue` 阶段 hook 触发 `rollback_pending_tasks` 把未完成任务回滚为 `queued`,全部见 `src/main.rs:130-203`。broadcast channel 只用于 RequestCoalescer 的同 URL 并发等待,不参与停机流程

---

## 💾 Caching Strategy

### Architecture

Caching uses **oxcache 0.5** with a single in-memory backend (moka). There is no Redis or multi-tier cache.

```mermaid
flowchart TD
    subgraph L1 [oxcache in-memory (moka)]
        L1Feat1[oxcache::Cache K,V]
        L1Feat2[Configurable capacity & TTL]
        L1Feat3[LRU eviction]
        L1Feat4[Bloom filter]
        L1Feat5[Batch-write]
        L1Feat6[Metrics]
    end

    subgraph Source [Source]
        S1[Actual scraping]
        S2[Database queries]
        S3[DNS resolution]
    end

    L1 -->|"miss"| Source
```

### Cache Types

**Search Result Cache:**
```
search:query:lang=en:country=US:limit=10
TTL: configurable (default 60s)
```

**DNS Cache:**
```
dns:hostname:port
TTL: configurable (default 300s)
```

**Regex Cache:**
```
regex:{hash(pattern)}
TTL: configurable (default 600s)
```

### Cache Service Trait

```rust
#[async_trait]
pub trait CacheService: Send + Sync {
    fn get(&self, key: &str) -> Pin<Box<dyn Future<Output = Result<Option<String>>> + Send + '_>>;
    fn set(&self, key: &str, value: &str, ttl_seconds: u64) -> Pin<Box<dyn Future<Output = Result<()>> + Send + '_>>;
    fn delete(&self, key: &str) -> Pin<Box<dyn Future<Output = Result<()>> + Send + '_>>;
    fn exists(&self, key: &str) -> Pin<Box<dyn Future<Output = Result<bool>> + Send + '_>>;
}
```

### Cache Invalidation

- **Time-based TTL** - Automatic expiration per entry
- **Capacity-based** - LRU eviction when capacity exceeded
- **Bloom filter** - Quick negative existence check before cache lookup

---

## ⏱️ Rate Limiting

**Technology:** limiteron 0.3 (PostgreSQL-backed)

Rate limiting is implemented using **limiteron 0.3**, which provides PostgreSQL-backed token bucket rate limiting with the following features:

| Feature | Purpose |
|---------|---------|
| Token bucket | Per-key rate limiting with refill rate |
| Ban manager | Automatic ban on repeated violations |
| Quota control | Per-team quota management |
| Circuit breaker | Stop-the-world on sustained overload |
| Telemetry | Request tracking and monitoring |
| Parallel checker | High-concurrency rate limit checks |
| Audit log | Rate limit violation logging |

### Middleware 实现

`src/presentation/middleware/rate_limit_middleware.rs` 是**唯一的**限流中间件。历史上曾并存另外两个平行方案(`src/presentation/middleware/mod.rs:21-24` 的历史注释记录了它们因无路由挂载点、零调用方而被移除的事实),现已不在代码中。该中间件本身只负责 429 判定与 `Retry-After` 响应头,实际速率判断经 `RateLimitingService` trait 委托给 `LimiteronService`(limiteron 0.3,PostgreSQL 后端):

```text
HTTP request → rate_limit_middleware (429 + Retry-After)
              → RateLimitingService trait
                ├── rate-limit on:  LimiteronService(PostgreSQL)
                └── rate-limit off: NoopRateLimitingService(always allow)
```

全仓无 Redis 依赖,因此不存在跨实例共享的分布式限流变体。tower 原生层语义(429/403 + `RateLimit-*` 响应头)由 limiteron 的 `tower-middleware` 提供,crawlrs 尚未接入。

### Multi-Level Limiting

```mermaid
flowchart TD
    subgraph Global [Global Rate Limit - System-wide]
        G1[Total requests/second]
        G2[Max concurrent connections]
        G3[Prevents system overload]
    end

    subgraph PerKey [Per-API Key Rate Limit]
        K1[Requests per minute]
        K2[Burst allowance - token bucket]
        K3[Configurable per key]
    end

    subgraph PerTeam [Per-Team Concurrency Limit]
        T1[Max concurrent tasks]
        T2[Semaphore-based enforcement]
        T3[Resource allocation per team]
    end

    Global --> PerKey
    PerKey --> PerTeam
```

### Concurrency Control

```rust
pub struct TeamSemaphore {
    permits: Arc<DashMap<crate::domain::id::Id, Arc<tokio::sync::Semaphore>>>,
    max_concurrent: Arc<DashMap<crate::domain::id::Id, usize>>,
}
```

The `TeamSemaphore` enforces per-team concurrency limits. Each team has a dedicated semaphore; the `team_semaphore_middleware` acquires/releases permits per request.

### Limiteron Service

```rust
pub struct LimiteronService {
    governor: Arc<Governor>,
    config: RateLimitingConfig,
    task_repository: Arc<dyn TaskRepository>,
    tasks_backlog_repository: Arc<dyn TasksBacklogRepository>,
    credits_repository: Arc<dyn CreditsRepository>,
}
```

Configured with rules for:
- Default team rate limits
- API key-level rate limits
- Burst allowances

---

## 🔁 自研库反向需求

自研库矩阵(limiteron / garrison / oxcache / sdforge,同作者共享 rc 链)已承载认证、限流、缓存、DTO 等基础面。本节登记 **crawlrs 侧已按库能力收敛后仍存在的库侧缺口**——均为向 limiteron / sdforge 反向提需求的依据;每条附本变更(`selfdev-lib-adoption-r1`,2026-10)中的用途与目标收敛路径。oxcache 在本变更消费面(memory 缓存 + `get_or_with_ttl` 内建 single-flight + `BloomFilter`)无缺口。

### limiteron

| # | 需求 | 现状(过渡态) | 用途 / 收敛路径 |
|---|------|---------------|----------------|
| L1 | `KeyedCircuitBreaker`:管理器级键控熔断池(对齐 `LimiterManager::get_concurrency_limiter` 形态) | crawlrs `DashMap<String, Arc<ManualCircuitBreaker>>` 薄封装(`src/engines/circuit_breaker.rs`),库只提供单实例 | 引擎熔断(S6)。库补齐键控池后删除 crawlrs DashMap 层,改经 manager 取用 |
| L2 | 滑窗绝对失败计数模式(对齐 crawlrs 迁移前语义) | 库为累计比例触发(`total >= min_requests` 且 `failures/total >= 0.5`),混合流量下灵敏度提高(行为变化已登记 CHANGELOG) | 引擎熔断(S6)。库补齐滑窗模式后按引擎恢复"窗口内 N 次失败跳闸"语义 |
| L3 | 同步 `try_admit` / state 快照 | crawlrs `AtomicU8` 镜像(record 路径异步刷新,selector 同步读,`src/engines/circuit_breaker.rs`) | engine_selector 同步候选过滤(禁 sync 上下文 block_on)。库提供同步 state 快照后删除镜像层 |
| L4 | `ConcurrencyLimiter` 动态 resize | crawlrs AIMD 保留自有 `AdaptiveSemaphore` 外层(`src/domain/services/team_semaphore.rs`);库 manager 同 key 参数不一致直接 panic,与动态 resize 互斥 | 团队并发 AIMD(S1)。库补齐 resize 后 AIMD 分支可全量池化 |
| L5 | `ConcurrencyLimiter` 公开 owned / 非阻塞获取入口(`try_acquire` 目前仅存在于库 `#[cfg(test)]` 私有 impl)+ 键控池许可可从 `&self` 逃逸借用 | `TeamSemaphore` Fixed 模式保留 crawlrs `DashMap + Arc<Semaphore>` 实现(`acquire` 返回 owned permit、`try_acquire` 同步返回 `Option`,调用方零改动) | 团队并发 Fixed 池(S1 判定不可吸收的直接原因)。库补齐 owned/try_acquire 后 Fixed 分支可下沉 manager 键控池 |

### sdforge

| # | 需求 | 现状(过渡态) | 用途 / 收敛路径 |
|---|------|---------------|----------------|
| S1 | `unified::http_status_for` 映射集补 502/504(BAD_GATEWAY / GATEWAY_TIMEOUT 对应 `ApiError` 变体) | `CrawlRsError::Network`/`Engine`→502、`Timeout`→504 为显式残留三分支(`src/common/error.rs`),其余 14 变体已委托库映射表 | HTTP 状态码映射(S3)。库补齐后删除残留分支,`status_code` 全量委托 |
| S2 | ~~`TranslationStore` 支持 fluent 资源装载 + 参数插值 + locale→default→key 回退链~~(消息层已收敛:查询统一走 `sdforge::i18n::translate_for`) | crawlrs `I18nBundle` 仅保留 locale 元数据与协商(`locales/*.ftl` 编译期内嵌,`src/i18n/catalog.rs` 启动期行解析注册进宿主注册表;fluent-bundle 已退役);格式化层(数字/日期/复数/排序)无消费场景,`I18nFormatter` 未引入 | 错误消息 i18n(S3)。格式化场景出现时再评估 trait-kit `i18n` feature |

> 收敛纪律:库侧补齐后按本表逐项二次收敛(删除 crawlrs 薄封装/残留分支),并在对应 specmark 变更中登记。本表只登记**消费面缺口**,不包含库内部缺陷。

---

## 🔒 Security Model

### Authentication

> **0.2.0 变更(`garrison-auth-migration`):** 认证引擎由 garrison v0.9 接管。旧的手写 SHA-256 `api_key_hash` 查表、`AuthRateLimiter`、`AuthScopeService` 已被删除;`api_keys.key_hash` / `scopes` 表标记为弃用(`deprecated_at` 列)。

**Garrison 接管的认证流程:**

1. 客户端在 `Authorization: Bearer` 头中传入 garrison 签发的 key,格式为 `garrison_key_id.garrison_secret`
2. `auth_middleware_inner` 调用 `GarrisonUtil::check_api_key` 校验 key、过期、状态、RBAC 权限
3. garrison 内部维护 oxcache + 自管 postgres schema,命中时无 DB 往返
4. `auth_bridge::bridge_to_auth_state` 将 garrison 返回的 `permissions` 通过 `map_perms_to_scope` 映射为 `ApiKeyScope`,注入 `AuthState`
5. crawlrs 侧仅保留 `api_keys` 表的 `id`/`team_id`/`key` 列,用于 `api_key_id→team_id` 反查;该映射由 `TEAM_ID_CACHE` (LRU) 缓存

```rust
pub struct AuthState {
    pub pool: Arc<DbPool>,
    pub team_id: crate::domain::id::Id,
    pub api_key_id: crate::domain::id::Id,
    pub scope: ApiKeyScope,
}
```

**401 / 429 由 garrison `firewall-bruteforce` 触发:**
- 默认策略:5 次失败 / 60 秒窗口 / 300 秒锁定
- 锁定期间所有请求返回 429(带 `Retry-After` 头)
- 失败计数与锁定状态由 garrison 自管 oxcache 持有

### Garrison 认证引擎

garrison v0.9 提供五大组件,crawlrs 通过 `auth` feature 隐式依赖:

| 组件 | 职责 | crawlrs 集成方式 |
|------|------|------------------|
| **DAO** | 自管 postgres schema(`garrison_api_keys` / `garrison_roles` / `garrison_permissions` 等) | 由 garrison 自行迁移,crawlrs 不感知 |
| **Interface** | `GarrisonUtil::check_api_key` / `ApiKeyHandler::generate_with_namespace` | `auth_middleware_inner` 与 `POST /v1/admin/api-keys` handler 直接调用 |
| **Config** | RBAC 角色 / 权限 / tenant 隔离配置 | garrison 预置 RBAC(`tenant_id=0`,所有 team 共享);API Key 经 `POST /v1/admin/api-keys` 或 `CRAWLRS__BOOTSTRAP_ADMIN_API_KEY` 启动引导签发 |
| **RBAC + JWT** | HS256 JWT 签发、角色-权限映射、API Key 生成与校验 | 必填 `CRAWLRS__AUTH__JWT_SECRET`(≥32 字节,弱密钥拒绝启动) |
| **firewall-bruteforce** | 失败计数、窗口锁定、IP 隔离 | 401/429 由 garrison 直接返回,crawlrs 透传 |
| **audit-log** | 认证事件落库(成功 / 失败 / 锁定) | 写入 crawlrs `audit_logs` 表 + garrison 自管 schema |

**预置 RBAC 模型(`tenant_id=0`,所有 team 共享):**

| 权限 | 角色 admin | 角色 user | 角色 read_only |
|------|-----------|-----------|----------------|
| `crawlrs:admin` | ✅ | ❌ | ❌ |
| `crawlrs:write` | ✅ | ✅ | ❌ |
| `crawlrs:read` | ✅ | ✅ | ✅ |

`auth_bridge::map_perms_to_scope` 将上述权限映射回 `ApiKeyScope`(确定性查找表,admin 蕴含 read+write):
- `crawlrs:admin` → `read=true, write=true, admin=true`
- `crawlrs:write` → `write=true`
- `crawlrs:read` → `read=true`

全部通过 `ApiKeyScope::with_custom_limits` 构造(`DEFAULT_SEARCH_LIMIT=100`, `DEFAULT_SCRAPE_LIMIT=50`),不调用 `full_access()`——避免无限制 `u32::MAX` 与项目配额语义冲突。

### Authorization

**Scope-based Access Control:**

```rust
pub async fn scope_middleware(req: Request<Body>, next: Next) -> Result<Response, StatusCode>
```

`scope_middleware`(`src/presentation/middleware/auth_middleware.rs:487`)从 `AuthState.scope` 读取 `ApiKeyScope`,调用 `determine_required_scope(path, method)` 推导所需 `ScopePermission`,再用 `ApiKeyScope::has_permission` 校验。`ScopePermission` 枚举:`Read` / `Write` / `Admin`(`src/domain/auth/mod.rs:35`)。注意该中间件未挂载生产路由。

Scopes: `scrape`, `crawl`, `search`, `extract`, `admin`

> **0.2.0 起**:`AuthState.scope` 由 `auth_bridge::map_perms_to_scope` 从 garrison permissions 桥接而来,下游 `scope_middleware` 无感知。

### 迁移指南(0.2.0 garrison-auth-migration)

**对现有部署的影响:**

1. **旧 API Key 作废**:原 `api_keys.key_hash` (SHA-256) 中的所有 key 失效,客户端必须经 garrison 重新领取
2. **`scopes` 表只读**:原有 scope 映射不再生效;新权限由 garrison RBAC 提供
3. **必填配置**:`CRAWLRS__AUTH__JWT_SECRET` 必须设置(HS256,≥32 字节),弱密钥将拒绝启动

**运维步骤:**

迁移已完成。新 team 通过 `POST /v1/admin/api-keys` 签发 API Key,无需运维工具介入。
开发/测试环境可通过 `CRAWLRS__BOOTSTRAP_ADMIN_API_KEY` 环境变量自动签发 admin key。

**`migrations/005_deprecate_legacy_api_keys.sql` 行为:**
- 仅给 `api_keys` 表追加 `deprecated_at` 列并回填 `NOW()`
- 仅给 `scopes` 表追加 `deprecated_at` 列并回填 `NOW()`
- 不删除任何数据、不删除任何列,便于回滚与历史审计

### SSRF Protection

SSRF validation occurs at two stages:
1. **Static validation** (synchronous pre-filter) - checks URL scheme, hostname patterns, IP ranges
2. **Full validation** (async with DNS resolution) - resolves hostnames, checks resolved IPs

**Blocked patterns (static validation):**
- `http://localhost:*` (loopback hostnames)
- `http://127.*` (IPv4 loopback)
- `http://10.*` (RFC 1918 class A)
- `http://192.168.*` (RFC 1918 class C)
- `http://172.16-31.*` (RFC 1918 class B)
- `http://[::1]*` (IPv6 loopback)
- `http://[fe80:*` (IPv6 link-local)
- `file://*` and `ftp://*` (non-HTTP schemes)

**Blocked ports:**
- 25 (SMTP), 465 (SMTPS), 587 (SMTP submission)
- 3306 (MySQL), 5432 (PostgreSQL), 6379 (Redis), 27017 (MongoDB)

**DNS rebinding protection:**
- Resolve hostname to IPs at request time
- Validate all resolved IPs against private ranges
- Cache DNS results with configurable TTL

```rust
pub enum SsrfValidationResult {
    Safe(ValidatedUrl),
    Blocked { url: String, reason: String },
    RequiresDnsResolution { url: String, hostname: String },
}
```

### Audit Logging

**Logged Events:**
- API requests (endpoint, method, timestamp, team ID)
- Authentication attempts (success/failure)
- Rate limit violations
- Denied requests (SSRF, invalid key, scope violation)
- Task lifecycle events (created, completed, failed)

---

## 🚢 Deployment Architecture

### Single Instance

```mermaid
flowchart TB
    subgraph Container [Docker Container]
        subgraph App [crawlrs Application]
            S1[Axum Server<br/>port 8899]
            S2[Scrape Workers + Common Workers<br/>4 worker types]
        end
    end

    subgraph Resources [External Resources]
        DB[(Postgres Database)]
        RC[(oxcache<br/>in-memory)]
    end

    App --> DB
    App --> RC
```

### Multi-Instance Deployment (Kubernetes)

```mermaid
flowchart TB
    subgraph K8s [Kubernetes Cluster]
        subgraph Pods [Application Pods]
            subgraph Pod1 [Pod 1]
                A1[crawlrs<br/>API + Workers]
            end

            subgraph Pod2 [Pod 2]
                A2[crawlrs<br/>API + Workers]
            end
        end

        subgraph Infrastructure [Infrastructure Services]
            DB[(Postgres<br/>Primary + Replica)]
            MON[Prometheus + Grafana<br/>Metrics collection<br/>Visualization]
        end
    end

    Pod1 --> DB
    Pod2 --> DB
    Pod1 --> MON
    Pod2 --> MON
```

### Deployment Components

1. **Application Pods**
   - Each pod runs both API server and worker pool
   - Horizontal Pod Autoscaler (CPU/memory based)
   - Load balancer (Ingress)
   - Health check endpoints: `/health`, `/ready`, `/metrics`

2. **PostgreSQL**
   - Primary + Read replicas
   - Connection pooling via dbnexus
   - Automated backups

3. **Monitoring**
   - Prometheus metrics scraping (via `metrics-exporter-prometheus`)
   - inklog structured logging
   - Grafana dashboards

---

## 📈 Scalability Considerations

### Horizontal Scaling

**Stateless Workers:**
- Workers share no local state
- Task coordination via PostgreSQL `FOR UPDATE SKIP LOCKED`
- No session affinity required

**Shared State:**
- PostgreSQL is the single source of truth
- oxcache is per-instance (in-memory), not shared

### Scaling Strategy

```mermaid
flowchart TD
    A[Load ↑] --> B[Scale workers]
    B --> C{Check: CPU < 80%,<br/>Memory < 80%}
    C -->|OK| D[Scale up]
    C -->|Not OK| E[Scale database]
```

### Bottleneck Identification

| Metric | Action |
|--------|--------|
| Queue depth > 1000 | Scale worker pods |
| Cache hit < 50% | Increase cache size |
| DB query time > 100ms | Add replicas / optimize queries |
| Memory > 80% | Scale vertically or horizontally |
| CPU > 80% | Scale horizontally |

---

## 🔮 Future Enhancements

### Planned Architecture Improvements

1. **Event-Driven Architecture**
   - Event bus for internal communication
   - Webhook event sourcing

2. **WebSocket Support**
   - Real-time task status updates
   - Push notifications (reduced polling)

3. **Redis Cache Layer**
   - Shared cache across instances
   - Replace per-instance oxcache for multi-pod deployments

---

## 📚 Documentation

- [API Reference](API_REFERENCE.md)
- [User Guide](USER_GUIDE.md)

---

**Last Updated:** 2026-09-15