omnivore-core
Core crawler and knowledge graph engine for Omnivore.
- Fast async crawling with Tokio
- Respectful politeness and robots.txt support
- HTML parsing and extraction utilities
- Pluggable storage and graph building
License
MIT OR Apache-2.0
- Coverage
- 0%
0 out of 607 items documented0 out of 223 items with examples - Size
- Source code size: 298.17 kB This is the summed size of all the files inside the crates.io package for this release.
- Documentation size: 9.04 MB This is the summed size of all files generated by rustdoc for all configured targets
- Links
- Homepage
-
Pranav-Karra-3301/omnivore
1 0 2 - crates.io
- Dependencies
-
- anyhow ^1.0 normal
- arc-swap ^1.7 normal
- async-trait ^0.1 normal
- base64 ^0.22 normal
- bytes ^1.9 normal
- chrono ^0.4 normal
- dashmap ^6.1 normal
- dirs ^5.0 normal
- encoding_rs ^0.8 normal
- flate2 ^1.0 normal
- futures ^0.3 normal
- governor ^0.7 normal
- html5ever ^0.29 normal
- hyper ^1.6 normal
- indexmap ^2.7 normal
- mime ^0.3 normal
- once_cell ^1.20 normal
- parking_lot ^0.12 normal
- petgraph ^0.6 normal
- priority-queue ^2.1 normal
- rand ^0.8 normal
- redis ^0.27 normal
- regex ^1.11 normal
- reqwest ^0.12 normal
- robotstxt ^0.3 normal
- rocksdb ^0.22 normal
- scraper ^0.22 normal
- select ^0.6 normal
- serde ^1.0 normal
- serde_json ^1.0 normal
- serde_yaml ^0.9 normal
- sha2 ^0.10 normal
- sqlx ^0.8 normal optional
- thirtyfour ^0.36 normal
- thiserror ^2.0 normal
- tokio ^1.43 normal
- toml ^0.8 normal
- tracing ^0.1 normal
- tracing-subscriber ^0.3 normal
- url ^2.5 normal
- uuid ^1.11 normal
- zstd ^0.13 normal
- criterion ^0.5 dev
- mockito ^1.6 dev
- proptest ^1.5 dev
- tempfile ^3.14 dev
- Versions
- Owners
-