#omnivore-core
Core crawler and knowledge graph engine for Omnivore.
-Fast async crawling with Tokio
- Respectful politeness and robots.txt support
- HTML parsing and extraction utilities
- Pluggable storage and graph building
##License
MIT OR Apache-2.0