vil_crawler 0.2.2

VIL Web Crawler — async concurrent BFS crawling with robots.txt support (I01)
Documentation 
1
2
3
4
5
6
7
8
9
10
11

# vil_crawler

VIL Web Crawler — async concurrent BFS crawling with robots.txt support (I01)

## Part of VIL

This crate is part of [VIL](https://github.com/OceanOS-id/VIL) — a process-oriented language and framework for building zero-copy, high-performance distributed systems.

## License

Licensed under either of [Apache License 2.0](../../LICENSE-APACHE) or [MIT License](../../LICENSE-MIT).