# Product OS : Crawler
Product OS : Crawler is a browser based cralwer that utilises Product OS : Browser to perform advanced url crawling leveraging headless browsing and automation.
### What is Product OS?
Product OS is a collection of packages that provide different tools and features that can work together to build products more easily for the Rust ecosystem.
## Installation
Use the Rust crate package manager [cargo](https://www.rust-lang.org/tools/install) to install Product OS : Crawler.
```bash
cargo add product-os-crawler
```
or add Product OS : Crawler to your cargo.toml ```[packages]``` section.
```rust
product-os-crawler = { version = "0.0.6", features = [], default-features = true, optional = false }
```
## Features
Product OS Crawler supports a number of features leveraging existing Rust libraries to crawl and perform instructions including:
- Basic crawling capabilities with back-off rules
- Revisit logic and full configuration for tuning
- Scoring system fully configurable to determine value of content
- Ability to hand-off to indexing and storage services using a processor
```rust
// Feature samples TODO
```
## Usage
```rust
// Examples TODO
```
## Contributing
Contributions are not currently available but will be available on a public repository soon.
## License
[GNU AGPLv3](https://choosealicense.com/licenses/agpl-3.0/)