unobtanium-crawler 3.0.0

The default web-crawler for unobtanium.
1
2
3
4
5
6
7
8
9
10
11
12

# unobtanium-crawler

[Manual](https://doc.unobtanium.rocks/manual/unobtanium-viewer) | [Codeberg](https://codeberg.org/unobtanium/unobtanium/src/branch/main/crawler)

The unobtanium-crawler is the web crawler of the unobtanium search engine.

It is responsible for crawling websites and turning the result into a searchable index.

It is published under an `AGPL-3.0-only` license.

[Learn more about unobtanium.](https://doc.unobtanium.rocks)