# Web crawler (`kibble crawl`)
`kibble crawl <url>` BFS-crawls a site (same-host, depth/page-limited, `robots.txt`-respecting, polite)
and writes each page as **markdown** into `data/ingest/crawl-<host>/`. Point a `[[source]]` at that dir
to feed `build`.
```bash
kibble crawl https://docs.example.com --depth 2 --max-pages 100
```
## Config `[crawl]`
`max_pages` (50) · `max_depth` (2) · `same_host` (true) · `delay_ms` (200, politeness) ·
`respect_robots` (true) · `user_agent` · `max_bytes` (2 MB/page) · `out` (`data/ingest`) ·
`allow_hosts` (SSRF bypass). CLI: `--depth`, `--max-pages`, `--out`, `--all-hosts`.
Every URL is SSRF-guarded and byte-capped; `robots.txt` is honored. **Static HTML only** — JS-rendered
pages need a headless browser (a follow-up).
> **`--all-hosts` caveat:** only the *seed* host's `robots.txt` is fetched and honored. When
> `--all-hosts` lets the crawl follow links off the seed host, those off-host pages are fetched
> **without** consulting their own `robots.txt` (per-host robots fetching is a follow-up). Use
> `--all-hosts` responsibly.