pub const AGENT_GUIDE: &str = "# Siteforge Agent Guide\n\nThis guide describes the supported `siteforge` 0.1 CLI and public Rust facade. Run `siteforge --agent` to print it verbatim without loading configuration or initializing archive services. It is also embedded in the crate as `siteforge::AGENT_GUIDE`.\n\n## Safety First\n\n- Archive only content that you are authorized to access and retain. Siteforge does not bypass authentication, paywalls, CAPTCHAs, DRM, or access controls.\n- Use absolute `http://` or `https://` seed URLs. Seed input is URL-parsed, but there is no private-address, loopback, link-local, or cloud-metadata deny-list. Do not run untrusted seeds in a network with sensitive internal services.\n- Do not put credentials in a URL. Seed, source, and final URLs are recorded in archive metadata and readable artifacts.\n- The HTTP client follows up to 10 redirects. Treat redirect destinations as part of the network access you are authorizing.\n- Same-domain mode limits discovered page links, not every request. Page assets can be downloaded from external hosts, and sitemap URLs can come from `robots.txt`.\n- `--allow-cross-domain` removes the host/path restriction for discovered page links. Use it only with tight `--include`, `--exclude`, `--max-depth`, and `--max-pages` controls.\n- `--header` and `--cookie` values are applied by the shared HTTP fetcher to its requests, including robots, sitemap, page, and asset fetches. A page can reference third-party assets, so do not supply credentials unless sending them to all fetched resources is acceptable.\n- HTTP headers and cookies are not passed to the browser renderer. Use an authorized, isolated `--browser-profile` when browser authentication is required.\n- Browser profiles may contain sensitive session data. Use a dedicated profile directory, protect it outside the archive, and do not share it as an archive artifact.\n- Siteforge reads and obeys applicable `robots.txt` rules for its user agent and applies the greater of configured delay, robots crawl-delay, and robots request-rate pacing. A failed robots fetch is treated as no rules.\n\nDiscovered links are normalized by resolving relative URLs, accepting only HTTP(S), removing fragments and default ports, sorting query pairs, and deduplicating. In default same-domain mode, a discovered page must match a seed host (with leading `www.` ignored) and path prefix. The check compares hosts, not schemes or ports. A seed ending in `/` scopes to that directory; a seed ending in a file-like segment scopes to its parent directory.\n\n## Configuration\n\nThe global `--config <PATH>` option loads YAML. Without it, siteforge uses the platform config path; if no file exists, `Config::default()` is used. Relative `archive_root` paths resolve from the process working directory.\n\nRun `siteforge config` to create the default config file if missing and print the active configuration. Run `siteforge config --edit` to open it with `$EDITOR`, defaulting to `vi`.\n\nImportant defaults are:\n\n| Setting | Default |\n| --- | ---: |\n| `archive_root` | `archives` |\n| `archive_profile` | `agent` |\n| `default_concurrency` | `4` |\n| `default_delay_ms` | `250` |\n| `max_depth` | `4` |\n| `max_pages` | `500` |\n| `max_page_size_bytes` | `52428800` |\n| `max_asset_size_bytes` | `26214400` |\n| `max_total_archive_size_bytes` | `5368709120` |\n| `timeout_secs` | `30` |\n| `retry_count` | `2` |\n\nThe default excludes are `**/login**`, `**/logout**`, `**/signup**`, and `**/cart**`. CLI `--include` values replace configured include patterns when at least one is supplied. CLI `--exclude` values extend configured excludes.\n\nFor page and asset body limits, `0` means unlimited. For the total archive size limit, `0` disables the cap. `--max-pages 0` does not mean unlimited: it causes the crawl loop to stop before fetching a page. Concurrency values below one are normalized to one.\n\nExplicit archive IDs must contain 1 to 128 ASCII letters, numbers, `.`, `_`, or `-`. If omitted, an ID is generated from the first seed host and a UTC timestamp. A new archive fails if its target directory already exists.\n\n## Archive Modes\n\n### Single Page And Seed Batches\n\nWithout `--full-site`, siteforge fetches only the supplied seed pages. The normal `agent` and `full` profiles can still download assets referenced by those pages.\n\n```bash\nsiteforge archive https://example.com/article --archive-id example-article\nsiteforge archive --input urls.txt --archive-id docs-batch -j 4\nsiteforge archive https://example.com/start --input more-urls.txt --archive-id combined\n```\n\n`--input` reads one URL per line. Blank lines and lines beginning with `#` are ignored. A positional URL and `--input` can be used together.\n\n### Full Crawl\n\n`--full-site` first attempts sitemap discovery from robots `Sitemap:` directives and `/sitemap.xml`, then follows normalized links that pass depth, glob, and domain/path scope checks.\n\n```bash\nsiteforge archive https://example.com/docs/ \\\n --full-site \\\n --archive-id example-docs \\\n --max-depth 3 \\\n --max-pages 200 \\\n --concurrency 4 \\\n --delay-ms 500 \\\n --include \'https://example.com/docs/**\' \\\n --exclude \'**/drafts/**\'\n```\n\n`max_depth` controls discovered links; seeds are depth zero and sitemap page entries are queued at depth one. `max_pages` limits successful fetch responses counted by the crawl. Page, asset, and total archive byte limits fail or record the affected work when exceeded.\n\n### Output Profiles\n\nAll profiles create the directory layout, so an omitted artifact class can appear as an empty directory.\n\n| Profile | Raw HTML/rendered DOM | Assets | YAML Markdown | Basic Markdown | JSON | Plain text | JSONL chunks |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| `full` | yes | yes | yes | yes | yes | yes | yes |\n| `agent` (default) | yes | yes | no | yes | yes | no | yes |\n| `lean` | no | no | no | yes | yes | no | yes |\n\nChoose a profile per run with `--archive-profile full|agent|lean`, set `archive_profile` in YAML, or assign `Config::archive_profile` in Rust.\n\n### Render Modes\n\n- Default/static mode leaves `render_js` false and parses fetched HTML.\n- `--render-js` renders every eligible HTML page before parsing.\n- `--render-js-auto` enables rendering only when the fetched HTML looks like a thin SPA shell.\n- `--render-js` and `--render-js-auto` conflict on the CLI.\n- `--render-wait-ms` sets the browser virtual-time budget.\n- `--browser-command` selects an executable or command template. Templates support `{url}`, `{wait_ms}`, and `{user_agent}`. Without `{url}`, siteforge appends Chromium-style headless arguments and the URL.\n- Without `--browser-command`, siteforge tries `chromium`, `chromium-browser`, `google-chrome`, and `google-chrome-stable`.\n- Render failures are logged as warnings and fetched HTML is still parsed.\n\nThe browser is launched with headless, GPU-disabled, no-sandbox, and disabled `/dev/shm` usage flags when siteforge builds the browser arguments. Only use browser rendering in an appropriately isolated environment.\n\n## Headers And Cookies\n\nRepeat `--header` for each `Name: Value` pair. `--cookie` accepts the complete Cookie header value.\n\n```bash\nsiteforge archive https://docs.example.test/private \\\n --archive-id private-doc \\\n --header \'Authorization: Bearer <token>\' \\\n --header \'Accept-Language: en\' \\\n --cookie \'session=<value>\'\n```\n\nMalformed header names/values and malformed cookie values fail fetcher construction. Request credentials are not written as dedicated manifest fields, but URLs, response headers, fetched content, and browser-rendered content can contain sensitive data. Review artifacts before sharing.\n\n`siteforge resume` has no header or cookie options. Resume reconstructs scope, concurrency, delay, page limit, output profile, and render mode from the archive state, but per-run headers and cookies are empty. Do not assume an authenticated interrupted crawl can resume authenticated requests.\n\n## Operational Workflows\n\n### Archive\n\nUse a dedicated archive ID, conservative limits, and `--progress never` in machine-readable automation. Progress modes are `auto`, `always`, and `never`; progress is written to stderr and the final summary to stdout.\n\n```bash\nsiteforge --config ./siteforge.yaml archive https://example.com/docs/ \\\n --full-site --archive-id docs-2026 --max-pages 100 --progress never\n```\n\nCtrl+C checkpoints crawl state, returns active frontier rows to pending, and skips final index/checksum generation so stopping is fast. The CLI prints a resume command. The library returns `SiteforgeError::ArchiveInterrupted(archive_id)`.\n\n### List And Inspect\n\n```bash\nsiteforge list\nsiteforge inspect docs-2026\n```\n\n`list` prints manifests under the active archive root in most-recently-updated order. `inspect` opens the manifest and crawl database and reports scope, counters, indexed page/asset totals, recent pages, and up to ten failed pages.\n\n### Search, Pack, And Export\n\n```bash\nsiteforge search \'installation\' --archive-id docs-2026 --limit 20\nsiteforge pack docs-2026 --for-ai --max-tokens 120000\nsiteforge export docs-2026 --format agent-tar\nsiteforge export docs-2026 --format zip --output ./deliverables\n```\n\nSearch can target one archive or all archives. `pack` requires `--for-ai`; it replaces prior `pack_*` files and writes token-budgeted Markdown under `packs/`. The minimum effective pack budget is 1,000 tokens.\n\nExport formats are `markdown`, `basic-markdown`, `jsonl`, `json`, `html`, `warc`, `zip`, `tar`, `agent-tar`, and `all`. Without `--output`, exports go under `exports/<format>/`. WARC output is WARC-like JSONL metadata, not a standards-complete WARC file. `agent-tar` contains agent/readable/chunk/pack/raw artifacts but omits the crawl database, `.thesa/`, and exports.\n\n### Verify\n\n```bash\nsiteforge verify docs-2026\nthesa verify archives/docs-2026\n```\n\nSiteforge verification checks required core paths and BLAKE3 entries in `checksums.json`. It reports missing files, size mismatches, and hash mismatches. It does not verify the `.thesa/` sidecar; use `thesa verify` for that sidecar.\n\nThe CLI currently prints verification problems but still returns success when `verify_archive` itself ran successfully. Automation must parse the output or call the Rust API and require the returned problem vector to be empty.\n\n### Resume\n\n```bash\nsiteforge resume docs-2026 --progress always\n```\n\nResume requires an existing `manifest.json`, resets `in_progress` frontier rows to pending, and continues the same archive. On successful completion it regenerates readable indexes, agent entrypoints, checksums, and the thesa sidecar. See the credential limitation above.\n\n### Diagnostics\n\n```bash\nsiteforge diagnostics\nsiteforge diagnostics --archive-id docs-2026\n```\n\nGlobal diagnostics print the archive root, optional Linux `/proc` process memory/open-file values, configured limits, and up to 50 archive summaries. Per-archive diagnostics add archive/database/checksum sizes, manifest settings, crawl counters, and up to 20 job snapshots. Diagnostics are observational and do not repair archives.\n\n## Dry Run\n\nThere is no `--dry-run` CLI flag and no dry-run method in the public facade. `siteforge archive` creates an archive and performs network requests; `--progress never` only disables progress rendering.\n\nFor a non-network preflight, use `siteforge archive --help`, inspect `siteforge config`, validate the seed list separately, and choose explicit limits and an isolated `archive_root`. A one-page run with `--max-pages 1` is a real crawl, not a dry run.\n\n## Archive Outputs\n\nA completed archive uses this layout:\n\n```text\n<archive_root>/<archive_id>/\n manifest.json\n AGENTS.md\n agent-index.json\n crawl.db\n checksums.json\n .thesa/\n manifest.json\n checksums.blake3\n raw/\n pages/\n rendered/\n assets/\n readable/\n README.md\n AGENTS.md\n basic_markdown/\n markdown/\n json/\n text/\n chunks/\n chunks.jsonl\n packs/\n logs/\n exports/\n```\n\n`manifest.json` records seeds, scope, statistics, and a configuration snapshot. `crawl.db` stores frontier/job state, metadata, and search indexes. `checksums.json` excludes itself, `exports/`, `.thesa/`, and transient SQLite WAL/SHM files. Packs and exports are produced only when their commands/API methods are run.\n\n### Agent Entrypoints\n\n- Start with top-level `AGENTS.md` for ingestion order and profile-specific hints. The same content is copied to `readable/AGENTS.md`.\n- Read `agent-index.json` for `archive_id`, update time, seeds, standard artifact paths, page records, and asset records.\n- Each indexed page includes IDs, source/final URLs, status, and optional paths for basic Markdown, YAML Markdown, JSON, text, and raw HTML.\n- Each indexed asset includes IDs, source URL, optional local path, MIME type, byte size, and OCR status.\n- Use `readable/README.md` as the human-readable archive index.\n- Use `readable/basic_markdown/` for broad agent ingestion and `chunks/chunks.jsonl` for retrieval. Code chunks are kept whole.\n- Generate `packs/pack_*.md` when bounded, source-attributed context bundles are preferable.\n\nInterrupted archives can lack final `AGENTS.md`, `agent-index.json`, `readable/README.md`, `checksums.json`, and `.thesa/` artifacts until resume completes.\n\n## Public Rust API\n\nThe high-level facade is `Siteforge`. `Config`, `ArchiveOutputProfile`, `ArchiveRunOptions`, `ArchiveSummary`, `ExportFormat`, `SiteforgeError`, and `Result` are re-exported at the crate root. Lower-level modules are public for specialized use.\n\n```rust\nuse siteforge::{ArchiveOutputProfile, ArchiveRunOptions, Config, Siteforge};\n\n#[tokio::main]\nasync fn main() -> siteforge::Result<()> {\n let mut config = Config::default();\n config.archive_root = \"archives\".into();\n config.archive_profile = ArchiveOutputProfile::Agent;\n\n let siteforge = Siteforge::new(config);\n let options = ArchiveRunOptions::full_site_url(\n \"https://example.com/docs/\",\n siteforge.config(),\n )?\n .with_archive_id(\"example-docs\")\n .with_max_depth(3)\n .with_max_pages(200)\n .with_concurrency(4)\n .with_delay_ms(500);\n\n let summary = siteforge.archive(options).await?;\n let problems = siteforge.verify(&summary.archive_id)?;\n if !problems.is_empty() {\n return Err(siteforge::SiteforgeError::message(problems.join(\"; \")));\n }\n Ok(())\n}\n```\n\nConvenience methods `archive_page` and `archive_full_site` use configuration defaults and one seed. Their `_with_progress` variants and `archive_with_progress` accept a bounded Tokio sender. `ArchiveRunOptions` supports seed-only and full-site constructors plus builders for archive ID, mode, scope, limits, concurrency, delay, headers, and cookie.\n\nThe facade also provides `archive_layout`, `resume`, `resume_with_progress`, `list_archives`, `search`, `export`, `create_ai_packs`, and `verify`. There are no facade methods named `inspect`, `diagnostics`, or `dry_run`; those first two are CLI workflows and the third is unsupported.\n\nThe embedded guide is available without filesystem lookup:\n\n```rust\nlet guide: &\'static str = siteforge::AGENT_GUIDE;\nassert!(guide.contains(\"# Siteforge Agent Guide\"));\n```\n\n## Errors And Completion Semantics\n\nPublic operations return `siteforge::Result<T>`, whose error is `SiteforgeError`. Defined categories include I/O, HTTP, URL, SQLite, JSON, YAML, glob, ZIP, missing input, invalid/archive-not-found IDs, robots denial, unsupported content, size limits, unavailable OCR, TUI failures, interruption, and message errors.\n\n- Missing URL and input is an error: `siteforge archive` requires a positional URL, `--input`, or both.\n- Invalid globs, headers, cookies, config, URLs, archive IDs, or browser commands fail the operation.\n- HTTP status 400 or above is recorded as a failed page. Non-HTML page responses are recorded as skipped.\n- Robots denial is recorded as skipped rather than aborting the whole archive.\n- Fetch and page-processing failures are recorded in `crawl.db`; a crawl can complete with nonzero `pages_failed`. Always inspect the summary or run `inspect`.\n- Asset fetch and OCR failures are normally recorded/logged without failing the entire archive.\n- A JS render failure falls back to fetched HTML.\n- A total archive size failure can leave an incomplete archive that is useful for diagnosis but lacks final indexes/checksums.\n- The CLI handles Ctrl+C interruption as a saved state and prints `siteforge resume <archive_id>`; library callers receive `ArchiveInterrupted`.\n- New archive operations never overwrite an existing archive directory. Use resume only for its persisted frontier, not as an overwrite mechanism.\n\n## End-To-End Examples\n\nConservative public documentation crawl:\n\n```bash\nsiteforge archive https://example.com/docs/ \\\n --full-site \\\n --archive-id example-docs \\\n --archive-profile agent \\\n --max-depth 2 \\\n --max-pages 100 \\\n --concurrency 2 \\\n --delay-ms 1000\nsiteforge inspect example-docs\nsiteforge verify example-docs\nsiteforge pack example-docs --for-ai --max-tokens 60000\nsiteforge export example-docs --format agent-tar\n```\n\nJS-heavy single page using an isolated browser profile:\n\n```bash\nsiteforge archive https://app.example.test/help \\\n --archive-id app-help \\\n --render-js-auto \\\n --browser-command chromium \\\n --browser-profile /secure/siteforge-browser-profile \\\n --render-wait-ms 5000 \\\n --max-pages 1\n```\n\nBatch of explicit pages with no link traversal:\n\n```text\n# urls.txt\nhttps://example.com/guide/start\nhttps://example.com/guide/reference\n```\n\n```bash\nsiteforge archive --input urls.txt \\\n --archive-id selected-guides \\\n --archive-profile lean \\\n --max-pages 2 \\\n --progress never\nsiteforge list\nsiteforge inspect selected-guides\n```\n";Expand description
Agent-oriented operating guide shipped with and embedded in the crate.