Module ingest 

Hosted ingest: getting a stranger’s repository onto the box (paid-path plan S2, gap G1).

Until now POST /v1/index accepted only a path on the server host, which is a fine contract for the operator’s own keys and a useless one for a paying tenant: there was no way in. This module is the way in. Both hosted modes land the tree in a server-owned workdir — <data_dir>/ingest/<index_id>/ — never at a caller-chosen host path:

• Clone. A public https:// git URL, shallow-cloned with the system git. The URL is validated before git ever runs: HTTPS only, no embedded credentials, port 443 only, and the host must resolve exclusively to public addresses (see validate_repo_url). The clone itself is hardened: argument array (never a shell), GIT_TERMINAL_PROMPT=0 so a private repo fails instead of prompting, redirects refused so a checked URL cannot bounce the clone somewhere unchecked, submodules not recursed, LFS smudge skipped, a wall-clock timeout, and a cap on the cloned tree’s size.
• Archive upload. A .zip or .tar.gz body, extracted entry by entry with every path proven to stay inside the workdir before it is written: traversal entries (../…, absolute paths) are refused, and so are link entries — a symlink planted by one entry and written through by the next is the classic extraction escape, so archives carrying links do not ingest at all. Extracted bytes are counted as they are written, not trusted from headers, so a lying header cannot smuggle an expansion bomb past the cap.

The caps below are the hosted plan’s values, named so the abuse-envelope step (plan S3) could turn them into per-plan policy without hunting literals — which it did: they are now the HOSTED row of IngestCaps, resolved per key by crate::limits, and every function here takes its caps as a parameter. A failed ingest removes its workdir: a partial tree is never left behind for a later sweep to mistake for a live index.

Structs

IngestCaps

The ingest caps one plan resolves to, passed into every function here so no function reads a global: a hosted key’s clone, upload, and extraction are bounded by these numbers, and the operator’s self-hosted keys resolve to IngestCaps::UNLIMITED instead — trusting the operator with their own box is the self-hosted plan’s whole definition (see crate::limits).

ResolvePin

One validated DNS answer the clone is pinned to: git will be told to connect the URL’s host to exactly this address (see validate_repo_url).

ValidatedRepoUrl

A repo_url that passed validation, with the DNS answers it was approved against. The clone consumes this, never the raw string, so the checked URL and the connected addresses cannot drift apart between the two steps.

Enums

ArchiveKind

The archive formats ingest accepts.

Constants

CLONE_TIMEOUT_SECS

Wall-clock budget for one git clone. A clone that outlives it is killed, not waited on: ingest is synchronous from the client’s side of the request, and an unbounded clone is an unbounded request.

MAX_ARCHIVE_ENTRIES

Most entries one archive may carry. Bounds the work a single upload can make the extractor do before any byte is counted.

MAX_TREE_BYTES

Largest tree an ingest may produce, in bytes — extracted bytes for an archive, on-disk bytes for a clone. Counted as written/measured, never trusted from a header.

MAX_UPLOAD_BYTES

Largest archive body POST /v1/index accepts, in bytes.

Functions

archive_kind

Decides what a request body is: the declared content type first, the magic bytes when the declaration is missing or generic (application/octet-stream). None is neither archive format — the handler answers invalid_body.

clone_into

Clones a validated URL into workdir (created here) and checks the tree against the plan’s byte cap. On any failure the workdir is removed: a partial clone is not an index and must not linger as one.

count_tree_files

Counts the regular files under root an index of this tree would be built from, symlinks not followed, directories in [skipped_dir_name] not entered. Stops as soon as the count reaches stop_after — the per-index file cap’s check passes limit + 1, so a returned limit + 1 means “more than the limit” and the walk never pays for an exact count of a tree it is about to refuse.

extract_into

Extracts an archive body into workdir (created here) and returns the directory to index. On any failure the workdir is removed.

validate_repo_url

Fully validates a repo_url, including where its host points.