Expand description
The GitHub tree-API source: index a repository (or a subdirectory of one) without crawling anything.
One GET /repos/{owner}/{repo}/git/trees/{ref}?recursive=1 lists every
file with its blob SHA. That SHA is the change token: entries whose SHA
matches a caller-supplied validator are recorded as revalidated and never
fetched, so an unchanged repository costs exactly one HTTPS request per
sync. New or changed blobs are fetched from raw.githubusercontent.com,
classified by path (raw serves everything as text/plain), and yielded
with the blob SHA as their entity tag — which a persistent backend stores
and hands back as next sync’s validators, closing the loop statelessly.
Discovery does not depend on link structure: a corpus whose README links
via absolute URLs, or whose directories hold .yaml/.py with no markdown
connectivity at all, is enumerated completely. This is the property the
crawler cannot provide, and the reason this source exists.
Structs§
- GitHub
Source - A GitHub repository source over a
Fetcher. - Github
Auth - Bearer auth confined to GitHub’s two content hosts.
- Github
Spec - A parsed GitHub repository reference:
{owner, repo, ref, subdir}.