Struct GitHubSource 

pub struct GitHubSource<F: Fetcher> { /* private fields */ }

A GitHub repository source over a Fetcher.

The repository reference lives on the source itself; the root argument of Source::discover is ignored (there is nothing to crawl from — the tree call enumerates everything).

Implementations

impl<F: Fetcher> GitHubSource<F>

pub fn new(fetcher: F, spec: GithubSpec) -> Self

Create a source for spec over fetcher (pair it with GithubAuth).

pub fn bases( self, api_base: impl Into<CompactString>, raw_base: impl Into<CompactString>, ) -> Self

Override the API and raw base URLs (GitHub Enterprise, tests).

pub fn depth(self, depth: u16) -> Self

Cap how many directory levels below the subdir are indexed (0 = only files directly in it). Default: unlimited.

pub fn max_bytes(self, max_bytes: u64) -> Self

Per-file byte ceiling. Oversized files are skipped without a fetch — the tree listing already carries each blob’s size.

pub fn concurrency(self, concurrency: usize) -> Self

Maximum blob fetches in flight at once (clamped to ≥ 1).

pub fn accept_extension(self, extension: impl AsRef<str>) -> Self

Only index files whose path ends with extension (additive).

pub fn validators( self, entries: impl IntoIterator<Item = (UrlKey, CompactString)>, ) -> Self

Provide known blob SHAs (keyed by canonical raw-URL key) so unchanged files are revalidated without any fetch.

pub fn take_revalidated(&self) -> Vec<Url>

Files revalidated (blob SHA unchanged, nothing fetched) during the most recent discover. Same contract as the crawler’s: this is the only place those checks are visible, and a freshness tracker consumes it.

pub fn failed_count(&self) -> u32

Files whose blob fetch failed during the most recent discover.

Trait Implementations

impl<F: Debug + Fetcher> Debug for GitHubSource<F>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<F: Fetcher> Source for GitHubSource<F>

type Pages<'a> = impl Stream<Item = Result<Page<'a>, Error>> + Send + 'a where Self: 'a

The stream of discovered pages.

fn kind(&self) -> &'static str

A stable label for this source kind ("crawl", "fs").

fn discover<'a>(&'a self, _root: &'a SourceRef) -> Self::Pages<'a>

Begin discovery from root.