Module fetchers

Expand description

Fetcher system for specialized content fetching

Design: Each fetcher handles specific URL patterns with custom logic. FetcherRegistry dispatches to the first matching fetcher.

Structs§

ArXivFetcher: ArXiv paper fetcher
DefaultFetcher: Default HTTP fetcher
DocsSiteFetcher: Documentation site fetcher with llms.txt support
FetcherRegistry: Registry of fetchers that dispatches to the appropriate handler
GitHubCodeFetcher: GitHub source file fetcher
GitHubIssueFetcher: GitHub issue/PR fetcher
GitHubRepoFetcher: GitHub repository fetcher
HackerNewsFetcher: Hacker News fetcher
PackageRegistryFetcher: Package registry fetcher
RSSFeedFetcher: RSS/Atom feed fetcher
StackOverflowFetcher: Stack Overflow fetcher
TwitterFetcher: Twitter/X tweet fetcher
WikipediaFetcher: Wikipedia fetcher
YouTubeFetcher: YouTube video fetcher