cadi-scraper

CADI Scraper/Chunker utility for converting source code repos and file data into reusable CADI chunks.

Overview

cadi-scraper automatically analyzes source code projects and converts them into optimized, content-addressed chunks ready for distribution through CADI registries. It handles multiple programming languages, diverse file formats, and provides intelligent semantic chunking.

Features

Multi-language support: Rust, TypeScript, Python, JavaScript, Go, C/C++
Format-agnostic: Source code, Markdown, JSON, YAML, HTML, CSS
5 chunking strategies: By-file, Semantic, Fixed-size, Hierarchical, By-line-count
Automatic metadata extraction: Titles, descriptions, licenses, frameworks, API surfaces
Rate-limited fetching: HTTP and filesystem access with configurable throttling
Semantic analysis: AST-based code understanding via tree-sitter
Framework detection: Identifies 20+ popular frameworks (React, Django, Spring, etc.)
License detection: Recognizes SPDX licenses automatically
Async/await: High-performance concurrent processing

Installation

Add this to your Cargo.toml:

[dependencies]
cadi-scraper = "1.0"

Quick Start

Basic Scraping

use cadi_scraper::{Scraper, ScraperConfig, ScraperInput, ChunkingStrategy};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let config = ScraperConfig {
        chunking_strategy: ChunkingStrategy::Semantic,
        max_chunk_size: 50_000,
        ..Default::default()
    };
    
    let scraper = Scraper::new(config);
    
    let input = ScraperInput::LocalPath("./my-project".into());
    let output = scraper.scrape(input).await?;
    
    println!("Created {} chunks", output.chunks.len());
    println!("Total bytes: {}", output.statistics.total_bytes);
    
    Ok(())
}

CLI Usage

# Install
cargo install cadi

# Scrape a project
cadi scrape ./my-project --strategy semantic --output ./chunks

# Publish to registry
cadi publish --registry https://registry.example.com \
  --auth-token TOKEN \
  --namespace myorg/project

# See all options
cadi scrape --help

Chunking Strategies

By-File

Creates one chunk per file. Fast, simple, preserves file structure.

ChunkingStrategy::ByFile

Semantic

Analyzes code structure and chunks at logical boundaries (functions, classes, modules).

ChunkingStrategy::Semantic

Fixed-Size

Creates fixed-byte chunks, useful for uniform processing.

ChunkingStrategy::FixedSize

Hierarchical

Creates parent chunks per file with children chunks for functions/classes.

ChunkingStrategy::Hierarchical

By-Line-Count

Creates chunks every N lines (default 100).

ChunkingStrategy::ByLineCount

Configuration

Via Environment Variables

export CADI_CHUNKING_STRATEGY="semantic"
export CADI_MAX_CHUNK_SIZE="52428800"  # 50MB
export CADI_INCLUDE_OVERLAP="true"
export CADI_EXTRACT_API_SURFACE="true"
export CADI_DETECT_LICENSES="true"

Via Config File

Create ~/.cadi/scraper-config.yaml:

chunking_strategy: semantic
max_chunk_size: 52428800
include_overlap: true
extract_api_surface: true
detect_licenses: true
languages:
  rust:
    enabled: true
    custom_patterns: []
  python:
    enabled: true
    custom_patterns: []

Programmatically

let config = ScraperConfig {
    chunking_strategy: ChunkingStrategy::Semantic,
    max_chunk_size: 50_000,
    include_overlap: true,
    hierarchy: true,
    extract_api: true,
    detect_licenses: true,
    ..Default::default()
};

Output

Scraping produces ScraperOutput with:

pub struct ScraperOutput {
    pub chunks: Vec<ScrapedChunk>,      // Generated chunks
    pub manifest: Manifest,              // Dependency graph
    pub statistics: ScrapingStatistics,  // Metrics
    pub errors: Vec<String>,             // Non-fatal errors
}

Advanced Usage

Custom Language Patterns

let mut config = ScraperConfig::default();
config.languages.insert("rust".to_string(), LanguageConfig {
    enabled: true,
    custom_patterns: vec![
        r"#\[derive\((.*?)\)\]".to_string(),
    ],
});

Publishing Chunks

use cadi_registry::RegistryClient;

let output = scraper.scrape(input).await?;
let client = RegistryClient::new(registry_url, auth_token);

for chunk in output.chunks {
    client.publish_chunk(&chunk).await?;
}

Batch Processing

let inputs = vec![
    ScraperInput::LocalPath("./project1".into()),
    ScraperInput::LocalPath("./project2".into()),
    ScraperInput::Url("https://github.com/user/repo".into()),
];

for input in inputs {
    let output = scraper.scrape(input).await?;
    // Process output...
}

Framework Detection

Automatically detects:

Frontend: React, Vue, Angular, Svelte, Next.js
Backend: Express, Fastify, Django, FastAPI, Spring, Rails
Async Runtimes: Tokio, async-std
Testing: Jest, pytest, RSpec
Build Tools: Webpack, Vite, Cargo, Maven

License Detection

Recognizes SPDX identifiers:

MIT
Apache-2.0
GPL-3.0
BSD-3-Clause
ISC
And many more...

Performance

Typical performance on modern hardware:

By-file chunking: ~100 MB/sec
Semantic chunking: ~50 MB/sec
Metadata extraction: Included in above
Rate limiting: Configurable (default 10 req/sec)

Error Handling

use cadi_scraper::error::Error;

match scraper.scrape(input).await {
    Ok(output) => {
        if !output.errors.is_empty() {
            eprintln!("Warnings: {:?}", output.errors);
        }
    }
    Err(Error::InvalidInput(msg)) => eprintln!("Invalid input: {}", msg),
    Err(Error::Fetch(msg)) => eprintln!("Fetch failed: {}", msg),
    Err(e) => eprintln!("Error: {}", e),
}

Integration

Part of the CADI ecosystem:

cadi-core: Chunk and manifest types
cadi-registry: Publish scraped chunks
cadi: CLI integration
cadi-builder: Transform scraped chunks

Documentation

Full API docs: docs.rs/cadi-scraper
User guide: Check repository SCRAPER-GUIDE.md
Examples: See repository examples/ directory

License

MIT License

cadi-scraper 1.1.0