Crate blz_core 

blz-core

Core functionality for blz - a fast, local search cache for llms.txt documentation.

This crate provides the foundational components for parsing, storing, and searching llms.txt documentation files locally. It’s designed for speed (sub-10ms search latency), offline-first usage, and exact line citations.

Architecture

The crate is organized around several key components:

• Configuration: Global and per-source settings management
• Parsing: Tree-sitter based markdown parsing with structured output
• Types: Core data structures representing sources, search results, and metadata
• Error Handling: Comprehensive error types with categorization and recovery hints

Quick Start

use blz_core::{Config, MarkdownParser, Result};

// Load global configuration
let config = Config::load()?;

// Parse markdown content
let mut parser = MarkdownParser::new()?;
let result = parser.parse("# Hello World\n\nThis is content.")?;

println!("Found {} heading blocks", result.heading_blocks.len());
println!("Generated TOC with {} entries", result.toc.len());

Performance Characteristics

• Parse time: < 150ms per MB of markdown content
• Memory usage: < 2x source document size during parsing
• Thread safety: All types are Send + Sync where appropriate

Error Handling

All operations return Result<T, Error> with structured error information:

use blz_core::{Error, MarkdownParser};

let mut parser = MarkdownParser::new()?;
match parser.parse("malformed content") {
    Ok(result) => println!("Parsed successfully"),
    Err(Error::Parse(msg)) => eprintln!("Parse error: {}", msg),
    Err(e) if e.is_recoverable() => eprintln!("Recoverable error: {}", e),
    Err(e) => eprintln!("Fatal error: {}", e),
}

Re-exports

pub use config::Config;

pub use config::DefaultsConfig;

pub use config::FetchConfig;

pub use config::FollowLinks;

pub use config::IndexConfig;

pub use config::PathsConfig;

pub use config::ToolConfig;

pub use config::ToolMeta;

pub use discovery::ProbeResult;

pub use discovery::probe_domain;

pub use error::Error;

pub use error::Result;

pub use fetcher::FetchResult;

pub use fetcher::Fetcher;

pub use heading::HeadingPathVariants;

pub use heading::HeadingSegmentVariants;

pub use heading::normalize_text_for_search;

pub use heading::path_variants;

pub use heading::segment_variants;

pub use health::CacheInfo;

pub use health::HealthCheck;

pub use health::HealthReport;

pub use health::HealthStatus;

pub use health::SourceHealth;

pub use health::SourceHealthEntry;

pub use health::SourceKind;

pub use index::SearchIndex;

pub use json_builder::build_llms_json;

pub use language_filter::FilterStats;

pub use language_filter::LanguageFilter;

pub use mapping::build_anchors_map;

pub use mapping::compute_anchor_mappings;

pub use parser::MarkdownParser;

pub use parser::ParseResult;

pub use profiling::PerformanceMetrics;

pub use profiling::ResourceMonitor;

pub use registry::Registry;

pub use storage::Storage;

pub use types::*;

Modules

config

Configuration management for global and per-source settings Configuration management for blz cache system.

discovery

Documentation source discovery Documentation source discovery for domains.

error

Error types and result aliases Error types and handling for blz-core operations.

fetcher

HTTP fetching with conditional requests support

firecrawl

Firecrawl CLI detection and web scraping integration Firecrawl CLI integration for web documentation scraping.

generate

Generation pipeline for creating llms.txt from web scraping Generation pipeline for creating llms.txt from web scraping.

heading

Heading sanitization and normalization helpers

health

Health check types for diagnostics and source health monitoring Health check types for diagnostics and source health monitoring.

index

Search index implementation using Tantivy Tantivy-backed search index for llms.txt content.

json_builder

JSON builder helpers for llms.json structures Helper utilities for building blz-core JSON structures.

language_filter

Language filtering for multilingual llms.txt files Language filtering for llms.txt documentation entries.

mapping

Anchor remapping utilities between versions

numeric

Safe numeric conversion helpers Safe numeric conversion helpers with documented precision trade-offs.

page_cache

Page cache for scraped web content Page cache for storing scraped web content.

parser

Tree-sitter based markdown parser Markdown parsing using tree-sitter for structured content analysis.

profile

Application profile detection helpers

profiling

Performance profiling utilities

refresh

Refresh helpers shared across CLI and MCP Refresh helpers shared by CLI and MCP consumers.

registry

Built-in registry of known documentation sources

storage

Local filesystem storage for cached documentation

types

Core data types and structures Core data structures for blz cache system.

url_resolver

URL resolver for llms.txt variants URL resolution utilities for smart llms.txt variant detection.