Crate blz_core 

blz-core

Core functionality for blz - a fast, local search cache for llms.txt documentation.

This crate provides the foundational components for parsing, storing, and searching llms.txt documentation files locally. It’s designed for speed (sub-10ms search latency), offline-first usage, and exact line citations.

Architecture

The crate is organized around several key components:

• Configuration: Global and per-source settings management
• Parsing: Tree-sitter based markdown parsing with structured output
• Types: Core data structures representing sources, search results, and metadata
• Error Handling: Comprehensive error types with categorization and recovery hints

Quick Start

use blz_core::{Config, MarkdownParser, Result};

// Load global configuration
let config = Config::load()?;

// Parse markdown content
let mut parser = MarkdownParser::new()?;
let result = parser.parse("# Hello World\n\nThis is content.")?;

println!("Found {} heading blocks", result.heading_blocks.len());
println!("Generated TOC with {} entries", result.toc.len());

Performance Characteristics

• Parse time: < 150ms per MB of markdown content
• Memory usage: < 2x source document size during parsing
• Thread safety: All types are Send + Sync where appropriate

Error Handling

All operations return Result<T, Error> with structured error information:

use blz_core::{Error, MarkdownParser};

let mut parser = MarkdownParser::new()?;
match parser.parse("malformed content") {
    Ok(result) => println!("Parsed successfully"),
    Err(Error::Parse(msg)) => eprintln!("Parse error: {}", msg),
    Err(e) if e.is_recoverable() => eprintln!("Recoverable error: {}", e),
    Err(e) => eprintln!("Fatal error: {}", e),
}

Re-exports

pub use config::Config;

pub use config::DefaultsConfig;

pub use config::FetchConfig;

pub use config::FollowLinks;

pub use config::IndexConfig;

pub use config::PathsConfig;

pub use config::ToolConfig;

pub use config::ToolMeta;

pub use error::Error;

pub use error::Result;

pub use fetcher::FetchResult;

pub use fetcher::Fetcher;

pub use heading::HeadingPathVariants;

pub use heading::HeadingSegmentVariants;

pub use heading::normalize_text_for_search;

pub use heading::path_variants;

pub use heading::segment_variants;

pub use index::SearchIndex;

pub use json_builder::build_llms_json;

pub use language_filter::FilterStats;

pub use language_filter::LanguageFilter;

pub use mapping::build_anchors_map;

pub use mapping::compute_anchor_mappings;

pub use parser::MarkdownParser;

pub use parser::ParseResult;

pub use profiling::PerformanceMetrics;

pub use profiling::ResourceMonitor;

pub use registry::Registry;

pub use storage::Storage;

pub use types::*;

Modules

config

Configuration management for global and per-source settings Configuration management for blz cache system.

error

Error types and result aliases Error types and handling for blz-core operations.

fetcher

HTTP fetching with conditional requests support

heading

Heading sanitization and normalization helpers

index

Search index implementation using Tantivy

json_builder

JSON builder helpers for llms.json structures Helper utilities for building blz-core JSON structures.

language_filter

Language filtering for multilingual llms.txt files Language filtering for llms.txt documentation entries.

mapping

Anchor remapping utilities between versions

parser

Tree-sitter based markdown parser Markdown parsing using tree-sitter for structured content analysis.

profile

Application profile detection helpers

profiling

Performance profiling utilities

refresh

Refresh helpers shared across CLI and MCP Refresh helpers shared by CLI and MCP consumers.

registry

Built-in registry of known documentation sources

storage

Local filesystem storage for cached documentation

types

Core data types and structures Core data structures for blz cache system.

url_resolver

URL resolver for llms.txt variants URL resolution utilities for smart llms.txt variant detection.