Crate scribe_scaling 

Scribe Scaling

Advanced scaling optimizations for handling large repositories (10k-100k+ files) efficiently. This crate implements progressive loading, intelligent caching, parallel processing, and adaptive threshold management for optimal performance at scale.

Core Features

• Progressive Loading: Metadata-first streaming architecture that avoids loading all files into memory
• Intelligent Caching: Persistent caching with signature-based invalidation
• Parallel Processing: Async/parallel pipeline with backpressure management
• Dynamic Thresholds: Repository-aware adaptive configuration
• Advanced Signatures: Multi-level signature extraction with budget pressure adaptation
• Repository Profiling: Automatic detection of repo type and optimal configuration

Performance Targets

• Small repos (≤1k files): <1s selection, <50MB memory
• Medium repos (1k-10k files): <5s selection, <200MB memory
• Large repos (10k-100k files): <15s selection, <1GB memory
• Enterprise repos (100k+ files): <30s selection, <2GB memory

Architecture

The scaling system is built around a streaming, metadata-first approach:

Repository Discovery → Metadata Stream → Filtered Stream → Analysis Pipeline → Selection
      ↓                     ↓                ↓                   ↓             ↓
  Fast scanning      Lightweight load    Smart filtering   Parallel work   Optimized result

Re-exports

pub use engine::ScalingEngine;

pub use engine::ScalingConfig;

pub use engine::ProcessingResult;

pub use selector::ScalingSelector;

pub use selector::ScalingSelectionConfig;

pub use selector::ScalingSelectionResult;

pub use selector::SelectionAlgorithm;

pub use positioning::ContextPositioner;

pub use positioning::ContextPositioningConfig;

pub use positioning::PositionedSelection;

pub use positioning::ContextPositioning;

pub use streaming::StreamingConfig;

pub use streaming::FileMetadata;

pub use streaming::FileChunk;

pub use caching::CacheConfig;

pub use parallel::ParallelConfig;

pub use adaptive::AdaptiveConfig;

pub use signatures::SignatureLevel;

pub use signatures::SignatureConfig;

pub use profiling::RepositoryProfiler;

pub use profiling::RepositoryProfile;

pub use profiling::RepositoryType;

pub use memory::MemoryConfig;

pub use memory::MemoryStats;

pub use metrics::ScalingMetrics;

pub use metrics::BenchmarkResult;

pub use error::ScalingError;

pub use error::ScalingResult;

Modules

adaptive

Adaptive configuration system that adjusts thresholds based on repository characteristics.

caching

Intelligent caching system with persistent storage.

engine

Main scaling engine that integrates all optimization components.

error

Error handling for scaling optimizations.

memory

Memory management and pooling for efficient resource utilization.

metrics

Performance metrics and benchmarking for scaling optimizations.

parallel

Parallel processing configuration for scaling operations.

positioning

Context Positioning Optimization

profiling

Repository profiling for automatic type detection and configuration optimization.

selector

Intelligent Scaling Selector - Minimal Self-contained Selection

signatures

Advanced signature extraction with multiple levels and budget pressure adaptation.

streaming

Progressive loading and streaming for memory-efficient file processing.

Constants

VERSION

Current version of the scaling crate

Functions

create_scaling_engine

Create a scaling engine with automatic repository profiling

default_scaling_config

Default scaling configuration optimized for most repositories

quick_scale_estimate

Quick scaling analysis for immediate performance estimates