BitNet Core

The core foundation library for BitNet neural networks, providing sophisticated memory management, device abstraction, tensor infrastructure, and GPU acceleration optimized for Apple Silicon and high-performance computing.

🎯 Purpose

bitnet-core serves as the foundational layer for the BitNet ecosystem, focusing on:

• Advanced Memory Management: Production-ready hybrid memory pool system
• Device Abstraction: Unified interface for CPU, Metal GPU, and future accelerators
• Metal GPU Acceleration: Complete Metal compute pipeline with shader compilation
• Tensor Infrastructure: Basic tensor operations and metadata management
• Performance Optimization: Zero-copy operations and SIMD-friendly data structures

✅ What's Implemented

🟢 Memory Management System (Production Ready)

Hybrid Memory Pool Architecture

• SmallBlockPool: Fixed-size allocation for blocks < 1MB with O(1) operations
• LargeBlockPool: Buddy allocation algorithm for blocks ≥ 1MB with coalescing
• DeviceSpecificPools: Separate memory pools for CPU and Metal GPU memory
• Thread Safety: Fine-grained locking with minimal contention

Advanced Memory Tracking

• Real-time Metrics: Allocation patterns, peak usage, fragmentation analysis
• Memory Pressure Detection: Automatic detection of memory pressure with callbacks
• Leak Detection: Comprehensive tracking of unreleased allocations
• Performance Profiling: Timeline analysis and allocation pattern recognition

Automatic Cleanup System

• Intelligent Compaction: Automatic memory defragmentation
• Configurable Strategies: Idle, pressure-based, and periodic cleanup
• Device-Specific Cleanup: Optimized cleanup for different device types
• Safety Validation: Prevents corruption of active tensors

🟢 Device Abstraction Layer (Production Ready)

Device Management

• Automatic Device Selection: Intelligent selection of optimal compute device
• Device Capabilities: Runtime detection of device features and limitations
• Memory Bandwidth Detection: Automatic detection of memory bandwidth characteristics
• Cross-Platform Support: Unified API across different hardware platforms

Device-Specific Optimizations

• CPU Optimizations: Cache-friendly memory layouts and SIMD alignment
• Metal GPU Support: Optimized memory management for Apple Silicon GPUs
• Future Extensibility: Architecture ready for CUDA and other accelerators

🟢 Metal GPU Acceleration (Production Ready)

Metal Compute Pipeline

• Device Management: Automatic Metal device detection and initialization
• Command Buffer Management: Advanced command buffer pooling and lifecycle management
• Shader Compilation: Dynamic Metal shader compilation with caching
• Pipeline Creation: Automatic compute pipeline state management

BitNet-Specific Shaders

• BitLinear Operations: GPU-accelerated BitLinear forward/backward passes
• Quantization Kernels: 1-bit weight and 8-bit activation quantization
• Activation Functions: Optimized ReLU, GELU, Swish, Sigmoid, Tanh, and more
• Mixed Precision: Support for mixed precision operations

Advanced Metal Features

• Buffer Pooling: High-performance Metal buffer allocation and reuse
• Synchronization: Events, fences, and sync points for GPU operations
• Resource Tracking: Automatic dependency management for GPU resources
• Error Handling: Comprehensive error recovery and validation

🟡 Tensor Infrastructure (Basic Implementation)

Tensor Metadata System

• BitNetDType: Custom data types optimized for quantized operations
• TensorMetadata: Comprehensive tensor shape, stride, and device information
• TensorHandle: Safe reference counting and lifetime management
• Memory Layout: Optimized memory layouts for different tensor operations

Basic Tensor Operations

• Tensor Creation: Basic tensor allocation and initialization
• Memory Management: Integration with the hybrid memory pool system
• Device Placement: Automatic tensor placement on appropriate devices
• Metadata Tracking: Comprehensive tracking of tensor properties

🔴 What Needs Implementation

High Priority

1. Advanced Tensor Operations

   • Matrix multiplication optimizations
   • Element-wise operations (add, mul, etc.)
   • Reduction operations (sum, mean, max, etc.)
   • Broadcasting and reshaping operations

2. SIMD Optimizations

   • AVX2/AVX-512 implementations for x86_64
   • NEON optimizations for ARM64
   • Auto-vectorization hints and intrinsics

3. Memory Layout Optimizations

   • Strided tensor support
   • Memory-efficient tensor views
   • Zero-copy tensor slicing

Medium Priority

1. Advanced Device Features

   • Multi-GPU support and load balancing
   • Device-to-device memory transfers
   • Asynchronous operations and streams

2. Performance Monitoring

   • Detailed performance counters
   • Operation-level profiling
   • Memory bandwidth utilization tracking

3. Error Handling

   • Comprehensive error recovery
   • Graceful degradation on memory pressure
   • Device failure handling

Low Priority

1. Serialization Support

   • Tensor serialization/deserialization
   • Memory pool state persistence
   • Cross-platform compatibility

2. Advanced Memory Features

   • Memory-mapped file support
   • Shared memory between processes
   • Memory compression for inactive tensors

🚀 Quick Start

Metal GPU Acceleration

use bitnet_core::metal::*;

// Initialize Metal context
let (device, command_queue, _library) = initialize_metal_context()?;
println!("Metal device: {}", device.name());

// Create BitNet shader collection
let shaders = BitNetShaders::new(device.clone())?;

// Create and execute a ReLU operation
let input_data = vec![1.0f32, -2.0, 3.0, -4.0];
let input_buffer = create_buffer(&device, &input_data)?;
let output_buffer = create_empty_buffer(
    &device,
    input_data.len() * 4,
    metal::MTLResourceOptions::StorageModeShared,
)?;

// Create command buffer and encoder
let command_buffer = command_queue.new_command_buffer();
let encoder = shaders.create_compute_encoder_with_pipeline(
    &command_buffer,
    BitNetShaderFunction::ReluForward
)?;

// Set buffers and dispatch
encoder.set_buffer(0, Some(&input_buffer), 0);
encoder.set_buffer(1, Some(&output_buffer), 0);
set_compute_bytes(&encoder, &[input_data.len() as u32], 2);

let (threads, threadgroup) = shaders.calculate_dispatch_params(
    BitNetShaderFunction::ReluForward,
    input_data.len()
)?;
dispatch_compute(&encoder, threads, threadgroup);

encoder.end_encoding();
command_buffer.commit();
command_buffer.wait_until_completed();

// Read results
let output_data: Vec<f32> = read_buffer(&output_buffer)?;
println!("ReLU result: {:?}", output_data); // [1.0, 0.0, 3.0, 0.0]

Basic Memory Pool Usage

use bitnet_core::memory::{HybridMemoryPool, MemoryPoolConfig};
use bitnet_core::device::auto_select_device;

// Create memory pool with default configuration
let pool = HybridMemoryPool::new()?;
let device = auto_select_device();

// Allocate 1MB of memory with 64-byte alignment
let handle = pool.allocate(1024 * 1024, 64, &device)?;

// Get memory metrics
let metrics = pool.get_metrics();
println!("Total allocated: {} bytes", metrics.total_allocated);
println!("Peak usage: {} bytes", metrics.peak_allocated);

// Deallocate memory
pool.deallocate(handle)?;

Advanced Memory Tracking

use bitnet_core::memory::{
    MemoryPoolConfig, TrackingConfig, TrackingLevel,
    MemoryPressureLevel
};

// Configure advanced tracking
let mut config = MemoryPoolConfig::default();
config.enable_advanced_tracking = true;
config.tracking_config = Some(TrackingConfig {
    level: TrackingLevel::Detailed,
    enable_pressure_detection: true,
    enable_leak_detection: true,
    ..Default::default()
});

let pool = HybridMemoryPool::with_config(config)?;

// Register pressure callback
pool.register_pressure_callback(Box::new(|level| {
    match level {
        MemoryPressureLevel::Critical => {
            eprintln!("CRITICAL: Memory pressure detected!");
        },
        MemoryPressureLevel::High => {
            println!("HIGH: Memory pressure detected");
        },
        _ => {}
    }
}));

// Get detailed metrics
if let Some(detailed) = pool.get_detailed_metrics() {
    println!("Pressure level: {:?}", detailed.pressure_level);
    println!("Fragmentation: {:.2}%", detailed.fragmentation_ratio * 100.0);
}

Advanced Metal Operations

use bitnet_core::metal::*;

// Initialize with custom configuration
let config = ShaderCompilerConfig {
    shader_directory: PathBuf::from("custom/shaders"),
    enable_caching: true,
    optimization_level: OptimizationLevel::Full,
    ..Default::default()
};

let shaders = BitNetShaders::new_with_config(device.clone(), config)?;

// Execute BitLinear forward pass
let encoder = create_bitlinear_forward_encoder(&shaders, &command_buffer)?;
dispatch_bitlinear_forward(
    &encoder,
    &input_buffer,
    &weights_buffer,
    Some(&bias_buffer),
    &output_buffer,
    input_size,
    output_size,
    batch_size,
    threads,
    threadgroup,
);

// Execute quantization
let quant_encoder = create_quantization_encoder(
    &shaders,
    &command_buffer,
    BitNetShaderFunction::QuantizeWeights1Bit
)?;
dispatch_quantization(
    &quant_encoder,
    &input_buffer,
    &output_buffer,
    &scale_buffer,
    element_count,
    group_size,
    threads,
    threadgroup,
);

Device Abstraction

use bitnet_core::device::{auto_select_device, DeviceCapabilities};

// Automatic device selection
let device = auto_select_device();
println!("Selected device: {:?}", device);

// Check device capabilities
let caps = DeviceCapabilities::for_device(&device);
println!("Supports Metal: {}", caps.supports_metal);
println!("Memory bandwidth: {} GB/s", caps.memory_bandwidth_gbps);

Basic Tensor Operations

use bitnet_core::memory::tensor::{BitNetTensor, BitNetDType, TensorMetadata};
use bitnet_core::device::auto_select_device;

let device = auto_select_device();
let pool = HybridMemoryPool::new()?;

// Create tensor metadata
let metadata = TensorMetadata::new(
    vec![128, 256],  // shape
    BitNetDType::F32,
    device.clone()
);

// Create tensor
let tensor = BitNetTensor::new(metadata, &pool)?;
println!("Tensor shape: {:?}", tensor.shape());
println!("Tensor device: {:?}", tensor.device());

📊 Performance Characteristics

Metal GPU Performance (Apple M1 Pro)

Memory Pool Performance (Apple M1 Pro)

Memory Tracking Overhead

🏗️ Architecture

Memory Management Architecture

HybridMemoryPool
├── SmallBlockPool (< 1MB allocations)
│   ├── Fixed-size block allocation
│   ├── Fast O(1) allocation/deallocation
│   └── Minimal fragmentation
├── LargeBlockPool (≥ 1MB allocations)
│   ├── Buddy allocation algorithm
│   ├── Efficient large block handling
│   └── Memory coalescing
├── DeviceSpecificPools
│   ├── CPU memory pools
│   ├── Metal GPU memory pools
│   └── Future: CUDA memory pools
└── AdvancedTracking
    ├── Memory pressure detection
    ├── Allocation pattern analysis
    ├── Leak detection and reporting
    └── Performance profiling

Module Structure

bitnet-core/src/
├── device/                 # Device abstraction layer
│   └── mod.rs             # Device selection and capabilities
├── memory/                # Memory management system
│   ├── mod.rs            # Main memory pool interface
│   ├── small_block.rs    # Small block allocator
│   ├── large_block.rs    # Large block allocator
│   ├── device_pool.rs    # Device-specific pools
│   ├── handle.rs         # Memory handle management
│   ├── metrics.rs        # Memory metrics and monitoring
│   ├── tracking/         # Advanced memory tracking
│   │   ├── mod.rs       # Tracking system interface
│   │   ├── tracker.rs   # Main tracking implementation
│   │   ├── patterns.rs  # Allocation pattern analysis
│   │   ├── pressure.rs  # Memory pressure detection
│   │   ├── timeline.rs  # Timeline analysis
│   │   ├── profiler.rs  # Performance profiling
│   │   └── config.rs    # Tracking configuration
│   ├── cleanup/          # Automatic cleanup system
│   │   ├── mod.rs       # Cleanup system interface
│   │   ├── manager.rs   # Cleanup manager
│   │   ├── scheduler.rs # Cleanup scheduling
│   │   ├── strategies.rs # Cleanup strategies
│   │   ├── metrics.rs   # Cleanup metrics
│   │   ├── config.rs    # Cleanup configuration
│   │   └── device_cleanup.rs # Device-specific cleanup
│   └── tensor/           # Tensor memory management
│       ├── mod.rs       # Tensor system interface
│       ├── tensor.rs    # Tensor implementation
│       ├── handle.rs    # Tensor handle management
│       ├── metadata.rs  # Tensor metadata
│       └── dtype.rs     # BitNet data types
├── metal/                # Metal GPU acceleration
│   ├── mod.rs           # Metal device and command buffer management
│   ├── shader_compiler.rs # Dynamic shader compilation and caching
│   ├── shader_utils.rs  # High-level BitNet shader utilities
│   └── shaders/         # Metal compute shaders
│       ├── README.md    # Shader documentation
│       ├── bitlinear.metal # BitLinear layer operations
│       ├── quantization.metal # Quantization kernels
│       └── activation.metal # Activation functions
├── tensor/               # Basic tensor operations
│   └── mod.rs           # Tensor operation interface
└── lib.rs               # Library root and re-exports

🧪 Testing

Run the comprehensive test suite:

# Run all tests
cargo test --package bitnet-core

# Run specific test modules
cargo test --package bitnet-core memory
cargo test --package bitnet-core device
cargo test --package bitnet-core tensor
cargo test --package bitnet-core metal

# Run with detailed output
cargo test --package bitnet-core -- --nocapture

# Run Metal-specific tests (macOS only)
cargo test --package bitnet-core metal_device_availability_tests
cargo test --package bitnet-core --features metal

# Run integration tests
cargo test --package bitnet-core --test integration_test

Running Examples

# Metal shader compilation demo
cargo run --example shader_compilation_demo --features metal

# Memory tracking demo
cargo run --example memory_tracking_demo

# Cleanup system demo
cargo run --example cleanup_system_demo

# Tensor lifecycle demo
cargo run --example tensor_lifecycle

📈 Benchmarks

Run performance benchmarks:

# Run all benchmarks
cargo bench --package bitnet-benchmarks

# Run memory-specific benchmarks
cargo bench --package bitnet-benchmarks -- memory

# Generate benchmark reports
cargo bench --package bitnet-benchmarks -- --output-format html

🔧 Configuration

Metal GPU Configuration

use bitnet_core::metal::*;

// Shader compiler configuration
let shader_config = ShaderCompilerConfig {
    shader_directory: PathBuf::from("custom/shaders"),
    enable_caching: true,
    cache_directory: Some(PathBuf::from("target/shader_cache")),
    debug_info: false,
    optimization_level: OptimizationLevel::Full,
    compile_options: CompileOptions {
        language_version: LanguageVersion::Metal3_0,
        fast_math: true,
        defines: [("CUSTOM_DEFINE", "1")].into(),
        ..Default::default()
    },
};

// Command buffer pool configuration
let cb_config = CommandBufferPoolConfig {
    max_command_buffers: 32,
    default_timeout: Duration::from_secs(30),
    auto_cleanup: true,
    cleanup_interval: Duration::from_secs(5),
    enable_reuse: true,
};

// Buffer pool configuration
let buffer_config = BufferPoolConfig {
    max_buffers_per_size: 16,
    max_total_memory: 256 * 1024 * 1024, // 256MB
    cleanup_timeout: Duration::from_secs(60),
    auto_cleanup: true,
};

// Create configured Metal context
let (device, command_queue, _) = initialize_metal_context()?;
let shaders = BitNetShaders::new_with_config(device.clone(), shader_config)?;
let manager = create_command_buffer_manager_with_config(&device, &command_queue, cb_config);
let buffer_pool = create_buffer_pool_with_config(&device, buffer_config);

Memory Pool Configuration

use bitnet_core::memory::{MemoryPoolConfig, TrackingConfig, CleanupConfig};

let config = MemoryPoolConfig {
    // Pool sizing
    initial_small_pool_size: 64 * 1024 * 1024,  // 64MB
    max_small_pool_size: 512 * 1024 * 1024,     // 512MB
    initial_large_pool_size: 128 * 1024 * 1024, // 128MB
    max_large_pool_size: 2 * 1024 * 1024 * 1024, // 2GB
    
    // Tracking configuration
    enable_advanced_tracking: true,
    tracking_config: Some(TrackingConfig {
        level: TrackingLevel::Standard,
        enable_pressure_detection: true,
        enable_leak_detection: true,
        pressure_threshold_ratio: 0.8,
        leak_detection_interval: Duration::from_secs(60),
    }),
    
    // Cleanup configuration
    enable_automatic_cleanup: true,
    cleanup_config: Some(CleanupConfig {
        idle_cleanup_interval: Duration::from_secs(30),
        pressure_cleanup_threshold: 0.9,
        enable_compaction: true,
        max_cleanup_time: Duration::from_millis(100),
    }),
};

let pool = HybridMemoryPool::with_config(config)?;

🤝 Contributing

Contributions are welcome! Priority areas for bitnet-core:

1. Metal Shaders: Add new BitNet-specific compute kernels
2. Tensor Operations: Implement missing tensor operations
3. SIMD Optimizations: Add platform-specific optimizations
4. Device Support: Extend device abstraction for new hardware
5. Performance: Optimize critical paths and reduce overhead

Metal Development

When contributing Metal shaders:

1. Add .metal files to src/metal/shaders/
2. Update BitNetShaderFunction enum
3. Add function mapping in shader_utils.rs
4. Include comprehensive tests and benchmarks
5. Document shader parameters and usage

See the main project README for contribution guidelines.

📄 License

Licensed under the MIT License. See LICENSE for details.