Module file_chunk_id

adaptive_pipeline_domain::value_objects

Module file_chunk_id 

Source
Expand description

§File Chunk Identifier Value Object - Processing Infrastructure

This module provides a comprehensive file chunk identifier value object that implements type-safe chunk identification, temporal ordering, and processing sequence management for the adaptive pipeline system’s file processing infrastructure.

§Overview

The file chunk identifier system provides:

  • Type-Safe Identification: Strongly-typed chunk identifiers with compile-time validation
  • Temporal Ordering: ULID-based time-ordered creation sequence for chunk processing
  • Processing Sequence: Natural ordering for chunk processing workflows
  • Traceability: Complete chunk lifecycle tracking and debugging support
  • Serialization: Consistent serialization across storage backends and APIs
  • Validation: Comprehensive chunk-specific validation and business rules

§Architecture

The file chunk ID system follows a layered architecture with clear separation of concerns:

┌─────────────────────────────────────────────────────────────────┐
│                  File Chunk ID System                          │
│                                                                     │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │               FileChunkId Value Object                 │    │
│  │  - Type-safe chunk identifier wrapper                  │    │
│  │  - ULID-based temporal ordering                        │    │
│  │  - Immutable value semantics (DDD pattern)             │    │
│  │  - Chunk-specific business rules                       │    │
│  └─────────────────────────────────────────────────────────┘    │
│                                                                     │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │              FileChunkMarker Type                      │    │
│  │  - Category identification ("file_chunk")              │    │
│  │  - Chunk-specific validation rules                     │    │
│  │  - Timestamp validation and constraints                │    │
│  │  - Business rule enforcement                           │    │
│  └─────────────────────────────────────────────────────────┘    │
│                                                                     │
│  ┌─────────────────────────────────────────────────────────┐    │
│  │               Generic ID Foundation                    │    │
│  │  - ULID generation and management                      │    │
│  │  - Timestamp extraction and validation                 │    │
│  │  - Serialization and deserialization                  │    │
│  │  - Cross-platform compatibility                       │    │
│  └─────────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────┘

§Key Features

§1. Type-Safe Chunk Identification

Strongly-typed chunk identifiers with comprehensive validation:

  • Compile-Time Safety: Cannot be confused with other entity IDs
  • Runtime Validation: Timestamp and format validation at creation time
  • Immutable Semantics: Value objects that cannot be modified after creation
  • Business Rule Enforcement: Chunk-specific validation rules

§2. Temporal Ordering and Processing Sequence

ULID-based temporal ordering for chunk processing:

  • Time-Ordered Creation: Natural chronological ordering of chunks
  • Processing Sequence: Deterministic chunk processing order
  • Timestamp Extraction: Easy access to creation timestamps
  • Chronological Sorting: Built-in sorting capabilities

§3. Traceability and Debugging

Comprehensive chunk lifecycle tracking:

  • Creation Tracking: Clear identification of chunk creation times
  • Processing Flow: Easy tracking of chunk processing workflows
  • Debugging Support: Rich debugging information and validation
  • Audit Trail: Complete chunk lifecycle audit capabilities

§4. Serialization and Storage

Consistent serialization across platforms:

  • JSON Serialization: Standard JSON representation
  • Database Storage: Optimized database storage patterns
  • Cross-Platform: Consistent representation across languages
  • API Integration: RESTful API compatibility

§Usage Examples

§Basic Chunk ID Creation and Management

§Creating Chunk IDs from Different Sources

§Chunk Processing Sequence and Ordering

§Serialization and Deserialization

§Chunk Processing Workflow Integration

§Error Handling and Validation

§Integration Patterns

§Database Storage

§API Integration

§Performance Characteristics

  • Creation Time: ~2μs for new chunk ID generation
  • Validation Time: ~1μs for chunk ID validation
  • Serialization: ~3μs for JSON serialization
  • Deserialization: ~4μs for JSON deserialization
  • Memory Usage: ~32 bytes per chunk ID instance
  • Comparison Speed: O(1) for equality, O(log n) for ordering
  • Thread Safety: Immutable value objects are fully thread-safe

§Validation Rules

The chunk ID validation enforces several business rules:

  • Non-Nil Constraint: Chunk IDs cannot be nil (all zeros)
  • Timestamp Validation: Timestamps cannot be more than 1 day in the future
  • Format Validation: Must be valid ULID format
  • Category Validation: Must belong to “file_chunk” category

§Best Practices

§Chunk ID Management

  • Use Natural Ordering: Leverage ULID’s temporal ordering for processing
  • Validate Early: Always validate chunk IDs at system boundaries
  • Consistent Serialization: Use standard string representation across systems
  • Error Handling: Implement proper error handling for invalid IDs

§Processing Workflows

  • Sequential Processing: Process chunks in chronological order when possible
  • Status Tracking: Maintain chunk processing status for monitoring
  • Batch Operations: Group chunks for efficient batch processing
  • Recovery Handling: Implement recovery mechanisms for failed chunks

§Performance Optimization

  • Efficient Collections: Use BTreeSet/BTreeMap for ordered chunk collections
  • Minimal Conversions: Avoid unnecessary string conversions
  • Batch Validation: Validate multiple chunks together when possible
  • Memory Management: Reuse chunk ID instances where appropriate

§Cross-Platform Compatibility

The chunk ID format is designed for cross-platform compatibility:

  • Rust: FileChunkId newtype wrapper with full validation
  • Go: FileChunkID struct with equivalent interface
  • Python: FileChunkId class with similar validation
  • JSON: Direct string representation for API compatibility
  • Database: TEXT column with ULID string storage

Structs§

FileChunkId
File chunk identifier value object for type-safe chunk management