1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
//! Lock-free high-performance ingestion subsystem
//!
//! This module provides a complete lock-free data ingestion system with
//! configurable ACID guarantees. It enables linear scaling of write
//! throughput across CPU cores while maintaining transactional safety.
//!
//! # Architecture Overview
//!
//! ```text
//! ┌─────────────────────────────────────────────────────────────────────────┐
//! │ Lock-Free Ingestion System │
//! ├─────────────────────────────────────────────────────────────────────────┤
//! │ │
//! │ User API │
//! │ ┌─────────────────────────────────────────────────────────────────┐ │
//! │ │ LockFreeIngestionEngine │ │
//! │ │ begin_transaction() → insert/update/delete → commit/abort │ │
//! │ └─────────────────────────────────────────────────────────────────┘ │
//! │ │ │
//! │ ┌───────────────────────────┼───────────────────────────────────┐ │
//! │ │ ▼ │ │
//! │ │ ┌───────────────┐ ┌───────────────┐ ┌──────────────────┐ │ │
//! │ │ │ Config │ │ RowIdGen │ │ WriteCoordinator │ │ │
//! │ │ │ SafetyLevel │ │ Hierarchical │ │ TxnBuffers │ │ │
//! │ │ │ - Full │ │ or Batched │ │ GroupCommit │ │ │
//! │ │ │ - Batched │ │ │ │ │ │ │
//! │ │ │ - Async │ │ Lock-free │ │ Lock-free │ │ │
//! │ │ │ - Unsafe │ │ unique IDs │ │ buffering │ │ │
//! │ │ └───────────────┘ └───────────────┘ └──────────────────┘ │ │
//! │ │ │ │
//! │ │ ┌────────────────────────────────────────────────────────┐ │ │
//! │ │ │ PartitionedWalManager │ │ │
//! │ │ │ ┌────────┐ ┌────────┐ ┌────────┐ ┌────────┐ │ │ │
//! │ │ │ │ WAL P0 │ │ WAL P1 │ │ WAL P2 │ ... │ WAL Pn │ │ │ │
//! │ │ │ └────────┘ └────────┘ └────────┘ └────────┘ │ │ │
//! │ │ │ │ │ │
//! │ │ │ Two-Phase Commit for cross-partition transactions │ │ │
//! │ │ │ Parallel fsync for linear I/O scaling │ │ │
//! │ │ └────────────────────────────────────────────────────────┘ │ │
//! │ └───────────────────────────────────────────────────────────────┘ │
//! │ │
//! └─────────────────────────────────────────────────────────────────────────┘
//! ```
//!
//! # Safety Levels
//!
//! The system supports four safety levels that trade durability for performance:
//!
//! | Level | Durability | Performance | Use Case |
//! |----------|------------|-------------|-----------------------------|
//! | Full | ✓ Zero loss| 1x baseline | Financial, critical data |
//! | Batched | ~Up to N | 3-5x | High throughput, small loss |
//! | Async | ~Recent | 5-10x | Analytics, logs |
//! | Unsafe | ✗ All | 10-50x | Bulk load, temp data |
//!
//! # Key Features
//!
//! - **Lock-free row ID generation**: No coordination between threads
//! - **Per-thread write buffers**: No contention on hot paths
//! - **Partitioned WAL**: Linear I/O scaling with CPU cores
//! - **Group commit**: Amortized fsync cost across transactions
//! - **Two-phase commit**: ACID atomicity for multi-partition writes
//! - **Hierarchical IDs**: Self-describing IDs, no persistence needed
//!
//! # Usage Examples
//!
//! ## Full ACID Mode (Default)
//!
//! ```ignore
//! use heliosdb::storage::lockfree::*;
//!
//! let config = LockFreeIngestionConfig::default();
//! let engine = LockFreeIngestionEngine::new(config, "/path/to/wal")?;
//!
//! let txn = engine.begin_transaction()?;
//! let row_id = engine.generate_row_id("users");
//! engine.insert(&txn, "users", row_id, &serialized_data)?;
//! engine.commit(txn)?; // WAL fsynced before return
//! ```
//!
//! ## Bulk Load Mode (Maximum Performance)
//!
//! ```ignore
//! let config = LockFreeIngestionConfig::for_maximum_performance();
//! let engine = LockFreeIngestionEngine::new(config, "/path/to/wal")?;
//!
//! // Load millions of rows with minimal overhead
//! let result = engine.bulk_insert("events", rows_iterator)?;
//! println!("Loaded {} rows", result.rows_inserted);
//!
//! // Checkpoint before shutdown
//! engine.checkpoint()?;
//! ```
//!
//! ## Batched Mode (Controlled Loss Window)
//!
//! ```ignore
//! let config = LockFreeIngestionConfig {
//! safety_level: IngestionSafetyLevel::Batched {
//! batch_size: 1000,
//! batch_timeout_ms: 100,
//! },
//! ..Default::default()
//! };
//! ```
//!
//! # ACID Guarantees
//!
//! The lock-free architecture maintains ACID properties through:
//!
//! - **Atomicity**: Transaction buffers ensure all-or-nothing
//! - **Consistency**: WAL recovery restores consistent state
//! - **Isolation**: MVCC with read timestamps (no read locks needed)
//! - **Durability**: Configurable via safety level
//!
//! Even in "unsafe" mode, Atomicity, Consistency, and Isolation are preserved.
//! Only Durability is configurable.
// Re-export configuration types
pub use ;
// Re-export row ID generation
pub use ;
// Re-export write buffer types
pub use ;
// Re-export WAL management
pub use ;
// Re-export high-level API
pub use ;