tegdb 0.5.0

The name TegridyDB (short for TegDB) is inspired by the Tegridy Farm in South Park and tries to correct some of the wrong database implementations, such as null support, implicit conversion support, etc.
Documentation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
# TegDB

TegDB is a lightweight, embedded database engine with a SQL-like interface designed for simplicity, performance, and reliability. It provides ACID transactions, crash recovery, and efficient RDBMS.

> **Design Philosophy**: TegDB prioritizes simplicity and reliability over complexity. It uses a single-threaded design to eliminate concurrency bugs, reduce memory overhead, and provide predictable performance - making it ideal for embedded systems and applications where resource efficiency matters more than parallel processing.

## Key Features

### 🚀 **Performance & Footprint**

- Key->offset B+tree index; values are stored on disk, small values can be inlined to reduce IO
- Bounded cache (configurable cap) to boost hot-read hit rates
- Primary key optimized queries (O(log n) lookups)
- Streaming query processing with early LIMIT termination
- Efficient binary serialization

### 🔒 **ACID Transactions**

- Atomicity: All-or-nothing transaction execution
- Consistency: Schema validation and constraint enforcement  
- Isolation: Write-through with snapshot-like behavior
- Durability: Write-ahead logging with commit markers

### 🛡️ **Reliability**

- Crash recovery from write-ahead log
- File locking prevents concurrent access corruption
- Graceful handling of partial writes and corruption
- Automatic rollback on transaction drop
- Strong durability by default: per-transaction fsync, configurable group commit
- Observability: metrics for bytes read/written, cache hits/misses, fsync counts

### 📦 **Simple Design**

- Single-threaded architecture eliminates race conditions
- Minimal dependencies (only `fs2` for file locking)
- Clean separation of concerns across layers
- Extensive test coverage including ACID compliance

### 🔌 **Extension System**

- PostgreSQL-inspired plugin architecture
- Built-in string functions: UPPER, LOWER, LENGTH, TRIM, SUBSTR, REPLACE, CONCAT, REVERSE
- Built-in math functions: ABS, CEIL, FLOOR, ROUND, SQRT, POW, MOD, SIGN
- Create custom scalar and aggregate functions
- Type-safe function signatures with validation

## Getting Started

### Quick Start (CLI + MinIO in 2–3 minutes)

This walkthrough uses released builds and the CLI tools, no code required.

1) Install the CLIs

```bash
# Clone the repository
git clone https://github.com/minifish-org/tegdb.git
cd tegdb

# Build both CLI tools (tg and tgstream)
cargo build --release

# Or build individually:
# cargo build --release --bin tg
# cargo build --release --bin tgstream

# Copy binaries to PATH (or add target/release to your PATH)
cp target/release/tg ~/.cargo/bin/
cp target/release/tgstream ~/.cargo/bin/

# Ensure ~/.cargo/bin is on your PATH
export PATH="$HOME/.cargo/bin:$PATH"
```

**Alternative**: If you prefer installing from crates.io:

```bash
cargo install tegdb --version 0.3.0 --bin tg
cargo install tegdb --version 0.3.0 --bin tgstream
```

2) Start MinIO locally and create a bucket

```bash
# Run MinIO
docker run -d --name minio -p 9000:9000 -p 9001:9001 \
  -e MINIO_ROOT_USER=minioadmin -e MINIO_ROOT_PASSWORD=minioadmin \
  quay.io/minio/minio server /data --console-address :9001

# Create a bucket using the MinIO Console at http://localhost:9001 (Login: minioadmin/minioadmin)
# In the Console: Buckets → Create Bucket → Name: tegdb-backups
```

3) Configure AWS-compatible env vars for MinIO

```bash
export AWS_ACCESS_KEY_ID=minioadmin
export AWS_SECRET_ACCESS_KEY=minioadmin
export AWS_REGION=us-east-1
export AWS_ENDPOINT_URL=http://127.0.0.1:9000
export TGSTREAM_BUCKET=tegdb-backups
```

4) Create and query a database with the `tg` CLI

```bash
# Use an absolute file URL ending with .teg
DB=file:///$(pwd)/quickstart.teg

# Create table and insert a row
tg "$DB" --command "CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT(32));"
tg "$DB" --command "INSERT INTO users (id, name) VALUES (1, 'Alice');"

# Query
tg "$DB" --command "SELECT * FROM users;"
```

5) Enable continuous cloud backup to MinIO with `tgstream`

```bash
# Create config file (use absolute path - replace /path/to with your actual path)
cat > tgstream.toml <<EOF
database_path = "$(pwd)/quickstart.teg"

[s3]
bucket = "tegdb-backups"
prefix = "dbs/quickstart"
region = "us-east-1"
endpoint = "http://127.0.0.1:9000"
access_key_id = "minioadmin"
secret_access_key = "minioadmin"

[base]
interval_minutes = 1
segment_size_mb = 50

[segment]
min_bytes = 1024
debounce_ms = 1500

[retention]
bases = 3
max_segments_bytes = 107374182400

gzip = true
EOF

# Start replication (best run under a supervisor/tmux)
tgstream run --config tgstream.toml

# In another terminal, verify backup is working:
tgstream list --config tgstream.toml
# You should see base snapshots appearing every 15 minutes
```

6) Restore database from backup

```bash
# List available backups
tgstream list --config tgstream.toml

# Restore to latest state
tgstream restore --config tgstream.toml --to $(pwd)/restored.teg

# Verify restored data
tg "file:///$(pwd)/restored.teg" --command "SELECT * FROM users;"
# Should show: Alice
```

**Example restore scenario**: If your original database gets corrupted or deleted, you can restore it from MinIO:
```bash
# Original database is lost/corrupted
rm quickstart.teg

# Restore from backup
tgstream restore --config tgstream.toml --to $(pwd)/quickstart.teg

# Continue using the restored database
tg "file:///$(pwd)/quickstart.teg" --command "SELECT * FROM users;"
```

## Using TegDB as a Library

Add TegDB to your `Cargo.toml`:

```toml
[dependencies]
tegdb = "0.3.0"
```

### Basic Usage

```rust
use tegdb::Database;

fn main() -> tegdb::Result<()> {
    // Open or create a database (must use absolute file:// path ending in .teg)
    let mut db = Database::open("file:///tmp/my_app.teg")?;
    
    // Create a table
    db.execute("CREATE TABLE users (id INTEGER PRIMARY KEY, name TEXT(32), age INTEGER)")?;
    
    // Insert data
    db.execute("INSERT INTO users (id, name, age) VALUES (1, 'Alice', 30)")?;
    db.execute("INSERT INTO users (id, name, age) VALUES (2, 'Bob', 25)")?;
    
    // Query data
    let result = db.query("SELECT name, age FROM users WHERE age > 25")?;
    
    println!("Found {} users:", result.len());
    for row in result.rows() {
        if let (Some(name), Some(age)) = (row.first(), row.get(1)) {
            println!("User: {:?}, Age: {:?}", name, age);
        }
    }
    
    Ok(())
}
```

### Transaction Example

```rust
use tegdb::Database;

fn main() -> tegdb::Result<()> {
    let mut db = Database::open("file:///tmp/bank.teg")?;
    
    // Create accounts table
    db.execute("CREATE TABLE accounts (id INTEGER PRIMARY KEY, name TEXT(32), balance INTEGER)")?;
    db.execute("INSERT INTO accounts (id, name, balance) VALUES (1, 'Alice', 1000)")?;
    db.execute("INSERT INTO accounts (id, name, balance) VALUES (2, 'Bob', 500)")?;
    
    // Transfer funds using explicit transaction
    let mut tx = db.begin_transaction()?;
    
    // Debit from Alice's account
    tx.execute("UPDATE accounts SET balance = balance - 100 WHERE id = 1")?;
    
    // Credit to Bob's account  
    tx.execute("UPDATE accounts SET balance = balance + 100 WHERE id = 2")?;
    
    // Commit the transaction (or it will auto-rollback on drop)
    tx.commit()?;
    
    println!("Transfer completed successfully!");
    Ok(())
}
```

## SQL Support

TegDB supports a comprehensive subset of SQL:

### Data Definition Language (DDL)

```sql
-- Create tables with constraints
CREATE TABLE products (
    id INTEGER PRIMARY KEY,
    name TEXT NOT NULL,
    price REAL,
    category TEXT
);

-- Drop tables
DROP TABLE IF EXISTS old_table;
```

### Data Manipulation Language (DML)

```sql
-- Insert single or multiple rows
INSERT INTO products (id, name, price) VALUES (1, 'Widget', 19.99);
INSERT INTO products (id, name, price) VALUES 
    (2, 'Gadget', 29.99),
    (3, 'Tool', 39.99);

-- Update with conditions
UPDATE products SET price = 24.99 WHERE name = 'Widget';

-- Delete with conditions
DELETE FROM products WHERE price < 20.00;

-- Query with filtering and limits
SELECT name, price FROM products 
WHERE category = 'Electronics' 
LIMIT 10;
```

### Transaction Control

```sql
BEGIN;
UPDATE accounts SET balance = balance - 100 WHERE id = 1;
UPDATE accounts SET balance = balance + 100 WHERE id = 2;
COMMIT;
-- or ROLLBACK;
```

### Supported Data Types

- `INTEGER` - 64-bit signed integers
- `REAL` - 64-bit floating point numbers  
- `TEXT` - UTF-8 strings (requires length specification, e.g., TEXT(100))
- `NULL` - Null values

## Extension System

TegDB provides a PostgreSQL-inspired extension system for adding custom functions. Extensions can be loaded via SQL commands (PostgreSQL-style) or programmatically via the Rust API.

### Loading Extensions via SQL (PostgreSQL-style)

The recommended way to manage extensions is using SQL commands:

```sql
-- Load built-in extensions
CREATE EXTENSION tegdb_string;
CREATE EXTENSION tegdb_math;

-- Use extension functions in SQL
SELECT UPPER('hello'), SQRT(144);

-- Load custom extension from dynamic library
CREATE EXTENSION my_extension;
-- Or specify explicit path
CREATE EXTENSION my_extension WITH PATH '/path/to/libmy_extension.so';

-- Extensions persist automatically - they'll be loaded on next database open
-- Remove an extension
DROP EXTENSION my_extension;
```

### Loading Extensions via Rust API

You can also register extensions programmatically:

```rust
use tegdb::{Database, StringFunctionsExtension, MathFunctionsExtension, SqlValue};

let mut db = Database::open("file:///tmp/test.teg")?;

// Register built-in extensions
db.register_extension(Box::new(StringFunctionsExtension))?;
db.register_extension(Box::new(MathFunctionsExtension))?;

// Call functions directly
let result = db.call_function("UPPER", &[SqlValue::Text("hello".to_string())])?;
assert_eq!(result, SqlValue::Text("HELLO".to_string()));

let result = db.call_function("SQRT", &[SqlValue::Integer(144)])?;
assert_eq!(result, SqlValue::Real(12.0));
```

### Available Functions

**String Functions** (tegdb_string):
- `UPPER(text)` - Convert to uppercase
- `LOWER(text)` - Convert to lowercase
- `LENGTH(text)` - String length
- `TRIM(text)` / `LTRIM(text)` / `RTRIM(text)` - Trim whitespace
- `SUBSTR(text, start, length)` - Extract substring
- `REPLACE(text, from, to)` - Replace occurrences
- `CONCAT(text, ...)` - Concatenate strings (variadic)
- `REVERSE(text)` - Reverse string

**Math Functions** (tegdb_math):
- `ABS(number)` - Absolute value
- `CEIL(number)` / `FLOOR(number)` - Ceiling/floor
- `ROUND(number, decimals)` - Round to decimal places
- `SQRT(number)` - Square root
- `POW(base, exponent)` - Power
- `MOD(a, b)` - Modulo
- `SIGN(number)` - Sign (-1, 0, or 1)

### Creating Custom Extensions

Define custom extensions by implementing the `Extension` trait:

```rust
use tegdb::{Extension, ScalarFunction, FunctionSignature, ArgType, DataType, SqlValue};

// Define a custom function
struct DoubleFunction;

impl ScalarFunction for DoubleFunction {
    fn name(&self) -> &'static str { "DOUBLE" }
    
    fn signature(&self) -> FunctionSignature {
        FunctionSignature::new(vec![ArgType::Numeric], DataType::Real)
    }
    
    fn execute(&self, args: &[SqlValue]) -> Result<SqlValue, String> {
        match &args[0] {
            SqlValue::Integer(i) => Ok(SqlValue::Integer(i * 2)),
            SqlValue::Real(r) => Ok(SqlValue::Real(r * 2.0)),
            _ => Err("Expected numeric argument".to_string()),
        }
    }
}

// Define an extension
struct MyExtension;

impl Extension for MyExtension {
    fn name(&self) -> &'static str { "my_extension" }
    fn version(&self) -> &'static str { "1.0.0" }
    fn scalar_functions(&self) -> Vec<Box<dyn ScalarFunction>> {
        vec![Box::new(DoubleFunction)]
    }
}

// Register and use
db.register_extension(Box::new(MyExtension))?;
let result = db.call_function("DOUBLE", &[SqlValue::Integer(21)])?;
assert_eq!(result, SqlValue::Integer(42));
```

### Extension Management

**Via SQL:**
```sql
-- Extensions are automatically persisted and loaded on database open
-- No need to re-run CREATE EXTENSION after restarting
```

**Via Rust API:**
```rust
// List registered extensions
for (name, version) in db.list_extensions() {
    println!("{} v{}", name, version);
}

// Check if a function exists
if db.has_function("UPPER") {
    println!("UPPER function is available");
}

// Unregister an extension
db.unregister_extension("my_extension")?;
```

### Creating Loadable Extensions

To create a dynamic library extension that can be loaded via `CREATE EXTENSION`:

1. **Create a Rust library project:**
   ```bash
   cargo new --lib my_extension
   cd my_extension
   ```

2. **Configure Cargo.toml:**
   ```toml
   [lib]
   crate-type = ["cdylib"]
   
   [dependencies]
   tegdb = { path = "../tegdb" }  # Or from crates.io
   ```

3. **Implement the extension:**
   ```rust
   use tegdb::{Extension, ExtensionWrapper, ScalarFunction, SqlValue, FunctionSignature, ArgType, DataType};
   
   struct MyFunction;
   
   impl ScalarFunction for MyFunction {
       fn name(&self) -> &'static str { "MY_FUNCTION" }
       fn signature(&self) -> FunctionSignature {
           FunctionSignature::new(vec![ArgType::TextLike], DataType::Text(None))
       }
       fn execute(&self, args: &[SqlValue]) -> Result<SqlValue, String> {
           match &args[0] {
               SqlValue::Text(s) => Ok(SqlValue::Text(s.to_uppercase())),
               _ => Err("Expected text argument".to_string()),
           }
       }
   }
   
   struct MyExtension;
   
   impl Extension for MyExtension {
       fn name(&self) -> &'static str { "my_extension" }
       fn version(&self) -> &'static str { "1.0.0" }
       fn scalar_functions(&self) -> Vec<Box<dyn ScalarFunction>> {
           vec![Box::new(MyFunction)]
       }
   }
   
   #[no_mangle]
   pub extern "C" fn create_extension() -> *mut ExtensionWrapper {
       Box::into_raw(Box::new(ExtensionWrapper {
           extension: Box::new(MyExtension),
       }))
   }
   ```

4. **Build and install:**
   ```bash
   cargo build --release
   cp target/release/libmy_extension.so ~/.tegdb/extensions/
   # Or place in ./extensions/ relative to your database
   ```

5. **Use in SQL:**
   ```sql
   CREATE EXTENSION my_extension;
   SELECT MY_FUNCTION('hello');
   ```

For complete examples, see `examples/extension_demo.rs` and `examples/extension_template.rs`.

## Performance Characteristics

### Time Complexity

- **Primary key lookups**: O(log n)
- **Range scans**: O(log n + k) where k = result size
- **Inserts/Updates/Deletes**: O(log n)
- **Schema operations**: O(1) with caching

### Memory Usage

- **In-memory index**: BTreeMap holds key -> value offset (values on disk), small values can inline
- **Bounded cache**: Byte-capped value/page cache for hot data
- **Lazy allocation**: Undo logs only allocated when needed
- **Streaming queries**: LIMIT processed without loading full result

### Storage Format

- **Fixed header**: 64-byte header with magic `TEGDB\0`, version (1), limits, flags
- **Append-only log**: Fast writes after the header, no seek overhead
- **Binary serialization**: Compact data representation
- **Key->offset layout**: B+tree holds offsets; values stored in the data region, small values inline
- **Automatic compaction**: Reclaims space from old entries while preserving header
- **Crash recovery**: Replay from last commit marker

## Cloud Backup & Replication (tgstream)

TegDB includes `tgstream`, a standalone streaming backup tool that continuously replicates your database to cloud storage (S3, MinIO, etc.), similar to Litestream for SQLite.

### Features

- **Incremental Replication**: Tracks file offsets and uploads only committed changes
- **Base Snapshots**: Periodic full database snapshots for fast recovery
- **Automatic Rotation Detection**: Handles database compaction/rotation automatically
- **Point-in-Time Recovery**: Restore to any previous state using base + segments
- **Retention Policies**: Configurable retention for snapshots and segments
- **Compression**: Optional gzip compression to reduce storage costs

### Installation

Install from crates.io:

```bash
cargo install tegdb --version 0.3.0 --bin tgstream
# Or install both binaries:
cargo install tegdb --version 0.3.0 --bin tg --bin tgstream
```

### Configuration

Create a configuration file `tgstream.toml`:

```toml
database_path = "/absolute/path/to/your/database.teg"

[s3]
bucket = "my-backup-bucket"
prefix = "dbs/mydb"
region = "us-east-1"
# Optional: For MinIO or custom S3-compatible storage
endpoint = "http://localhost:9000"  # MinIO endpoint
access_key_id = "minioadmin"        # MinIO access key
secret_access_key = "minioadmin"     # MinIO secret key

[base]
interval_minutes = 60  # Create new base snapshot every hour
segment_size_mb = 100   # Or after 100MB of segments

[segment]
min_bytes = 1024        # Minimum segment size to upload
debounce_ms = 2000      # Wait 2 seconds before uploading

[retention]
bases = 3               # Keep last 3 base snapshots
max_segments_bytes = 107374182400  # 100GB max segments

gzip = true             # Enable compression
```

For AWS S3, you can omit `endpoint`, `access_key_id`, and `secret_access_key` and use environment variables or IAM roles instead:

```bash
export AWS_ACCESS_KEY_ID=your-key
export AWS_SECRET_ACCESS_KEY=your-secret
export AWS_REGION=us-east-1
```

### Commands

```bash
# Run continuous replication
tgstream run --config tgstream.toml

# Create a one-off snapshot
tgstream snapshot --config tgstream.toml

# Restore database from backup
tgstream restore --config tgstream.toml --to /path/to/restored.teg

# List available snapshots
tgstream list --config tgstream.toml

# Prune old snapshots
tgstream prune --config tgstream.toml
```

### How It Works

1. **Monitoring**: Tegstream monitors your `.teg` file for new committed transactions
2. **Segment Uploads**: After each commit, new data is uploaded as incremental segments
3. **Base Snapshots**: Periodically (every N minutes or after N MB of segments), a full snapshot is created
4. **State Tracking**: Local state file tracks progress, file metadata, and prevents duplicate uploads
5. **Restore**: Downloads base snapshot + all subsequent segments to reconstruct the database

The tool is designed to be run as a background service alongside your application, providing continuous off-site backup with minimal overhead.

```mermaid
flowchart LR
    subgraph App
        TG[tg CLI / App]
    end
    DB[(.teg file)]
    TS[tgstream]
    S3[(S3/MinIO Bucket)]

    TG -- SQL --> DB
    TS -- monitor commits --> DB
    TS -- base snapshots --> S3
    TS -- incremental segments --> S3
    S3 -- base+segments --> Restore[tgstream restore]
```

## Architecture Overview

TegDB implements a clean layered architecture with four distinct layers:

```mermaid
flowchart TB
    A[Database API\nSQLite-like interface\nwith schema caching]
    B[SQL Executor\nQuery optimization\n+ execution]
    C[SQL Parser\nnom-based AST]
    D[Storage Engine\nKV + WAL + TX]

    A --> B --> C --> D
```

### Core Components

- **Storage Engine**: BTreeMap-based in-memory storage with append-only log persistence
- **Transaction System**: Write-through transactions with undo logging and commit markers
- **SQL Support**: Full SQL parser and executor supporting DDL and DML operations
- **Index-Organized Tables**: Primary key optimization with direct key lookups
- **Schema Caching**: Database-level schema caching for improved performance
- **Crash Recovery**: WAL-based recovery using transaction commit markers

See [ARCHITECTURE.md](ARCHITECTURE.md) for detailed information about:

- Layer-by-layer implementation details
- Storage format and recovery mechanisms  
- Memory management and performance optimizations
- Transaction system and ACID guarantees
- Query optimization and execution strategies

## Advanced Usage

### Engine Configuration

```rust
use std::time::Duration;
use tegdb::storage_engine::{DurabilityConfig, DurabilityLevel, EngineConfig};

let config = EngineConfig {
    max_key_size: 1024,        // 1KB max key size
    max_value_size: 256 * 1024, // 256KB max value size
    auto_compact: true,         // Auto-compact on open
    initial_capacity: None,     // No default key cap; set Some(...) to enforce a hard key cap
    preallocate_size: None,     // No default disk preallocation; set Some(bytes) to cap/preallocate
    // Durability: default is Immediate (fsync every commit)
    durability: DurabilityConfig {
        level: DurabilityLevel::Immediate,
        group_commit_interval: Duration::from_millis(0), // Set >0 to enable group commit
    },
    // Inline hot/small values to avoid disk reads; the rest spill to data pages.
    inline_value_threshold: 64,          // bytes
    cache_size_bytes: 8 * 1024 * 1024,   // value/page cache cap
    // Compaction: absolute threshold + ratio + min delta bytes since last compact
    compaction_absolute_threshold_bytes: 10 * 1024 * 1024,
    compaction_ratio: 2.0,
    compaction_min_delta_bytes: 2 * 1024 * 1024,
    ..Default::default()
};

// Note: Custom config requires low-level API
```

Key defaults:
- Values are stored on disk; the B+tree indexes key -> value offset. Small values (<= `inline_value_threshold`) stay inline.
- A byte-capped value cache (`cache_size_bytes`) keeps hot values in memory.
- Durability defaults to per-commit `fsync`; set `DurabilityLevel::GroupCommit` with a non-zero `group_commit_interval` to coalesce flushes.
- Default compaction uses an absolute threshold (10 MiB), fragmentation ratio (2.0), and a minimum written delta (2 MiB) since the last compaction.
- No default hard cap on key count or disk size; set `initial_capacity` and `preallocate_size` to enforce limits in production.

Metrics (observability):

```rust
let metrics = engine.metrics();
println!(
    "bytes_read={}, bytes_written={}, cache_hits={}, cache_misses={}, fsyncs={}",
    metrics.bytes_read,
    metrics.bytes_written,
    metrics.cache_hits,
    metrics.cache_misses,
    metrics.fsync_count
);
```

### Low-Level Engine API

For advanced use cases, you can access low-level APIs via module paths:

```rust
use tegdb::storage_engine::{StorageEngine, EngineConfig};

// Direct key-value operations (requires absolute PathBuf)
let mut engine = StorageEngine::new(std::path::PathBuf::from("/tmp/data.teg"))?;
engine.set(b"key", b"value".to_vec())?;
let value = engine.get(b"key");

// Transaction control
let mut tx = engine.begin_transaction();
tx.set(b"key1", b"value1".to_vec())?;
tx.set(b"key2", b"value2".to_vec())?;
tx.commit()?;
```

## Development

### Building from Source

```bash
# Standard build
cargo build

# Run tests
cargo test

# Run benchmarks  
cargo bench
```

### Testing

TegDB includes comprehensive tests covering:

- ACID transaction properties
- Crash recovery scenarios  
- SQL parsing and execution
- Performance benchmarks
- Edge cases and error conditions

```bash
# Run the full native test suite
./run_all_tests.sh

# Run with verbose output
./run_all_tests.sh --verbose

# CI-friendly run (preserves test output)
./run_all_tests.sh --ci
```

### Code Quality

Use the following commands to keep the tree clean:

```bash
# Format source code
cargo fmt --all

# Run Clippy with the same settings as CI
cargo clippy --all-targets --all-features -- -D warnings

# Run the full CI-equivalent precheck suite
./ci_precheck.sh
```

### Benchmarks

Run performance benchmarks against other embedded databases:

```bash
cargo bench
```

Included benchmarks compare against:

- SQLite
- sled  
- redb

## Design Principles

1. **Simplicity First**: Prefer simple, understandable solutions
2. **Reliability**: Prioritize correctness over performance optimizations
3. **Standard Library**: Use std library when possible to minimize dependencies
4. **Single Threaded**: Eliminate concurrency complexity and bugs
5. **Resource Efficient**: Optimize for memory and CPU usage

## Limitations

### Current Limitations

- **Single-threaded**: No concurrent access support
- **No secondary indexes**: Only primary key optimization
- **Limited SQL**: Subset of full SQL standard
- **No foreign keys**: Basic constraint support only
- **No joins**: Single table queries only

### Future Enhancements

- Secondary index support
- JOIN operation support  
- More SQL features (subqueries, aggregation)
- Compression for large values
- Streaming for very large result sets
- Enhanced backup features (multi-DB support, encryption-at-rest)

## License

Licensed under AGPL-3.0. See [LICENSE](LICENSE) for details.

The AGPL-3.0 ensures that any modifications to TegDB remain open source and available to the community.

## Contributing

Contributions welcome! Please:

1. Follow the design principles above
2. Include comprehensive tests
3. Update documentation for new features
4. Ensure benchmarks still pass

See [CONTRIBUTING.md](CONTRIBUTING.md) for detailed guidelines.