sentinel-crypto 2.0.1

Cryptographic utilities for Cyberpath Sentinel, including hashing and signing.
Documentation

Cyberpath License: Apache 2.0 Codecov Crates.io Version docs.rs

A document-based DBMS written in Rust that stores all data as files on disk, where tables are represented by folders and each document's primary key is the filename. Every piece of data is inspectable, auditable, and compliant by design.

Why Cyberpath Sentinel?

Modern databases prioritize speed. Cyberpath Sentinel prioritizes trust, transparency, and compliance.

Perfect For

  • Audit Logs - Every entry is a file, versioned with Git
  • Certificate Management - Secure, inspectable, with OS-level ACLs
  • Compliance Rules & Policies - GDPR right-to-delete is literally rm file
  • Encryption Key Management - Keys stored as files with filesystem security
  • Regulatory Reporting - All data is immediately forensic-friendly
  • Edge Devices & Disconnected Systems - No server required, works with Git sync
  • Zero-Trust Infrastructure - Inspect everything before trusting it

NOT For

  • Real-time bidding systems
  • High-frequency trading platforms
  • Streaming analytics pipelines
  • Multi-million row transactional systems (yet)

Massive Advantages

Auditability & Security

  • Every document is inspectable with cat or your favorite editor
  • Versioned transparently with Git—see who changed what, when, and why
  • Secured with OS-level ACLs; no database user management nonsense
  • Cryptographic hashing enables forensic integrity verification

Operational Simplicity

  • Use standard UNIX tools: rsync for replication, tar for backups, grep for queries
  • No database daemon to manage, update, or patch
  • Deploy to any device with a filesystem; scaling is adding folders
  • Disaster recovery: git clone and you're done

Compliance-Ready

  • GDPR right-to-delete: rm file.json and it's gone (with audit trail)
  • Immutable audit logs with append-only patterns
  • PII can be encrypted at rest and in transit with cryptographic keys
  • Regulatory bodies love "show me the data"—here it is, plain text

Zero Lock-In

  • Data is pure JSON/BSON—no proprietary binary formats
  • Migrate to PostgreSQL, MongoDB, or DuckDB using standard tools
  • Your data isn't trapped in a vendor ecosystem
  • Export to CSV, XML, or custom formats trivially

Perfect Secure Ecosystem

  • Integrates seamlessly with security tools and compliance frameworks
  • Designed for organizations managing sensitive security and compliance data
  • Audit trails that satisfy SOC 2, ISO 27001, HIPAA requirements
  • Natural fit for DevOps, SRE, and security operations teams

Real Trade-offs (We're Honest)

Concurrency Complexity

  • Multi-writer scenarios require file locks or Write-Ahead Logging (WAL)
  • Not optimized for thousands of concurrent writes
  • Proposed Solution: Locking strategies, eventual consistency, and deterministic replication

Query Performance

  • No native B-tree indices; initial queries scan files
  • Proposed Solution: In-memory caching, lazy indexing, and hash-based sharding

Scaling Limits

  • Single folder performance degrades around 4M files
  • Proposed Solution: Hash-based sharding, distributed stores, and hierarchical folders

Partial Write Safety

  • Power failure mid-write requires careful handling
  • Proposed Solution: Write-Ahead Logging, atomic rename patterns, and checksums

Not for High-Throughput

  • Bad for 100K+ operations per second
  • Good for audit logs, configuration management, and compliance data

⚡ Quick Start

Installation

# Install from crates.io
cargo install sentinel-dbms

# Or add to your Cargo.toml
[dependencies]
sentinel-dbms = "0.1"

# Or build from source
git clone https://github.com/cyberpath-HQ/sentinel
cd sentinel
cargo build --release

Library Usage

use sentinel_dbms::{Store, QueryBuilder, Operator, SortOrder};
use serde_json::json;
use futures::TryStreamExt;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Create a store (async I/O with optional passphrase for signing)
    let store = Store::new("./data", None).await?;

    // Access a collection (creates data/users/ folder if needed)
    let users = store.collection("users").await?;

    // Insert a document (creates data/users/user-123.json)
    users.insert("user-123", json!({
        "name": "Alice Johnson",
        "email": "alice@example.com",
        "role": "admin",
        "department": "Engineering",
        "age": 30
    })).await?;

    // Retrieve a document
    if let Some(user) = users.get("user-123").await? {
        println!("User: {}", serde_json::to_string_pretty(&user)?);
    }

    // Update a document
    users.update("user-123", json!({
        "name": "Alice Smith",
        "email": "alice.smith@example.com",
        "role": "admin",
        "department": "Engineering",
        "age": 31
    })).await?;

    // Stream all documents (memory-efficient for large collections)
    let stream = users.all();
    let docs: Vec<_> = stream.try_collect().await?;
    println!("Total documents: {}", docs.len());

    // Query with filters using predicate functions
    let adults = users.filter(|doc| {
        doc.data().get("age")
            .and_then(|v| v.as_i64())
            .map_or(false, |age| age >= 18)
    });
    let adult_docs: Vec<_> = adults.try_collect().await?;
    println!("Adults: {}", adult_docs.len());

    // Advanced querying with QueryBuilder
    let query = QueryBuilder::new()
        .filter("age", Operator::GreaterThan, json!(25))
        .filter("role", Operator::Equals, json!("admin"))
        .sort("name", SortOrder::Ascending)
        .limit(10)
        .offset(0)
        .project(vec!["name".to_string(), "email".to_string()])
        .build();

    let result = users.query(query).await?;
    println!("Query executed in {:?}", result.execution_time);

    // Stream query results
    let stream = result.documents;
    futures::pin_mut!(stream);
    while let Some(doc) = stream.try_next().await? {
        println!("Found: {:?}", doc.data());
    }

    // Delete a document (soft delete to .deleted/ folder)
    users.delete("user-123").await?;

    Ok(())
}

CLI Usage

The Sentinel CLI provides commands for managing stores and documents from the terminal:

# Initialize a store
sentinel init --path ./my-store

# Create a collection
sentinel create-collection --store ./my-store --name users

# Insert a document
sentinel insert \
  --store ./my-store \
  --collection users \
  --id user-1 \
  --data '{"name": "Alice", "email": "alice@example.com"}'

# Query documents
sentinel query \
  --store ./my-store \
  --collection users \
  --filter "age>25" \
  --filter "role=admin" \
  --sort "name:asc" \
  --limit 10 \
  --project "name,email"

# Get a specific document
sentinel get --store ./my-store --collection users --id user-1

# List all documents in a collection
sentinel list --store ./my-store --collection users

# Update a document
sentinel update \
  --store ./my-store \
  --collection users \
  --id user-1 \
  --data '{"name": "Alice Smith", "email": "alice.smith@example.com"}'

# Delete a document
sentinel delete --store ./my-store --collection users --id user-1

Folder Structure

data/
├── users/
│   ├── user-123.json
│   ├── user-456.json
│   └── .deleted/
│       └── user-789.json
├── audit_logs/
│   ├── audit-2026-01-01.json
│   └── audit-2026-01-02.json
└── certs/
    ├── cert-a1b2c3.pem
    └── cert-d4e5f6.pem

📋 Features & Status

✅ Implemented Features

  • Document Storage - JSON files stored as inspectable documents
  • Collections - Folder-based namespaces for organizing documents
  • Async CRUD Operations - Full Create, Read, Update, Delete with Tokio
  • Document Metadata - Automatic version, timestamps, hash, and signature
  • Streaming API - Memory-efficient streaming for large datasets
  • Advanced Querying - Filter, sort, limit, offset, and projection
  • Query Builder - Fluent API for building complex queries
  • Cryptography Module - Modular hashing, signing, encryption, key derivation
  • Multiple Algorithms:
    • Hashing: BLAKE3
    • Signing: Ed25519
    • Encryption: XChaCha20-Poly1305, AES-256-GCM-SIV, Ascon-128
    • Key Derivation: Argon2id, PBKDF2
  • Soft Deletes - Documents moved to .deleted/ folder
  • CLI Tool - Complete command-line interface with all operations
  • Passphrase Protection - Encrypt signing keys with passphrases
  • Global Crypto Config - Flexible configuration for algorithm selection
  • Comprehensive Testing - Extensive unit and integration tests
  • Benchmarking - Performance benchmarks with Criterion

🚧 In Progress

  • Write-Ahead Logging (WAL) for durability
  • File locking for concurrent writes
  • Lazy indexing for improved query performance
  • In-memory caching with LRU eviction

📋 Planned Features

  • Full-text search capabilities
  • Replication and sync (Git integration)
  • Backup and restore utilities
  • Compliance reporting dashboards
  • Multi-version concurrency control (MVCC)
  • Access control lists (ACLs)
  • Content-addressable storage
  • Merkle tree verification for integrity

Architecture

Design Principles

  1. Filesystem is the Database - Leverage OS reliability and tooling
  2. Immutability by Default - Audit trails, append-only logs, deletions are soft
  3. Security First - File permissions, encryption, cryptographic verification
  4. UNIX Philosophy - Do one thing well, compose with standard tools
  5. Zero External Dependencies - Works offline, on edge devices

Core Components

┌─────────────────────────────────────────┐
│  Cyberpath Sentinel Client (Rust)       │
├─────────────────────────────────────────┤
│  Query Engine & Filtering               │
│  Transaction Manager (WAL)              │
│  Caching Layer (in-memory)              │
├─────────────────────────────────────────┤
│  File I/O & Concurrency Control         │
│  Encryption & Signing                   │
│  Checksum Verification                  │
├─────────────────────────────────────────┤
│  Filesystem (ext4, NTFS, APFS, etc.)    │
│  OS-level ACLs & Permissions            │
└─────────────────────────────────────────┘

Security & Compliance

Built-In Security

  • Filesystem Permissions - Leverage OS ACLs for access control
  • Encryption at Rest - Optional AES-256 encryption for sensitive files
  • Checksums & Integrity - SHA-256 hashing for corruption detection
  • Immutable Audit Logs - Append-only journals with cryptographic signatures
  • Soft Deletes - Deleted files moved to .deleted/ folder (recoverable)

Compliance Ready

  • GDPR - Right-to-delete is filesystem deletion
  • SOC 2 - Audit trails are intrinsic to the system
  • HIPAA - Encryption at rest and in transit
  • PCI-DSS - File-level access controls
  • ISO 27001 - Security controls built into architecture

Performance Characteristics

Best Case Scenarios

Operation Time Complexity Notes
Insert O(1) Single file write
Get O(1) Direct filename lookup
Delete O(1) Rename to .deleted/
Update O(1) Atomic file rename
List O(n) Scan directory for filenames
Filter O(n) Scan all files in collection
Index O(n) Build lazy indices on first query

Optimization Strategies

  • Caching - LRU cache for frequently accessed documents
  • Sharding - Hash-based sharding for 4M+ file collections
  • Lazy Indexing - Create indices on first query, reuse thereafter
  • Write Coalescing - Batch multiple writes to reduce fsync calls

Deployment Options

Single Machine

# Initialize store
cyberpath-sentinel init --path /var/cyberpath

# Run server
cyberpath-sentinel serve --path /var/cyberpath --port 8080

Replicated Cluster (Git-backed)

# Primary node
git init --bare /data/cyberpath.git
cyberpath-sentinel serve --path /data/cyberpath --git-push origin main

# Secondary node
git clone /data/cyberpath.git /data/cyberpath
cyberpath-sentinel serve --path /data/cyberpath --git-pull origin main

Encrypted Cloud Storage

# Backup to S3 with encryption
cyberpath-sentinel backup --path /data --s3-bucket compliance-backups --encryption AES256

Documentation

Contributing

We welcome contributions! This is an ambitious project, and we need help with:

  • Core DBMS features (transactions, indexing, caching)
  • Encryption and security implementations
  • Performance optimization and benchmarking
  • Documentation and tutorials
  • Real-world use case implementations

See CONTRIBUTING.md for guidelines.

License

This project is licensed under the Apache 2.0 License - see the LICENSE file for details.

Vision

Cyberpath Sentinel is building the gold standard for transparent, auditable data storage. We're not trying to replace PostgreSQL or MongoDB, we're creating something new for organizations that choose accountability over speed, and transparency over convenience.

In five years, we want Cyberpath Sentinel to be synonymous with:

  • Compliance as Code - Your data is your audit trail
  • Security by Design - Every document is inspectable and verifiable
  • Trust Infrastructure - The UNIX of data storage
  • Edge Intelligence - Secure, offline-first data stores on every device

Contact & Community

Made with ❤️ for security teams, compliance officers, and developers who believe data should be transparent.