title: "AllSource Core" status: CURRENT last_updated: 2026-02-15 category: service port: 3900 technology: Rust version: "0.10.1"

AllSource Core

High-performance event store built in Rust. Columnar storage, schema validation, stream processing, leader-follower replication. 1,300 tests passing.

Current Version: v0.10.1 | crates.io | docs.rs

Key Numbers

Installation

As a Library

cargo add allsource-core@0.10

Or in Cargo.toml:

[dependencies]
allsource-core = "0.10"

As a Server

# From source
git clone https://github.com/all-source-os/all-source.git
cd all-source/apps/core
cargo run --release

# Docker
docker pull ghcr.io/all-source-os/allsource-core:0.10.1
docker run -p 3900:3900 ghcr.io/all-source-os/allsource-core:0.10.1

Binaries

Storage & Durability

AllSource Core is the database. Event data survives restarts.

Data flow on ingestion:

1. WAL write (durable)
2. DashMap index update
3. Projection update
4. Pipeline processing
5. Parquet flush (periodic)
6. WebSocket broadcast
7. Auto-snapshot (if threshold reached)

Replication

Leader-follower replication via WAL shipping. No external coordination service required.

Leader (port 3900, replication port 3910)
  |-- WAL Ship --> Follower 1 (port 3900)
  |-- WAL Ship --> Follower 2 (port 3900)

Modes: async (default), semi-sync (1 ACK), sync (all ACKs).

Configure via environment:

• ALLSOURCE_ROLE=leader|follower
• ALLSOURCE_REPLICATION_PORT=3910
• ALLSOURCE_LEADER_URL=http://leader:3910

The allsource-sentinel binary monitors health and triggers automatic failover.

Features

Event Store

• Append-only immutable event log
• Time-travel queries (entity state at any timestamp)
• Batch ingestion
• Multi-tenancy with per-tenant isolation and quotas

Schema Registry

• JSON Schema validation at ingestion time
• Schema versioning with compatibility checking (backward, forward, full, none)
• Subject-based organization

Stream Processing (6 operators)

• Filter: eq, ne, gt, lt, contains
• Map: Transform field values (uppercase, lowercase, trim, math)
• Reduce: Aggregations (count, sum, avg, min, max) with grouping
• Window: Time-based (tumbling, sliding, session)
• Enrich: External data lookup
• Branch: Conditional event routing

Event Replay

• Point-in-time replay with progress tracking
• Projection rebuilding
• Configurable batch sizes, async execution, cancellable

Analytics

• Event frequency analysis
• Statistical summaries
• Event correlation

Security

• JWT authentication with RBAC (Admin, Operator, Reader)
• API key authentication for service-to-service
• Per-tenant rate limiting
• IP filtering (allowlist/blocklist)
• Audit logging (event-sourced, backed by Core's own WAL)

Optional Features (Cargo feature flags)

API Reference

All endpoints use the /api/v1/ prefix. Core returns wrapped responses: {"events": [...], "count": N}.

Events

POST   /api/v1/events                    # Ingest event (returns 200)
POST   /api/v1/events/batch              # Batch ingest
GET    /api/v1/events/query              # Query events (?entity_id=, ?event_type=, ?since=, ?limit=)
GET    /api/v1/entities/:id/state        # Entity state (?as_of= for time-travel)
GET    /api/v1/entities/:id/snapshot     # Latest snapshot
GET    /api/v1/stats                     # Store statistics
WS     /api/v1/events/stream             # Real-time WebSocket stream

Schemas

POST   /api/v1/schemas                   # Register schema
GET    /api/v1/schemas                   # List subjects
GET    /api/v1/schemas/:subject          # Get schema (?version=)
GET    /api/v1/schemas/:subject/versions # List versions
POST   /api/v1/schemas/validate          # Validate event against schema
PUT    /api/v1/schemas/:subject/compatibility  # Set compatibility mode

Projections & Snapshots

GET    /api/v1/projections               # List projections
POST   /api/v1/snapshots                 # Create snapshot
GET    /api/v1/snapshots                 # List snapshots
GET    /api/v1/snapshots/:id/latest      # Get latest snapshot

Pipelines

POST   /api/v1/pipelines                 # Register pipeline
GET    /api/v1/pipelines                 # List pipelines
GET    /api/v1/pipelines/:id             # Get pipeline
DELETE /api/v1/pipelines/:id             # Remove pipeline
GET    /api/v1/pipelines/stats           # All pipeline stats
GET    /api/v1/pipelines/:id/stats       # Pipeline stats
PUT    /api/v1/pipelines/:id/reset       # Reset pipeline state

Replay

POST   /api/v1/replay                    # Start replay
GET    /api/v1/replay                    # List replays
GET    /api/v1/replay/:id               # Get progress
POST   /api/v1/replay/:id/cancel        # Cancel replay
DELETE /api/v1/replay/:id               # Delete replay

Analytics & Operations

GET    /api/v1/analytics/frequency       # Event frequency
GET    /api/v1/analytics/summary         # Statistical summary
GET    /api/v1/analytics/correlation     # Event correlation
POST   /api/v1/compaction/trigger        # Trigger compaction
GET    /api/v1/compaction/stats          # Compaction stats
GET    /health                           # Health check (root path)
GET    /metrics                          # Prometheus metrics

Tenants, Auth, Audit, Config

POST   /api/v1/tenants                   # Create tenant
GET    /api/v1/tenants                   # List tenants
GET    /api/v1/tenants/:id              # Get tenant
PUT    /api/v1/tenants/:id              # Update tenant
DELETE /api/v1/tenants/:id              # Delete tenant
POST   /api/v1/auth/login               # Login
POST   /api/v1/auth/api-keys            # Create API key
GET    /api/v1/audit/events             # Query audit log
GET    /api/v1/config                    # Get configuration
PUT    /api/v1/config                    # Update configuration

Architecture

Clean Architecture with strict layered dependencies:

src/
├── domain/                 # Business logic (zero external deps)
│   ├── entities/           # Event, Tenant, Schema, Projection, EventStream, AuditEvent...
│   ├── value_objects/      # TenantId, EntityId, EventType, PartitionKey, Money...
│   └── repositories/       # Trait definitions only
│
├── application/            # Use cases & orchestration
│   ├── use_cases/          # IngestEvent, QueryEvents, ManageTenant, ManageSchema...
│   ├── services/           # Analytics, Pipeline, Replay, Schema validation
│   └── dto/                # Request/Response DTOs
│
├── infrastructure/         # Technical implementations
│   ├── web/                # HTTP handlers, WebSocket, API routes
│   ├── persistence/        # WAL, Parquet storage, Snapshots, Compaction, Indexes
│   ├── replication/        # WAL shipper, WAL receiver, protocol
│   ├── security/           # Auth, rate limiting, middleware
│   ├── repositories/       # In-memory, event-sourced implementations
│   ├── search/             # Vector search, keyword search (optional)
│   ├── cluster/            # Cluster coordination
│   ├── observability/      # Metrics, tracing
│   ├── config/             # Configuration management
│   └── di/                 # Dependency injection (ServiceContainer)
│
├── security/               # Security integration tests
├── store.rs                # EventStore core implementation
├── main.rs                 # Server entry point
├── lib.rs                  # Library exports
└── bin/
    ├── allsource-admin.rs  # Admin CLI
    └── allsource-sentinel.rs # Failover sentinel

Dependency rule: Inner layers never depend on outer layers. Domain has zero external dependencies.

Development

Prerequisites

• Rust 1.92+ (MSRV)
• Cargo

Commands

cargo build --release       # Build
cargo test --lib            # Run unit tests (1,300 tests)
cargo test                  # Run all tests including integration
cargo bench                 # Run benchmarks
RUST_LOG=debug cargo run    # Run with debug logging

Quality Gates

make check          # Run all quality gates
make format         # Auto-fix formatting
make ci             # Full CI pipeline locally

Enforced: rustfmt, clippy (zero warnings), cargo-sort, all tests passing, release build.

Configuration

License

MIT