---
title: "AllSource Core"
status: CURRENT
last_updated: 2026-02-15
category: service
port: 3900
technology: Rust
version: "0.10.1"
---
# AllSource Core
[](https://crates.io/crates/allsource-core)
[](https://docs.rs/allsource-core)
[](https://github.com/all-source-os/all-source/actions/workflows/ci.yml)
[](../../LICENSE)
High-performance event store built in Rust. Columnar storage, schema validation, stream processing, leader-follower replication. **1,300 tests passing.**
## Key Numbers
| Event ingestion | 469K events/sec |
| Entity query (p99) | 11.9 us |
| Concurrent writes (8 threads) | 8.0 ms/batch |
| Parquet batch write | 3.5 ms / 1000 events |
| Library tests | 1,300 |
| API endpoints | 50+ |
## Installation
### As a Library
```bash
cargo add allsource-core@0.10
```
Or in `Cargo.toml`:
```toml
[dependencies]
allsource-core = "0.10"
```
### As a Server
```bash
# From source
git clone https://github.com/all-source-os/all-source.git
cd all-source/apps/core
cargo run --release
# Docker
docker pull ghcr.io/all-source-os/allsource-core:0.10.1
docker run -p 3900:3900 ghcr.io/all-source-os/allsource-core:0.10.1
```
### Binaries
| `allsource-core` | Main event store server (port 3900) |
| `allsource-admin` | Admin CLI for maintenance operations |
| `allsource-sentinel` | Automated leader election and failover |
## Storage & Durability
AllSource Core **is** the database. Event data survives restarts.
| **WAL** | Crash recovery | CRC32 checksums, configurable fsync (default 100ms) |
| **Parquet** | Columnar persistence | Snappy compression, periodic flush |
| **DashMap** | In-memory reads | Lock-free concurrent map, O(1) lookups |
| **Snapshots** | Point-in-time state | Automatic creation, entity-level |
Data flow on ingestion:
1. WAL write (durable)
2. DashMap index update
3. Projection update
4. Pipeline processing
5. Parquet flush (periodic)
6. WebSocket broadcast
7. Auto-snapshot (if threshold reached)
## Replication
Leader-follower replication via WAL shipping. No external coordination service required.
```
Leader (port 3900, replication port 3910)
|-- WAL Ship --> Follower 1 (port 3900)
|-- WAL Ship --> Follower 2 (port 3900)
```
Modes: `async` (default), `semi-sync` (1 ACK), `sync` (all ACKs).
Configure via environment:
- `ALLSOURCE_ROLE=leader|follower`
- `ALLSOURCE_REPLICATION_PORT=3910`
- `ALLSOURCE_LEADER_URL=http://leader:3910`
The `allsource-sentinel` binary monitors health and triggers automatic failover.
## Features
### Event Store
- Append-only immutable event log
- Time-travel queries (entity state at any timestamp)
- Batch ingestion
- Multi-tenancy with per-tenant isolation and quotas
### Schema Registry
- JSON Schema validation at ingestion time
- Schema versioning with compatibility checking (backward, forward, full, none)
- Subject-based organization
### Stream Processing (6 operators)
- **Filter**: eq, ne, gt, lt, contains
- **Map**: Transform field values (uppercase, lowercase, trim, math)
- **Reduce**: Aggregations (count, sum, avg, min, max) with grouping
- **Window**: Time-based (tumbling, sliding, session)
- **Enrich**: External data lookup
- **Branch**: Conditional event routing
### Event Replay
- Point-in-time replay with progress tracking
- Projection rebuilding
- Configurable batch sizes, async execution, cancellable
### Analytics
- Event frequency analysis
- Statistical summaries
- Event correlation
### Security
- JWT authentication with RBAC (Admin, Operator, Reader)
- API key authentication for service-to-service
- Per-tenant rate limiting
- IP filtering (allowlist/blocklist)
- Audit logging (event-sourced, backed by Core's own WAL)
### Optional Features (Cargo feature flags)
| Vector search | `vector-search` | fastembed, instant-distance |
| Keyword search | `keyword-search` | tantivy |
| RocksDB storage | `rocksdb-storage` | rocksdb |
## API Reference
All endpoints use the `/api/v1/` prefix. Core returns wrapped responses: `{"events": [...], "count": N}`.
### Events
```
POST /api/v1/events # Ingest event (returns 200)
POST /api/v1/events/batch # Batch ingest
GET /api/v1/events/query # Query events (?entity_id=, ?event_type=, ?since=, ?limit=)
GET /api/v1/entities/:id/state # Entity state (?as_of= for time-travel)
GET /api/v1/entities/:id/snapshot # Latest snapshot
GET /api/v1/stats # Store statistics
WS /api/v1/events/stream # Real-time WebSocket stream
```
### Schemas
```
POST /api/v1/schemas # Register schema
GET /api/v1/schemas # List subjects
GET /api/v1/schemas/:subject # Get schema (?version=)
GET /api/v1/schemas/:subject/versions # List versions
POST /api/v1/schemas/validate # Validate event against schema
PUT /api/v1/schemas/:subject/compatibility # Set compatibility mode
```
### Projections & Snapshots
```
GET /api/v1/projections # List projections
POST /api/v1/snapshots # Create snapshot
GET /api/v1/snapshots # List snapshots
GET /api/v1/snapshots/:id/latest # Get latest snapshot
```
### Pipelines
```
POST /api/v1/pipelines # Register pipeline
GET /api/v1/pipelines # List pipelines
GET /api/v1/pipelines/:id # Get pipeline
DELETE /api/v1/pipelines/:id # Remove pipeline
GET /api/v1/pipelines/stats # All pipeline stats
GET /api/v1/pipelines/:id/stats # Pipeline stats
PUT /api/v1/pipelines/:id/reset # Reset pipeline state
```
### Replay
```
POST /api/v1/replay # Start replay
GET /api/v1/replay # List replays
GET /api/v1/replay/:id # Get progress
POST /api/v1/replay/:id/cancel # Cancel replay
DELETE /api/v1/replay/:id # Delete replay
```
### Analytics & Operations
```
GET /api/v1/analytics/frequency # Event frequency
GET /api/v1/analytics/summary # Statistical summary
GET /api/v1/analytics/correlation # Event correlation
POST /api/v1/compaction/trigger # Trigger compaction
GET /api/v1/compaction/stats # Compaction stats
GET /health # Health check (root path)
GET /metrics # Prometheus metrics
```
### Tenants, Auth, Audit, Config
```
POST /api/v1/tenants # Create tenant
GET /api/v1/tenants # List tenants
GET /api/v1/tenants/:id # Get tenant
PUT /api/v1/tenants/:id # Update tenant
DELETE /api/v1/tenants/:id # Delete tenant
POST /api/v1/auth/login # Login
POST /api/v1/auth/api-keys # Create API key
GET /api/v1/audit/events # Query audit log
GET /api/v1/config # Get configuration
PUT /api/v1/config # Update configuration
```
## Architecture
Clean Architecture with strict layered dependencies:
```
src/
├── domain/ # Business logic (zero external deps)
│ ├── entities/ # Event, Tenant, Schema, Projection, EventStream, AuditEvent...
│ ├── value_objects/ # TenantId, EntityId, EventType, PartitionKey, Money...
│ └── repositories/ # Trait definitions only
│
├── application/ # Use cases & orchestration
│ ├── use_cases/ # IngestEvent, QueryEvents, ManageTenant, ManageSchema...
│ ├── services/ # Analytics, Pipeline, Replay, Schema validation
│ └── dto/ # Request/Response DTOs
│
├── infrastructure/ # Technical implementations
│ ├── web/ # HTTP handlers, WebSocket, API routes
│ ├── persistence/ # WAL, Parquet storage, Snapshots, Compaction, Indexes
│ ├── replication/ # WAL shipper, WAL receiver, protocol
│ ├── security/ # Auth, rate limiting, middleware
│ ├── repositories/ # In-memory, event-sourced implementations
│ ├── search/ # Vector search, keyword search (optional)
│ ├── cluster/ # Cluster coordination
│ ├── observability/ # Metrics, tracing
│ ├── config/ # Configuration management
│ └── di/ # Dependency injection (ServiceContainer)
│
├── security/ # Security integration tests
├── store.rs # EventStore core implementation
├── main.rs # Server entry point
├── lib.rs # Library exports
└── bin/
├── allsource-admin.rs # Admin CLI
└── allsource-sentinel.rs # Failover sentinel
```
**Dependency rule**: Inner layers never depend on outer layers. Domain has zero external dependencies.
## Development
### Prerequisites
- Rust 1.92+ (MSRV)
- Cargo
### Commands
```bash
cargo build --release # Build
cargo test --lib # Run unit tests (1,300 tests)
cargo test # Run all tests including integration
cargo bench # Run benchmarks
RUST_LOG=debug cargo run # Run with debug logging
```
### Quality Gates
```bash
make check # Run all quality gates
make format # Auto-fix formatting
make ci # Full CI pipeline locally
```
Enforced: rustfmt, clippy (zero warnings), cargo-sort, all tests passing, release build.
## Configuration
| `PORT` | `3900` | HTTP server port |
| `RUST_LOG` | `info` | Log level |
| `ALLSOURCE_ROLE` | `leader` | Node role: `leader` or `follower` |
| `ALLSOURCE_REPLICATION_PORT` | `3910` | Replication listener port (leader) |
| `ALLSOURCE_LEADER_URL` | - | Leader replication URL (follower) |
| `ALLSOURCE_READ_ONLY` | `false` | Read-only mode (alternative to role=follower) |
| `ALLSOURCE_DATA_DIR` | `data/` | Data directory for WAL and Parquet files |
## License
[MIT](../../LICENSE)