<p align="center">
<img src="apps/tecton-ui/src-tauri/icons/logo.svg" alt="Tecton logo" width="120" />
</p>
<h1 align="center">Tecton</h1>
<p align="center">
<strong>Unified data infrastructure</strong> — modular storage + compute in a blazing-fast Rust stack.
</p>
<p align="center">
<a href="LICENSE"><img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" alt="License: Apache-2.0" /></a>
<a href="https://crates.io/crates/tecton"><img src="https://img.shields.io/crates/v/tecton.svg" alt="crates.io" /></a>
<a href="https://docs.rs/tecton-core"><img src="https://img.shields.io/docsrs/tecton-core" alt="docs.rs" /></a>
<img src="https://img.shields.io/badge/Language-Rust-orange.svg" alt="Language: Rust" />
<img src="https://img.shields.io/badge/Status-MVP-brightgreen.svg" alt="Status: MVP" />
<img src="https://img.shields.io/badge/UI-Tauri%20%2B%20React-0d9488.svg" alt="UI: Tauri + React" />
</p>
---
# Introduction
**Tecton** is a modular, single-binary (or modular-crate) alternative to legacy Hadoop / MinIO-style stacks. It brings together:
- **Storage** — an S3-compatible HTTP API powered by [OpenDAL](https://opendal.apache.org/) (local FS, AWS S3, Azure Blob, GCS)
- **Compute** — SQL over Parquet/CSV via [Apache DataFusion](https://datafusion.apache.org/) and Arrow
- **Desktop UI** — a modern Tauri + React control plane with dark/light themes
## Why Tecton?
| **Speed** | Async Rust (`tokio`), zero-cost abstractions, release LTO builds |
| **Security** | No hardcoded secrets; env-based cloud credentials; input validation on CLI & API |
| **Modularity** | Run storage, compute, or both — pick the surface you need |
| **Offline-first** | Local filesystem backend and desktop UI work without the cloud |
---
## Architecture Overview
```text
tecton/
├── crates/
│ ├── tecton-core # Config, logging (tracing), errors (thiserror), shared types
│ ├── tecton-io # OpenDAL storage engine + Axum S3-compatible API
│ └── tecton-compute # DataFusion SQL engine over Parquet/CSV
├── apps/
│ ├── tecton-cli # crates.io package `tecton` (CLI binary)
│ └── tecton-ui # Tauri desktop app (React + Vite + Tailwind)
├── scripts/ # Dataset generators, icons, benchmarks
└── tecton.toml.example # Sample configuration
```
| `tecton-core` | Shared utilities, `TectonConfig`, `TectonError`, system status types |
| `tecton-io` | Multi-cloud object storage + lightweight HTTP API |
| `tecton-compute` | DataFusion SQL, CSV→Parquet auto-optimize, streaming results |
| `tecton` (CLI, from `apps/tecton-cli`) | Binary: `start`, `status`, `query`, `ls`, `benchmark`, `version` |
| `tecton-ui` | Desktop app (not published to crates.io) |
---
## Install (crates.io)
```bash
cargo install tecton
```
| CLI | https://crates.io/crates/tecton |
| Core | https://crates.io/crates/tecton-core |
| IO | https://crates.io/crates/tecton-io |
| Compute | https://crates.io/crates/tecton-compute |
---
## Prerequisites
- **Rust** 1.80+ ([rustup](https://rustup.rs/))
- **Node.js** 20+ and npm
- **Python** 3.10+ with `numpy` + `pyarrow` (only for dataset generators)
- **Tauri prerequisites** for your OS — see [Tauri getting started](https://v2.tauri.app/start/)
- Windows: WebView2 (usually preinstalled on Win10/11)
- macOS: Xcode CLT
- Linux: `webkit2gtk` and related packages
Optional: copy `tecton.toml.example` → `tecton.toml` to customize bind address, storage root, and dataset directory.
---
## Build from source
```bash
# Clone
git clone https://github.com/rabbittrix/tecton.git
cd tecton
# Build all Rust workspace members (release)
cargo build --release
# CLI binary lands at:
# target/release/tecton (Linux/macOS)
# target/release/tecton.exe (Windows)
# Desktop UI dependencies (from repo root or apps/tecton-ui)
npm install
# Dev mode (Vite + Tauri) — works from repo root
npm run tauri:dev
# Production UI bundle
npm run tauri:build
```
---
## Usage
### CLI
```bash
# Start storage + compute (default)
tecton start --mode full
# Storage API only (S3-compatible HTTP on 127.0.0.1:9000)
tecton start --mode storage --port 9000
# Compute only
tecton start --mode compute
# Health / status snapshot (JSON)
tecton status
# List objects
tecton ls datasets/
# Run a read-only SQL query (table name is always `data`)
tecton query --file apps/tecton-ui/src-tauri/data/BANK_DATA.csv \
--sql "SELECT gender, COUNT(*) AS n FROM data GROUP BY gender ORDER BY n DESC"
# Query a Parquet shard folder (billion-row stress tests)
tecton query --file apps/tecton-ui/src-tauri/data/bank_billion \
--sql "SELECT bank_name, COUNT(*) AS n FROM data GROUP BY bank_name ORDER BY n DESC LIMIT 20"
# Benchmark cold (CSV→Parquet) vs warm (Parquet cache)
tecton benchmark --file apps/tecton-ui/src-tauri/data/BANK_DATA.csv
```
### HTTP (when storage is running)
```bash
curl http://127.0.0.1:9000/health
curl -X PUT --data-binary @file.parquet http://127.0.0.1:9000/datasets/file.parquet
curl http://127.0.0.1:9000/?prefix=datasets/
```
### Desktop UI
- **Dashboard** — system status (Running/Stopped), storage usage, quick actions
- **Analyze** — open a CSV/Parquet file or a Parquet shard folder, run SQL, view chart/table/JSON
- **Theme toggle** — dark / light mode (defaults to dark)
- **Support / Donate** — SEPA IBAN / QR + GitHub Sponsors
UI result cues: execution time (ms), **Auto-optimized to Parquet** badge, truncation warning when streamed rows are capped, and a loading overlay during queries.
---
## Test datasets
Synthetic datasets live under `apps/tecton-ui/src-tauri/data/` (large generated files are gitignored; small `*.sample.csv` seeds are committed).
| `BANK_DATA.sample.csv` | Banking | seed | CSV (committed) |
| `BANK_DATA.csv` | Banking | ~10M rows / ~961 MiB | CSV |
| `bank_billion/` | Banking | **1 000 000 001 rows** / ~10.4 GiB | Parquet shards |
| `EDP_DATA.sample.csv` | Energy (EDP-style metering/billing) | seed | CSV (committed) |
| `EDP_DATA.csv` | Energy | ~10M rows / ~1.4 GiB | CSV |
| `edp_billion/` | Energy | up to **1B rows** | Parquet shards |
| `TELECOM_EU_ME.sample.csv` | Telecom EU + Middle East | seed | CSV (committed) |
| `TELECOM_EU_ME.csv` | Telecom | ~10M rows / ~1.2 GiB | CSV |
| `telecom_eu_me_billion/` | Telecom | up to **1B rows** | Parquet shards |
> Prefer **Parquet shard folders** for billion-row tests. A single 1B-row CSV is typically 100 GiB+ and slow to query.
### Generate datasets
```bash
# Bank — expand CSV (~10M)
python scripts/generate_bank_data.py --rows 10000001
# Bank — billion-row Parquet shards (resume-safe)
python scripts/generate_bank_billion.py --rows 1000000001 --shard-rows 2000000
# EDP + Telecom — medium CSVs (~10M each)
python scripts/generate_edp_telecom_billion.py --dataset both --rows 10000000 --format csv
# EDP + Telecom — billion-row Parquet shards
python scripts/generate_edp_telecom_billion.py --dataset both --rows 1000000000 --format parquet
```
### Example SQL
```sql
-- Banking
SELECT gender, COUNT(*) AS count FROM data GROUP BY gender ORDER BY count DESC;
-- EDP
SELECT region, COUNT(*) AS n, ROUND(AVG(consumption_kwh), 2) AS avg_kwh
FROM data GROUP BY region ORDER BY n DESC;
-- Telecom EU / ME
SELECT country, operator, COUNT(*) AS subscribers, ROUND(AVG(arpu_eur), 2) AS avg_arpu
FROM data GROUP BY country, operator ORDER BY subscribers DESC LIMIT 50;
```
In the UI: **Analyze → Open file** for CSVs/Parquet files, or **Open folder** for `bank_billion`, `edp_billion`, or `telecom_eu_me_billion`.
---
## Performance
Tecton auto-optimizes large CSVs so interactive analysis stays fast after the first run.
### Auto-optimization
| Threshold | CSV files **> 100 MiB** |
| First query | Scan CSV → write sibling `.parquet` → run SQL on Parquet |
| Later queries | Reuse the cached `.parquet` (same basename) |
| `SELECT *` safety | Bare `SELECT *` without `LIMIT` gets `LIMIT 1000`; stream hard-cap is **10 000** rows |
### Benchmark (`tecton benchmark`)
Measured on Windows (release build) with `BANK_DATA.csv` (~**961 MiB**, **10 000 001** rows, 7 columns):
```sql
SELECT gender, COUNT(*) AS count FROM data GROUP BY gender ORDER BY count DESC
```
| Cold (CSV → Parquet + query) | **2 282 ms** | 2 282 ms | 179.7 MiB | Converted CSV to Parquet |
| Warm (Parquet cache) | **57 ms** | 56 ms | 60.1 MiB | Using existing Parquet cache |
- **Speedup:** ~**40×** cold → warm (**97.5%** less wall time)
- **Parquet cache size:** ~94 MiB (~10× smaller than the CSV)
```bash
cargo run -p tecton-cli --release -- benchmark --file apps/tecton-ui/src-tauri/data/BANK_DATA.csv
```
### Comparison with traditional tools
Same machine, same dataset and aggregation (wall clock includes load + `GROUP BY gender`):
| **Tecton warm (Parquet)** | **57 ms** | Cached after first conversion |
| **Tecton cold (CSV→Parquet)** | **2 282 ms** | Includes one-time conversion (~94 MiB Parquet) |
| pandas `read_csv` + `groupby` | **4 276 ms** | Materializes the full CSV in memory |
| pandas `read_parquet` + `groupby` | **1 003 ms** | Uses Tecton-produced Parquet cache |
```bash
python scripts/compare_bench.py
```
> Numbers vary by CPU, disk cache, and OS buffering. Prefer `tecton benchmark` on your machine for capacity planning.
---
## Configuration
Environment variables use the `TECTON_` prefix (nested keys with `__`):
```bash
export TECTON_SERVER__PORT=9000
export TECTON_STORAGE__BACKEND=fs
export TECTON_LOG_LEVEL=debug
```
Cloud backends (`s3`, `azblob`, `gcs`) read credentials from the standard environment variables (`AWS_*`, Azure / GCP ADC) — never from source code.
---
## Donation
If Tecton helps your business, consider supporting its development. Your support keeps the project independent, secure, and moving fast.
### SEPA bank transfer (EU)
| **Beneficiary** | Roberto de Souza |
| **IBAN** | `PT50 3560 0001 9001 8573 6595 0` |
| **BIC / SWIFT** | `REVOPTP2` |
Scan a **SEPA QR** (EPC) from the desktop app Dashboard → Support / Donate, or enter the IBAN above in your banking app.
Also: **[GitHub Sponsors](https://github.com/sponsors/rabbittrix)**
---
## Author & Contact
| **Author** | Roberto de Souza |
| **Email** | [rabbittrix@hotmail.com](mailto:rabbittrix@hotmail.com) |
| **Repository** | [https://github.com/rabbittrix/tecton.git](https://github.com/rabbittrix/tecton.git) |
| **License** | [Apache-2.0](LICENSE) |
---
<p align="center">
<sub>Built by Expert software archeteture Roberto de Souza with Rust · OpenDAL · DataFusion · Tauri</sub>
</p>