tecton-0.1.1 is not a library.

Introduction

Tecton is a modular, single-binary (or modular-crate) alternative to legacy Hadoop / MinIO-style stacks. It brings together:

Storage — an S3-compatible HTTP API powered by OpenDAL (local FS, AWS S3, Azure Blob, GCS)
Compute — SQL over Parquet/CSV via Apache DataFusion and Arrow
Desktop UI — a modern Tauri + React control plane with dark/light themes

Why Tecton?

Principle	What it means
Speed	Async Rust (`tokio`), zero-cost abstractions, release LTO builds
Security	No hardcoded secrets; env-based cloud credentials; input validation on CLI & API
Modularity	Run storage, compute, or both — pick the surface you need
Offline-first	Local filesystem backend and desktop UI work without the cloud

Architecture Overview

tecton/
├── crates/
│   ├── tecton-core      # Config, logging (tracing), errors (thiserror), shared types
│   ├── tecton-io        # OpenDAL storage engine + Axum S3-compatible API
│   └── tecton-compute   # DataFusion SQL engine over Parquet/CSV
├── apps/
│   ├── tecton-cli       # crates.io package `tecton` (CLI binary)
│   └── tecton-ui        # Tauri desktop app (React + Vite + Tailwind)
├── scripts/             # Dataset generators, icons, benchmarks
└── tecton.toml.example  # Sample configuration

Crate / App	Role
`tecton-core`	Shared utilities, `TectonConfig`, `TectonError`, system status types
`tecton-io`	Multi-cloud object storage + lightweight HTTP API
`tecton-compute`	DataFusion SQL, CSV→Parquet auto-optimize, streaming results
`tecton` (CLI, from `apps/tecton-cli`)	Binary: `start`, `status`, `query`, `ls`, `benchmark`, `version`
`tecton-ui`	Desktop app (not published to crates.io)

Install (crates.io)

cargo install tecton

Crate	Link
CLI	https://crates.io/crates/tecton
Core	https://crates.io/crates/tecton-core
IO	https://crates.io/crates/tecton-io
Compute	https://crates.io/crates/tecton-compute

Prerequisites

Rust 1.80+ (rustup)
Node.js 20+ and npm
Python 3.10+ with numpy + pyarrow (only for dataset generators)
Tauri prerequisites for your OS — see Tauri getting started
- Windows: WebView2 (usually preinstalled on Win10/11)
- macOS: Xcode CLT
- Linux: webkit2gtk and related packages

Optional: copy tecton.toml.example → tecton.toml to customize bind address, storage root, and dataset directory.

Build from source

# Clone

git clone https://github.com/rabbittrix/tecton.git

cd tecton


# Build all Rust workspace members (release)

cargo build --release


# CLI binary lands at:

#   target/release/tecton        (Linux/macOS)

#   target/release/tecton.exe    (Windows)


# Desktop UI dependencies (from repo root or apps/tecton-ui)

npm install


# Dev mode (Vite + Tauri) — works from repo root

npm run tauri:dev


# Production UI bundle

npm run tauri:build

Usage

CLI

# Start storage + compute (default)

tecton start --mode full


# Storage API only (S3-compatible HTTP on 127.0.0.1:9000)

tecton start --mode storage --port 9000


# Compute only

tecton start --mode compute


# Health / status snapshot (JSON)

tecton status


# List objects

tecton ls datasets/


# Run a read-only SQL query (table name is always `data`)

tecton query --file apps/tecton-ui/src-tauri/data/BANK_DATA.csv \

  --sql "SELECT gender, COUNT(*) AS n FROM data GROUP BY gender ORDER BY n DESC"


# Query a Parquet shard folder (billion-row stress tests)

tecton query --file apps/tecton-ui/src-tauri/data/bank_billion \

  --sql "SELECT bank_name, COUNT(*) AS n FROM data GROUP BY bank_name ORDER BY n DESC LIMIT 20"


# Benchmark cold (CSV→Parquet) vs warm (Parquet cache)

tecton benchmark --file apps/tecton-ui/src-tauri/data/BANK_DATA.csv

HTTP (when storage is running)

curl http://127.0.0.1:9000/health

curl -X PUT --data-binary @file.parquet http://127.0.0.1:9000/datasets/file.parquet

curl http://127.0.0.1:9000/?prefix=datasets/

Desktop UI

Dashboard — system status (Running/Stopped), storage usage, quick actions
Analyze — open a CSV/Parquet file or a Parquet shard folder, run SQL, view chart/table/JSON
Theme toggle — dark / light mode (defaults to dark)
Support / Donate — SEPA IBAN / QR + GitHub Sponsors

UI result cues: execution time (ms), Auto-optimized to Parquet badge, truncation warning when streamed rows are capped, and a loading overlay during queries.

Test datasets

Synthetic datasets live under apps/tecton-ui/src-tauri/data/ (large generated files are gitignored; small *.sample.csv seeds are committed).

Path	Domain	Scale	Format
`BANK_DATA.sample.csv`	Banking	seed	CSV (committed)
`BANK_DATA.csv`	Banking	~10M rows / ~961 MiB	CSV
`bank_billion/`	Banking	1 000 000 001 rows / ~10.4 GiB	Parquet shards
`EDP_DATA.sample.csv`	Energy (EDP-style metering/billing)	seed	CSV (committed)
`EDP_DATA.csv`	Energy	~10M rows / ~1.4 GiB	CSV
`edp_billion/`	Energy	up to 1B rows	Parquet shards
`TELECOM_EU_ME.sample.csv`	Telecom EU + Middle East	seed	CSV (committed)
`TELECOM_EU_ME.csv`	Telecom	~10M rows / ~1.2 GiB	CSV
`telecom_eu_me_billion/`	Telecom	up to 1B rows	Parquet shards

Prefer Parquet shard folders for billion-row tests. A single 1B-row CSV is typically 100 GiB+ and slow to query.

Generate datasets

# Bank — expand CSV (~10M)

python scripts/generate_bank_data.py --rows 10000001


# Bank — billion-row Parquet shards (resume-safe)

python scripts/generate_bank_billion.py --rows 1000000001 --shard-rows 2000000


# EDP + Telecom — medium CSVs (~10M each)

python scripts/generate_edp_telecom_billion.py --dataset both --rows 10000000 --format csv


# EDP + Telecom — billion-row Parquet shards

python scripts/generate_edp_telecom_billion.py --dataset both --rows 1000000000 --format parquet

Example SQL

-- Banking
SELECT gender, COUNT(*) AS count FROM data GROUP BY gender ORDER BY count DESC;

-- EDP
SELECT region, COUNT(*) AS n, ROUND(AVG(consumption_kwh), 2) AS avg_kwh
FROM data GROUP BY region ORDER BY n DESC;

-- Telecom EU / ME
SELECT country, operator, COUNT(*) AS subscribers, ROUND(AVG(arpu_eur), 2) AS avg_arpu
FROM data GROUP BY country, operator ORDER BY subscribers DESC LIMIT 50;

In the UI: Analyze → Open file for CSVs/Parquet files, or Open folder for bank_billion, edp_billion, or telecom_eu_me_billion.

Performance

Tecton auto-optimizes large CSVs so interactive analysis stays fast after the first run.

Auto-optimization

Behavior	Detail
Threshold	CSV files > 100 MiB
First query	Scan CSV → write sibling `.parquet` → run SQL on Parquet
Later queries	Reuse the cached `.parquet` (same basename)
`SELECT *` safety	Bare `SELECT ` without `LIMIT` gets `LIMIT 1000`; stream hard-cap is 10 000* rows

Benchmark (`tecton benchmark`)

Measured on Windows (release build) with BANK_DATA.csv (~961 MiB, 10 000 001 rows, 7 columns):

SELECT gender, COUNT(*) AS count FROM data GROUP BY gender ORDER BY count DESC

Run	Wall clock	Engine time	Peak RSS	Optimization
Cold (CSV → Parquet + query)	2 282 ms	2 282 ms	179.7 MiB	Converted CSV to Parquet
Warm (Parquet cache)	57 ms	56 ms	60.1 MiB	Using existing Parquet cache

Speedup: ~40× cold → warm (97.5% less wall time)
Parquet cache size: ~94 MiB (~10× smaller than the CSV)

cargo run -p tecton-cli --release -- benchmark --file apps/tecton-ui/src-tauri/data/BANK_DATA.csv

Comparison with traditional tools

Same machine, same dataset and aggregation (wall clock includes load + GROUP BY gender):

Tool / path	Wall time	Notes
Tecton warm (Parquet)	57 ms	Cached after first conversion
Tecton cold (CSV→Parquet)	2 282 ms	Includes one-time conversion (~94 MiB Parquet)
pandas `read_csv` + `groupby`	4 276 ms	Materializes the full CSV in memory
pandas `read_parquet` + `groupby`	1 003 ms	Uses Tecton-produced Parquet cache

python scripts/compare_bench.py

Numbers vary by CPU, disk cache, and OS buffering. Prefer tecton benchmark on your machine for capacity planning.

Configuration

Environment variables use the TECTON_ prefix (nested keys with __):

export TECTON_SERVER__PORT=9000

export TECTON_STORAGE__BACKEND=fs

export TECTON_LOG_LEVEL=debug

Cloud backends (s3, azblob, gcs) read credentials from the standard environment variables (AWS_*, Azure / GCP ADC) — never from source code.

Donation

If Tecton helps your business, consider supporting its development. Your support keeps the project independent, secure, and moving fast.

SEPA bank transfer (EU)

Field	Value
Beneficiary	Roberto de Souza
IBAN	`PT50 3560 0001 9001 8573 6595 0`
BIC / SWIFT	`REVOPTP2`

Scan a SEPA QR (EPC) from the desktop app Dashboard → Support / Donate, or enter the IBAN above in your banking app.

Also: GitHub Sponsors

Author & Contact


Author	Roberto de Souza
Email	rabbittrix@hotmail.com
Repository	https://github.com/rabbittrix/tecton.git
License	Apache-2.0

tecton 0.1.1