tecton 0.1.0

Unified data infrastructure CLI — S3-compatible storage + DataFusion SQL
tecton-0.1.0 is not a library.

Introduction

Tecton is a modular, single-binary (or modular-crate) alternative to legacy Hadoop / MinIO-style stacks. It brings together:

  • Storage — an S3-compatible HTTP API powered by OpenDAL (local FS, AWS S3, Azure Blob, GCS)
  • Compute — SQL over Parquet/CSV via Apache DataFusion and Arrow
  • Desktop UI — a modern Tauri + React control plane with dark/light themes

Why Tecton?

Principle What it means
Speed Async Rust (tokio), zero-cost abstractions, release LTO builds
Security No hardcoded secrets; env-based cloud credentials; input validation on CLI & API
Modularity Run storage, compute, or both — pick the surface you need
Offline-first Local filesystem backend and desktop UI work without the cloud

Architecture Overview

tecton/
├── crates/
│   ├── tecton-core      # Config, logging (tracing), errors (thiserror), shared types
│   ├── tecton-io        # OpenDAL storage engine + Axum S3-compatible API
│   └── tecton-compute   # DataFusion SQL engine over Parquet/CSV
├── apps/
│   ├── tecton-cli       # Headless server/CLI binary (`tecton`)
│   └── tecton-ui        # Tauri desktop app (React + Vite + Tailwind)
├── scripts/             # Dataset generators, icons, benchmarks
└── tecton.toml.example  # Sample configuration
Crate / App Role
tecton-core Shared utilities, TectonConfig, TectonError, system status types
tecton-io Multi-cloud object storage + lightweight HTTP API (/health, put/get/list/delete)
tecton-compute DataFusion SQL, CSV→Parquet auto-optimize, streaming results
tecton-cli clap CLI: start, status, query, ls, benchmark, version
tecton-ui Dashboard + Analyze (SQL, charts, Parquet folder datasets)

Prerequisites

  • Rust 1.80+ (rustup)
  • Node.js 20+ and npm
  • Python 3.10+ with numpy + pyarrow (only for dataset generators)
  • Tauri prerequisites for your OS — see Tauri getting started
    • Windows: WebView2 (usually preinstalled on Win10/11)
    • macOS: Xcode CLT
    • Linux: webkit2gtk and related packages

Optional: copy tecton.toml.example → tecton.toml to customize bind address, storage root, and dataset directory.

Installation & Build

# Clone

git clone https://github.com/rabbittrix/tecton.git

cd tecton


# Build all Rust workspace members (release)

cargo build --release


# CLI binary lands at:

#   target/release/tecton        (Linux/macOS)

#   target/release/tecton.exe    (Windows)


# Desktop UI dependencies

cd apps/tecton-ui

npm install


# Dev mode (Vite + Tauri)

npm run tauri:dev


# Production UI bundle

npm run tauri:build

Usage

CLI

# Start storage + compute (default)

tecton start --mode full


# Storage API only (S3-compatible HTTP on 127.0.0.1:9000)

tecton start --mode storage --port 9000


# Compute only

tecton start --mode compute


# Health / status snapshot (JSON)

tecton status


# List objects

tecton ls datasets/


# Run a read-only SQL query (table name is always `data`)

tecton query --file apps/tecton-ui/src-tauri/data/BANK_DATA.csv \

  --sql "SELECT gender, COUNT(*) AS n FROM data GROUP BY gender ORDER BY n DESC"


# Query a Parquet shard folder (billion-row stress tests)

tecton query --file apps/tecton-ui/src-tauri/data/bank_billion \

  --sql "SELECT bank_name, COUNT(*) AS n FROM data GROUP BY bank_name ORDER BY n DESC LIMIT 20"


# Benchmark cold (CSV→Parquet) vs warm (Parquet cache)

tecton benchmark --file apps/tecton-ui/src-tauri/data/BANK_DATA.csv

HTTP (when storage is running)

curl http://127.0.0.1:9000/health

curl -X PUT --data-binary @file.parquet http://127.0.0.1:9000/datasets/file.parquet

curl http://127.0.0.1:9000/?prefix=datasets/

Desktop UI

  • Dashboard — system status (Running/Stopped), storage usage, quick actions
  • Analyze — open a CSV/Parquet file or a Parquet shard folder, run SQL, view chart/table/JSON
  • Theme toggle — dark / light mode (defaults to dark)
  • Support / Donate — link to sponsorship options

UI result cues: execution time (ms), Auto-optimized to Parquet badge, truncation warning when streamed rows are capped, and a loading overlay during queries.

Test datasets

Synthetic datasets live under apps/tecton-ui/src-tauri/data/ (large generated files are gitignored; small *.sample.csv seeds are committed).

Path Domain Scale Format
BANK_DATA.sample.csv Banking seed CSV (committed)
BANK_DATA.csv Banking ~10M rows / ~961 MiB CSV
bank_billion/ Banking 1 000 000 001 rows / ~10.4 GiB Parquet shards
EDP_DATA.sample.csv Energy (EDP-style metering/billing) seed CSV (committed)
EDP_DATA.csv Energy ~10M rows / ~1.4 GiB CSV
edp_billion/ Energy up to 1B rows Parquet shards
TELECOM_EU_ME.sample.csv Telecom EU + Middle East seed CSV (committed)
TELECOM_EU_ME.csv Telecom ~10M rows / ~1.2 GiB CSV
telecom_eu_me_billion/ Telecom up to 1B rows Parquet shards

Prefer Parquet shard folders for billion-row tests. A single 1B-row CSV is typically 100 GiB+ and slow to query.

Generate datasets

# Bank — expand CSV (~10M)

python scripts/generate_bank_data.py --rows 10000001


# Bank — billion-row Parquet shards (resume-safe)

python scripts/generate_bank_billion.py --rows 1000000001 --shard-rows 2000000


# EDP + Telecom — medium CSVs (~10M each)

python scripts/generate_edp_telecom_billion.py --dataset both --rows 10000000 --format csv


# EDP + Telecom — billion-row Parquet shards

python scripts/generate_edp_telecom_billion.py --dataset both --rows 1000000000 --format parquet

Example SQL

-- Banking
SELECT gender, COUNT(*) AS count FROM data GROUP BY gender ORDER BY count DESC;

-- EDP
SELECT region, COUNT(*) AS n, ROUND(AVG(consumption_kwh), 2) AS avg_kwh
FROM data GROUP BY region ORDER BY n DESC;

-- Telecom EU / ME
SELECT country, operator, COUNT(*) AS subscribers, ROUND(AVG(arpu_eur), 2) AS avg_arpu
FROM data GROUP BY country, operator ORDER BY subscribers DESC LIMIT 50;

In the UI: Analyze → Open file for CSVs/Parquet files, or Open folder for bank_billion, edp_billion, or telecom_eu_me_billion.

Performance

Tecton auto-optimizes large CSVs so interactive analysis stays fast after the first run.

Auto-optimization

Behavior Detail
Threshold CSV files > 100 MiB
First query Scan CSV → write sibling .parquet → run SQL on Parquet
Later queries Reuse the cached .parquet (same basename)
SELECT * safety Bare SELECT * without LIMIT gets LIMIT 1000; stream hard-cap is 10 000 rows

Benchmark (tecton benchmark)

Measured on Windows (release build) with BANK_DATA.csv (~961 MiB, 10 000 001 rows, 7 columns):

SELECT gender, COUNT(*) AS count FROM data GROUP BY gender ORDER BY count DESC
Run Wall clock Engine time Peak RSS Optimization
Cold (CSV → Parquet + query) 2 282 ms 2 282 ms 179.7 MiB Converted CSV to Parquet
Warm (Parquet cache) 57 ms 56 ms 60.1 MiB Using existing Parquet cache
  • Speedup: ~40× cold → warm (97.5% less wall time)
  • Parquet cache size: ~94 MiB (~10× smaller than the CSV)
cargo run -p tecton-cli --release -- benchmark --file apps/tecton-ui/src-tauri/data/BANK_DATA.csv

Comparison with traditional tools

Same machine, same dataset and aggregation (wall clock includes load + GROUP BY gender):

Tool / path Wall time Notes
Tecton warm (Parquet) 57 ms Cached after first conversion
Tecton cold (CSV→Parquet) 2 282 ms Includes one-time conversion (~94 MiB Parquet)
pandas read_csv + groupby 4 276 ms Materializes the full CSV in memory
pandas read_parquet + groupby 1 003 ms Uses Tecton-produced Parquet cache 
python scripts/compare_bench.py

Numbers vary by CPU, disk cache, and OS buffering. Prefer tecton benchmark on your machine for capacity planning.

Configuration

Environment variables use the TECTON_ prefix (nested keys with __):

export TECTON_SERVER__PORT=9000

export TECTON_STORAGE__BACKEND=fs

export TECTON_LOG_LEVEL=debug

Cloud backends (s3, azblob, gcs) read credentials from the standard environment variables (AWS_*, Azure / GCP ADC) — never from source code.

Donation

If Tecton helps your business, consider supporting its development!

Donate via GitHub Sponsors / PayPal / Coffee

Your support keeps the project independent, secure, and moving fast.

Author & Contact
Author Roberto de Souza
Email rabbittrix@hotmail.com
Repository https://github.com/rabbittrix/tecton.git
License Apache-2.0