📊 avila-arrow - Native Columnar Format

Native columnar data format optimized for AvilaDB and Brazilian scientific computing.

🚀 Features

• 🔬 Scientific Types

  • Quaternions (4D rotations for physics)
  • Tensor4D (spacetime tensors)
  • Complex numbers (FFT, quantum mechanics)
  • Spinors (particle physics)

• ⚡ High Performance

  • Zero-copy data access
  • SIMD-optimized operations
  • Columnar storage (cache-friendly)
  • < 1ms batch creation for 1M rows

• 🇧🇷 Brazilian Research Optimized

  • LIGO gravitational wave data
  • LISA mission support
  • Astrophysics time-series
  • Particle physics experiments

• 🔧 AvilaDB Native

  • Direct integration with AvilaDB
  • Query pushdown optimization
  • Streaming ingestion
  • Incremental updates

📦 Installation

[dependencies]
avila-arrow = "0.1"

🎯 Quick Start

Basic Schema and RecordBatch

use avila_arrow::{Schema, Field, DataType, RecordBatch};
use avila_arrow::array::{Float64Array, Int64Array};

// Create schema
let schema = Schema::new(vec![
    Field::new("id", DataType::Int64),
    Field::new("value", DataType::Float64),
]);

// Create arrays
let ids = Int64Array::from(vec![1, 2, 3, 4, 5]);
let values = Float64Array::from(vec![1.1, 2.2, 3.3, 4.4, 5.5]);

// Create batch
let batch = RecordBatch::try_new(
    schema,
    vec![Box::new(ids), Box::new(values)],
)?;

println!("Batch has {} rows", batch.num_rows());

Scientific Types - Quaternions

use avila_arrow::{Schema, Field, DataType, RecordBatch};
use avila_arrow::scientific::Quaternion;

// Create schema with quaternion column
let schema = Schema::new(vec![
    Field::new("rotation", DataType::Quaternion),
]);

// Create quaternions (w, x, y, z)
let rotations = vec![
    Quaternion::new(1.0, 0.0, 0.0, 0.0),  // Identity
    Quaternion::new(0.707, 0.707, 0.0, 0.0),  // 90° around X
    Quaternion::new(0.707, 0.0, 0.707, 0.0),  // 90° around Y
];

// Create batch
let batch = RecordBatch::from_quaternions("rotation", rotations)?;

Scientific Types - Tensor4D

use avila_arrow::scientific::Tensor4D;

// Spacetime tensor for General Relativity
let metric = Tensor4D::schwarzschild_metric(
    mass: 1.0,  // Solar mass
    r: 10.0,    // Distance
);

// Access components
let g00 = metric.get(0, 0);  // Time-time component
let g11 = metric.get(1, 1);  // Radial component

Complex Numbers for FFT

use avila_arrow::{Schema, Field, DataType};
use avila_arrow::scientific::Complex64;

// Schema for FFT results
let schema = Schema::new(vec![
    Field::new("frequency", DataType::Float64),
    Field::new("amplitude", DataType::Complex64),
]);

// Create complex array
let amplitudes = vec![
    Complex64::new(1.0, 0.0),
    Complex64::new(0.5, 0.866),  // 60° phase
    Complex64::new(0.0, 1.0),    // 90° phase
];

🔬 Scientific Use Cases

LIGO Gravitational Wave Analysis

use avila_arrow::{Schema, Field, DataType, RecordBatch};

// Schema for gravitational wave strain data
let schema = Schema::new(vec![
    Field::new("gps_time", DataType::Float64),
    Field::new("strain_h1", DataType::Complex64),  // Hanford detector
    Field::new("strain_l1", DataType::Complex64),  // Livingston detector
    Field::new("snr", DataType::Float64),
]);

// High-performance columnar storage for 10GB+ datasets

Particle Physics - Spinor Fields

use avila_arrow::scientific::Spinor;

// Dirac spinor for fermions
let electron_state = Spinor::new(
    Complex64::new(1.0, 0.0),  // Spin up
    Complex64::new(0.0, 0.0),  // Spin down
);

// Efficient columnar storage for millions of particles

Astrophysics - 4D Spacetime

use avila_arrow::scientific::Tensor4D;

// Store metric tensors for black hole simulation
let tensors = vec![
    Tensor4D::kerr_metric(mass, spin, r, theta),
    Tensor4D::schwarzschild_metric(mass, r),
];

📊 Performance

Benchmarks on AWS c6i.xlarge (São Paulo region):

🧪 Testing

# Run all tests
cargo test -p avila-arrow

# Run benchmarks
cargo bench -p avila-arrow

# Test scientific types
cargo test -p avila-arrow --features scientific

🗺️ Roadmap

• Core schema and field types
• Basic arrays (Int, Float, String)
• RecordBatch implementation
• Quaternion type
• Tensor4D type
• Complex64 type
• Spinor type
• Arrow IPC compatibility
• Parquet file format
• AvilaDB integration
• SIMD optimizations

🎯 Design Principles

1. Zero-Copy First: Minimize data movement
2. SIMD-Ready: Vectorized operations for scientific types
3. Type-Safe: Compile-time guarantees for data integrity
4. Brazilian-Optimized: Built for LIGO, LISA, and local research

🤝 Contributing

We welcome contributions! See CONTRIBUTING.md for guidelines.

📄 License

Licensed under either of:

• Apache License, Version 2.0 (LICENSE-APACHE)
• MIT license (LICENSE-MIT)

at your option.

Built with ❤️ for the AVL Platform and Brazilian research infrastructure 🇧🇷