u-insight

A statistical analysis and data profiling engine in Rust with C FFI bindings.

What's New in 0.9.0

Kendall tau-b correlation added to CorrelationMethod (Pearson / Spearman / Kendall)
Outlier fences exposed on OutlierResult — lower_fence, upper_fence, center, spread
detect_outliers_slice(&[f64], method) helper for raw-slice input
vif_analysis() and condition_number() standalone multicollinearity diagnostics
WASM: correlation_matrix accepts optional _method field ("pearson" | "spearman" | "kendall"); new detect_univariate_outliers, vif_diagnostic, condition_number_diagnostic
C#: CorrelationMethodKind enum + Correlate(data, method) parameter
BREAKING — Rust: Non-finite numeric inputs now return InsightError::DegenerateData (was NonNumericColumn); audit covered 11 call sites
BREAKING — Rust: OutlierResult gained 4 new fields (exhaustive pattern matches must be updated)
BREAKING — FFI: insight_correlation signature gained a method: u32 parameter (use INSIGHT_CORR_PEARSON = 0 to keep prior behaviour)
BREAKING — C#: Correlate(...) now takes an optional CorrelationMethodKind parameter (default Pearson keeps existing call-sites compiling)

Overview

u-insight transforms raw tabular data into actionable statistical insights. It operates in two distinct layers with opposite assumptions about input data quality:

CSV (raw)
  │
  ├─→ Profiling ─→ "What is the state of this data?"
  │     Tolerates dirty data (missing values, type mismatches expected)
  │
  │   (external preprocessing)
  │
  └─→ Analysis  ─→ "What can we learn from this data?"
        Requires clean numeric data (no NaN, no missing)

Built on u-analytics (statistical algorithms), u-numflow (math primitives).

Modules

Data Layer

Module	Description
`dataframe`	Column-major tabular data model (DataFrame, Column, DataType)
`csv_parser`	CSV parsing with automatic type inference
`error`	Error types (InsightError)

Profiling Layer (dirty data tolerated)

Module	Description
`profiling`	Column-level and dataset-level data profiling — descriptive stats, missing analysis, outlier flagging (IQR/Z-score/Modified Z-score), diagnostic flags

Analysis Layer (clean data required)

Module	Description
`analysis`	Correlation (Pearson/Spearman), regression (simple/multiple OLS), Cramer's V contingency analysis
`clustering`	K-Means++ (auto-K, Gap Statistic), Mini-Batch K-Means, DBSCAN, Hierarchical Agglomerative (Single/Complete/Average/Ward), HDBSCAN
`distribution`	ECDF, histogram bins (Sturges/Scott/FD), QQ-plot, normality tests (KS, Jarque-Bera, Shapiro-Wilk, Anderson-Darling), Grubbs test, distribution fitting
`pca`	Principal Component Analysis with auto-scaling option
`isolation_forest`	Isolation Forest anomaly detection (Liu et al. 2008)
`lof`	Local Outlier Factor (LOF) density-based anomaly detection
`mahalanobis`	Mahalanobis distance multivariate outlier detection
`feature_importance`	Variance threshold, correlation filter, VIF, condition number, composite importance, ANOVA F-test selection, Mutual Information, Permutation Importance

FFI Layer

Module	Description
`ffi`	C FFI bindings — 32 functions, 20 `#[repr(C)]` structs, auto-generated C header via cbindgen

Quick Start

use u_insight::csv_parser::CsvParser;
use u_insight::profiling::profile_dataframe;

// 1. Parse CSV
let csv = "name,value,active\nAlice,1.5,true\nBob,2.3,false\nCharlie,3.1,true\n";
let df = CsvParser::new().parse_str(csv).unwrap();

// 2. Profile
let profiles = profile_dataframe(&df);

Clustering

use u_insight::clustering::{kmeans, dbscan, KMeansConfig, DbscanConfig};

let data = vec![
    vec![0.0, 0.0], vec![0.5, 0.5],
    vec![10.0, 10.0], vec![10.5, 10.5],
];

// K-Means
let km = kmeans(&data, &KMeansConfig::new(2)).unwrap();
assert_eq!(km.k, 2);

// DBSCAN
let db = dbscan(&data, &DbscanConfig::new(1.5, 2)).unwrap();
assert_eq!(db.n_clusters, 2);

Distribution Analysis

use u_insight::distribution::{distribution_analysis, DistributionConfig};

let data: Vec<f64> = (0..50).map(|i| (i as f64 - 25.0) * 0.2).collect();
let result = distribution_analysis(&data, &DistributionConfig::default()).unwrap();
println!("Normal: {}", result.normality.is_normal);

C FFI

u-insight builds as cdylib + staticlib for cross-language interop. A C header (u_insight.h) is auto-generated by cbindgen at build time.

Profiling

Function	Description
`insight_profile_csv`	Profile a CSV string → opaque context
`insight_profile_free`	Free profile context
`insight_profile_row_count`	Row count from profile
`insight_profile_col_count`	Column count from profile
`insight_profile_column`	Get column summary

Clustering

Function	Description
`insight_kmeans`	K-Means++ clustering
`insight_mini_batch_kmeans`	Mini-Batch K-Means clustering
`insight_dbscan`	DBSCAN density-based clustering
`insight_hierarchical`	Hierarchical Agglomerative clustering (4 linkages)
`insight_hdbscan`	HDBSCAN clustering with membership probabilities
`insight_gap_statistic`	Gap statistic for optimal K selection

Dimensionality Reduction

Function	Description
`insight_pca`	Principal Component Analysis

Anomaly Detection

Function	Description
`insight_isolation_forest`	Isolation Forest anomaly detection
`insight_lof`	Local Outlier Factor detection
`insight_mahalanobis`	Mahalanobis distance outlier detection

Statistical Analysis

Function	Description
`insight_correlation`	Pearson correlation matrix
`insight_regression`	Simple linear regression
`insight_cramers_v`	Cramer's V contingency analysis

Distribution

Function	Description
`insight_distribution`	Normality testing (KS, JB, SW, AD)

Feature Importance

Function	Description
`insight_feature_importance`	Composite feature importance scores
`insight_anova_select`	ANOVA F-test feature selection
`insight_mutual_info`	Mutual information feature ranking
`insight_permutation_importance`	Permutation importance for regression

Memory Management

Function	Description
`insight_free_labels`	Free u32 label arrays
`insight_free_i32_array`	Free i32 arrays
`insight_free_f64_array`	Free f64 arrays
`insight_free_anova_features`	Free ANOVA feature arrays
`insight_free_mi_features`	Free MI feature arrays
`insight_free_perm_features`	Free permutation importance arrays

Error & Version

Function	Description
`insight_last_error`	Last error message (thread-local)
`insight_clear_error`	Clear error state
`insight_version`	Library version string

All FFI functions use catch_unwind to prevent panics from crossing the FFI boundary.

C# Binding (UInsight)

Install via NuGet — native libraries are bundled automatically:

dotnet add package UInsight

using UInsight;

using var client = new InsightClient();
Console.WriteLine(client.GetVersion());

var data = new double[,] { {0,0}, {1,1}, {10,10}, {11,11} };
var result = client.KMeans(data, k: 2);
Console.WriteLine($"K={result.K}, WCSS={result.Wcss:F2}");

The binding is in bindings/csharp/UInsight/ with:

Interop/NativeLibrary.cs — [LibraryImport] declarations for all 32 FFI functions
Interop/NativeStructs.cs — [StructLayout] mappings for all 20 C structs
InsightClient.cs — High-level managed API (automatic memory management)
InsightException.cs — Error code to exception conversion

Test Status

357 lib tests + 49 doc-tests = 406 total
0 clippy warnings
Build: lib + cdylib + staticlib
C header: auto-generated via cbindgen (20 structs, 32 functions)

Scope & Non-Goals

In Scope:

Data profiling (dirty data → quality report + diagnostic flags)
Statistical analysis (clean data → patterns + relationships)
Correlation, regression, clustering, PCA, anomaly detection
Feature importance and selection (ANOVA, MI, Permutation)
Distribution analysis and normality testing
C FFI for cross-language use
C# binding (UInsight NuGet package)

Out of Scope:

Visualization / charting
Data cleaning / transformation / imputation
ML model training / deployment
Deep learning

Requirements

Rust 1.75+
Dependencies: u-analytics, u-numflow

WebAssembly / npm

Available as an npm package via wasm-pack.

npm install @iyulab/u-insight

Quick Start

import init, { describe, kmeans } from '@iyulab/u-insight';

await init();
const stats = describe({ col1: [1, 2, 3], col2: [4, 5, 6] });

Functions

`describe(data) -> [ColumnResult]`

Descriptive statistics per column. Input: column-major { "col1": [1,2,3] }.

Output: Array of { name, data_type, numeric: { count, min, max, mean, median, std_dev, variance, skewness, kurtosis, q1, q3, iqr, p5, p95, ... } }.

`correlation_matrix(data) -> CorrelationResult`

Pearson correlation matrix. Input: column-major { "col1": [1,2,3], "col2": [4,5,6] }.

Output:

{ "names": ["col1","col2"], "matrix": [1,0.99,0.99,1], "n": 2, "high_pairs": [{ "col_a": "col1", "col_b": "col2", "r": 0.99, "p_value": 0.01 }] }

`kmeans(data, k) -> KMeansResult`

K-Means++ clustering on row-major data [[x,y,...], ...].

Output:

{ "k": 3, "labels": [0,0,1,1,2,2], "centroids": [[...]], "wcss": 5.2, "iterations": 12, "cluster_sizes": [2,2,2] }

`pca(data, n_components) -> PcaResult`

Principal Component Analysis on row-major data.

Output:

{ "n_components": 2, "n_features": 4, "eigenvalues": [3.1,0.9], "explained_variance_ratio": [0.77,0.23], "cumulative_variance_ratio": [0.77,1.0], "loadings": [[...]], "scores": [[...]], "means": [...], "stds": [...] }

`dbscan(data, config) -> DbscanResult`

DBSCAN density-based clustering. config: { "epsilon": 1.5, "min_samples": 3 }.

Output:

{ "labels": [0,0,null,1,1], "n_clusters": 2, "noise_count": 1, "cluster_sizes": [2,2], "core_points": [true,true,false,true,true] }

`hierarchical(data, config) -> HierarchicalResult`

Hierarchical agglomerative clustering. config: { "linkage": "ward", "n_clusters": 3 } or { "linkage": "single", "distance_threshold": 5.0 }.

Output:

{ "merges": [{ "cluster_a": 0, "cluster_b": 1, "distance": 1.2, "size": 2 }], "labels": [0,0,1,1,2], "n_clusters": 3 }

`isolation_forest(data, config) -> IsolationForestResult`

Isolation Forest anomaly detection. config: { "n_estimators": 100, "contamination": 0.1, "seed": 42 }.

Output:

{ "scores": [0.45, 0.82], "anomalies": [false, true], "threshold": 0.65, "anomaly_count": 1, "anomaly_fraction": 0.5 }

`lof(data, config) -> LofResult`

Local Outlier Factor anomaly detection. config: { "k": 20, "threshold": 1.5 }.

Output:

{ "scores": [1.0, 2.3], "anomalies": [false, true], "threshold": 1.5, "anomaly_count": 1, "anomaly_fraction": 0.5 }

`distribution_analysis(data, config) -> DistributionResult`

Distribution analysis on a 1-D array. config: { "bin_method": "freedman_diaconis", "significance_level": 0.05, "compute_ecdf": true, "compute_histogram": true, "compute_qq_plot": true, "fit_distributions": false }.

Output:

{ "n": 100, "ecdf": { "values": [...], "probabilities": [...] }, "histogram": { "n_bins": 10, "bin_width": 0.5, "edges": [...], "counts": [...] }, "qq_plot": { "theoretical": [...], "sample": [...] }, "normality": { "shapiro_wilk": { "statistic": 0.98, "p_value": 0.45, "rejected": false }, "is_normal": true }, "fits": [] }

`regression(data) -> RegressionResult`

OLS regression analysis.

Input:

{ "predictors": { "x1": [1,2,3,4,5] }, "target": [2.1, 3.9, 6.1, 7.9, 10.1], "target_name": "y" }

Output:

{ "target_name": "y", "predictor_names": ["x1"], "r_squared": 0.99, "adj_r_squared": 0.99, "coefficients": [0.1, 2.0], "p_values": [0.9, 0.0001], "vif": [1.0], "f_p_value": 0.0001 }

`feature_importance(data) -> FeatureImportanceResult`

Feature importance via permutation, ANOVA, or mutual information.

Input:

{ "features": { "f1": [1,2,3], "f2": [5,4,3] }, "target": [0,0,1], "method": "permutation", "n_repeats": 5, "seed": 42 }

Output:

{ "method": "permutation", "features": [{ "name": "f1", "index": 0, "score": 0.8, "std_dev": 0.1 }], "baseline_score": 0.5 }

u-analytics -- Statistical analytics
u-numflow -- Mathematical primitives

License

MIT License

u-insight 0.9.0

u-insight

What's New in 0.9.0

Overview

Modules

Data Layer

Profiling Layer (dirty data tolerated)

Analysis Layer (clean data required)

FFI Layer

Quick Start

Clustering

Distribution Analysis

C FFI

Profiling

Clustering

Dimensionality Reduction

Anomaly Detection

Statistical Analysis

Distribution

Feature Importance

Memory Management

Error & Version

C# Binding (UInsight)

Test Status

Scope & Non-Goals

Requirements

WebAssembly / npm

Quick Start

Functions

describe(data) -> [ColumnResult]

correlation_matrix(data) -> CorrelationResult

kmeans(data, k) -> KMeansResult

pca(data, n_components) -> PcaResult

dbscan(data, config) -> DbscanResult

hierarchical(data, config) -> HierarchicalResult

isolation_forest(data, config) -> IsolationForestResult

lof(data, config) -> LofResult

distribution_analysis(data, config) -> DistributionResult

regression(data) -> RegressionResult

feature_importance(data) -> FeatureImportanceResult

Related

License

`describe(data) -> [ColumnResult]`

`correlation_matrix(data) -> CorrelationResult`

`kmeans(data, k) -> KMeansResult`

`pca(data, n_components) -> PcaResult`

`dbscan(data, config) -> DbscanResult`

`hierarchical(data, config) -> HierarchicalResult`

`isolation_forest(data, config) -> IsolationForestResult`

`lof(data, config) -> LofResult`

`distribution_analysis(data, config) -> DistributionResult`

`regression(data) -> RegressionResult`

`feature_importance(data) -> FeatureImportanceResult`