<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><title>Millwright · Data & EDA</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="stylesheet" href="site.css">
</head><body><header class="top">
<div class="wrap">
<div class="brand"><a href="../index.html"><span class="mark">⚙</span>millwright</a><span class="ver">docs</span></div>
<nav>
<a href="index.html">home</a>
<a href="data.html" class="active">data & EDA</a>
<a href="pipelines.html">pipelines</a>
<a href="insight.html">insight</a>
<a href="deploy.html">deploy</a>
<a href="python.html">python</a>
<a href="../index.html">design brief</a>
<a class="repo" href="https://github.com/mi7plus/millwright">GitHub ↗</a>
</nav>
</div>
</header>
<main>
<div class="wrap">
<div class="hero">
<div class="eyebrow">01 · data & EDA</div>
<h1>The boundary type,<br>and the typed layer in front of it.</h1>
<p class="lede"><code class="inl">Frame</code> is the numeric boundary the whole API speaks. <code class="inl">Table</code> (feature <code class="inl">eda</code>) is the polars-backed, dtype-aware world that ingests real CSV/Parquet and lowers into it.</p>
</div>
</div>
<!-- FRAME -->
<section id="frame">
<div class="wrap">
<div class="head col">
<div class="eyebrow">Frame & Dataset</div>
<h2>One contiguous <span class="mono">f64</span> buffer, plus a schema.</h2>
<p class="muted">Everything the <em>public</em> API speaks is a <code class="inl">Frame</code>: a contiguous, row-major <code class="inl">f64</code> buffer with named columns. It is what lets a linfa model and a smartcore <code class="inl">DenseMatrix</code> meet in one signature without your code ever naming their array versions — each backend converts <code class="inl">Frame</code> ⇄ its native type inside the adapter only. A <code class="inl">Dataset</code> pairs a frame with a target.</p>
</div>
<pre><span class="k">use</span> millwright::prelude::*;
<span class="k">let</span> x = <span class="f">Frame</span>::from_rows(
<span class="f">vec!</span>[<span class="f">vec!</span>[<span class="k">0.0</span>, <span class="k">0.1</span>], <span class="f">vec!</span>[<span class="k">0.4</span>, <span class="k">0.2</span>], <span class="f">vec!</span>[<span class="k">9.0</span>, <span class="k">9.1</span>], <span class="f">vec!</span>[<span class="k">9.4</span>, <span class="k">8.7</span>]],
<span class="f">vec!</span>[<span class="s">"a"</span>.into(), <span class="s">"b"</span>.into()],
)?;
<span class="k">assert_eq!</span>(x.shape(), (<span class="k">4</span>, <span class="k">2</span>)); <span class="c">// (rows, cols)</span>
<span class="k">let</span> train = <span class="f">Dataset</span>::new(x.clone(), <span class="f">vec!</span>[<span class="k">0.0</span>, <span class="k">0.0</span>, <span class="k">1.0</span>, <span class="k">1.0</span>])?;
<span class="k">let</span> _features = train.features(); <span class="c">// &Frame</span>
<span class="k">let</span> _target = train.target(); <span class="c">// &[f64]</span></pre>
<p class="tiny">The task — classification vs. regression — is inferred from the target: an all-integral target is class labels, anything else is regression. A pure-numeric CSV loads directly with <code class="inl">Frame::from_csv</code>; typed data uses <code class="inl">Table</code> below.</p>
</div>
</section>
<!-- INGEST -->
<section id="ingest">
<div class="wrap">
<div class="head col">
<div class="eyebrow">Ingest & explore</div>
<h2><span class="mono">Table</span> reads it, <span class="mono">Profile</span> reports it.</h2>
<p class="muted">Behind the <code class="inl">eda</code> feature, a polars-backed <code class="inl">Table</code> reads real CSV/Parquet — strings, categories, dates, booleans, nulls — and a <code class="inl">Profile</code> returns a <em>typed</em> analysis (not just an HTML blob) and drafts the preprocessing.</p>
</div>
<pre><span class="k">let</span> table = <span class="f">Table</span>::from_csv(<span class="s">"customers.csv"</span>)?; <span class="c">// or ::from_parquet(…)</span>
<span class="c">// a typed profile — overview, per-column stats, missingness, correlations, alerts</span>
<span class="k">let</span> profile = <span class="f">Profile</span>::of_with_target(&table, <span class="s">"churned"</span>)?;
<span class="f">println!</span>(<span class="s">"{}"</span>, profile.summary());
<span class="k">for</span> alert <span class="k">in</span> profile.alerts() {
<span class="f">println!</span>(<span class="s">"{alert}"</span>); <span class="c">// "[city] categorical (3 levels) → OneHotEncoder"</span>
}
profile.to_html(<span class="s">"eda_report.html"</span>)?; <span class="c">// a shareable report</span></pre>
<p class="muted">Because Millwright owns EDA <em>and</em> the pipeline, the profile drafts the starting preprocessing from its own findings — the loop scikit-learn can't close:</p>
<pre><span class="c">// lower the typed table to the numeric world</span>
<span class="k">let</span> train = table.into_dataset(<span class="s">"churned"</span>)?; <span class="c">// categoricals encoded, nulls → NaN</span>
<span class="c">// EDA drafts the pipeline; you just add the model</span>
<span class="k">let mut</span> pipe = profile.suggest_pipeline() <span class="c">// impute · encode · scale, from the alerts</span>
.estimator(<span class="s">"rf"</span>, <span class="f">RandomForest</span>::new());
pipe.fit(&train)?;</pre>
<p class="run">cargo run --example explore --features "eda smartcore-backend"</p>
</div>
</section>
<!-- DTYPES -->
<section id="dtypes">
<div class="wrap">
<div class="head col">
<div class="eyebrow">Dtype-aware</div>
<h2>Types flow through the pipeline.</h2>
<p class="muted">A <code class="inl">Frame</code> carries a per-column <code class="inl">Dtype</code> (defaulting to <code class="inl">Numeric</code>). <code class="inl">Table</code> marks the columns it knows are <code class="inl">Categorical</code> as it lowers — so preprocessing doesn't have to <em>guess</em>: scalers, <code class="inl">Winsorize</code>, and <code class="inl">PowerTransform</code> pass categorical columns through untouched, and <code class="inl">OneHotEncoder</code> encodes by dtype rather than a value heuristic.</p>
</div>
<pre><span class="c">// a genuinely-integer feature is NOT wrongly one-hot'd; the categorical one is</span>
<span class="k">let</span> f = <span class="f">Frame</span>::from_rows(rows, cols)?
.with_dtypes(<span class="f">vec!</span>[<span class="f">Dtype</span>::Categorical, <span class="f">Dtype</span>::Numeric])?;
<span class="k">let</span> out = <span class="f">OneHotEncoder</span>::infer().fit_transform(&f)?; <span class="c">// expands only column 0</span>
<span class="c">// or one-hot at the Table boundary, with real category names</span>
<span class="k">let</span> train = table.into_dataset_with(<span class="s">"churned"</span>, <span class="f">CategoryEncoding</span>::OneHot)?;</pre>
<p class="tiny">Nominal categories become <code class="inl">"{col}={value}"</code> indicator columns instead of ordinal codes — the correct representation for linear and tree models.</p>
</div>
</section>
<div class="wrap">
<div class="pager">
<a href="index.html"><span class="dir">← prev</span><b>Home</b></a>
<a class="next" href="pipelines.html"><span class="dir">next →</span><b>Pipelines & Models</b></a>
</div>
</div>
</main>
<footer>
<div class="wrap">
<span class="mono">⚙ millwright docs</span>
<span class="mono"><a href="../index.html">design brief</a> · <a href="https://crates.io/crates/millwright">crates.io</a> · <a href="https://pypi.org/project/millwright/">PyPI</a> · <a href="https://docs.rs/millwright">docs.rs</a> · <a href="https://github.com/mi7plus/millwright">GitHub</a></span>
</div>
</footer>
</body></html>