Expand description
A fast, extensible probabilistic cross-categorization engine.
Lace is a probabilistic cross-categorization engine written in rust with an optional interface to python. Unlike traditional machine learning methods, which learn some function mapping inputs to outputs, Lace learns a joint probability distribution over your dataset, which enables users to…
- predict or compute likelihoods of any number of features conditioned on any number of other features
- identify, quantify, and attribute uncertainty from variance in the data, epistemic uncertainty in the model, and missing features
- determine which variables are predictive of which others
- determine which records/rows are similar to which others on the whole or given a specific context
- simulate and manipulate synthetic data
- work natively with missing data and make inferences about missingness (missing not-at-random)
- work with continuous and categorical data natively, without transformation
- identify anomalies, errors, and inconsistencies within the data
- edit, backfill, and append data without retraining
and more, all in one place, without any explicit model building.
§Design
Lace learns a probabilistic model of tabular data using cross-categorization. The general steps to operation are
- Create a
prelude::Codebook
which describes your data. One can be autogenerated but it is best to check it before use. - Create an
prelude::Engine
with your data and codebook. - Train the
prelude::Engine
and monitor the model likelihood for convergence. - Ask questions via the
prelude::OracleT
implementation ofprelude::Engine
to explore your data.
§Example
(For a complete tutorial, see the Lace Book)
The following example uses the pre-trained animals
example dataset.
Each row represents an animal and each column represents a feature of that animal.
The feature is present if the cell value is 1 and is absent if the value is 0.
First, we create an oracle and import some enum
s that allow us to call
out some of the row and column indices in plain English.
use lace::prelude::*;
use lace::examples::Example;
let oracle = Example::Animals.oracle().unwrap();
Let’s ask about the statistical dependence between whether something swims and is fast or has flippers. We expect that something swimming is more indicative of whether it swims than whether something is fast, therefore we expect the dependence between swims and flippers to be higher.
let depprob_fast = oracle.depprob(
"swims",
"fast",
).unwrap();
let depprob_flippers = oracle.depprob(
"swims",
"flippers",
).unwrap();
assert!(depprob_flippers > depprob_fast);
We have the same expectation of mutual information. Mutual information requires more input from the user. We need to know what type of mutual information, and how many samples to take if we need to estimate the mutual information.
let mut rng = rand::thread_rng();
let mi_fast = oracle.mi(
"swims",
"fast",
1000,
MiType::Iqr,
).unwrap();
let mi_flippers = oracle.mi(
"swims",
"flippers",
1000,
MiType::Iqr,
).unwrap();
assert!(mi_flippers > mi_fast);
We can likewise ask about the similarity between rows – in this case, animals.
let wrt: Option<&[usize]> = None;
let rowsim_wolf = oracle.rowsim(
"wolf",
"chihuahua",
wrt,
RowSimilarityVariant::ViewWeighted,
).unwrap();
let rowsim_rat = oracle.rowsim(
"rat",
"chihuahua",
wrt,
RowSimilarityVariant::ViewWeighted,
).unwrap();
assert!(rowsim_rat > rowsim_wolf);
And we can add context to similarity.
let context = vec!["swims"];
let rowsim_otter = oracle.rowsim(
"beaver",
"otter",
Some(&context),
RowSimilarityVariant::ViewWeighted,
).unwrap();
let rowsim_dolphin = oracle.rowsim(
"beaver",
"dolphin",
Some(&context),
RowSimilarityVariant::ViewWeighted,
).unwrap();
§Feature flags
formats
: createEngine
s andCodebook
s from IPC, CSV, JSON, and Parquet data filesbencher
: Build benchmarking utilitiesctrc_handler
: enables and update handler than captures Ctrl+C
Re-exports§
pub use config::EngineUpdateConfig;
Modules§
- Data loaders and utilities
- Default values
- Misc, generally useful helper functions
- Function optimization utilities
- Common import for general use.
Structs§
- An oracle without data for sensitive data applications
- The engine runs states in parallel
- Builds
Engine
s - Describes table-extending actions taken when inserting data
- Holds the components required to compute mutual information
- Oracle answers questions
- A list of data for insertion into a certain row
- Stores some diagnostic info in the
State
at every iteration - A datum for insertion into a certain column
- Defines how/where data may be inserted, which day may and may not be overwritten, and whether data may extend the domain
Enums§
- Defines the behavior of the data table when new rows are appended
- The variant on conditional entropy to compute
- Represents the types of data lace can work with
- Feature type
- Describes a the conditions (or not) on a conditional distribution
- Defines insert data behavior – where data may be inserted.
- Mutual Information Type
- Holds a
String
name or ausize
index - Defines which data may be overwritten
- The variant of row similarity to compute
- MCMC transitions in the
State
- Describes the support extension action taken
Traits§
- Trait defining items that can converted into a usize column index
- Returns and summarizes data
- Returns references to crosscat states
- Trait defining an item that can be converted into a row index