Skip to main content

Crate hessboost

Crate hessboost 

Source
Expand description

§hessboost

Fast, deterministic gradient boosting in Rust (with Python bindings). hessboost provides multi-core tree building with runtime-detected NEON and AVX2 SIMD, strict parameter validation, reproducible models on any thread count, and stable model storage. It supports modern extensions like conformal prediction, explainable boosting machines (EBMs), distributional modeling, and tree-based diffusion, alongside bidirectional XGBoost JSON/UBJSON model interchange.

§Quick start

Build a DMatrix, set TrainingParams with its builder, train with train (or Trainer for eval sets, early stopping, custom hooks, and continued training), then predict:

use hessboost::prelude::*;

// 6 rows × 2 features, row-major, plus a label per row.
let x = [0.0, 0.0,  1.0, 0.0,  0.0, 1.0,  1.0, 1.0,  0.5, 0.5,  0.2, 0.9];
let y = [0.0,       1.0,       1.0,       0.0,       0.5,       0.7];
let dtrain = DMatrix::from_dense(&x, 6, 2)?.with_labels(&y)?;

let params = TrainingParams::builder()
    .objective(Objective::SquaredError(RegLoss::default()))
    .tree_method(TreeMethod::Hist)
    .max_depth(3)
    .eta(0.1)
    .build()?;

let model = train(&params, &dtrain, 50)?;
let preds = model.predict(&dtrain, Iterations::Best)?;
assert_eq!((preds.n_rows(), preds.width()), (6, 1)); // `[row][output]`

model.save("model.bin", ModelFormat::Binary)?; // native format

prelude holds only this workflow’s items (including TreeMethod and Objective); everything else is imported from its module.

§Modules

§What’s here

  • Boosters: BoosterKind::GbTree, BoosterKind::Dart, BoosterKind::GbLinear, boosted random forests (num_parallel_tree), BoosterKind::Boulevard, and BoosterKind::Ebm.
  • Lifecycle: continued training and process_type=update refresh (Trainer::init_model), a per-round hook for progress, custom stopping, and cancellation (Trainer::on_round), slicing (BoostedModel::slice), iteration_range prediction as Rust ranges (every prediction method’s Iterations argument).
  • Tree methods: exact, hist, approx; depthwise/lossguide growth; uniform or gradient_based row sampling; column sampling with optional per-feature weights (DMatrix::with_feature_weights).
  • Objectives (Objective, each with its parameters): regression (squared, squared-log, pseudo-Huber, smoothed absolute, quantile/expectile lists), binary (logistic, logitraw, hinge) and multiclass, counts, LambdaMART and XE-NDCG ranking, survival (survival:cox, survival:aft on censored bounds), plus custom losses (Objective::Custom, e.g. a CustomLoss).
  • Multi-output: label matrices (DMatrix::with_label_matrix), one tree per output or vector-leaf trees (MultiStrategy::MultiOutputTree).
  • Metrics (EvalMetric, each with its own parameters): rmse, rmsle, mae, mape, mphe, logloss, error, auc, aucpr, mlogloss, merror, poisson/gamma/tweedie-nloglik, ndcg, map, pre, quantile, expectile, cox/aft-nloglik, interval-regression-accuracy, plus a custom hook (Trainer::custom_metric, reported after built-in metrics). XGBoost-compatible parameter dictionaries parse through TrainingParams::from_xgboost: @k ranking cutoffs and @rho on tweedie-nloglik, other suffixes refused.
  • Modeling: monotone and interaction constraints, native categorical splits, early stopping, feature importance, QuadratureTreeSHAP values and interactions (predict_contribs / predict_interactions).
  • I/O: libsvm/CSV loaders, native binary + JSON, XGBoost JSON and UBJSON import/export (XGBoost interchange), LightGBM 4.x text model import (ModelFormat::LightgbmText; see LightGBM import), all through one ModelFormat with byte-level detection; models compiled into the binary with EmbeddedModel.
  • Validation: cross-validation (cv), custom, forward-chaining (time-ordered, purged by a row gap), or purged forward (timestamped rows, purged by each label window, Fold::purged_forward) Folds with fold-mean early stopping (CrossValidation); whole-query folds of ranking data; ordered target statistics fitted inside each fold (CrossValidation::target_stats).
  • Advanced & experimental methods (opt-in):
    • split-conformal and conformalized-quantile intervals with finite-sample marginal coverage (conformal);
    • Boulevard boosting (Zhou & Hooker, JMLR 2022) and its dropout (BRAT-D) and parallel (BRAT-P) variants (Fang, Tan & Hooker, NeurIPS 2025) with CLT-based confidence intervals for f(x), prediction and reproduction intervals, an honest leaf refit, and exact or Nyström variance (BoosterKind::Boulevard, inference);
    • explainable boosting machines (GA²M: cyclic per-feature boosting, early-stopped outer bags, FAST pair terms, numerical and categorical terms; Lou et al., KDD 2012/2013, InterpretML) with per-term shape functions, and their Boulevard variant (Fang, Tan, Pipping & Hooker, AISTATS 2026) with confidence bands on every shape (BoosterKind::Ebm, ebm, EbmInference);
    • CatBoost-style ordered target statistics (data::target_stats);
    • LightGBM options extra_trees, path_smooth, linear_tree leaves (TrainingParams::extra_trees, path_smooth, linear_tree, LinearLeaves);
    • LightGBM class-balanced bagging for binary classification (BalancedBagging: pos_bagging_fraction, neg_bagging_fraction), in place of subsample;
    • LightGBM XE-NDCG ranking (Objective::RankXendcg; its keyed per-round draws differ from LightGBM’s random stream) and query-level bagging (QueryBagging, bagging_by_query);
    • CatBoost-style symmetric trees (GrowPolicy::Symmetric), routed by bit pattern in batch prediction;
    • Boosted Trees on a Diet reuse penalties (toad_penalty_feature, toad_penalty_threshold) and a bit-packed layout with bit-identical margins (model::compact);
    • LightGBM-style quantized gradients (QuantizedGrad, use_quantized_grad);
    • PerpetualBooster-style budget training: one budget instead of eta/depth/rounds (training::budget);
    • in-place row addition and deletion (incremental learning and machine unlearning) for trained hist models, exact or approximate (training::online);
    • distributional boosting (NGBoost / XGBoostLSS style): dist:normal, dist:lognormal, dist:gamma, dist:poisson, dist:negbinomial per-row distributions (predict_distribution, objective::distributional), scored by nll / crps;
    • CatBoost’s Stochastic Gradient Langevin Boosting and model shrinkage (langevin, model_shrink, posterior_sampling) with virtual ensembles: knowledge, data, and total uncertainty from one model’s exactly rebuilt truncations (predict_uncertainty, model::uncertainty);
    • nonparametric p(y | x) by tree-based conditional diffusion (Treeffuser) and flow matching (DiffGBM) for scalar or vector labels, sampled deterministically (diffusion);
    • ForestFlow / ForestDiffusion tabular generation and imputation (diffusion::forest);
    • native Metal on macOS 10.15+ (metal feature): bit-identical GPU prediction (to_gpu, ~3x faster at scale) and bit-identical GPU histograms (device = metal; exact integer sums, CPU fallback outside their exact domain). Documented only in macOS builds with the feature (cargo doc --features metal); elsewhere backend::metal is a stub.

examples/ has one program per topic (train_regression, binary_classification, multiclass, ranking, rank_xendcg, shap, model_io, custom_objective, constraints, conformal, boulevard_inference, ebm, compact_model, distributional, virtual_ensembles, tree_diffusion, forest_flow, budget, balanced_bagging, online_update, ordered_target_stats, pfn_boost, metal with --features metal on macOS). Run one with cargo run --release --example binary_classification.

§Compatibility notes

While hessboost is a standalone library, it offers extensive compatibility with XGBoost configurations and models: TrainingParams::from_xgboost reads XGBoost params dictionaries (keys, aliases, and value spellings), and unsupported settings are refused. Deterministic configurations reproduce XGBoost predictions within 1e-4 (quantile cuts bit for bit), and imported XGBoost models predict and explain identically. RNG-driven options (subsampling, forests, DART) match in quality only — the random streams differ.

§Not implemented

  • Distributed and external-memory training; GPU training outside macOS.
  • XGBoost options available at one setting only (so they are not TrainingParams fields; from_xgboost accepts exactly that setting): gblinear uses updater = coord_descent with feature_selector = cyclic; LambdaMART uses lambdarank_pair_method = topk (no lambdarank_unbiased or ndcg_exp_gain); DART has no sample_type or normalize_type (it samples uniformly and normalizes by tree); categorical splits use XGBoost’s defaults max_cat_to_onehot = 4 and max_cat_threshold = 64.
  • The metrics gamma-deviance, error@t (XGBoost’s classification threshold suffix), and the - variants of the ranking metrics (ndcg-, ndcg@k-, map-, map@k-); these names are refused.
  • XGBoost import and export of gblinear models.
  • scale_pos_weight = 0: XGBoost’s bound is >= 0; hessboost’s RegLoss::new needs a positive weight, so configurations and XGBoost files with 0 are refused.

Modules§

backend
Optional compute backends.
config
Training configuration types.
conformal
Distribution-free prediction intervals by split conformal prediction.
data
Datasets: the DMatrix container, the MetaInfo view objectives and metrics read, the libsvm/CSV loaders, and the opt-in target_stats encoder for categorical columns.
diffusion
Nonparametric probabilistic regression: conditional diffusion and flow matching with boosted trees as the score or velocity model (opt-in).
ebm
Explainable boosting machines (EBMs): cyclic GA²M boosting, per-term shape functions, and (with Boulevard averaging) confidence bands on them. Opt-in: train with BoosterKind::Ebm, read the shape functions with shape_functions, and, for an Ebm::boulevard model, their bands with EbmInference.
error
Error types for hessboost.
inference
Statistical inference for Boulevard boosting: confidence intervals for the regression function f(x), prediction intervals for new labels, and reproduction intervals, with asymptotic (central-limit) guarantees under the assumptions below and validated only in the regimes listed in Validation. Opt-in: train with BoosterKind::Boulevard, then fit a BoulevardInference on the training rows. A Boulevard EBM (crate::ebm, ebm_boulevard) gets bands on its shape functions from EbmInference, built on the same solvers.
metric
Evaluation metrics used for reporting and early stopping.
model
Trained models: BoostedModel, its prediction, explanation, and persistence, and the size-optimized compact form.
objective
Learning objectives: gradients, Hessians, prediction transforms, and base score estimation.
prelude
The train-and-predict workflow in one import: use hessboost::prelude::*;.
training
Training: the boosting loop (train, Trainer), cross-validation (cv, or CrossValidation over caller-supplied or time-ordered Folds), and opt-in budget training.
tree
The trees of a trained model (BoostedModel::trees), for inspection: RegTree, its Nodes, and the LinearLeaves of linear_tree models.