Skip to main content

Crate hessboost

Crate hessboost 

Source
Expand description

§hessboost

Fast, deterministic gradient boosting in Rust (with Python bindings). hessboost provides multi-core tree building with runtime-detected NEON and AVX2 SIMD, strict parameter validation, reproducible models on any thread count, and stable model storage. It supports modern extensions like conformal prediction, explainable boosting machines (EBMs), distributional modeling, and tree-based diffusion, alongside bidirectional XGBoost JSON/UBJSON model interchange.

§Quick start

Build a DMatrix, set TrainingParams with its builder, train with train (or Trainer for eval sets, early stopping, custom hooks, and continued training), then predict:

use hessboost::prelude::*;

// 6 rows × 2 features, row-major, plus a label per row.
let x = [0.0, 0.0,  1.0, 0.0,  0.0, 1.0,  1.0, 1.0,  0.5, 0.5,  0.2, 0.9];
let y = [0.0,       1.0,       1.0,       0.0,       0.5,       0.7];
let dtrain = DMatrix::from_dense(&x, 6, 2)?.with_labels(&y)?;

let params = TrainingParams::builder()
    .objective(Objective::SquaredError(RegLoss::default()))
    .tree_method(TreeMethod::Hist)
    .max_depth(3)
    .eta(0.1)
    .build()?;

let model = train(&params, &dtrain, 50)?;
let preds = model.predict(&dtrain, Iterations::Best)?;
assert_eq!((preds.n_rows(), preds.width()), (6, 1)); // `[row][output]`

model.save("model.bin", ModelFormat::Binary)?; // native format

prelude holds only this workflow’s items (including TreeMethod and Objective); everything else is imported from its module.

§Modules

§What’s here

  • Boosters: BoosterKind::GbTree, BoosterKind::Dart, BoosterKind::GbLinear, boosted random forests (num_parallel_tree), BoosterKind::Boulevard, and BoosterKind::Ebm.
  • Lifecycle: continued training and process_type=update refresh (Trainer::init_model), a per-round hook for progress, custom stopping, and cancellation (Trainer::on_round), slicing (BoostedModel::slice), iteration_range prediction as Rust ranges (every prediction method’s Iterations argument).
  • Prediction: of a DMatrix, of borrowed dense rows without copying them (predict_rows), and of one row without allocating (predict_row, predict_row_into), bit for bit alike in every batch: each value (each row for softmax and the sorted quantiles) is transformed on its own, with XGBoost’s scalar expf, sigmoid, and softmax (transform_margins_into). What-if prediction of rows that differ from one base row in a few features, in one walk of each tree and bit for bit their rows’ predictions (RowVariants, predict_variants_into), and sound margin bounds over a box of feature values (margin_bounds_into). Read-only model metadata: tree weights, the outputs each tree feeds, nodes, depths, and category sets, gblinear weights, and the model-shrinkage record (tree_weights, tree_outputs, RegTree, linear, shrinkage).
  • Tree methods: exact, hist, approx; depthwise/lossguide growth; uniform or gradient_based row sampling; column sampling with optional per-feature weights (DMatrix::with_feature_weights).
  • Data sharing: a matrix’s copies with other metadata share its values, and those with its sample weights and feature types the hist index binned from them, built once for every training (concurrent ones included) and kept while in use or the latest asked for, until the values are freed; row selections are views of their source’s values (DMatrix::select_rows), binned by their own rows or by their source’s cuts (ViewCuts). A dense matrix can read a caller’s buffer in place (DMatrix::from_dense_shared; Python’s copy=False), and a BinnedMatrix keeps only a matrix’s bins and metadata (XGBoost’s QuantileDMatrix), training as the matrix does with hist growth and refusing what would read the values.
  • Objectives (Objective, each with its parameters): regression (squared, squared-log, pseudo-Huber, smoothed absolute, quantile/expectile lists), exceedance probabilities over a threshold ladder (reg:exceedance: P(y ≥ k_j) per threshold, never increasing in j), binary (logistic, logitraw, hinge) and multiclass, counts, LambdaMART and XE-NDCG ranking, survival (survival:cox, survival:aft on censored bounds), plus custom losses (Objective::Custom, e.g. a CustomLoss). The quantile and absolute losses can set every leaf to its rows’ exact residual quantile instead of the surrogate’s Newton step (LeafEstimation::Exact).
  • Multi-output: label matrices (DMatrix::with_label_matrix), one tree per output or vector-leaf trees (MultiStrategy::MultiOutputTree).
  • Metrics (EvalMetric, each with its own parameters): rmse, rmsle, mae, mape, mphe, logloss, error, auc, aucpr, mlogloss, merror, poisson/gamma/tweedie-nloglik, ndcg, map, pre, quantile, expectile, exceedance-logloss, cox/aft-nloglik, interval-regression-accuracy, plus a custom hook (Trainer::custom_metric, reported after built-in metrics). XGBoost-compatible parameter dictionaries parse through TrainingParams::from_xgboost: @k ranking cutoffs and @rho on tweedie-nloglik, other suffixes refused.
  • Modeling: monotone and interaction constraints, native categorical splits, early stopping, feature importance, QuadratureTreeSHAP values and interactions (predict_contribs / predict_interactions), one model predicting several models’ mean margin (bagging, BoostedModel::average).
  • I/O: libsvm/CSV loaders, native binary + JSON, XGBoost JSON and UBJSON import/export (XGBoost interchange), LightGBM 4.x text model import (ModelFormat::LightgbmText; see LightGBM import), all through one ModelFormat with byte-level detection; models compiled into the binary with EmbeddedModel.
  • Validation: cross-validation (cv), custom, forward-chaining (time-ordered, purged by a row gap), or purged forward (timestamped rows, purged by each label window, in test blocks of the last decision times or starting at given times such as calendar periods, over every row or a row subset such as the labeled rows: Fold::purged_forward, Fold::purged_forward_at, and their _rows variants) Folds with fold-mean or leakage-safe per-fold early stopping on a purged tail of each fold’s training rows, the last share of their time span or row count (CrossValidation, CvStopping, PurgedTail, TailCut); each fold’s round count and every row’s out-of-fold margins (CvReport); whole-query folds of ranking data; ordered target statistics fitted inside each fold (CrossValidation::target_stats); continuing a model in every fold; refitting on every row at the chosen round count (CrossValidation::refit).
  • Advanced & experimental methods (opt-in):
    • split-conformal and conformalized-quantile intervals with finite-sample marginal coverage (conformal);
    • Boulevard boosting (Zhou & Hooker, JMLR 2022) and its dropout (BRAT-D) and parallel (BRAT-P) variants (Fang, Tan & Hooker, NeurIPS 2025) with CLT-based confidence intervals for f(x), prediction and reproduction intervals, an honest leaf refit, and exact or Nyström variance (BoosterKind::Boulevard, inference);
    • explainable boosting machines (GA²M: cyclic per-feature boosting, early-stopped outer bags, FAST pair terms, numerical and categorical terms; Lou et al., KDD 2012/2013, InterpretML) with per-term shape functions, and their Boulevard variant (Fang, Tan, Pipping & Hooker, AISTATS 2026) with confidence bands on every shape (BoosterKind::Ebm, ebm, EbmInference);
    • CatBoost-style ordered target statistics (data::target_stats);
    • split-group child constraints: a split only where both children hold rows of at least min_child_groups distinct groups, each row’s group its independent unit of evidence (one symbol on one day, an episode), counted exactly over each node’s sampled rows (min_child_groups, DMatrix::with_split_groups);
    • LightGBM options extra_trees, path_smooth, linear_tree leaves (TrainingParams::extra_trees, path_smooth, linear_tree, LinearLeaves), and linear leaves that clamp their inputs to the ranges they were fit on, so they stay flat beyond them (LinearTree::clamp, linear_clamp);
    • LightGBM class-balanced bagging for binary classification (BalancedBagging: pos_bagging_fraction, neg_bagging_fraction), in place of subsample;
    • LightGBM XE-NDCG ranking (Objective::RankXendcg; its keyed per-round draws differ from LightGBM’s random stream) and query-level bagging (QueryBagging, bagging_by_query);
    • CatBoost-style symmetric trees (GrowPolicy::Symmetric), routed by bit pattern in batch prediction;
    • Boosted Trees on a Diet reuse penalties (toad_penalty_feature, toad_penalty_threshold) and a bit-packed layout with bit-identical margins (model::compact);
    • LightGBM-style quantized gradients (QuantizedGrad, use_quantized_grad);
    • PerpetualBooster-style budget training: one budget instead of eta/depth/rounds (training::budget);
    • in-place row addition and deletion (incremental learning and machine unlearning) for trained hist models, exact or approximate (training::online);
    • distributional boosting (NGBoost / XGBoostLSS style): dist:normal, dist:lognormal, dist:gamma, dist:poisson, dist:negbinomial, dist:student_t (heavy-tailed location-scale with learned degrees of freedom) per-row distributions (predict_distribution, objective::distributional), scored by nll / crps;
    • CatBoost’s Stochastic Gradient Langevin Boosting and model shrinkage (langevin, model_shrink, posterior_sampling) with virtual ensembles: knowledge, data, and total uncertainty from one model’s exactly rebuilt truncations (predict_uncertainty, model::uncertainty);
    • nonparametric p(y | x) by tree-based conditional diffusion (Treeffuser) and flow matching (DiffGBM) for scalar or vector labels, sampled deterministically (diffusion);
    • ForestFlow / ForestDiffusion tabular generation and imputation (diffusion::forest);
    • native Metal on macOS 10.15+ (metal feature): bit-identical GPU prediction (to_gpu, ~3x faster at scale) and bit-identical GPU histograms (device = metal; exact integer sums, CPU fallback outside their exact domain). Documented only in macOS builds with the feature (cargo doc --features metal); elsewhere backend::metal is a stub.
    • portable GPU support through wgpu (wgpu feature; Vulkan, Metal, DirectX 12): the same bit-identical prediction (to_wgpu) and histograms (device = wgpu), runnable without a GPU on a software adapter such as Mesa’s lavapipe (backend::wgpu).
    • NVIDIA CUDA on Linux (cuda feature): the same bit-identical prediction (to_cuda) and training (device = cuda/cuda:<n>; exact integer sums or the CPU’s ordered f64 sums, CPU fallback for the rest), with the driver loaded at run time; building the feature needs CUDA 13’s headers and libclang, running it no CUDA toolkit (backend::cuda).

examples/ has one program per topic (train_regression, binary_classification, multiclass, ranking, rank_xendcg, shap, model_io, custom_objective, constraints, conformal, boulevard_inference, ebm, compact_model, distributional, virtual_ensembles, tree_diffusion, forest_flow, budget, balanced_bagging, online_update, ordered_target_stats, pfn_boost, metal with --features metal on macOS, wgpu with --features wgpu). Run one with cargo run --release --example binary_classification.

§Compatibility notes

While hessboost is a standalone library, it offers extensive compatibility with XGBoost configurations and models: TrainingParams::from_xgboost reads XGBoost params dictionaries (keys, aliases, and value spellings), and unsupported settings are refused. Deterministic configurations reproduce XGBoost predictions within 1e-4 (quantile cuts bit for bit), and imported XGBoost models predict and explain identically. RNG-driven options (subsampling, forests, DART) match in quality only — the random streams differ.

§Not implemented

  • Distributed and external-memory training.
  • XGBoost options available at one setting only (so they are not TrainingParams fields; from_xgboost accepts exactly that setting): gblinear uses updater = coord_descent with feature_selector = cyclic; LambdaMART uses lambdarank_pair_method = topk (no lambdarank_unbiased or ndcg_exp_gain); DART has no sample_type or normalize_type (it samples uniformly and normalizes by tree); categorical splits use XGBoost’s defaults max_cat_to_onehot = 4 and max_cat_threshold = 64; and, writing no log output, hessboost takes verbosity = 0 only.
  • The metrics gamma-deviance, error@t (XGBoost’s classification threshold suffix), and the - variants of the ranking metrics (ndcg-, ndcg@k-, map-, map@k-); these names are refused.
  • XGBoost import and export of gblinear models.
  • gblinear on categorical features: refused, as XGBoost refuses it (type the column numerical to fit its codes as one weight).
  • scale_pos_weight = 0: XGBoost’s bound is >= 0; hessboost’s RegLoss::new needs a positive weight, so configurations and XGBoost files with 0 are refused.

Modules§

backend
Optional compute backends.
config
Training configuration types.
conformal
Distribution-free prediction intervals by split conformal prediction.
data
Datasets: the DMatrix container, the BinnedMatrix of its histogram bins, the MetaInfo view objectives and metrics read, the libsvm/CSV loaders, and the opt-in target_stats encoder for categorical columns.
diffusion
Nonparametric probabilistic regression: conditional diffusion and flow matching with boosted trees as the score or velocity model (opt-in).
ebm
Explainable boosting machines (EBMs): cyclic GA²M boosting, per-term shape functions, and (with Boulevard averaging) confidence bands on them. Opt-in: train with BoosterKind::Ebm, read the shape functions with shape_functions, and, for an Ebm::boulevard model, their bands with EbmInference.
error
Error types for hessboost.
inference
Statistical inference for Boulevard boosting: confidence intervals for the regression function f(x), prediction intervals for new labels, and reproduction intervals, with asymptotic (central-limit) guarantees under the assumptions below and validated only in the regimes listed in Validation. Opt-in: train with BoosterKind::Boulevard, then fit a BoulevardInference on the training rows. A Boulevard EBM (crate::ebm, ebm_boulevard) gets bands on its shape functions from EbmInference, built on the same solvers.
metric
Evaluation metrics used for reporting and early stopping.
model
Trained models: BoostedModel, its prediction, explanation, and persistence, and the size-optimized compact form.
objective
Learning objectives: gradients, Hessians, prediction transforms, and base score estimation.
prelude
The train-and-predict workflow in one import: use hessboost::prelude::*;.
training
Training: the boosting loop (train, Trainer), cross-validation (cv, or CrossValidation over caller-supplied or time-ordered Folds, with CrossValidation::refit retraining on every row), and opt-in budget training.
tree
The trees of a trained model (BoostedModel::trees), for inspection: RegTree, its Nodes, and the LinearLeaves of linear_tree models.