Module online 

In-place data addition and deletion for trained GBDT models (incremental and decremental learning, machine unlearning; opt-in).

An OnlineModel keeps a model together with its training data and, when approximate updates are enabled, the node statistics that let it update the model after rows are added to or deleted from that data without retraining from scratch. It follows Lin, Chung, Lao and Zhao, Online Gradient Boosting Decision Tree: In-Place Updates for Efficient Adding/Deleting Data (arXiv 2502.01634; reference code https://github.com/huawei-lin/InplaceOnlineGBDT), which extends their decremental method (Machine Unlearning in Gradient Boosting Decision Trees, KDD 2023) to additions.

Method

The iterations are walked in order. In each tree the nodes are visited from the root down; the rows that changed (added, deleted, or whose gradient is refreshed, below) update the node’s cached per-bin gradient histogram, and the node’s current split is ranked among every candidate of the updated histogram with the histogram builder’s own scoring. With OnlineParams::approximate’s tolerance σ, a split ranked within the top max(1, ⌊σ · candidates⌋) is kept (the paper’s split robustness tolerance); otherwise the subtree under the node is regrown with the histogram builder on the rows now reaching it. Leaves whose statistics changed get their weights recomputed.

Gradients are refreshed lazily (the paper’s adaptive lazy update): rows that were added, or that reached a regrown subtree in an earlier tree, get their gradients recomputed from the updated model; every other row keeps the gradient cached for it, since its margin only drifts by the (small) changes of the leaf weights it reaches. The histogram’s bin boundaries and the model’s intercept stay those of the original training, so an added value at or above a feature’s top bin boundary (beyond the training data’s range) is refused in this mode.

Exactness

OnlineParams::exact is the exact mode: every node’s split must still be the best one on recomputed statistics, bin boundaries and intercept, which is by definition what retraining produces, so the exact mode defers every tree to the builder: the result equals train on OnlineModel::data bit for bit, at about the cost of retraining (a reference to measure the approximate mode against, and the answer when unlearning must be exact). OnlineParams::approximate is approximate: the kept splits, the fixed bins and intercept, and the lazily refreshed gradients make the model differ from a retrain, by a gap that grows with the fraction of rows changed and with σ, in exchange for touching only the changed rows and the regrown subtrees.

Supported configurations

Updating is sound only where retraining the same parameters on the updated data is deterministic in the data alone and every node’s subtree depends on the node’s rows only: gbtree with tree_method = hist (or auto), depth-wise growth with a positive max_depth and no max_leaves, one output, num_parallel_tree = 1, no row or column sampling (a retrain draws its samples sequentially over the rows, so they would change with any added or deleted row), no monotone or interaction constraints, none of the opt-in split options (extra_trees, path_smooth, linear_tree, quantized gradients, reuse penalties), CPU, and a built-in objective whose gradients are per row and whose leaves are plain Newton steps (not ranking, survival:cox, reg:absoluteerror, reg:quantileerror, or a custom loss). The data may not carry weights, base margins, groups, label bounds, or feature weights, and the approximate mode needs numerical features. Everything else is refused.

Memory

The approximate mode caches, per tree, one gradient pair per row and one histogram (total_bins pairs of f64) per internal node: trees × (8 · rows + 16 · total_bins · internal nodes) bytes. An update works on a copy of it, swapped in on success, so an abandoned update restores the exact state it started from; the copy briefly doubles that memory.

Accuracy and speed

On 20,000-row synthetic tasks (Friedman #1 regression, its thresholded classification, and a 30-feature variant; 100 trees of depth 6, σ = 0.1), updates of 0.1% and 1% of the rows ran 1.3–4.8x faster than retraining, with test losses within about 0.5% of the retrained model’s; at 5% they were slower than retraining (regrown subtrees refresh most rows). Deleting rows raises the model’s loss on them, but by a fraction of what retraining does (e.g. regression RMSE on 200 deleted rows 0.877 → 0.903, retrained 1.080): the approximate mode forgets only partially, so unlearning that must be complete needs the exact mode.

Interruption

OnlineModel::train_with and OnlineModel::update_with call a per-iteration hook as Trainer::on_round does. Breaking stops training after the iteration (as it stops Trainer) and abandons an update, leaving the model, data and state unchanged. OnlineModel::update_with_commit also asks for a last confirmation once the update is computed, before it is applied.

Example

use hessboost::prelude::*;
use hessboost::training::online::{OnlineModel, OnlineParams};

let x: Vec<f32> = (0..200).map(|i| (i % 50) as f32 / 50.0).collect();
let y: Vec<f32> = x.iter().map(|v| 3.0 * v).collect();
let data = DMatrix::from_dense(&x, 200, 1)?.with_labels(&y)?;
let params = TrainingParams::builder()
    .tree_method(TreeMethod::Hist)
    .max_depth(3)
    .build()?;

let mut online = OnlineModel::train(&params, &data, 20, OnlineParams::default())?;
// Forget the first ten rows, learn two new ones.
let new = DMatrix::from_dense(&[0.5, 0.25], 2, 1)?.with_labels(&[1.5, 0.75])?;
let report = online.update(Some(&new), &(0..10).collect::<Vec<_>>())?;
assert_eq!(online.data().n_rows(), 192);
assert!(report.nodes_kept > 0);

Structs

OnlineModel

A trained model with its training data, updatable in place. See the module docs.

OnlineParams

Settings of OnlineModel: its OnlineMode, the exact mode (Self::exact, see the module docs) or the approximate one with a split robustness tolerance (Self::approximate). The default is approximate at 0.1, the paper’s recommendation.

UpdateReport

What an OnlineModel::update did.

Enums

OnlineMode

How an OnlineModel updates (see the module docs), read from OnlineParams::mode; built by OnlineParams::exact and OnlineParams::approximate.