Skip to main content

unique_records

Function unique_records 

Source
pub fn unique_records(dumps: &[Loaded]) -> Vec<(&Loaded, &Rec)>
Expand description

Returns a reference to every record across dumps, sorted into the global order (ts_nanos, instance_id, shard_id, event_id, fields), with duplicates removed.

A duplicate is any record sharing another’s full RecordKey — which happens whenever overlapping dumps are combined (successive dumps of one process re-capture shared ring contents). Because the key leads with the fields the converters sort by and ends with the bytes that distinguish records, sorting on it both produces the output order and lands byte-identical records side by side, so a single linear dedup scan removes them. No hash set: the only extra memory is the Vec of borrowed (&Loaded, &Rec) pairs (the field bytes stay in the dump buffer), which the caller needs anyway. Callers therefore receive already-sorted, de-duplicated records and need no further sort.