Skip to main content

Module merge

Module merge 

Source
Expand description

backbeat merge: combine several .bb dumps into one multi-instance .bb.

The dump format is inherently multi-instance — Meta, Intern, and Shard sections are each tagged with the instance_id of the process that produced them, and the schema registry is content addressed by a stable EventId. Merging therefore has two modes:

  • Default (dedup + trim). Decode each input’s records, drop duplicates — the same logged event re-captured by overlapping dumps (e.g. one process’s successive ring snapshots) — and re-pack the survivors into fresh, compact shards. The output is the smallest faithful dump: one row per distinct event. This is what you want before analysis, and it avoids paying to store/transfer redundant ring data.

  • --no-dedup (raw splice). Copy every input’s Meta/Intern/Shard bodies through verbatim and union the registries. Nothing is decoded, so it is cheap and lossless, but overlapping dumps keep their duplicates. Use this to quickly concatenate a host’s dumps for upload; convert always dedups on the way out, so the duplicates never reach the final table.

Either way the registry is unioned by id (identical event types collapse to one entry) and instance_ids are preserved, so converting the merged file yields exactly what converting the inputs together would.

Functions§

merge
Merges inputs into one multi-instance dump written to output.