Structs§
- Combiner
Config - Configuration for the file combiner
- Statistics
- Represents the statistics collected during the file combining process.
Functions§
- combine_
files - Combines files from the specified directory into a single output file.
- get_bpe
- Returns the appropriate BPE based on the tokenizer name.