Module normalize 

Detect an SFT dataset’s schema and map each record to a {messages} Row.

Structs

ColumnMap

NormalizeReport

Enums

DropReason

Schema

Functions

collect_data_files

All jsonl/parquet files under raw_dir, recursively. Metadata files (README, .gitattributes, dataset_infos.json, and other bare .json) are excluded by only accepting .jsonl and .parquet.

detect_schema

Inspect the first record of sample and decide the schema. An override_map short-circuits detection to the generic user/assistant shape (PromptResponse), letting the caller map exotic columns explicitly.

map_record

normalize_dataset

Read raw_dir’s data files, detect the schema, map each record, and (when chat-shaped) write normalized {messages} JSONL to out_file. Always returns a report. For Unknown/TextOnly the report has wrote_output = false, rows_out = 0, and no file is created.

parse_column_map

Parse --map user=<col>,assistant=<col>[,system=<col>]. user and assistant are required.