Skip to main content

widen_csv_numeric_columns

hyperdb_mcp::schema

Function widen_csv_numeric_columns 

Source 
pub fn widen_csv_numeric_columns<R: Read>(
    reader: R,
    has_header: bool,
    columns: &mut [ColumnSchema],
) -> Result<(), McpError>
Expand description

Second-pass streaming widen: re-read the given CSV source and, for columns the first-pass inference classified as INT, BIGINT, or DOUBLE PRECISION, promote the type if a value outside its current range appears anywhere in the file (not just the first 1 000 rows).

The first pass handles column naming and ambiguous-type resolution; this pass exists specifically to catch “big value hidden near the end of a CSV” — the exact bug that prompted this code path, where OWID keeps world- aggregate populations (~8 billion) in the last rows of a file whose first thousand rows only contain country-sized numbers.

Promotion rules per numeric column:

  • INT → BIGINT if any value exceeds i32 range.
  • INT / BIGINT → NUMERIC(38,0) if any value exceeds i64 range.
  • INT / BIGINT → DOUBLE PRECISION if any value contains a decimal point or exponent (mixed integer/float column).

Columns with non-numeric inferred types are left untouched. Nullability is preserved. Empty fields are ignored.

§Errors

Returns ErrorCode::SchemaMismatch when the CSV parser fails on any row (typically unbalanced quotes, mismatched delimiters, or non-UTF-8 bytes reported by the csv crate).

§Panics

Does not panic in practice. The stats.get_mut(&col_idx).expect("preallocated") invariant holds because stats is preallocated with one entry per candidate_idxs value, and the loop only uses indices from that same slice.