Module avro_types 

Columns cast to types Polars’ Avro writer can hold, for an Avro export only.

The writer knows booleans, 32- and 64-bit integers and floats, strings, binary, dates, naive millisecond and microsecond datetimes, and lists and structs of those. Anything else fails the whole export with “not yet implemented”, so each is cast to the nearest type it does know. The casts are strict: a value that does not fit (a u64 past i64::MAX) fails the export by column name instead of turning null.

It also writes decimals, but wrongly: it drops the sign byte of a positive value whose leading byte is 0x80 or more, so every reader sees 327.68 as -327.68. Decimals are written as their exact text instead.

And it writes names as they are, with an empty record name, but an Avro name is [A-Za-z_][A-Za-z0-9_]* and strict readers refuse the file. [write] names the record and gives each column and struct field a valid name in the file’s schema, with the original as the field’s doc. It also writes the header once, where Polars’ writer repeats it for every chunk, and cuts blocks by size rather than one per chunk.

Constants

RECORD_NAME

The record name of an export. Polars’ default is empty, which strict readers refuse; its nested records are r1, r2, …, so this never clashes.

Functions

lazy_for_avro

lf with every column Avro cannot hold cast to one it can, under its own name. Planned, not run.

renames

Whether an Avro export renames this column or a struct field inside it.

write

Write df, prepared by lazy_for_avro, as an uncompressed Avro file: Polars’ AvroWriter’s encoding, with valid names, one header, and blocks of about [BLOCK_BYTES].