Module combine 

One schema out of several files.

read_csv('data/*.csv') is one stream of rows and a stream has one schema, so the files a pattern matched have to agree on one. A Parquet file states its schema and the first file’s is taken as the answer, but a CSV file states nothing, so every file is sniffed and the answers are combined. That is not an optimisation choice, it is a correctness one: a directory of daily exports where one day’s file happens to hold whole numbers in a column that is otherwise decimal would otherwise come out BIGINT or DOUBLE depending on which day sorted first.

Both rules here were measured against duckdb v1.4.1 rather than reasoned about.

The types combine by widen. Two integers stay an integer, an integer and a double become a double, and everything else becomes text. Notably a date and a timestamp become text rather than a timestamp, which is not what a type lattice would say and is what the binary does.

The names have to match, and a file that is missing a column is mismatch, which is a different sentence from the one the Parquet reader gives for the same situation because the two readers in DuckDB are two pieces of code that each wrote their own.

Functions

across

The columns of a read that covers several files, given each file’s own sniffed columns.

mismatch

DuckDB’s message for a globbed file that does not have a column the first file has.

widen

The type a column has to be for values from both files to fit in it.