Module split 

One CSV file read on many threads.

A Parquet file comes cut into row groups and says where each one starts. A CSV file does not, and where a record starts is only known by reading every byte before it, since a newline inside a quoted field is part of a value and not the end of a line. Read that way a large file is read on one thread however many the query was given.

So the file is cut into ranges of about RANGE bytes and each range guesses. A range that is not the first assumes its first record starts just after the first line ending in it, which is right unless that line ending is inside a quoted field, and it splits records from there to learn where its last record ends. That is where the next range really starts if the guess was right, so the true starts are known a range at a time as fast as the guesses can be checked, and a range converts nothing until its own start is one of them. Converting is most of the work, and it is done once, from the right place, on every thread at the same time.

A wrong guess costs a second split of that range from its true start, which is also what a range that nobody guessed for gets, so a guess only ever saves time. A range whose owner is slow to guess is split by whichever thread is waiting on it, which also means no range can be left waiting for one that is never read.

Errors are where the ranges have to agree with one thread exactly. A value that does not fit its column is reported with its line number, which is a count of the records before it, and when several ranges fail at once the one reported has to be the first a single thread would have met. A range that fails therefore does not report its own error. The file is read again on one thread from the end of the last range that converted without one, in the same chunks a whole read makes, and the first error that read meets is the one every range reports. That is slow, and it only happens to a query that is failing.

Structs

Part

One range of a Split, read a chunk at a time like a whole Reader.

Split

A file cut into ranges that are read on separate threads and come back in file order.

Constants

RANGE

How many bytes of the file a range covers.

Functions

size

How many bytes of a file a range covers, which is RANGE unless a test said otherwise.