Module gps 

GPS logs: NMEA 0183 and GPX, read into a table.

Neither format can be scanned in place, so an open reads the file once, start to end, a piece at a time, and writes the rows to a temporary Arrow IPC file a batch at a time. The dataset scans that file lazily like any other, and memory stays at one batch however long the log. The file is claimed through the open’s [Writer], so quitting or replacing the open removes it.

A GPX file can find a new extension field a million points in: the rows after it go to a new segment, a second IPC file with the wider schema, and the frame is the segments joined with the missing columns null. An NMEA table’s columns are fixed, so it is always one segment.

The readers take bytes as they come and hand back rows as they are ready, so a log still being written can later be followed by reading on from where they stopped.

Modules

gpx

GPX tracks, routes and waypoints: one row per trkpt, rtept or wpt.

nmea

NMEA 0183 logs: the sentences a GPS receiver writes, one per line.

table

Rows built a column at a time, handed over as a DataFrame a batch at a time.