s3ueeze

Have you had a bunch offiles in S3 that you want to munge together? s3ueeze will walk through all the sources you pass to it, grab all the files therein, and write them to a single output file you specify.

Formats

s3ueeze works with any newline-delimited data format. To optimize memory usage, s3ueeze works line-wise on the data, so there's no ordering guarantee. Lines from multiple input files will be interleaved in the output.

JSON Lines (https://jsonlines.org/)