reddit-search

a tool for searching the pushshift reddit dumps written in rust. Available from crates.io via cargo install reddit-search

The dumps are available via torrent from here: https://academictorrents.com/details/7c0645c94321311bb05bd879ddee4d0eba08aaee

a sample dump from january 2012 can be downloaded from IPFS at: https://ipfs.io/ipfs/QmTGmjq6jkXi7oVsT9vdMwNdnGimWUpB1YagpBRCZRDpdd?filename=RC_2012-01.zst

usage

To see command line parameters, use reddit-search -h or --help

Sample usage commands

Basic Usage

reddit-search --input <input file path> --output <output file path> --fields <field:value> ...

Iterating over the whole dataset

Unix-like Shell (Bash, ZSH, etc)

for file in /path/to/dumps; do; reddit-search --append -i $file -o output.json -f field:value; done

Powershell

Get-ChildItem C:\path\to\dumps | ForEach-Object { reddit-search.exe --input $_.FullName -f KEY:DATA -o OUTPUT_FILENAME.json --append }

Descriptions of the fields contained within reddit dumps

Note that not all data contains all of these fields (for example, a comment from 2007 would not have the "gilded" field since that system was not implemented until later.)

Boolean values are saved numerically (0 is false, 1 is true)