pq

protobuf to json deserializer, written in Rust

pq is a tool which deserializes protobuf messages given a set of pre-compiled .fdset files. It can understand varint-delimited streams, and it can connect to Kafka.

pq will pretty-print when outputting to a tty, but you should pipe it to jq for more fully-featured json handling.

Dependency on serde-protobuf

Due to serde-protobuf being (sadly) almost abandonware at this point, I copy-pasted the source files into pq's src. Using a git override was causing me to be unable to publish with Rust 1.19.0.

The files:

• de.rs
• descriptor.rs
• serde_protobuf.rs
• serde_protobuf_error.rs
• value.rs

are not written by me. Original author is https://github.com/dflemstr.

Download

pq is on crates.io: cargo install pq. You can also download a static binary from the releases page.

Usage

To set up, put your *.fdset files in ~/.pq (specify an alternate directory with the FDSET_PATH= env var):

$ protoc -o dog.fdset dog.proto
$ protoc -o person.fdset person.proto
$ cp *.fdset ~/.pq/

Pipe a single compiled protobuf message:

$ pq --msgtype com.example.dog.Dog <./tests/samples/dog
{
  "age": 4,
  "breed": "poodle",
  "temperament": "excited"
}

Pipe a varint-delimited stream:

$ pq --msgtype com.example.dog.Dog --stream varint <./tests/samples/dog_stream
{
  "age": 10,
  "breed": "gsd",
  "temperament": "aggressive"
}

Consume from a Kafka stream:

$ pq kafka my_topic --brokers 192.168.0.1:9092 --beginning --count 1 --msgtype com.example.dog.Dog
{
  "age": 10,
  "breed": "gsd",
  "temperament": "aggressive"
}

Convert a Kafka stream to varint-delimited:

$ pq kafka my_topic --brokers=192.168.0.1:9092 --count 1 --convert varint |\
> pq --msgtype com.example.dog.Dog --stream varint
{
  "age": 10,
  "breed": "gsd",
  "temperament": "aggressive"
}