quilt-rs

Library and CLI provide a set of commands for managing data packages, allowing users to install, commit, push, and pull packages from S3 storage, as well as browse and track changes in package contents.

It supports operations like installing specific paths from packages, managing package metadata, and tracking package lineage with features for viewing status and handling workflows.

Testing

cargo test
cargo install cargo-watch
cargo watch # -x test

Publishing

Use Release job in CI, or

cargo update
cargo test
cargo publish

Coverage

cargo install cargo-tarpaulin
cargo tarpaulin --out html
open tarpaulin-report.html

Update Dependencies

cargo install cargo-upgrades
cargo upgrades

Verify files integrity

• sha256sum calculates SHA256 hash of a file.
• base64 converts binary data to base64.
• xxd -r -p converts HEX produced by SHA256 to binary
• split -b 8388608 splits file into 8 * 1024 * 1024 bytes

0Mb

sha256sum ./FILE | xxd -r -p | base64

Chunksum is stable:

• 47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=
• e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

<= 8Mb

sha256sum ./FILE | xxd -r -p | sha256sum | xxd -r -p | base64

> 8Mb

split -b 8388608 ./FILE --filter='sha256sum' | xxd -r -p | sha256sum | xxd -r -p | base64

Verify packages

split -l 1 ~/MANIFEST.jsonl --filter="jq -cSM 'del(.physical_keys)'" | tr -d '\n' | sha256sum