filedupes
Library that can be used to lazily (not load entire file directly) check for duplicates on two vectors of filecontent
Uses file hashing code from dupe-krill
Example
use filedupes::*;
fn main() {
// get the files in the closest subfolder
let mut origs = get_files(&"./",2);
// just do a quick removal of all original duplicates, sort will sort on size
origs.sort();
origs.dedup();
// compare to all files in all the subfolders
let mut checks = get_files(&"./", std::usize::MAX);
println!("{} Originals checked against {} files", origs.len(), checks.len());
let dupesets = check_duplicates(&origs, &mut checks);
println!( "Amount of duplicates of first {}", dupesets.first().unwrap().dupes.len() );
}
Binary
the crate contains a binary called ripdupe which is a small software tool that can check for duplicates and either remove them or just show and summarize them.
Example searching for duplicates of cats.jpg
cargo run --release -- -o testdata/cats.jpg -c testdata/
A small software tool that can compare one set of file, to find duplicates in another set. It has some differences to similar tools due to the fact it can limit the traversal depth in folders, thus allow to only change or include files that not in subfolders.
Example only taking the toplevel files in a folder, and looking for duplicates in the closest subfolders
cargo run --release -- -o testdata/ --odepth 1 -c testdata/ --cdepth 2
Warning
I take no responsibility of files lost, since this program is specifically created to delete duplicate files