Deduplication Signature

Deduplication Signature generates a hash of textual fields for deduplication. Currently it only supports TextProfileSignature.

Installation

Cargo

Add this to the Cargo.toml:

[dependencies]
dedup_signature = "^0.1.0"

Getting Started

Please follow the installation procedure and then run the following code:

extern crate dedup_signature;
use dedup_signature::generator::*;

let profile_generator = TextProfileSignature{ ..TextProfileSignature::default() };

let text = r#"Liberty, in philosophy, involves free will as contrasted with determinism.[1] In politics, liberty consists of the social and political freedoms enjoyed by all citizens.[2] In theology, liberty is freedom from the bondage of sin.[3] Generally, liberty seems to be distinct from freedom in that freedom concerns itself primarily, if not exclusively, with the ability to do as one wills and what one has the power to do; whereas liberty also takes into account the rights of all involved. As such, liberty can be thought of as freedom limited by rights, and therefore cannot be abused."#;

let sign = profile_generator.generate_sign(&text);

assert_eq!("6274be1f2560d8c9b8d344513d0b3942", sign);

Documentation for options

Name	Type	Description	Default value
min_token_length	int	The minimum token length to consider	2
quant_rate	float	When multiplied by the maximum token frequency, this determines count quantization	0.01

License

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.