DIDI File Format

This is a learning project that demonstrates how to create a custom file format using Rust. The DIDI file format includes capabilities for encoding data to save on storage and metadata to indicate if a specific string is present in the file.

Features

• Encoding Data: Stores data by converting it into a specific format using Run-Length Encoding (RLE).
• Metadata: Indicates whether a specific string is present anywhere in the file.

Installation

Add this to your Cargo.toml:

[dependencies]
didi_file_format = "0.1.0"

Usage

Here’s a basic example of how to use the DIDI file format in your Rust project.

Writing to a DIDI File

use didi_file_format::write_didi;

fn main() -> std::io::Result<()> {
    let data = "aaabbcccc";
    let search_string = "a3";

    write_didi("example.didi", data, search_string)?;
    Ok(())
}

Reading from a DIDI File

use didi_file_format::read_didi;

fn main() -> std::io::Result<()> {
    let (decoded_data, stored_search_string) = read_didi("example.didi")?;
    println!("Decoded Data: {}", decoded_data);
    println!("Stored Search String: {}", stored_search_string);
    Ok(())
}

Using the Sniffer Function

use didi_file_format::sniffer;

fn main() -> std::io::Result<()> {
    let stored_search_string = sniffer("example.didi")?;
    println!("Sniffer found the stored search string: {}", stored_search_string);
    Ok(())
}

Generating a Large File and Comparing Performance

To generate a large file and compare the performance of finding a string using the DIDI file format vs. manually searching the file, use the compare_performance example.

use std::fs;
use std::io;
use std::time::Instant;
use didi_file_format::{write_didi, sniffer};

fn generate_large_file(repetitions: usize) -> io::Result<()> {
    let letters = vec!["a", "b", "c"];
    let mut generated_string = String::new();

    for i in 0..repetitions {
        let letter = letters[i % letters.len()];
        for _ in 0..10 {
            generated_string.push_str(letter);
        }
    }

    // Append custom string at the end
    generated_string.push_str("custom_formats_are_interesting");

    // Save to a text file
    fs::write("large_example.txt", &generated_string)?;

    Ok(())
}

fn manual_search(file_path: &str, search_string: &str) -> bool {
    if let Ok(content) = fs::read_to_string(file_path) {
        return content.contains(search_string);
    }
    false
}

fn main() -> io::Result<()> {
    let search_string = "specific_string";
    let file_path = "large_example.txt";

    if !file_exists(file_path)? {
        generate_large_file(100000)?;
    } else {
        println!("File already exists: {}", file_path);
    }

    let data = fs::read_to_string(file_path)?;  // Read the generated file as a string

    write_didi("large_example.didi", &data, search_string)?;

    let start_sniffer = Instant::now();
    let found_sniffer = sniffer("large_example.didi")?;
    let duration_sniffer = start_sniffer.elapsed();

    let start_manual = Instant::now();
    let found_manual = manual_search("large_example.didi", search_string);
    let duration_manual = start_manual.elapsed();

    println!("Sniffer found: {}", found_sniffer);
    println!("Sniffer duration: {:?}", duration_sniffer);
    println!("Manual search found: {}", found_manual);
    println!("Manual search duration: {:?}", duration_manual);

    Ok(())
}

fn file_exists(file_path: &str) -> io::Result<bool> {
    match fs::metadata(file_path) {
        Ok(_) => Ok(true),
        Err(ref e) if e.kind() == io::ErrorKind::NotFound => Ok(false),
        Err(e) => Err(e),
    }
}

License

This project is licensed under the MIT License. See the LICENSE file for details.

Contributing

Contributions are welcome! Please open an issue or submit a pull request.

Contact

For any questions or feedback, feel free to reach out.

Enjoy using the DIDI file format!