file-crawler 0.1.1

A fast, concurrent, async and customisable file crawler
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

# Custom-File-Crawler - a fast, concurrent, customisable and fully documented file crawler!
Features:
- fully multithreaded by default (2x - 3x speedup)
- Two options:
  - async
  - non-async
- set maximum traversal depth
- set file / folder regex
- lazyness (see lazy evaluation)
- execution a closure for every file (in the defined scope)
- having persistent data across closure invocations
- (set start directory)

## Example usage
 use file_crawler::prelude::*;

 use std::fs::File;
 use std::path::PathBuf;
 use std::sync::Arc;
 use std::sync::atomic::AtomicU32;

 let a_count=
 Crawler::new()
    .start_dir("C\\user\\foo")
    .context(AtomicU32::new(0))
    .run(|ctx: Arc<AtomicU32>, path: PathBuf| {
        let mut contents=String::new();
        let file=File::open(path)?;
        //NOTE: this can cause an error for files not readable as UTF-8
        //which returns an error and therefore terminates the crawler
        file.read_to_string(&mut contents)?;
        contents.chars().for_each(|char| if char=='a' { ctx.fetch_add(1); });
        Ok(())
    })?;
 println!("Appearance of the letter 'a' in \"C\\user\\foo\": {}", a_count)