Expand description
Feature extraction shared by the indexer and the detector of pelicode.
Source code is parsed with Tree-sitter, split into function units, normalized at several levels, and turned into exact hashes, winnowing fingerprints, MinHash signatures and LSH buckets.
Re-exports§
pub use config::Config;pub use config::IdentMode;pub use config::Level;pub use config::NORMALIZER_VERSION;pub use extract::FileFeatures;pub use extract::FunctionFeatures;pub use extract::LevelFeatures;pub use extract::extract_bytes;pub use extract::extract_file;pub use extract::extract_source;pub use hash::Fingerprint;pub use hash::Hash128;pub use language::Language;pub use scan::SourceFile;pub use scan::discover;pub use scan::extract_all;pub use scan::slash_path;
Modules§
- config
- Parameters of feature extraction.
- extract
- Function extraction and feature computation for a single source file.
- hash
- Exact hashes, winnowing fingerprints, MinHash signatures and LSH buckets.
- language
- Language adapters: the per-language rules for function extraction and normalization.
- scan
- Discovery and parallel extraction of source files under a directory.
Enums§
- Error
- Errors of feature extraction.