Struct TreeTokenizer 

pub struct TreeTokenizer {
    pub universe: Universe,
    pub tree: HashMap<String, Lapper<u32, u32>>,
}

A tokenizer that uses an interval tree to find overlaps

Attributes

• universe - the universe of regions
• tree - the interval tree

Methods

• from - create a new TreeTokenizer from a bed file
• tokenize_region - tokenize a region into the vocabulary of the tokenizer
• tokenize_region_set - tokenize a region set into the vocabulary of the tokenizer
• tokenize_bed_set - tokenize a bed set into the vocabulary of the tokenizer
• unknown_token - get the unknown token

Fields

universe: Universe

tree: HashMap<String, Lapper<u32, u32>>

Implementations

impl TreeTokenizer

pub fn tokenize_region_set_batch( &self, region_sets: &Vec<RegionSet>, ) -> Option<Vec<TokenizedRegionSet<'_>>>

Trait Implementations

impl From<&Path> for TreeTokenizer

fn from(value: &Path) -> Self

Arguments

• value - the path to the bed file

Returns

A new TreeTokenizer

impl Tokenizer for TreeTokenizer

fn tokenize_region(&self, region: &Region) -> Option<TokenizedRegionSet<'_>>

Arguments

• region - the region to be tokenized

Returns

A TokenizedRegionSet that corresponds to one or more regions in the tokenizers vocab (or universe).

fn tokenize_region_set( &self, region_set: &RegionSet, ) -> Option<TokenizedRegionSet<'_>>

Tokenize a region set into the vocabulary of the tokenizer

fn padding_token(&self) -> Region

fn unknown_token(&self) -> Region

fn tokenize_bed_set( &self, bed_set: &BedSet, ) -> Option<Vec<TokenizedRegionSet<'_>>>

Tokenize a bed set into the vocabulary of the tokenizer