Crate gtars_tokenizers 

gtars-tokenizers

Wrapper around gtars-overlaprs for producing tokens for machine learning models.

Purpose

This module wraps the core overlap infrastructure from gtars-overlaprs to convert genomic regions into vocabulary tokens for machine learning pipelines. It is specifically designed for ML applications that need to represent genomic intervals as discrete tokens.

Design Philosophy

All overlap computation is delegated to gtars-overlaprs. This module focuses on:

• Token vocabulary management
• Encoding/decoding strategies
• Integration with ML frameworks (HuggingFace, etc.)

Use Cases

• Transformer Models: Convert genomic regions to token sequences
• Feature Extraction: Represent intervals as discrete features for ML
• Language Model Input: Prepare genomic data for NLP-based models

Main Components

• Tokenizer: Maps regions to vocabulary tokens using overlap detection
• Universe: Vocabulary of genomic regions (peaks/intervals)

Example

use std::path::Path;
use gtars_tokenizers::Tokenizer;
use gtars_core::models::Region;

let tokenizer = Tokenizer::from_bed(Path::new("../tests/data/tokenizers/peaks.bed")).unwrap();

let regions = vec![Region {
    chr: "chr1".to_string(),
    start: 100,
    end: 200,
    rest: None,
}];
let tokens = tokenizer.tokenize(&regions);

Re-exports

pub use encoding::*;

pub use error::*;

pub use tokenizer::*;

pub use universe::*;

pub use utils::*;

Modules

config

consts

encoding

error

tokenizer

universe

This module defines the Universe struct, which represents a collection of regions. It also provides methods to convert between region strings and their corresponding IDs.

utils

This module contains utility functions for tokenizers. Basic things like universe prapration and special token handling are done here.