pub struct StreamingTokenizer<T: SignalTokenizer> { /* private fields */ }Expand description
Streaming tokenizer for processing long sequences in chunks
Implementations§
Source§impl<T: SignalTokenizer> StreamingTokenizer<T>
impl<T: SignalTokenizer> StreamingTokenizer<T>
Sourcepub fn new(
tokenizer: T,
chunk_size: usize,
overlap: usize,
) -> TokenizerResult<Self>
pub fn new( tokenizer: T, chunk_size: usize, overlap: usize, ) -> TokenizerResult<Self>
Create a new streaming tokenizer
§Arguments
tokenizer- The underlying tokenizerchunk_size- Size of each chunkoverlap- Overlap between chunks (for continuity)
Sourcepub fn encode_streaming(
&self,
signal: &Array1<f32>,
) -> TokenizerResult<Vec<Array1<f32>>>
pub fn encode_streaming( &self, signal: &Array1<f32>, ) -> TokenizerResult<Vec<Array1<f32>>>
Encode a long signal in chunks
Sourcepub fn decode_streaming(
&self,
chunks: &[Array1<f32>],
) -> TokenizerResult<Array1<f32>>
pub fn decode_streaming( &self, chunks: &[Array1<f32>], ) -> TokenizerResult<Array1<f32>>
Decode chunks and reconstruct signal with overlap handling
Sourcepub fn chunk_size(&self) -> usize
pub fn chunk_size(&self) -> usize
Get chunk size
Auto Trait Implementations§
impl<T> Freeze for StreamingTokenizer<T>where
T: Freeze,
impl<T> RefUnwindSafe for StreamingTokenizer<T>where
T: RefUnwindSafe,
impl<T> Send for StreamingTokenizer<T>where
T: Send,
impl<T> Sync for StreamingTokenizer<T>where
T: Sync,
impl<T> Unpin for StreamingTokenizer<T>where
T: Unpin,
impl<T> UnwindSafe for StreamingTokenizer<T>where
T: UnwindSafe,
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Mutably borrows from an owned value. Read more
Source§impl<T> Instrument for T
impl<T> Instrument for T
Source§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
Source§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
Converts
self into a Left variant of Either<Self, Self>
if into_left is true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
Converts
self into a Left variant of Either<Self, Self>
if into_left(&self) returns true.
Converts self into a Right variant of Either<Self, Self>
otherwise. Read more