Skip to main content

TokenDecoder

wordchipper

Trait TokenDecoder 

Source 
pub trait TokenDecoder<T: TokenType>: Send + Sync {
    // Required method
    fn try_decode_to_bytes(
        &self,
        tokens: &[T],
    ) -> WCResult<DecodeResult<Vec<u8>>>;

    // Provided methods
    fn try_decode_batch_to_bytes(
        &self,
        batch: &[&[T]],
    ) -> WCResult<BatchDecodeResult<Vec<u8>>> { ... }
    fn try_decode_to_string(
        &self,
        tokens: &[T],
    ) -> WCResult<DecodeResult<String>> { ... }
    fn try_decode_batch_to_strings(
        &self,
        batch: &[&[T]],
    ) -> WCResult<BatchDecodeResult<String>> { ... }
}
Expand description

The common trait for &[T] -> Vec<u8>/String> decoders.

§Style Hints

When there is no local ambiguity with other decoders, instance names for implementing types should prefer decoder; and use the preferred name for the implementing type when there is conflict with other encoders.

Required Methods§

Source

fn try_decode_to_bytes(&self, tokens: &[T]) -> WCResult<DecodeResult<Vec<u8>>>

Decodes tokens into bytes.

§Arguments
  • tokens - A slice of tokens to decode.
§Returns

A Result<DecodeResult<Vec<u8>>>.

Provided Methods§

Source

fn try_decode_batch_to_bytes( &self, batch: &[&[T]], ) -> WCResult<BatchDecodeResult<Vec<u8>>>

Decodes a batch of tokens.

§Arguments
  • batch - A batch of tokens.
§Returns

A Result<Vec<DecodeResult<Vec<u8>>>>.

Source

fn try_decode_to_string(&self, tokens: &[T]) -> WCResult<DecodeResult<String>>

Decodes tokens into a string.

UTF-8 lossy decoding is used to handle invalid UTF-8 sequences.

§Arguments
  • tokens - A slice of tokens to decode.
§Returns

A Result<Vec<DecodeResult<String>>>.

Source

fn try_decode_batch_to_strings( &self, batch: &[&[T]], ) -> WCResult<BatchDecodeResult<String>>

Decodes a batch of tokens.

UTF-8 lossy decoding is used to handle invalid UTF-8 sequences.

§Arguments
  • batch - A batch of tokens.
§Returns

A Result<BatchDecodeResult<String>>.

Implementors§