Trait TokenSource 

pub trait TokenSource {
    type Token: Hash + Eq;
    type Tokenizer: Iterator<Item = Self::Token>;

    // Required methods
    fn tokenize(&self) -> Self::Tokenizer;
    fn estimate_tokens(&self) -> u32;
}

Available on crate feature blob only.

A trait for types that can be split into tokens for diffing.

Implementing this trait allows a type to be used with InternedInput to create interned token sequences for computing diffs. For example, &str implements this trait by default to split text into lines.

Required Associated Types

type Token: Hash + Eq

The type of token this source produces. Must be hashable and comparable for equality.

type Tokenizer: Iterator<Item = Self::Token>

An iterator that yields tokens from this source.

Required Methods

fn tokenize(&self) -> Self::Tokenizer

Creates an iterator that yields all tokens from this source.

fn estimate_tokens(&self) -> u32

Provides an estimate of the number of tokens this source will produce.

This is used to pre-allocate memory for better performance. The estimate does not need to be exact.

Implementations on Foreign Types

impl<'a> TokenSource for &'a str

By default, a line diff is produced for a string

type Token = &'a str

type Tokenizer = Lines<'a>

fn tokenize(&self) -> <&'a str as TokenSource>::Tokenizer

fn estimate_tokens(&self) -> u32

impl<'a> TokenSource for &'a BStr

By default, a line diff is produced for a BStr.

type Token = &'a BStr

type Tokenizer = BStrLines<'a>

fn tokenize(&self) -> <&'a BStr as TokenSource>::Tokenizer

fn estimate_tokens(&self) -> u32

impl<'a> TokenSource for &'a [u8]

By default, a line diff is produced for a bytes

type Token = &'a [u8]

type Tokenizer = ByteLines<'a>

fn tokenize(&self) -> <&'a [u8] as TokenSource>::Tokenizer

fn estimate_tokens(&self) -> u32

Implementors

impl<'a> TokenSource for ByteLinesWithoutTerminator<'a>

type Token = &'a [u8]

type Tokenizer = ByteLinesWithoutTerminator<'a>

impl<'a> TokenSource for BStrLines<'a>

type Token = &'a BStr

type Tokenizer = BStrLines<'a>

impl<'a> TokenSource for ByteLines<'a>

By default, a line diff is produced for a string

type Token = &'a [u8]

type Tokenizer = ByteLines<'a>

impl<'a> TokenSource for Lines<'a>

By default, a line diff is produced for a string

type Token = &'a str

type Tokenizer = Lines<'a>

impl<'a> TokenSource for Words<'a>

type Token = &'a str

type Tokenizer = Words<'a>