Trait TokenSource 

pub trait TokenSource {
    type Token: Hash + Eq;
    type Tokenizer: Iterator<Item = Self::Token>;

    // Required methods
    fn tokenize(&self) -> Self::Tokenizer;
    fn estimate_tokens(&self) -> u32;
}

A trait for types that can be split into tokens for diffing.

Implementing this trait allows a type to be used with InternedInput to create interned token sequences for computing diffs. For example, &str implements this trait by default to split text into lines.

Required Associated Types

type Token: Hash + Eq

The type of token this source produces. Must be hashable and comparable for equality.

type Tokenizer: Iterator<Item = Self::Token>

An iterator that yields tokens from this source.

Required Methods

fn tokenize(&self) -> Self::Tokenizer

Creates an iterator that yields all tokens from this source.

fn estimate_tokens(&self) -> u32

Provides an estimate of the number of tokens this source will produce.

This is used to pre-allocate memory for better performance. The estimate does not need to be exact.

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementations on Foreign Types

impl<'a> TokenSource for &'a str

By default, a line diff is produced for a string

type Token = &'a str

type Tokenizer = Lines<'a>

fn tokenize(&self) -> Self::Tokenizer

fn estimate_tokens(&self) -> u32

impl<'a> TokenSource for &'a BStr

By default, a line diff is produced for a BStr.

type Token = &'a BStr

type Tokenizer = BStrLines<'a>

fn tokenize(&self) -> Self::Tokenizer

fn estimate_tokens(&self) -> u32

impl<'a> TokenSource for &'a [u8]

By default, a line diff is produced for a bytes

type Token = &'a [u8]

type Tokenizer = ByteLines<'a>

fn tokenize(&self) -> Self::Tokenizer

fn estimate_tokens(&self) -> u32

Implementors

impl<'a> TokenSource for BStrLines<'a>

type Token = &'a BStr

type Tokenizer = BStrLines<'a>

impl<'a> TokenSource for ByteLines<'a>

By default, a line diff is produced for a string

type Token = &'a [u8]

type Tokenizer = ByteLines<'a>

impl<'a> TokenSource for Lines<'a>

By default, a line diff is produced for a string

type Token = &'a str

type Tokenizer = Lines<'a>

impl<'a> TokenSource for Words<'a>

type Token = &'a str

type Tokenizer = Words<'a>