Trait Splitter

pub trait Splitter: Sized {
    type Error: ScanError;
    type TokenType;

    // Required method
    fn split<'input>(
        &mut self,
        data: &'input [u8],
    ) -> Result<(Option<(&'input [u8], Self::TokenType)>, usize), Self::Error>;
}

Split function used to tokenize the input

Required Associated Types

type Error: ScanError

type TokenType

Required Methods

fn split<'input>( &mut self, data: &'input [u8], ) -> Result<(Option<(&'input [u8], Self::TokenType)>, usize), Self::Error>

The arguments are an initial substring of the remaining unprocessed data.

If the returned error is non-nil, scanning stops and the error is returned to the client.

The function is never called with an empty data slice.

Dyn Compatibility

This trait is not dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety", so this trait is not object safe.

Implementors

impl Splitter for Tokenizer

use sqlite3_parser::lexer::sql::Tokenizer;
use sqlite3_parser::lexer::Scanner;

let tokenizer = Tokenizer::new();
let input = "PRAGMA parser_trace=ON;".as_bytes();
let mut s = Scanner::new(input, tokenizer);
let (token1, _) = s.scan().unwrap().unwrap();
s.scan().unwrap().unwrap();
assert!(b"PRAGMA".eq_ignore_ascii_case(token1));

type Error = Error

type TokenType = TokenType