Struct Index

pub struct Index { /* private fields */ }

Index efficiently maps vocabulary tokens to state transitions.

Implementations

impl Index

The Index structure is designed to efficiently map tokens from a given vocabulary to state transitions within a finite-state automaton.

Usage:

The Index is typically constructed by combining a vocabulary and regular expressions. Once built, it can be used to efficiently evaluate token sequences or to validate input data.

Example:

use outlines_core::prelude::*;

let regex = "0|[1-9][0-9]*";
let vocabulary = Vocabulary::from_pretrained("openai-community/gpt2", None)?;
let index = Index::new(regex, &vocabulary)?;

let initial_state = index.initial_state();
println!("Initial state is {}", initial_state);
println!("Is initial state a final state? {}", index.is_final_state(&initial_state));

let allowed_tokens = index.allowed_tokens(&initial_state).expect("Some allowed tokens");
println!("Allowed tokens at initial state are {:?}", allowed_tokens);

let token_id = allowed_tokens.first().expect("First token");
println!("Next state for the token_id {} is {:?}", token_id, index.next_state(&initial_state, token_id));

println!("Final states are {:?}", index.final_states());
println!("Index has exactly {} transitions", index.transitions().len());

Performance:

• Complexity: The Index can accommodate large vocabularies and complex regular expressions. However, its size may grow significantly with the complexity of the input.
• Construction Cost: Building the Index involves processing the vocabulary and regular expressions, which may require a considerable amount of time and computational resources.

pub fn new(regex: &str, vocabulary: &Vocabulary) -> Result<Self>

Builds an Index from regular expression and vocabulary tokens.

pub fn initial_state(&self) -> StateId

Returns the ID of the initial state in the automaton.

pub fn final_states(&self) -> &HashSet<StateId>

Returns set of final states.

pub fn transitions(&self) -> &HashMap<StateId, HashMap<TokenId, StateId>>

Returns state transitions map of tokens ids and their corresponding transition states.

pub fn is_final_state(&self, state: &StateId) -> bool

Checks if state is in final states set or not.

pub fn allowed_tokens(&self, state: &StateId) -> Option<Vec<TokenId>>

Lists allowed tokens for a give state ID or None if it is not found in Index.

pub fn allowed_tokens_iter( &self, state: &StateId, ) -> Option<impl Iterator<Item = &TokenId>>

pub fn next_state(&self, state: &StateId, token_id: &TokenId) -> Option<StateId>

Returns transition state for a given state and token id or None otherwise.

pub fn vocab_size(&self) -> usize

Trait Implementations

impl<'__de, __Context> BorrowDecode<'__de, __Context> for Index

fn borrow_decode<__D: BorrowDecoder<'__de, Context = __Context>>( decoder: &mut __D, ) -> Result<Self, DecodeError>

Attempt to decode this type with the given BorrowDecode.

impl Clone for Index

fn clone(&self) -> Index

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Index

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<__Context> Decode<__Context> for Index

fn decode<__D: Decoder<Context = __Context>>( decoder: &mut __D, ) -> Result<Self, DecodeError>

Attempt to decode this type with the given Decode.

impl Display for Index

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Encode for Index

fn encode<__E: Encoder>(&self, encoder: &mut __E) -> Result<(), EncodeError>

Encode a given type.

impl PartialEq for Index

fn eq(&self, other: &Index) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for Index