Crate alkale

Alkale is a library focused on streamlining repetitive aspects of hand-written tokenizers. Generally, only structures inside of self will need to be directly referenced by end-users, namely TokenizerContext and TokenizerResult.

The common sub-module contains many helpful method implementation for common tokenizer patterns such as tokenizing identifiers and numbers.

Modules

• common
  This module contains many helpful “built-in” helper methods added to TokenizerContext.
• notification
  Module that contains everything related to notifications. Notifications are how tokenizer output is handled by Alkale. (e.g. warnings, errors, etc)
• span
  Module containing types for span information.
• token
  Module for tokens and related types.

Structs

• TokenizerContext
  Provides many helpful methods for tokenization. This should be made once per tokenizer, and may be passed around as needed. When tokenization is done, result can convert this object into usable output.
• TokenizerIterator
  An iterator over a TokenizerContext’s tokens. This iterator completely ignores the context’s notifications.
• TokenizerResult
  The result of a tokenizer. This should generally be returned by tokenizer functions. This type is generally created by calling result on a TokenizerContext.