Module mode 

Tokenization modes and penalty configurations.

This module defines the different tokenization modes available and their penalty configurations for controlling segmentation behavior.

Structs

RbMode

Tokenization mode.

RbPenalty

Penalty configuration for decompose mode.

Functions

define

Defines Mode and Penalty classes in the given Ruby module.