Module nlp 

Natural Language Processing - Tokenizers, Embeddings, Text Processing

This module provides NLP utilities for text preprocessing and representation.

Structs

BPETokenizer

BPE (Byte Pair Encoding) Tokenizer

CharTokenizer

Character-level tokenizer

TfidfVectorizer

TF-IDF Vectorizer

Word2Vec

Word2Vec Skip-gram model (simplified)

WordTokenizer

Word-level tokenizer