Module processing 

Document processing and transformation utilities. Processing module for ReasonKit Core

Provides document and text processing utilities for the RAG pipeline.

Overview

This module handles:

• Text normalization and cleaning
• Token counting and estimation
• Text chunking strategies
• Processing pipeline orchestration

Modules

chunking

Document chunking module Document Chunking Module

Structs

NormalizationOptions

Text normalization options

ProcessingPipeline

Processing pipeline for documents

Functions

count_words

Count words in text

estimate_tokens

Estimate token count for text (rough approximation: ~4 chars per token)

extract_sentences

Extract sentences from text

normalize_text

Normalize text according to options

split_paragraphs

Split text into paragraphs