Module token_counter

Token counting and content prioritization module.

This module provides functionality for accurate token counting using the GPT-4 tokenizer and intelligent content prioritization to fit within token limits.

Structs

ContentPrioritizer

Content prioritizer that manages sections based on priority and token limits.

ContentSection

A content section with associated metadata for prioritization.

TokenCounter

Token counter using the GPT-4 tokenizer for accurate token counting.