Module indexer 

Document indexing module.

This module provides functionality to build and maintain document indices:

• Tree Building — Convert raw nodes to hierarchical trees
• Thinning — Merge small nodes with parents
• Merging — Combine adjacent small nodes
• Incremental — Update indices when documents change

Example

use vectorless::indexer::TreeBuilder;
use vectorless::document::RawNode;

let raw_nodes = vec![
    RawNode { level: 1, title: "Section 1".into(), ..Default::default() },
    RawNode { level: 2, title: "Subsection".into(), ..Default::default() },
];

let tree = TreeBuilder::new()
    .with_root_title("My Document")
    .build(raw_nodes);

Re-exports

pub use crate::config::IndexerConfig;

Structs

DiffResult

Diff result between old and new document structure.

IncrementalIndexer

Incremental indexer for updating existing indices.

ThinningConfig

Configuration for tree thinning.

ThinningConfigBuilder

Builder for ThinningConfig.

TreeBuilder

Builder for constructing document trees from raw nodes.

Functions

calculate_total_tokens

Calculate total token counts for all nodes (recursive, includes children).

diff_trees

Compare two document trees and find differences.

merge_adjacent_small_nodes

Merge adjacent sibling nodes that are below token threshold.

merge_children_into_parent

Merge all small children into parent.

subtree_token_count

Calculate total token count for a subtree.

thin_raw_nodes

Apply thinning to raw nodes before tree construction.

thin_tree

Thin a document tree by merging small nodes.