Module nlp 

Natural language processing utilities Advanced NLP Module

This module provides advanced natural language processing capabilities:

• Semantic chunking algorithms
• Custom NER training pipeline
• Syntax analysis

Features

Semantic Chunking

• Multiple chunking strategies (sentence, paragraph, topic, semantic, hybrid)
• Intelligent boundary detection
• Coherence scoring
• Configurable chunk sizes and overlap

Custom NER

• Pattern-based entity extraction
• Dictionary/gazetteer matching
• Rule-based extraction with priorities
• Training dataset management
• Active learning support

Syntax Analysis

• Part-of-speech tagging
• Dependency parsing
• Noun phrase extraction

Re-exports

pub use semantic_chunking::ChunkingConfig;

pub use semantic_chunking::ChunkingStats;

pub use semantic_chunking::ChunkingStrategy;

pub use semantic_chunking::SemanticChunk;

pub use semantic_chunking::SemanticChunker;

pub use custom_ner::AnnotatedExample;

pub use custom_ner::CustomNER;

pub use custom_ner::DatasetStatistics;

pub use custom_ner::EntityType;

pub use custom_ner::ExtractedEntity;

pub use custom_ner::ExtractionRule;

pub use custom_ner::RuleType;

pub use custom_ner::TrainingDataset;

pub use syntax_analyzer::Dependency;

pub use syntax_analyzer::DependencyRelation;

pub use syntax_analyzer::NounPhrase;

pub use syntax_analyzer::POSTag;

pub use syntax_analyzer::SyntaxAnalyzer;

pub use syntax_analyzer::SyntaxAnalyzerConfig;

pub use syntax_analyzer::Token;

Modules

custom_ner

Custom NER Training Pipeline

semantic_chunking

Semantic Chunking

syntax_analyzer

Rule-based Syntax Analysis