Skip to main content

extract_chunks

shodh_memory::memory::query_parser

Function extract_chunks 

Source 
pub fn extract_chunks(text: &str) -> ChunkExtraction
Expand description

Extract chunks from text using shallow parsing

This function:

  1. Splits text into sentences
  2. Tags each word with its POS (noun, verb, adjective, etc.)
  3. Returns chunks that can be used for:
    • Entity extraction (all nouns, not just YAKE top-N)
    • Co-occurrence edge creation (words in same sentence)

Unlike YAKE, this doesn’t rank by frequency - ALL content words are extracted.