Skip to main content

Module cjk

llm_kernel::graph

Module cjk 

Source
Expand description

CJK-aware graph search (Rust-side segmentation; no schema change). CJK-aware graph search (Rust-side segmentation, no schema change).

The bundled FTS5 table uses the trigram tokenizer, which matches poorly for short CJK queries (a 1–2 character Japanese/Korean query produces no usable trigram). This module adds a parallel CJK search path that:

  1. Segments the query in Rust — each CJK character becomes its own token, Latin/digit runs stay intact (segment_cjk).
  2. Matches tokens as substrings (LIKE '%tok%') against title, body, and tags, requiring every token to be present (AND semantics), ranked by importance.

Because this is plain LIKE over the existing columns, it touches no SQLite DDL — init_graph_schema is identical with or without graph-cjk, so a database created without the feature is safe to use with it and vice-versa. The cost is a scan per query (no FTS index), acceptable for the small-to-medium graphs this foundation targets.

Functions§

search_nodes_cjk
Search nodes for a CJK (or mixed) query via contiguous substring matching.
segment_cjk
Insert a space after every CJK character so whitespace tokenization treats each CJK character as a separate token. Latin/digit/punctuation runs and existing whitespace are left untouched.