//! 韩文音节可转换成的 Hanja：libhangul 在提交 717409ce61524bb3d8426060a384822f21354c62 的单音节读音（BSD-3-Clause，见 `resources/licenses/libhangul-hanja-BSD-3-Clause.txt`），msime-dictionary 把它原样收为 `sources/korean/hanja.txt`；`resources/dictionary-sources.lock.json` 用 `libhangul` 引用记下这个提交。
//!
//! `hanja.tsv` is generated by `msime-dict-build hanja` and embedded, so the table needs no download and no file on disk, and stays in clean read-only pages. Each line is `syllable<TAB>hanja<TAB>gloss`: the syllables come in code point order, each one's Hanja in libhangul's order, which is by frequency of use, and the gloss is the source's 훈음 (나라 이름 한), empty for the roughly three quarters of rows that have none. Only the syllable index is built at run time; the rows are read from the table as they are asked for.
use std::collections::HashMap;
use std::ops::Range;
use std::sync::OnceLock;
use crate::types::{CandidateSource, QueryRequest, SchemeType, WordItem};
const TABLE: &str = include_str!("hanja.tsv");
/// 只读浏览内嵌汉字表，编码用韩文读音；此表按来源顺序排序，没有数值权重。
pub(crate) fn browse(query: &str, offset: usize, limit: usize) -> Vec<crate::dictionary::DictRow> {
TABLE
.lines()
.filter_map(|line| {
let mut fields = line.split('\t');
let (key, value) = (fields.next()?, fields.next()?);
(query.is_empty() || key.starts_with(query) || value.contains(query))
.then_some((key, value))
})
.skip(offset)
.take(limit)
.map(|(key, value)| crate::dictionary::DictRow {
key: key.to_owned(),
value: value.to_owned(),
weight: 0,
})
.collect()
}
/// Byte range and row count of each syllable's lines in `TABLE`.
fn index() -> &'static HashMap<&'static str, (Range<usize>, usize)> {
static INDEX: OnceLock<HashMap<&'static str, (Range<usize>, usize)>> = OnceLock::new();
INDEX.get_or_init(|| {
let mut index: HashMap<&'static str, (Range<usize>, usize)> = HashMap::new();
let mut start = 0;
for line in TABLE.split_inclusive('\n') {
let end = start + line.len();
if let Some((syllable, _)) = line.split_once('\t') {
index
.entry(syllable)
.and_modify(|(range, count)| {
range.end = end;
*count += 1;
})
.or_insert((start..end, 1));
}
start = end;
}
index
})
}
/// The Hanja of `syllable` in table order, each with its gloss (empty where the source has none). Nothing for text the table does not list as one syllable, such as a lone jamo or two syllables.
pub fn readings(syllable: &str) -> impl Iterator<Item = (&'static str, &'static str)> {
let lines = index()
.get(syllable)
.map_or("", |(range, _)| &TABLE[range.clone()]);
lines.lines().filter_map(|line| {
let mut fields = line.split('\t');
let (_, hanja, gloss) = (fields.next()?, fields.next()?, fields.next()?);
Some((hanja, gloss))
})
}
fn reading_count(syllable: &str) -> usize {
index().get(syllable).map_or(0, |(_, count)| *count)
}
/// The 훈음 of `hanja` read as `syllable`; empty when the source has none or the pair is not in the table.
pub fn gloss(syllable: &str, hanja: &str) -> &'static str {
readings(syllable)
.find(|(candidate, _)| *candidate == hanja)
.map_or("", |(_, gloss)| gloss)
}
/// The candidate rows of an open Hanja list: the composing syllable's Hanja in table order. Each row carries the key letters as its code, as Japanese rows carry their romaji, so choosing one completes the composition; it is a `Database` row rather than `Generated`, which the runtime reads as a whole-sentence reading, and it is never learned (the session commits Korean rows without learning).
pub fn candidates(request: &QueryRequest) -> Vec<WordItem> {
let mut candidates = Vec::with_capacity(reading_count(&request.normalized_segmentation));
query_into(request, &mut candidates);
candidates
}
/// 将 Hanja 候选写入已有行缓冲，避免韩文列表每次刷新都重新分配候选字符串。
pub fn query_into(request: &QueryRequest, destination: &mut Vec<WordItem>) {
let readings = readings(&request.normalized_segmentation);
let mut used = 0;
for (hanja, _) in readings {
if let Some(item) = destination.get_mut(used) {
item.pinyin.clear();
item.pinyin.push_str(&request.raw_input_with_cases);
item.canonical_pinyin.clear();
item.word.clear();
item.word.push_str(hanja);
item.weight = 0;
item.source = CandidateSource::Database;
item.scheme = SchemeType::Korean;
item.fixed_position = 0;
item.fuzzy = false;
item.corrected_from.clear();
item.sentence_association = false;
item.sentence_words.clear();
item.touch_supported = false;
item.confident_correction = false;
} else {
let mut item = WordItem::new(
&request.raw_input_with_cases,
hanja,
0,
CandidateSource::Database,
"",
);
item.scheme = SchemeType::Korean;
destination.push(item);
}
used += 1;
}
destination.truncate(used);
}
#[cfg(test)]
mod tests {
use super::*;
fn is_syllable(text: &str) -> bool {
let mut characters = text.chars();
matches!((characters.next(), characters.next()), (Some(c), None) if ('\u{ac00}'..='\u{d7a3}').contains(&c))
}
/// CJK Unified Ideographs, Extension A, and the twelve unified ideographs of the compatibility block that NFC leaves alone (U+FA0E 﨎, U+FA11 﨑 and so on), the rule `msime-dict-build hanja` writes the table by.
fn is_kept_hanja(text: &str) -> bool {
const UNIFIED_IN_COMPATIBILITY_BLOCK: [char; 12] = [
'\u{fa0e}', '\u{fa0f}', '\u{fa11}', '\u{fa13}', '\u{fa14}', '\u{fa1f}', '\u{fa21}',
'\u{fa23}', '\u{fa24}', '\u{fa27}', '\u{fa28}', '\u{fa29}',
];
let mut characters = text.chars();
matches!((characters.next(), characters.next()), (Some(c), None) if ('\u{4e00}'..='\u{9fff}').contains(&c) || ('\u{3400}'..='\u{4dbf}').contains(&c) || UNIFIED_IN_COMPATIBILITY_BLOCK.contains(&c))
}
#[test]
fn the_embedded_table_is_well_formed() {
let mut previous = "";
let mut rows = 0;
for line in TABLE.lines() {
let fields: Vec<&str> = line.split('\t').collect();
assert_eq!(fields.len(), 3, "{line}");
assert!(is_syllable(fields[0]), "{line}");
assert!(is_kept_hanja(fields[1]), "{line}");
assert!(previous <= fields[0], "syllables out of order at {line}");
previous = fields[0];
rows += 1;
}
assert!(rows > 28_000, "{rows}");
// The twelve unified ideographs of the compatibility block are real Hanja that NFC leaves alone, so the table keeps them.
assert!(readings("기").any(|(hanja, _)| hanja == "\u{fa11}"));
assert!(readings("효").any(|(hanja, _)| hanja == "\u{fa27}"));
// Every syllable's lines are one contiguous range, so the index covers every row exactly once.
let indexed: usize = index()
.keys()
.map(|syllable| readings(syllable).count())
.sum();
assert_eq!(indexed, rows);
assert_eq!(index().len(), 555);
}
#[test]
fn readings_keep_the_source_order_and_gloss() {
let han: Vec<_> = readings("한").take(3).collect();
assert_eq!(
han,
[
("韓", "나라 이름 한, 한나라 한"),
("漢", "한수 한"),
("寒", "찰 한")
]
);
assert_eq!(readings("국").next(), Some(("國", "나라 국")));
assert_eq!(gloss("한", "漢"), "한수 한");
// A row without a 훈음 has an empty gloss, and a pair the table lacks has none either.
assert!(readings("한").any(|(_, gloss)| gloss.is_empty()));
assert_eq!(gloss("한", "國"), "");
}
#[test]
fn only_a_listed_syllable_has_readings() {
assert_eq!(readings("ㄱ").count(), 0);
assert_eq!(readings("한국").count(), 0);
assert_eq!(readings("").count(), 0);
assert_eq!(readings("a").count(), 0);
}
#[test]
fn candidates_are_korean_database_rows_keyed_by_the_letters() {
let request = QueryRequest {
scheme: SchemeType::Korean,
raw_input: "gks".into(),
raw_input_with_cases: "gks".into(),
normalized_segmentation: "한".into(),
valid: true,
..QueryRequest::default()
};
let rows = candidates(&request);
assert_eq!(rows.len(), readings("한").count());
assert_eq!(rows.capacity(), rows.len());
assert_eq!(rows[0].word, "韓");
assert_eq!(rows[1].word, "漢");
for row in &rows {
assert_eq!(row.pinyin, "gks");
assert_eq!(row.canonical_pinyin, "");
assert_eq!(row.source, CandidateSource::Database);
assert_eq!(row.scheme, SchemeType::Korean);
}
}
#[test]
fn query_into_reuses_existing_candidate_rows() {
let request = QueryRequest {
scheme: SchemeType::Korean,
raw_input: "gks".into(),
raw_input_with_cases: "gks".into(),
normalized_segmentation: "한".into(),
valid: true,
korean_hanja: true,
..QueryRequest::default()
};
let mut destination = candidates(&request);
let word_pointer = destination[0].word.as_ptr();
// 复用的行带着上一次（双拼触点纠错整句）的落点支持和首选标记，写入时要清掉。
destination[0].touch_supported = true;
destination[0].confident_correction = true;
query_into(&request, &mut destination);
assert_eq!(destination, candidates(&request));
assert_eq!(destination[0].word.as_ptr(), word_pointer);
}
}