pub fn normalize_fields(
fields: HashMap<String, QuillValue>,
) -> HashMap<String, QuillValue>Expand description
Normalizes document fields by applying all preprocessing steps.
This function orchestrates input normalization for document fields:
- Strips Unicode bidirectional formatting characters from all string values
- For the body field: converts
<<text>>to«text»(guillemets) - For other fields: strips chevrons entirely (
<<text>>→text)
§Processing Order
The normalization order is important:
- Bidi stripping - Must happen first so markdown delimiters are recognized
- Guillemet preprocessing - Converts user syntax to internal markers
§Examples
use quillmark_core::normalize::normalize_fields;
use quillmark_core::QuillValue;
use std::collections::HashMap;
let mut fields = HashMap::new();
fields.insert("title".to_string(), QuillValue::from_json(serde_json::json!("<<hello>>")));
fields.insert("body".to_string(), QuillValue::from_json(serde_json::json!("**bold** \u{202D}**more**")));
let result = normalize_fields(fields);
// Title has chevrons stripped
assert_eq!(result.get("title").unwrap().as_str().unwrap(), "hello");
// Body has bidi chars stripped (guillemet would apply if there were any <<>>)
assert_eq!(result.get("body").unwrap().as_str().unwrap(), "**bold** **more**");