pub fn normalize_fields(
fields: HashMap<String, QuillValue>,
) -> HashMap<String, QuillValue>Expand description
Normalizes document fields by applying all preprocessing steps.
This function orchestrates input normalization for document fields:
- Strips Unicode bidirectional formatting characters from all string values
- For the body field: fixes HTML comment fences to preserve trailing text
Double chevrons (<< and >>) are passed through unchanged in all fields.
§Processing Order
The normalization order is important:
- Bidi stripping - Must happen first so markdown delimiters are recognized
- HTML comment fence fixing - Ensures text after
-->is preserved
§Examples
use quillmark_core::normalize::normalize_fields;
use quillmark_core::QuillValue;
use std::collections::HashMap;
let mut fields = HashMap::new();
fields.insert("title".to_string(), QuillValue::from_json(serde_json::json!("<<hello>>")));
fields.insert("BODY".to_string(), QuillValue::from_json(serde_json::json!("**bold** \u{202D}**more**")));
let result = normalize_fields(fields);
// Title has chevrons preserved (only bidi stripped)
assert_eq!(result.get("title").unwrap().as_str().unwrap(), "<<hello>>");
// Body has bidi chars stripped, chevrons preserved
assert_eq!(result.get("BODY").unwrap().as_str().unwrap(), "**bold** **more**");