normalize_fields

Function normalize_fields 

Source
pub fn normalize_fields(
    fields: HashMap<String, QuillValue>,
) -> HashMap<String, QuillValue>
Expand description

Normalizes document fields by applying all preprocessing steps.

This function orchestrates input normalization for document fields:

  1. Strips Unicode bidirectional formatting characters from all string values
  2. For the body field: converts <<text>> to «text» (guillemets)
  3. For other fields: strips chevrons entirely (<<text>>text)

§Processing Order

The normalization order is important:

  1. Bidi stripping - Must happen first so markdown delimiters are recognized
  2. Guillemet preprocessing - Converts user syntax to internal markers

§Examples

use quillmark_core::normalize::normalize_fields;
use quillmark_core::QuillValue;
use std::collections::HashMap;

let mut fields = HashMap::new();
fields.insert("title".to_string(), QuillValue::from_json(serde_json::json!("<<hello>>")));
fields.insert("BODY".to_string(), QuillValue::from_json(serde_json::json!("**bold** \u{202D}**more**")));

let result = normalize_fields(fields);

// Title has chevrons stripped
assert_eq!(result.get("title").unwrap().as_str().unwrap(), "hello");

// Body has bidi chars stripped (guillemet would apply if there were any <<>>)
assert_eq!(result.get("BODY").unwrap().as_str().unwrap(), "**bold** **more**");