Skip to main content

normalize_fields

Function normalize_fields 

Source
pub fn normalize_fields(
    fields: HashMap<String, QuillValue>,
) -> HashMap<String, QuillValue>
Expand description

Normalizes document fields by applying all preprocessing steps.

This function orchestrates input normalization for document fields:

  1. Strips Unicode bidirectional formatting characters from all string values
  2. For the body field: fixes HTML comment fences to preserve trailing text

Double chevrons (<< and >>) are passed through unchanged in all fields.

§Processing Order

The normalization order is important:

  1. Bidi stripping - Must happen first so markdown delimiters are recognized
  2. HTML comment fence fixing - Ensures text after --> is preserved

§Examples

use quillmark_core::normalize::normalize_fields;
use quillmark_core::QuillValue;
use std::collections::HashMap;

let mut fields = HashMap::new();
fields.insert("title".to_string(), QuillValue::from_json(serde_json::json!("<<hello>>")));
fields.insert("BODY".to_string(), QuillValue::from_json(serde_json::json!("**bold** \u{202D}**more**")));

let result = normalize_fields(fields);

// Title has chevrons preserved (only bidi stripped)
assert_eq!(result.get("title").unwrap().as_str().unwrap(), "<<hello>>");

// Body has bidi chars stripped, chevrons preserved
assert_eq!(result.get("BODY").unwrap().as_str().unwrap(), "**bold** **more**");