Module iscc::normalization

Content Normalization

Functions

text_normalize

We define a text normalization function that is specific to our application. It takes text and an optional boolean keep_ws parameter as an input and returns normalized Unicode text for further algorithmic processing. The text_normalize function performs the following operations in the given order while each step works with the results of the previous operation: