Skip to main content

Utf8Validator

Trait Utf8Validator 

Source
pub trait Utf8Validator {
    // Required methods
    unsafe fn new() -> Self
       where Self: Sized;
    unsafe fn update(&mut self, input: &[u8]);
    unsafe fn finalize(self) -> Result<(), Utf8Error>;
}
Available on crate feature public_imp only.
Expand description

A low-level interface for streaming validation of UTF-8 data. It is meant to be integrated in high-performance data processing pipelines.

Data can be streamed in arbitrarily-sized chunks using the Self::update() method. There is no way to find out if the input so far was valid UTF-8 during the validation. Only when the validation is completed with the Self::finalize() method the result of the validation is returned. Use ChunkedUtf8Validator if possible for highest performance.

This implementation requires CPU SIMD features specified by the module it resides in. It is undefined behavior to use it if the required CPU features are not available which is why all trait methods are unsafe.

General usage:

use simdutf8::basic::imp::Utf8Validator;
use std::io::{stdin, Read, Result};

fn main() -> Result<()> {
    unsafe {
        if !std::is_x86_feature_detected!("avx2") {
            panic!("This example only works with CPUs supporting AVX 2");
        }

        let mut validator = simdutf8::basic::imp::x86::avx2::Utf8ValidatorImp::new();
        let mut buf = vec![0; 8192];
        loop {
            let bytes_read = stdin().read(buf.as_mut())?;
            if bytes_read == 0 {
                break;
            }
            validator.update(&buf);
        }

        if validator.finalize().is_ok() {
            println!("Input is valid UTF-8");
        } else {
            println!("Input is not valid UTF-8");
        }
    }

    Ok(())
}

Required Methods§

Source

unsafe fn new() -> Self
where Self: Sized,

Creates a new validator.

§Safety

This implementation requires CPU SIMD features specified by the module it resides in. It is undefined behavior to call it if the required CPU features are not available.

Source

unsafe fn update(&mut self, input: &[u8])

Updates the validator with input.

§Safety

This implementation requires CPU SIMD features specified by the module it resides in. It is undefined behavior to call it if the required CPU features are not available.

Source

unsafe fn finalize(self) -> Result<(), Utf8Error>

Finishes the validation and returns Ok(()) if the input was valid UTF-8.

§Errors

A basic::Utf8Error is returned if the input was not valid UTF-8. No further information about the location of the error is provided.

§Safety

This implementation requires CPU SIMD features specified by the module it resides in. It is undefined behavior to call it if the required CPU features are not available.

Dyn Compatibility§

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors§

Source§

impl Utf8Validator for simdutf8::basic::imp::x86::avx512::Utf8ValidatorImp

Available on avx512_stable and (runtime_detection, or crate feature public_imp, or target features avx512bw and avx512f and avx512vbmi2 and avx512vbmi) and (x86 or x86-64) only.
Source§

impl Utf8Validator for simdutf8::basic::imp::x86::avx2::Utf8ValidatorImp

Available on (crate feature public_imp, or runtime_detection and not (avx512_stable and target feature avx512bw and target feature avx512f and target feature avx512vbmi2 and target feature avx512vbmi), or target feature avx2 and non-runtime_detection and not (avx512_stable and target feature avx512bw and target feature avx512f and target feature avx512vbmi2 and target feature avx512vbmi)) and (x86 or x86-64) only.
Source§

impl Utf8Validator for simdutf8::basic::imp::x86::sse42::Utf8ValidatorImp

Available on (crate feature public_imp, or runtime_detection and neither avx512_stable and target feature avx512bw and target feature avx512f and target feature avx512vbmi2 and target feature avx512vbmi, nor target feature avx2 and non-avx512_stable, or target feature sse4.2 and non-runtime_detection and non-target feature avx2 and not (avx512_stable and target feature avx512bw and target feature avx512f and target feature avx512vbmi2 and target feature avx512vbmi)) and (x86 or x86-64) only.
Source§

impl Utf8Validator for simdutf8::basic::imp::portable::simd128::Utf8ValidatorImp

Available on crate feature portable_public_imp only.
Source§

impl Utf8Validator for simdutf8::basic::imp::portable::simd256::Utf8ValidatorImp

Available on crate feature portable_public_imp only.