Crate icu_provider

icu_provider is one of the ICU4X components.

Unicode’s experience with ICU4X’s parent projects, ICU4C and ICU4J, led the team to realize that data management is the most critical aspect of deploying internationalization, and that it requires a high level of customization for the needs of the platform it is embedded in. As a result ICU4X comes with a selection of providers that should allow for ICU4X to naturally fit into different business and technological needs of customers.

icu_provider defines traits and structs for transmitting data through the ICU4X locale data pipeline. The primary trait is DataProvider. It is parameterized by a KeyedDataMarker, which contains the data type and a DataKey. It has one method, DataProvider::load, which transforms a DataRequest into a DataResponse.

• DataKey is a fixed identifier for the data type, such as "plurals/cardinal@1".
• DataRequest contains additional annotations to choose a specific variant of the key, such as a locale.
• DataResponse contains the data if the request was successful.

In addition, there are three other traits which are widely implemented:

• AnyProvider returns data as dyn Any trait objects.
• BufferProvider returns data as [u8] buffers.
• DynamicDataProvider returns structured data but is not specific to a key.

The most common types required for this crate are included via the prelude:

use icu_provider::prelude::*;

Types of Data Providers

All nontrivial data providers can fit into one of two classes.

1. AnyProvider: Those whose data originates as structured Rust objects
2. BufferProvider: Those whose data originates as unstructured [u8] buffers

✨ Key Insight: A given data provider is generally either an AnyProvider or a BufferProvider. Which type depends on the data source, and it is not generally possible to convert one to the other.

See also crate::constructors.

AnyProvider

These providers are able to return structured data cast into dyn Any trait objects. Users can call as_downcasting() to get an object implementing DataProvider by downcasting the trait objects.

Examples of AnyProviders:

• DatagenProvider reads structured data from CLDR source files and returns ICU4X data structs.
• AnyPayloadProvider wraps a specific data struct and returns it.
• The BakedDataProvider which encodes structured data directly in Rust source

BufferProvider

These providers are able to return unstructured data typically represented as serde-serialized buffers. Users can call as_deserializing() to get an object implementing DataProvider by invoking Serde Deserialize.

Examples of BufferProviders:

• FsDataProvider reads individual buffers from the filesystem.
• BlobDataProvider reads buffers from a large in-memory blob.

Provider Adapters

ICU4X offers several built-in modules to combine providers in interesting ways. These can be found in the icu_provider_adapters crate.

Testing Provider

This crate also contains a concrete provider for demonstration purposes:

• HelloWorldProvider returns “hello world” strings in several languages.

Types and Lifetimes

Types compatible with Yokeable can be passed through the data provider, so long as they are associated with a marker type implementing DataMarker.

Data structs should generally have one lifetime argument: 'data. This lifetime allows data structs to borrow zero-copy data.

Data generation API

This functionality is enabled with the “datagen” Cargo feature

The datagen module contains several APIs for data generation. See icu_datagen for the reference data generation implementation.

Re-exports

• pub use crate::any::AnyMarker;
• pub use crate::any::AnyPayload;
• pub use crate::any::AnyProvider;
• pub use crate::any::AnyResponse;
• pub use crate::any::AsDowncastingAnyProvider;
• pub use crate::any::AsDynamicDataProviderAnyMarkerWrap;
• pub use crate::any::MaybeSendSync;
• pub use crate::buf::BufferMarker;
• pub use crate::buf::BufferProvider;
• pub use crate::marker::DataMarker;
• pub use crate::marker::KeyedDataMarker;
• pub use crate::serde::AsDeserializingBufferProvider;

Modules

• any
  Traits for data providers that produce Any objects.
• buf
  Traits for data providers that produce opaque buffers.
• constructors
  📚 This module documents ICU4X constructor signatures.
• datagen
  This module contains various utilities required to generate ICU4X data files, typically via the icu_datagen reference crate. End users should not need to consume anything in this module as a library unless defining new types that integrate with icu_datagen.
• dynutil
  Utilities for using trait objects with DataPayload.
• hello_world
  Data provider returning multilingual “Hello World” strings for testing.
• marker
  Marker types and traits for DataProvider.
• prelude
  Core selection of APIs and structures for the ICU4X data provider.
• serde
  Provides the DeserializingBufferProvider wrapper, which deserializes data using Serde.

Macros

• data_key
  See DataKey.
• impl_casting_upcast
  Implements UpcastDataPayload from several data markers to a single data marker that all share the same DataMarker::Yokeable.
• impl_dynamic_data_provider
  Implements DynamicDataProvider for a marker type S on a type that already implements DynamicDataProvider or DataProvider for one or more M, where M is a concrete type that is convertible to S via UpcastDataPayload.
• make_exportable_provider
  This macro can be used on a data provider to allow it to be used for data generation.

Structs

• AuxiliaryKeys
  The “auxiliary key” is an annotation on DataLocale that can contain an arbitrary information that does not fit into the LanguageIdentifier or Keywords.
• Cart
  The type of the “cart” that is used by DataPayload.
• DataError
  The error type for ICU4X data provider operations.
• DataKey
  Used for loading data from an ICU4X data provider.
• DataKeyHash
  A compact hash of a DataKey. Useful for keys in maps.
• DataKeyMetadata
  Metadata statically associated with a particular DataKey.
• DataKeyPath
  The string path of a data key. For example, “foo@1”
• DataLocale
  A locale type optimized for use in fallbacking and the ICU4X data pipeline.
• DataPayload
  A container for data payloads returned from a data provider.
• DataRequest
  The request type passed into all data provider implementations.
• DataRequestMetadata
  Metadata for data requests. This is currently empty, but it may be extended with options for tuning locale fallback, buffer layout, and so forth.
• DataResponse
  A response object containing an object as payload and metadata about it.
• DataResponseMetadata
  A response object containing metadata about the returned data.

Enums

• DataErrorKind
  A list specifying general categories of data provider error.

Traits

• DataProvider
  A data provider that loads data for a specific DataKey.
• DynamicDataProvider
  A data provider that loads data for a specific data type.

Attribute Macros

• data_struct
  The #[data_struct] attribute should be applied to all types intended for use in a DataStruct.