Struct ConfigValueGroup 

pub struct ConfigValueGroup {
    pub target_size: u64,
    pub max_target_size: u64,
    pub cache_size_limit: ByteSize,
    pub chunk_index_table_max_size: usize,
    pub cache_subdir: String,
}

ConfigValueGroup struct containing all configurable values

Fields

target_size: u64

The target shard size in bytes.

The default value is 67108864.

Use the environment variable HF_XET_SHARD_TARGET_SIZE to set this value.

max_target_size: u64

Maximum shard size; small shards are aggregated until they are at most this.

The default value is 67108864.

Use the environment variable HF_XET_SHARD_MAX_TARGET_SIZE to set this value.

cache_size_limit: ByteSize

The (soft) maximum size in bytes of the shard cache. Default is 16 GB.

As a rough calculation, a cache of size X will allow for dedup against data of size 1000 * X. The default would allow a 16 TB repo to be deduped effectively.

Note the cache is pruned to below this value at the beginning of a session, but during a single session new shards may be added such that this limit is exceeded.

The default value is 16gb.

Use the environment variable HF_XET_SHARD_CACHE_SIZE_LIMIT to set this value.

chunk_index_table_max_size: usize

The maximum size of the chunk index table that’s stored in memory. After this, no new chunks are loaded for deduplication.

The default value is 67108864.

Use the environment variable HF_XET_SHARD_CHUNK_INDEX_TABLE_MAX_SIZE to set this value.

cache_subdir: String

Subdirectory name for shard cache within the endpoint cache directory.

The default value is “shard-cache”.

Use the environment variable HF_XET_SHARD_CACHE_SUBDIR to set this value.

Implementations

impl ConfigValueGroup

pub fn new() -> Self

Create a new instance with default values only (no environment variable overrides).

pub fn apply_env_overrides(&mut self)

Apply environment variable overrides to this configuration group.

The group name is derived from the module path. For example, in module xet_config::groups::data, the env var for TEST_INT would be HF_XET_DATA_TEST_INT.

pub fn field_names() -> &'static [&'static str]

Returns the list of field names in this configuration group.

pub fn get(&self, name: &str) -> Result<String, ConfigError>

Get a configuration field’s string representation by name.

Trait Implementations

impl AsRef<ConfigValueGroup> for ConfigValueGroup

fn as_ref(&self) -> &ConfigValueGroup

Converts this type into a shared reference of the (usually inferred) input type.

impl Clone for ConfigValueGroup

fn clone(&self) -> ConfigValueGroup

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ConfigValueGroup

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for ConfigValueGroup

fn default() -> Self

Returns the “default value” for a type.