Struct PythonTorchDatasetConfig 

pub struct PythonTorchDatasetConfig {
    pub root: PathBuf,
    pub dataset_id: DatasetId,
    pub dataset_view_id: DatasetViewId,
    pub source_uri: String,
    pub format: String,
    pub manifest_hash: ContentId,
    pub preprocessing_hash: ContentId,
    pub tokenizer_hash: Option<ContentId>,
    pub sizing: DatasetSizing,
    pub planner: MicroShardPlannerConfig,
    pub microshards_per_batch: usize,
    pub metadata: BTreeMap<String, String>,
}

Declares the shard-backed dataset view exposed to the p2p runtime.

Fields

root: PathBuf

Root containing fetch-manifest.json and shard files.

dataset_id: DatasetId

Stable dataset id.

dataset_view_id: DatasetViewId

Stable dataset view id.

source_uri: String

Source URI surfaced in dataset metadata.

format: String

Dataset format tag.

manifest_hash: ContentId

Dataset manifest hash.

preprocessing_hash: ContentId

Preprocessing hash for the view.

tokenizer_hash: Option<ContentId>

Optional tokenizer hash.

sizing: DatasetSizing

Dataset sizing used to plan microshards.

planner: MicroShardPlannerConfig

Planner config used to derive microshard ids.

microshards_per_batch: usize

Number of cached microshards grouped into one Python batch ref.

metadata: BTreeMap<String, String>

Arbitrary dataset metadata propagated into the registration.

Implementations

impl PythonTorchDatasetConfig

pub fn registration(&self) -> DatasetRegistration

Returns a local-upstream dataset registration.

pub fn plan(&self) -> Result<MicroShardPlan>

Plans the microshards for this dataset view.

Trait Implementations

impl Clone for PythonTorchDatasetConfig

fn clone(&self) -> PythonTorchDatasetConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for PythonTorchDatasetConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for PythonTorchDatasetConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for PythonTorchDatasetConfig

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.