Struct DataItem 

pub struct DataItem {
    pub label: Label,
    pub media_type: MediaType,
    pub source_name: String,
    pub extension: String,
    pub metadata: HashMap<String, String>,
    /* private fields */
}

A single data sample flowing through the dataset pipeline.

Fields

label: Label

Dataset label.

media_type: MediaType

Media kind for the sample.

source_name: String

Logical source name.

extension: String

File extension including the leading dot.

metadata: HashMap<String, String>

String metadata attached to the sample.

Implementations

impl DataItem

pub fn new( bytes: Vec<u8>, label: Label, media_type: MediaType, source_name: impl Into<String>, ) -> AppResult<Self>

Create an item from bounded in-memory bytes.

pub fn new_bytes( bytes: Vec<u8>, label: Label, media_type: MediaType, source_name: impl Into<String>, ) -> AppResult<Self>

Create an item from bounded in-memory bytes.

pub fn new_bytes_with_limits( bytes: Vec<u8>, label: Label, media_type: MediaType, source_name: impl Into<String>, limits: &DatasetLimits, ) -> AppResult<Self>

Create an item from bounded in-memory bytes with explicit limits.

pub fn new_file( path: impl Into<PathBuf>, label: Label, media_type: MediaType, source_name: impl Into<String>, ) -> Self

Create an item from a file path for streaming large payloads.

pub fn payload(&self) -> &DataPayload

Borrow the item payload.

pub fn try_with_payload( self, payload: DataPayload, limits: &DatasetLimits, ) -> AppResult<Self>

Replace the payload after validating it against explicit limits.

pub fn source_offset(&self) -> Option<usize>

Return the source-provided resume cursor after this item, if available.

pub fn with_source_offset(self, offset: usize) -> Self

Attach a source-provided resume cursor.

pub fn with_extension(self, ext: impl Into<String>) -> Self

Set the output extension.

pub fn with_metadata( self, key: impl Into<String>, value: impl Into<String>, ) -> Self

Attach string metadata.

pub fn validate(&self, limits: &DatasetLimits) -> AppResult<()>

Validate the item against configured limits and path safety rules.

pub fn write_to_path( &self, path: &Path, limits: &DatasetLimits, ) -> AppResult<u64>

Write this item to path using the configured payload limits.

Trait Implementations

impl Clone for DataItem

fn clone(&self) -> DataItem

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl DatasetItem for DataItem

fn label(&self) -> Label

Classification label used to route and count the item. Defaults to Label::Real.

fn source_offset(&self) -> Option<usize>

Source-provided resume cursor observed after this item, if any.

impl Debug for DataItem

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl ItemSink<DataItem> for LocalBlobSink

fn name(&self) -> &str

Stable sink name.

fn prepare<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = AppResult<()>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Prepare the destination before any item is written (create directories, seed counters).

fn write<'life0, 'async_trait>( &'life0 self, item: DataItem, ) -> Pin<Box<dyn Future<Output = AppResult<()>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Materialize a single collected item.

fn finish<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = AppResult<()>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Flush and finalize the destination after the last item is written.

impl Transform<DataItem, DataItem> for ResizeTransform

Available on crate feature image-transform only.

fn name(&self) -> &str

Stable transform name.

fn apply( &self, item: DataItem, limits: &DatasetLimits, ) -> AppResult<Option<DataItem>>

Apply the transform, returning Ok(None) when the item is filtered out.