Enum DataSource 

pub enum DataSource {
    File {
        path: String,
        format: FileFormat,
        size_bytes: u64,
        modified_at: Option<String>,
        parquet_metadata: Option<ParquetMetadata>,
    },
    Query {
        engine: QueryEngine,
        statement: String,
        database: Option<String>,
        execution_id: Option<String>,
    },
    DataFrame {
        name: String,
        source_library: DataFrameLibrary,
        row_count: usize,
        column_count: usize,
        memory_bytes: Option<u64>,
    },
    Stream {
        topic: String,
        batch_id: String,
        partition: Option<u32>,
        consumer_group: Option<String>,
        source_system: StreamSourceSystem,
        session_id: Option<String>,
        first_record_at: Option<String>,
        last_record_at: Option<String>,
    },
}

Source-agnostic data source metadata.

Variants

File

File-based data source (CSV, JSON, Parquet, etc.)

Fields

path: String

Absolute or relative path to the file

format: FileFormat

Detected or specified file format

size_bytes: u64

File size in bytes

modified_at: Option<String>

Last modification timestamp (ISO 8601 / RFC 3339)

parquet_metadata: Option<ParquetMetadata>

Parquet-specific metadata (only present for Parquet files)

Query

SQL query-based data source

Fields

engine: QueryEngine

Database engine used for the query

statement: String

SQL statement executed

database: Option<String>

Target database name (if applicable)

execution_id: Option<String>

Unique execution identifier for tracing

DataFrame

In-memory DataFrame source (pandas/polars via PyCapsule)

Fields

name: String

User-provided name for identification

source_library: DataFrameLibrary

Source library (pandas, polars, pyarrow)

row_count: usize

Number of rows at profiling time

column_count: usize

Number of columns

memory_bytes: Option<u64>

Memory usage in bytes (if available)

Stream

Streaming data source

Fields

topic: String

Stream identifier (e.g., Kafka topic, Kinesis stream name)

batch_id: String

Batch identifier for ordering and deduplication

partition: Option<u32>

Partition for parallel processing (optional)

consumer_group: Option<String>

Consumer group for Kafka-style coordination (optional)

source_system: StreamSourceSystem

Source system identifier (kafka, kinesis, pulsar, http, etc.)

session_id: Option<String>

Session ID for multi-tenant scenarios

first_record_at: Option<String>

Timestamp of first record in batch (ISO 8601)

last_record_at: Option<String>

Timestamp of last record in batch (ISO 8601)

Implementations

impl DataSource

pub fn identifier(&self) -> String

Get a human-readable identifier for this data source.

pub fn size_mb(&self) -> Option<f64>

Get file size in megabytes if this is a file-based source or dataframe.

pub fn is_file(&self) -> bool

Check if this is a file-based source.

pub fn is_query(&self) -> bool

Check if this is a query-based source.

pub fn is_dataframe(&self) -> bool

Check if this is a DataFrame-based source.

pub fn is_stream(&self) -> bool

Check if this is a Stream-based source.

pub fn file_path(&self) -> Option<&str>

Get the file path if this is a file-based source.

pub fn stream_topic(&self) -> Option<&str>

Get the stream topic if this is a stream-based source.

pub fn batch_id(&self) -> Option<&str>

Get the batch ID if this is a stream-based source.

Trait Implementations

impl Clone for DataSource

fn clone(&self) -> DataSource

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for DataSource

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for DataSource

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for DataSource

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.