pub enum DataSource {
File {
path: String,
format: FileFormat,
size_bytes: u64,
modified_at: Option<String>,
parquet_metadata: Option<ParquetMetadata>,
csv_metadata: Option<CsvSourceMetadata>,
},
Query {
engine: QueryEngine,
statement: String,
database: Option<String>,
execution_id: Option<String>,
},
DataFrame {
name: String,
source_library: DataFrameLibrary,
row_count: usize,
column_count: usize,
memory_bytes: Option<u64>,
},
Bytes {
name: String,
format: FileFormat,
size_bytes: u64,
csv_metadata: Option<CsvSourceMetadata>,
},
Stream {
topic: String,
batch_id: String,
partition: Option<u32>,
consumer_group: Option<String>,
source_system: Option<StreamSourceSystem>,
format: Option<FileFormat>,
session_id: Option<String>,
first_record_at: Option<String>,
last_record_at: Option<String>,
csv_metadata: Option<CsvSourceMetadata>,
},
}Expand description
Source-agnostic data source metadata.
Variants§
File
File-based data source (CSV, JSON, Parquet, etc.)
Fields
format: FileFormatDetected or specified file format
parquet_metadata: Option<ParquetMetadata>Parquet-specific metadata (only present for Parquet files)
csv_metadata: Option<CsvSourceMetadata>How a CSV source was split into fields (only present for CSV)
Query
SQL query-based data source
Fields
engine: QueryEngineDatabase engine used for the query
DataFrame
In-memory DataFrame source (pandas/polars via PyCapsule)
Fields
source_library: DataFrameLibrarySource library (pandas, polars, pyarrow)
Bytes
In-memory byte buffer profiled without a path behind it. The buffer may
be text (CSV/JSON/JSONL) or binary (Parquet); format says which.
Distinguished from DataSource::DataFrame: the caller handed over
bytes, not a schema-bearing dataframe, so provenance must say so.
Fields
format: FileFormatFormat the buffer was parsed as
csv_metadata: Option<CsvSourceMetadata>How a CSV source was split into fields (only present for CSV)
Stream
Streaming data source
Fields
source_system: Option<StreamSourceSystem>Source system identifier (kafka, kinesis, pulsar, http, etc.), as the caller declared it. Absent when the caller declared none: the report does not guess a transport (#1219).
format: Option<FileFormat>Format the stream was parsed as: arrow for an Arrow C stream
(#1230). Absent in reports written before 0.14 (#1042).
csv_metadata: Option<CsvSourceMetadata>How a CSV source was split into fields (only present for CSV)
Implementations§
Source§impl DataSource
impl DataSource
Sourcepub fn identifier(&self) -> String
pub fn identifier(&self) -> String
Get a human-readable identifier for this data source.
Sourcepub fn size_mb(&self) -> Option<f64>
pub fn size_mb(&self) -> Option<f64>
Get file size in megabytes if this is a file-based source or dataframe.
Sourcepub fn is_dataframe(&self) -> bool
pub fn is_dataframe(&self) -> bool
Check if this is a DataFrame-based source.
Sourcepub fn format(&self) -> Option<&FileFormat>
pub fn format(&self) -> Option<&FileFormat>
The format a file, bytes or stream source was parsed as. None for a
query or dataframe source, and for a stream that recorded none.
Sourcepub fn csv_metadata(&self) -> Option<&CsvSourceMetadata>
pub fn csv_metadata(&self) -> Option<&CsvSourceMetadata>
How a CSV source was split into fields, when the source records it.
Sourcepub fn with_csv_metadata(self, metadata: CsvSourceMetadata) -> Self
pub fn with_csv_metadata(self, metadata: CsvSourceMetadata) -> Self
Record how a CSV source was split into fields. A query or dataframe source has no delimiter, so it is returned unchanged.
Sourcepub fn stream_topic(&self) -> Option<&str>
pub fn stream_topic(&self) -> Option<&str>
Get the stream topic if this is a stream-based source.
Trait Implementations§
Source§impl Clone for DataSource
impl Clone for DataSource
Source§impl Debug for DataSource
impl Debug for DataSource
Source§impl<'de> Deserialize<'de> for DataSource
impl<'de> Deserialize<'de> for DataSource
Source§fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>where
__D: Deserializer<'de>,
fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>where
__D: Deserializer<'de>,
Source§impl JsonSchema for DataSource
impl JsonSchema for DataSource
Source§fn schema_id() -> Cow<'static, str>
fn schema_id() -> Cow<'static, str>
Source§fn json_schema(generator: &mut SchemaGenerator) -> Schema
fn json_schema(generator: &mut SchemaGenerator) -> Schema
Source§fn inline_schema() -> bool
fn inline_schema() -> bool
$ref keyword. Read more