Struct FileSourceConfig 

pub struct FileSourceConfig {

    pub path: String,
    pub format: FileSourceFormat,
    pub compression: CompressionConfig,
    pub recursive: bool,
    pub concurrency: usize,
    pub batch_size: usize,
    pub incremental: Option<IncrementalSpec>,
    pub stable_for_secs: Option<u64>,
    pub strict: bool,
    pub max_object_bytes: u64,
    pub max_decoded_bytes: Option<u64>,
    pub max_files: Option<usize>,
    pub headers: BTreeMap<String, String>,
    pub http_retries: u32,
    pub http_connect_timeout_secs: u64,
    pub http_read_timeout_secs: u64,
    pub csv: CsvOptions,
    pub excel: ExcelOptions,
    pub xml: XmlOptions,
    pub avro: AvroOptions,
    pub orc: OrcOptions,
    pub parquet: ParquetReadOptions,
    pub encryption: Option<EncryptionSpec>,
}

Configuration for the local (and http(s)://) file source.

Experimental (PRINCIPLES.md §3): this block’s shape may change in a minor release; any change is called out in the changelog.

Fields

path: String

A file, a directory, a glob (data/**/*.jsonl), or one http:// / https:// URL. A directory lists its regular files (descending into subdirectories with recursive: true); every listing is sorted by path.

format: FileSourceFormat

File format, auto (by extension, per file) by default. Under auto a file with no recognised extension is skipped with a warning, or refused when strict: true. JSON Lines, Avro, ORC and Parquet stream; JSON arrays, CSV, XML and Excel are read whole per file.

compression: CompressionConfig

Compression codec. auto (default) resolves per file from its suffix (.gz, .zst).

recursive: bool

Descend into subdirectories when path is a directory.

concurrency: usize

Files opened ahead of the one being decoded (ordered prefetch). 0 reads serially.

batch_size: usize

Records per emitted page. 0 emits one page per file.

incremental: Option<IncrementalSpec>

Read only files that are new since the previous run. Needs a pipeline state: block; the bookmark is kept under file:<path>.

stable_for_secs: Option<u64>

Skip files modified less than this many seconds ago — files still being written. They are read by a later run once they settle.

strict: bool

Refuse (instead of skipping with a warning) a file whose extension names no format under format: auto.

max_object_bytes: u64

Largest file, once decompressed, that a format read whole (JSON array, XML, Excel, compressed or remote Avro/ORC/Parquet, encrypted files) may reach; a larger one fails with an error instead of exhausting memory. Default 2 GiB.

max_decoded_bytes: Option<u64>

Most bytes one object may decode to in memory where a format can expand past its input: an Avro file’s records (per batch where the file streams), an Excel workbook’s parts and shared strings. A file past it fails with an error instead of exhausting memory. Default 64 × max_object_bytes, at most 16 GiB.

max_files: Option<usize>

Read at most this many files (after sorting and filtering).

headers: BTreeMap<String, String>

Request headers for an http(s):// path (e.g. Authorization).

http_retries: u32

Retries for a failed HTTP request (connection error, 429, 5xx), with exponential backoff that honours Retry-After.

http_connect_timeout_secs: u64

Seconds to wait for an HTTP connection to open. A timeout is retried.

http_read_timeout_secs: u64

Seconds an HTTP response may go without sending a byte — waiting for the headers or mid-body. A timeout before the body starts is retried; one mid-body fails the file.

csv: CsvOptions

CSV dialect, used for CSV files.

excel: ExcelOptions

Worksheet selection, used for Excel files.

xml: XmlOptions

Record framing, used for XML files.

avro: AvroOptions

Reader schema, used for Avro files.

orc: OrcOptions

Column projection, used for ORC files.

parquet: ParquetReadOptions

Column projection, used for Parquet files.

encryption: Option<EncryptionSpec>

Available on crate feature encryption only.

Decrypt files sealed by the file or jsonl sink’s encryption block (AES-256-GCM). An encrypted file is read into memory whole: JSON Lines and raw text written line by line are then opened one line at a time, every other file as one sealed body. A file that is not sealed fails the read.

Implementations

impl FileSourceConfig

pub fn new(path: impl Into<String>) -> Self

A config reading path with every default.

pub fn format(self, format: FileSourceFormat) -> Self

Set the format.

pub fn recursive(self, recursive: bool) -> Self

Descend into subdirectories.

pub fn with_batch_size(self, batch_size: usize) -> Self

Set the records per page.

pub fn incremental(self, by: IncrementalBy) -> Self

Turn on incremental mode.

pub fn is_http(&self) -> bool

Whether path is an http(s):// URL.

pub fn parquet_columns<I, S>(self, columns: I) -> Self

where I: IntoIterator<Item = S>, S: Into<String>,

Project Parquet files to columns.

pub fn encryption_set(&self) -> bool

Whether an encryption block is set.

pub fn format_options(&self) -> FormatOptions

The per-format option blocks. CSV is strict about ragged rows unless csv.flexible says otherwise.

pub fn validate(&self) -> Result<(), FaucetError>

Load-time checks: batch size bounds, a non-empty path, a parseable Avro reader schema, and header names/values HTTP can carry.

Trait Implementations

impl Clone for FileSourceConfig

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for FileSourceConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for FileSourceConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl JsonSchema for FileSourceConfig

fn schema_name() -> Cow<'static, str>

The name of the generated JSON Schema.

fn schema_id() -> Cow<'static, str>

Returns a string that uniquely identifies the schema produced by this type.

fn json_schema(generator: &mut SchemaGenerator) -> Schema

Generates a JSON Schema for this type.

fn inline_schema() -> bool

Whether JSON Schemas generated for this type should be included directly in parent schemas, rather than being re-used where possible using the $ref keyword.

impl Serialize for FileSourceConfig

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.