Struct JsonFormat 

pub struct JsonFormat { /* private fields */ }

JSON FileFormat implementation supporting both line-delimited and array formats.

Supported Formats

Line-Delimited JSON (default, newline_delimited = true)

{"key1": 1, "key2": "val"}
{"key1": 2, "key2": "vals"}

JSON Array Format (newline_delimited = false)

[
    {"key1": 1, "key2": "val"},
    {"key1": 2, "key2": "vals"}
]

Note: JSON array format is processed using streaming conversion, which is memory-efficient even for large files.

Implementations

impl JsonFormat

pub fn with_options(self, options: JsonOptions) -> Self

Set JSON options

pub fn options(&self) -> &JsonOptions

Retrieve JSON options

pub fn with_schema_infer_max_rec(self, max_rec: usize) -> Self

Set a limit in terms of records to scan to infer the schema

• defaults to DEFAULT_SCHEMA_INFER_MAX_RECORD

pub fn with_file_compression_type( self, file_compression_type: FileCompressionType, ) -> Self

Set a FileCompressionType of JSON

• defaults to FileCompressionType::UNCOMPRESSED

pub fn with_newline_delimited(self, newline_delimited: bool) -> Self

Set whether to read as newline-delimited JSON (NDJSON).

When true (default), expects newline-delimited format:

{"a": 1}
{"a": 2}

When false, expects JSON array format:

[{"a": 1}, {"a": 2}]

pub fn is_newline_delimited(&self) -> bool

Returns whether this format expects newline-delimited JSON.

Trait Implementations

impl Debug for JsonFormat

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for JsonFormat

fn default() -> JsonFormat

Returns the “default value” for a type.

impl FileFormat for JsonFormat

fn as_any(&self) -> &dyn Any

Returns the table provider as Any so that it can be downcast to a specific implementation.

fn get_ext(&self) -> String

Returns the extension for this FileFormat, e.g. “file.csv” -> csv

fn get_ext_with_compression( &self, file_compression_type: &FileCompressionType, ) -> Result<String>

Returns the extension for this FileFormat when compressed, e.g. “file.csv.gz” -> csv

fn compression_type(&self) -> Option<FileCompressionType>

Returns whether this instance uses compression if applicable

fn infer_schema<'life0, 'life1, 'life2, 'life3, 'async_trait>( &'life0 self, _state: &'life1 dyn Session, store: &'life2 Arc<dyn ObjectStore>, objects: &'life3 [ObjectMeta], ) -> Pin<Box<dyn Future<Output = Result<SchemaRef>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait, 'life3: 'async_trait,

Infer the common schema of the provided objects. The objects will usually be analysed up to a given number of records or files (as specified in the format config) then give the estimated common schema. This might fail if the files have schemas that cannot be merged.

fn infer_stats<'life0, 'life1, 'life2, 'life3, 'async_trait>( &'life0 self, _state: &'life1 dyn Session, _store: &'life2 Arc<dyn ObjectStore>, table_schema: SchemaRef, _object: &'life3 ObjectMeta, ) -> Pin<Box<dyn Future<Output = Result<Statistics>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait, 'life3: 'async_trait,

Infer the statistics for the provided object. The cost and accuracy of the estimated statistics might vary greatly between file formats.

fn create_physical_plan<'life0, 'life1, 'async_trait>( &'life0 self, _state: &'life1 dyn Session, conf: FileScanConfig, ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn ExecutionPlan>>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Take a list of files and convert it to the appropriate executor according to this file format.

fn create_writer_physical_plan<'life0, 'life1, 'async_trait>( &'life0 self, input: Arc<dyn ExecutionPlan>, _state: &'life1 dyn Session, conf: FileSinkConfig, order_requirements: Option<LexRequirement>, ) -> Pin<Box<dyn Future<Output = Result<Arc<dyn ExecutionPlan>>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Take a list of files and the configuration to convert it to the appropriate writer executor according to this file format.

fn file_source(&self, table_schema: TableSchema) -> Arc<dyn FileSource>

Return the related FileSource such as CsvSource, JsonSource, etc.

fn infer_ordering<'life0, 'life1, 'life2, 'life3, 'async_trait>( &'life0 self, _state: &'life1 dyn Session, _store: &'life2 Arc<dyn ObjectStore>, _table_schema: Arc<Schema>, _object: &'life3 ObjectMeta, ) -> Pin<Box<dyn Future<Output = Result<Option<LexOrdering>, DataFusionError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait, 'life3: 'async_trait, Self: 'async_trait,

Infer the ordering (sort order) for the provided object from file metadata.

fn infer_stats_and_ordering<'life0, 'life1, 'life2, 'life3, 'async_trait>( &'life0 self, state: &'life1 dyn Session, store: &'life2 Arc<dyn ObjectStore>, table_schema: Arc<Schema>, object: &'life3 ObjectMeta, ) -> Pin<Box<dyn Future<Output = Result<FileMeta, DataFusionError>> + Send + 'async_trait>>

where 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait, 'life3: 'async_trait, Self: 'async_trait,

Infer both statistics and ordering from a single metadata read.