Struct DefaultJsonHandler 

pub struct DefaultJsonHandler<E: TaskExecutor> { /* private fields */ }

Available on (crate features default-engine-native-tls or default-engine-rustls or arrow-conversion) and crate feature default-engine-base only.

Implementations

impl<E: TaskExecutor> DefaultJsonHandler<E>

pub fn new(store: Arc<DynObjectStore>, task_executor: Arc<E>) -> Self

pub fn with_buffer_size(self, buffer_size: usize) -> Self

Set the maximum number read requests to buffer in memory at once in Self::read_json_files().

Defaults to 1000.

Memory constraints can be imposed by constraining the buffer size and batch size. Note that overall memory usage is proportional to the product of these two values.

1. Batch size governs the size of RecordBatches yielded in each iteration of the stream
2. Buffer size governs the number of concurrent tasks (which equals the size of the buffer

pub fn with_batch_size(self, batch_size: usize) -> Self

Limit the number of rows per batch. That is, for batch_size = N, then each RecordBatch yielded by the stream will have at most N rows.

Defaults to 1000 rows (json objects).

See Decoder::with_buffer_size for details on constraining memory usage with buffer size and batch size.

Trait Implementations

impl<E: Debug + TaskExecutor> Debug for DefaultJsonHandler<E>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<E: TaskExecutor> JsonHandler for DefaultJsonHandler<E>

fn parse_json( &self, json_strings: Box<dyn EngineData>, output_schema: SchemaRef, ) -> DeltaResult<Box<dyn EngineData>>

Parse the given json strings and return the fields requested by output schema as columns in EngineData. json_strings MUST be a single column batch of engine data, and the column type must be string

fn read_json_files( &self, files: &[FileMeta], physical_schema: SchemaRef, predicate: Option<PredicateRef>, ) -> DeltaResult<FileDataReadResultIterator>

Read and parse the JSON format file at given locations and return the data as EngineData with the columns requested by physical schema. Note: The FileDataReadResultIterator must emit data from files in the order that files is given. For example if files [“a”, “b”] is provided, then the engine data iterator must first return all the engine data from file “a”, then all the engine data from file “b”. Moreover, for a given file, all of its EngineData and constituent rows must be in order that they occur in the file. Consider a file with rows (1, 2, 3). The following are legal iterator batches: iter: [EngineData(1, 2), EngineData(3)] iter: [EngineData(1), EngineData(2, 3)] iter: [EngineData(1, 2, 3)] The following are illegal batches: iter: [EngineData(3), EngineData(1, 2)] iter: [EngineData(1), EngineData(3, 2)] iter: [EngineData(2, 1, 3)]

fn write_json_file( &self, path: &Url, data: Box<dyn Iterator<Item = DeltaResult<FilteredEngineData>> + Send + '_>, overwrite: bool, ) -> DeltaResult<()>

Atomically (!) write a single JSON file. Each row of the input data should be written as a new JSON object appended to the file. this write must: (1) serialize the data to newline-delimited json (each row is a json object literal) (2) write the data to storage atomically (i.e. if the file already exists, fail unless the overwrite flag is set)