Struct Schema 

pub struct Schema {
    pub fields: Vec<Field>,
    pub metadata: HashMap<String, String>,
}

Lance Schema.

Fields

fields: Vec<Field>

Top-level fields in the dataset.

metadata: HashMap<String, String>

Metadata of the schema

Implementations

impl Schema

pub fn unenforced_primary_key(&self) -> Vec<&Field>

The unenforced primary key fields in the schema, ordered by position.

Fields with explicit positions (1, 2, 3, …) are ordered by their position value. Fields without explicit positions (using the legacy boolean flag) are ordered by their schema field id and come after fields with explicit positions.

pub fn compare_with_options( &self, expected: &Schema, options: &SchemaCompareOptions, ) -> bool

pub fn explain_difference( &self, expected: &Schema, options: &SchemaCompareOptions, ) -> Option<String>

pub fn has_dictionary_types(&self) -> bool

pub fn check_compatible( &self, expected: &Schema, options: &SchemaCompareOptions, ) -> Result<(), Error>

pub fn to_compact_string(&self, indent: Indentation) -> String

Convert to a compact string representation.

This is intended for display purposes and not for serialization.

pub fn resolve(&self, column: impl AsRef<str>) -> Option<Vec<&Field>>

Given a string column reference, resolve the path of fields

For example, given a.b.c we will return the fields [a, b, c] Field names containing dots must be quoted: parent.“child.with.dot”

Returns None if we can’t find a segment at any point

pub fn project<T>(&self, columns: &[T]) -> Result<Schema, Error>

where T: AsRef<str>,

Project the columns over the schema.

let schema = Schema::from(...);
let projected = schema.project(&["col1", "col2.sub_col3.field4"])?;

pub fn project_or_drop<T>(&self, columns: &[T]) -> Result<Schema, Error>

where T: AsRef<str>,

Project the columns over the schema, dropping unrecognized columns

pub fn project_preserve_system_columns<T>( &self, columns: &[T], ) -> Result<Schema, Error>

where T: AsRef<str>,

Project the columns over the schema, preserving system columns.

pub fn validate(&self) -> Result<(), Error>

Check that the top level fields don’t contain . in their names to distinguish from nested fields.

pub fn intersection(&self, other: &Schema) -> Result<Schema, Error>

Intersection between two Schema.

pub fn intersection_ignore_types(&self, other: &Schema) -> Result<Schema, Error>

Intersection between two Schema, ignoring data types.

pub fn fields_pre_order(&self) -> impl Iterator<Item = &Field>

Iterates over the fields using a pre-order traversal

This is a DFS traversal where the parent is visited before its children

pub fn field_paths(&self) -> Vec<String>

Get all field paths in the schema as a list of strings.

This returns all field paths in the schema, including nested fields. For example, if there’s a struct field “user” with a field “name”, this will return “user.name” as one of the paths.

pub fn project_by_ids( &self, column_ids: &[i32], include_all_children: bool, ) -> Schema

Returns a new schema that only contains the fields in column_ids.

This projection can filter out both top-level and nested fields

If include_all_children is true, then if a parent field id is passed, then all children of that field will be included in the projection regardless of whether their ids were passed. If this is false, then only the child fields with the passed ids will be included.

pub fn project_by_schema<S>( &self, projection: S, on_missing: OnMissing, on_type_mismatch: OnTypeMismatch, ) -> Result<Schema, Error>

where S: TryInto<Schema, Error = Error>,

Project the schema by another schema, and preserves field metadata, i.e., Field IDs.

Parameters

• projection: The schema to project by. Can be arrow_schema::Schema or Schema.

pub fn exclude<T>(&self, schema: T) -> Result<Schema, Error>

where T: TryInto<Schema> + Debug,

Exclude the fields from other Schema, and returns a new Schema.

pub fn field(&self, name: &str) -> Option<&Field>

Get a field by its path. Return None if the field does not exist. Field names containing dots must be quoted: parent.“child.with.dot”

pub fn field_case_insensitive(&self, name: &str) -> Option<&Field>

Get a field by its path, with case-insensitive matching.

This first tries an exact match, then falls back to case-insensitive matching. Returns the actual field from the schema (preserving original case). Field names containing dots must be quoted: parent.“child.with.dot”

pub fn resolve_case_insensitive( &self, column: impl AsRef<str>, ) -> Option<Vec<&Field>>

Given a string column reference, resolve the path of fields with case-insensitive matching.

This first tries an exact match, then falls back to case-insensitive matching. Returns the actual fields from the schema (preserving original case).

pub fn field_id(&self, column: &str) -> Result<i32, Error>

pub fn top_level_field_ids(&self) -> Vec<i32>

pub fn field_ids(&self) -> Vec<i32>

pub fn field_by_id_mut(&mut self, id: impl Into<i32>) -> Option<&mut Field>

Get field by its id.

pub fn field_by_id(&self, id: impl Into<i32>) -> Option<&Field>

pub fn field_ancestry_by_id(&self, id: i32) -> Option<Vec<&Field>>

Get the sequence of fields from the root to the field with the given id.

pub fn mut_field_by_id(&mut self, id: impl Into<i32>) -> Option<&mut Field>

pub fn max_field_id(&self) -> Option<i32>

Get the maximum field id in the schema.

Note: When working with Datasets, you should prefer Manifest::max_field_id() over this method. This method does not take into account the field IDs of dropped fields.

pub fn set_dictionary(&mut self, batch: &RecordBatch) -> Result<(), Error>

Recursively attach set up dictionary values to the dictionary fields.

pub fn set_field_id(&mut self, max_existing_id: Option<i32>)

Walk through the fields and assign a new field id to each field that does not have one (e.g. is set to -1)

If this schema is on an existing dataset, pass the result of Manifest::max_field_id to max_existing_id. If for some reason that id is lower than the maximum field id in this schema, the field IDs will be reassigned starting from the maximum field id in this schema.

If this schema is not associated with a dataset, pass None to max_existing_id. This is the same as passing Self::max_field_id().

pub fn extend(&mut self, fields: &[Field]) -> Result<(), Error>

Create a new schema by adding fields to the end of this schema

pub fn merge<S>(&self, other: S) -> Result<Schema, Error>

where S: TryInto<Schema, Error = Error>,

Merge this schema from the other schema.

After merging, the field IDs from other schema will be reassigned, following the fields in self.

pub fn all_fields_nullable(&self) -> bool

pub fn field_path(&self, field_id: i32) -> Result<String, Error>

Returns the properly formatted path from root to the field. Field names containing dots are quoted (e.g., struct.field.with.dot)

pub fn verify_primary_key(&self) -> Result<(), Error>

Trait Implementations

impl Clone for Schema

fn clone(&self) -> Schema

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Schema

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl DeepSizeOf for Schema

fn deep_size_of_children(&self, context: &mut Context) -> usize

Returns an estimation of the heap-managed storage of this object. This does not include the size of the object itself.

fn deep_size_of(&self) -> usize

Returns an estimation of a total size of memory owned by the object, including heap-managed storage.

impl Default for Schema

fn default() -> Schema

Returns the “default value” for a type.

impl Display for Schema

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl From<&Fields> for Schema

Convert list of protobuf Field to a Schema.

fn from(fields: &Fields) -> Schema

Converts to this type from the input type.

impl From<FieldsWithMeta> for Schema

Convert list of protobuf Field and Metadata to a Schema.

fn from(fields_with_meta: FieldsWithMeta) -> Schema

Converts to this type from the input type.

impl From<Schema> for ProjectionRequest

fn from(schema: Schema) -> Self

Converts to this type from the input type.

impl PartialEq for Schema

fn eq(&self, other: &Schema) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Projectable for Schema

fn schema(&self) -> &Schema

impl TryFrom<&Schema> for Schema

Convert arrow2::datatype::Schema to Lance

type Error = Error

The type returned in the event of a conversion error.

fn try_from(schema: &Schema) -> Result<Schema, Error>

Performs the conversion.

impl TryFrom<&Schema> for Schema

Make API cleaner to accept both Schema and Arrow Schema.

type Error = Error

The type returned in the event of a conversion error.

fn try_from(schema: &Schema) -> Result<Schema, Error>

Performs the conversion.