pub enum DataType {
Show 33 variants Null, Boolean, Int8, Int16, Int32, Int64, UInt8, UInt16, UInt32, UInt64, Float16, Float32, Float64, Timestamp(TimeUnitOption<String>), Date32, Date64, Time32(TimeUnit), Time64(TimeUnit), Duration(TimeUnit), Interval(IntervalUnit), Binary, FixedSizeBinary(i32), LargeBinary, Utf8, LargeUtf8, List(Box<Field>), FixedSizeList(Box<Field>, i32), LargeList(Box<Field>), Struct(Vec<Field>), Union(Vec<Field>, Vec<i8>, UnionMode), Dictionary(Box<DataType>, Box<DataType>), Decimal(usizeusize), Map(Box<Field>, bool),
}
Expand description

The set of datatypes that are supported by this implementation of Apache Arrow.

The Arrow specification on data types includes some more types. See also Schema.fbs for Arrow’s specification.

The variants of this enum include primitive fixed size types as well as parametric or nested types. Currently the Rust implementation supports the following nested types:

  • List<T>
  • Struct<T, U, V, ...>

Nested types can themselves be nested within other arrays. For more information on these types please see the physical memory layout of Apache Arrow.

Variants

Null

Null type

Boolean

A boolean datatype representing the values true and false.

Int8

A signed 8-bit integer.

Int16

A signed 16-bit integer.

Int32

A signed 32-bit integer.

Int64

A signed 64-bit integer.

UInt8

An unsigned 8-bit integer.

UInt16

An unsigned 16-bit integer.

UInt32

An unsigned 32-bit integer.

UInt64

An unsigned 64-bit integer.

Float16

A 16-bit floating point number.

Float32

A 32-bit floating point number.

Float64

A 64-bit floating point number.

Timestamp(TimeUnitOption<String>)

A timestamp with an optional timezone.

Time is measured as a Unix epoch, counting the seconds from 00:00:00.000 on 1 January 1970, excluding leap seconds, as a 64-bit integer.

The time zone is a string indicating the name of a time zone, one of:

  • As used in the Olson time zone database (the “tz database” or “tzdata”), such as “America/New_York”
  • An absolute time zone offset of the form +XX:XX or -XX:XX, such as +07:30
Timestamps with a non-empty timezone

If a Timestamp column has a non-empty timezone value, its epoch is 1970-01-01 00:00:00 (January 1st 1970, midnight) in the UTC timezone (the Unix epoch), regardless of the Timestamp’s own timezone.

Therefore, timestamp values with a non-empty timezone correspond to physical points in time together with some additional information about how the data was obtained and/or how to display it (the timezone).

For example, the timestamp value 0 with the timezone string “Europe/Paris” corresponds to “January 1st 1970, 00h00” in the UTC timezone, but the application may prefer to display it as “January 1st 1970, 01h00” in the Europe/Paris timezone (which is the same physical point in time).

One consequence is that timestamp values with a non-empty timezone can be compared and ordered directly, since they all share the same well-known point of reference (the Unix epoch).

Timestamps with an unset / empty timezone

If a Timestamp column has no timezone value, its epoch is 1970-01-01 00:00:00 (January 1st 1970, midnight) in an unknown timezone.

Therefore, timestamp values without a timezone cannot be meaningfully interpreted as physical points in time, but only as calendar / clock indications (“wall clock time”) in an unspecified timezone.

For example, the timestamp value 0 with an empty timezone string corresponds to “January 1st 1970, 00h00” in an unknown timezone: there is not enough information to interpret it as a well-defined physical point in time.

One consequence is that timestamp values without a timezone cannot be reliably compared or ordered, since they may have different points of reference. In particular, it is not possible to interpret an unset or empty timezone as the same as “UTC”.

Conversion between timezones

If a Timestamp column has a non-empty timezone, changing the timezone to a different non-empty value is a metadata-only operation: the timestamp values need not change as their point of reference remains the same (the Unix epoch).

However, if a Timestamp column has no timezone value, changing it to a non-empty value requires to think about the desired semantics. One possibility is to assume that the original timestamp values are relative to the epoch of the timezone being set; timestamp values should then adjusted to the Unix epoch (for example, changing the timezone from empty to “Europe/Paris” would require converting the timestamp values from “Europe/Paris” to “UTC”, which seems counter-intuitive but is nevertheless correct).

Date32

A 32-bit date representing the elapsed time since UNIX epoch (1970-01-01) in days (32 bits).

Date64

A 64-bit date representing the elapsed time since UNIX epoch (1970-01-01) in milliseconds (64 bits). Values are evenly divisible by 86400000.

Time32(TimeUnit)

A 32-bit time representing the elapsed time since midnight in the unit of TimeUnit.

Time64(TimeUnit)

A 64-bit time representing the elapsed time since midnight in the unit of TimeUnit.

Duration(TimeUnit)

Measure of elapsed time in either seconds, milliseconds, microseconds or nanoseconds.

Interval(IntervalUnit)

A “calendar” interval which models types that don’t necessarily have a precise duration without the context of a base timestamp (e.g. days can differ in length during day light savings time transitions).

Binary

Opaque binary data of variable length.

FixedSizeBinary(i32)

Opaque binary data of fixed size. Enum parameter specifies the number of bytes per value.

LargeBinary

Opaque binary data of variable length and 64-bit offsets.

Utf8

A variable-length string in Unicode with UTF-8 encoding.

LargeUtf8

A variable-length string in Unicode with UFT-8 encoding and 64-bit offsets.

List(Box<Field>)

A list of some logical data type with variable length.

FixedSizeList(Box<Field>, i32)

A list of some logical data type with fixed length.

LargeList(Box<Field>)

A list of some logical data type with variable length and 64-bit offsets.

Struct(Vec<Field>)

A nested datatype that contains a number of sub-fields.

Union(Vec<Field>, Vec<i8>, UnionMode)

A nested datatype that can represent slots of differing types. Components:

  1. Field for each possible child type the Union can hold
  2. The corresponding type_id used to identify which Field
  3. The type of union (Sparse or Dense)

Dictionary(Box<DataType>, Box<DataType>)

A dictionary encoded array (key_type, value_type), where each array element is an index of key_type into an associated dictionary of value_type.

Dictionary arrays are used to store columns of value_type that contain many repeated values using less memory, but with a higher CPU overhead for some operations.

This type mostly used to represent low cardinality string arrays or a limited set of primitive types as integers.

Decimal(usizeusize)

Exact decimal value with precision and scale

  • precision is the total number of digits
  • scale is the number of digits past the decimal

For example the number 123.45 has precision 5 and scale 2.

Map(Box<Field>, bool)

A Map is a logical nested type that is represented as

List<entries: Struct<key: K, value: V>>

The keys and values are each respectively contiguous. The key and value types are not constrained, but keys should be hashable and unique. Whether the keys are sorted can be set in the bool after the Field.

In a field with Map type, the field has a child Struct field, which then has two children: key type and the second the value type. The names of the child fields may be respectively “entries”, “key”, and “value”, but this is not enforced.

Implementations

Generate a JSON representation of the data type.

Returns true if this type is numeric: (UInt*, Unit*, or Float*).

Returns true if this type is valid as a dictionary key (e.g. super::ArrowDictionaryKeyType

Compares the datatype with another, ignoring nested field names and metadata.

Trait Implementations

Returns a copy of the value. Read more

Performs copy-assignment from source. Read more

Formats the value using the given formatter. Read more

Deserialize this value from the given Serde deserializer. Read more

Formats the value using the given formatter. Read more

Feeds this value into the given Hasher. Read more

Feeds a slice of this type into the given Hasher. Read more

This method returns an Ordering between self and other. Read more

Compares and returns the maximum of two values. Read more

Compares and returns the minimum of two values. Read more

Restrict a value to a certain interval. Read more

This method tests for self and other values to be equal, and is used by ==. Read more

This method tests for !=.

This method returns an ordering between self and other values if one exists. Read more

This method tests less than (for self and other) and is used by the < operator. Read more

This method tests less than or equal to (for self and other) and is used by the <= operator. Read more

This method tests greater than (for self and other) and is used by the > operator. Read more

This method tests greater than or equal to (for self and other) and is used by the >= operator. Read more

Serialize this value into the given Serde serializer. Read more

The type returned in the event of a conversion error.

The type returned in the event of a conversion error.

The type returned in the event of a conversion error.

Performs the conversion.

Auto Trait Implementations

Blanket Implementations

Gets the TypeId of self. Read more

Immutably borrows from an owned value. Read more

Mutably borrows from an owned value. Read more

Compare self to key and return true if they are equal.

Returns the argument unchanged.

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

The resulting type after obtaining ownership.

Creates owned data from borrowed data, usually by cloning. Read more

Uses borrowed data to replace owned data, usually by cloning. Read more

Converts the given value to a String. Read more

The type returned in the event of a conversion error.

Performs the conversion.

The type returned in the event of a conversion error.

Performs the conversion.