lance 0.8.0

A columnar data format that is 100x faster than Parquet for random access.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140

# Developing Lance

Lance in one diagram

![img.png](img.png)

The write path is analogous but there's no need for io planning. Instead,
we need to version data correctly and create the right manifest data to
be written

## Data format spec

TODO: release data format spec and add link here

## Rust codebase guide

| Directory | Description                                  |
|-----------|----------------------------------------------|
| dataset   | dataset-level scanner and writer abstraction |
| encodings | base level file serde                        |
| format    | structs to match the format spec             |
| index     | vector indices right now                     |
| io        | readers, writers, and IO planning            |
| utils     | distance metrics and testing utilities       |
| arrow     | extensions for arrow-rs                      |
| bin       | lq utility cli                               |

## Rust Roadmap

### Predicate support

Support for predicate pushdown needs to be ported over from C++

### NA support

For fixed stride data (plain encoder) we currently do not handle NA's.

### Versioning

Appends, overwrites, and adding columns need to be ported from C++.
Deletes and updates need to be implemented.

### Type support

See `Arrow types` section below for what's missing. 

### Non-python duckdb reader

Integrate DuckDB-rs and provide TableFunction for lance with replacement scan

## Python

1. Need to expose things like row count from Rust
2. Need to support `filter=` for predicate pushdown (and how to pass filters to Rust)
3. Convenience functions for checkout and tags to better support experimentation

## Development

### Build

1. Rust

Under /rust directory, run `cargo build` or `cargo build --release` for optimized package.

2. Python

Under /rust/pylance directory, run `maturin develop` or `maturin develop --release`.

### Testing

1. Rust

Under /rust, run `cargo test`

2. Python

Under /rust/pylance, run `pytest python/tests`

### Encodings

|                   | Read | Write | Null |
|-------------------|------|-------|------|
| Plain             | Yes  | Yes   | No   |
| Var-length Binary | Yes  | Yes   | Yes  |
| Dictionary        | Yes  | Yes   | No   |
| RLE               | No   | No    | No   |


### Arrow types

|                         | Read | Write | Null |
|-------------------------|------|-------|------|
| NA                      | NO   | NO    | NO   |
| Bool                    | Yes  | Yes   | No   |
| UINT8                   | Yes  | Yes   | No   |
| INT8                    | Yes  | Yes   | No   |
| UINT16                  | Yes  | Yes   | No   |
| INT16                   | Yes  | Yes   | No   |
| UINT32                  | Yes  | Yes   | No   |
| INT32                   | Yes  | Yes   | No   |
| UINT64                  | Yes  | Yes   | No   |
| INT64                   | Yes  | Yes   | No   |
| HALF_FLOAT              | Yes  | Yes   | No   |
| FLOAT                   | Yes  | Yes   | No   |
| DOUBLE                  | Yes  | Yes   | No   |
| DECIMAL                 | No   | No    | No   |
| DECIMAL128              | No   | No    | No   |
| DECIMAL256              | No   | No    | No   |
| STRING                  | Yes  | Yes   | Yes  |
| LARGE_STRING            | Yes  | Yes   | Yes  |
| BINARY                  | Yes  | Yes   | Yes  |
| LARGE_BINARY            | Yes  | Yes   | Yes  |
| FIXED_SIZE_LIST         | Yes  | Yes   | No   |
| FIXED_SIZE_BINARY       | Yes  | Yes   | No   |
| DATE32                  | Yes  | Yes   | No   |
| DATE64                  | Yes  | Yes   | No   |
| TIMESTAMP               | Yes  | Yes   | No   |
| TIME32                  | Yes  | Yes   | No   |
| TIME64                  | Yes  | Yes   | No   |
| INTERVAL_MONTHS         | No   | No    | No   |
| INTERVAL_DAY_TIME       | No   | No    | No   |
| INTERVAL_MONTH_DAY_NANO | No   | No    | No   |
| DURATION                | No   | No    | No   |
| LIST                    | Yes  | Yes   | No   |
| LARGE_LIST              | Yes  | Yes   | No   |
| STRUCT                  | Yes  | Yes   | No   |
| DICTIONARY              | Yes  | Yes   | No   |
| MAP                     | No   | No    | No   |
| DENSE_UNION             | No   | No    | No   |
| SPARSE_UNION            | No   | No    | No   |


## Extension types

We want to add extension types for ML:
- Images, Videos, PDF, Document
- Boxes, Labels, Polygon, etc

Currently arrow-rs doesn't have an extension mechanism nor does it talk to the
C++ extension registry.