iceberg-rs 0.1.0

An implementation of Apache Iceberg in Rust
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113

# Iceberg-rs

 **A Rust implementation of Iceberg.**

---
The [Iceberg](https://iceberg.apache.org/) is an open table format for analytic datasets. It is very convenient to access the Iceberg table information to do things like view schema details, track changes, and view snapshots. This project is a Rust implementation of the [Iceberg spec](https://iceberg.apache.org/spec/).

## Iceberg-rs in action

```rust
use iceberg_rs::model::table::TableMetadataV2;

fn main() {
   let data = r#"
       {
           "format-version" : 2,
           "table-uuid": "fb072c92-a02b-11e9-ae9c-1bb7bc9eca94",
           "location": "s3://b/wh/data.db/table",
           "last-sequence-number" : 1,
           "last-updated-ms": 1515100955770,
           "last-column-id": 1,
           "schemas": [
               {
                   "schema-id" : 1,
                   "type" : "struct",
                   "fields" :[
                       {
                           "id": 1,
                           "name": "struct_name",
                           "required": true,
                           "field_type": "fixed[1]"
                       }
                   ]
               }
           ],
           "current-schema-id" : 1,
           "partition-specs": [
               {
                   "spec-id": 1,
                   "fields": [
                       {  
                           "source-id": 4,  
                           "field-id": 1000,  
                           "name": "ts_day",  
                           "transform": "day"
                       } 
                   ]
               }
           ],
           "default-spec-id": 1,
           "last-partition-id": 1,
           "properties": {
               "commit.retry.num-retries": "1"
           },
           "metadata-log": [
               {  
                   "metadata-file": "s3://bucket/.../v1.json",  
                   "timestamp-ms": 1515100
               }
           ],
           "sort-orders": [],
           "default-sort-order-id": 0
       }
   "#;
    let metadata = serde_json::from_str::<TableMetadataV2>(&data).unwrap();
 
}
```

## Features

Currently supported:
* Parsing table metadata v2.

Coming soon:
* Manifest files. 
* Manifest lists.
* v1 table metadata support.
* Validation.

## Iceberg Model

Diagram showing the main `nouns` of the Iceberg model.
```mermaid
erDiagram
    TABLE ||--o{ PARTITION_SPEC : partitions
    PARTITION_SPEC ||--o{ PARTITION_FIELD : fields
    PARTITION_FIELD ||--|| TRANSFORM : transformed
    TABLE ||--o{ SNAPSHOT : snapshots
    TABLE ||--|{ SCHEMA : schemas
    SCHEMA ||--o{ STRUCT_FIELD : fields
    TABLE ||--o{ SORT_ORDER : fields
    SORT_ORDER ||--o{ SORT_FIELD : fields
```

## Contributing

If you’d like to contribute, please fork the repository and use a feature branch. We warmly welcomed pull requests.

Please validate all submission with `make validate`.

#### License

<sup>
Licensed under either of <a href="LICENSE-APACHE">Apache License, Version
2.0</a> or <a href="LICENSE-MIT">MIT license</a> at your option.
</sup>

<br>

<sub>
Unless you explicitly state otherwise, any contribution intentionally submitted to include in Iceberg-rs by you, as defined in the Apache-2.0 license, shall be dual licensed as above, with no additional terms or conditions.
</sub>