lxdb-format 0.1.0

Binary format records and sections for LXDB datasets.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103

# LXDB Binary Format Specification


## File Layout


Every LXDB file is composed of the following sections.

+----------------------+
| Header               |
+----------------------+
| Metadata             |
+----------------------+
| Token Records        |
+----------------------+
| Token String Table   |
+----------------------+
| Relation Records     |
+----------------------+
| Adjacency Index      |
+----------------------+

Each section is preceded by a `SectionHeader`.

Sections may appear in a different order in future versions.

Readers MUST use the section headers instead of assuming offsets.

---

## Header


Contains:

- Magic bytes
- Format version

---

## Metadata


Contains dataset information such as:

- Dataset ID
- Language
- Version
- Name

---

## Token records


Each token record occupies 24 bytes.

| Offset | Size | Field |
|---:|---:|---|
| 0 | 4 | Token ID |
| 4 | 4 | Reserved |
| 8 | 8 | String table offset |
| 16 | 4 | String length |
| 20 | 4 | Reserved |

The string table offset is relative to the beginning of the `TokenStringTable` section payload.

---

## Token String Table


A contiguous UTF-8 byte buffer.

Strings are never null terminated.

Each TokenRecord specifies:

- offset
- length

---

## Relation Records


Array of `RelationRecord`.

Relations are sorted by source token.

---

## Adjacency Index


Array of `AdjacencyRecord`.

One record per token.

Allows O(1) lookup of outgoing relations.

---

## Design Goals


- Zero-copy where possible
- Forward compatible
- Compact
- Sequential reads
- Immutable
- Language agnostic