litedoc-core 0.1.0

Deterministic document parser for machine-generated output with error recovery
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144

# LiteDoc

[![CI](https://github.com/nssalian/litedoc/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/nssalian/litedoc/actions/workflows/ci.yml)
[![crates.io](https://img.shields.io/crates/v/litedoc-core.svg)](https://crates.io/crates/litedoc-core)
[![docs.rs](https://docs.rs/litedoc-core/badge.svg)](https://docs.rs/litedoc-core)

Deterministic document format for AI agents and LLM output. Explicit block fencing, zero-copy parsing, and error recovery.

## Why LiteDoc?

Markdown is ambiguous. Indentation rules vary, edge cases abound, and parsers disagree.[1] LiteDoc uses explicit `::block` fencing for deterministic parsing that recovers gracefully from malformed input, which is ideal for machine-generated content in LLM pipelines where output parsing can fail when formatting is off.[2]

## Status

v0.1.0 - Initial release with Rust parser and CLI. APIs are intended to be stable within v0.1 but may evolve.

## Stability & Compatibility

LiteDoc follows semantic versioning. For the v0.1 line, we will not make breaking
changes to the core format or public APIs without a version bump and a migration
note in the changelog.

| Document | Description |
|----------|-------------|
| [LITEDOC_SPEC.md](LITEDOC_SPEC.md) | Language specification |
| [LITEDOC_AST.md](LITEDOC_AST.md) | AST reference |

## Performance

| Metric | LiteDoc | Markdown | Improvement |
|--------|---------|----------|-------------|
| Parse speed | 5.451 µs | 5.660 µs | 3.7% faster |
| Inline parsing | 490 ns | 1.313 µs | 63% faster |
| Error recovery | 0.89 | 0.67 | 33% better |

## Install

```bash
cargo add litedoc-core            # Rust library
pip install litedoc-py                  # Python library
cargo install litedoc-cli          # CLI tool
```

## Usage

### Rust

```rust
use litedoc_core::{Block, Parser, Profile};

let mut parser = Parser::new(Profile::Litedoc);
let result = parser.parse_with_recovery(input);

for block in &result.document.blocks {
    match block {
        Block::List(list) => process_list(list),
        Block::Table(table) => process_table(table),
        _ => {}
    }
}
```

### Python

```python
import pyld

doc = pyld.parse("# Hello\n\nWorld")
for block in doc.blocks:
    match block:
        case pyld.Heading(level=level):
            print(f"H{level}")
        case pyld.Paragraph(content=content):
            print(content)
```

## CLI

```bash
ldcli agent_output.ld            # Parse and display structure
ldcli -j agent_output.ld         # Output as JSON
ldcli validate agent_output.ld   # Check for errors
ldcli stats agent_output.ld      # Show statistics
```

## Format

```text
::list
- First item
- Second item
::

::quote
Quoted text
::

::table
| A | B |
|---|---|
| 1 | 2 |
::
```

Metadata:

```text
--- meta ---
agent: summarizer-v2
task_id: abc123
timestamp: 1704067200
confidence: 0.92
tags: [summary, final]
---
```

## Benchmarks

```bash
cargo bench -p litedoc-core
```

```bash
cargo test -p litedoc-core robustness_report -- --nocapture
```

CSV output:

```bash
ROBUSTNESS_CSV=1 cargo test -p litedoc-core robustness_report -- --nocapture
ROBUSTNESS_BENCH_CSV=1 cargo bench -p litedoc-core robustness_benchmark -- --nocapture
```

## References

[1] CommonMark Spec, “Why is a spec needed?” (notes original Markdown syntax is not unambiguous and implementations diverged).
https://spec.commonmark.org/0.31.2/

[2] LangChain docs: OUTPUT_PARSING_FAILURE (example of JSON-in-Markdown parsing failures).
https://docs.langchain.com/oss/python/langchain/errors/OUTPUT_PARSING_FAILURE

## License

Apache-2.0