1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
//! Language Server Protocol (LSP) implementation for Lex
//!
//! This crate provides language server capabilities for the Lex format, enabling rich editor
//! support in any LSP-compatible editor (VSCode, Neovim, Emacs, Sublime, etc.).
//!
//! Position Encoding — read before touching any `Position`
//!
//! In this server, `Position.character` is a **UTF-8 byte offset** into the line —
//! NOT a UTF-16 code unit (the LSP default) nor a `char` count. This is the
//! load-bearing convention every position↔text conversion here must honor.
//!
//! It originates upstream: lex-core's `SourceLocation::byte_to_position` computes
//! `column = byte_offset - line_start`, and `to_lsp_position` forwards that value to
//! LSP unchanged. Every consumer of a `Position` in this crate must therefore read
//! `.character` back as a byte offset to stay consistent.
//!
//! Practical consequence: to take the text up to a caret, slice on the byte offset
//! (`line.get(..pos.character as usize)`, which returns `None` rather than panicking
//! on an out-of-bounds or mid-`char` index) — never
//! `line.chars().take(pos.character as usize)`.
//! A `char`-based count silently over-reads past the caret on any line containing
//! multi-byte characters (this caused a real bug in #740). See `slice_text_by_range`
//! in `server.rs` for the canonical byte-offset slicing routine, including the
//! multi-byte-boundary guards.
//!
//! Caveat — not yet uniform: `indent_level_from_position` in `server.rs` still uses
//! the `chars().take()` form. It is currently harmless there (it only consumes leading
//! ASCII indentation, where byte and `char` counts coincide), but it does not yet
//! follow this convention and is a separate cleanup, not a reason to copy the pattern.
//!
//! Design Decision: tower-lsp
//!
//! After evaluating the Rust LSP ecosystem, we chose tower-lsp as our framework:
//!
//! Considered Options:
//! 1. tower-lsp: High-level async framework built on Tower (~204K monthly downloads)
//! 2. lsp-server: Low-level sync library from rust-analyzer (~309K monthly downloads)
//! 3. async-lsp: Low-level async with full Tower integration (~135 GitHub stars)
//!
//! Why tower-lsp:
//! - Best balance of ease-of-use and functionality for a new LSP project
//! - Strong ecosystem support with extensive documentation and examples
//! - Modern async/await patterns ideal for Lex's structured parsing needs
//! - Built-in LSP 3.18 support with proposed features
//! - Active community with production usage in many language servers
//! - Good integration with Rust async ecosystem (tokio, futures)
//!
//! Trade-offs:
//! - Less flexible than async-lsp for custom Tower layers (acceptable for our needs)
//! - Requires &self for trait methods, forcing Arc<Mutex<>> for mutable state (standard pattern)
//! - Notification ordering is async (not an issue for initial feature set)
//!
//! Future Migration Path:
//! If we later need precise notification ordering or custom middleware, we can migrate
//! to async-lsp with minimal disruption as both use similar async patterns.
//!
//! Feature Set
//!
//! Lex is a structured document format, not a programming language. LSP features are selected
//! to optimize document authoring and navigation workflows:
//!
//! Core Features:
//!
//! a. Syntax Highlighting
//! 1. Semantic Tokens (textDocument/semanticTokens/*):
//! - Syntax highlighting for sessions, lists, definitions, annotations
//! - Inline formatting: bold, italic, code, math
//! - References, footnotes, citations
//! - Verbatim blocks with language-specific highlighting
//! 2. Document Symbols (textDocument/documentSymbol):
//! - Hierarchical outline view of document structure
//! - Sessions with nesting (1., 1.1., 1.1.1., etc.)
//! - Definitions, annotations, lists as navigable symbols
//! 3. Hover Information (textDocument/hover):
//! - Preview footnote/citation content on hover
//! - Show annotation metadata
//! - Preview definition content when hovering over reference
//! 4. Folding Ranges (textDocument/foldingRange):
//! - Fold sessions and nested content
//! - Fold list items with children
//! - Fold annotations, definitions, verbatim blocks
//!
//! b. Navigation
//!
//! 5. Go to Definition / Find References (textDocument/definition, textDocument/references):
//! - Find all references to footnotes/citations
//! - Jump from footnote reference [42] to annotation
//! - Jump from citation [@spec2025] to bibliography entry
//! - Jump from internal reference [TK-rootlist] to target
//! 6. Document Links (textDocument/documentLink):
//! - Clickable links in text
//! - Verbatim block src parameters (images, includes)
//! - External references
//!
//! c. Editing
//!
//! 7. Document Formatting (textDocument/formatting, textDocument/rangeFormatting):
//! - Fix indentation issues
//! - Normalize blank lines
//! - Align list markers //!
//!
//!
//!
//! Architecture
//!
//! The server follows a layered architecture:
//!
//! LSP Layer (tower-lsp):
//! - Handles JSON-RPC communication
//! - Protocol handshaking and capability negotiation
//! - Request/response routing
//!
//! Server Layer (this crate):
//! - Implements LanguageServer trait
//! - Manages document state and parsing
//! - Coordinates feature implementations
//! - Very thing, mostly calls the the feature layers over lex-parser
//! - Thin tests just asserting the right things are being called and returned
//!
//! Feature Layer:
//! - Each feature operates on Lex AST
//! - Stateless transformations where possible
//! - All logic and dense unit tests
//!
//!
//! Testing Strategy
//!
//! Following Lex project conventions:
//! - Use official sample files from specs/ for all tests
//! - Use lexplore loader for consistent test data
//! - Use ast_assertions library for AST validation
//! - Test each feature in isolation and integration
//! - Test against kitchensink and trifecta fixtures
//!
//! Non-Features
//!
//! The following LSP features are intentionally excluded as they don't apply to document formats:
//! - Code Lens: Not applicable to documents
//! - Type Hierarchy: No type system
//! - Implementation: No interfaces/implementations
//! - Moniker: For cross-repo linking, not needed
//! - Linked Editing Range: For paired tags (HTML/XML)
//! - Diagnostics: we don't have a clear vision for how that would work.
//!
//! Error Handling and Robustness
//!
//! The server is designed to be highly robust and crash-resistant, following these principles:
//!
//! 1. No Panics:
//! - We strictly avoid `unwrap()` and `expect()` in production code paths.
//! - All potential failure points (parsing, serialization, IO) return `Result`.
//! - Errors are propagated up the stack and handled gracefully.
//!
//! 2. Graceful Degradation:
//! - If a feature fails (e.g., semantic tokens calculation), we log the error and return
//! an empty result or `None` rather than crashing the server.
//! - This ensures that a bug in one feature doesn't bring down the entire editor experience.
//!
//! 3. Error Propagation:
//! - The `lex-parser` crate returns `Result` types for all parsing operations.
//! - The `lexd-lsp` server maps these internal errors to appropriate LSP error codes
//! (e.g., `InternalError`, `InvalidRequest`) when communicating with the client.
//!
//! 4. Property-Based Testing:
//! - We use `proptest` to fuzz the server with random inputs (commands, document text)
//! to uncover edge cases and ensure stability under unexpected conditions.
//!
//! Usage
//!
//! This crate provides both a library and binary:
//!
//! Library:
//! ```rust
//! use lexd_lsp::LexLanguageServer;
//! use tower_lsp::Server;
//!
//! #[tokio::main]
//! async fn main() {
//! let stdin = tokio::io::stdin();
//! let stdout = tokio::io::stdout();
//!
//! let (service, socket) = LspService::new(|client| LexLanguageServer::new(client));
//! Server::new(stdin, stdout, socket).serve(service).await;
//! }
//! ```
//!
//! Binary:
//! $ lexd-lsp
//! Starts the language server on stdin/stdout for editor integration.
//!
pub use LexLanguageServer;