lsp-bench 0.2.0

Benchmark framework for Language Server Protocol (LSP) servers
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124

# Solidity LSP Benchmark Analysis

Analysis of `v4-core` (`src/libraries/Pool.sol`) — 10 iterations per benchmark.

## Servers

| Server | Description | Version |
|--------|-------------|---------|
| [mmsaki](https://github.com/mmsaki/solidity-language-server) | Solidity Language Server by mmsaki | `solidity-language-server 0.1.14+commit.3d6a3d1.macos.aarch64` |
| [solc](https://docs.soliditylang.org) | Official Solidity compiler LSP | `0.8.33+commit.64118f21.Darwin.appleclang` |
| [nomicfoundation](https://github.com/NomicFoundation/hardhat-vscode) | Hardhat/Nomic Foundation Solidity Language Server | `@nomicfoundation/solidity-language-server 0.8.25` |
| [juanfranblanco](https://github.com/juanfranblanco/vscode-solidity) | VSCode Solidity by Juan Blanco | `vscode-solidity-server 0.0.187` |
| [qiuxiang](https://github.com/qiuxiang/solidity-ls) | Solidity Language Server by qiuxiang | `solidity-ls 0.5.4` |

## Capability Matrix

| Benchmark | mmsaki | solc | nomicfoundation | juanfranblanco | qiuxiang |
|-----------|--------|------|-----------------|----------------|----------|
| initialize | ok | ok | ok | ok | ok |
| textDocument/diagnostic | ok | ok | timeout | crash | timeout |
| textDocument/definition | ok | empty | timeout | crash | timeout |
| textDocument/declaration | ok | no | timeout | crash | timeout |
| textDocument/hover | ok | empty | timeout | crash | timeout |
| textDocument/references | ok | no | timeout | crash | timeout |
| textDocument/documentSymbol | ok | no | timeout | crash | timeout |
| textDocument/documentLink | ok | no | timeout | crash | timeout |

| Server | Working | Failed | Success Rate |
|--------|---------|--------|--------------|
| mmsaki | 8/8 | 0/8 | 100% |
| solc | 2/8 | 6/8 | 25% |
| nomicfoundation | 1/8 | 7/8 | 12% |
| juanfranblanco | 1/8 | 7/8 | 12% |
| qiuxiang | 1/8 | 7/8 | 12% |

## initialize

| Server | Status | Mean | p50 | p95 | Spread | Spike | Min | Max | Range | Overhead | vs mmsaki |
|--------|--------|------|-----|-----|--------|-------|-----|-----|-------|----------|-----------|
| mmsaki | ok | 4.13ms | 4.2ms | 4.6ms | 0.4ms | 1.09x | 3.77ms | 4.59ms | 0.82ms | **1.0x (fastest)** | - |
| solc | ok | 116.04ms | 116.5ms | 117.8ms | 1.3ms | 1.01x | 114.28ms | 117.81ms | 3.53ms | **28.1x** | **28.1x slower** |
| nomicfoundation | ok | 882.34ms | 877.4ms | 902.3ms | **25.0ms** | 1.03x | 869.44ms | 902.32ms | **32.88ms** | **213.6x** | **213.6x slower** |
| juanfranblanco | ok | 524.35ms | 524.8ms | 526.9ms | 2.1ms | 1.00x | 520.96ms | 526.85ms | 5.89ms | **127.0x** | **127.0x slower** |
| qiuxiang | ok | 70.60ms | 71.0ms | 71.9ms | 1.0ms | 1.01x | 69.11ms | 71.93ms | 2.82ms | **17.1x** | **17.1x slower** |

## textDocument/diagnostic

| Server | Status | Mem | Mean | p50 | p95 | Spread | Spike | Min | Max | Range | Overhead | vs mmsaki |
|--------|--------|-----|------|-----|-----|--------|-------|-----|-----|-------|----------|-----------|
| mmsaki | ok | 39.7 MB | 454.45ms | 452.1ms | 473.6ms | **21.5ms** | 1.05x | 449.46ms | 473.56ms | **24.10ms** | 3.3x | - |
| solc | ok | 26.2 MB | 136.80ms | 136.7ms | 138.3ms | 1.6ms | 1.01x | 135.92ms | 138.35ms | 2.43ms | **1.0x (fastest)** | 3.3x faster |
| nomicfoundation | timeout | 509.9 MB | - | - | - | - | - | - | - | - | - | timeout |
| juanfranblanco | crash | 0.0 MB | - | - | - | - | - | - | - | - | - | crash |
| qiuxiang | timeout | 69.7 MB | - | - | - | - | - | - | - | - | - | timeout |

## textDocument/definition

| Server | Status | Mem | Mean | p50 | p95 | Spread | Spike | Min | Max | Range | vs mmsaki |
|--------|--------|-----|------|-----|-----|--------|-------|-----|-----|-------|-----------|
| mmsaki | ok | 26.4 MB | 8.95ms | 9.2ms | 9.5ms | 0.3ms | 1.04x | 8.51ms | 9.52ms | 1.01ms | - |
| solc | empty | 26.2 MB | - | - | - | - | - | - | - | - | empty |
| nomicfoundation | timeout | 509.7 MB | - | - | - | - | - | - | - | - | timeout |
| juanfranblanco | crash | 0.0 MB | - | - | - | - | - | - | - | - | crash |
| qiuxiang | timeout | 68.7 MB | - | - | - | - | - | - | - | - | timeout |

## textDocument/declaration

| Server | Status | Mem | Mean | p50 | p95 | Spread | Spike | Min | Max | Range | vs mmsaki |
|--------|--------|-----|------|-----|-----|--------|-------|-----|-----|-------|-----------|
| mmsaki | ok | 35.7 MB | 9.04ms | 8.9ms | 9.9ms | 1.0ms | 1.11x | 8.58ms | 9.86ms | 1.28ms | - |
| solc | no | 26.1 MB | - | - | - | - | - | - | - | - | empty |
| nomicfoundation | timeout | 510.8 MB | - | - | - | - | - | - | - | - | timeout |
| juanfranblanco | crash | 0.0 MB | - | - | - | - | - | - | - | - | crash |
| qiuxiang | timeout | 69.4 MB | - | - | - | - | - | - | - | - | timeout |

## textDocument/hover

| Server | Status | Mem | Mean | p50 | p95 | Spread | Spike | Min | Max | Range | vs mmsaki |
|--------|--------|-----|------|-----|-----|--------|-------|-----|-----|-------|-----------|
| mmsaki | ok | 30.0 MB | 14.01ms | 14.0ms | 14.7ms | 0.7ms | 1.05x | 13.51ms | 14.67ms | 1.16ms | - |
| solc | empty | 26.0 MB | - | - | - | - | - | - | - | - | empty |
| nomicfoundation | timeout | 511.8 MB | - | - | - | - | - | - | - | - | timeout |
| juanfranblanco | crash | 0.0 MB | - | - | - | - | - | - | - | - | crash |
| qiuxiang | timeout | 69.9 MB | - | - | - | - | - | - | - | - | timeout |

## textDocument/references

| Server | Status | Mem | Mean | p50 | p95 | Spread | Spike | Min | Max | Range | vs mmsaki |
|--------|--------|-----|------|-----|-----|--------|-------|-----|-----|-------|-----------|
| mmsaki | ok | 30.3 MB | 11.06ms | 10.6ms | 14.4ms | 3.8ms | 1.36x | 10.24ms | 14.40ms | 4.16ms | - |
| solc | no | 26.1 MB | - | - | - | - | - | - | - | - | empty |
| nomicfoundation | timeout | 511.7 MB | - | - | - | - | - | - | - | - | timeout |
| juanfranblanco | crash | 0.0 MB | - | - | - | - | - | - | - | - | crash |
| qiuxiang | timeout | 70.1 MB | - | - | - | - | - | - | - | - | timeout |

## textDocument/documentSymbol

| Server | Status | Mem | Mean | p50 | p95 | Spread | Spike | Min | Max | Range | vs mmsaki |
|--------|--------|-----|------|-----|-----|--------|-------|-----|-----|-------|-----------|
| mmsaki | ok | 25.8 MB | 8.72ms | 8.8ms | 9.2ms | 0.3ms | 1.04x | 8.26ms | 9.17ms | 0.91ms | - |
| solc | no | 26.1 MB | - | - | - | - | - | - | - | - | empty |
| nomicfoundation | timeout | 513.5 MB | - | - | - | - | - | - | - | - | timeout |
| juanfranblanco | crash | 0.0 MB | - | - | - | - | - | - | - | - | crash |
| qiuxiang | timeout | 70.0 MB | - | - | - | - | - | - | - | - | timeout |

## textDocument/documentLink

| Server | Status | Mem | Mean | p50 | p95 | Spread | Spike | Min | Max | Range | vs mmsaki |
|--------|--------|-----|------|-----|-----|--------|-------|-----|-----|-------|-----------|
| mmsaki | ok | 26.7 MB | 64.32ms | 64.5ms | 65.2ms | 0.7ms | 1.01x | 63.34ms | 65.24ms | 1.90ms | - |
| solc | no | 25.7 MB | - | - | - | - | - | - | - | - | empty |
| nomicfoundation | timeout | 511.9 MB | - | - | - | - | - | - | - | - | timeout |
| juanfranblanco | crash | 0.0 MB | - | - | - | - | - | - | - | - | crash |
| qiuxiang | timeout | 69.5 MB | - | - | - | - | - | - | - | - | timeout |

## Peak Memory (RSS)

| mmsaki | solc | nomicfoundation | juanfranblanco | qiuxiang |
|--------|------|-----------------|----------------|----------|
| 39.7 MB | 26.2 MB | 513.5 MB | 0.0 MB | 70.1 MB |

---

*Generated from [`benchmarks/v4-core/2026-02-13T10-31-12Z.json`](benchmarks/v4-core/2026-02-13T10-31-12Z.json) — benchmark run: 2026-02-13T10:31:12Z*