# SqC — Real-World Benchmark Results
**Last Updated**: 2026-03-06
Automated benchmark results across 5 real-world C codebases using sqc, cppcheck, and clang-tidy.
---
## Latest Results (sqc v0.3.5)
MCP-based benchmark infrastructure across 3 hosts (cppcheck 2.10, clang-tidy 21.1.6, sqc v0.3.5 commit `8b8e1eec`).
### Violation Counts — All Three Tools
| **libcrc** | 16 | 2,130 | 734 | 43 | 2 |
| **sqlite** | 310 | 402,321 | 129,035 | 1,181 | 135 |
| **mosquitto** | 384 | 88,717 | 29,824 | 747 | 44 |
| **curl** | 697 | 240,412 | 63,207 | 519 | 114 |
| **hostap** | 505 | 541,441 | 179,833 | 2,118 | 2,279 |
| **Total** | **1,912** | **1,275,021** | **402,633** | **4,608** | **2,574** |
**Interpretation**: sqc covers 283 CERT-C rules (advisory + mandatory) while cppcheck and clang-tidy implement ~20 checks each. The 100x difference in raw counts reflects rule coverage breadth, not false positive rate.
### Scan Timing — sqc v0.3.13 (4-core laptop, single process)
| **libcrc** | 16 | 2,130 | 6s | 734 |
| **mosquitto** | 384 | 88,717 | 4m 10s | 26,735 |
| **curl** | 697 | 240,412 | TBD | 63,207 |
| **sqlite** | 310 | 402,321 | TBD | 129,035 |
| **hostap** | 505 | 541,441 | TBD | 179,833 |
**Environment**: 4-core laptop (AMD/Intel mobile), single sqc process, `-d` cross-file analysis enabled, warm filesystem cache. Timing measured with `time` command (wall-clock).
**Notes**:
- Scan time correlates with LOC more than file count. sqlite's amalgamated `sqlite3.c` (~250K lines) dominates its scan time despite having fewer files than curl
- When comparing across machines, record CPU model and core count
- First-run (cold cache) adds ~50-60% overhead vs warm cache
- **mosquitto** is used as the CI/CD benchmark target. Expected CI time: ~8-12 min on a standard CI agent
### sqc Version History
| **libcrc** | 842 | 811 | 790 | 777 | 777 | 734 | -43 (-5.5%) |
| **mosquitto** | 39,177 | 33,638 | 33,200 | 29,997 | 29,989 | 29,824 | -173 (-0.6%) |
| **sqlite** | 180,011 | 147,091 | 144,581 | 130,774 | 130,802 | 129,035 | -1,739 (-1.3%) |
| **curl** | 93,576 | 73,816 | 73,239 | 64,393 | 64,389 | 63,207 | -1,186 (-1.8%) |
| **hostap** | 234,421 | 206,906 | 204,560 | 184,952 | 185,197 | 179,833 | -5,119 (-2.8%) |
| **Total** | **548,027** | **462,262** | **456,370** | **410,893** | **411,154** | **402,633** | **-8,260 (-2.0%)** |
**v0.3.5 changes**: DCL13-C alias tracking fix, DCL30-C function parameter skip, EXP33-C uninitialized variable improvements, struct field type resolution for INT32-C/INT30-C, ARR32-C array bounds fixes, POS49-C thread function fixes, STR04-C/STR34-C string type fixes.
**Top rule changes (4 comparable codebases, v0.2.21→v0.3.5)**:
- ARR32-C −2,178 (array declarator improvements)
- EXP33-C −2,118 (init tracking for function calls, conditional branches)
- POS49-C −1,787 (thread function recognition)
- STR04-C −229 (string literal type checks)
- EXP05-C −146 (const qualifier detection)
- INT34-C −100 (shift operand type fixes)
- INT32-C −93 (net: struct field resolution added new findings, bounds-check detection removed FPs)
- INT30-C +287 (struct field resolution now correctly identifies unsigned struct fields → more findings)
**v0.2.22 changes**: INT30-C if-statement upper-bound guard detection. Extends `is_bounded_by_loop_condition()` to suppress `++`/`+= 1`/`var + 1` inside `if (var < limit)` true branch. INT30-C deltas: curl -9, hostap -7, mosquitto -8 (total -24).
**v0.2.16→v0.2.21 changes**: const_eval value-range analysis (INT32-C/INT30-C macro constant folding), API00-C static function skip, INT01-C dedup fix, EXP34-C stack array NotNull, DCL13-C alias tracking, INT31-C pointer cast skip, ARR36-C type filter, INT30-C guard expansion, ARR02-C string-literal arrays, POS02-C socket/setsockopt, PRE31-C literal stripping, MEM05-C ALL_CAPS VLA.
**Top rule changes (curl, v0.2.16→v0.2.21)**: API00-C −6,102, DCL13-C −470, INT32-C −350, MEM05-C −323, EXP34-C −287, INT30-C −171, INT01-C −142, ARR02-C −98, EXP37-C −81, POS02-C −61.
**v0.2.13→v0.2.16 changes**: Call-site null propagation (EXP34-C Phase 2). Prescan collects argument null states at call sites; callee params seeded with joined caller states instead of blanket PossiblyNull. Call-site flagging re-enabled for DefinitelyNull args passed to functions that don't null-check.
**v0.2.7→v0.2.13 changes**: Cross-file analysis (`-d` directories), Windows API whitelist, bounds-check detection (INT32-C/INT30-C), CFG-based null state dataflow (EXP34-C Phase 1), multiple FP reduction rounds. Total: -85,765 (-15.6%).
### Improvement from Baseline (sqlite: v0.2.4 → v0.3.5)
| Total violations | 427,377 | 180,011 | 144,581 | 130,774 | 129,035 | **-298,342 (-69.8%)** |
| STR31-C | 206,651 | 222 | ~200 | ~200 | -206,451 (rewrite) |
| EXP34-C | 41,886 | 8,734 | ~8,500 | ~8,200 | -33,686 (CFG dataflow + call-site propagation) |
| ARR36-C | 3,034 | 600 | ~600 | ~550 | -2,484 |
| EXP30-C | 2,623 | 300 | ~300 | ~300 | -2,323 |
| API02-C | 1,542 | 166 | ~166 | ~166 | -1,376 |
### Key Observations
- **Steady decline across all projects**: Every codebase shows consistent reduction from v0.2.7 through v0.3.5
- **v0.2.16→v0.2.21 is the largest inter-version drop**: -10.0% overall, driven by API00-C static skip (dominant), const_eval, and multiple targeted FP fixes
- **v0.2.21→v0.3.5 continues trend**: -2.0% overall, driven by EXP33-C init tracking, ARR32-C array bounds, and POS49-C thread fixes
- **Struct field resolution validated**: INT30-C +287 increase is correct behavior — struct fields now properly classified as unsigned, generating new true findings
- **Advisory rules dominate**: DCL07-C, DCL31-C, DCL08-C, DCL13-C, EXP19-C, API00-C are code-style/quality rules. Severity filtering would significantly reduce noise
- **mosquitto is cleanest**: 30K violations (vs. 180K for hostap)
- **Cumulative reduction from v0.2.7**: -145,394 violations (-26.5%) across all 5 codebases (4 comparable)
---
## Cross-Tool Capability Analysis
### Comparable Checks
| Unchecked return value | ERR33-C | `cert-err33-c` | — | sqc 5x count (broader function list) |
| Unsafe numeric conversion | ERR34-C | `cert-err34-c` | — | sqc finds MORE (126 vs 33 on mosquitto) |
| Null pointer dereference | EXP34-C | `NullDereference` | `nullPointer` | sqc 4,300:1 ratio (see below) |
| Uninitialized variable | EXP33-C | — | `uninitvar` | Different sub-patterns of CWE-457 |
| String/buffer safety | STR rules | `DeprecatedOrUnsafe...` | — | Different scope |
### EXP34-C: Known High FP Rate on Real Code
| mosquitto | 8,657 | 2 | 0 |
| curl | 22,350 | 0 | 177 |
sqc uses CFG-based null state dataflow with inter-procedural call-site propagation (Phase 2 complete as of v0.2.16). cppcheck uses data-flow analysis and only fires when it can prove a null-dereference path. The gap is narrowing but sqc still flags more conservatively.
### What sqc Uniquely Covers
- **POS49-C** (POSIX misuse): 4,534 on curl — no competitor equivalent
- **INT32-C / INT30-C** (signed/unsigned overflow): significant counts — competitors skip
- **MEM30-C / MEM31-C** (use-after-free, memory management)
- **API00-C / API02-C**: no competitor equivalent
- **270+ additional rules** across integer, floating-point, environment, concurrency, POSIX
---
## Historical Results (v0.2.7)
First MCP-based benchmark run (cppcheck 2.10, clang-tidy 21.1.6, sqc v0.2.7 commit `54819432`).
| **libcrc** | 842 | 40 | 4 |
| **sqlite** | 180,011 | 517 | 204 |
| **mosquitto** | 39,177 | 364 | 160 |
| **curl** | 93,576 | 297 | 1,314 |
| **hostap** | 234,421 | 1,675 | 2,957 |
| **Total** | **548,027** | **2,893** | **4,639** |
---
## Baseline References (v0.2.3)
Earlier results for comparison (before STR31-C rewrite and major FP reductions).
### libcrc (v0.2.3)
| **sqc** | 954 | EXP14-C (106), ERR33-C (68), EXP12-C (62), INT30-C (60) |
| **cppcheck** | 40 | variableScope (36), unusedFunction (2) |
| **clang-tidy** | 52 | cert-err33-c (26), DeprecatedOrUnsafe... (24) |
### sqlite (v0.2.3)
| **sqc** | 424,842 | STR31-C (206,651) = 49% — `detect_manual_string_loop` bug |
| **cppcheck** | 1,182 | variableScope (505), toomanyconfigs (189) |
| **clang-tidy** | 2,291 | cert-err33-c (1,025), DeprecatedOrUnsafe... (453) |
### curl (v0.2.3)
| **sqc** | 207,476 | STR31-C (93,140) = 45% — same runaway bug |
| **cppcheck** | 551 | toomanyconfigs (253), variableScope (95) |
| **clang-tidy** | 1,653 | clang-diagnostic-error (1,024), cert-err33-c (366) |
### mosquitto (v0.2.3)
| **sqc** | 47,417 | EXP34-C (7,631) dominates (STR31-C NOT triggered here) |
| **cppcheck** | 598 | 50 `uninitvar` at error severity — highest-confidence real defects |
| **clang-tidy** | 907 | cert-err33-c (477), cert-err34-c (111) |
### hostap (v0.2.3)
| **sqc** | 473,862 | STR31-C (170,586) = 36% |
| **cppcheck** | 1,066 | 89 `uninitvar` at error severity |
| **clang-tidy** | 1,083 | cert-err34-c (377) dominates |
---
## STR31-C `detect_manual_string_loop` Bug (FIXED)
**Severity**: High — caused 36–49% of all sqc violations on 3 of 5 projects.
**Root cause**: Final fallback iterated ALL lines in source file looking for `memcpy` + `strlen`/`string`. One match anywhere caused every loop to generate a violation. `jimsh0.c` alone produced 180,297 violations.
**Fix**: Deleted file-wide fallback; condition-only matching; body-only write detection; improved `is_string_memcpy`.
**After fix**: `jimsh0.c` STR31-C dropped from 180,297 to 10.