sarah 0.1.5

Count tokens, lines and function sizes per file to help AI agents decide what to refactor
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160

# sarah

Sarah is our beloved mother that tracks our behavior and make sure we don't screw up.
Enforcing tokens , line count, folder structure and more.

Counts tokens, lines and function sizes per file so AI agents (and humans) can see
*when*, *what* and *how* to refactor a repo.

Built on existing crates: `ignore` (gitignore-aware parallel walking), `bpe-openai`
(tokens; exact `tiktoken` counts), `tokei` (code/comment/blank lines), `tree-sitter-tags`
(function spans), `rayon` (parallelism; files over 256 KB are tokenized in chunks).

## Install

Download a binary for Linux (x86_64/aarch64, static musl), macOS (Apple Silicon/Intel)
or Windows (x86_64) from [Releases](https://github.com/aiman2039/sarah/releases), or:

```sh
cargo install sarah
```

## Usage

```sh
sarah [PATH]                      # human table
sarah -f json                     # full report for agents
sarah -f md                       # markdown to paste into an agent's context
sarah --fail-on-warn              # exit 1 if anything exceeds limits (CI / agent loops)
sarah -f md --json-out r.json     # markdown on stdout, full JSON in a file
sarah --dirs                      # add per-folder totals to the table
sarah -f github                   # GitHub Actions ::warning annotations
sarah -f sarif > sarah.sarif      # SARIF 2.1.0 for code scanning
sarah -f json > baseline.json     # save today's warnings...
sarah --baseline baseline.json    # ...then report only new ones, or ones that got worse
sarah --changed-since origin/main # only warnings for files changed on this branch
```

Key flags: `--sort tokens|lines|name`, `-n/--top N`, `--tokenizer o200k|cl100k`,
`--max-file-tokens 8000`, `--max-file-lines 500`, `--max-fn-lines 80`,
`--max-fn-tokens N` and `--max-fn-depth N` (off by default),
`-i/--include GLOB`, `-e/--exclude GLOB`, `--hidden`, `--no-functions`,
`--no-default-excludes` (lockfiles, `*.min.js`, `*.map` are skipped by default),
`--config FILE` (default: `<PATH>/sarah.toml` when it exists).

Exit codes: `0` success, `1` warnings with `--fail-on-warn`, `2` error.

Function extraction: Rust, Python, JavaScript, TypeScript/TSX, Go, Java, C, C++, C#,
Ruby, PHP. Each function has a nesting `depth`: the deepest chain of `if`/loop/`match`/
`switch`/`try` blocks (`else if` chains count once).

The JSON report also has `directories`: per folder, its direct `files` and `dirs`, and
`total_files`, `lines` and `tokens` for the whole subtree.

## Config: `sarah.toml`

Optional. Without it, sarah behaves as above. CLI flags win over the file.

```toml
preset = "rust"            # rust | python | go | ts: built-in folder rules, applied first
exclude = ["vendor/**"]    # added to --exclude (include = [...] too)

# Size limits per glob. Later matches override earlier ones, field by field.
[[files]]
path = "**"
max_fn_lines = 60
max_fn_depth = 4

[[files]]
path = "tests/**"
max_fn_lines = 200          # max_file_tokens, max_file_lines, max_fn_tokens also work

# Folder rules. Later matches override earlier ones, field by field.
[[folder]]
path = "src/**"             # `*` = one path part, `**` = any; the root is "." (or "**")
max_files = 12              # files directly in the folder
max_dirs = 8                # sub-folders directly in the folder
max_depth = 4               # levels below the root; reported once per branch
max_tokens = 40000          # tokens in the whole subtree (one agent context)
min_files = 2               # direct entries (files + folders); fewer = flatten it
require = ["mod.rs"]        # each glob must match a file in the folder
forbid = ["utils*", "misc*"] # no file or folder name may match
allow_only = ["cli", "walk"] # sub-folder names must match one of these
naming = "snake_case"       # snake_case | kebab-case | camelCase | PascalCase
no_sibling_module = true    # no `x.rs` next to `x/`

[[folder]]
path = "src/generated/**"
max_files = 100
require = []                # an empty list turns a rule off
```

Folder warnings use kinds such as `folder_too_many_files` and `folder_bad_name`. They
work with `--fail-on-warn`, `--baseline`, `--changed-since` and annotations.

`--baseline` matches warnings by path, kind and `subject` (function name or glob), and
shows a warning again when its file, function or folder grew. `--changed-since REV`
compares with the merge base of `REV` and `HEAD`, and counts uncommitted and untracked
files.

Token counts use OpenAI encodings; Claude's tokenizer isn't public, so treat them as a close approximation.

Symlinks: every file is counted once, keyed by its realpath. File links are followed
when their target is inside the root; directory links are never descended (an in-root
target is walked by its real path anyway), so link cycles can't loop. Broken links,
links leaving the root, directory links and duplicate targets are reported under
`skipped` instead of disappearing.

## GitHub Action

Installs the released binary for the runner (Linux, macOS, Windows; x64 and ARM64 where
released), writes the markdown report to the job summary and exposes the JSON report.

```yaml
- uses: actions/checkout@v7
- id: sarah
  uses: aiman2039/sarah@v0.1.3   # a release tag; @master installs the latest release
  with:
    path: .
    args: --max-fn-lines 60 --exclude 'vendor/**'
    fail-on-warn: true
- run: echo "${{ steps.sarah.outputs.tokens }} tokens, ${{ steps.sarah.outputs.warnings }} warnings"
```

| Input | Default | |
|---|---|---|
| `path` | `.` | Directory to analyze |
| `args` | | Extra `sarah` arguments |
| `fail-on-warn` | `false` | Fail the step when limits are exceeded |
| `summary` | `true` | Write the markdown report to the job summary |
| `annotations` | `true` | Annotate the files (and lines) that have warnings |
| `version` | action ref if it is a tag, else latest | Release tag to install |
| `token` | `github.token` | Token used to download the release |

Outputs: `json` (report path), `tokens`, `warnings`.

## Development

Hooks install on the first `cargo test` (cargo-husky). `pre-commit` runs
`scripts/check.sh fmt` and `scripts/check.sh clippy`. Clippy runs pedantic plus the
safety lints in `[lints.clippy]` in `Cargo.toml`, with `-D warnings`.

```sh
sh scripts/check.sh          # fmt + clippy + tests + cargo-audit + cargo-deny (same as CI)
```

## Releasing

```sh
scripts/bump-version.sh               # 0.1.0 -> 0.1.1 in Cargo.toml + Cargo.lock (no commit)
git commit -am "Release v0.1.1" && git push
scripts/release.sh                    # trigger the Release workflow for origin/master's version

scripts/git-release.sh --check-only   # run every check, change nothing
scripts/git-release.sh                # all of the above in one step: check, bump, commit, tag, push, release
```

The Release workflow (`.github/workflows/release.yaml`) runs CI on macOS/Linux/Windows,
builds all five targets in parallel, uploads them to a draft release with
`SHA256SUMS.txt`, publishes it, then runs the action end to end against the new release.
CI runs only as part of Release.