loctok

Count LOC (lines of code) & TOK (LLM tokens), fast.

Features

• Gitignore-aware scan (respects .gitignore, global gitignore, and git excludes)
• Tiktoken encodings: o200k_base (default), cl100k_base, p50k_base, p50k_edit, r50k_base
• By-language summary table, JSON report, or file tree view
• Extension filter via --ext rs,py,ts (case-insensitive, no leading dots)
• Optional inclusion of hidden files via --hidden
• Fast parallel scanning (Rayon) with a live progress indicator (stderr)

Install

Option 1: via npm (prebuilt binary)

npx loctok          # run directly
# or
npm i -g loctok     # install globally

Option 2: build from source (Rust 1.70+ recommended)

# From crates.io (recommended)
cargo install loctok

# From Git (without publishing)
cargo install --git https://github.com/yourname/loctok --bin loctok

# From the project root (for local testing)
cargo install --path .

CLI Usage

# Count tokens in current directory
loctok

# Count tokens in a given path
loctok path/to/dir

# JSON output
loctok --format json > counts.json

# File tree with per-node LOC and tokens
loctok --format tree

# Use a specific encoding
loctok --encoding cl100k_base

# Include only certain extensions (no dots)
loctok --ext rs,md,ts

# Include hidden files (dotfiles)
loctok --hidden

# Progress prints to stderr; to silence in scripts, redirect:
loctok --format json 2>/dev/null

Run loctok --help to see all options.

Examples

The examples below were produced by running against tests/fixtures in this repo.

By-language table (default)

> loctok

595.171834ms (655.27 files/s)
╭────────────┬───────────────┬─────────────╮
│ Language   │ lines of code │ token count │
├────────────┼───────────────┼─────────────┤
│ Rust       │       109,910 │     894,106 │
│ Other      │        13,705 │     174,612 │
│ YAML       │         4,668 │      91,801 │
│ TypeScript │         6,224 │      53,639 │
│ Markdown   │         1,584 │      17,791 │
│ TOML       │         1,260 │      11,727 │
│ SVG        │           222 │      10,950 │
│ JSON       │           261 │       4,001 │
│ Vue        │           214 │       1,524 │
│ Text       │           119 │       1,296 │
│ CSS        │            69 │         420 │
│ JavaScript │            26 │         277 │
│ HTML       │            13 │         112 │
│ Shell      │             5 │          29 │
│ SUM:       │       138,280 │   1,262,285 │
╰────────────┴───────────────┴─────────────╯

Tree view

> loctok --format tree

379.958459ms (21.05 files/s)

Name                           LOC       TOK
--------------------------------------------
    ┌── lib.rs                 332     3,213
    ├── main.rs                573     4,707
┌── src/                       905     7,920
│           ┌── kept2.txt        1         3
│       ┌── nested/              1         3
│       ├── kept.txt             1         3
│   ┌── fixtures/                2         6
│   ├── integration.rs          69       790
├── tests/                      71       796
├── Cargo.lock                 877    10,198
├── Cargo.toml                  26       201
├── README.md                  110     1,174
./                           1,989    20,289

JSON

> loctok --format json

{
  "by_language": [
    {
      "language": "Other",
      "lines": 877,
      "tokens": 10198
    },
    {
      "language": "Rust",
      "lines": 974,
      "tokens": 8710
    },
    {
      "language": "Markdown",
      "lines": 120,
      "tokens": 1316
    },
    {
      "language": "TOML",
      "lines": 26,
      "tokens": 201
    },
    {
      "language": "Text",
      "lines": 2,
      "tokens": 6
    }
  ],
  "encoding": "o200k_base",
  "files": [
    {
      "lines": 26,
      "path": "./Cargo.toml",
      "tokens": 201
    },
    {
      "lines": 1,
      "path": "./tests/fixtures/nested/kept2.txt",
      "tokens": 3
    },
    {
      "lines": 1,
      "path": "./tests/fixtures/kept.txt",
      "tokens": 3
    },
    {
      "lines": 69,
      "path": "./tests/integration.rs",
      "tokens": 790
    },
    {
      "lines": 877,
      "path": "./Cargo.lock",
      "tokens": 10198
    },
    {
      "lines": 120,
      "path": "./README.md",
      "tokens": 1316
    },
    {
      "lines": 332,
      "path": "./src/lib.rs",
      "tokens": 3213
    },
    {
      "lines": 573,
      "path": "./src/main.rs",
      "tokens": 4707
    }
  ],
  "models": [
    "GPT-4o",
    "GPT-4.1",
    "o1",
    "o3",
    "o4"
  ],
  "path": ".",
  "token_number": 200000,
  "total": 20431
}

Behavior and Notes

• Respects .gitignore, global gitignore, and git excludes; also adds .gitignore as a custom ignore file in non-git contexts.
• Only UTF‑8 text files are counted; non‑UTF‑8 files are skipped silently.
• Language grouping is inferred from file extensions.
• Progress updates print to stderr. They are ephemeral on TTYs (single updating line) and line-based otherwise. Redirect or pipe stderr to silence in scripts.

Development

• Run tests: cargo test
• Optional: cargo fmt and cargo clippy

License

MIT or Apache-2.0