Module tokens 

lf tokens — how big is this codebase, in the units a model pays for.

Lines are what a human counts; tokens are what a run costs. They disagree wildly — a minified file or a lockfile is cheap in lines and ruinous in tokens — and the context project is about the second number, so that is the one this measures.

Tokens come from the same count_tokens (tiktoken cl100k_base) the prompt assembler uses to budget context, so the tree here and the context budget there are speaking about the same quantity.

Only tracked files are counted: git ls-files already honours .gitignore, so target/, .build/, and node_modules/ never enter the total. A file that is not valid UTF-8 is a binary and is skipped rather than estimated. A tracked symlink weighs its link text, matching the blob Git stores; the target is already counted at its own tracked path.

Structs

CodeNode

One directory or file in the codebase tree. Wire type: every field required.

CodeSlice

One file extension’s weight in a snapshot. ext is lowercase and carries no dot; a file with no extension is (none), and the long tail is other.

CodeSnapshot

How big the codebase was, per top-level directory, on each day it changed.

Functions

run