Module pycode 

sphinx.pycode — the ModuleAnalyzer/DefinitionFinder surface the literalinclude :pyobject: filter consumes.

find_tags is the port of ModuleAnalyzer.find_tags() (SP/pycode/__init__.py:167-170 → Parser.parse_definition, SP/pycode/parser.py:623-627 → DefinitionFinder, SP/pycode/parser.py:514-588, sphinx 9.1.0). Ground truth: research spec §3.5 (cited as [INC §3.5]), docs/superpowers/plans/2026-09-01-m2-wave4.5-research-spec-include-literalinclude.md.

DefinitionFinder is a token-stream scanner, not an AST walk: it reads the tokenize stream and keeps two stacks — a dotted context of class/def names and an indents stack of open blocks. That mechanic, not Python’s grammar, is what decides every line number, so the port keeps the same shape:

• a definition’s start line is the @ line of its FIRST decorator when one is pending, else the line of the NAME after class/def (:563-567);
• a one-liner (def f(): return 1) ends at the NAME’s line — even when its signature spans continuation lines (:573-576, probe oneliner_multiline_sig);
• a block definition ends at DEDENT.end_row - 1, walked back over every line matching emptyline_re = ^\s*(#.*)?$ — blank lines AND comment-only lines (:578-588, probe comment_trap);
• add_definition drops a def whose immediately enclosing open block is also a def (:526-532), so nested functions vanish but methods survive as Class.method — and a def nested in a def through an intervening if/for/with block SURVIVES, because that block pushed an 'other' entry (probe def_in_if_in_def: outer.inner is recorded);
• the context stack tracks only class/def names, so a def inside a module-level if is recorded under its bare name (probe def_in_if);
• async def needs no special case: async is an ordinary NAME token and def drives the scan (probe deco_async).

The tokenizer simplification and its bounds

Sphinx runs CPython’s tokenize; this module runs a line-oriented scanner that produces only what DefinitionFinder consumes: INDENT/DEDENT with CPython’s two-column (tabs-as-8 and tabs-as-1) bookkeeping, NEWLINE vs NL, COMMENT, NAME, NUMBER, STRING and maximal-munch OP. It is deliberately NOT a Python parser. Tested bounds (each pinned below against the real ModuleAnalyzer):

• logical vs physical lines: a bracketed continuation emits NL, not NEWLINE, and suppresses indentation processing (probe continuation_sig); so does a backslash continuation, which makes def f(a): \ + return a a ONE-LINER (probe backslash_continuation);
• blank and comment-only lines never produce INDENT/DEDENT, which is why a # comment at column 0 inside a class does not close it (probe def_then_dedent_to_comment_col0);
• strings (all prefixes, triple quotes, escapes) are skipped whole, so def /class inside a docstring creates no tag (probe triple_quoted_trap);
• @ only starts a decorator when the previous token is NEWLINE/NL/ INDENT/DEDENT or the stream just started — the matrix-multiplication operator inside a statement is not a decorator (probe matmul_at), but one after an in-bracket NL IS mistaken for one by sphinx, and this port reproduces that (probe at_after_nl_in_parens: start line 2, not 3);
• filter_whitespace (parser.py:31-32) replaces form feeds with spaces BEFORE lines are split, and line splitting is str.splitlines(True), both mirrored here so line numbers agree (probe formfeed).

Three places where the scanner is knowingly coarser than tokenize, none of which can change a tag in a file that is valid Python:

• number munching is greedy over alphanumerics, so 1if x else 2 becomes ONE NUMBER where CPython emits NUMBER 1 + NAME if. Nothing the finder keys on (def/class/@/:) can be swallowed that way, since a number never precedes them at a header’s top level;
• a name of 1-2 letters from rRbBuUfF immediately before a quote is taken for a string prefix, so an INVALID combination (bb"x", uu'y') is lexed as one STRING where CPython emits NAME + STRING. Such a file is a syntax error in Python, so sphinx warns on it anyway (the tokenizes-but-does-not-parse class below);
• the dedent branch checks CPython’s two conditions in CPython’s order — unindent-matches-no-outer-level first, tab inconsistency second — so which failure fires is faithful; only the rendered detail differs, since sphinx surfaces them wrapped in IndentationError/TabError reprs (below).

Errors, and the divergence they carry

Sphinx’s Parser.parse() runs ast.parse BEFORE DefinitionFinder (parser.py:607-621), so find_tags fails for EVERY file that is not valid Python, with PycodeError(f'parsing {srcname!r} failed: {exc!r}') (__init__.py:158-160) — rendered, probed verbatim, as parsing '/abs/broken.py' failed: SyntaxError('invalid syntax', ('<unknown>', 1, 7, 'def f(:\n', 1, 8)). That tail is a CPython SyntaxError repr: version-specific wording, '<unknown>' from ast.parse’s default filename, and byte offsets. This port has no Python parser, so it cannot reproduce either the tail or the full failure SET. The decision (task 15, evidence above):

• our failures are a strict SUBSET of sphinx’s — only what the scanner itself cannot get past (unterminated string, unclosed bracket at EOF, unindent that matches no outer level, inconsistent tabs/spaces), each of which also fails ast.parse;
• the caller (crate::rst’s literalinclude reader) keeps sphinx’s parsing %r failed: prefix and substitutes this error’s [Display] for the un-reproducible {exc!r} tail;
• a file that tokenizes but does not PARSE (x = = 1 after a clean def) yields tags here and a warning in sphinx. Documented divergence, not a bug to fix without a Python parser.

Other documented divergences from the sphinx path:

• sphinx reads the file itself with tokenize.open (BOM + PEP 263 coding cookie); this interface takes the reader’s already-decoded text, so a non-UTF-8 file with a coding cookie and no :encoding: option differs;
• the reader has already applied :tab-width: expansion when it hands the text over (read_file, SP/directives/code.py:221-240), while sphinx’s analyzer re-reads the RAW file. Expansion never changes the line COUNT, but at a tab-width other than 8 it can change the indentation COLUMNS this scanner measures, and with them the block structure and every end line derived from it: a \t is column 8 to CPython’s tokenizer but four spaces after expandtabs(4), so against a six-space line it flips from deeper to shallower — an INDENT where sphinx sees a DEDENT, or a clean parse here where sphinx raises IndentationError. Triggering it takes all three of :tab-width: (≠ 8), :pyobject:, and a file that mixes tab and space indentation;
• PEP 701 f-strings that reuse the outer quote inside a replacement field (f"{"a"}") are lexed here as pre-3.12 string literals.

Structs

PycodeError

sphinx.errors.PycodeError — every analyzer failure funnels through LiteralInclude.run()’s broad except into a reporter warning whose text is this error’s Display.

Enums

TagKind

The first member of a find_tags entry — sphinx’s 'class' / 'def' tag strings.

Functions

find_tags

ModuleAnalyzer.find_tags() over already-decoded source text: dotted_name -> (kind, start_line, end_line), both line numbers 1-based inclusive, exactly as lines[start - 1:end] expects.