urls-le 0.1.0

Useful? A star is how other developers find it — ★ GitHub · letools.dev/tools/urls-le

grep -o 'https\?://[^ ]*' finds URLs. It also finds them inside fenced code blocks, misses the one in a TOML value it can't see the string boundary of, and hands you back half a URL because it guessed where the delimiter was. urls-le knows the difference between a URL in a Markdown link and one in a <!-- comment -->, and it gives you the line and column of each.

Then it stops. This tool extracts; what the URLs mean is your call.

It is the second frontend of URLs-LE, the VS Code extension — one product, two frontends, one repository, so the two can never read a document differently. The corpus both build against lives at crate/fixtures/, and CI fails on drift.

Sixty seconds

urls-le .                       # every URL in the tree
urls-le --dedupe docs/          # one line per distinct URL
cat README.md | urls-le --stdin --format markdown

# the point of the whole thing:
urls-le . | jq -r '.urls[].value' | sort -u | lychee -

./config.toml:1:13  https://example.org/start
./docs/guide.md:3:13  https://example.com/api
./docs/guide.md:3:44  https://example.com/ref
3 URLs in 2 files

Exit codes follow grep — 0 URLs found, 1 none found, 2 the question was malformed. Finding none is an answer, not an error, which is what makes if urls-le src/; then … work.

Install

No runtime, no network, nothing written.

Why it has no opinions

The obvious next features are all judgments: fetch each URL to see if it 404s, flag http:// as insecure, flag a password in the userinfo, flag a private address. Every one is absent, deliberately.

An http:// URL is wrong in a production config and correct in a test fixture. A localhost URL is a bug in one repository and the whole point in another. A tool that decides for you is a tool you configure, then argue with, then mute — and the muting takes the extraction with it.

Link checking is also already solved. lychee, muffet and htmltest do it well, and each needs a network, which would stop this being a cheap deterministic CI step. The value here is the list, not a verdict on it — so pipe it to one of those and let it have the opinions.

A contract test asserts that no flag asks for a judgment, so the boundary is enforced rather than merely written down.

What it reads

Eleven formats, the same eleven the extension supports: Markdown, HTML, CSS, JavaScript, TypeScript, JSON, YAML, .properties, TOML, INI, XML.

Each format decides which part of the document to scan, and nothing else — that is what one shared scanner buys. Markdown skips fenced blocks and inline code spans. HTML skips comments. JSON reads string literals only. .properties skips comment lines. TOML and INI parse first and locate each value back in the source, falling back to a whole-document scan when the file does not parse — so a broken config still yields its URLs.

A directory is walked the way ripgrep walks one: .gitignore honoured, hidden files skipped, --no-ignore and --hidden to reach the rest. A file named explicitly is always read.

Ported as-is, including the awkward parts

• A URL ends at whitespace or at the extension's delimiter set — angle brackets, quotes, braces, pipes, backslash, caret, backtick, square brackets, semicolon, close-paren, apostrophe.
• Trailing punctuation is kept. https://x.com/a. keeps its dot, because a dot can be part of a URL and stripping it would corrupt the ones that genuinely end that way. A documented limitation, not a bug.
• Every occurrence is its own result, with its own position. --dedupe is opt-in, because which repeats matter is your call.
• mailto: needs an @; tel: needs a subject.
• Limits are behaviour: 10 MB of content and 50,000 URLs per document, both reported when they bind so a truncated result is never mistaken for a complete one.

Options

--dedupe             collapse repeated URLs to their first occurrence
--format <format>    force a format instead of inferring from the name
--stdin              read one document from stdin
--follow-symlinks    descend symlinked directories when walking a tree
--hidden             walk hidden files and directories too
--no-ignore          walk files that .gitignore excludes

As an MCP server

urls-le mcp

Two tools, both returning { ok, data, diagnostics, meta }:

• extract_urls — content in, URLs out. Touches no filesystem. The npm server ships the same tool with byte-identical output; one corpus runs against both.
• urls_le_scan — files or directories in, the same reports the CLI writes.

ok means the scan ran, never that it found something. A document with no URLs is a result, not an error.

The other four ways to run it

All ten LE tools are on letools.dev.

Also by nolindnaidoo

Rust

• pixelcoords — Freeze your screen, mark regions, get pixel-exact coordinates and crops pixelcoords.dev · crates.io · docs.rs
• pixelactions — Consume human-verified coordinates, perform the interaction, confirm it landed pixelactions.dev · crates.io
• paths-le — Find every path in a codebase and report whether it still points at anything crates.io
• secrets-le — Find hardcoded credentials, and never print one crates.io
• regex-le — Find every regex in a codebase and report which can be driven into catastrophic backtracking crates.io
• scrape-le — Check whether a page is scrapeable before the scraper is written crates.io

Contact Developer — nolindnaidoo.com · GitHub · LinkedIn

License

MIT — see LICENSE.