scrape-le 0.1.0

Check whether a page is scrapeable before the scraper is written
scrape-le-0.1.0 is not a library.

Useful? A star is how other developers find it — ★ GitHub · letools.dev

Most scraper failures are discovered after the scraper is written: the page renders a challenge, robots.txt forbade the path all along, the third request hits a rate limit. scrape-le asks first — it loads the page the way a scraper would, gathers the evidence, and returns a verdict with the receipts, as JSON on stdout and an exit code a script can branch on.

It is the second frontend of Scrape-LE, the VS Code extension — one product, two frontends, one repository, so the two can never answer a URL differently. The signature corpus both build against lives at signatures/ and fixtures/, and CI fails on drift.

Sixty seconds

scrape-le https://example.com/search      # one URL
scrape-le --input urls.txt                # a batch, streamed as it completes
scrape-le doctor                          # is a browser here, and which

restricted — 1 finding  (https://example.com/search · 1842ms · exit 1)
  blocks  robots     Disallow: /search for User-agent: *
  checks  antibot ✓  ·  rate-limit ✓  ·  robots ✓  ·  auth ✓

The report is JSON on stdout, the summary above is stderr, and the exit code is the answer: 0 clear · 1 a real no · 2 the question was malformed. Every finding carries its evidence — not "Cloudflare detected" but which signal, from which source — so a false positive is diagnosable rather than mysterious.

Install

Route Command Worth knowing
cargo cargo install scrape-le Any platform, needs Rust 1.88+.
From source git clone https://github.com/nolindnaidoo/scrape-lecd scrape-le/crate && cargo build --release The same build CI runs.

You also need a Chromium — Chrome, Chromium, Brave or Edge. It is never downloaded for you; scrape-le doctor says whether one was found and what runs without it. Homebrew, winget and prebuilt binaries follow the pixelcoords/pixelactions playbook and arrive with later releases.

Verdicts

Verdict Means Exit
clear every check ran, and nothing found would stop a naive scraper 0
restricted something would stop or limit you — see the findings 1
blocked the page could not be reached or rendered at all 1
inconclusive nothing blocking was found, but not every check ran 1

clear requires completeness; restricted does not. A Disallow or a 401 is true whether or not the page rendered, so a partial run can say restricted honestly — but clear is a claim about absence, and absence cannot be claimed for a check that did not run. That is why a --no-render run can never come back clear, and why the report names which checks were partial.

Commands and flags
scrape-le <url> check one URL
--input <file|-> batch: a JSON array, a CSV with a url column, or one URL per line — detected by content, not extension
--no-render skip the browser; caps the verdict at inconclusive
--agent <token> evaluate robots.txt as this crawler (RFC 9309 group selection) instead of User-agent: *
--signatures <file> add or replace vendor signatures from a TOML file
--concurrency <n> hosts checked at once (default 4); same-host URLs are always sequential
--ignore-crawl-delay do not honour a declared Crawl-delay; recorded in the report when used
doctor is a browser available, which one, what will run
mcp serve the same checks over MCP on stdio

Two MCP servers, one tool contract

scrape-le mcp offers three tools; the published npm server scrape-le-mcp offers one. The overlap is deliberate and enforced:

Tool npm server this binary
analyze_robots_txt — content in, analysis out, no network yes yes, byte-identical
scrape_le_check — fetch, render, full verdict — yes
scrape_le_doctor — browser availability — yes

The npm one runs anywhere with no install and no browser, which is why it ships inside the VS Code extension and works over npx. This one needs the binary and a Chromium. Writing analyze_robots_txt once means one tool name works whichever server a host has configured; a shared fixture corpus runs against both implementations and fails either build if they drift.

Batches

One rule generates the rest: never two concurrent requests to the same host. A tool whose premise is asking whether it is acceptable to hit a site cannot hammer that site while asking, and a batch of a hundred URLs is very often a hundred paths on one site. So hosts run in parallel, URLs within a host run sequentially, Crawl-delay is honoured between them, robots.txt is fetched once per host, exact-duplicate URLs are checked once, and reports stream as they complete carrying their input index. The exit code is the worst verdict in the batch.

Design commitments

These hold for every release, starting with the first:

  • A real browser, never downloaded. It drives a Chromium you already have — Chrome, Chromium, Brave, or Edge — and if none is found it says so and names the fix, rather than fetching 130 MB on first run. The tool stays useful without one: robots.txt, status, redirects, and rate-limit headers are plain HTTP.
  • Rendering is the default, because most anti-bot detection is invisible to a raw fetch, and a tool whose value is an honest answer must not default to the mode that produces the least honest one.
  • Exit codes are the API. Scripts branch on them; for a batch, the exit code is the worst verdict in it.
  • Network scope is the URL under check plus that origin's /robots.txt. Nothing else, ever. Generic User-Agent, no telemetry.
  • No async runtime. Sync CDP (headless_chrome), ureq, and std threads — batching runs 4 hosts concurrently, sequential within a host, and never sends two concurrent requests to the same host.

Non-goals

Not a scraper — no selectors, no extraction, no pagination, no crawling beyond the single URL and its robots.txt. Not a bypass tool — it never solves a captcha, never rotates a proxy, never impersonates a TLS fingerprint. Detection informs a person; absence of a signature is not permission to scrape.

Documentation

Also by nolindnaidoo

Rust

VS Code Extensions — every tool in the family, one page: letools.dev

  • String-LE - Extract string values for i18n from JSON, YAML, CSV, TOML, INI, and .env
  • Numbers-LE - Extract numeric values from JSON, YAML, CSV, TOML, INI, and .env
  • EnvSync-LE - Spot missing keys across your .env files, with a markdown report
  • Paths-LE - Extract file paths from JS/TS imports, JSON, HTML, CSS, TOML, CSV, and .env
  • Secrets-LE - Detect and sanitize credentials locally, before you commit
  • Scrape-LE - Check whether a page is scrapeable before you write the scraper
  • Colors-LE - Extract and analyze colors from CSS, SCSS, LESS, Stylus, HTML, JS/TS, and SVG
  • URLs-LE - Extract URLs from documentation, configs, and code
  • Regex-LE - Find, test, and validate the regex patterns in the current file
  • Dates-LE - Extract and analyze dates from logs, configs, and code

Contact Developer — nolindnaidoo.com · GitHub · LinkedIn

License

MIT — see LICENSE.