flq

Fetch the web as lean, token-efficient text. HTML → .flq

Raw HTML wastes 70-90% of LLM tokens on markup. flq strips a page down to its content — headings, text, lists, tables, code — in a format that tokenizes tight and reads clean.

Install

cargo install flq

Usage

flq https://example.com                          # fetch one page
flq https://example.com/1 https://example.com/2  # multiple (=== separated)
curl -s https://example.com | flq -              # HTML from stdin
flq page.html                                    # local file

curl-compatible flags

flq URL -o page.flq                    # output to file
flq URL -H "Authorization: Bearer x"   # custom header
flq URL -A "my-agent/1.0"              # user-agent
flq URL -m 10                          # timeout (seconds; k/m suffix ok, e.g. 1k)
flq URL -b "session=abc"               # cookies
flq URL -I                             # response headers only
flq URL -s                             # silent

Extraction

flq URL -f headings                    # headings only
flq URL -f code                        # code blocks only
flq URL -f tables,links                # combine filters
flq URL -g "install"                   # grep body lines
flq URL --select "article.post"        # scope by CSS selector
flq URL --raw                          # raw HTML passthrough

Filters: headings code tables links img text

Agent workflow

flq URL --max-tokens 2k                # token budget (k/m suffix), truncates cleanly
flq URL --no-meta                      # body only, no frontmatter
flq URL -j                             # JSON: {title,url,body,tokens,truncated}
cat urls.txt | flq --pipe              # NDJSON stream, one page per line
flq URL --depth 1 --max-pages 10       # crawl same-host links
flq URL --cache                        # cache locally (~/.cache/flq)
flq URL --cache --cache-ttl 86400      # custom TTL
flq URL --diff                         # diff against cached version

Format

title: Page Title
url: https://example.com
lang: ja
desc: meta description
---
# heading
paragraph text
- list item
  - nested item
> quote
@ https://link | link text
! https://img.png | alt text
table[3]{name,price,stock}:
Widget A,1200,45
Widget B,800,120
Gadget C,500,300

Prose stays markdown — LLMs parse it natively and every marker is one token. Tables use schema-once TOON-style headers: field names are declared once, rows stream as CSV. Links (@) and images (!) are off by default; enable with -f links / -f img.

JSON output

flq https://example.com -j

{
  "title": "Page Title",
  "url": "https://example.com",
  "body": "# heading\nparagraph...",
  "tokens": 142,
  "truncated": false
}

tokens is an estimate (~4 ASCII chars or ~1.3 CJK chars per token).

As a library

use flq::{Opts, process};

let opts = Opts {
    max_tokens: Some(2000),
    no_meta: true,
    ..Default::default()
};
let page = process("https://example.com", &opts)?;
println!("{} ({} tokens)", page.body, page.tokens);

MCP server

flq doubles as a Model Context Protocol server, so agents like Claude Code can fetch and parse the web directly.

flq --mcp                               # run as MCP server (stdio JSON-RPC)
claude mcp add flq -- flq --mcp         # register with Claude Code

Tools: fetch_page, fetch_pages, crawl_site, head. See docs/mcp.md for the full tool reference and setup, and skills/flq/SKILL.md for an agent skill that teaches when and how to reach for flq.

License

MIT

flq 1.0.0

flq