replysieve

Keep the real comments, drop the junk. replysieve judges a user comment in about a second and tells you to approve it, reject it, or send it for review, with a reason. It runs free checks first and then asks a small decision model on OpenRouter, so it is much cheaper and faster than asking a chat LLM.

It works as a CLI, an MCP server for coding agents, and a Rust library.

cargo install replysieve
export OPENROUTER_API_KEY=sk-or-...   # https://openrouter.ai/keys

replysieve check --title "Taskly, a todo app with offline sync" \
  "Does offline sync merge edits if I change the same task on two phones?"

{"verdict":"approve","reason":"useful","confidence":1.0,"level":"medium","model":"typesafe/jev-1.14-20261014","latency_ms":712}

Use it from Claude Code or Codex

claude mcp add replysieve -e OPENROUTER_API_KEY=sk-or-... -- replysieve mcp
codex mcp add replysieve --env OPENROUTER_API_KEY=sk-or-... -- replysieve mcp

Then ask your agent something like "check these comments with replysieve at heavy strictness". It gets two tools, check_comment and check_batch.

Strictness levels

Your own rules apply at every level. The level only changes the built-in checks.

Verdict JSON

If no model answers, the verdict is needs_review with reason model_unavailable. replysieve never approves silently on failure.

Context, batches and duplicates

# Context helps the heavy level tell on-topic from off-topic
replysieve check --level heavy \
  --title "Brewly" --description "Coffee beans by mail, roasted weekly" \
  --parent "Has anyone tried the decaf?" \
  "The decaf is surprisingly good, a bit flat in milk though"

# A batch, one JSON object per line, verdicts come back in the same order
replysieve check --file comments.jsonl
#   {"text": "...", "title": "...", "description": "...", "parent": "..."}

# Reject exact repeats of recent texts (one per line)
replysieve check --recent recent.txt "Try my app, link in bio"

Presets and custom rules

--preset comments (default), product-review or launch-thread tells the model what kind of place the comment came from and adds stock phrases for that setting.

Put your own rules and defaults in a TOML file and pass --config replysieve.toml.

level = "medium"
preset = "launch-thread"
models = ["~typesafe/jev-latest"]        # tried in order
rules = [
  "No links to other products",
  "No questions about pricing, those go to support",
]
stock_phrases = ["nice ship"]

Command-line flags override the file. --model can be repeated to set the fallback order.

Rust library

use replysieve::{Checker, Comment, Level, Options};

# async fn run() -> Result<(), replysieve::Error> {
let checker = Checker::new(
    Options { level: Level::Heavy, ..Options::default() },
    std::env::var("OPENROUTER_API_KEY").ok(),
);
let comment = Comment {
    title: Some("Taskly, a todo app with offline sync".into()),
    ..Comment::new("Does sync merge edits made on two phones?")
};
let verdict = checker.check(&comment, &[]).await?;
println!("{:?} {:?}", verdict.verdict, verdict.reason);
# Ok(())
# }

Docs are on docs.rs/replysieve.

Accuracy

replysieve eval runs the 302 labeled example comments that ship with the crate (product reviews, Reddit-style threads, X replies and launch comments) and reports, per level, how much junk was rejected and how many good comments were wrongly rejected. With ~typesafe/jev-latest on 2026-11-08:

At heavy, the misses are on-topic comments with no specific point that the model was unsure about, so they go to review rather than being rejected. The set was drafted with an AI model and reviewed by hand, so treat these numbers as a check that the pipeline works, not as a promise for your own comments. Run replysieve eval --file your-set.jsonl --misses on your own labeled comments.

Privacy

replysieve is stateless. It never writes comment text to disk or logs, at any verbosity (-v prints only timing and model fallback events). Comment text goes only to OpenRouter, using your own key.

Not in 0.1.0

An HTTP server and Docker image are planned for 0.2.0. English only. No images or video. replysieve gives verdicts and leaves deleting, hiding or banning to you.

License

Licensed under either of Apache License, Version 2.0 or MIT license at your option. Contributions are welcome, see CONTRIBUTING.md.