Crate leadforge 

leadforge

High-performance async lead generation engine built in Rust.

leadforge provides a streaming, concurrent pipeline for fetching, filtering, and returning structured job data from external sources such as Hacker News.

The crate is designed around three core ideas:

• Streaming pipelines → process data as it arrives
• Bounded concurrency → maximize throughput without overload
• Resilience → retries, timeouts, and error classification

---

Architecture Overview

The execution model follows a layered pipeline:

Command
↓
run()
↓
source module (e.g. hn)
↓
async stream (concurrent requests)
↓
retry + timeout handling
↓
filtering
↓
Vec<Output>

Each data source (e.g. Hacker News) implements its own ingestion logic, while sharing the same high-level execution pattern.

---

Example

use leadforge::{run, LeadForgeCommand};
use std::time::Duration;

#[tokio::main]
async fn main() {
    let command = LeadForgeCommand::HackerNews {
        limit: 10,
        max_concurrency: 50,
        max_retries: 3,
        timeout: Duration::from_secs(10),
        keyword: Some("rust".to_string()),
    };

    let results = run(command).await.unwrap();

    for job in results {
        println!("{:?}", job);
    }
}

---

Design Goals

• Efficiency: avoid unnecessary allocations and blocking
• Scalability: handle large ID ranges and high request volume
• Composability: expose a clean API usable from CLI or other programs
• Robustness: gracefully handle unreliable networks and APIs

---

Modules

• hn → Hacker News ingestion pipeline

Additional sources can be added by following the same pattern.

---

CLI Usage

While leadforge is designed as a reusable library, it is primarily consumed through its command-line interface.

Basic Example

leadforge hacker-news --limit 10

With Filtering

leadforge hacker-news --keyword rust --limit 5

Controlling Concurrency and Retries

leadforge hn \
  --limit 20 \
  --max-concurrency 50 \
  --max-retries 5 \
  --timeout 5

---

Output Format

Results are emitted as JSON to stdout, making them easy to integrate with other tools.

Example:

[
  {
    "id": 47251163,
    "by": "yeldarb",
    "score": 1,
    "time": 1772646584,
    "title": "Roboflow (YC S20) Is Hiring a Security Engineer for AI Infra",
    "url": "https://roboflow.com/careers",
    "text": null
  }
]

---

Shell Integration

Because output is written to stdout, leadforge composes naturally with standard Unix tools:

leadforge hn --keyword rust | jq

leadforge hn --limit 50 > jobs.json

This makes it suitable for:

• automation scripts
• data pipelines
• lead generation workflows

Enums

LeadForgeCommand

High-level command describing what data to fetch.

LeadForgeError

Errors that can occur during lead fetching and processing.

Functions

run

Executes a LeadForgeCommand and returns collected results.