PaperForge

PaperForge is a blocking Rust library for rights-aware scholarly discovery and verified, user-directed local archival. It uses only official provider APIs: arXiv Atom, Europe PMC core search plus JATS full text, and NASA NTRS citation search/downloads.

use paperforge::{ArchiveOptions, PaperForge, PaperProvider, parse_paper_query};

let query = parse_paper_query(r#"graph networks +"molecular property" -survey title:protein author:"Jane Smith" topic:cs.LG after:2022 before:2026 @recent @limit:25"#)?;
let forge = PaperForge::with_user_agent(
    PaperProvider::Arxiv,
    "thesa/1.0 (researcher@example.org)",
)?;
let papers = forge.discover(&query)?;
if let Some(paper) = papers.first() {
    let result = forge.archive(std::slice::from_ref(paper), &ArchiveOptions::default())?;
    assert!(result.is_success());
}
# Ok::<(), paperforge::PaperForgeError>(())

Query Language

Bare words and quoted phrases search all textual fields. Prefix a clause with + to require it or - to exclude it. Fields are title:, abstract:, author:, category:/topic:, doi:, journal:, and exact id:/arxiv:/pmid:/pmcid:/ntrs:. Date bounds accept a year, month, or day through after: and before:; bounds are inclusive. Exact arXiv, Europe PMC, NASA NTRS, and DOI URLs are accepted only from their known HTTPS hosts.

Directives are @recent, @top, @sort:relevance|date|citations, @limit:1..100, and @provider:arxiv|europe-pmc|nasa-ntrs. @recent ranks by local relevance tier and then date; Europe PMC @top ranks by tier and then real citation counts. arXiv and NASA do not provide citation counts, so those clients reject @top and citation sorting. A provider directive must match the instantiated client.

"protein folding" +transformer -survey after:2021 @recent
author:"Jane Smith" journal:Nature @sort:citations @limit:10
arxiv:2401.12345v2
https://europepmc.org/articles/PMC1234567
https://ntrs.nasa.gov/citations/20230001234

Provider syntax is generated from validated typed clauses; raw query fragments are never forwarded. Unsupported arXiv quote/backslash values are rejected rather than emitted as invalid provider grammar. Bare optional clauses are OR candidates when no required clause exists; required clauses do not become restricted by optionals. Every provider result is rechecked locally using Unicode normalization, case folding, punctuation boundaries, contiguous phrases, exact identities, exclusions, and date bounds. Multiple identities are fetched as alternatives and unioned before local filtering. Initial deduplication is intentionally provider-local.

Discovery examines a fixed, provider-local budget of 100 raw records independent of the requested output limit. Ranking is deterministic within the eligible subset of that bounded page, not a claim about every matching provider record. Use discover_detailed to inspect raw candidate_count, eligible_candidate_count, candidate_budget, and candidate_cap_reached; a full raw page sets the cap even when provider safety gates reject records. discover remains a convenience returning only the ranked papers.

let result = forge.discover_detailed(&query)?;
if result.candidate_cap_reached {
    eprintln!("ranking used the first {} provider candidates", result.candidate_budget);
}
# Ok::<(), paperforge::PaperForgeError>(())

Providers And Rights

• arXiv metadata is recorded as CC0-1.0. An explicit arxiv:license is retained as document-license evidence; when absent, the manifest records that absence and the exact arXiv perpetual non-exclusive terms instead of fabricating an element. Public arXiv distribution does not imply permission to redistribute a paper. Versioned identifiers are preserved. One process-global gate serializes all arXiv requests and spaces starts by at least three seconds; callers running multiple processes or machines must coordinate them externally.
• Europe PMC searches only OPEN_ACCESS:Y core records and requires a PMCID, inEPMC:Y, and open-access status. Archives are official JATS XML, not PDFs. Search-result license evidence is preflight evidence only. After download, every bounded JATS <permissions>/<license> record is retained separately. Equivalent records become the effective document license; absent evidence resolves to Unknown/JATS-unresolved, differing records resolve to Unknown/JATS-ambiguous, and conflicts are marked.
• NASA NTRS records pass only when distribution is PUBLIC, dissemination is DOCUMENT_AND_METADATA, overall export control, ITAR, and EAR are all NO, CUI is false, downloads are available, and a non-draft first-party STI application/pdf exists. NASA DOI exact lookup is not supported in 0.1 and is rejected. NASA publication does not by itself establish public-domain status; copyright determinations and public-use terms are retained.

OpenAlex, Crossref, and Semantic Scholar are intentionally not providers in 0.1. They are useful metadata services, but metadata openness does not license third-party PDFs.

Archive Safety

Archives default to papers/<provider-id>/. paperforge-paper.json records source metadata, query evidence, separate metadata/document rights, provider revision, document details, relative path, byte count, BLAKE3, and completion state. Downloads are HTTPS-only, host-pinned, redirect-free, size-capped (256 MiB by default), MIME-checked, and verified as %PDF- or JATS <article> XML. PaperForge never extracts archives.

Each paper is written to a sibling staging directory and verified before rollback-aware replacement. Symlinked output components, stale transaction directories, and unrecognized existing directories are refused. skip_existing verifies current provider revision/search-rights evidence, manifests, bytes, document identity/magic, and BLAKE3; unavailable revision evidence forces revalidation. Recognized corrupt PaperForge output is repaired atomically. inspect_archive and verify_archive both verify complete summary identity, rights, trusted URLs, document identity/magic, bytes, and BLAKE3 before returning final post-JATS evidence.

See AGENT_GUIDE.md for a safe autonomous workflow.

Development

The minimum supported Rust version is 1.85.

cargo fmt --check
cargo test
cargo clippy --all-targets -- -D warnings
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps
cargo package