paperforge 0.1.0

Rights-aware scholarly paper discovery and verified local archival
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73

# PaperForge

PaperForge is a blocking Rust library for rights-aware scholarly discovery and verified, user-directed local archival. It uses only official provider APIs: arXiv Atom, Europe PMC core search plus JATS full text, and NASA NTRS citation search/downloads.

```rust
use paperforge::{ArchiveOptions, PaperForge, PaperProvider, parse_paper_query};

let query = parse_paper_query(r#"graph networks +"molecular property" -survey title:protein author:"Jane Smith" topic:cs.LG after:2022 before:2026 @recent @limit:25"#)?;
let forge = PaperForge::with_user_agent(
    PaperProvider::Arxiv,
    "thesa/1.0 (researcher@example.org)",
)?;
let papers = forge.discover(&query)?;
if let Some(paper) = papers.first() {
    let result = forge.archive(std::slice::from_ref(paper), &ArchiveOptions::default())?;
    assert!(result.is_success());
}
# Ok::<(), paperforge::PaperForgeError>(())
```

## Query Language

Bare words and quoted phrases search all textual fields. Prefix a clause with `+` to require it or `-` to exclude it. Fields are `title:`, `abstract:`, `author:`, `category:`/`topic:`, `doi:`, `journal:`, and exact `id:`/`arxiv:`/`pmid:`/`pmcid:`/`ntrs:`. Date bounds accept a year, month, or day through `after:` and `before:`; bounds are inclusive. Exact arXiv, Europe PMC, NASA NTRS, and DOI URLs are accepted only from their known HTTPS hosts.

Directives are `@recent`, `@top`, `@sort:relevance|date|citations`, `@limit:1..100`, and `@provider:arxiv|europe-pmc|nasa-ntrs`. `@recent` ranks by local relevance tier and then date; Europe PMC `@top` ranks by tier and then real citation counts. arXiv and NASA do not provide citation counts, so those clients reject `@top` and citation sorting. A provider directive must match the instantiated client.

```text
"protein folding" +transformer -survey after:2021 @recent
author:"Jane Smith" journal:Nature @sort:citations @limit:10
arxiv:2401.12345v2
https://europepmc.org/articles/PMC1234567
https://ntrs.nasa.gov/citations/20230001234
```

Provider syntax is generated from validated typed clauses; raw query fragments are never forwarded. Unsupported arXiv quote/backslash values are rejected rather than emitted as invalid provider grammar. Bare optional clauses are OR candidates when no required clause exists; required clauses do not become restricted by optionals. Every provider result is rechecked locally using Unicode normalization, case folding, punctuation boundaries, contiguous phrases, exact identities, exclusions, and date bounds. Multiple identities are fetched as alternatives and unioned before local filtering. Initial deduplication is intentionally provider-local.

Discovery examines a fixed, provider-local budget of 100 raw records independent of the requested output limit. Ranking is deterministic within the eligible subset of that bounded page, not a claim about every matching provider record. Use `discover_detailed` to inspect raw `candidate_count`, `eligible_candidate_count`, `candidate_budget`, and `candidate_cap_reached`; a full raw page sets the cap even when provider safety gates reject records. `discover` remains a convenience returning only the ranked papers.

```rust
let result = forge.discover_detailed(&query)?;
if result.candidate_cap_reached {
    eprintln!("ranking used the first {} provider candidates", result.candidate_budget);
}
# Ok::<(), paperforge::PaperForgeError>(())
```

## Providers And Rights

- arXiv metadata is recorded as `CC0-1.0`. An explicit `arxiv:license` is retained as document-license evidence; when absent, the manifest records that absence and the exact arXiv perpetual non-exclusive terms instead of fabricating an element. Public arXiv distribution does not imply permission to redistribute a paper. Versioned identifiers are preserved. One process-global gate serializes all arXiv requests and spaces starts by at least three seconds; callers running multiple processes or machines must coordinate them externally.
- Europe PMC searches only `OPEN_ACCESS:Y` core records and requires a PMCID, `inEPMC:Y`, and open-access status. Archives are official JATS XML, not PDFs. Search-result license evidence is preflight evidence only. After download, every bounded JATS `<permissions>`/`<license>` record is retained separately. Equivalent records become the effective document license; absent evidence resolves to `Unknown/JATS-unresolved`, differing records resolve to `Unknown/JATS-ambiguous`, and conflicts are marked.
- NASA NTRS records pass only when distribution is `PUBLIC`, dissemination is `DOCUMENT_AND_METADATA`, overall export control, ITAR, and EAR are all `NO`, CUI is false, downloads are available, and a non-draft first-party `STI` `application/pdf` exists. NASA DOI exact lookup is not supported in 0.1 and is rejected. NASA publication does not by itself establish public-domain status; copyright determinations and public-use terms are retained.

OpenAlex, Crossref, and Semantic Scholar are intentionally not providers in 0.1. They are useful metadata services, but metadata openness does not license third-party PDFs.

## Archive Safety

Archives default to `papers/<provider-id>/`. `paperforge-paper.json` records source metadata, query evidence, separate metadata/document rights, provider revision, document details, relative path, byte count, BLAKE3, and completion state. Downloads are HTTPS-only, host-pinned, redirect-free, size-capped (256 MiB by default), MIME-checked, and verified as `%PDF-` or JATS `<article>` XML. PaperForge never extracts archives.

Each paper is written to a sibling staging directory and verified before rollback-aware replacement. Symlinked output components, stale transaction directories, and unrecognized existing directories are refused. `skip_existing` verifies current provider revision/search-rights evidence, manifests, bytes, document identity/magic, and BLAKE3; unavailable revision evidence forces revalidation. Recognized corrupt PaperForge output is repaired atomically. `inspect_archive` and `verify_archive` both verify complete summary identity, rights, trusted URLs, document identity/magic, bytes, and BLAKE3 before returning final post-JATS evidence.

See [`AGENT_GUIDE.md`](AGENT_GUIDE.md) for a safe autonomous workflow.

## Development

The minimum supported Rust version is 1.85.

```bash
cargo fmt --check
cargo test
cargo clippy --all-targets -- -D warnings
RUSTDOCFLAGS="-D warnings" cargo doc --no-deps
cargo package
```