photoforge 0.1.0

Rights-aware Wikimedia Commons image discovery and verified photo archival
Documentation

PhotoForge

PhotoForge 0.1.0 is a blocking Rust library for rights-aware Wikimedia Commons image discovery and verified, user-directed local archival. It uses the official https://commons.wikimedia.org/w/api.php Action API and original Commons downloads. It never scrapes file pages or mirrors a catalog, has no binary or CLI, and accepts no user credentials.

PhotoForge's name and ranking vocabulary do not mean every accepted bitmap is a photograph, depicts what its metadata claims, is safe-for-work, or is suitable for a particular audience or use. The retained metadata and strict automatic license filter are evidence for source review, not legal conclusions.

use photoforge::{ArchiveOptions, PhotoForge, parse_photo_target};

let forge = PhotoForge::with_user_agent(
    "my-photo-agent/1.0 (contact@example.org)",
)?
.with_discovery_limit(10);

let target = parse_photo_target("search:blue flower")?;
let discovery = forge.discover_detailed(&target)?;
if let Some(photo) = discovery.photos.first() {
    println!("Review {} at {}", photo.title, photo.source_url);
    println!("Attribution: {}", photo.rights.attribution);
    let result = forge.archive(std::slice::from_ref(photo), &ArchiveOptions::default())?;
    assert!(result.is_success());
}
# Ok::<(), photoforge::PhotoForgeError>(())

Public API

PhotoForge::with_user_agent(&str) requires a nonblank, control-free caller User-Agent. The client disables redirects and system/environment proxies and has a 90-second timeout. with_discovery_limit clamps to 1 through 50. provider, discover, discover_detailed, archive, and archive_with_progress are blocking methods. The free inspect_archive and verify_archive functions fully verify an archive; equivalent client methods are provided for integration convenience. photoforge::AGENT_GUIDE is byte-for-byte the included AGENT_GUIDE.md.

parse_photo_target supports:

Input Target Provider behavior
blue flower or search:blue flower PhotoTarget::Search Literal bounded text search plus filetype:bitmap
category:Night photographs PhotoTarget::Category Direct exact category-members generator
file:Example.jpg PhotoTarget::File Exact current file metadata, following Commons title redirects
https://commons.wikimedia.org/wiki/File:Example.jpg PhotoTarget::File Exact canonical HTTPS Commons file page
top, popular, trending PhotoTarget::Top Bounded slice of curated Category:Featured pictures on Wikimedia Commons, ranked locally
latest, newest, new PhotoTarget::Latest Bounded raw allimages window, requested newest-first

Empty values, HTTP, credentials, fragments, queries, nonstandard ports, other hosts, other Commons paths, and raw non-file URLs are rejected. PhotoTarget::is_collection() is false only for File.

Discovery Bounds

Every free-text token is separately quoted and backslashes/quotes are escaped before the fixed filetype:bitmap operator is added. User text cannot become a CirrusSearch field, negation, or category operator. Every collection target requests at most min(limit * 3, 50) raw candidates. Pages are provider-order-sorted, truncated to that raw budget, and deduplicated before metadata is requested. A separate batched metadata request addresses only those candidate page IDs, so metadata continuation cannot silently enlarge the candidate set.

One discovery or exact-fetch operation permits at most ten actual HTTP sends, including every retry and continuation. Candidate and metadata work share that budget and one five-second aggregate retry deadline, so retries reduce continuation capacity. Archive performs one independently bounded exact-fetch operation per selected photo before its download.

DiscoveryResult reports candidate_count, eligible_candidate_count, candidate_budget, and candidate_cap_reached. candidate_count is the number of unique valid IDs for which metadata was requested. The cap means ranking covers a bounded provider window, not every Commons match. Search requires normalized whole-token coverage across title, object name, categories, and description, then prioritizes exact/title coverage. Category and top ranking use title, featured/quality assessment, and global-usage evidence. Top is not a claim of global popularity or trending status: it is only a bounded slice of one Commons featured category ranked locally. Latest preserves provider order and means newest eligible results found within its bounded newest-first raw window, not newest eligible files across the full catalog. Global usage is a popularity signal, not quality or rights evidence; usage_count_complete says whether its bounded continuation completed.

Strict Initial Policy

Automatic eligibility requires all of the following:

  • Commons media type exactly BITMAP and MIME exactly image/jpeg or image/png.
  • Positive dimensions no larger than 32,768 pixels per side and no more than 100 million pixels.
  • Exactly one normalized license: CC0-1.0, CC-BY-4.0, or CC-BY-SA-4.0.
  • A matching Creative Commons host, family, version, and path URL. A matching HTTP URL is canonicalized to HTTPS; missing or mismatched URLs are rejected.
  • Usable creator or custom-attribution evidence for BY and BY-SA. Own work credit never replaces a creator.
  • Nonempty UsageTerms must exactly identify the accepted family/version through a supported Commons label. NC, ND, multiple, ambiguous, and unknown terms are rejected.
  • Nonempty AttributionRequired must be a strict boolean consistent with the license: true for BY/BY-SA and false for CC0.
  • No explicit non-free status, contradictory BY/BY-SA Copyrighted=false, or nonempty Restrictions, Permission, or DeletionReason.

Older, ported, custom, ambiguous, multiple, NC, ND, FAL, GFDL, PDM, and generic public-domain metadata are outside 0.1's automatic scope. GIF, SVG, WebP, TIFF/BigTIFF, and AVIF are unsupported. Filtering them is not a statement that they are unusable; callers can review and handle them outside PhotoForge.

PhotoRights retains the normalized license and URL, raw license label, creator, custom attribution, selected attribution, source credit, permission, usage terms, raw attribution-required value, restrictions, copyrighted/non-free/deletion evidence, metadata retrieval time, and Commons guidance URL. Untrusted HTML fields are converted to bounded plain text. Creator/custom-attribution placeholders containing signals such as unknown, anonymous, own work, uploader, see description, or not provided are unusable. PhotoFile retains original URL, size, SHA-1, MIME, media type, dimensions, and current revision ID/timestamp when supplied.

Rights And Source Review

A Commons license label does not guarantee that the uploader owned every relevant right or entered metadata correctly. Review the current source page and underlying work before use.

Copyright permission does not settle privacy, publicity, personality, or moral rights. Check trademarks and trade dress; depicted artworks, sculptures, performances, buildings, and architecture; freedom-of-panorama rules; cultural-property, heritage, sacred-object, and community restrictions; model/property releases; and jurisdiction-specific requirements. Metadata can be stale, incomplete, or erroneous, and deletion discussions may emerge after archival.

For CC BY, preserve appropriate creator credit, title/source where appropriate, license link, and modification indication. For CC BY-SA, also determine whether adaptations trigger ShareAlike and how to satisfy it. CC0 does not eliminate non-copyright rights or the need to avoid misleading endorsement. The selected attribution is a starting point, not a guarantee of legal sufficiency. Review COMMONS_REUSE_URL and the exact source_url.

Archive Integrity

ArchiveOptions::default() writes to photos, does not skip, and caps each original at 256 MiB. Zero and limits above the hard 512 MiB retained-image maximum are rejected. Public verification enforces the same hard maximum even for externally supplied manifests. Immediately before every download, PhotoForge refetches exact canonical metadata by title, reruns eligibility, and requires the page ID to match discovery. The fresh summary is archived.

The layout is exactly:

<output>/<numeric-page-id>/
  photo.jpg | photo.png
  photoforge-photo.json

The format-1 manifest has an explicit photoforge ownership marker, full fresh PhotoSummary, fixed relative path, exact bytes, Commons SHA-1, local BLAKE3, and completion marker. This manifest is the portable persistent evidence. ArchivedPhoto::output and other PathBuf fields describe only the local runtime filesystem. verify_archive deliberately rejects every additional file, directory, symlink, special file, or sidecar inside this directory. Wrappers such as Thesa must place their own metadata outside the provider archive.

Downloads accept only the exact original HTTPS shape upload.wikimedia.org/wikipedia/commons/<1-hex>/<2-hex>/<filename>, without credentials, queries, fragments, altered ports, thumbnails, archives, temporary paths, or redirects. The source-page title and decoded original filename must match the canonical Commons title. Supplied Content-Length must be numeric and equal current Commons size. Streaming enforces both exact size and the caller cap, verifies Commons SHA-1, and flushes/syncs bytes.

JPEG validation requires FF D8 FF, safe segment lengths, one valid SOF matching metadata, a valid scan, EOI, and no non-padding trailer. PNG validation requires exact signature, one first 13-byte IHDR, matching dimensions/encoding fields, checked chunk lengths and CRCs, IDAT, final IEND, no trailing bytes, and no APNG chunks. After those structural checks, PhotoForge fully decodes the entire image with the declared JPEG/PNG decoder. Decoder format and dimensions must match, with explicit 32,768-pixel side, 100-million-pixel, 512 MiB allocation, and 512 MiB retained-file limits. The same full bounded decode runs during staging and public verification.

Each page ID has an fs2 destination lock with a photoforge-lock-v1 ownership header and append-only, fsynced transaction states. Blank, foreign, replaced, and hard-linked lock files are refused without modification; same-file compares the locked handle with the current path before state writes, and Unix/WASI or Windows-by-handle metadata supplies link count. Other platforms conservatively refuse transaction writes when link count cannot be established. Bounded state compaction retains the ownership header. Staging and recognized-destination replacement use output-root siblings, durable state transitions, backups, verification, rollback, root-directory sync, and conservative interrupted-state recovery. Recovery restores or removes an old backup only after full identity, exact-file, format, dimension, SHA-1, and BLAKE3 verification; tampered backups remain interrupted for operator review. Foreign or malformed destinations are preserved. A manifest-recognized corrupt archive may be repaired. With skip_existing, a skip is returned only after fresh nonvolatile metadata matches and full verification succeeds; every skip has a complete ArchivedPhoto in skipped_photos, and its progress event is emitted after aggregate counters are updated.

Archive calls deduplicate by numeric page ID, preserve first-selected order in selected_ids, attempt every unique selection, retain successful records, and accumulate independent ArchiveFailure values. Inspect failed/failures; a top-level Ok does not imply every photo succeeded. Progress phases are Starting, Downloading, Verifying, Installing, Skipped, Failed, and Completed; callbacks are synchronous and should remain fast.

Network And Scope

API traffic is pinned directly to https://commons.wikimedia.org/w/api.php, bypasses configured system proxies, includes maxlag=5, caps responses at 16 MiB, and uses bounded continuation. Only HTTP 429, HTTP 5xx, and Action API maxlag are retried, with at most three attempts inside the operation-wide ten-send and five-second retry bounds. Only numeric Retry-After is honored. Downloads are not retried automatically.

PhotoForge 0.1 has no Openverse, Google Images, social-media, stock-site, or other provider integration. It accepts no cookies, API keys, login sessions, or user credentials. It does not provide bulk mirroring, arbitrary URLs, scraping, content moderation, face recognition, safety classification, or legal clearance.

Development

The minimum supported Rust version is 1.85 (edition 2024).

TMPDIR=/home/x4/Workspace/AgenticSpace/.tmp CARGO_BUILD_JOBS=1 cargo fmt --check
TMPDIR=/home/x4/Workspace/AgenticSpace/.tmp CARGO_BUILD_JOBS=1 cargo test
TMPDIR=/home/x4/Workspace/AgenticSpace/.tmp CARGO_BUILD_JOBS=1 cargo clippy --all-targets -- -D warnings
TMPDIR=/home/x4/Workspace/AgenticSpace/.tmp CARGO_BUILD_JOBS=1 RUSTDOCFLAGS="-D warnings" cargo doc --no-deps
TMPDIR=/home/x4/Workspace/AgenticSpace/.tmp CARGO_BUILD_JOBS=1 cargo publish --dry-run --locked --allow-dirty