wikrs/lib.rs
1//! **wikrs** — fast, honest wikitext extraction and parsing.
2//!
3//! Turns MediaWiki wikitext (the markup inside Wikipedia XML dumps) into clean
4//! plain text or a structured AST, and **emits a diagnostic when it hits input
5//! it can't faithfully handle instead of silently corrupting the output**.
6//! Validated on the full English Wikipedia (7.19M articles, 98.0% of pages
7//! convert with zero residual markup).
8//!
9//! # Quick start
10//!
11//! ```
12//! // Parse wikitext into an AST + diagnostics, then render plain text.
13//! let parsed = wikrs::parser::parse("'''Earth''' is a [[Planet|planet]].");
14//! assert!(parsed.diagnostics.is_empty());
15//! assert_eq!(wikrs::render::plain(&parsed.nodes), "Earth is a planet.");
16//!
17//! // Or the Stage 1 one-shot stripper (fast, lossy, no diagnostics).
18//! assert_eq!(
19//! wikrs::extract::strip("'''Earth''' is a [[Planet|planet]]."),
20//! "Earth is a planet."
21//! );
22//! ```
23//!
24//! Reading a whole dump ([`dump::open`], or [`dump::open_multistream`] for
25//! parallel bz2 decoding) yields [`dump::Page`]s whose `text` feeds the same
26//! two entry points. The `wikrs` CLI wraps exactly this pipeline.
27//!
28//! Pre-1.0: the API surface is the modules documented below; items marked
29//! `#[doc(hidden)]` are internal plumbing with no stability promise.
30
31#![warn(missing_docs)]
32
33pub mod ast;
34pub mod diag;
35pub mod dump;
36pub mod extract;
37pub mod parser;
38pub mod render;
39
40// Internal plumbing, public only for the CLI / dev tooling (xtask, tests).
41// No semver promise — do not build on these.
42#[doc(hidden)]
43pub mod diff;
44#[doc(hidden)]
45pub mod mdnorm;
46#[doc(hidden)]
47pub mod output;
48
49// Crate-internal machinery (tokenizer feeds parser; entities feed render/strip).
50pub(crate) mod entities;
51pub(crate) mod tokenizer;