Crate ruwex 

ruwex — a fast Rust rewrite of wikiextractor, extracting and cleaning text from Wikimedia XML dumps.

The crate is usable as a library; the wikiextractor binary is a thin CLI wrapper over pipeline::run. See the README for usage and the compatibility notes.

Re-exports

pub use config::ExtractorConfig;

pub use config::OutputFormat;

pub use config::default_namespaces;

pub use config::default_workers;

pub use config::parse_size;

pub use dump::reader::PageSource;

pub use dump::Page;

pub use dump::SiteInfo;

pub use expand::LazyTemplateSource;

pub use expand::TemplateDb;

pub use expand::TemplateSource;

pub use output::format::render_page;

pub use output::writer::DocSink;

pub use output::writer::ShardedWriter;

pub use output::writer::StdoutSink;

pub use pipeline::Stats;

pub use pipeline::run;

pub use pipeline::run_with_templates;

pub use title_index::TitleIndex;

Modules

clean

Wikitext cleaning: markup → plain text (or simple HTML), a faithful port of wikiextractor’s clean() and compact().

config

dump

Input layer: reading pages out of MediaWiki XML dumps in their various packagings (plain XML, single-stream bz2, seekable multistream bz2).

expand

Template expansion, ported from wikiextractor’s Extractor.expandTemplates: magic words, parser functions, and template instantiation against a TemplateDb. With an empty database (--no-templates) every ordinary inclusion resolves to the empty string, exactly like the original.

output

Output layer: document formatting and sharded file writing.

pipeline

Orchestration: source → worker pool → order-preserving writer.

title_index

Fast page-title lookup for multistream dumps.

tools

The auxiliary command-line tools shipped alongside wikiextractor: extractPage (pull a single page’s raw XML from a dump) and cirrus-extract (extract from CirrusSearch JSON dumps).

Enums

Error

Type Aliases

Result