Skip to main content

Crate ngano

Crate ngano 

Source
Expand description

ngano is an open catalogue and streaming loader for African-language speech datasets.

It ships a snapshot of the catalogue inside the binary, so browsing works with no network at all, and it streams rows from any catalogued dataset that is hosted on Hugging Face, mapping heterogeneous columns onto one canonical schema shared with the Python and JavaScript SDKs.

§Browse the catalogue, offline

use ngano::{Catalogue, Filter};

let cat = Catalogue::bundled()?;
let hits = cat.datasets(&Filter::new().language("sna").commercial(true).task("ASR"));
for d in &hits {
    println!("{} ({})", d.name, d.licence);
}

let stats = cat.stats();
println!("{} datasets, {:.0} counted hours", stats.datasets, stats.hours);

§Languages

A language is a BCP 47 tag whose primary subtag is an ISO 639-3 code, so Shona is sna. A region subtag marks a country-specific variety, so Nigerian English is eng-NG and Mozambican Portuguese is por-MZ. Filter::language accepts a bare code, a tag or a name, in any case, and Catalogue::resolve_language shows what a given spelling resolves to. A bare code never picks up a regional variety unless you ask for it with Filter::include_varieties.

use ngano::{Catalogue, Filter};

let cat = Catalogue::bundled()?;
assert_eq!(cat.resolve_language("isiZulu"), vec!["zul".to_string()]);
let nigerian_english = cat.datasets(&Filter::new().language("eng-NG"));
assert!(!nigerian_english.is_empty());

§Stream rows

use futures::TryStreamExt;
use ngano::{Filter, Interleave, Loader};

let mut stream = Loader::new()
    .filter(Filter::new().language("sna").country("ZW").commercial(true))
    .split("train")
    .hf_token(std::env::var("HF_TOKEN").ok())
    .interleave(Interleave::RoundRobin)
    .limit(Some(1000))
    .stream()
    .await?;

while let Some(row) = stream.try_next().await? {
    println!("{:?}", row.transcript.as_deref());
    if let Some(audio) = &row.audio {
        let bytes = audio.read().await?; // nothing was fetched before this
        println!("{} bytes", bytes.len());
    }
}

§One repo, by name

Loader::repo streams a Hugging Face repo whether or not the catalogue lists it, which is the Rust spelling of load_dataset("google/fleurs", config="sw_ke") in the Python SDK.

let stream = ngano::Loader::new()
    .repo("google/fleurs")
    .config("sw_ke")
    .limit(Some(100))
    .stream()
    .await?;

§Hours figures

Some sources self-report totals of 20,000 hours or more. Those records carry Dataset::unverified_size and are excluded from every hours aggregate this crate computes.

§Runtime, features and MSRV

The async API expects a Tokio runtime, which is what reqwest uses for its own timers. Default features give rustls TLS; native-tls swaps it, blocking adds synchronous wrappers, and cli builds the ngano binary. The minimum supported Rust version is 1.82.

Modules§

blockingblocking
Synchronous wrappers, behind the blocking feature.

Structs§

AudioRef
A lazy handle on one audio clip.
Binding
Where one canonical field gets its value from.
Catalogue
An in-memory catalogue of datasets, countries and languages.
ColumnMap
A resolved mapping from one dataset’s columns onto the canonical schema.
Country
A country in the catalogue’s coverage map.
Dataset
One catalogue record: a published dataset, as described by its source.
FieldMap
The parsed contents of field_map.json.
Filter
A declarative dataset filter.
Language
A language in the catalogue’s coverage map, keyed on its BCP 47 tag.
LanguageCode
One entry of the language registry: a canonical tag and everything known about it, including every catalogue spelling that resolves to it.
LanguageRegistry
The language registry: every known tag, with the name and alias tables used to resolve one.
Loader
Builder for a row stream.
Row
One utterance in the canonical schema shared by the Python, JavaScript and Rust SDKs.
RowStream
A stream of canonical rows across one or more datasets.
Stats
Aggregates computed over a Catalogue.

Enums§

Interleave
How rows from several datasets are combined into one stream.
NganoError
Every fallible operation in ngano returns this error.

Constants§

CANONICAL_FIELDS
Canonical fields, in the precedence order used when two fields could claim the same column.
DEFAULT_API_BASE
The public ngano API base URL.
DEFAULT_HF_BASE
The public datasets-server base URL.
MAX_PAGE_ROWS
The server refuses length above this, so pages are capped here.
VERSION
The version of this crate.

Functions§

canonicalise_tag
Put a tag into canonical case: lowercase primary subtag, uppercase region. None when the value is not tag-shaped at all.
normalise_column
Lowercase a name and strip underscores and hyphens, so WAV_Path, wav-path and wavpath all compare equal.

Type Aliases§

Result
A short alias for results produced by this crate.