Expand description
ngano is an open catalogue and streaming loader for African-language
speech datasets.
It ships a snapshot of the catalogue inside the binary, so browsing works with no network at all, and it streams rows from any catalogued dataset that is hosted on Hugging Face, mapping heterogeneous columns onto one canonical schema shared with the Python and JavaScript SDKs.
§Browse the catalogue, offline
use ngano::{Catalogue, Filter};
let cat = Catalogue::bundled()?;
let hits = cat.datasets(&Filter::new().language("sna").commercial(true).task("ASR"));
for d in &hits {
println!("{} ({})", d.name, d.licence);
}
let stats = cat.stats();
println!("{} datasets, {:.0} counted hours", stats.datasets, stats.hours);§Languages
A language is a BCP 47 tag whose primary subtag is an ISO 639-3 code, so
Shona is sna. A region subtag marks a country-specific variety, so
Nigerian English is eng-NG and Mozambican Portuguese is por-MZ.
Filter::language accepts a bare code, a tag or a name, in any case, and
Catalogue::resolve_language shows what a given spelling resolves to. A
bare code never picks up a regional variety unless you ask for it with
Filter::include_varieties.
use ngano::{Catalogue, Filter};
let cat = Catalogue::bundled()?;
assert_eq!(cat.resolve_language("isiZulu"), vec!["zul".to_string()]);
let nigerian_english = cat.datasets(&Filter::new().language("eng-NG"));
assert!(!nigerian_english.is_empty());§Stream rows
use futures::TryStreamExt;
use ngano::{Filter, Interleave, Loader};
let mut stream = Loader::new()
.filter(Filter::new().language("sna").country("ZW").commercial(true))
.split("train")
.hf_token(std::env::var("HF_TOKEN").ok())
.interleave(Interleave::RoundRobin)
.limit(Some(1000))
.stream()
.await?;
while let Some(row) = stream.try_next().await? {
println!("{:?}", row.transcript.as_deref());
if let Some(audio) = &row.audio {
let bytes = audio.read().await?; // nothing was fetched before this
println!("{} bytes", bytes.len());
}
}§One repo, by name
Loader::repo streams a Hugging Face repo whether or not the catalogue
lists it, which is the Rust spelling of load_dataset("google/fleurs", config="sw_ke") in the Python SDK.
let stream = ngano::Loader::new()
.repo("google/fleurs")
.config("sw_ke")
.limit(Some(100))
.stream()
.await?;§Hours figures
Some sources self-report totals of 20,000 hours or more. Those records carry
Dataset::unverified_size and are excluded from every hours aggregate this
crate computes.
§Runtime, features and MSRV
The async API expects a Tokio runtime, which is what reqwest uses for its
own timers. Default features give rustls TLS; native-tls swaps it,
blocking adds synchronous wrappers, and cli builds the ngano binary.
The minimum supported Rust version is 1.82.
Modules§
- blocking
blocking - Synchronous wrappers, behind the
blockingfeature.
Structs§
- Audio
Ref - A lazy handle on one audio clip.
- Binding
- Where one canonical field gets its value from.
- Catalogue
- An in-memory catalogue of datasets, countries and languages.
- Column
Map - A resolved mapping from one dataset’s columns onto the canonical schema.
- Country
- A country in the catalogue’s coverage map.
- Dataset
- One catalogue record: a published dataset, as described by its source.
- Field
Map - The parsed contents of
field_map.json. - Filter
- A declarative dataset filter.
- Language
- A language in the catalogue’s coverage map, keyed on its BCP 47 tag.
- Language
Code - One entry of the language registry: a canonical tag and everything known about it, including every catalogue spelling that resolves to it.
- Language
Registry - The language registry: every known tag, with the name and alias tables used to resolve one.
- Loader
- Builder for a row stream.
- Row
- One utterance in the canonical schema shared by the Python, JavaScript and Rust SDKs.
- RowStream
- A stream of canonical rows across one or more datasets.
- Stats
- Aggregates computed over a
Catalogue.
Enums§
- Interleave
- How rows from several datasets are combined into one stream.
- Ngano
Error - Every fallible operation in
nganoreturns this error.
Constants§
- CANONICAL_
FIELDS - Canonical fields, in the precedence order used when two fields could claim the same column.
- DEFAULT_
API_ BASE - The public ngano API base URL.
- DEFAULT_
HF_ BASE - The public datasets-server base URL.
- MAX_
PAGE_ ROWS - The server refuses
lengthabove this, so pages are capped here. - VERSION
- The version of this crate.
Functions§
- canonicalise_
tag - Put a tag into canonical case: lowercase primary subtag, uppercase region.
Nonewhen the value is not tag-shaped at all. - normalise_
column - Lowercase a name and strip underscores and hyphens, so
WAV_Path,wav-pathandwavpathall compare equal.
Type Aliases§
- Result
- A short alias for results produced by this crate.