1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
//! # Pretrained Whisper models
//!
//! From a model's name to a loaded model, through the data layer's
//! pretrained machinery ([`data::pretrained`](crate::data::pretrained)).
//! The prefab map is always here: a geometry needs no features. The tables,
//! the maps, the hook and the factory need the `store_pytorch` and `cache`
//! features.
//!
//! # The index
//!
//! [`default_whisper_factory`] is the index a caller holds, over
//! [`default_whisper_providers`] in search order:
//!
//! - [`WELL_KNOWN_TABLE`]: the checkpoints bunsen knows by name, as
//! `well-known:openai/base`, `openai/base`, or `base`. Its one group,
//! [`OPENAI`], is upstream's `_MODELS`, with upstream's aliases (`large`,
//! `turbo`).
//! - With the `whisper-weights` feature, the bundled table:
//! `bundled:openai/base`, served in place from `bunsen-bundled-whisper`'s
//! build directory.
//! - The data layer's [`HfProvider`](crate::data::pretrained::HfProvider):
//! `hf:openai/whisper-large-v3` for a Hugging Face repo, its safetensors
//! checkpoint under [`CHECKPOINT`]. It lists nothing.
//!
//! A caller with a provider of its own, a mirror say, adds it to the
//! default factory with
//! [`with_provider`](crate::data::pretrained::PretrainedFactory::with_provider);
//! the hook stays the kit's, chosen by the row, never the caller's.
//!
//! # Rows, prefabs and maps
//!
//! A row names its prefab in [`WHISPER_PREFABS`], the geometry its
//! checkpoint must have, and fuses a checkpoint map from
//! [`OPENAI_CHECKPOINTS`] with the vocabulary map its token layout
//! selects ([`WhisperVocabulary`]). A prefab is a shape without weights,
//! so a name means a shape before any bytes are fetched; a map is where
//! the bytes come from and the digest that pins them. A path arrives as a
//! one-resource map with no prefab, and gets its vocabulary from the same
//! rule, [`WhisperVocabulary::for_layout`].
//!
//! # Loading
//!
//! [`load_bundle`](crate::data::pretrained::PretrainedFactory::load_bundle)
//! is the whole pathway from a name to a
//! [`WhisperBundle`](crate::kits::speech::whisper::driver::WhisperBundle)
//! behind an `Arc`. The factory resolves the name to a
//! [`Deferred`](crate::data::pretrained::Deferred) model carrying the
//! kit's hook, [`WhisperConstruct`], with the [`WhisperReader`] its
//! checkpoint's `kind` names (`OpenAI`'s `.pt`, or `transformers`'
//! `model.safetensors`, one file or shards). The hook's plan brings the
//! checkpoint local, scans it, checks it against the geometry the prefab
//! promised, and settles the vocabulary; the cache brings the rest local;
//! and the hook's construct reads the checkpoint into a model at the
//! precision it ships in and the vocabulary into ranks. A checkpoint on
//! disk is not the factory's, but becomes a deferred model the same way,
//! through [`Deferred::from_map`](crate::data::pretrained::Deferred::from_map).
//!
//! # Example
//!
//! The index answers without loading anything:
//!
//! ```rust
//! # #[cfg(all(feature = "store_pytorch", feature = "cache"))] {
//! use bunsen::kits::speech::whisper::{
//! driver::WhisperSpecialIds,
//! pretrained::{
//! VOCABULARY,
//! WHISPER_PREFABS,
//! WhisperVocabulary,
//! default_whisper_factory,
//! },
//! };
//!
//! let factory = default_whisper_factory()?;
//!
//! // `large` is upstream's alias for `large-v3`, and the row names its
//! // prefab: the geometry the checkpoint must scan to.
//! let (_, row) = factory.lookup("large")?;
//! assert_eq!(row.name, "openai/large-v3");
//! let prefab =
//! WHISPER_PREFABS.try_lookup_prefab(row.prefab.as_deref().unwrap())?;
//! let geometry = prefab.to_config().geometry();
//! assert_eq!((geometry.n_mels, geometry.vocab_size), (128, 51866));
//!
//! // The row's vocabulary is the one that geometry's token layout selects.
//! let ids = WhisperSpecialIds::from_vocab_size(geometry.vocab_size)?;
//! let rule = WhisperVocabulary::for_layout(&ids).map().to_map();
//! assert_eq!(
//! row.resources.try_get(VOCABULARY)?.file,
//! rule.try_get(VOCABULARY)?.file,
//! );
//! # }
//! # Ok::<(), bunsen::errors::BunsenError>(())
//! ```
pub use *;
pub use *;
pub use *;
pub use *;
pub use *;
pub use *;
pub use *;
pub use *;