1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
//! The Whisper stream driver: audio pushed in chunks, timed transcript out.
//!
//! What a caller touches lives here: the driver and its config, the stream
//! context, the [`WhisperEngine`] they run, the bundle, and the token
//! layout. The driver and the context are the
//! [`asr_driver`](crate::kits::speech::asr_driver) core's
//! [`AsrStreamDriver`](crate::kits::speech::asr_driver::AsrStreamDriver) and
//! [`AsrStreamContext`](crate::kits::speech::asr_driver::AsrStreamContext) over
//! that engine; the parts that do not depend on Whisper (scheduling, the
//! emission policy and what it emits, the clock, the voice-activity gate and
//! its regions, the level) live there. [`support`] holds the internals
//! underneath, and nothing in it is part of the driver's API.
//!
//! # Driver and context
//!
//! A transcription is split in two. The [`WhisperStreamDriver`] is the
//! shared, immutable half: the model and its token layout (a
//! [`WhisperBundle`] behind an `Arc`), the mel front end, the logit
//! filters, the decode settings, and the voice-activity model when one is
//! attached. A [`WhisperStreamContext`] is one stream in progress, and the
//! only stateful type here: the front end's carry, the frames not yet
//! decoded, the seek pointer, the committed transcript, the stream's clock,
//! its level and its voice-activity state.
//!
//! Stream state is injected, never owned by the model. The context holds
//! it, and the driver and the model are only read, so one driver serves any
//! number of streams in one process, each in a context of its own, and
//! [`advance_ready`](crate::kits::speech::asr_driver::advance_ready) batches
//! the decodes of many contexts through the one model. The same split runs
//! underneath: [`SileroVad`](crate::kits::speech::silero_vad::SileroVad) and
//! its [`SileroVadContext`](crate::kits::speech::silero_vad::SileroVadContext),
//! the mel converter and its
//! [`PerceptiveAudioConversionContext`](crate::ops::signal::perceptive_audio::PerceptiveAudioConversionContext),
//! and the text decoder and the
//! [`TextDecoderCache`](crate::kits::speech::whisper::blocks::TextDecoderCache)
//! each decode opens.
//!
//! One driver serves three deployments, configured rather than forked:
//! server-batch offline inference; conservative real time, for a
//! programmatic consumer; and best-effort real time, for a human reading
//! during the utterance. The three
//! [`EmissionPolicy`](crate::kits::speech::asr_driver::EmissionPolicy) presets
//! are those three.
//!
//! # Lifecycle
//!
//! 1. A [`WhisperBundle`]: from a name, through
//! [`default_whisper_factory`](crate::kits::speech::whisper::pretrained::default_whisper_factory)
//! and a [`PretrainedCache`](crate::data::pretrained::PretrainedCache), or
//! over a model already in hand.
//! 2. A [`WhisperStreamDriverConfig`] (language, task, timestamps, search,
//! [`EmissionPolicy`](crate::kits::speech::asr_driver::EmissionPolicy),
//! fallback ladder), built into a driver over the bundle by
//! [`init_from_bundle`](WhisperStreamDriverConfig::init_from_bundle).
//! [`with_vad`](WhisperStreamDriver::with_vad) attaches a voice-activity
//! model, which the real-time presets need.
//! 3. One context per stream, from
//! [`new_context`](WhisperStreamDriver::new_context), given the stream's
//! [`StreamClock`](crate::kits::speech::asr_driver::StreamClock) and a
//! [`LevelPolicy`](crate::kits::speech::asr_driver::LevelPolicy).
//! 4. Samples in through [`write_read`](WhisperStreamContext::write_read),
//! [`TranscriptEvent`](crate::kits::speech::asr_driver::TranscriptEvent)s
//! out; [`end_read`](WhisperStreamContext::end_read) ends the stream and
//! decodes what is left.
//!
//! # The `write_read` fold
//!
//! [`write_read`](WhisperStreamContext::write_read) is a fold over the
//! samples it is handed, in two halves.
//! [`write`](WhisperStreamContext::write) appends them to staging, moves
//! whole hops through the mel front end, offers the new frames to the
//! stream's level and appends them to the frame ring, and runs the
//! voice-activity model over whole chunks, stepping its gate.
//! [`read`](WhisperStreamContext::read) runs every decode the emission
//! policy says is due, commits each, and then drafts if a draft is due. The
//! halves are separate so that a batch of streams can be written one at a
//! time and read together by
//! [`advance_ready`](crate::kits::speech::asr_driver::advance_ready).
//!
//! The hop alignment never reaches the caller. Staging holds whatever is
//! not yet a whole hop, so a sound card's 480- or 1024-sample buffers are
//! written as they come.
//!
//! No decode state crosses a window boundary: each decode opens its own
//! text-decoder cache. What carries from one window to the next is
//! host-side: the seek pointer, the prompt carry (the tail of the committed
//! transcript), the language once detected, and the clock. The ring holds
//! every frame from the seek pointer onward and nothing before it.
//!
//! # Emission: commit and draft
//!
//! The policy has two axes,
//! [`DecodeTriggers`](crate::kits::speech::asr_driver::DecodeTriggers) (when a
//! decode runs) and a
//! [`CommitRule`](crate::kits::speech::asr_driver::CommitRule) (when its output
//! is final), and the output has two
//! variants, [`Committed`](crate::kits::speech::asr_driver::TranscriptEvent::Committed) and
//! [`Draft`](crate::kits::speech::asr_driver::TranscriptEvent::Draft). **A
//! `window_full` or `endpoint` decode commits, per the commit rule; an
//! `interval` decode drafts.** What the three deployments differ by falls out
//! of those two sentences.
//!
//! A draft covers the audio after the last commit and replaces the previous
//! draft whole, so there is no retraction protocol, no sequence numbers,
//! and never two drafts to reconcile.
//!
//! | Preset | Triggers | Commit rule | Drafts |
//! |---|---|---|---|
//! | [`offline`](crate::kits::speech::asr_driver::EmissionPolicy::offline) | `window_full` | `Complete` | never |
//! | [`conservative`](crate::kits::speech::asr_driver::EmissionPolicy::conservative) | `window_full`, `endpoint` | `LastTimestamp` | with timestamps on, a decode's unfinished tail |
//! | [`responsive`](crate::kits::speech::asr_driver::EmissionPolicy::responsive) | `window_full`, `endpoint`, `interval` (600 ms) | `LastTimestamp` | as conservative, plus one per interval of speech |
//!
//! ## What a decode may touch
//!
//! A decode reads the context through `&self`: cutting its frames from the
//! ring, packaging them against the stream's level
//! ([`reference`](crate::kits::speech::asr_driver::LevelTracker::reference)),
//! the model's forward, and the prompt. Only a commit moves the stream (the
//! seek pointer, the transcript, the prompt carry); a draft moves nothing but
//! its own pacing. So a draft is the same decode as a commit, not a second code
//! path, and adding the `interval` trigger cannot change what is
//! committed: `responsive` commits exactly what `conservative` commits on
//! the same audio, which the context's tests pin. That holds on a driver
//! that detects the language too: only a commit fixes a stream's language,
//! and a draft before the first commit detects one for its own decode and
//! keeps nothing.
//!
//! # The stream clock
//!
//! Every context carries a
//! [`StreamClock`](crate::kits::speech::asr_driver::StreamClock): sorted
//! `(sample, time)` anchors and a sample rate, mapping a sample index to media
//! time. A bare stream
//! gets [`StreamClock::uniform`](crate::kits::speech::asr_driver::StreamClock::uniform), one anchor at `(0, 0.0)`, which
//! reproduces exactly the arithmetic upstream does from its seek pointer.
//! Everything richer (a capture callback's timestamp, a container's
//! presentation time, a dropped buffer becoming a new anchor rather than a
//! permanent shift) is an addition to that, not a departure:
//! [`anchor_write_read`](WhisperStreamContext::anchor_write_read) anchors
//! the next sample before writing. Making the general case the only case
//! costs nothing, because the bare case *is* the general case with one
//! anchor.
//!
//! A segment's times are its frames put on the clock: timestamp token `i`
//! of a window that starts at sample `s` is `clock.time_at(s + i * grid)`,
//! where `grid` is the [`encoder_grid`](WhisperStreamDriver::encoder_grid),
//! 320 samples at 16 kHz.
//!
//! # The level: where the clamp's reference comes from
//!
//! Whisper floors its log-mels 8 dB below a reference maximum, and upstream
//! takes that maximum over the **whole clip** before cutting windows. A
//! stream cannot see the whole clip, so where the reference comes from is
//! the stream's [`LevelPolicy`](crate::kits::speech::asr_driver::LevelPolicy),
//! the same policy every engine on the core uses; Whisper's statistic is a
//! [`MaxLevel`](crate::kits::speech::asr_driver::MaxLevel). The context knows
//! only two points: every arriving frame is offered to its
//! [`LevelTracker`](crate::kits::speech::asr_driver::LevelTracker) once
//! ([`observe`](crate::kits::speech::asr_driver::LevelTracker::observe), `&mut
//! self`), and a window's reference is asked for just before packaging
//! ([`reference`](crate::kits::speech::asr_driver::LevelTracker::reference),
//! `&self`). That split is forced, not chosen: a decode reads the context
//! through `&self`, so the call it makes on the way cannot mutate either, and
//! packaging a window for a draft leaves the level as the commit will find it.
//!
//! Two policies cover four choices of reference.
//! [`Running`](crate::kits::speech::asr_driver::LevelPolicy::Running) in a
//! context written the whole clip before its first read is upstream's global
//! reference; written as audio arrives, it is the running one; in a context
//! opened per region, it is the per-region one.
//! [`PerWindow`](crate::kits::speech::asr_driver::LevelPolicy::PerWindow)
//! floors each window against its own maximum, as
//! [`package_mels`](crate::kits::speech::whisper::blocks::WhisperFrontEndConfig::package_mels)
//! does. Packaging itself, and the clamp range, belong to the model's front
//! end
//! ([`package_window`](crate::kits::speech::whisper::blocks::WhisperFrontEndConfig::package_window));
//! the level supplies only the reference.
//!
//! # Voice activity: regions as sub-streams
//!
//! Under a policy with the `endpoint` trigger, the context runs the
//! attached voice-activity model over every chunk, and a
//! [`VoiceActivityFilter`](crate::kits::speech::asr_driver::VoiceActivityFilter) (from the driver's [`VoiceActivityFilterConfig`](crate::kits::speech::asr_driver::VoiceActivityFilterConfig))
//! turns its speech probabilities into
//! [`SpeechRegion`](crate::kits::speech::asr_driver::SpeechRegion)s. A region
//! the filter closes is padded, snapped outward onto the encoder grid so that
//! its edges fall on timestamp positions, and becomes a decode unit of its
//! own: its frames are cut from the ring, decoded a window at a time, and
//! committed with times off the parent stream's clock. That is
//! region-as-stream without a second stream object. Under `window_full` a
//! full window is then decoded only while speech is in progress inside it;
//! a full window of silence is skipped, not decoded, which is the gating
//! half of voice activity. A policy without `endpoint` ignores an attached
//! model.
//!
//! A caller that decodes regions as streams of their own, say from the
//! whole-clip
//! [`speech_regions`](crate::kits::speech::asr_driver::VoiceActivityFilterConfig::speech_regions),
//! opens a context per region with the region's
//! [`clock`](crate::kits::speech::asr_driver::SpeechRegion::clock): the
//! parent's, sliced, so the region's times stay absolute.
pub use *;
pub use *;
pub use *;
pub use *;