Skip to main content

Module audio

Module audio 

Source
Expand description

Audio intelligence + call recording (PROCESSOR-DESIGN §6.1).

This module is the home for the audio-intelligence processors that sit on the hot media path inside the cascaded pipeline — VAD, end-of-turn, noise filters, and mixers — alongside the existing call recorder. The analyzer traits (VadAnalyzer/TurnAnalyzer) are frozen in crate::service; the ONNX (ort) impls compile only under the vad-ort feature so flowcat-core stays dependency-light by default.

Module map:

  • vad — VadAnalyzer impls (Silero behind vad-ort, a mock always).
  • turn — TurnAnalyzer impls (Smart-Turn behind vad-ort, a mock always).
  • strategy — turn start/stop/mute + interruption strategies (pure logic).
  • filter — RNNoise/Koala audio-filter processors (Krisp deferred).
  • mixer — audio mixers.

§Call recording (existing)

Tap both legs (caller + bot), mono-mix, render to WAV bytes.

The recorder accumulates audio from each leg and produces a single mono WAV (via hound) suitable for upload to the object store (see DESIGN.md “Audio path”).

The two legs typically arrive at different sample rates (caller ≈ 8 kHz μ-law decoded, bot ≈ 24 kHz from Gemini Live), so each leg is buffered with its own rate and resampled to the recorder’s common sample_rate only at render time. Mixing is a plain per-sample sum clamped back into i16.

Re-exports§

pub use filter::AudioFilter;
pub use filter::AudioFilterProcessor;
pub use filter::IdentityFilter;
pub use mixer::AudioMixer;
pub use mixer::MixerProcessor;
pub use mixer::SilenceMixer;
pub use mixer::SoundfileMixer;
pub use strategy::AlwaysInterrupt;
pub use strategy::AlwaysMute;
pub use strategy::ExternalTurnStart;
pub use strategy::FirstSpeechMute;
pub use strategy::FunctionCallMute;
pub use strategy::InterruptionStrategy;
pub use strategy::MinWordsInterrupt;
pub use strategy::MinWordsTurnStart;
pub use strategy::MuteStrategy;
pub use strategy::MuteUntilFirstBotComplete;
pub use strategy::SpeechTimeoutStop;
pub use strategy::TurnAnalyzerStop;
pub use strategy::TurnStartDecision;
pub use strategy::TurnStartStrategy;
pub use strategy::TurnStopDecision;
pub use strategy::TurnStopStrategy;
pub use strategy::VadTurnStart;
pub use turn::MockTurn;
pub use turn::TurnProcessor;
pub use turn::TurnSilenceTracker;
pub use vad::window_volume;
pub use vad::MockVad;
pub use vad::VadProcessor;
pub use vad::VadStateMachine;

Modules§

filter
Input-leg audio-filter processors (noise suppression).
mixer
Output-leg audio mixers.
strategy
Turn start/stop/mute + interruption strategies (pure framework logic).
turn
End-of-turn / semantic-completion analyzer impls + the TurnProcessor.
vad
Voice-activity-detector impls + the VadProcessor.

Structs§

AudioRecorder
Accumulates inbound (caller) and outbound (bot) audio on a shared wall-clock timeline and mixes them down to a single mono WAV byte buffer.