Module av 

oxideav-core types as oxinfer’s in-memory media (the oxideav feature): decoded pictures, audio and video as VideoFrame / AudioFrame plus the CodecParameters that describe them, converted to what the towers take (RGB8, mono f32 at the model’s rate), and RGB output (Stable Diffusion) back to a frame.

Only oxideav-core is used: no codec, container or file code, so these work in a build without the media feature, e.g. inside an oxideav pipeline filter that already has decoded frames.

oxideav frames carry no geometry or format of their own; CodecParameters gives width, height and pixel_format (video), sample_rate, channels and sample_format (audio). YCbCr is converted with the matrix and range of the colour signal (the frame’s VideoFrame::color_signal record, else CodecParameters::color_signal): unspecified means BT.601, limited range (full for the YuvJ* formats). The arithmetic (Q15 fixed point, nearest-neighbour chroma, deep samples narrowed to their top 8 bits) is the one of oxideav-pixfmt, so every format both convert gives the same RGB bytes as oxideav_image::Image::to_rgb8, which the path-based inputs use.

Structs

AudioClip

Decoded audio: frames in stream order and the stream parameters describing them (sample_rate, channels, sample_format). An audio attachment, or Whisper input.

Picture

A decoded picture: one frame and the stream parameters describing it (width, height, pixel_format, optionally the colour signal). An image attachment, or a decision’s image.

VideoClip

Decoded video: frames in presentation order, the stream parameters describing them and the time base of their pts. Frames are sampled as the model’s video processor does (see crate::video::VideoOptions); the prompt timestamps of the sampled frames come from their pts when every frame has one and time_base is set, otherwise from their index and the frame rate. The frame rate is params.frame_rate, else the one the pts imply, else DEFAULT_FPS.

Functions

audio_frames_to_mono

Decoded audio frames to mono f32 in [-1, 1] (channels averaged) at rate Hz, resampled with crate::audio::resample when the stream’s rate differs. params gives sample_rate, channels (default 1) and sample_format (inferred from the frame layout when missing). Every SampleFormat, interleaved or planar, is supported.

complete_params

The parameters a decoder left out (width, height, pixel_format), inferred from the frame’s plane layout: some decoders do not report their output format in the stream parameters.

mono_audio_frame

One frame of mono F32 samples at sample_rate with presentation time pts (in samples, i.e. a time base of 1 / sample_rate) and the parameters describing it: text-to-speech output (crate::tts), whole or streamed.

rgb_frame

A tightly packed Rgb24 frame (pts 0) and its rawvideo parameters (width, height, pixel_format), e.g. to hand a generated image to an oxideav encoder. The colour signal is left unspecified, as oxideav-image does for raw RGB: an sRGB signal would carry the identity matrix, which an RGB to YCbCr conversion downstream would then apply.

video_frame_to_rgb

Converts a decoded frame to tightly packed RGB8. params gives width, height and pixel_format; when one is missing it is inferred from the plane layout as for decoded files (complete_params). Alpha is dropped (not composited), deeper samples are narrowed to their top 8 bits, palettes (Pal8) are expanded. YCbCr uses the colour signal’s matrix (BT.601 when unspecified; BT.709, FCC, BT.470BG, SMPTE 170M / 240M, BT.2020 NCL and identity are implemented) and range (limited unless signalled, or a YuvJ* format). Formats without an RGB meaning here (CMYK, float, …) are an error naming the format.