Module perf 

Performance recording and analysis for streaming LLM responses

This module provides mechanisms to record streaming responses with minimal overhead during collection, then analyze the recorded data for performance insights.

Modules

logprobs

Module for recording logprobs from a streaming response.

Structs

RecordedStream

Container for recorded streaming responses. This forms the core object on which analysis is performed.

RecordingStream

Recording stream that wraps an AsyncEngineStream and records responses Following the pattern of ResponseStream for AsyncEngine compatibility

TimestampedResponse

A response wrapper that adds timing information with minimal overhead

Enums

RecordingMode

Recording mode determines how the recorder behaves with the stream

Traits

CapacityHint

Trait for requests that can provide hints about expected response count This enables capacity pre-allocation for better performance

Functions

record_response_stream

Create a recording stream from ResponseStream (convenience wrapper)

record_stream

Create a recording stream that wraps an AsyncEngineStream Returns a pinned stream and a receiver for the recorded data

record_stream_with_capacity

Create a recording stream with capacity hint

record_stream_with_context

Create a recording stream from a raw stream and context Returns a pinned stream and a receiver for the recorded data

record_stream_with_context_and_capacity

Create a recording stream from a raw stream and context with capacity hint

record_stream_with_request_hint

Create a recording stream with capacity hint from request

Type Aliases

RecordedStreamReceiver

Type alias for a receiver of recorded stream data

RecordingResult

Type alias for the return type of recording functions