Struct LLMDriftProfile 

pub struct LLMDriftProfile {
    pub config: LLMDriftConfig,
    pub metrics: Vec<LLMMetric>,
    pub scouter_version: String,
    pub workflow: Workflow,
    pub metric_names: Vec<String>,
}

Fields

config: LLMDriftConfig

metrics: Vec<LLMMetric>

scouter_version: String

workflow: Workflow

metric_names: Vec<String>

Implementations

impl LLMDriftProfile

pub fn new( config: LLMDriftConfig, metrics: Vec<LLMMetric>, workflow: Option<Bound<'_, PyAny>>, ) -> Result<LLMDriftProfile, ProfileError>

Create a new LLMDriftProfile LLM evaluations are run asynchronously on the scouter server.

Logic flow:

1. If a user provides only a list of metrics, a workflow will be created from the metrics using from_metrics method.
2. If a user provides a workflow, It will be parsed and validated using from_workflow method.
   • The user must also provide a list of metrics that will be used to evaluate the output of the workflow.
   • The metric names must correspond to the final task names in the workflow In addition, baseline metrics and threshold will be extracted from the LLMMetric.

Arguments

• config - LLMDriftConfig - The configuration for the LLM drift profile
• metrics - Option<Bound<’_, PyList>> - Optional list of metrics that will be used to evaluate the LLM
• workflow - Option<Bound<’_, PyAny>> - Optional workflow to use for the LLM drift profile

Returns

• Result<Self, ProfileError> - The LLMDriftProfile

Errors

• ProfileError::MissingWorkflowError - If the workflow is

pub fn __str__(&self) -> String

pub fn model_dump_json(&self) -> String

pub fn model_dump(&self, py: Python<'_>) -> Result<Py<PyDict>, ProfileError>

pub fn save_to_json( &self, path: Option<PathBuf>, ) -> Result<PathBuf, ProfileError>

pub fn model_validate(data: &Bound<'_, PyDict>) -> LLMDriftProfile

pub fn model_validate_json(json_string: String) -> LLMDriftProfile

pub fn from_file(path: PathBuf) -> Result<LLMDriftProfile, ProfileError>

pub fn update_config_args( &mut self, space: Option<String>, name: Option<String>, version: Option<String>, alert_config: Option<LLMAlertConfig>, ) -> Result<(), TypeError>

pub fn create_profile_request(&self) -> Result<ProfileRequest, TypeError>

Create a profile request from the profile

impl LLMDriftProfile

pub fn from_metrics( config: LLMDriftConfig, metrics: Vec<LLMMetric>, ) -> Result<LLMDriftProfile, ProfileError>

Creates an LLMDriftProfile from a configuration and a list of metrics.

Arguments

• config - LLMDriftConfig - The configuration for the LLM
• metrics - Vec - The metrics that will be used to evaluate the LLM
• scouter_version - Option - The version of scouter that the profile is created with.

Returns

• Result<Self, ProfileError> - The LLMDriftProfile

pub fn from_workflow( config: LLMDriftConfig, workflow: Workflow, metrics: Vec<LLMMetric>, ) -> Result<LLMDriftProfile, ProfileError>

Creates an LLMDriftProfile from a workflow and a list of metrics. This is useful when the workflow is already defined and you want to create a profile from it. This is also for more advanced use cases where the workflow may need to execute many dependent tasks. Because of this, there are a few requirements:

1. All beginning tasks in the the workflow must have “input” and/or “output” parameters defined.
2. All ending tasks in the workflow must have a response type of “Score”.
3. The user must also supply a list of metrics that will be used to evaluate the output of the workflow.
   • The metric names must correspond to the final task names in the workflow.

Arguments

• config - LLMDriftConfig - The configuration for the LLM
• workflow - Workflow - The workflow that will be used to evaluate the L
• metrics - Vec - The metrics that will be used to evaluate the LLM
• scouter_version - Option - The version of scouter that the profile is created with.

Returns

• Result<Self, ProfileError> - The LLMDriftProfile

pub fn get_metric_value(&self, metric_name: &str) -> Result<f64, ProfileError>

Trait Implementations

impl Clone for LLMDriftProfile

fn clone(&self) -> LLMDriftProfile

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for LLMDriftProfile

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for LLMDriftProfile

fn deserialize<__D>( __deserializer: __D, ) -> Result<LLMDriftProfile, <__D as Deserializer<'de>>::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl<'py> IntoPyObject<'py> for LLMDriftProfile

type Target = LLMDriftProfile

The Python output type

type Output = Bound<'py, <LLMDriftProfile as IntoPyObject<'py>>::Target>

The smart pointer type to use.

type Error = PyErr

The type returned in the event of a conversion error.

fn into_pyobject( self, py: Python<'py>, ) -> Result<<LLMDriftProfile as IntoPyObject<'py>>::Output, <LLMDriftProfile as IntoPyObject<'py>>::Error>

Performs the conversion.

impl PartialEq for LLMDriftProfile

fn eq(&self, other: &LLMDriftProfile) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl ProfileBaseArgs for LLMDriftProfile

fn get_base_args(&self) -> ProfileArgs

fn to_value(&self) -> Value

impl PyClass for LLMDriftProfile

type Frozen = False

Whether the pyclass is frozen.

impl PyTypeInfo for LLMDriftProfile

const NAME: &'static str = "LLMDriftProfile"

Class name.

const MODULE: Option<&'static str> = ::core::option::Option::None

Module name, if any.

fn type_object_raw(py: Python<'_>) -> *mut PyTypeObject

Returns the PyTypeObject instance for this type.

fn type_object(py: Python<'_>) -> Bound<'_, PyType>

Returns the safe abstraction over the type object.

fn is_type_of(object: &Bound<'_, PyAny>) -> bool

Checks if object is an instance of this type or a subclass of this type.

fn is_exact_type_of(object: &Bound<'_, PyAny>) -> bool

Checks if object is an instance of this type.

impl Serialize for LLMDriftProfile

fn serialize<__S>( &self, __serializer: __S, ) -> Result<<__S as Serializer>::Ok, <__S as Serializer>::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl DerefToPyAny for LLMDriftProfile

impl StructuralPartialEq for LLMDriftProfile