Struct PromptTestConfig 

pub struct PromptTestConfig {

    pub inherits: Option<String>,
    pub name: Option<String>,
    pub prompt: Option<String>,
    pub messages: Option<Vec<MessageParam>>,
    pub system: Option<String>,
    pub model: Option<String>,
    pub max_tokens: Option<u32>,
    pub temperature: Option<f32>,
    pub top_p: Option<f32>,
    pub top_k: Option<u32>,
    pub stop_sequences: Option<Vec<String>>,
    pub tools: Option<Vec<ToolUnionParam>>,
    pub tool_choice: Option<ToolChoice>,
    pub expected_contains: Option<Vec<String>>,
    pub expected_not_contains: Option<Vec<String>>,
    pub min_response_length: Option<usize>,
    pub max_response_length: Option<usize>,
    pub expected_tool_calls: Option<Vec<String>>,
    pub expect_error: Option<bool>,
    pub expected_error_message: Option<String>,
    pub output_format: Option<OutputFormat>,
}

Configuration for a prompt test with support for inheritance and file references.

This structure represents a complete prompt test configuration that can be loaded from YAML files, inherit from base configurations, and automatically resolve file references for content.

File Reference Resolution

When loading from files, the following fields support automatic file resolution:

• prompt: If the value is a relative path ending with “prompt.yaml”, the file content is loaded
• system: If the value is a relative path ending with “system.md”, the file content is loaded

Security Considerations

File resolution is restricted for security:

• Only relative paths are resolved (absolute paths remain as literal strings)
• Only specific filenames (“prompt.yaml”, “system.md”) trigger file loading
• Files are resolved relative to the configuration file’s directory

Examples

Creating a basic configuration:

let config = PromptTestConfig::new("What is the capital of France?")
    .with_model("claude-haiku-4-5")
    .expect_contains("Paris");

Loading from file with automatic file references:

If test.yaml contains:

name: "Geography Test"
prompt: "prompt.yaml"    # Content loaded from prompt.yaml
system: "system.md"      # Content loaded from system.md
model: "claude-haiku-4-5"
expected_contains:
  - "capital"

Then:

let config = PromptTestConfig::from_file("test.yaml")?;
// config.prompt now contains the contents of prompt.yaml
// config.system now contains the contents of system.md

Fields

inherits: Option<String>

Base configuration to inherit from (filename within prompts directory).

name: Option<String>

Name of the test (optional).

prompt: Option<String>

The prompt text to send (for single-turn conversations).

When loading from files, if this field contains a relative path ending with “prompt.yaml”, the content will be automatically loaded from that file.

messages: Option<Vec<MessageParam>>

Multi-turn conversation messages (alternative to prompt).

system: Option<String>

Optional system prompt.

When loading from files, if this field contains a relative path ending with “system.md”, the content will be automatically loaded from that file.

model: Option<String>

Model to use for testing.

max_tokens: Option<u32>

Maximum tokens to generate.

temperature: Option<f32>

Temperature setting (0.0 to 1.0).

top_p: Option<f32>

Top-p setting (0.0 to 1.0).

top_k: Option<u32>

Top-k setting.

stop_sequences: Option<Vec<String>>

Stop sequences.

tools: Option<Vec<ToolUnionParam>>

Tools available for the conversation.

tool_choice: Option<ToolChoice>

How the model should use the provided tools.

expected_contains: Option<Vec<String>>

Expected content that should appear in the response.

expected_not_contains: Option<Vec<String>>

Expected content that should NOT appear in the response.

min_response_length: Option<usize>

Minimum expected response length.

max_response_length: Option<usize>

Maximum expected response length.

expected_tool_calls: Option<Vec<String>>

Expected tool calls (name of tools that should be called).

expect_error: Option<bool>

Whether this test is expected to fail with an API error.

expected_error_message: Option<String>

Expected error message (substring match).

output_format: Option<OutputFormat>

Output format for structured outputs.

When set, constrains Claude’s response to follow a specific JSON schema, ensuring valid, parseable output for downstream processing.

Implementations

impl PromptTestConfig

pub fn new(prompt: impl Into<String>) -> Self

Create a new prompt test configuration with just a prompt.

Examples

let config = PromptTestConfig::new("What is the capital of France?");
assert_eq!(config.prompt, Some("What is the capital of France?".to_string()));

pub fn new_conversation(messages: Vec<MessageParam>) -> Self

Create a new multi-turn conversation test.

Examples

let messages = vec![
    MessageParam::user("Hello"),
    MessageParam::assistant("Hi there! How can I help you?"),
    MessageParam::user("What's the weather like?"),
];
let config = PromptTestConfig::new_conversation(messages);
assert!(config.messages.is_some());
assert!(config.prompt.is_none());

pub fn with_name(self, name: impl Into<String>) -> Self

Set the test name.

pub fn with_system(self, system: impl Into<String>) -> Self

Set the system prompt.

pub fn with_model(self, model: impl Into<String>) -> Self

Set the model to use.

pub fn with_max_tokens(self, max_tokens: u32) -> Self

Set the maximum tokens.

pub fn with_temperature(self, temperature: f32) -> Self

Set the temperature.

pub fn expect_contains(self, content: impl Into<String>) -> Self

Add expected content that should appear in the response.

pub fn expect_not_contains(self, content: impl Into<String>) -> Self

Add content that should NOT appear in the response.

pub fn with_min_length(self, min_length: usize) -> Self

Set minimum expected response length.

pub fn with_max_length(self, max_length: usize) -> Self

Set maximum expected response length.

pub fn with_tool(self, tool: ToolUnionParam) -> Self

Add a tool to the configuration.

pub fn with_tool_choice(self, tool_choice: ToolChoice) -> Self

Set the tool choice configuration.

pub fn expect_tool_call(self, tool_name: impl Into<String>) -> Self

Expect a specific tool to be called.

pub fn expect_error(self) -> Self

Expect the API call to fail with an error.

pub fn expect_error_message(self, message: impl Into<String>) -> Self

Expect a specific error message (substring match).

pub fn with_output_format(self, output_format: OutputFormat) -> Self

Set the output format for structured outputs.

When set, constrains Claude’s response to follow a specific JSON schema, ensuring valid, parseable output for downstream processing.

Examples

let config = PromptTestConfig::new("Extract the person's name and age")
    .with_output_format(OutputFormat::json_schema(json!({
        "type": "object",
        "properties": {
            "name": { "type": "string" },
            "age": { "type": "integer" }
        },
        "required": ["name", "age"],
        "additionalProperties": false
    })));

pub fn from_file<P: AsRef<Path>>(path: P) -> Result<Self, Box<dyn Error>>

Load a prompt test configuration from a YAML file with inheritance and file reference support.

This method provides several key features:

Configuration Inheritance

Supports configuration inheritance via the inherits field, allowing you to build configuration hierarchies. Security restrictions apply: only relative paths are allowed, and parent directory traversal is only permitted for base.yaml files.

File Reference Resolution

Automatically loads content from external files when:

• prompt field contains a relative path ending with “prompt.yaml”
• system field contains a relative path ending with “system.md”

Files are resolved relative to the configuration file’s directory. Absolute paths are treated as literal strings for security reasons.

Examples

Basic usage:

let config = PromptTestConfig::from_file("test_config.yaml")?;
println!("Loaded test: {:?}", config.name);

File references:

If your config file contains:

name: "My Test"
prompt: "prompt.yaml"     # This file will be loaded
system: "system.md"       # This file will be loaded
model: "claude-haiku-4-5"

The content of prompt.yaml and system.md (relative to the config file) will be automatically loaded into the prompt and system fields.

Inheritance with file references:

inherits: "../base.yaml"   # Inheritance (base.yaml only for parent dirs)
name: "Specialized Test"
prompt: "custom_prompt.yaml"  # File reference

Errors

Returns an error if:

• The file cannot be read
• The YAML is invalid
• Referenced prompt.yaml or system.md files cannot be read
• Inheritance files use absolute paths or unsafe traversal
• Inherited files cannot be found or loaded

pub fn from_file_with_base_dir<P: AsRef<Path>>( path: P, base_dir: Option<&Path>, ) -> Result<Self, Box<dyn Error>>

Load a prompt test configuration from a YAML file with a specific base directory.

This is the core method that handles both configuration inheritance and file reference resolution. The base_dir parameter allows you to override the directory used for resolving relative file paths.

File Reference Resolution

Files are automatically loaded when:

• The prompt field contains a relative path ending with “prompt.yaml”
• The system field contains a relative path ending with “system.md”

Only these specific filenames are resolved for security reasons. Other filenames or absolute paths are treated as literal strings.

Base Directory Resolution

• If base_dir is provided, all relative paths are resolved relative to it
• If base_dir is None, paths are resolved relative to the config file’s directory

Examples

// Load config with custom base directory
let config = PromptTestConfig::from_file_with_base_dir(
    "config.yaml",
    Some(Path::new("/custom/base/dir"))
)?;

Security

File resolution is restricted to specific patterns for security:

• Only relative paths ending with “prompt.yaml” or “system.md” are resolved
• Absolute paths are treated as literal strings
• Parent directory traversal in inheritance is only allowed for “base.yaml” files

Errors

Returns an error if:

• The configuration file cannot be read
• Referenced prompt.yaml or system.md files cannot be read
• The YAML syntax is invalid
• Inheritance security restrictions are violated

pub fn to_file<P: AsRef<Path>>(&self, path: P) -> Result<(), Box<dyn Error>>

Save a prompt test configuration to a YAML file.

pub async fn run(&self, client: &Anthropic) -> Result<PromptTestResult, Error>

Run the prompt test using the provided Anthropic client.

This method executes the prompt against the Anthropic API and validates all configured assertions. It handles both successful responses and API errors gracefully, allowing tests to verify error conditions.

Examples

let client = Anthropic::new(None)?;
let config = PromptTestConfig::new("Hello, world!")
    .expect_contains("hello")
    .with_min_length(5);

let result = config.run(&client).await?;
assert!(result.api_success);
println!("Response: {}", result.response);

Errors

Returns a crate::Error if:

• Neither prompt nor messages is provided
• Invalid parameter values (e.g., temperature out of range)
• Other validation errors during request building

Trait Implementations

impl Clone for PromptTestConfig

fn clone(&self) -> PromptTestConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for PromptTestConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for PromptTestConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for PromptTestConfig

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.