# Test Documentation
This document describes the test suite structure and provides guidance for running tests.
## Test Organization
### Unit Tests (No External Dependencies)
Unit tests are located inline within modules and run without external services:
```bash
cargo test
```
### Integration Tests (Require API Keys)
Integration tests are marked with `#[ignore]` and require external services:
```bash
cargo test --features openai,ollama -- --ignored
```
### Integration Test Directory
End-to-end workflow tests are in `tests/` directory:
- `architecture.rs`: Architecture and type system tests
## Required Environment Variables
| OpenAI | `OPENAI_API_KEY` |
| Anthropic Claude | `ANTHROPIC_API_KEY` |
| Google Gemini | `GOOGLE_API_KEY` |
| Mistral AI | `MISTRAL_API_KEY` |
| Ollama (local) | `OLLAMA_HOST` (optional, default: `http://localhost:11434`) |
| AWS Bedrock | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION` |
| GitHub | `GITHUB_TOKEN` |
| AWS S3 | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION`, `S3_BUCKET` |
## Running Tests by Category
### All Non-Ignored Tests
```bash
cargo test
```
### All Tests (Including Integration)
```bash
cargo test --features openai,ollama,mistralai,gemini,bedrock -- --ignored
```
### OpenAI Tests Only
```bash
cargo test --features openai test_openai -- --ignored
```
### Ollama Tests Only
```bash
cargo test --features ollama test_ollama -- --ignored
```
## Test Categories by Module
### LLM Provider Tests
| `src/llm/openai/mod.rs` | OpenAI | `OPENAI_API_KEY` |
| `src/llm/claude/client.rs` | Anthropic Claude | `ANTHROPIC_API_KEY` |
| `src/llm/gemini/client.rs` | Google Gemini | `GOOGLE_API_KEY` |
| `src/llm/mistralai/client.rs` | Mistral AI | `MISTRAL_API_KEY` |
| `src/llm/ollama/client.rs` | Ollama | Running Ollama (`ollama serve`) |
| `src/llm/deepseek/client.rs` | DeepSeek | `DEEPSEEK_API_KEY` |
| `src/llm/qwen/client.rs` | Alibaba Qwen | `QWEN_API_KEY` |
| `src/llm/bedrock/client.rs` | AWS Bedrock | AWS credentials + Bedrock access |
| `src/llm/huggingface/client.rs` | HuggingFace | `HF_TOKEN` |
### Embedding Tests
| `src/embedding/mistralai/mistralai_embedder.rs` | Mistral AI | `MISTRAL_API_KEY` |
| `src/embedding/ollama/ollama_embedder.rs` | Ollama | Running Ollama |
### Chain Tests
| `src/chain/llm_chain.rs` | `test_invoke_chain` | `OPENAI_API_KEY` |
| `src/chain/sequential/chain.rs` | Sequential chain tests | API key |
| `src/chain/question_answering.rs` | Q&A chain tests | API key + Vector store |
| `src/chain/conversational_retrieval_qa/` | Conversational RAG | API key + Vector store |
### Agent Tests
| `src/agent/chat/chat_agent.rs` | Chat agent | API key |
| `tests/architecture.rs` | Architecture tests | None (unit tests) |
### Tool Tests
| `src/tools/duckduckgo/duckduckgo_search.rs` | DuckDuckGo | None (should work) |
| `src/tools/wolfram/wolfram.rs` | Wolfram Alpha | `WOLFRAM_APP_ID` |
| `src/tools/serpapi/serpapi.rs` | SerpApi | `SERPAPI_API_KEY` |
| `src/tools/text2speech/openai/client.rs` | OpenAI TTS | `OPENAI_API_KEY` |
### Document Loader Tests
| `src/document_loaders/git_commit_loader/` | Git commits | Git repository |
| `src/document_loaders/github_loader.rs` | GitHub | `GITHUB_TOKEN` |
| `src/document_loaders/aws_s3_loader.rs` | AWS S3 | AWS credentials + bucket |
| `src/document_loaders/web_loaders/` | Web scraping | Network access |
| `src/document_loaders/office_loaders/excel_loader.rs` | Excel | Excel file |
## Vector Store Integration Tests
Vector store tests typically require:
- Running database service (PostgreSQL, MongoDB, etc.)
- Appropriate feature flag enabled
```bash
# PostgreSQL with pgvector
cargo test --features postgres
# Qdrant
cargo test --features qdrant
# MongoDB
cargo test --features mongodb
```
## Semantic Router Tests
| `src/semantic_router/route_layer/route_layer.rs` | Route layer | API key (for embeddings) |
## Best Practices for Adding Tests
### 1. Adding Unit Tests
Unit tests should be placed in `#[cfg(test)]` modules within the source file:
```rust
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_something() {
// Test implementation
}
}
```
### 2. Adding Integration Tests
Integration tests that require API keys should:
- Be marked with `#[ignore]`
- Include the required environment variables in the ignore reason
- Include a comment with setup instructions
```rust
#[tokio::test]
#[ignore = "Requires OPENAI_API_KEY - see tests/README.md"]
async fn test_with_api() {
let api_key = std::env::var("OPENAI_API_KEY")
.expect("OPENAI_API_KEY must be set");
// Test implementation
}
```
### 3. Adding E2E Workflow Tests
End-to-end tests should be added to `tests/` directory:
```rust
// tests/my_workflow_test.rs
use oris_runtime::chain::Chain;
#[tokio::test]
#[ignore = "Requires OPENAI_API_KEY"]
async fn test_complete_workflow() {
// Test implementation
}
```
## CI/CD Configuration
See `.github/workflows/` for CI configuration. Integration tests run on schedule with API keys from secrets.
## Troubleshooting
### Test Fails with "Not implemented"
Some tests may fail with "Not implemented" if the underlying feature is not yet complete. Check:
- Feature flags are enabled
- Required dependencies are present
- The feature is actively being developed
### API Connection Errors
1. Verify environment variables are set
2. Check API keys are valid
3. Ensure network connectivity
4. For local services (Ollama), verify the service is running
### Timeout Errors
Some tests may timeout if:
- API rate limits are hit
- Network latency is high
- Service is unresponsive
Increase timeout or run tests individually:
```bash
cargo test --test my_test -- --nocapture
```