harmony-protocol 0.1.0

# Harmony Response Format API Reference

## ⚠️ **IMPORTANT: Model Integration Required**

This library provides conversation formatting and parsing - **you must integrate with OpenAI's API or compatible model**. The typical workflow is:

1. Use this library to format conversations into tokens
2. Send tokens to OpenAI model via API
3. Receive response tokens from the model
4. Use this library to parse response tokens back to messages

## Core Types

### Message
```rust
pub struct Message {
    pub author: Author,
    pub recipient: Option<String>,
    pub content: Vec<Content>,
    pub channel: Option<String>,
    pub content_type: Option<String>,
}
```

**Methods:**
- `from_role_and_content(role: Role, content: impl Into<Content>) -> Self`
- `from_author_and_content(author: Author, content: impl Into<Content>) -> Self`
- `adding_content(self, content: impl Into<Content>) -> Self`
- `with_channel(self, channel: impl Into<String>) -> Self`
- `with_recipient(self, recipient: impl Into<String>) -> Self`
- `with_content_type(self, content_type: impl Into<String>) -> Self`

### Author
```rust
pub struct Author {
    pub role: Role,
    pub name: Option<String>,
}
```

**Methods:**
- `new(role: Role, name: impl Into<String>) -> Self`

### Role
```rust
pub enum Role {
    User,
    Assistant,
    System,
    Developer,
    Tool,
}
```

**Methods:**
- `as_str(&self) -> &str`
- `try_from(value: &str) -> Result<Self, &'static str>`

### Content
```rust
pub enum Content {
    Text(TextContent),
    SystemContent(SystemContent),
    DeveloperContent(DeveloperContent),
}
```

### SystemContent
```rust
pub struct SystemContent {
    pub model_identity: Option<String>,
    pub reasoning_effort: Option<ReasoningEffort>,
    pub tools: Option<BTreeMap<String, ToolNamespaceConfig>>,
    pub conversation_start_date: Option<String>,
    pub knowledge_cutoff: Option<String>,
    pub channel_config: Option<ChannelConfig>,
}
```

**Methods:**
- `new() -> Self`
- `with_model_identity(self, model_identity: impl Into<String>) -> Self`
- `with_reasoning_effort(self, effort: ReasoningEffort) -> Self`
- `with_tools(self, ns_config: ToolNamespaceConfig) -> Self`
- `with_conversation_start_date(self, date: impl Into<String>) -> Self`
- `with_knowledge_cutoff(self, cutoff: impl Into<String>) -> Self`
- `with_channel_config(self, config: ChannelConfig) -> Self`
- `with_required_channels<I, T>(self, channels: I) -> Self`
- `with_browser_tool(self) -> Self`
- `with_python_tool(self) -> Self`

### DeveloperContent
```rust
pub struct DeveloperContent {
    pub instructions: Option<String>,
    pub tools: Option<BTreeMap<String, ToolNamespaceConfig>>,
}
```

**Methods:**
- `new() -> Self`
- `with_instructions(self, instructions: impl Into<String>) -> Self`
- `with_tools(self, ns_config: ToolNamespaceConfig) -> Self`
- `with_function_tools(self, tools: Vec<ToolDescription>) -> Self`

### ReasoningEffort
```rust
pub enum ReasoningEffort {
    Low,
    Medium,
    High,
}
```

### ChannelConfig
```rust
pub struct ChannelConfig {
    pub valid_channels: Vec<String>,
    pub channel_required: bool,
}
```

**Methods:**
- `require_channels<I, T>(channels: I) -> Self`

### ToolNamespaceConfig
```rust
pub struct ToolNamespaceConfig {
    pub name: String,
    pub description: Option<String>,
    pub tools: Vec<ToolDescription>,
}
```

**Methods:**
- `new(name: impl Into<String>, description: Option<String>, tools: Vec<ToolDescription>) -> Self`
- `browser() -> Self`
- `python() -> Self`

### ToolDescription
```rust
pub struct ToolDescription {
    pub name: String,
    pub description: String,
    pub parameters: Option<serde_json::Value>,
}
```

**Methods:**
- `new(name: impl Into<String>, description: impl Into<String>, parameters: Option<serde_json::Value>) -> Self`

### Conversation
```rust
pub struct Conversation {
    pub messages: Vec<Message>,
}
```

**Methods:**
- `from_messages<I>(messages: I) -> Self`

## Encoding API

### HarmonyEncoding
```rust
pub struct HarmonyEncoding {
    // Internal fields
}
```

**Methods:**
- `name(&self) -> &str`
- `tokenizer_name(&self) -> &str`
- `max_message_tokens(&self) -> usize`
- `tokenizer(&self) -> &CoreBPE`
- `stop_tokens(&self) -> anyhow::Result<HashSet<Rank>>`
- `stop_tokens_for_assistant_actions(&self) -> anyhow::Result<HashSet<Rank>>`

**Rendering Methods:**
- `render_conversation<'a, I>(&self, conversation: I, config: Option<&RenderConversationConfig>) -> anyhow::Result<Vec<Rank>>`
- `render_conversation_for_completion<'a, I>(&self, conversation: I, next_turn_role: Role, config: Option<&RenderConversationConfig>) -> anyhow::Result<Vec<Rank>>`
- `render_conversation_for_training<'a, I>(&self, conversation: I, config: Option<&RenderConversationConfig>) -> anyhow::Result<Vec<Rank>>`
- `render(&self, message: &Message, render_options: Option<&RenderOptions>) -> anyhow::Result<Vec<Rank>>`

**Parsing Methods:**
- `parse_messages_from_completion_tokens<I>(&self, tokens: I, role: Option<Role>) -> anyhow::Result<Vec<Message>>`

### StreamableParser
```rust
pub struct StreamableParser {
    // Internal fields
}
```

**Methods:**
- `new(encoding: HarmonyEncoding, role: Option<Role>) -> anyhow::Result<Self>`
- `process(&mut self, token: Rank) -> anyhow::Result<&mut Self>`
- `process_eos(&mut self) -> anyhow::Result<&mut Self>`
- `current_content(&self) -> anyhow::Result<String>`
- `current_role(&self) -> Option<Role>`
- `current_content_type(&self) -> Option<String>`
- `current_recipient(&self) -> Option<String>`
- `current_channel(&self) -> Option<String>`
- `last_content_delta(&self) -> anyhow::Result<Option<String>>`
- `into_messages(self) -> Vec<Message>`
- `messages(&self) -> &[Message]`
- `tokens(&self) -> &[Rank]`
- `state_json(&self) -> anyhow::Result<String>`

### RenderConversationConfig
```rust
pub struct RenderConversationConfig {
    pub auto_drop_analysis: bool,
}
```

**Methods:**
- `default() -> Self`

### RenderOptions
```rust
pub struct RenderOptions {
    pub conversation_has_function_tools: bool,
}
```

**Methods:**
- `default() -> Self`

## Registry API

### HarmonyEncodingName
```rust
pub enum HarmonyEncodingName {
    HarmonyGptOss,
}
```

**Methods:**
- `to_string(&self) -> String`
- `from_str(s: &str) -> Result<Self, anyhow::Error>`

### Functions
- `load_harmony_encoding(name: HarmonyEncodingName) -> anyhow::Result<HarmonyEncoding>`

## Tokenization API

### CoreBPE
```rust
pub struct CoreBPE {
    // Internal fields
}
```

**Methods:**
- `new<E, SE>(encoder: E, special_tokens_encoder: SE, pattern: &str) -> Result<Self, Box<dyn std::error::Error + Send + Sync>>`
- `encode_ordinary(&self, text: &str) -> Vec<Rank>`
- `encode(&self, text: &str, allowed_special: &HashSet<&str>) -> (Vec<Rank>, usize)`
- `encode_with_special_tokens(&self, text: &str) -> Vec<Rank>`
- `decode_bytes<S, E>(&self, tokens: S) -> Result<Vec<u8>, DecodeKeyError>`
- `decode_utf8<S, E>(&self, tokens: S) -> Result<String, DecodeError>`
- `special_tokens(&self) -> HashSet<&str>`
- `is_special_token(&self, token: Rank) -> bool`

### Types
```rust
pub type Rank = u32;
```

## Error Types

### DecodeKeyError
```rust
pub struct DecodeKeyError {
    pub token: Rank,
}
```

### DecodeError
```rust
pub struct DecodeError {
    pub message: String,
}
```

## Special Token Values

| Token | Value | Purpose |
|-------|-------|---------|
| `<|startoftext|>` | 199998 | Text start marker |
| `<|endoftext|>` | 199999 | Text end marker |
| `<|return|>` | 200002 | Training completion |
| `<|constrain|>` | 200003 | Constrained format |
| `<|channel|>` | 200005 | Channel marker |
| `<|start|>` | 200006 | Message start |
| `<|end|>` | 200007 | Message end |
| `<|message|>` | 200008 | Content start |
| `<|call|>` | 200012 | Tool call end |
| `<|reserved_*|>` | 200014-201088 | Reserved tokens |

## Usage Examples

### Basic Usage
```rust
use openai_harmony::{
    load_harmony_encoding, HarmonyEncodingName, Role, Message,
    Conversation, SystemContent
};

let enc = load_harmony_encoding(HarmonyEncodingName::HarmonyGptOss)?;
let convo = Conversation::from_messages([
    Message::from_role_and_content(Role::System, SystemContent::new()),
    Message::from_role_and_content(Role::User, "Hello!"),
]);
let tokens = enc.render_conversation_for_completion(&convo, Role::Assistant, None)?;
```

### With Tools
```rust
let system_content = SystemContent::new()
    .with_browser_tool()
    .with_required_channels(["analysis", "commentary", "final"]);
let message = Message::from_role_and_content(Role::System, system_content);
```

### Streaming Parser
```rust
let mut parser = StreamableParser::new(enc.clone(), Some(Role::Assistant))?;
for token in response_tokens {
    parser.process(token)?;
    if let Ok(Some(delta)) = parser.last_content_delta() {
        print!("{}", delta);
    }
}
let messages = parser.into_messages();
```