claude-code-mux 0.6.2

# Google AI Integration Guide

## Overview

This document covers two separate providers for Google's AI models:

1. **Google Gemini** (`provider_type: "gemini"`)
   - For Google AI Studio (AI.google.dev)
   - Supports OAuth 2.0 (Code Assist API) and API Key
   - **Only Gemini models**

2. **Vertex AI** (`provider_type: "vertex-ai"`) ✅
   - For Google Cloud Platform
   - Uses Application Default Credentials (ADC)
   - **Supports Gemini, Claude, Llama, and other models**

## ⚠️ IMPORTANT: OAuth Uses Code Assist API

The OAuth client_id `681255809395-...` from gemini-cli is registered **ONLY** for Google's Code Assist API (`cloudcode-pa.googleapis.com`), **NOT** for the public Gemini API (`generativelanguage.googleapis.com`).

Therefore:
- **OAuth** → Code Assist API (`cloudcode-pa.googleapis.com/v1internal:generateContent`)
- **API Key** → Public Gemini API (`generativelanguage.googleapis.com/v1beta/models/{model}:generateContent`)

---

## Google Gemini Provider (`provider_type: "gemini"`)

### Authentication Methods

#### 1. OAuth 2.0 (Google AI Pro/Ultra) - Code Assist API

**OAuth Configuration:**
```
client_id: 681255809395-oo8ft2oprdrnp9e3aqf6av3hmdib135j.apps.googleusercontent.com
client_secret: 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
auth_url: https://accounts.google.com/o/oauth2/v2/auth
token_url: https://oauth2.googleapis.com/token
redirect_uri: http://localhost:13456/api/oauth/callback
scopes:
  - https://www.googleapis.com/auth/cloud-platform
  - https://www.googleapis.com/auth/userinfo.email
  - https://www.googleapis.com/auth/userinfo.profile
```

**OAuth Flow:**
1. Generate authorization URL with PKCE
2. User authorizes in browser → redirected to callback URL
3. Exchange authorization code for access token + refresh token
4. **Call loadCodeAssist API to get/validate project ID**
5. Store tokens with project_id for automatic refresh

**Project ID Handling:**

| Account Type | Project ID Source | Action Required |
|--------------|-------------------|-----------------|
| **Individual Account** (free-tier) | Auto-generated by API | ✅ None - works automatically |
| **Workspace Account** (standard-tier) | User-defined | ⚠️ Set `GOOGLE_CLOUD_PROJECT` env var |
| **Licensed Users** | User-defined | ⚠️ Set `GOOGLE_CLOUD_PROJECT` env var |

**For Workspace/Licensed Users Only:**
1. Go to [Google Cloud Console](https://console.cloud.google.com/)
2. Create a new project or select an existing one
3. Enable "Cloud AI Companion API" for the project
4. Copy the Project ID (NOT the project name)
5. Set environment variable before starting server:
   ```bash
   export GOOGLE_CLOUD_PROJECT=your-project-id
   # OR
   export GOOGLE_CLOUD_PROJECT_ID=your-project-id
   ```

**LoadCodeAssist API:**
```
POST https://cloudcode-pa.googleapis.com/v1internal:loadCodeAssist
Authorization: Bearer {access_token}
Content-Type: application/json

{
  "cloudaicompanionProject": null,
  "metadata": {
    "ideType": "IDE_UNSPECIFIED",
    "platform": "PLATFORM_UNSPECIFIED",
    "pluginType": "GEMINI"
  }
}

Response:
{
  "cloudaicompanionProject": "projects/123456789",
  "currentTier": { ... }
}
```

**API Endpoint (OAuth - Code Assist API):**
```
https://cloudcode-pa.googleapis.com/v1internal:generateContent
```

**Request Format (Code Assist API):**
```json
{
  "model": "gemini-2.0-flash-exp",
  "user_prompt_id": "gemini-1234567890",
  "request": {
    "contents": [...],
    "systemInstruction": {...},
    "generationConfig": {...}
  }
}
```

**Response Format (Code Assist API):**
```json
{
  "response": {
    "candidates": [...],
    "usageMetadata": {...}
  },
  "traceId": "..."
}
```

**Headers:**
```
Authorization: Bearer {access_token}
Content-Type: application/json
```

#### 2. API Key (Google AI Studio)

**API Key Configuration:**
- Get API key from: https://aistudio.google.com/app/apikey

**API Endpoint:**
```
https://generativelanguage.googleapis.com/v1beta/models/{model}:generateContent?key={api_key}
```

**Headers:**
```
Content-Type: application/json
```

### Configuration Example

**OAuth-based Gemini provider:**
```toml
[[providers]]
name = "gemini-oauth"
provider_type = "gemini"
auth_type = "oauth"
oauth_provider = "google"
models = ["gemini-2.0-flash-exp", "gemini-2.5-pro"]
enabled = true
```

**API Key-based Gemini provider:**
```toml
[[providers]]
name = "gemini-studio"
provider_type = "gemini"
auth_type = "apikey"
api_key = "your-api-key-from-ai-studio"
models = ["gemini-1.5-pro", "gemini-1.5-flash"]
enabled = true
```

---

## Vertex AI Provider (`provider_type: "vertex-ai"`)

Vertex AI is Google Cloud's unified AI platform that provides access to multiple model families:
- **Gemini models** (2.5 Flash, 2.0 Flash, 1.5 Pro, etc.)
- **Claude models** (3.7 Sonnet, 3.5 Haiku, etc.)
- **Llama models** (Llama 4, Llama 3.x)
- **Other models** via Model Garden

### Authentication

Vertex AI uses **Application Default Credentials (ADC)** for authentication.

**Setup Steps:**
1. Install Google Cloud SDK: https://cloud.google.com/sdk/docs/install
2. Authenticate with ADC:
   ```bash
   gcloud auth application-default login
   ```
3. This creates credentials that the server will use automatically

**Alternative Authentication Methods:**
- Service Account JSON key
- Google Cloud API key
- Workload Identity (for GKE deployments)

### Configuration

**Required Fields:**
- `project_id`: Your Google Cloud Project ID (not project name)
- `location`: GCP region (e.g., us-central1, europe-west1)

**Available Locations:**
```
us-central1, us-east1, us-west1, us-west4
europe-west1, europe-west2, europe-west4
asia-east1, asia-northeast1, asia-southeast1
australia-southeast1
```

**TOML Configuration Example:**
```toml
[[providers]]
name = "vertex-gemini"
provider_type = "vertex-ai"
project_id = "my-gcp-project-id"
location = "us-central1"
models = [
    "publishers/google/models/gemini-2.0-flash-exp",
    "publishers/google/models/gemini-1.5-pro",
]
enabled = true
```

### Admin UI Setup

1. In Admin UI, select **"Vertex AI"** as provider type
2. Enter your **Project ID** (from Google Cloud Console)
3. Select **Location** from dropdown (11 regions)
4. Add model names with `publishers/google/models/` prefix
5. Save configuration
6. **Before starting the server**, authenticate with ADC (see above)

**Admin UI Features:**
- ✅ Select from 11 pre-configured regions
- ✅ Validates Project ID and Location before saving
- ✅ Shows Vertex AI badge on provider cards
- ✅ Edit/Update Vertex AI credentials anytime

### API Endpoint

```
https://{location}-aiplatform.googleapis.com/v1/projects/{project}/locations/{location}/publishers/google/models/{model}:generateContent
```

**Example:**
```
https://us-central1-aiplatform.googleapis.com/v1/projects/my-project/locations/us-central1/publishers/google/models/gemini-2.0-flash-exp:generateContent
```

**Headers:**
```
Authorization: Bearer {access_token}  // From ADC
Content-Type: application/json
```

### Supported Models

Vertex AI supports models from multiple providers. Model names must include the `publishers/{publisher}/models/` prefix.

**Gemini Models:**
```
publishers/google/models/gemini-2.0-flash-exp
publishers/google/models/gemini-1.5-pro
publishers/google/models/gemini-1.5-flash
```

**Claude Models:**
```
publishers/anthropic/models/claude-3-7-sonnet
publishers/anthropic/models/claude-3-5-haiku
```

**Llama Models:**
```
publishers/meta/models/llama-4-maverick
publishers/meta/models/llama-3-2-90b
```

---

## API Format

### Request Format (Gemini generateContent API)

```json
{
  "contents": [
    {
      "role": "user",
      "parts": [
        {
          "text": "Hello, how are you?"
        }
      ]
    }
  ],
  "generationConfig": {
    "temperature": 1.0,
    "topP": 0.95,
    "topK": 40,
    "maxOutputTokens": 8192,
    "stopSequences": []
  },
  "systemInstruction": {
    "parts": [
      {
        "text": "You are a helpful assistant."
      }
    ]
  }
}
```

### Response Format (Non-streaming)

```json
{
  "candidates": [
    {
      "content": {
        "parts": [
          {
            "text": "I'm doing well, thank you for asking!"
          }
        ],
        "role": "model"
      },
      "finishReason": "STOP",
      "index": 0,
      "safetyRatings": [...]
    }
  ],
  "usageMetadata": {
    "promptTokenCount": 10,
    "candidatesTokenCount": 15,
    "totalTokenCount": 25
  }
}
```

### Streaming Response Format

```
data: {"candidates": [{"content": {"parts": [{"text": "Hello"}],"role": "model"}}]}

data: {"candidates": [{"content": {"parts": [{"text": " there"}],"role": "model"}}]}

data: {"candidates": [{"content": {"parts": [{"text": "!"}],"role": "model"}],"finishReason": "STOP","usageMetadata": {...}}]}
```

## Anthropic → Gemini Conversion

### Role Mapping
```
Anthropic → Gemini
user      → user
assistant → model
```

### Content Block Mapping

**Text:**
```rust
// Anthropic
ContentBlock::Text { text: "Hello" }

// Gemini
{ "text": "Hello" }
```

**Image:**
```rust
// Anthropic
ContentBlock::Image {
  source: ImageSource {
    type: "base64",
    media_type: "image/png",
    data: "iVBORw0KG..."
  }
}

// Gemini
{
  "inline_data": {
    "mime_type": "image/png",
    "data": "iVBORw0KG..."
  }
}
```

**Thinking (Extended Thinking):**
```rust
// Anthropic
ContentBlock::Thinking {
  thinking: "Let me think...",
  signature: "..."
}

// Gemini (convert to text)
{ "text": "Let me think..." }
```

### System Prompt
```rust
// Anthropic
request.system = Some("You are a helpful assistant")

// Gemini
{
  "systemInstruction": {
    "parts": [
      { "text": "You are a helpful assistant" }
    ]
  }
}
```

### Generation Config
```rust
// Anthropic → Gemini
max_tokens       → maxOutputTokens
temperature      → temperature
top_p            → topP
stop_sequences   → stopSequences

// Gemini only
topK: 40 (default)
```

### Tools (Function Calling)

**Anthropic:**
```json
{
  "tools": [
    {
      "name": "get_weather",
      "description": "Get weather information",
      "input_schema": {
        "type": "object",
        "properties": {
          "location": { "type": "string" }
        }
      }
    }
  ]
}
```

**Gemini:**
```json
{
  "tools": [
    {
      "functionDeclarations": [
        {
          "name": "get_weather",
          "description": "Get weather information",
          "parameters": {
            "type": "object",
            "properties": {
              "location": { "type": "string" }
            }
          }
        }
      ]
    }
  ]
}
```

## Model Names Reference

### Gemini Provider Models

For `provider_type: "gemini"` (OAuth or API Key authentication), use simple model names:

**Stable Models (Recommended):**
```
gemini-2.5-pro              # Latest production model
gemini-2.0-flash-exp        # Fast experimental model
gemini-1.5-pro              # Previous stable model
gemini-1.5-flash            # Fast stable model
gemini-1.5-flash-8b         # Ultra-fast model
```

**Preview Models (Limited Access):**
```
gemini-3-pro-preview        # Requires preview access (may return 404)
gemini-exp-1206             # Experimental model
```

⚠️ **Note**: Preview models like `gemini-3-pro-preview` require special access and may not be available to all users. If you receive a 404 error, use `gemini-2.5-pro` or `gemini-2.0-flash-exp` instead.

### Vertex AI Provider Models

For `provider_type: "vertex-ai"`, use full model paths with `publishers/{publisher}/models/` prefix.

**Note:** This section was already documented above under "Vertex AI Provider → Supported Models".

## Error Handling

### Common Errors

**401 Unauthorized:**
```json
{
  "error": {
    "code": 401,
    "message": "Request is missing required authentication credential.",
    "status": "UNAUTHENTICATED"
  }
}
```

**403 Forbidden:**
```json
{
  "error": {
    "code": 403,
    "message": "User does not have permission to access this resource.",
    "status": "PERMISSION_DENIED"
  }
}
```

**429 Rate Limit:**
```json
{
  "error": {
    "code": 429,
    "message": "Resource has been exhausted (e.g. check quota).",
    "status": "RESOURCE_EXHAUSTED"
  }
}
```

## Implementation Checklist

### Gemini Provider (`provider_type: "gemini"`)
- [x] GeminiProvider struct with OAuth/API Key support
- [x] OAuth configuration in auth/oauth.rs
- [x] OAuth token refresh logic
- [x] Request transformation (Anthropic → Gemini)
- [x] Response transformation (Gemini → Anthropic)
- [x] **Project ID retrieval via loadCodeAssist API** (OAuth only)
- [x] **Token storage with project_id field** (OAuth only)
- [x] **Code Assist API integration** (OAuth)
- [x] **Public Gemini API integration** (API Key)
- [x] **JSON Schema metadata cleaning for tools**
- [ ] Streaming support
- [x] Tool/Function calling support
- [x] Image support (inline_data)
- [x] Error handling and mapping
- [x] Admin UI integration
  - [x] OAuth flow UI (similar to Anthropic/OpenAI)
  - [x] API Key input
  - [x] Provider type selector showing "📱 Google Gemini"

### Vertex AI Provider (`provider_type: "vertex-ai"`)
- [x] Separate provider type in registry
- [x] Reuses GeminiProvider with ADC authentication
- [x] Project ID and Location configuration fields
- [x] ADC token acquisition logic
- [x] Vertex AI API endpoint construction
- [x] Admin UI integration
  - [x] Separate provider type selector "Vertex AI"
  - [x] Project ID input field
  - [x] Location dropdown (11 regions)
  - [x] ADC authentication instructions
  - [x] Vertex AI badge on provider cards
  - [x] Edit/Update support for Vertex AI providers
- [x] Multi-model support (Gemini, Claude, Llama)
- [x] Model name validation with `publishers/` prefix

## References

- Gemini CLI OAuth: `/tmp/gemini-cli/packages/core/src/code_assist/oauth2.ts`
- Gemini API Docs: https://ai.google.dev/gemini-api/docs
- Vertex AI Docs: https://cloud.google.com/vertex-ai/docs/generative-ai/start/quickstarts/api-quickstart