influence 0.1.5

# Influence - Specification

## Overview

Influence is a Rust CLI application for downloading models from HuggingFace mirrors and running local LLM inference with Candle. The tool prioritizes privacy and local-only operation with no cloud API dependencies.

## Version

**Current Version:** 0.1.0

## Architecture

### Components

1. **CLI Module** (`cli.rs`)
   - Command-line interface using Clap derive API
   - Main commands: search, download, serve, generate, chat, embed
   - Type-safe argument parsing

2. **Download Module** (`download.rs`)
   - Downloads models from HuggingFace mirrors
   - Dynamic file discovery via HuggingFace API
   - Fallback to hardcoded file lists
   - Progress tracking with indicatif
   - Handles gated models gracefully
   - Automatic directory management

3. **Search Module** (`search.rs`)
   - Queries HuggingFace model API
   - Filters by author/organization
   - Displays model metadata
   - Shows download commands

4. **Local Module** (`local.rs`)
   - Local model loading and inference
   - Tokenizer loading from HuggingFace format
   - Model architecture detection
   - Llama-style model loading from .safetensors
   - Forward pass with KV caching
   - Temperature-based token sampling
   - Streaming token generation

5. **Influencer Module** (`influencer.rs`)
   - Command generation orchestration
   - Requires local model path
   - No cloud API dependency

6. **Error Module** (`error.rs`)
   - Centralized error handling with thiserror
   - Custom error types for each failure mode
   - Helpful error messages

7. **Server Module** (`server.rs`)
   - Axum-based HTTP server (REST + SSE)

8. **Config Module** (`config.rs`)
   - Environment variable configuration
   - .env file support via dotenvy
   - Fallback to sensible defaults

### Configuration System

Influence uses a layered configuration approach:

1. **Built-in defaults** - Sensible defaults for all parameters
2. **Environment variables** - `.env` file or system environment
3. **CLI arguments** - Explicit command-line flags (highest priority)

**Priority:** CLI args > Environment vars > Defaults

**Supported environment variables:**
- `INFLUENCE_MODEL_PATH` - Default model directory
- `INFLUENCE_TEMPERATURE` - Generation temperature (default: 0.7)
- `INFLUENCE_TOP_P` - Nucleus sampling threshold (default: 0.9)
- `INFLUENCE_TOP_K` - Top-k sampling limit (default: none)
- `INFLUENCE_REPEAT_PENALTY` - Repetition penalty (default: 1.1)
- `INFLUENCE_MAX_TOKENS` - Max tokens to generate (default: 512)
- `INFLUENCE_DEVICE` - Compute device: auto|cpu|metal|cuda (default: auto)
- `INFLUENCE_DEVICE_INDEX` - GPU ordinal (default: 0)
- `INFLUENCE_PORT` - Server port (default: 8080)
- `INFLUENCE_WARMUP_TOKENS` - Metal warmup tokens (default: 6)
- `INFLUENCE_MIRROR` - HuggingFace mirror URL
- `INFLUENCE_OUTPUT_DIR` - Download output directory

### Design Principles

- **DRY**: No code duplication
- **KISS**: Simple, straightforward implementation
- **SoC**: Clear separation of concerns
- **Local-First**: No cloud API dependencies for generation
- **Maintainability**: Modular structure with clear responsibilities

## API Design

### CLI Commands

#### `search`
```bash
influence search <query> [options]
```

Search for models on HuggingFace.

**Arguments:**
- `query` - Search query string

**Options:**
- `-l, --limit <n>` - Maximum results (default: 20)
- `-a, --author <org>` - Filter by author/organization

**Output:**
- Model ID
- Author
- Downloads/Likes count
- Pipeline tag
- Download command

#### `download`
```bash
influence download -m <model> [options]
```

Download a model from HuggingFace mirror.

**Required:**
- `-m, --model <model>` - Model name (e.g., 'TinyLlama/TinyLlama-1.1B-Chat-v1.0')

**Options:**
- `-r, --mirror <url>` - Mirror URL (default: hf-mirror.com)
- `-o, --output <path>` - Output directory (default: ./models/)

**Process:**
1. Validate model exists on mirror
2. Fetch file list from HuggingFace API
3. Download files with progress bars
4. Handle 403 errors for gated models
5. Save to local directory

**Downloaded Files:**
- `config.json` - Model configuration
- `tokenizer.json` - Tokenizer configuration
- `tokenizer_config.json` - Additional tokenizer settings
- `special_tokens_map.json` - Special token mappings
- `*.safetensors` - Model weights

#### `generate`
```bash
influence generate <prompt> [options]
```

Generate text using local LLM inference.

**Required:**
- `prompt` - Text prompt
- `-m, --model-path <path>` - Path to local model directory

**Options:**
- `--max-tokens <n>` - Maximum tokens to generate (default: 512)
- `--temperature <t>` - Sampling temperature (default: 0.7)
- `--system <text>` - Optional system prompt prepended before user prompt
- `--device <auto|cpu|metal|cuda>` - Compute backend selection (default: auto)
- `--device-index <n>` - GPU ordinal for metal/cuda backends (default: 0)

**Process:**
1. Load tokenizer from model directory
2. Detect model architecture from config.json
3. Load .safetensors weights
4. Initialize KV cache
5. Tokenize prompt
6. Generate tokens with temperature sampling
7. Stream output to stdout

### `serve`

Serve the local model over HTTP.

**Endpoints:**
- `POST /v1/generate` - returns JSON `{ "text": "..." }`
- `POST /v1/generate_stream` - returns SSE events `event: token` with token chunks in `data:`

**Ollama-compatible endpoints (partial):**
- `POST /api/generate`
  - `stream=false` (default): returns JSON `{ model, response, done, created_at }`
  - `stream=true`: returns `application/x-ndjson` with one JSON object per line (each line is `{ model, response, done, created_at }`)
  - Supported `options` mapping: `temperature`, `top_p`, `top_k`, `repeat_penalty`, `num_predict`
- `POST /api/embeddings`
  - Returns JSON `{ embedding: [f32...] }`
  - Current limitation: embeddings only supported for encoder-only BERT models
- `POST /api/tags`
  - Returns JSON `{ models: [{ name, model, modified_at }] }` for the currently served model

**Error Conditions:**
- Model path not provided
- Model directory not found
- Missing tokenizer files
- Missing config.json
- Missing .safetensors files
- Unsupported model architecture (Mamba, MoE, encoder-only)

#### `serve`
```bash
influence serve [options]
```

Serve the local model over HTTP.

**Options:**
- `-m, --model-path <path>` - Path to model directory
- `-p, --port <n>` - Port to serve on (default: 8080)
- `--device <auto|cpu|metal|cuda>` - Compute backend selection (default: auto)
- `--device-index <n>` - GPU ordinal for metal/cuda backends (default: 0)

## Configuration

### No Configuration Files

The CLI follows the KISS principle:
- All configuration via command-line arguments
- No config files to manage
- No environment variables required
- Predictable behavior

### Model Directory Structure

Models are stored in a flat structure:
```
models/
└── <org>_<model-name>/
    ├── config.json
    ├── tokenizer.json
    ├── tokenizer_config.json
    ├── special_tokens_map.json
    ├── model.safetensors (or sharded model-*.safetensors)
    └── README.md (optional)
```

## Data Flow

### Download Flow

```
Parse CLI Arguments
  → Validate model name
  → Determine output directory
  → Create HTTP client
  → Check model exists (HEAD request)
  → Fetch file list (HuggingFace API or fallback)
  → For each file:
      → Check if already exists
      → Download with progress bar
      → Save to disk
      → Verify file integrity
  → Complete
```

### Generate Flow

```
Parse CLI Arguments
  → Validate --model-path provided
  → Read model directory
  → Load tokenizer (tokenizer.json)
  → Read config.json
  → Detect architecture (model_type field)
  → Check for unsupported architectures
  → Load .safetensors weights (mmap)
  → Build model graph
  → Initialize KV cache
  → Tokenize prompt
  → Generate tokens:
      → Forward pass
      → Get logits for last token
      → Apply temperature
      → Sample token (argmax)
      → Check EOS
      → Stream token to stdout
  → Complete
```

## Model Support

### Supported Architectures

**Currently Supported:**
- Llama (meta-llama/Llama-2-7b-hf, TinyLlama/TinyLlama-1.1B-Chat-v1.0)
- Mamba (mamba family configs)
- GraniteMoeHybrid (attention-only configs)
- Encoder-only embeddings: BERT (`influence embed ...`)

**Not Supported:**
- Mixture of Experts (MoE) models
- GraniteMoeHybrid configs containing Mamba layers (not supported by candle-transformers yet)
- Encoder-only text generation (BERT, RoBERTa, ALBERT)

### Model File Requirements

Each model directory must contain:
- `config.json` - Model architecture and hyperparameters
- `tokenizer.json` or `tokenizer_config.json` - Tokenizer
- `*.safetensors` - Model weights (memory-mapped)

### Architecture Detection

Detection logic in `local.rs`:
1. Read `config.json`
2. Check for `layer_types` field (indicates Mamba/MoE)
3. Parse `model_type` field
4. Map to architecture enum
5. Return helpful error for unsupported types

## Error Handling

### Error Types

```rust
pub enum InfluenceError {
    DownloadError(String),
    ModelNotFound(String),
    InvalidConfig(String),
    LocalModelError(String),
    IoError(std::io::Error),
    HttpError(reqwest::Error),
    JsonError(serde_json::Error),
    CandleError(String),
    TokenizerError(String),
}
```

### Error Messages

- User-friendly descriptions
- Actionable suggestions
- Model-specific guidance
- Architecture incompatibility explanations

## Testing Strategy

### Unit Tests
- Individual function testing
- Error case handling
- Configuration parsing

### Integration Tests
- End-to-end command testing
- Mock HTTP servers for downloads
- Model loading with test fixtures

### Test Coverage
- Core functionality
- Error paths
- Architecture detection

## Performance Considerations

### KV Caching:
- Fresh KV cache created per generation request (stateless)
- Reduces redundant computation within a single generation
- Cache automatically freed after generation completes
- Future: Session-based cache reuse for multi-turn conversations

### Memory Mapping:
- Zero-copy model loading with mmap
- Reduces memory overhead

### Streaming:
- Display tokens as they're generated
- Better user experience for long responses
- **Memory**: Model size + cache + tokenizer
- **Compute**: CPU-bound (GPU support planned)
- **Latency**: First token slower, subsequent tokens faster (due to cache)

### Metal Warmup (macOS)

On Metal, the first few decode steps can be significantly slower due to kernel compilation. Influence performs a small warmup when loading Llama models on Metal to reduce visible first-token latency during generation.

- Default: warm up `6` single-token decode steps during model load
- Environment variable: `INFLUENCE_WARMUP_TOKENS` (default: 6, set to 0 to disable)
- Runs a few decode steps during model load to pre-compile Metal kernels
- Reduces first-token latency at the cost of slightly longer model load time

### KV Cache Management:
- Fresh cache created for each generation request
- Stateless design ensures predictable behavior
- No cross-request cache persistence (planned for future sessions feature)
- Memory efficient: cache freed immediately after generation

## Security Considerations

1. **No Remote Code Execution**: Pure local inference
2. **HTTPS Only**: All downloads use HTTPS
3. **File Permissions**: Respects system permissions
4. **No API Keys**: No credentials to leak
5. **Input Validation**: All arguments validated

## Future Enhancements

### Priority 1 (Usability)
- Better sampling methods (top-k, nucleus)
- Chat mode with conversation history
- System prompt support
- Repetition penalty control

### Priority 2 (Performance)
- GPU support (CUDA/Metal)
- Batch generation
- Model caching
- Quantized model support

### Priority 3 (Features)
- HTTP API server
- Configuration file support
- Download resume capability
- Model validation after download
- Integration test suite

## Dependencies

### Core
- `tokio` 1.42 - Async runtime
- `clap` 4.5 - CLI parsing
- `reqwest` 0.12 - HTTP client

### ML Inference
- `candle-core` 0.9 - Core ML operations
- `candle-nn` 0.9 - Neural network components
- `candle-transformers` 0.9 - Transformer models
- `tokenizers` 0.21 - HuggingFace tokenizers

### Utilities
- `tracing` 0.1 - Logging
- `indicatif` 0.17 - Progress bars
- `serde` 1.0 - Serialization
- `anyhow` 1.0 - Error context
- `thiserror` 2.0 - Error types