shimmy 1.1.0

Lightweight 5MB Ollama alternative for local AI inference. Fast startup, reliable inference engine.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150

# API Reference


Shimmy provides multiple API interfaces for local LLM inference.

## HTTP REST API


### Generate Text


**Endpoint:** `POST /api/generate`

**Request Body:**
```json
{
  "model": "string",           // Model name (required)
  "prompt": "string",          // Input prompt (required)
  "max_tokens": 100,          // Maximum tokens to generate (optional, default: 100)
  "temperature": 0.7,         // Sampling temperature (optional, default: 0.7)
  "stream": false             // Enable streaming response (optional, default: false)
}
```

**Non-Streaming Response:**
```json
{
  "choices": [
    {
      "text": "Generated text response",
      "index": 0,
      "finish_reason": "length"
    }
  ],
  "usage": {
    "prompt_tokens": 10,
    "completion_tokens": 20,
    "total_tokens": 30
  }
}
```

**Streaming Response:**
Server-Sent Events with data chunks:
```
data: {"choices":[{"text":"Hello","index":0}]}

data: {"choices":[{"text":" world","index":0}]}

data: [DONE]
```

### List Models


**Endpoint:** `GET /api/models`

**Response:**
```json
{
  "models": [
    {
      "id": "default",
      "name": "Default Model",
      "description": "Base GGUF model"
    }
  ]
}
```

### Health Check


**Endpoint:** `GET /api/health`

**Response:**
```json
{
  "status": "healthy",
  "models_loaded": 1,
  "memory_usage": "2.1GB"
}
```

## WebSocket API


**Endpoint:** `ws://localhost:11435/ws/generate`

### Connect and Send

```json
{
  "model": "default",
  "prompt": "Hello world",
  "max_tokens": 50,
  "temperature": 0.7
}
```

### Receive Tokens

```json
{"token": "Hello"}
{"token": " world"}
{"done": true}
```

## CLI Interface


### Commands


```bash
# Start server

shimmy serve --bind 127.0.0.1:11435 --port 11435

# Generate text

shimmy generate --prompt "Hello" --max-tokens 50 --temperature 0.7

# List available models

shimmy list

# Probe model loading

shimmy probe [model-name]

# Show diagnostics

shimmy diag
```

### Global Options


- `--verbose, -v`: Enable verbose logging
- `--help, -h`: Show help information
- `--version, -V`: Show version information

## Error Responses


All endpoints return consistent error formats:

```json
{
  "error": {
    "code": "model_not_found",
    "message": "The specified model was not found",
    "details": "Model 'invalid-model' is not available"
  }
}
```

Common error codes:
- `model_not_found`: Requested model is not available
- `invalid_request`: Request format is invalid
- `generation_failed`: Text generation failed
- `server_error`: Internal server error

## Rate Limiting


Currently no rate limiting is implemented. For production use, consider placing shimmy behind a reverse proxy with rate limiting capabilities.