shlesha 0.5.7

High-performance extensible transliteration library with hub-and-spoke architecture
Documentation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
# Shlesha API Bindings

This document provides comprehensive information about using Shlesha across different programming languages and platforms.

## Overview

Shlesha offers full-featured bindings for multiple languages:
- **Rust** (native library)
- **Python** (via PyO3)
- **JavaScript/WebAssembly** (via wasm-bindgen)
- **Command Line Interface**

All bindings provide the same core functionality:
- Bidirectional transliteration between all supported scripts
- Metadata collection for unknown tokens
- Script discovery and validation
- Graceful error handling

## Python Bindings

### Installation

```bash
# Development installation
pip install maturin
maturin develop --features python

# Production build
maturin build --features python --release
pip install target/wheels/*.whl
```

### API Reference

#### Classes

##### `Shlesha`
Main transliterator class.

```python
transliterator = shlesha.Shlesha()
```

**Methods:**
- `transliterate(text, from_script, to_script) -> str`
- `transliterate_with_metadata(text, from_script, to_script) -> TransliterationResult`
- `list_supported_scripts() -> List[str]`
- `supports_script(script) -> bool`
- `load_schema(schema_path) -> None`
- `get_script_info() -> Dict[str, str]`

##### `TransliterationResult`
Result object containing output and metadata.

**Properties:**
- `output: str` - Transliterated text
- `metadata: Optional[TransliterationMetadata]` - Conversion metadata

##### `TransliterationMetadata`
Metadata about the transliteration process.

**Properties:**
- `source_script: str` - Source script name
- `target_script: str` - Target script name
- `used_extensions: str` - Extensions used
- `unknown_tokens: List[UnknownToken]` - Unknown tokens found

##### `UnknownToken`
Information about unknown/untranslatable tokens.

**Properties:**
- `script: str` - Script where token was found
- `token: str` - The unknown token
- `position: int` - Position in the text
- `unicode: str` - Unicode representation
- `is_extension: bool` - Whether from runtime extension

#### Convenience Functions

```python
# Direct transliteration
result = shlesha.transliterate("धर्म", "devanagari", "iast")

# Get supported scripts
scripts = shlesha.get_supported_scripts()

# Create transliterator instance
transliterator = shlesha.create_transliterator()
```

### Examples

```python
import shlesha

# Basic usage
transliterator = shlesha.Shlesha()
result = transliterator.transliterate("धर्म", "devanagari", "iast")
print(result)  # "dharma"

# With metadata
result = transliterator.transliterate_with_metadata("धर्मkr", "devanagari", "iast")
print(result.output)  # "dharmakr"
if result.metadata:
    print(f"Unknown tokens: {len(result.metadata.unknown_tokens)}")

# Script information
if transliterator.supports_script("gujarati"):
    result = transliterator.transliterate("dharma", "iast", "gujarati")
    print(result)  # "ધર્મ"
```

## WebAssembly Bindings

### Building

```bash
# Install wasm-pack
cargo install wasm-pack

# Build for web browsers
wasm-pack build --target web --out-dir pkg --features wasm

# Build for Node.js
wasm-pack build --target nodejs --out-dir pkg-node --features wasm

# Build for bundlers (webpack, etc.)
wasm-pack build --target bundler --out-dir pkg-bundler --features wasm
```

### API Reference

#### Classes

##### `WasmShlesha`
Main transliterator class for WebAssembly.

```javascript
const transliterator = new WasmShlesha();
```

**Methods:**
- `transliterate(text, fromScript, toScript) -> string`
- `transliterateWithMetadata(text, fromScript, toScript) -> WasmTransliterationResult`
- `listSupportedScripts() -> Array<string>`
- `supportsScript(script) -> boolean`
- `loadSchema(schemaPath) -> void`
- `getScriptInfo() -> Object`
- `getSupportedScriptCount() -> number`

##### `WasmTransliterationResult`
Result object with output and metadata access methods.

**Methods:**
- `getOutput() -> string`
- `hasMetadata() -> boolean`
- `getSourceScript() -> string | null`
- `getTargetScript() -> string | null`
- `getUnknownTokenCount() -> number`
- `getUnknownTokens() -> Array<Object>`

#### Convenience Functions

```javascript
// Direct transliteration
const result = transliterate("धर्म", "devanagari", "iast");

// Get supported scripts
const scripts = getSupportedScripts();

// Create transliterator instance
const transliterator = createTransliterator();

// Get version
const version = getVersion();
```

### Usage Examples

#### Web Browser

```html
<!DOCTYPE html>
<html>
<head>
    <meta charset="UTF-8">
    <title>Shlesha Demo</title>
</head>
<body>
    <script type="module">
        import init, { WasmShlesha, transliterate } from './pkg/shlesha.js';
        
        async function main() {
            await init();
            
            // Using class
            const transliterator = new WasmShlesha();
            const result = transliterator.transliterate("धर्म", "devanagari", "iast");
            console.log(result); // "dharma"
            
            // Using convenience function
            const result2 = transliterate("dharma", "iast", "devanagari");
            console.log(result2); // "धर्म"
            
            // With metadata
            const withMeta = transliterator.transliterateWithMetadata("धर्मkr", "devanagari", "iast");
            console.log(withMeta.getOutput()); // "dharmakr"
            console.log(withMeta.getUnknownTokenCount()); // 2
        }
        
        main();
    </script>
</body>
</html>
```

#### Node.js

```javascript
const { WasmShlesha, transliterate } = require('./pkg-node/shlesha.js');

// Direct usage
const result = transliterate("धर्म", "devanagari", "iast");
console.log(result); // "dharma"

// Class usage
const transliterator = new WasmShlesha();
const scripts = transliterator.listSupportedScripts();
console.log(`Supports ${scripts.length} scripts`);
```

## Command Line Interface

### Installation

```bash
cargo install --path . --features cli
```

### Usage

```bash
# Basic transliteration
shlesha transliterate --from devanagari --to iast "धर्म"

# With metadata display (inline)
shlesha transliterate --from devanagari --to iast --show-metadata "धर्मkr"

# With detailed metadata
shlesha transliterate --from devanagari --to iast --verbose "धर्मkr"

# List supported scripts
shlesha scripts

# Read from stdin
echo "धर्म" | shlesha transliterate --from devanagari --to iast

# Process file
shlesha transliterate --from devanagari --to iast < input.txt > output.txt
```

### Flags

- `--from`, `-f`: Source script name
- `--to`, `-t`: Target script name  
- `--show-metadata`: Show unknown tokens inline as `[script:token]`
- `--verbose`, `-v`: Show detailed metadata breakdown
- `--help`, `-h`: Show help information

## Error Handling

All bindings implement graceful error handling:

### Python
```python
try:
    result = transliterator.transliterate("text", "invalid_script", "iast")
except RuntimeError as e:
    print(f"Error: {e}")
```

### JavaScript
```javascript
try {
    const result = transliterator.transliterate("text", "invalid_script", "iast");
} catch (error) {
    console.error(`Error: ${error.message}`);
}
```

### CLI
Exit codes:
- `0`: Success
- `1`: Transliteration error or invalid arguments

## Performance Considerations

### Python
- Use the class instance for multiple conversions to avoid repeated initialization
- Metadata collection has minimal overhead when not used

### WebAssembly
- WASM module initialization is asynchronous - await `init()` before use
- Consider using Web Workers for large batch processing
- Module size is optimized for web delivery

### CLI
- Supports stdin/stdout for pipeline integration
- Efficient for single conversions and batch processing

## Building from Source

### Requirements
- Rust 1.70+
- Python 3.8+ (for Python bindings)
- Node.js 14+ (for WASM testing)

### Python Development
```bash
# Install development dependencies
pip install maturin pytest

# Build and install in development mode
maturin develop --features python

# Run tests
pytest python/tests/
```

### WASM Development
```bash
# Install wasm-pack
cargo install wasm-pack

# Build and test
wasm-pack build --features wasm
wasm-pack test --node --features wasm

# Serve demo
python3 -m http.server 8000
# Open http://localhost:8000/demo.html
```

## Integration Examples

### Python Data Processing
```python
import pandas as pd
import shlesha

transliterator = shlesha.Shlesha()

# Process DataFrame column
df['transliterated'] = df['sanskrit_text'].apply(
    lambda x: transliterator.transliterate(x, "devanagari", "iast")
)
```

### JavaScript Web App
```javascript
// Async transliteration function
async function translateText(text, from, to) {
    if (!window.shleshaReady) {
        await init();
        window.transliterator = new WasmShlesha();
        window.shleshaReady = true;
    }
    
    return window.transliterator.transliterate(text, from, to);
}

// Use in form handler
document.getElementById('translateBtn').onclick = async () => {
    const text = document.getElementById('inputText').value;
    const result = await translateText(text, 'devanagari', 'iast');
    document.getElementById('output').textContent = result;
};
```

### Shell Script Processing
```bash
#!/bin/bash

# Batch convert files
for file in *.txt; do
    echo "Converting $file..."
    shlesha transliterate --from devanagari --to iast < "$file" > "converted_$file"
done

# Process with metadata
echo "धर्मkr" | shlesha transliterate --from devanagari --to iast --verbose > conversion_report.txt
```

## Troubleshooting

### Python Issues
- **Import errors**: Ensure maturin development install completed successfully
- **Runtime errors**: Check Python version compatibility (3.8+)
- **Performance**: Use class instances for multiple conversions

### WASM Issues  
- **Module not loading**: Ensure proper async initialization with `await init()`
- **CORS errors**: Serve files over HTTP, not file:// protocol
- **Memory issues**: Consider chunking large text processing

### CLI Issues
- **Command not found**: Ensure `~/.cargo/bin` is in your PATH
- **Permission errors**: Check file permissions for input/output files
- **Encoding issues**: Ensure terminal supports UTF-8 encoding

## Contributing

See the main README.md for contribution guidelines. When adding binding features:

1. Update all three binding implementations (Python, WASM, CLI)
2. Add corresponding tests for each binding
3. Update this documentation
4. Ensure backward compatibility