harmorp 0.1.2

Enhanced Nazief-Adriani Indonesian stemmer: iterative ECS, nasal-assimilation restoration, phonotactic guards, FST dictionary, zero-alloc hot path
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126

# Algorithm Documentation

## Enhanced Confix-Stripping (ECS)

This stemmer implements the Enhanced Confix-Stripping (ECS) variant of the Nazief-Adriani algorithm as described in Asian et al., 2007.

## Overview

The ECS algorithm improves upon the original Nazief-Adriani (1996) by:
1. Iterative confix-stripping (up to 4 passes)
2. Nasal-assimilation restoration
3. Phonotactic validity guards
4. Two-path candidate generation

## Algorithm Steps

### 1. Clitic Removal

First, remove the possessive/determiner clitic `-nya`:

```
bukunya → buku
merekan → mereka
```

### 2. Iterative Confix-Stripping

For up to 4 passes, the algorithm strips:
- One prefix family
- One derivational suffix

#### Prefix Families (in priority order)

| Family | Examples |
|--------|----------|
| `me(N)-` | membaca, menulis, mengambil |
| `pe(N)-` | pembaca, penulis, pengambil |
| `ber-` | berjalan, bertanya |
| `ter-` | terlihat, terkenal |
| `se-` | selesai, serupa |
| `ke-` | kesalahan, kebersihan |
| `di-` | dibaca, ditulis |

The `N` in `me(N)-` and `pe(N)-` represents nasal assimilation:
- `meng-` → `g` before vowels: mengambil → ambil
- `men-` → `t` before vowels: menulis → tulis
- `men-` → `s` before vowels: menyapu → sapu
- `meny-` → `s` before vowels: menyanyi → nyanyi

#### Derivational Suffixes (in priority order)

1. `-kan`
2. `-an`
3. `-i`

### 3. Inflectional Suffix Fallback

If no prefix matched, remove inflectional suffixes:
- `-lah`
- `-kah`
- `-tah`
- `-pun`

### 4. Nasal-Assimilation Restoration

When a prefix with nasal assimilation is stripped, the algorithm reconstructs the dropped consonant:

```
menulis → (strip men-) → nulis → (restore t) → tulis
mengambil → (strip meng-) → ngambil → (restore g) → ambil
```

Without a dictionary, the algorithm prefers the longer candidate when ambiguous.

With a dictionary, the first candidate found in the FST wins.

### 5. Phonotactic Validity Guards

Indonesian phonotactics forbid CC-onset (consonant clusters at word start). The algorithm discards candidates that would create invalid CC-onsets:

```
Valid: baca, tulis, jalan
Invalid: blajar (from belajar), ktulis (from ketulis)
```

This prevents over-stemming.

## Two-Path Candidate Generation

The algorithm explores both orderings:
1. Prefix-first then suffix
2. Suffix-first then prefix

Candidates from both paths are combined and ranked. The longer candidate is preferred when no dictionary is available.

## Example Walkthrough

### Example: `mempertimbangkan`

**Pass 1:**
- Strip `memper-` → `timbangan`
- Strip suffix `-an` → `timbang`

**Pass 2:**
- No more prefixes/suffixes to strip

**Result:** `timbang`

### Example: `pembelajaran`

**Pass 1:**
- Strip `pem-` → `belajaran`
- Strip suffix `-an` → `belajar`

**Pass 2:**
- Strip `be-` → `lajar`
- No suffix

**Result:** `lajar` (but with dictionary, `ajar` might be preferred)

## Accuracy

Without dictionary: ~85-90% accuracy on common Indonesian words
With dictionary: ~95-98% accuracy

The dictionary helps resolve ambiguous cases, especially with nasal-assimilation prefixes before vowel-initial roots.