seams 0.1.2

High-throughput sentence extractor for Project Gutenberg texts with dialog-aware detection
Documentation
# Dialog State Machine

This diagram represents the state machine implemented in `src/sentence_detector/dialog_detector.rs` for dialog-aware sentence detection.

## Overview Diagram

```mermaid
stateDiagram-v2
    [*] --> Narrative
    
    %% Group dialog states for readability
    state "Dialog States" as DialogGroup {
        DialogDoubleQuote
        DialogSingleQuote  
        DialogSmartDoubleOpen
        DialogSmartSingleOpen
        DialogParenthheticalRound
        DialogParenthheticalSquare
        DialogParenthheticalCurly
    }
    
    %% Narrative transitions
    Narrative --> DialogGroup : Enter Dialog
    Narrative --> Narrative : Stay in Narrative
    
    %% Dialog exits
    DialogGroup --> Narrative : Exit Dialog
    
    note right of Narrative
        Default state for regular text.
        Creates sentence boundaries on
        punctuation + capital letter.
        Handles N→D transitions.
    end note
    
    note right of DialogGroup
        Seven distinct dialog states.
        Each handles specific delimiters.
        All can transition back to Narrative
        or to other dialog states.
    end note
```

## Detailed State Transitions

### From Narrative State

```mermaid
flowchart TD
    N[Narrative] --> DD["DialogDoubleQuote<br/>N→D: punct + quote<br/>Open: comma + quote"]
    N --> DS["DialogSingleQuote<br/>N→D: punct + quote<br/>Open: comma + quote"]
    N --> SDO["DialogSmartDoubleOpen<br/>N→D: punct + quote<br/>Open: comma + quote"]
    N --> SSO["DialogSmartSingleOpen<br/>N→D: punct + quote<br/>Open: comma + quote"]
    N --> PR["DialogParenthheticalRound<br/>N→D: punct + paren<br/>Open: comma + paren"]
    N --> PS["DialogParenthheticalSquare<br/>N→D: punct + bracket<br/>Open: comma + bracket"]
    N --> PC["DialogParenthheticalCurly<br/>N→D: punct + brace<br/>Open: comma + brace"]
    N --> N2[Narrative] 
    
    N2 --> |"Sentence boundary:<br/>period + capital"| N
    N2 --> |"Hard separator:<br/>double newline"| N
```

### Dialog State Exits

```mermaid
flowchart TD
    AnyDialogState["Any Dialog State<br/>(All 7 states identical)"] --> ExitType{"Exit Pattern Type"}
    
    ExitType --> |"DialogEnd"| Narrative1["Narrative<br/>(Sentence boundary)"]
    ExitType --> |"DialogSoftEnd"| Narrative2["Narrative<br/>(Continue sentence)"]
    ExitType --> |"DialogEnd/DialogOpen"| Unknown["Unknown<br/>(D→D transition)"]
    
    subgraph "Exit Pattern Examples" as examples
        E1["Punctuated: close + punct + space + next<br/>Example: quote-close period space Capital"]
        E2["Continuation: punct + close + space + next<br/>Example: comma quote-close space next"]
        E3["Unpunctuated: close + space + next<br/>Example: quote-close space next"]
        E4["External punct: close + punct + space + next<br/>Example: quote-close exclamation space next"]
    end
    
    subgraph "Macro Generated Uniformity" as uniform
        U1["All dialog states use identical pattern structure<br/>Only delimiter character varies between states<br/>Generated by macro lines 493-596<br/>Eliminates 200+ lines of duplication"]
    end
```

### Dialog-to-Dialog Transitions

```mermaid
flowchart LR
    AnyDialog["Any Dialog State<br/>(Quote/Smart/Bracket)"] --> |"Pattern depends on type"| AnyDialog2["Any Dialog State<br/>(Quote/Smart/Bracket)"]
    
    subgraph "Transition Patterns" as patterns
        P1["Standard: close + space + opener + next_char<br/>Example: quote space paren Capital"]
        P2["Zero-char: close + opener + next_char<br/>Example: paren-close paren-open Capital<br/>(Bracket-style only)"]
    end
    
    AnyDialog -.-> P1
    AnyDialog -.-> P2
    
    subgraph "Macro Parameters" as macro
        M1["generate_dialog_patterns macro:<br/>dialog_state, close_char,<br/>has_zero_char_transitions,<br/>special_punct_pattern"]
    end
    
    subgraph "Implementation Details" as details
        D1["All 7 dialog states support full matrix<br/>Generated uniformly by macro lines 493-596<br/>Quote states: zero_char = false<br/>Bracket states: zero_char = true<br/>Target determined by opening delimiter"]
    end
```

## State Descriptions

- **Narrative**: Default state for regular text outside of dialog
- **DialogDoubleQuote**: ASCII double quote dialog (`"`)
- **DialogSingleQuote**: ASCII single quote dialog (`'`) 
- **DialogSmartDoubleOpen**: Smart double quote dialog (`"`)
- **DialogSmartSingleOpen**: Smart single quote dialog (`'`)
- **DialogParenthheticalRound**: Parenthetical dialog (`()`)
- **DialogParenthheticalSquare**: Square bracket dialog (`[]`)
- **DialogParenthheticalCurly**: Curly brace dialog (`{}`)

## Transition Types

- **N→D transition**: Creates sentence boundary AND enters dialog state
- **Dialog open**: Enters dialog state without sentence boundary  
- **Dialog end**: Creates sentence boundary and returns to Narrative
- **Dialog soft end**: Returns to Narrative without sentence boundary
- **Continuation punct**: Handles punctuation that continues sentences
- **Hard separator**: Double newline separators that force sentence boundaries