# Plaintext Identification in ciphey
## Overview
One of the most critical components of ciphey is its ability to identify when encoded text has been successfully decoded into plaintext. This document explains the mechanisms and strategies ciphey uses to determine whether a given string is valid plaintext.
## The Importance of Plaintext Detection
Accurate plaintext detection serves several crucial purposes in ciphey:
1. **Termination Condition**: It tells the search algorithm when to stop decoding
2. **Result Validation**: It confirms that the decoded output is meaningful
3. **Efficiency**: It prevents unnecessary decoding attempts on already-decoded text
4. **Accuracy**: It helps avoid false positives (incorrectly identifying gibberish as plaintext)
## The Checker System
ciphey uses a modular system of "checkers" to identify plaintext. Each checker specializes in recognizing different types of plaintext:
### Athena Checker
The Athena checker (`src/checkers/athena.rs`) is the main orchestrator that coordinates other checkers. When asked to check if text is plaintext, it:
1. Checks if a regex pattern is provided in the configuration
- If yes, it uses the RegexChecker to see if the text matches
- If the text matches, it optionally verifies with the human checker
2. If no regex is provided (or the regex didn't match), it tries:
- LemmeKnow checker first
- English checker second
- For each, if they identify the text as plaintext, it optionally verifies with the human checker
The Athena checker returns as soon as any of its sub-checkers identifies the text as plaintext, or returns a negative result if none do.
### LemmeKnow Checker
The LemmeKnow checker (`src/checkers/lemmeknow_checker.rs`) uses the [LemmeKnow](https://github.com/swanandx/lemmeknow) library, which is a Rust implementation of [PyWhat](https://github.com/bee-san/pyWhat). This library can identify over 100 different types of data formats and patterns, including:
- IP addresses (IPv4, IPv6)
- Email addresses
- URLs
- Credit card numbers
- Cryptocurrency addresses
- API keys and tokens
- File paths
- MAC addresses
- And many more
The checker works by:
1. Configuring LemmeKnow with a minimum rarity threshold (0.1 by default)
2. Passing the text to LemmeKnow's identify function
3. Checking if any patterns were identified
4. If patterns were found, marking the text as identified plaintext
This checker is particularly useful for identifying structured data that might not be natural language but is still valid plaintext.
### English Checker
The English checker (`src/checkers/english.rs`) determines if text is valid English language. It uses the [gibberish-or-not](https://crates.io/crates/gibberish-or-not) library to distinguish meaningful English text from random character sequences, with configurable sensitivity levels.
The process works as follows:
1. **Normalization**: The text is first normalized by:
- Converting to lowercase
- Removing all ASCII punctuation
- This helps ensure consistent checking regardless of formatting
2. **Gibberish Detection**: The normalized text is passed to the `is_gibberish` function with a sensitivity level
- If the function returns `false`, the text is considered valid English
- If it returns `true`, the text is considered gibberish
3. **Edge Case Handling**: Very short strings (less than 2 characters after normalization) are automatically considered not plaintext, as they're too short for reliable detection
#### Sensitivity Levels
The English checker supports three sensitivity levels:
- **Low Sensitivity**: Most strict classification, requires very high confidence to classify text as English. Used by classical ciphers like Caesar cipher that produce more English-like results.
- **Medium Sensitivity (Default)**: Balanced approach for general use, suitable for most applications. Used by most decoders in ciphey.
- **High Sensitivity**: Most lenient classification, favors classifying text as English. Useful when input is mostly gibberish and any English-like patterns are significant.
The English checker is effective for detecting natural language text but may struggle with specialized technical content or very short texts. The sensitivity level can be adjusted based on the specific decoder's needs.
### Regex Checker
The Regex checker (`src/checkers/regex_checker.rs`) allows users to provide a custom regular expression pattern to match against decoded text. This is useful when looking for specific formats or patterns in the output.
The checker simply:
1. Takes the regex pattern from the configuration
2. Attempts to match it against the input text
3. Returns true if there's a match, false otherwise
This checker is typically used when the user knows what they're looking for and can provide a specific pattern.
### Human Checker
The Human checker (`src/checkers/human_checker.rs`) provides a way to involve human judgment in the plaintext detection process. It's particularly useful for ambiguous cases or specialized content that automated checkers might not recognize correctly.
When enabled (off by default), it:
1. Displays the decoded text to the user
2. Asks if the text looks like valid plaintext
3. Returns the user's response
This checker is optional and can be enabled or disabled through the configuration.
## Plaintext Detection Process
The overall plaintext detection process in ciphey follows these steps:
1. **Initial Check**: When `perform_cracking` is called, ciphey first checks if the input is already plaintext using the Athena checker
- If it is, ciphey returns early with the input as the result
- This prevents unnecessary processing of already-decoded text
2. **During Search**: As the search algorithm explores possible decodings, each result is checked:
- The Athena checker is used to determine if the result is plaintext
- If it is, the search terminates and returns the result
- If not, the result is added to the search queue for further decoding
3. **Result Validation**: Before returning the final result, ciphey ensures it's valid plaintext
- This helps prevent returning partially decoded or incorrect results
## Handling Edge Cases
ciphey includes several mechanisms to handle edge cases in plaintext detection:
### Very Short Strings
Very short strings (less than 2-3 characters) are difficult to classify reliably. ciphey handles these by:
- Having specific logic in the English checker to reject very short strings
- Using multiple checkers to increase the chance of correct identification
### Specialized Content
Some valid plaintext might not be natural language (e.g., JSON, XML, code). ciphey addresses this through:
- The LemmeKnow checker, which can identify many structured data formats
- The regex checker, which allows users to provide custom patterns
- The human checker, which can be enabled for manual verification
### False Positives
To reduce false positives (incorrectly identifying gibberish as plaintext), ciphey:
- Uses multiple checkers with different approaches
- Configures the LemmeKnow checker with a minimum rarity threshold
- Allows for human verification in ambiguous cases
### False Negatives
To reduce false negatives (failing to identify valid plaintext), ciphey:
- Normalizes text before checking (removing punctuation, converting to lowercase)
- Uses multiple checkers with different strengths
- Provides configuration options to adjust the detection sensitivity
## Customizing Plaintext Detection
Users can customize the plaintext detection process through several configuration options:
- **Regex Pattern**: Provide a custom regex pattern to match against decoded text
- **Human Checker**: Enable or disable human verification of results
- **Timeout**: Adjust the maximum time spent trying to decode
- **Sensitivity Level**: Different decoders use different sensitivity levels based on their characteristics
These options allow users to tailor the plaintext detection to their specific needs and expectations.
## Future Improvements
The plaintext detection system in ciphey is continuously evolving. Planned improvements include:
1. **Better English Detection**: Enhancing the English checker to better handle technical content and edge cases
2. **More Specialized Checkers**: Adding checkers for specific formats like JSON, XML, etc.
3. **Machine Learning Approaches**: Exploring ML-based approaches to plaintext detection
4. **Context-Aware Detection**: Taking into account the context and expected output format
5. **User Feedback Integration**: Learning from user feedback to improve detection accuracy over time
## Conclusion
Plaintext identification is a fundamental component of ciphey that enables it to automatically decode text without requiring explicit knowledge of the encoding method. The modular checker system provides flexibility and extensibility, allowing ciphey to handle a wide range of plaintext formats and continuously improve its detection capabilities.