helpprobe 0.2.0

# helpprobe

A comprehensive CLI tool discovery and automation framework that extracts structured information from command help text. Transform any CLI tool's help output into machine-readable data for automation, scripting, documentation generation, and IDE integration.

## Features

### 🔍 Command Discovery
- **Structured Parsing**: Extracts options, arguments, subcommands, and usage patterns from help text
- **Multi-Format Support**: Handles various help text formats (clap, GNU, custom)
- **Recursive Discovery**: Automatically discovers nested subcommands and builds complete command trees
- **Smart Caching**: Caches probe results for performance with automatic version detection
- **Automatic Help Flag Detection**: Intelligently tries common help flags (`--help`, `-h`, `help`, etc.) when none are provided

### 📊 Rich Metadata Extraction
- **Enhanced Options**: Extracts option types (boolean, string, number, choice), defaults, and descriptions
- **Argument Analysis**: Identifies required/optional arguments, types (path, URL, email, number), and variadic patterns
- **Environment Variables**: Discovers environment variable mappings and defaults
- **Validation Rules**: Extracts validation constraints (ranges, patterns, choices)
- **Examples**: Parses example commands from help text

### 🛠️ Code Generation
- **Shell Completion**: Generate completion scripts for Bash, Zsh, Fish, PowerShell, and NuShell
- **Command Builders**: Generate type-safe command builder code in Rust, Python, JavaScript, and TypeScript
- **API Documentation**: Generate documentation in Markdown, HTML, OpenAPI, and JSON Schema formats

### ✅ Validation
- **Command Validation**: Validate command invocations against discovered specifications
- **Type Checking**: Verify argument types and option values
- **Error Detection**: Identify missing required options/arguments, unknown flags, and type mismatches

## Installation

### From Source

```bash
git clone <repository-url>
cd helpprobe
cargo build --release
```

The binary will be available at `target/release/helpprobe`.

### Using Cargo

```bash
cargo install --path .
```

## Usage

### Basic Usage

Probe a command's help text. The program automatically detects and uses appropriate help flags:

```bash
helpprobe -- ls
helpprobe -- docker
helpprobe -- git
helpprobe -- cargo
```

You can also explicitly provide help flags:

```bash
helpprobe -- ls --help
helpprobe -- docker -h
```

### JSON Output

Get structured JSON output:

```bash
helpprobe --json -- ls --help
```

### Generate Shell Completion

Generate completion scripts that enable tab completion for commands in your shell. These scripts provide intelligent suggestions when you press Tab, showing available options, subcommands, and arguments.

**What are completion scripts?**
- They are shell-specific scripts (`.sh` for bash, `.ps1` for PowerShell, `.nu` for NuShell, etc.) that tell your shell how to complete commands
- When you type a command and press Tab, the script suggests valid completions (options, subcommands, arguments)
- Each shell (bash, zsh, fish, etc.) has its own format

**Example:** After loading a completion script for `docker`, typing `docker <TAB>` will show all available subcommands (build, run, ps, etc.), and `docker run --<TAB>` will show all available options.

**How to use them:**
1. **Generate** the script: `helpprobe --generate-completion <shell> --output <file> -- <command>`
2. **Load** it:
   - **Temporary** (current session only): `source <file>` (bash/zsh) or `. <file>` (PowerShell)
   - **Permanent** (all future sessions): Add it to your shell's configuration file (see examples below)
3. **Test**: Type the command and press Tab to see completions

The `--output` flag is recommended (works in all shells, prompts if file exists):

```bash
# Bash
# Step 1: Generate the completion script (creates docker-completion.sh)
helpprobe --generate-completion bash --output docker-completion.sh -- docker

# Step 2: Load it in your current session (enables tab completion)
source docker-completion.sh

# Step 3: Test it - type "docker <TAB>" to see subcommands, or "docker run --<TAB>" for options

# To make it permanent, add to ~/.bashrc:
# echo "source $(pwd)/docker-completion.sh" >> ~/.bashrc

# Zsh
helpprobe --generate-completion zsh --output _cargo -- cargo
# Add to fpath: mv _cargo ~/.zsh/completions/ && echo 'fpath=(~/.zsh/completions $fpath)' >> ~/.zshrc
# Then reload: source ~/.zshrc

# Fish
helpprobe --generate-completion fish --output ~/.config/fish/completions/git.fish -- git
# Fish automatically loads completions from ~/.config/fish/completions/

# PowerShell
helpprobe --generate-completion powershell --output podman-completion.ps1 -- podman
# Load in current session: . .\podman-completion.ps1
# Or add to PowerShell profile for persistence: Add-Content $PROFILE ". .\podman-completion.ps1"

# NuShell
helpprobe --generate-completion nushell --output ~/.config/nushell/cargo-completion.nu -- cargo
# Add to config.nu: echo 'source ~/.config/nushell/cargo-completion.nu' >> ~/.config/nushell/config.nu
```

**Note:** Instead of `--output`, you can use shell redirection: `helpprobe --generate-completion bash -- docker > docker-completion.sh` (in NuShell, use `| save` instead of `>`).
```

### Generate Command Builders

Generate type-safe command builder code that provides a fluent API for programmatically constructing CLI commands:

```bash
# Rust
helpprobe --generate-builder rust --output docker_builder.rs -- docker

# Python
helpprobe --generate-builder python --output git_builder.py -- git

# JavaScript/TypeScript
helpprobe --generate-builder typescript --output cargo_builder.ts -- cargo
```

**Use Case Example:**

Instead of manually constructing command strings, use the generated builder for type-safe, IDE-autocompleted command construction:

```python
# Before: Manual command construction (error-prone, no autocomplete)
import subprocess
subprocess.run(["docker", "run", "-d", "--name", "myapp", "-p", "8080:80", "nginx"])

# After: Using generated builder (type-safe, autocompleted)
from docker_builder import DockerBuilder

# Fluent API with autocomplete and type safety
result = DockerBuilder() \
    .run() \
    .detach() \
    .name("myapp") \
    .publish("8080:80") \
    .image("nginx") \
    .execute()

print(result.stdout)
```

```rust
// Rust example
use docker_builder::DockerBuilder;

let output = DockerBuilder::new()
    .run()
    .detach()
    .name("myapp")
    .publish("8080:80")
    .image("nginx")
    .execute()?;
```

This is especially useful for:
- **Automation scripts** - Build commands programmatically from configuration
- **Testing** - Construct test commands with compile-time safety
- **Wrappers** - Create higher-level APIs around CLI tools
- **Configuration management** - Generate commands from config files

### Generate API Documentation

Generate API documentation in various formats:

```bash
# Markdown
helpprobe --generate-api-docs markdown -- docker --help > docker.md

# HTML
helpprobe --generate-api-docs html -- git --help > git.html

# OpenAPI 3.0
helpprobe --generate-api-docs openapi -- cargo --help > cargo-openapi.json

# JSON Schema
helpprobe --generate-api-docs jsonschema -- kubectl --help > kubectl-schema.json
```

### Validate Commands

Validate command invocations:

```bash
helpprobe --validate -- docker run --image nginx
helpprobe --validate -- git commit --message "test"
```

### Recursive Discovery

Discover all subcommands recursively:

```bash
helpprobe --discover-all -- docker --help
helpprobe --discover-all --max-depth 3 -- git --help
```

### Caching

Cache is enabled by default. Control caching behavior:

```bash
# Disable cache
helpprobe --no-cache -- ls --help

# Clear cache for a command
helpprobe --clear-cache -- ls --help

# Use custom cache directory
helpprobe --cache-dir /tmp/my-cache -- docker --help
```

## Library Usage

Use `helpprobe` as a library in your Rust projects. Add it to your `Cargo.toml`:

```toml
[dependencies]
helpprobe = "0.1.0"
tokio = { version = "1", features = ["full"] }
anyhow = "1.0"
```

### Basic Example

```rust
use helpprobe::{probe_command, ProbeConfig};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Use default configuration
    let config = ProbeConfig::default();
    
    // Probe a command (help flag is automatically detected if missing)
    let result = probe_command("ls", &[], &config).await?;
    
    println!("Found {} options", result.options.len());
    println!("Found {} subcommands", result.subcommands.len());
    println!("Found {} arguments", result.arguments.len());
    
    // Access parsed data
    for option in &result.options {
        println!("Option: {:?}", option.long_flags);
    }
    
    Ok(())
}
```

### Advanced Examples

#### With Caching

```rust
use helpprobe::{probe_command, ProbeConfig, CacheConfig};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let config = ProbeConfig {
        timeout_secs: 10,
        require_help_flag: false,
        cache: Some(CacheConfig {
            cache_dir: helpprobe::cache::default_cache_dir(),
            enabled: true,
            max_age_secs: Some(86400 * 7), // 7 days
        }),
    };
    
    let result = probe_command("docker", &["--help".to_string()], &config).await?;
    // Subsequent calls will use cached results
    Ok(())
}
```

#### Generate Shell Completion

```rust
use helpprobe::{probe_command, generate_shell_completion, Shell, ProbeConfig};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let config = ProbeConfig::default();
    let result = probe_command("git", &[], &config).await?;
    
    // Generate Bash completion script
    let bash_completion = generate_shell_completion(&result, Shell::Bash);
    println!("{}", bash_completion);
    
    // Generate Zsh completion script
    let zsh_completion = generate_shell_completion(&result, Shell::Zsh);
    println!("{}", zsh_completion);
    
    Ok(())
}
```

#### Generate Command Builder Code

```rust
use helpprobe::{probe_command, generate_command_builder, Language, ProbeConfig};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let config = ProbeConfig::default();
    let result = probe_command("docker", &[], &config).await?;
    
    // Generate Rust builder code
    let rust_builder = generate_command_builder(&result, Language::Rust);
    println!("{}", rust_builder);
    
    // Generate Python builder code
    let python_builder = generate_command_builder(&result, Language::Python);
    println!("{}", python_builder);
    
    Ok(())
}
```

#### Validate Command Invocations

```rust
use helpprobe::{probe_command, validate_command, ProbeConfig};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let config = ProbeConfig::default();
    let result = probe_command("docker", &[], &config).await?;
    
    // Validate a command invocation
    let validation = validate_command(
        &result,
        "docker",
        &["run", "--image", "nginx", "--port", "8080"]
    );
    
    if validation.is_valid {
        println!("Command is valid!");
    } else {
        println!("Validation errors:");
        for error in &validation.errors {
            println!("  - {}: {}", error.error_type, error.message);
        }
    }
    
    Ok(())
}
```

#### Discover All Subcommands

```rust
use helpprobe::{discover_all_subcommands, ProbeConfig};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let config = ProbeConfig::default();
    
    // Recursively discover all subcommands
    let tree = discover_all_subcommands("git", &config, Some(3)).await?;
    
    println!("Command: {}", tree.command);
    println!("Total commands: {}", tree.total_commands);
    println!("Root options: {}", tree.options.len());
    
    // Traverse subcommand hierarchy
    fn print_subcommands(subcommands: &[helpprobe::SubcommandSpec], depth: usize) {
        for subcmd in subcommands {
            let indent = "  ".repeat(depth);
            println!("{}{} - {}", indent, subcmd.full_path, 
                     subcmd.description.as_deref().unwrap_or(""));
            print_subcommands(&subcmd.subcommands, depth + 1);
        }
    }
    
    print_subcommands(&tree.subcommands, 0);
    
    Ok(())
}
```

#### Generate API Documentation

```rust
use helpprobe::{probe_command, generate_api_docs, DocFormat, ProbeConfig};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let config = ProbeConfig::default();
    let result = probe_command("kubectl", &[], &config).await?;
    
    // Generate Markdown documentation
    let markdown = generate_api_docs(&result, DocFormat::Markdown);
    println!("{}", markdown);
    
    // Generate OpenAPI specification
    let openapi = generate_api_docs(&result, DocFormat::OpenAPI);
    println!("{}", openapi);
    
    Ok(())
}
```

### Available Modules and Re-exports

Commonly used types are re-exported at the crate root for convenience:

- **Core API**: `probe_command`, `discover_all_subcommands`, `discover_subcommand_hierarchy`, `ProbeConfig`
- **Models**: `ProbeResult`, `OptionSpec`, `ArgumentSpec`, `SubcommandSpec`, `CommandTree`, `OptionType`, `ArgumentType`
- **Completion**: `generate_shell_completion`, `Shell`
- **Builder**: `generate_command_builder`, `Language`
- **API Docs**: `generate_api_docs`, `DocFormat`
- **Validation**: `validate_command`, `ValidationResult`, `ValidationError`, `ValidationErrorType`
- **Cache**: `CacheConfig`, `read_cache`, `write_cache`, `clear_cache`

Additional modules available:
- `helpprobe::parser` - Lower-level parsing functions for help text
- `helpprobe::runner` - Command execution utilities

## Command-Line Options

```
USAGE:
    helpprobe [OPTIONS] -- <COMMAND> [ARGS...]

OPTIONS:
    --cache-dir <DIR>              Cache directory path (default: ~/.cache/helpprobe)
    --clear-cache                   Clear cache for the specified command
    --discover-all                  Recursively discover all subcommands
    --force                         Run command even when no help flag is present
    --generate-api-docs <FORMAT>    Generate API documentation (markdown, html, openapi, jsonschema)
    --generate-builder <LANGUAGE>   Generate command builder code (rust, python, javascript, typescript)
    --generate-completion <SHELL>   Generate shell completion script (bash, zsh, fish, powershell, nushell)
    --output <FILE>                 Output file for completion/builder/docs (prompts if exists, use --force to skip)
    --json                          Emit JSON instead of human-readable text
    --max-depth <DEPTH>             Maximum depth for recursive discovery (default: 5)
    --no-cache                      Disable caching of probe results
    --timeout-secs <SECS>           Timeout in seconds for the target command (default: 3)
    --validate                      Validate a command invocation
    --verbose                       Show raw stdout/stderr from the command
    -h, --help                      Print help information
    -V, --version                   Print version information
```

## Output Format

The JSON output includes:

- **Command Information**: Command name, arguments, exit code
- **Options**: Short/long flags, descriptions, types, defaults, choices
- **Arguments**: Names, types, required/optional, variadic, descriptions
- **Subcommands**: Names, descriptions, full paths, parent relationships
- **Environment Variables**: Names, descriptions, option mappings, defaults
- **Validation Rules**: Types, patterns, ranges, messages
- **Examples**: Command examples with descriptions and tags
- **Raw Output**: Original stdout/stderr for reference

## Use Cases

### Shell Completion Systems
Generate completion scripts for any CLI tool automatically.

### IDE Integration
Provide autocomplete and validation for CLI commands in IDEs.

### Testing Frameworks
Validate command invocations before execution in test suites.

### Documentation Generators
Auto-generate up-to-date documentation from help text.

### API Clients
Create type-safe wrappers around CLI tools.

### Configuration Management
Discover environment variables and generate config templates.

### CI/CD Pipelines
Validate and construct commands programmatically in automation scripts.

## Examples

### Example 1: Generate Bash Completion

```bash
helpprobe --generate-completion bash -- docker --help > /etc/bash_completion.d/docker
```

### Example 2: Validate Command in Script

```bash
if helpprobe --validate -- docker run --invalid-flag; then
    echo "Command is valid"
else
    echo "Command has errors"
    exit 1
fi
```

### Example 3: Generate Python Builder

```bash
helpprobe --generate-builder python -- git --help > git_builder.py

# Then use it:
python -c "
from git_builder import GitBuilder
cmd = GitBuilder().add().file('test.txt').build()
print(cmd)
"
```

### Example 4: Discover All Subcommands

```bash
helpprobe --discover-all --json -- docker --help | jq '.total_commands'
```

## Changelog

See [CHANGELOG.md](CHANGELOG.md) for a detailed list of changes, new features, and bug fixes.

## Compatibility Specification

If you're developing a CLI tool and want to ensure optimal compatibility with `helpprobe`, see [HELP_PROBE_SPEC.md](HELP_PROBE_SPEC.md) for guidelines on formatting help output. This specification covers:

- Usage line formatting
- Options and flags formatting
- Subcommands formatting
- Arguments formatting
- Section headers and organization
- Best practices for maximum discoverability

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the repository
2. Create your feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add some amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request

Please make sure to:
- Add tests for new features
- Update documentation as needed
- Follow existing code style
- Run `cargo fmt` and `cargo clippy` before submitting

## License

This project is dual-licensed under:

- **MIT License** ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT)
- **Apache License, Version 2.0** ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)

You may choose either license for your use. This dual-licensing approach is common in the Rust ecosystem and provides maximum flexibility for users and contributors.

## Acknowledgments

Built with Rust for performance and reliability. Uses heuristic parsing to handle the wide variety of help text formats in the wild.