CLI Testing Specialist

Version: 1.0.9 Last Updated: 2025-11-12 Status: Production Ready License: MIT

A comprehensive testing framework that automatically validates the quality and security of CLI tools. Built with Rust for maximum performance and reliability.

📑 Table of Contents

Overview
Quick Start
Installation
Target Tools
Features
Report Formats
CI/CD Integration
Security Features
Configuration
File Structure
License
Contributing
Support

Overview

CLI Testing Specialist is a production-ready testing framework that automatically generates and executes comprehensive test suites for CLI tools.

Key Features

🎯 100% Inference Accuracy: Execution-based no-args behavior detection (v1.0.9)
🔒 Security Testing: OWASP-compliant automated security scanning
✅ Comprehensive Validation: 9 test categories, 45-55 test cases (configurable)
🎯 Input Validation: Automatic validation of numeric/path/enum options
🛡️ Destructive Operation Testing: Confirmation prompt and safety validation
🐚 Multi-Shell Support: bash/zsh compatibility testing
📊 Detailed Reports: Markdown/JSON/HTML/JUnit XML formats (Interactive HTML with filtering)
🔄 CI/CD Ready: GitHub Actions & GitLab CI integration examples
⚡ High Performance: Written in Rust for blazing-fast execution
📦 Single Binary: Zero runtime dependencies

Quick Start

# 1. Install cli-testing-specialist
cargo install --git https://github.com/sanae-abe/cli-testing-specialist

# 2. Analyze CLI tool
cli-testing-specialist analyze /usr/bin/curl -o curl-analysis.json

# 3. Generate tests (all categories)
cli-testing-specialist generate curl-analysis.json -o curl-tests -c all

# 4. Run tests and generate reports
cli-testing-specialist run curl-tests -f all -o curl-reports

# 5. View HTML report
open curl-reports/curl-tests-report.html  # macOS
# xdg-open curl-reports/curl-tests-report.html  # Linux

Installation

From Source (Recommended)

# Install from GitHub
cargo install --git https://github.com/sanae-abe/cli-testing-specialist

# Verify installation
cli-testing-specialist --version

From Crates.io (Coming Soon)

cargo install cli-testing-specialist

Dependencies

Required for Test Execution

BATS (Bash Automated Testing System): Test execution framework

# macOS
brew install bats-core

# Ubuntu/Debian
sudo apt-get install bats

# Manual installation
git clone https://github.com/bats-core/bats-core.git
cd bats-core
sudo ./install.sh /usr/local

CLI Testing Specialist Binary

Zero runtime dependencies: Single self-contained binary

Target Tools

cli-testing-specialist is optimized for standard CLI tools. See docs/TARGET-TOOLS.md for detailed guidance.

✅ High Compatibility (70-90% success rate)

Standard CLI tools (curl, git, ls, cat)
Tools with standard options (--help, --version)
Non-interactive tools
Example: package-publisher (Node.js CLI)

⚠️ Medium Compatibility (30-60% success rate)

Configuration-driven tools (cmdrun with commands.toml)
Custom UI implementations (cldev with dialoguer, i18n)
Recommendation: Use "informational mode" in CI

❌ Low Compatibility (not recommended)

Interactive shells (mysql, psql, redis-cli)
Container management (docker, podman)
Domain-specific tools with custom protocols

🌐 Language Support Status

✅ Tested & Supported:

C/C++ (getopt, custom parsers) - curl, git
Rust (clap) - backup-suite, cmdrun, cldev
Node.js (commander) - package-publisher

⚠️ Untested (Estimated 70-80% compatible):

Go (cobra, urfave/cli) - gh, kubectl, docker
Python (argparse requires fix, click/typer likely work)
Ruby (thor, gli)

📋 Planned Testing: v1.1.0+

See docs/TARGET-TOOLS.md for complete guidelines, language-specific details, and best practices.

Features

Category	Description	Default
Basic Validation	Help, version, exit codes	✅ Enabled
Help	Comprehensive subcommand help validation	✅ Enabled
Security	Command injection, null bytes, path traversal	✅ Enabled
Path Handling	Special characters, deep hierarchies, Unicode	✅ Enabled
Multi-Shell	bash/zsh compatibility	✅ Enabled
Input Validation	Numeric/path/enum option validation	✅ Enabled
Destructive Operations	Confirmation prompts, --yes/--force flags	✅ Enabled
Performance	Startup time, memory usage	✅ Enabled
Directory Traversal	Large file count, deep nesting, symlink loops	⚠️ Opt-in*

* Directory Traversal tests are opt-in via --include-intensive flag to prevent CI environment issues (disk space, resource limits).

Test Generation Options

# Default: All categories except Directory Traversal
cli-testing-specialist generate analysis.json -c all

# Include resource-intensive tests
cli-testing-specialist generate analysis.json -c all --include-intensive

# Specific categories only
cli-testing-specialist generate analysis.json -c basic,security,path

Report Formats

1. Markdown Format (`.md`)

Human-readable format for GitHub/GitLab display

cli-testing-specialist run ./tests -f markdown -o ./reports

2. JSON Format (`.json`)

Optimal for CI/CD integration and programmatic processing

cli-testing-specialist run ./tests -f json -o ./reports

# Parse with jq
jq '[.suites[].tests[] | select(.status == "passed")] | length' reports/tests-report.json

3. HTML Format (`.html`)

Interactive browser display with search and filtering

cli-testing-specialist run ./tests -f html -o ./reports
open reports/tests-report.html

HTML Features:

📊 Visual statistics cards (Passed/Failed/Skipped/Duration)
📈 Progress bar showing success rate
📋 Test suite breakdown with detailed results
🎨 Clean, professional design (Bootstrap 5)
🚀 Zero CDN dependencies (embedded CSS)
📱 Fully responsive layout
⚡ Fast loading with self-contained HTML

4. JUnit XML Format (`.xml`)

CI/CD integration (GitHub Actions, GitLab CI, Jenkins)

cli-testing-specialist run ./tests -f junit -o ./reports

5. All Formats at Once

cli-testing-specialist run ./tests -f all -o ./reports

For details, see docs/REPORT-FORMATS.md.

CI/CD Integration

GitHub Actions

name: CLI Testing

on: [push, pull_request]

jobs:
  cli-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install cli-testing-specialist
        run: |
          cargo install --git https://github.com/sanae-abe/cli-testing-specialist

      - name: Build your CLI
        run: cargo build --release

      - name: Analyze CLI
        run: |
          cli-testing-specialist analyze \
            ./target/release/your-cli \
            -o analysis.json

      - name: Generate tests
        run: |
          cli-testing-specialist generate \
            analysis.json \
            -o tests \
            -c all

      - name: Install BATS
        run: sudo apt-get install -y bats

      - name: Run tests
        run: |
          cli-testing-specialist run \
            tests \
            -f all \
            -o reports

      - name: Upload reports
        uses: actions/upload-artifact@v4
        with:
          name: test-reports
          path: reports/

GitLab CI

cli-test:
  image: rust:latest
  script:
    - cargo install --git https://github.com/sanae-abe/cli-testing-specialist
    - cargo build --release
    - cli-testing-specialist analyze ./target/release/your-cli -o analysis.json
    - cli-testing-specialist generate analysis.json -o tests -c all
    - apt-get update && apt-get install -y bats
    - cli-testing-specialist run tests -f all -o reports
  artifacts:
    paths:
      - reports/

Security Features

Security Test Philosophy

IMPORTANT: Security tests expect tools to reject malicious inputs with non-zero exit codes.

// Command injection test
cli-test --name 'test; rm -rf /'
// Expected: exit code 1 (rejection) ✅
// NOT exit code 0 (success) ❌

Security Test Categories

Injection Attacks: Command injection, null byte injection
- Expected behavior: Tool rejects with exit code 1
- Tags: injection, critical
Path Traversal: Directory traversal attempts
- Expected behavior: Tool rejects with exit code 1
- Tags: path-traversal, critical
Buffer Overflow: Extremely long inputs
- Expected behavior: Graceful handling (informational)
- Tags: buffer-overflow, informational

Input Validation

CLI binary path verification
Path canonicalization (prevent path traversal)
Timeout enforcement (prevent hang)
Safe command execution

Configuration

Command-Line Options

# Analyze with custom output
cli-testing-specialist analyze /usr/bin/curl -o custom-analysis.json

# Generate specific categories
cli-testing-specialist generate analysis.json -c basic,security,path

# Include intensive tests
cli-testing-specialist generate analysis.json -c all --include-intensive

# Run with timeout
cli-testing-specialist run tests --timeout 120 -f all -o reports

# Skip specific categories
cli-testing-specialist run tests --skip destructive-ops,directory-traversal

Environment Variables

# Skip directory traversal tests
export SKIP_DIRECTORY_TRAVERSAL_TESTS=1
cli-testing-specialist run tests -f all -o reports

File Structure

cli-testing-specialist/
├── Cargo.toml              # Rust project configuration
├── Cargo.lock
├── README.md               # This file
├── LICENSE                 # MIT License
├── src/
│   ├── main.rs             # Entry point
│   ├── lib.rs              # Library exports
│   ├── cli/                # CLI interface (clap)
│   ├── analyzer/           # CLI analysis engine
│   ├── generator/          # Test case generation
│   ├── runner/             # BATS test execution
│   ├── reporter/           # Report generation (MD/JSON/HTML/JUnit)
│   ├── types/              # Type definitions
│   ├── error.rs            # Error types
│   └── utils/              # Utilities
├── tests/                  # Integration tests
├── benches/                # Performance benchmarks
└── docs/
    ├── RUST_V1_DESIGN.md   # Design document
    ├── TARGET-TOOLS.md     # Target tool guidelines
    └── REPORT-FORMATS.md   # Report format guide

License

MIT License - see LICENSE file for details.

Contributing

Contributions are welcome! Please:

Fork the repository
Create a feature branch
Make your changes with tests
Submit a pull request

For major changes, please open an issue first to discuss the proposed changes.

Support

Documentation: docs/ directory
- Design Document - Architecture and implementation
- Target Tools Guide - Compatibility guidelines
- Report Formats - Report format reference
Issues: GitHub Issues
Discussions: GitHub Discussions

Changelog

v1.0.9 (2025-11-12) - Execution-based Inference 🎯

Revolutionary No-Args Behavior Detection:

100% Inference Accuracy: Directly executes binaries to measure actual exit codes
Solves cldev-type CLI Problem: Identical Usage patterns ([OPTIONS] <COMMAND>), different behaviors
Safety Measures: 1-second timeout, output discarding, non-TTY mode, interactive tools detection
Test Success Rate: 93.3% → 100% (15/15 tests passed across cldev/cmdrun/backup-suite)

HTML Report Improvements:

Fixed filter bug: Skipped filter now correctly hides error detail rows
Interactive filtering works perfectly for All/Passed/Failed/Skipped states

Technical:

New method: BehaviorInferrer::execute_and_measure()
Dependency: wait-timeout = "0.2" for process timeout handling
109 unit tests passing (100% pass rate)

v1.0.8 (2025-11-12)

No-Args Test Assertion Relaxation:

Removed strict output assertions (exit codes only)
Test Success Rate: 86.7% → 93.3%
Reason: CLIs show different error formats (short message vs full help)

v1.0.7 (2025-11-12)

Clippy Warning Fix:

Renamed TestCategory::default() to standard_categories()
Added Git Hooks configuration in .claude/CLAUDE.md

v1.0.6 (2025-11-12)

Required Arguments Detection:

Automatic extraction from Usage lines (<ID>, <FILE>)
Test template improvements (dummy arguments, dynamic option selection)
Test Success Rate: 85.0% → 92.9% for cmdrun

v1.0.5 (2025-11-12)

Dependency Updates:

All 7 Dependabot PRs merged (GitHub Actions, indicatif 0.18, thiserror 2.0, colored 3.0, criterion 0.7)
MSRV bumped to Rust 1.80
0 vulnerabilities with cargo audit

v1.0.4 (2025-01-12)

Documentation Improvements 📚:

Added comprehensive docs/TARGET-TOOLS.md guide for tool compatibility assessment
Tool classification system: High/Medium/Low compatibility with success rate estimates

v1.0.3 (2025-01-12)

Critical Security Test Fix 🔒:

Fixed security test design to accept any non-zero exit code (not just exit code 1)
Now correctly handles Unix convention: exit code 2 for usage errors

v1.0.2 (2025-01-12)

Security Fix 🔒:

Security tests now correctly expect tools to reject attacks
Directory Traversal tests now opt-in via --include-intensive flag

v1.0.1 (2025-01-11)

Initial production release
9 test categories with 45-55 test cases
4 report formats (Markdown/JSON/HTML/JUnit)

Built with ❤️ using Rust

cli-testing-specialist 1.0.9