sphinx-ultra 0.3.0

High-performance Rust-based Sphinx documentation builder for large codebases
Documentation

Sphinx Ultra Rust Builder

Crates.io CI Documentation Release License: MIT Rust Sponsor

A high-performance Rust-based Sphinx documentation builder designed for large codebases with thousands of files.

โš ๏ธ Development Status

๐Ÿšง This project is currently under active development and is NOT recommended for production usage.

Current Focus: The primary goal is validation and experimentation rather than producing perfectly matched Sphinx builds. We are:

  • โœ… Validating the core architecture and performance concepts
  • โœ… Testing parallel processing capabilities on large documentation sets
  • โœ… Experimenting with Rust-based parsing and rendering
  • โš ๏ธ NOT aiming for 100% Sphinx compatibility yet
  • โš ๏ธ NOT ready for production documentation workflows

Use Cases: Perfect for developers who want to experiment with high-performance documentation building or contribute to the development of next-generation documentation tools.

โœจ Features

โœ… Currently Implemented

  • ๐Ÿš€ Blazing Fast: Parallel processing with Rust's performance
  • ๐Ÿ“Š Scalable: Handle large documentation projects efficiently (tested with 50+ files in ~44ms)
  • ๐Ÿ”„ Incremental Builds: Smart caching system for faster rebuilds
  • ๐Ÿ“ File Processing: Support for RST and Markdown files
  • ๐Ÿ”ง Configuration: Multiple configuration formats (conf.py, YAML, JSON)
  • ๐Ÿ“‚ File Pattern Matching: 100% Sphinx-compatible include_patterns and exclude_patterns support
  • ๐Ÿ“Š Statistics: Project analysis and build metrics
  • โš ๏ธ Validation: Document validation with warning/error reporting
  • ๐Ÿ—๏ธ CLI Interface: Complete command-line interface (build, clean, stats)
  • ๐Ÿ“ฆ Static Assets: Automatic copying of static files and assets
  • ๐ŸŽฏ Domain System: Complete cross-reference validation with Python and RST domains
  • ๐Ÿ”— Reference Validation: Comprehensive validation of :func:, :class:, :doc:, :ref: references
  • ๐Ÿ’ก Smart Suggestions: Intelligent suggestions for broken references

๐Ÿšง Partially Implemented

  • ๐Ÿ” Search Index: Framework exists but search functionality not active
  • ๐Ÿ› ๏ธ Extensions: Basic extension system with limited Sphinx extension support
  • ๐ŸŽจ Themes: Basic theme structure but no advanced theming

๐Ÿ“‹ Planned Features

For detailed development roadmap, see Validation Features Plan which outlines our validation-focused approach.

Phase 1 (Next 2 months):

  • ๐Ÿ—๏ธ Domain System: Sphinx-compatible domain registration and cross-reference validation
  • ๐Ÿ“ Directive Validation: Complete directive/role validation system
  • ๐Ÿ“– Document Structure: TOC tree and hierarchy validation

Phase 2-4 (Months 3-8):

  • ๐Ÿ” Content Constraints: Field validation and workflow checking
  • ๏ฟฝ Extension Framework: Plugin validation and compatibility
  • ๐Ÿ“š Code Documentation: Autodoc-style validation
  • ๐ŸŒ Internationalization: Translation completeness validation

Advanced UI Features (search indexing, complex templating, etc.) are intentionally deferred until validation foundation is solid.

  • ๐ŸŒ Live Server: Development server with live reload
  • ๏ฟฝ๏ธ File Watching: Automatic rebuilds on file changes
  • ๐Ÿ”Œ Plugin System: Full plugin architecture for custom functionality
  • ๐Ÿ“ฑ Mobile Friendly: Responsive design optimization
  • ๐Ÿ–ผ๏ธ Image Optimization: Automatic image processing and optimization
  • ๐Ÿ“ฆ Asset Bundling: Advanced asset optimization and bundling

Note: This project is in active development. The core build functionality works reliably, but advanced features are still being developed.

๐Ÿš€ Quick Start

Prerequisites

  • Rust 1.70+
  • Cargo

Installation

# Clone and build from source
git clone https://github.com/salioglu/sphinx-ultra.git
cd sphinx-ultra
cargo build --release

# The binary will be available at target/release/sphinx-ultra

Basic Usage

# Build documentation
./target/release/sphinx-ultra build --source docs --output _build

# Clean build artifacts
./target/release/sphinx-ultra clean --output _build

# Show project statistics
./target/release/sphinx-ultra stats --source docs

# Get help
./target/release/sphinx-ultra --help

Available Commands

  • build: Build documentation from source files
  • clean: Remove build artifacts and output files
  • stats: Display project statistics and analysis

Build Options

# Parallel processing
sphinx-ultra build --jobs 8 --source docs --output _build

# Incremental builds (faster rebuilds)
sphinx-ultra build --incremental --source docs --output _build

# Clean before build
sphinx-ultra build --clean --source docs --output _build

# Save warnings to file
sphinx-ultra build --warning-file warnings.log --source docs --output _build

# Fail on warnings (useful for CI)
sphinx-ultra build --fail-on-warning --source docs --output _build

๐Ÿ”ง Configuration

Sphinx Ultra supports multiple configuration formats and can auto-detect your setup:

Configuration Priority

  1. conf.py (Sphinx standard) - Automatically detected and parsed
  2. sphinx-ultra.yaml - Native YAML configuration
  3. sphinx-ultra.yml - Alternative YAML format
  4. sphinx-ultra.json - JSON configuration
  5. Default settings - Used if no config file found

Sphinx conf.py Support

Sphinx Ultra can read and parse existing Sphinx conf.py files:

# conf.py (existing Sphinx configuration works)
project = 'My Documentation'
version = '1.0'
extensions = ['sphinx.ext.autodoc', 'sphinx.ext.viewcode']
html_theme = 'sphinx_rtd_theme'

YAML Configuration

Create a sphinx-ultra.yaml file for native configuration:

# Project information
project: "My Documentation"
version: "1.0.0"
copyright: "2024, My Company"

# Build settings
parallel_jobs: 8
max_cache_size_mb: 500
cache_expiration_hours: 24

# Output configuration
output:
  html_theme: "sphinx_rtd_theme"
  syntax_highlighting: true
  highlight_theme: "github"
  search_index: true
  minify_html: false

# File pattern matching (Sphinx-compatible)
include_patterns:
  - "**/*.rst"
  - "**/*.md"
exclude_patterns:
  - "_build/**"
  - "drafts/**"

# Extensions (limited support currently)
extensions:
  - "sphinx.ext.autodoc"
  - "sphinx.ext.viewcode"
  - "sphinx.ext.intersphinx"

# Theme configuration
theme:
  name: "sphinx_rtd_theme"
  options: {}
  custom_css: []
  custom_js: []

# Optimization settings
optimization:
  parallel_processing: true
  incremental_builds: true
  document_caching: true

Configuration Fields

Most standard Sphinx configuration options are supported including:

  • Project metadata (project, version, copyright, author)
  • HTML output options (theme, static paths, CSS/JS files)
  • Extension configuration
  • Template and static file paths
  • File pattern matching (include_patterns, exclude_patterns) - Full compatibility guide
  • Build optimization settings

๐Ÿ“ˆ Performance Benchmarks

Real performance test results on documentation projects:

Files Build Time Processing Rate Memory Usage
2 files 8ms 250 files/sec ~10MB
51 files 44ms 1,159 files/sec ~15MB
100+ files ~85ms* 1,176 files/sec* ~20MB*

*Projected based on linear scaling

Performance Features

  • Parallel Processing: Utilizes all CPU cores for maximum throughput
  • Smart Caching: Incremental builds only process changed files
  • Memory Efficient: Low memory footprint even for large projects
  • Fast Parsing: Optimized RST and Markdown parsing
  • Minimal I/O: Efficient file operations and batch processing

Comparison Notes

While we don't have direct Sphinx comparison benchmarks yet, the processing speeds above represent significant performance improvements for documentation builds. The actual performance gain depends on:

  • Number of files and their complexity
  • Available CPU cores
  • Disk I/O speed
  • Whether incremental builds are enabled

๐Ÿ—๏ธ Architecture

The Rust builder consists of several key components:

  • Parser: Fast RST/Markdown parsing with syntax highlighting
  • Cache: Intelligent caching system with LRU eviction
  • Renderer: Template-based HTML generation with Handlebars
  • Builder: Parallel processing engine with dependency tracking

๐Ÿ” Advanced Usage

Incremental Builds

Enable faster rebuilds by only processing changed files:

sphinx-ultra build --incremental --source docs --output _build

Parallel Processing

Control the number of parallel jobs:

# Use 16 parallel jobs for maximum performance on large projects
sphinx-ultra build --jobs 16 --source docs --output _build

# Use 1 job for debugging or memory-constrained environments  
sphinx-ultra build --jobs 1 --source docs --output _build

Warning and Error Handling

# Save all warnings and errors to a log file
sphinx-ultra build --warning-file build.log --source docs --output _build

# Treat warnings as errors (useful for CI/CD)
sphinx-ultra build --fail-on-warning --source docs --output _build

# Combine both for strict CI builds
sphinx-ultra build -w build.log -W --source docs --output _build

Configuration File Usage

# Use a specific configuration file
sphinx-ultra build --config my-config.yaml --source docs --output _build

# Configuration auto-detection order:
# 1. conf.py (if present)
# 2. sphinx-ultra.yaml  
# 3. sphinx-ultra.yml
# 4. sphinx-ultra.json
# 5. Default configuration

Clean Builds

# Clean output directory before building
sphinx-ultra build --clean --source docs --output _build

# Or clean manually
sphinx-ultra clean --output _build

Project Analysis

# Get detailed project statistics
sphinx-ultra stats --source docs

Output includes:

  • Number of source files discovered
  • Total lines of documentation
  • Average and largest file sizes
  • Directory depth analysis
  • Cross-reference count

๐Ÿ› Debugging and Troubleshooting

Enable Verbose Logging

# Debug-level logging for detailed build information
sphinx-ultra --verbose build --source docs --output _build

# Or set environment variable
RUST_LOG=debug sphinx-ultra build --source docs --output _build

Common Issues

Configuration Loading Errors

  • Ensure YAML/JSON syntax is valid
  • Check that required fields are present
  • Use --config to specify config file explicitly

Build Failures

  • Check file permissions in source and output directories
  • Verify source files are valid RST/Markdown
  • Review warning output for specific issues

Performance Issues

  • Reduce parallel jobs if memory-constrained: --jobs 1
  • Enable incremental builds: --incremental
  • Check for large files that may slow processing

Getting Help

  • Use sphinx-ultra --help for command overview
  • Use sphinx-ultra build --help for build options
  • Check project issues on GitHub
  • Enable verbose logging for debugging

๐Ÿค Contributing

We welcome contributors! This project is in active development and needs help with:

  • ๐Ÿงช Testing: Try the builder on various documentation projects
  • ๐Ÿ› Bug Reports: Report issues with parsing, rendering, or performance
  • ๐Ÿ’ก Feature Ideas: Suggest improvements or new capabilities
  • ๐Ÿ“ Documentation: Help improve setup guides and usage examples
  • ๐Ÿ”ง Code: Contribute to core features, optimizations, or new functionality

Development Setup

# Clone and build
git clone https://github.com/salioglu/sphinx-ultra.git
cd sphinx-ultra
./dev.sh setup

# Install git pre-commit hooks (recommended)
./dev.sh install-hooks

# Run development commands
./dev.sh fmt           # Format code
./dev.sh clippy        # Run linter
./dev.sh test          # Run tests
./dev.sh pre-commit    # Run all pre-commit checks
./dev.sh check         # Run all checks including tests

# Build documentation
./dev.sh docs

Please see CONTRIBUTING.md for detailed guidelines.

Priority Areas: We need help with:

  • ๐Ÿงช Testing: Try the builder on various documentation projects and report results
  • ๐Ÿ› Bug Reports: Report issues with parsing, rendering, or performance
  • ๐Ÿ’ก Feature Validation: Test existing features and suggest improvements
  • ๐Ÿ“ Documentation: Help improve setup guides and usage examples
  • ๐Ÿ”ง Core Features: Contribute to parsing, theming, or search functionality
  • ๐ŸŽจ Themes: Develop modern, responsive documentation themes
  • ๐Ÿ”Œ Extensions: Expand Sphinx extension compatibility

What Currently Works Well

  • Basic RST and Markdown processing
  • Fast parallel builds
  • Configuration auto-detection
  • File validation and warning systems
  • Incremental caching

What Needs Development

  • Advanced theming and templating
  • Search index functionality
  • Live development server
  • Full Sphinx directive compatibility

๏ฟฝ Releases

This project uses an automated release system with version validation to ensure consistency.

For Users

Download pre-built binaries from the Releases page.

For Maintainers

# Setup release environment (one-time)
./scripts/setup.sh

# Create a new patch release (0.1.2 โ†’ 0.1.3)
./scripts/release.sh --patch

# Create a new minor release (0.1.2 โ†’ 0.2.0)
./scripts/release.sh --minor

# Create a new major release (0.1.2 โ†’ 1.0.0)
./scripts/release.sh --major

# Preview what a release would do
./scripts/release.sh --dry-run --patch

The release script automatically:

  • โœ… Runs tests to ensure quality
  • โœ… Updates Cargo.toml version
  • โœ… Creates and pushes git tags
  • โœ… Triggers GitHub Actions to build and publish

Version Safety: The system prevents version mismatches between git tags and Cargo.toml. See scripts/README.md for detailed documentation.

๏ฟฝ๐Ÿ“„ License

This project is licensed under the MIT License - see the LICENSE file for details.