memscope-rs - Advanced Rust Memory Analysis Toolkit
A comprehensive memory analysis toolkit with specialized tracking strategies for single-threaded, multi-threaded, and async Rust applications.
๐ฏ Four Specialized Tracking Strategies
memscope-rs provides four intelligent tracking strategies automatically selected based on your application patterns:
Strategy | Use Case | Performance | Best For |
---|---|---|---|
๐งฉ Core Tracker | Development & debugging | Zero overhead | Precise analysis with track_var! macros |
๐ Lock-free Multi-threaded | High concurrency (100+ threads) | Thread-local sampling | Production monitoring, zero contention |
โก Async Task-aware | async/await applications | < 5ns per allocation | Context-aware async task tracking |
๐ Unified Backend | Complex hybrid applications | Adaptive routing | Automatic strategy selection and switching |
๐ Quick Start Examples
๐งฉ Core Tracking (Zero Overhead)
use ;
๐ Lock-free Multi-threaded (100+ Threads)
use lockfree;
โก Async Task-aware Tracking
use async_memory;
async
๐ Unified Backend (Automatic Strategy Selection)
use ;
๐ฅ Key Features
๐ Advanced Export Formats
- JSON Export: Human-readable with interactive HTML dashboards
- Binary Export: High-performance format (5-10x faster, 60-80% smaller)
- Streaming Export: Memory-efficient for large datasets
- HTML Dashboards: Interactive real-time visualization
๐ก๏ธ Smart Pointer Support
- Automatic Detection: Rc, Arc, Box, and custom smart pointers
- Reference Counting: Accurate ref count tracking
- Lifecycle Analysis: Comprehensive ownership history
- Memory Safety: Enhanced safety analysis and validation
๐ง Production-Ready Features
- Zero Overhead: Reference tracking with no runtime cost
- Thread Safety: Robust multi-threading support up to 100+ threads
- Sampling Support: Configurable sampling for production environments
- Error Recovery: Panic-safe error handling and graceful degradation
๐ฏ Advanced Analysis
- FFI Boundary Tracking: C/C++ interop memory analysis
- Container Analysis: Vec, HashMap, BTreeMap specialized tracking
- Drop Chain Analysis: Complex destructor chain analysis
- Memory Passport: Detailed allocation lifecycle tracking
๐ Performance Benchmarks
๐ Tracking Overhead
Strategy | Overhead | Best Use Case |
---|---|---|
Reference Tracking | ~0% (zero-cost) | Development debugging |
Ownership Tracking | ~5-10% | Precise lifecycle analysis |
Lock-free Multi-threaded | ~2-8% (adaptive sampling) | High concurrency production |
Async Task-aware | < 5ns per allocation | Async applications |
๐ Export Performance
Format | Speed vs JSON | Size vs JSON | Use Case |
---|---|---|---|
Binary Export | 5-10x faster | 60-80% smaller | Production, large datasets |
JSON Export | Baseline | Baseline | Development, debugging |
Streaming Export | Memory-efficient | Variable | Large datasets, limited memory |
๐ง Scalability
Metric | Single-threaded | Multi-threaded | Async |
---|---|---|---|
Concurrency | 1 thread | 100+ threads | 50+ tasks |
Variables | 1M+ variables | 100K+ per thread | 10K+ per task |
Memory Usage | ~50KB + 100B/var | Thread-local pools | Task-local buffers |
๐ Export Performance (Real Test Data)
Module | Export Time | File Size | Use Case |
---|---|---|---|
Single-threaded | 1.3s | 1.2MB | Development analysis |
Multi-threaded | 211ms | 480KB | Production monitoring |
Async | 800ms | 800KB | Task performance analysis |
Hybrid | 2.1s | 2.5MB | Comprehensive analysis |
Based on actual test results from example applications
๐ฎ Interactive HTML Dashboards
All modules generate rich, interactive HTML dashboards:
- Memory Timeline: Real-time allocation/deallocation patterns
- Thread Analysis: Per-thread memory usage and performance metrics
- Task Insights: Async task lifecycle and resource usage
- Smart Pointer Tracking: Reference counting and relationship analysis
- Leak Detection: Automatic identification of potential memory leaks
- Performance Bottlenecks: CPU, I/O, and memory correlation analysis
๐ Try It Now
# Clone the repository
# Try each module:
# Generate HTML reports:
๐ Documentation
๐ฏ Core Tracking Modules
- Core Modules Overview - Complete comparison of all four tracking strategies
- Single-threaded Module - Zero-overhead
track_var!
macros with examples - Multi-threaded Module - Lock-free high-concurrency tracking for 20+ threads
- Async Module - Task-centric memory analysis for async/await applications
- Hybrid Module - Comprehensive cross-module analysis and visualization
๐ Complete Documentation
- Getting Started - Installation, quick start, and basic tutorials
- User Guide - Tracking macros, analysis, export formats, CLI tools
- API Reference - Complete API documentation with examples
- Examples - Real-world usage examples and integration guides
- Advanced Features - Binary format, custom allocators, performance optimization
๐ Multi-language Documentation
- English Documentation - Complete English documentation
- ไธญๆๆๆกฃ - ๅฎๆด็ไธญๆๆๆกฃ
Core Features
1. Variable Tracking
- Non-intrusive tracking: Use
track_var!
macro to track variables without breaking your existing code (we promise!) - Smart pointer support: Full support for
Rc<T>
,Arc<T>
,Box<T>
- because Rust loves its smart pointers - Lifecycle analysis: Automatic recording of variable lifecycles from birth to... well, drop
- Reference count monitoring: Real-time tracking of smart pointer reference count changes (watch those Rc clones!)
2. Memory Analysis
- Memory leak detection: Find those sneaky leaks hiding in your code
- Fragmentation analysis: Basic heap fragmentation reporting
- Usage pattern detection: Simple memory usage pattern recognition
- Performance issue identification: Spot memory-related bottlenecks
3. Data Export & Interactive Visualization
- JSON export: Export detailed memory allocation data for programmatic analysis
- Binary export: Efficient binary format for large datasets with faster I/O
- SVG visualization: Generate memory usage charts and timelines (pretty pictures!)
- ๐ฏ HTML Interactive Dashboard: Full-featured web-based dashboard with clickable charts, filterable data, and real-time analysis
- Binary โ HTML: Convert binary snapshots directly to interactive HTML dashboards
- JSON โ HTML: Transform JSON analysis data into rich web visualizations
- Multiple export modes: Fast mode, detailed mode, and "let the computer decide" mode
4. Safety Analysis
- FFI boundary tracking: Monitor memory interactions between Rust and C/C++ code
- Security violation detection: Identify potential memory safety issues
- Use-after-free detection: Catch those "oops, I used it after freeing it" moments
Available Commands and Tools
Example Programs
# Basic usage demonstration
# Comprehensive memory analysis showcase
# Complex lifecycle showcase
# Memory stress test (warning: may stress your computer too)
# Multi-threaded stress test
# Performance test
# Realistic usage with extensions
# Large-scale binary comparison
# Unsafe/FFI safety demo (for the brave souls)
# Async basic test
# Simple binary test
# JSON export test
Usage Examples
Basic Usage
use ;
Smart Pointer Tracking
use Rc;
use Arc;
// Track reference counted pointers
let rc_data = new;
track_var!;
// Track atomic reference counted pointers (for when you need thread safety)
let arc_data = new;
track_var!;
// Cloning operations are also tracked (watch the ref count go up!)
let rc_clone = clone;
track_var!;
Export Configuration
use ExportOptions;
let options = new
.include_system_allocations // Fast mode (recommended)
.verbose_logging // For when you want ALL the details
.buffer_size; // 128KB buffer (because bigger is better, right?)
if let Err = tracker.export_to_json_with_options
Performance Testing & Benchmarks
๐ฏ Quick Start Commands
# Clone and setup
# Build and test basic functionality
# Generate HTML report
๐ Available Benchmarks
# Fast benchmarks (recommended)
# Comprehensive benchmarks
# Stress testing
Build & Installation
System Requirements
- Rust: 1.85 or later (required for bincode 2.0.1 compatibility)
- OS: Linux, macOS, Windows (basically everywhere Rust runs)
- Memory: At least 4GB RAM recommended (for analyzing large projects)
From Source
# Clone the repository
# Build the project (grab a coffee, this might take a moment)
# Run tests
# Try an example
# Export to different formats
# View generated dashboards
# You can view the HTML interface examples in ./images/*.html
From Crates.io
# Add to your project
# Or manually add to Cargo.toml
Feature Flags
[]
= { = "0.1.5" }
Available features:
backtrace
- Enable stack trace collection (adds overhead, but gives you the full story)derive
- Enable derive macro support (experimental, use at your own risk)tracking-allocator
- Custom allocator support (enabled by default)
Output File Structure & Interactive Dashboard
After running programs, you'll find analysis results in the MemoryAnalysis/
directory:
โโโ basic_usage_memory_analysis.json // comprehensive memory data
โโโ basic_usage_lifetime.json // variable lifetime info
โโโ basic_usage_performance.json // performance metrics
โโโ basic_usage_security_violations.json // security analysis
โโโ basic_usage_unsafe_ffi.json // unsafe && ffi info
โโโ basic_usage_complex_types.json // complex types data
โโโ memory_report.html // interactive dashboard
๐ Interactive HTML Dashboard Features
The generated dashboard.html
provides a rich, interactive experience:
- ๐ Interactive Charts: Click and zoom on memory usage graphs
- ๐ Filterable Data Tables: Search and filter allocations by type, size, or lifetime
- ๐ Real-time Statistics: Live updating memory metrics and trends
- ๐ฏ Variable Drill-down: Click on any variable to see detailed lifecycle information
- ๐ฑ Responsive Design: Works on desktop, tablet, and mobile browsers
- ๐ Cross-references: Navigate between related allocations and smart pointer relationships
To view the dashboard:
# output html
# After running your tracked program
# Or simply double-click the HTML file in your file manager
Project Highlights
1. Non-intrusive Design
- Use macros for tracking without changing your code structure
- Variables work normally after tracking (no weird side effects)
- Selective tracking of key variables instead of global tracking (because sometimes less is more)
2. Smart Analysis
- Automatic identification of memory usage patterns and anomalies
- Smart pointer reference count change tracking
- Variable relationship analysis and dependency graph generation
3. Diverse Output Formats
- JSON data for programmatic processing and integration
- SVG charts for intuitive visualization
- HTML dashboard for interactive analysis (with actual buttons to click!)
4. Performance Optimization
- Fast export mode to reduce performance overhead
- Parallel processing support for large datasets
- Configurable buffer sizes for I/O optimization
5. Safety Analysis
- FFI boundary memory safety checks
- Automatic detection of potential security vulnerabilities
- Memory access pattern safety assessment
Comparison with Other Tools
Feature | memscope-rs | Valgrind | Heaptrack | jemalloc |
---|---|---|---|---|
Rust Native | โ | โ | โ | โ ๏ธ |
Variable Names | โ | โ | โ | โ |
Smart Pointer Analysis | โ | โ ๏ธ | โ ๏ธ | โ |
Visual Reports | โ | โ ๏ธ | โ | โ |
Production Ready | โ ๏ธ | โ | โ | โ |
Interactive Timeline | โ | โ | โ ๏ธ | โ |
Real-time Tracking | โ ๏ธ | โ | โ | โ |
Low Overhead | โ ๏ธ | โ ๏ธ | โ | โ |
Mature Ecosystem | โ | โ | โ | โ |
Honest Assessment
memscope-rs (this project)
- โ Strengths: Rust native, variable name tracking, smart pointer analysis, interactive visualization
- โ Current status: Stable for development use, comprehensive testing (2310+ tests passing)
- โ ๏ธ Limitations: Performance overhead, recommend staging testing before production use
Valgrind
- โ Strengths: Industry standard, battle-tested, comprehensive features, production-grade
- โ ๏ธ Limitations: Not Rust native, significant performance overhead, steep learning curve
- ๐ฏ Best for: Deep memory debugging, complex problem troubleshooting
Heaptrack
- โ Strengths: Mature profiling tool, good visualization, relatively low overhead
- โ ๏ธ Limitations: Mainly for C/C++, limited Rust-specific features
- ๐ฏ Best for: Performance analysis, memory usage optimization
jemalloc
- โ Strengths: Production-grade allocator, excellent performance, built-in analysis features
- โ ๏ธ Limitations: Mainly an allocator, basic analysis functionality
- ๐ฏ Best for: Production environments, performance optimization
When to Use memscope-rs
Good scenarios:
- ๐ Rust project development debugging - Want to understand specific variable memory usage
- ๐ Learning Rust memory management - Visualize ownership and borrowing concepts
- ๐งช Prototype validation - Quickly verify memory usage patterns
- ๐ฏ Smart pointer analysis - Deep dive into Rc/Arc reference count changes
Use with caution:
- โ ๏ธ Production environments - Recommend thorough testing in staging first
- โ ๏ธ High-performance requirements - Monitor tracking overhead in your specific use case
- โ ๏ธ Very large datasets - Performance may degrade with >1M allocations
- โ ๏ธ Complex memory issues - Consider using mature tools like Valgrind for deep debugging
Performance Characteristics
Based on actual testing (not marketing numbers):
Tracking Overhead
- Small programs: ~5-15% runtime overhead
- Memory usage: ~10-20% additional memory for tracking data
- Large datasets: Performance degrades with >1M allocations (optimization ongoing)
Export Performance
- Small datasets (< 1000 allocations): < 100ms
- Medium datasets (1000-10000 allocations): 100ms - 1s
- Large datasets (> 10000 allocations): Several seconds
Use Cases
โ Recommended Use Cases
Single-threaded Applications
Development debugging : Track memory usage during development
- Performance optimization : Identify memory bottlenecks and optimization opportunities
- Memory leak troubleshooting : Locate and fix memory leak issues
- Code review : Analyze code memory usage patterns
- Educational demos : Demonstrate Rust memory management mechanisms
- Algorithm analysis : Understand memory behavior of data structures and algorithms
โ Multi-threaded Applications
- Production-grade threading: Handles up to 30+ concurrent threads reliably
- Async/await support: Comprehensive Future and task tracking
- Lock-free optimizations: Reduced contention and improved performance
- Hybrid analysis: Automatic detection of mixed execution patterns
โ ๏ธ Use with Caution
- Very large datasets: Performance may degrade with >1M allocations
- High-frequency systems: Monitor performance impact in your specific use case
- Production environments: Recommend staging testing before deployment
Technical Architecture
The project uses a modular design:
- core/: Core tracking functionality and type definitions
- analysis/: Memory analysis algorithms and pattern recognition
- export/: Data export and visualization generation
- cli/: Command-line tools and user interface
- bin/: Executable analysis tools
Troubleshooting
Common Issues
Performance optimization:
# Use fast mode for reduced overhead
# Or disable expensive operations for large datasets
Export fails with large datasets:
// Use smaller buffer or exclude system allocations
let options = new
.include_system_allocations
.buffer_size;
High memory usage:
# Disable backtrace collection
Permission errors on output:
# Ensure write permissions
Platform-specific configuration:
# For optimal performance on different platforms
Contributing
This is experimental software, but we welcome contributions! Please:
- Test thoroughly - Make sure your changes don't break existing functionality
- Document limitations - Be honest about what doesn't work
- Performance test - Measure the impact of your changes
- Keep it simple - Avoid over-engineering (we have enough complexity already)
# Development workflow
License
Licensed under either of:
- Apache License, Version 2.0 (LICENSE-APACHE)
- MIT License (LICENSE-MIT)
at your option.
๐ ๏ธ Installation
Add to your Cargo.toml
:
[]
= "0.1.6"
# Optional features
[]
= ["parking-lot"]
= ["memscope-rs/derive"] # Derive macros
= ["memscope-rs/enhanced-tracking"] # Advanced analysis
= ["memscope-rs/system-metrics"] # System monitoring
๐ง CLI Tools
memscope-rs includes powerful command-line tools:
# Analyze existing memory data
# Generate comprehensive reports
# Run performance benchmarks
๐ Documentation
- API Documentation - Complete API reference
- User Guide - Step-by-step tutorials
- Examples - Real-world usage examples
- Performance Guide - Optimization tips
๐ Help Me Improve This Project
I need your feedback! While memscope-rs has comprehensive functionality, I believe it can be even better with your help.
๐ Found a Bug? Please Tell Me!
I've put tremendous effort into testing, but complex software inevitably has edge cases I haven't encountered. Your real-world usage scenarios are invaluable:
- Performance issues in your specific use case
- Compatibility problems with certain crates or Rust versions
- Unexpected behavior that doesn't match documentation
- Missing features that would make your workflow easier
๐ก How You Can Help
- Create Issues: Open an issue - no matter how small!
- Share Use Cases: Tell me how you're using memscope-rs
- Report Performance: Let me know if tracking overhead is higher than expected
- Documentation Gaps: Point out anything confusing or unclear
๐ Your Experience Matters
Every issue report helps make memscope-rs more robust for the entire Rust community. I'm committed to:
- Quick responses to reported issues
- Transparent communication about fixes and improvements
- Recognition for your contributions
Together, we can build the best memory analysis tool for Rust! ๐ฆ
๐ค Contributing
We welcome contributions! Please see our Contributing Guide for details.
Running Tests
๐ License
This project is licensed under either of:
- Apache License, Version 2.0 (LICENSE-APACHE)
- MIT License (LICENSE-MIT)
*Made with โค๏ธ and ๐ฆ by developers who care about memory (maybe too much) *