memscope-rs - Advanced Rust Memory Analysis Toolkit
A comprehensive memory analysis toolkit with specialized tracking strategies for single-threaded, multi-threaded, and async Rust applications.
๐ฏ Four Specialized Tracking Strategies
memscope-rs provides four intelligent tracking strategies automatically selected based on your application patterns:
Strategy | Use Case | Performance | Best For |
---|---|---|---|
๐งฉ Core Tracker | Development & debugging | Zero overhead | Precise analysis with track_var! macros |
๐ Lock-free Multi-threaded | High concurrency (100+ threads) | Thread-local sampling | Production monitoring, zero contention |
โก Async Task-aware | async/await applications | < 5ns per allocation | Context-aware async task tracking |
๐ Unified Backend | Complex hybrid applications | Adaptive routing | Automatic strategy selection and switching |
๐ Quick Start Examples
๐งฉ Core Tracking (Zero Overhead)
use ;
๐ Lock-free Multi-threaded (100+ Threads)
use lockfree;
โก Async Task-aware Tracking
use async_memory;
async
๐ Unified Backend (Automatic Strategy Selection)
use ;
๐ฅ Key Features
๐ Advanced Export Formats
- JSON Export: Human-readable with interactive HTML dashboards
- Binary Export: High-performance format (5-10x faster, 60-80% smaller)
- Streaming Export: Memory-efficient for large datasets
- HTML Dashboards: Interactive real-time visualization
๐ก๏ธ Smart Pointer Support
- Automatic Detection: Rc, Arc, Box, and custom smart pointers
- Reference Counting: Accurate ref count tracking
- Lifecycle Analysis: Comprehensive ownership history
- Memory Safety: Enhanced safety analysis and validation
๐ง Production-Ready Features
- Zero Overhead: Reference tracking with no runtime cost
- Thread Safety: Robust multi-threading support up to 100+ threads
- Sampling Support: Configurable sampling for production environments
- Error Recovery: Panic-safe error handling and graceful degradation
๐ฏ Advanced Analysis
- FFI Boundary Tracking: C/C++ interop memory analysis
- Container Analysis: Vec, HashMap, BTreeMap specialized tracking
- Drop Chain Analysis: Complex destructor chain analysis
- Memory Passport: Detailed allocation lifecycle tracking
๐ Performance Benchmarks
๐ Tracking Overhead
Strategy | Overhead | Best Use Case |
---|---|---|
Reference Tracking | ~0% (zero-cost) | Development debugging |
Ownership Tracking | ~5-10% | Precise lifecycle analysis |
Lock-free Multi-threaded | ~2-8% (adaptive sampling) | High concurrency production |
Async Task-aware | < 5ns per allocation | Async applications |
๐ Export Performance
Format | Speed vs JSON | Size vs JSON | Use Case |
---|---|---|---|
Binary Export | 5-10x faster | 60-80% smaller | Production, large datasets |
JSON Export | Baseline | Baseline | Development, debugging |
Streaming Export | Memory-efficient | Variable | Large datasets, limited memory |
๐ง Scalability
Metric | Single-threaded | Multi-threaded | Async |
---|---|---|---|
Concurrency | 1 thread | 100+ threads | 50+ tasks |
Variables | 1M+ variables | 100K+ per thread | 10K+ per task |
Memory Usage | ~50KB + 100B/var | Thread-local pools | Task-local buffers |
๐ Export Performance (Real Test Data)
Module | Export Time | File Size | Use Case |
---|---|---|---|
Single-threaded | 1.3s | 1.2MB | Development analysis |
Multi-threaded | 211ms | 480KB | Production monitoring |
Async | 800ms | 800KB | Task performance analysis |
Hybrid | 2.1s | 2.5MB | Comprehensive analysis |
Based on actual test results from example applications
๐ฎ Interactive HTML Dashboards
All modules generate rich, interactive HTML dashboards:
- Memory Timeline: Real-time allocation/deallocation patterns
- Thread Analysis: Per-thread memory usage and performance metrics
- Task Insights: Async task lifecycle and resource usage
- Smart Pointer Tracking: Reference counting and relationship analysis
- Leak Detection: Automatic identification of potential memory leaks
- Performance Bottlenecks: CPU, I/O, and memory correlation analysis
๐ Try It Now
# Clone the repository
# Try each module:
# Generate HTML reports:
๐ Documentation
๐ฏ Core Tracking Modules
- Core Modules Overview - Complete comparison of all four tracking strategies
- Single-threaded Module - Zero-overhead
track_var!
macros with examples - Multi-threaded Module - Lock-free high-concurrency tracking for 20+ threads
- Async Module - Task-centric memory analysis for async/await applications
- Hybrid Module - Comprehensive cross-module analysis and visualization
๐ Complete Documentation
- Getting Started - Installation, quick start, and basic tutorials
- User Guide - Tracking macros, analysis, export formats, CLI tools
- API Reference - Complete API documentation with examples
- Examples - Real-world usage examples and integration guides
- Advanced Features - Binary format, custom allocators, performance optimization
๐ Multi-language Documentation
- English Documentation - Complete English documentation
- ไธญๆๆๆกฃ - ๅฎๆด็ไธญๆๆๆกฃ
Core Features
1. Variable Tracking
- Non-intrusive tracking: Use
track_var!
macro to track variables without breaking your existing code (we promise!) - Smart pointer support: Full support for
Rc<T>
,Arc<T>
,Box<T>
- because Rust loves its smart pointers - Lifecycle analysis: Automatic recording of variable lifecycles from birth to... well, drop
- Reference count monitoring: Real-time tracking of smart pointer reference count changes (watch those Rc clones!)
2. Memory Analysis
- Memory leak detection: Find those sneaky leaks hiding in your code
- Fragmentation analysis: Basic heap fragmentation reporting
- Usage pattern detection: Simple memory usage pattern recognition
- Performance issue identification: Spot memory-related bottlenecks
3. Data Export & Interactive Visualization
- JSON export: Export detailed memory allocation data for programmatic analysis
- Binary export: Efficient binary format for large datasets with faster I/O
- SVG visualization: Generate memory usage charts and timelines (pretty pictures!)
- ๐ฏ HTML Interactive Dashboard: Full-featured web-based dashboard with clickable charts, filterable data, and real-time analysis
- Binary โ HTML: Convert binary snapshots directly to interactive HTML dashboards
- JSON โ HTML: Transform JSON analysis data into rich web visualizations
- Multiple export modes: Fast mode, detailed mode, and "let the computer decide" mode
4. Safety Analysis
- FFI boundary tracking: Monitor memory interactions between Rust and C/C++ code
- Security violation detection: Identify potential memory safety issues
- Use-after-free detection: Catch those "oops, I used it after freeing it" moments
Available Commands and Tools
Example Programs
# Basic usage demonstration
# Comprehensive memory analysis showcase
# Complex lifecycle showcase
# Memory stress test (warning: may stress your computer too)
# Multi-threaded stress test
# Performance test
# Realistic usage with extensions
# Large-scale binary comparison
# Unsafe/FFI safety demo (for the brave souls)
# Async basic test
# Simple binary test
# JSON export test
Usage Examples
Basic Usage
use ;
Smart Pointer Tracking
use Rc;
use Arc;
// Track reference counted pointers
let rc_data = new;
track_var!;
// Track atomic reference counted pointers (for when you need thread safety)
let arc_data = new;
track_var!;
// Cloning operations are also tracked (watch the ref count go up!)
let rc_clone = clone;
track_var!;
Export Configuration
use ExportOptions;
let options = new
.include_system_allocations // Fast mode (recommended)
.verbose_logging // For when you want ALL the details
.buffer_size; // 128KB buffer (because bigger is better, right?)
if let Err = tracker.export_to_json_with_options
Performance Testing & Benchmarks
๐ฏ Quick Start Commands
# Clone and setup
# Build and test basic functionality
# Generate HTML report
๐ Available Benchmarks
# Fast benchmarks (recommended)
# Comprehensive benchmarks
# Stress testing
Build & Installation
System Requirements
- Rust: 1.85 or later (required for bincode 2.0.1 compatibility)
- OS: Linux, macOS, Windows (basically everywhere Rust runs)
- Memory: At least 4GB RAM recommended (for analyzing large projects)
From Source
# Clone the repository
# Build the project (grab a coffee, this might take a moment)
# Run tests (cross your fingers)
# Try an example
# Export to different formats
# View generated dashboards
# You can view the HTML interface examples in ./images/*.html
From Crates.io
# Add to your project
# Or manually add to Cargo.toml
Feature Flags
[]
= { = "0.1.5" }
Available features:
backtrace
- Enable stack trace collection (adds overhead, but gives you the full story)derive
- Enable derive macro support (experimental, use at your own risk)tracking-allocator
- Custom allocator support (enabled by default)
Output File Structure & Interactive Dashboard
After running programs, you'll find analysis results in the MemoryAnalysis/
directory:
โโโ basic_usage_memory_analysis.json // comprehensive memory data
โโโ basic_usage_lifetime.json // variable lifetime info
โโโ basic_usage_performance.json // performance metrics
โโโ basic_usage_security_violations.json // security analysis
โโโ basic_usage_unsafe_ffi.json // unsafe && ffi info
โโโ basic_usage_complex_types.json // complex types data
โโโ memory_report.html // interactive dashboard
๐ Interactive HTML Dashboard Features
The generated dashboard.html
provides a rich, interactive experience:
- ๐ Interactive Charts: Click and zoom on memory usage graphs
- ๐ Filterable Data Tables: Search and filter allocations by type, size, or lifetime
- ๐ Real-time Statistics: Live updating memory metrics and trends
- ๐ฏ Variable Drill-down: Click on any variable to see detailed lifecycle information
- ๐ฑ Responsive Design: Works on desktop, tablet, and mobile browsers
- ๐ Cross-references: Navigate between related allocations and smart pointer relationships
To view the dashboard:
# output html
# After running your tracked program
# Or simply double-click the HTML file in your file manager
Project Highlights
1. Non-intrusive Design
- Use macros for tracking without changing your code structure
- Variables work normally after tracking (no weird side effects)
- Selective tracking of key variables instead of global tracking (because sometimes less is more)
2. Smart Analysis
- Automatic identification of memory usage patterns and anomalies
- Smart pointer reference count change tracking
- Variable relationship analysis and dependency graph generation
3. Diverse Output Formats
- JSON data for programmatic processing and integration
- SVG charts for intuitive visualization
- HTML dashboard for interactive analysis (with actual buttons to click!)
4. Performance Optimization
- Fast export mode to reduce performance overhead
- Parallel processing support for large datasets
- Configurable buffer sizes for I/O optimization
5. Safety Analysis
- FFI boundary memory safety checks
- Automatic detection of potential security vulnerabilities
- Memory access pattern safety assessment
Comparison with Other Tools
Feature | memscope-rs | Valgrind | Heaptrack | jemalloc |
---|---|---|---|---|
Rust Native | โ | โ | โ | โ ๏ธ |
Variable Names | โ | โ | โ | โ |
Smart Pointer Analysis | โ | โ ๏ธ | โ ๏ธ | โ |
Visual Reports | โ | โ ๏ธ | โ | โ |
Production Ready | โ ๏ธ | โ | โ | โ |
Interactive Timeline | โ | โ | โ ๏ธ | โ |
Real-time Tracking | โ ๏ธ | โ | โ | โ |
Low Overhead | โ ๏ธ | โ ๏ธ | โ | โ |
Mature Ecosystem | โ | โ | โ | โ |
Honest Assessment
memscope-rs (this project)
- โ Strengths: Rust native, variable name tracking, smart pointer analysis, interactive visualization
- โ ๏ธ Current status: Experimental tool, good for development debugging, noticeable performance overhead
- โ Limitations: Not mature enough, not suitable for production, relatively limited functionality
Valgrind
- โ Strengths: Industry standard, battle-tested, comprehensive features, production-grade
- โ ๏ธ Limitations: Not Rust native, significant performance overhead, steep learning curve
- ๐ฏ Best for: Deep memory debugging, complex problem troubleshooting
Heaptrack
- โ Strengths: Mature profiling tool, good visualization, relatively low overhead
- โ ๏ธ Limitations: Mainly for C/C++, limited Rust-specific features
- ๐ฏ Best for: Performance analysis, memory usage optimization
jemalloc
- โ Strengths: Production-grade allocator, excellent performance, built-in analysis features
- โ ๏ธ Limitations: Mainly an allocator, basic analysis functionality
- ๐ฏ Best for: Production environments, performance optimization
When to Use memscope-rs
Good scenarios:
- ๐ Rust project development debugging - Want to understand specific variable memory usage
- ๐ Learning Rust memory management - Visualize ownership and borrowing concepts
- ๐งช Prototype validation - Quickly verify memory usage patterns
- ๐ฏ Smart pointer analysis - Deep dive into Rc/Arc reference count changes
Not recommended scenarios:
- ๐ซ Production monitoring - Use mature tools instead
- ๐ซ High-performance requirements - Tracking overhead might be unacceptable
- ๐ซ Complex memory issues - Valgrind and friends are better
- ๐ซ Large project comprehensive analysis - Functionality and stability not sufficient yet
Performance Characteristics
Based on actual testing (not marketing numbers):
Tracking Overhead
- Small programs: ~5-15% runtime overhead (not too bad!)
- Memory usage: ~10-20% additional memory for tracking data
- Large datasets: Performance degrades significantly (we're working on it)
Export Performance
- Small datasets (< 1000 allocations): < 100ms (blink and you'll miss it)
- Medium datasets (1000-10000 allocations): 100ms - 1s (time for a sip of coffee)
- Large datasets (> 10000 allocations): Several seconds (time for a full coffee break)
Known Limitations
- Thread safety: Basic support, may have issues under heavy concurrency
- Memory leaks: Tracking itself may leak memory in some scenarios (ironic, we know)
- Platform support: Limited testing on different platforms
- Error handling: Some errors are silently ignored (we're working on being more vocal)
Current Development Status
What works reliably:
- โ Single-threaded variable tracking: Core functionality works well in single-threaded environments
- โ
Multi-format data export:
- JSON export with complete allocation data
- Binary export for efficient large dataset handling
- Direct binary โ HTML conversion with interactive dashboards
- JSON โ HTML transformation with rich visualizations
- โ Interactive HTML dashboard: Feature-rich visualization with clickable elements, variable relationship graphs, 3D memory layout
- โ Smart pointer support: Full Rc, Arc, Box tracking with reference counting
- โ Memory analysis: Basic leak detection and pattern analysis
- โ CLI tools and examples: All demonstration programs run successfully
Known critical issues (honest assessment):
- โ ๏ธ Multi-threading deadlocks: Global tracker with multiple mutexes causes deadlocks(20 threads limit or be killed)
- โ ๏ธ Performance overhead: 5-15% runtime overhead, degrades significantly with large datasets
- โ ๏ธ 934 unsafe
unwrap()
calls: Potential panic points that need proper error handling- Risk: Application can panic unexpectedly during memory tracking operations
- Mitigation: Use
MEMSCOPE_TEST_MODE=1
for safer fallback behavior - Status: Active work in progress to replace with safe alternatives
- โ ๏ธ Thread safety: Basic support only, not thoroughly tested under concurrency
- โ ๏ธ Memory leaks in tracker itself: Tracking system can leak memory (ironic but true)
- โ ๏ธ Inconsistent API design: Some modules use different patterns and conventions
- โ ๏ธ Limited platform testing: Mainly tested on specific development environments
Production readiness:
- ๐ซ Not suitable for production: Current status is experimental/development tool only
- ๐ซ No stability guarantees: APIs may change, memory safety not fully validated
- โ Good for development debugging: Excellent for understanding memory patterns during development
- โ Educational value: Great for learning Rust memory management concepts
Planned improvements :
High Priority :
- ๐ Multi-threading safety: Implement lock-free tracking architecture
- ๐ Replace dangerous
unwrap()
calls: 154 calls need proper error handling - ๐ Performance optimization: Reduce overhead for large datasets
- ๐ Memory leak fixes: Fix tracker's own memory leaks
Medium Priority :
- ๐ API consistency: Standardize interfaces across modules
- ๐ Better error handling: Comprehensive error reporting system
- ๐ Cross-platform testing: Validate on Windows, macOS, Linux
- ๐ Documentation improvements: More examples and use cases
Future Goals :
- ๐ Production readiness: Stability and performance validation
- ๐ Advanced analysis: ML-based memory pattern detection
- ๐ Integration support: IDE plugins and CI/CD integration
- ๐ Real-time monitoring: Live memory tracking dashboard
Use Cases
โ Recommended Use Cases
Single-threaded Applications
Development debugging : Track memory usage during development
- Performance optimization : Identify memory bottlenecks and optimization opportunities
- Memory leak troubleshooting : Locate and fix memory leak issues
- Code review : Analyze code memory usage patterns
- Educational demos : Demonstrate Rust memory management mechanisms
- Algorithm analysis : Understand memory behavior of data structures and algorithms
โ ๏ธ Use with Caution
Multi-threaded Applications
- Only with workarounds : Use environment variables to disable problematic features
- Testing environments : Single-threaded test execution with
RUST_TEST_THREADS=1
- Development debugging : Limited tracking with
MEMSCOPE_DISABLE_GLOBAL=1
Required precautions for multi-threaded use:
# Choose one of these approaches:
# Safest: disables global tracking
# Skips heavy operations
# Uses simplified tracking
# Forces single-threaded execution
๐ซ Not Recommended
- Production environments: Not stable enough, use mature tools instead
- High-performance applications : Tracking overhead may be unacceptable
- Critical systems : Potential deadlocks and memory leaks in tracker itself
- Large-scale applications : Performance degrades significantly with large datasets
- Concurrent servers : Multi-threading limitations make it unsuitable
Technical Architecture
The project uses a modular design:
- core/: Core tracking functionality and type definitions
- analysis/: Memory analysis algorithms and pattern recognition
- export/: Data export and visualization generation
- cli/: Command-line tools and user interface
- bin/: Executable analysis tools
Troubleshooting
Common Issues
Application hangs or deadlocks in multi-threaded code:
# Use one of these environment variables:
# Completely disable global tracking
# Use simplified tracking logic
# Skip heavy operations
# Force single-threaded execution
Panic with "unwrap() called on None":
# Enable safer fallback behavior
# Or disable specific features
Export fails with large datasets:
// Use smaller buffer or exclude system allocations
let options = new
.include_system_allocations
.buffer_size;
High memory usage:
# Disable backtrace collection
Permission errors on output:
# Ensure write permissions
Performance degradation:
# Use fast mode with reduced tracking
# Or disable expensive operations
Contributing
This is experimental software, but we welcome contributions! Please:
- Test thoroughly - Make sure your changes don't break existing functionality
- Document limitations - Be honest about what doesn't work
- Performance test - Measure the impact of your changes
- Keep it simple - Avoid over-engineering (we have enough complexity already)
# Development workflow
License
Licensed under either of:
- Apache License, Version 2.0 (LICENSE-APACHE)
- MIT License (LICENSE-MIT)
at your option.
๐ ๏ธ Installation
Add to your Cargo.toml
:
[]
= "0.1.6"
# Optional features
[]
= ["parking-lot"]
= ["memscope-rs/derive"] # Derive macros
= ["memscope-rs/enhanced-tracking"] # Advanced analysis
= ["memscope-rs/system-metrics"] # System monitoring
๐ง CLI Tools
memscope-rs includes powerful command-line tools:
# Analyze existing memory data
# Generate comprehensive reports
# Run performance benchmarks
๐ Documentation
- API Documentation - Complete API reference
- User Guide - Step-by-step tutorials
- Examples - Real-world usage examples
- Performance Guide - Optimization tips
๐ Help Me Improve This Project
I need your feedback! While memscope-rs has comprehensive functionality, I believe it can be even better with your help.
๐ Found a Bug? Please Tell Me!
I've put tremendous effort into testing, but complex software inevitably has edge cases I haven't encountered. Your real-world usage scenarios are invaluable:
- Performance issues in your specific use case
- Compatibility problems with certain crates or Rust versions
- Unexpected behavior that doesn't match documentation
- Missing features that would make your workflow easier
๐ก How You Can Help
- Create Issues: Open an issue - no matter how small!
- Share Use Cases: Tell me how you're using memscope-rs
- Report Performance: Let me know if tracking overhead is higher than expected
- Documentation Gaps: Point out anything confusing or unclear
๐ Your Experience Matters
Every issue report helps make memscope-rs more robust for the entire Rust community. I'm committed to:
- Quick responses to reported issues
- Transparent communication about fixes and improvements
- Recognition for your contributions
Together, we can build the best memory analysis tool for Rust! ๐ฆ
๐ค Contributing
We welcome contributions! Please see our Contributing Guide for details.
Running Tests
๐ License
This project is licensed under either of:
- Apache License, Version 2.0 (LICENSE-APACHE)
- MIT License (LICENSE-MIT)
*Made with โค๏ธ and ๐ฆ by developers who care about memory (maybe too much) *