calltrace-rs 1.1.4

# CallTrace Troubleshooting Guide

This guide helps diagnose and resolve common issues when using CallTrace for function call tracing.

## Table of Contents

- [Quick Diagnostics](#quick-diagnostics)
- [Setup Issues](#setup-issues)
- [Compilation Problems](#compilation-problems)
- [Runtime Issues](#runtime-issues)
- [Performance Problems](#performance-problems)
- [Output Issues](#output-issues)
- [Advanced Debugging](#advanced-debugging)
- [Frequently Asked Questions](#frequently-asked-questions)

## Quick Diagnostics

Run these commands to quickly identify common issues:

```bash
# Check if CallTrace library exists
ls -la ./target/debug/libcalltrace.so
ls -la ./.target/debug/libcalltrace.so

# Verify GCC supports instrumentation
gcc --help=instrumentation | grep finstrument-functions

# Test basic library loading
ldd ./your_program
LD_PRELOAD=./target/debug/libcalltrace.so ldd ./your_program

# Check for required symbols
objdump -T ./target/debug/libcalltrace.so | grep cyg_profile
```

## Setup Issues

### Issue: "No such file or directory" when loading CallTrace

**Symptoms:**
```
ERROR: ld.so: object './target/debug/libcalltrace.so' from LD_PRELOAD cannot be preloaded
```

**Solutions:**
1. **Check library path:**
   ```bash
   # CallTrace uses .target directory, not target
   ls -la ./.target/debug/libcalltrace.so
   
   # Use correct path in LD_PRELOAD
   LD_PRELOAD=./.target/debug/libcalltrace.so ./your_program
   ```

2. **Build the library first:**
   ```bash
   cargo build                    # Debug build
   cargo build --release         # Release build
   ```

3. **Check library dependencies:**
   ```bash
   ldd ./.target/debug/libcalltrace.so
   ```

### Issue: "Permission denied" loading library

**Solution:**
```bash
# Ensure library is readable
chmod 644 ./.target/debug/libcalltrace.so

# Ensure execute permissions on directory
chmod 755 ./.target/debug/
```

### Issue: Library loads but no tracing occurs

**Diagnosis:**
```bash
# Enable debug output to see initialization
CALLTRACE_DEBUG=1 CALLTRACE_OUTPUT=debug.json LD_PRELOAD=./.target/debug/libcalltrace.so ./your_program
```

**Common causes:**
- Program not compiled with `-finstrument-functions`
- CallTrace disabled by environment variable
- Program exits before any function calls

## Compilation Problems

### Issue: "undefined reference to `__cyg_profile_func_enter`"

**Cause:** Program compiled with `-finstrument-functions` but CallTrace library not linked/preloaded.

**Solution:**
```bash
# Don't link CallTrace directly, use LD_PRELOAD instead
gcc -rdynamic -finstrument-functions -g your_program.c -o your_program
LD_PRELOAD=./.target/debug/libcalltrace.so ./your_program
```

### Issue: Missing debug information for argument capture

**Symptoms:**
```
CallTrace: DWARF analyzer initialization failed
```

**Solutions:**
1. **Add debug flag:**
   ```bash
   gcc -rdynamic -finstrument-functions -g your_program.c -o your_program
   ```

2. **Check debug info presence:**
   ```bash
   objdump -h your_program | grep debug
   readelf -S your_program | grep debug
   ```

3. **Verify DWARF format:**
   ```bash
   readelf -w your_program | head -20
   ```

### Issue: Symbols stripped from binary

**Symptoms:**
- Function names appear as addresses (e.g., "0x401020")
- `dladdr()` cannot resolve function names

**Solution:**
```bash
# Add -rdynamic flag for symbol export
gcc -rdynamic -finstrument-functions -g your_program.c -o your_program

# Verify symbols are present
nm your_program | grep main
objdump -t your_program | grep main
```

## Runtime Issues

### Issue: Program crashes with SIGSEGV

**Diagnosis:**
```bash
# Run with debug output
CALLTRACE_DEBUG=1 LD_PRELOAD=./.target/debug/libcalltrace.so ./your_program

# Use GDB for detailed analysis
gdb --args env LD_PRELOAD=./.target/debug/libcalltrace.so ./your_program
(gdb) run
(gdb) bt  # when it crashes
```

**Common causes:**
1. **Stack overflow from deep recursion:**
   ```bash
   # Increase max depth or disable deep tracing
   CALLTRACE_MAX_DEPTH=50 LD_PRELOAD=./.target/debug/libcalltrace.so ./your_program
   ```

2. **Thread safety issues:**
   - Ensure program is compiled with `-pthread` if using threads
   - Check for signal handler conflicts

3. **Memory corruption:**
   - Disable argument capture to isolate issue
   - Use Valgrind for memory error detection

### Issue: "Thread Local Storage access error"

**Cause:** CallTrace cleanup running after TLS destruction (fixed in recent versions).

**Solutions:**
1. **Update to latest CallTrace version**
2. **Use release build:**
   ```bash
   LD_PRELOAD=./.target/release/libcalltrace.so ./your_program
   ```

### Issue: Missing function calls in output

**Diagnosis:**
1. **Check call depth limit:**
   ```bash
   CALLTRACE_MAX_DEPTH=1000 LD_PRELOAD=./.target/debug/libcalltrace.so ./your_program
   ```

2. **Verify instrumentation:**
   ```bash
   objdump -d your_program | grep cyg_profile
   ```

3. **Check for inline functions:**
   - Inlined functions won't be traced
   - Compile with `-O0` to disable inlining

## Performance Problems

### Issue: Severe performance degradation

**Solutions:**
1. **Disable argument capture:**
   ```bash
   # Remove CALLTRACE_CAPTURE_ARGS or set to 0
   CALLTRACE_OUTPUT=trace.json LD_PRELOAD=./.target/debug/libcalltrace.so ./your_program
   ```

2. **Use release build:**
   ```bash
   cargo build --release
   LD_PRELOAD=./.target/release/libcalltrace.so ./your_program
   ```

3. **Limit call depth:**
   ```bash
   CALLTRACE_MAX_DEPTH=50 LD_PRELOAD=./.target/debug/libcalltrace.so ./your_program
   ```

4. **Profile the overhead:**
   ```bash
   # Compare with and without tracing
   time ./your_program
   time env LD_PRELOAD=./.target/debug/libcalltrace.so ./your_program
   ```

### Issue: Memory usage grows continuously

**Causes:**
- Very deep recursion
- Extremely high call frequency
- Memory leaks in traced program

**Solutions:**
```bash
# Monitor memory usage
valgrind --tool=massif LD_PRELOAD=./.target/debug/libcalltrace.so ./your_program

# Limit maximum depth
CALLTRACE_MAX_DEPTH=100 LD_PRELOAD=./.target/debug/libcalltrace.so ./your_program
```

## Output Issues

### Issue: No JSON file generated

**Diagnosis checklist:**
1. **Check CALLTRACE_OUTPUT is set:**
   ```bash
   echo $CALLTRACE_OUTPUT
   CALLTRACE_OUTPUT=output.json LD_PRELOAD=./.target/debug/libcalltrace.so ./your_program
   ```

2. **Verify write permissions:**
   ```bash
   touch test_output.json  # Can we write to this location?
   rm test_output.json
   ```

3. **Check program execution:**
   ```bash
   # Ensure program runs long enough for cleanup
   CALLTRACE_DEBUG=1 CALLTRACE_OUTPUT=output.json LD_PRELOAD=./.target/debug/libcalltrace.so ./your_program
   ```

### Issue: JSON file is empty or malformed

**Solutions:**
1. **Enable pretty printing:**
   ```bash
   CALLTRACE_PRETTY_JSON=1 CALLTRACE_OUTPUT=output.json LD_PRELOAD=./.target/debug/libcalltrace.so ./your_program
   ```

2. **Validate JSON syntax:**
   ```bash
   jq . output.json  # Will show syntax errors
   python -m json.tool output.json  # Alternative validator
   ```

3. **Check for partial writes:**
   - Program may have crashed before cleanup
   - Use crash handler to generate partial output

### Issue: Function names appear as addresses

**Cause:** Symbol resolution failed.

**Solutions:**
1. **Add -rdynamic flag:**
   ```bash
   gcc -rdynamic -finstrument-functions -g your_program.c -o your_program
   ```

2. **Check symbol table:**
   ```bash
   nm your_program | grep -v " U "  # Show defined symbols
   ```

3. **Test dladdr functionality:**
   ```bash
   # CallTrace will log symbol resolution status with debug enabled
   CALLTRACE_DEBUG=1 CALLTRACE_OUTPUT=debug.json LD_PRELOAD=./.target/debug/libcalltrace.so ./your_program
   ```

## Advanced Debugging

### Enable Comprehensive Debug Output

```bash
export CALLTRACE_DEBUG=1
export CALLTRACE_OUTPUT=debug_trace.json
export RUST_BACKTRACE=1

LD_PRELOAD=./.target/debug/libcalltrace.so ./your_program 2>debug.log
```

### Use GDB for Deep Analysis

```bash
gdb --args env LD_PRELOAD=./.target/debug/libcalltrace.so ./your_program

# Set breakpoints on CallTrace functions
(gdb) break __cyg_profile_func_enter
(gdb) break __cyg_profile_func_exit
(gdb) run
(gdb) bt  # Show call stack when breakpoint hits
```

### Analyze CallTrace Internals

```bash
# Check CallTrace library symbols
nm ./.target/debug/libcalltrace.so | grep cyg_profile

# Verify library initialization
strace -e trace=openat,write LD_PRELOAD=./.target/debug/libcalltrace.so ./your_program

# Monitor memory allocation
valgrind --tool=memcheck --leak-check=full LD_PRELOAD=./.target/debug/libcalltrace.so ./your_program
```

## Frequently Asked Questions

### Q: Can I use CallTrace with C++ programs?

**A:** Yes, but with limitations:
- Compile with `-finstrument-functions` flag
- Use `-rdynamic` for symbol export
- C++ name mangling affects function names (CallTrace attempts demangling)
- Template functions may not trace correctly

```bash
g++ -rdynamic -finstrument-functions -g your_program.cpp -o your_program
CALLTRACE_OUTPUT=cpp_trace.json LD_PRELOAD=./.target/debug/libcalltrace.so ./your_program
```

### Q: Does CallTrace work with optimized code?

**A:** Partially:
- `-O1`: Most functions traced, some may be inlined
- `-O2`/`-O3`: Many functions inlined, limited tracing
- For complete tracing, use `-O0` or `-fno-inline`

### Q: Can I trace system library calls?

**A:** No, CallTrace only traces functions compiled with `-finstrument-functions`. System libraries (libc, etc.) are not instrumented.

### Q: How much overhead does CallTrace add?

**A:**
- **Basic tracing**: ~50-100ns per function call
- **Argument capture**: ~500ns-2μs per function call  
- **Memory**: ~2MB per 10,000 function calls
- **Release builds** have lower overhead than debug builds

### Q: Can I trace recursive functions?

**A:** Yes, but:
- Set appropriate `CALLTRACE_MAX_DEPTH` to prevent stack overflow
- Deep recursion increases memory usage significantly
- Consider iterative alternatives for very deep recursion

### Q: Is CallTrace thread-safe?

**A:** Yes, CallTrace is fully thread-safe:
- Each thread gets its own call tree
- Concurrent access is protected by atomic operations and locks
- No data races or corruption in multi-threaded environments

### Q: Can I disable tracing for specific functions?

**A:** Currently no, but you can:
- Use `__attribute__((no_instrument_function))` in GCC to exclude functions
- Limit tracing depth with `CALLTRACE_MAX_DEPTH`
- Filter output during analysis

```c
// This function won't be traced
__attribute__((no_instrument_function))
void excluded_function() {
    // This won't appear in CallTrace output
}
```

### Q: How do I analyze the JSON output?

**A:** Several options:
1. **Command-line with jq:**
   ```bash
   jq '.metadata' trace.json  # View metadata
   jq '.call_trees | keys' trace.json  # List thread IDs
   ```

2. **Web browser:** Open JSON file directly for formatted view

3. **Custom tools:** Parse JSON programmatically in your preferred language

4. **Visualization:** Convert to formats like FlameGraph or Chrome Tracing

### Q: What platforms does CallTrace support?

**A:** Currently:
- **Architecture:** x86_64 (Intel/AMD 64-bit)
- **Operating System:** Linux with glibc
- **Compiler:** GCC with `-finstrument-functions` support

Future versions may support ARM64 and macOS.

### Q: Can I use CallTrace in production?

**A:** Yes, with considerations:
- Use release builds for lower overhead
- Disable argument capture for performance
- Monitor memory usage with high call volumes
- Test thoroughly in your specific environment

### Q: How do I report bugs or request features?

**A:**
1. Check this troubleshooting guide first
2. Search existing issues on the project repository
3. Provide minimal reproduction case
4. Include debug output and system information
5. Specify CallTrace version and compilation flags used