# eBPF Troubleshooting Guide
This guide provides comprehensive troubleshooting steps for diagnosing and fixing eBPF-related issues in denet.
## Using Debug Mode
Most tools in this guide support a `--debug` flag that enables verbose output:
```bash
# Enable debug mode for detailed eBPF diagnostic information
denet --enable-ebpf --debug run -- your_command
# Run setup script with verbose output
sudo ./setup/setup_tracefs_permissions.sh --debug
# Run diagnostic tool with verbose output
cargo run --bin ebpf_diag --features ebpf -- --debug
```
## Quick Diagnosis Tool
We've included a diagnostic tool specifically for eBPF issues:
```bash
# Build and run the diagnostic tool
cargo run --bin ebpf_diag --features ebpf
# For even more verbose debugging output
cargo run --bin ebpf_diag --features ebpf -- --debug
```
This will perform a comprehensive check of your system's eBPF capabilities and provide specific recommendations to fix any issues.
## Common Issues and Solutions
### 1. Permission Denied Errors
**Symptoms:**
- "Permission denied" error messages
- eBPF features don't work even with sudo
- Error logs mention "operation not permitted"
**Solutions:**
#### Check if running as root
```bash
# Run with sudo
sudo denet --enable-ebpf run -- your_command
```
#### Verify required capabilities
```bash
# Check if the binary has the capabilities
getcap $(which denet)
# Expected: cap_dac_read_search,cap_perfmon,cap_bpf=ep
# Add all three capabilities if missing
sudo setcap cap_bpf,cap_perfmon,cap_dac_read_search=ep $(which denet)
```
Tracepoint attach needs all three:
- `cap_bpf` — load the program, create maps
- `cap_perfmon` — `perf_event_open` for tracepoints
- `cap_dac_read_search` — read `/sys/kernel/tracing/events/*/id` (root-owned 0400)
#### Check tracefs permissions
```bash
# See if you can access tracefs
ls -la /sys/kernel/debug/tracing/events/syscalls
# Fix permissions using the setup script
sudo ./setup/setup_tracefs_permissions.sh
```
### 2. Missing Maps or Programs
**Symptoms:**
- "Map not found" or "Program not found" errors
- eBPF loads but doesn't track any syscalls
**Solutions:**
#### Check if the eBPF bytecode is properly built
```bash
# Clean and rebuild with eBPF feature
cargo clean
cargo build --release --features ebpf
```
#### Verify bytecode content
The diagnostic output will show information about the embedded bytecode. Make sure it contains:
- Proper ELF header
- Maps named "syscall_counts" and "pid_syscall_map"
- Tracepoint programs for syscalls
### 3. Kernel Support Issues
**Symptoms:**
- "Feature not supported" errors
- eBPF loads but fails to attach to tracepoints
**Solutions:**
#### Check kernel version
```bash
uname -r
```
You need kernel 4.18 or newer for full eBPF support.
#### Verify BPF is enabled in the kernel
```bash
grep CONFIG_BPF /boot/config-$(uname -r)
```
Should show `CONFIG_BPF=y`
#### Check JIT compilation
```bash
cat /proc/sys/net/core/bpf_jit_enable
```
Should be `1`
#### Check unprivileged BPF setting
```bash
cat /proc/sys/kernel/unprivileged_bpf_disabled
```
Values `1` and `2` both mean BPF requires `CAP_BPF` or root — they do **not** block a process that already has `CAP_BPF`. Value `2` is the same restriction as `1` but locked at runtime (requires reboot to change). If `setcap` is correctly applied, this setting is not the problem.
#### Check perf_event_paranoid (Ubuntu / hardened kernels)
```bash
cat /proc/sys/kernel/perf_event_paranoid
```
Standard Linux values are `0`–`2`. Ubuntu and some hardened kernels add levels `3` and `4` which **block `perf_event_open` for all non-root users, including those with `CAP_PERFMON`**. Tracepoint attachment fails silently with `EPERM` when this is set to `3` or `4`.
Fix:
```bash
# Temporary (until reboot)
sudo sysctl -w kernel.perf_event_paranoid=1
# Permanent
```
Level `1` allows processes with `CAP_PERFMON` to use `perf_event_open` while blocking fully unprivileged users.
### 4. Tracepoint Attachment Issues
**Symptoms:**
- "Failed to attach tracepoint" errors
- No syscalls being tracked
**Solutions:**
#### Check if tracefs is properly mounted
```bash
#### Verify specific tracepoints exist
```bash
ls -la /sys/kernel/debug/tracing/events/syscalls/sys_enter_read
```
#### Try with a simple syscall first
```bash
# Test just the read syscall
cargo run --bin ebpf_test --features ebpf
```
## Advanced Debugging
For detailed logging during eBPF initialization:
```bash
# Run with verbose logging
The enhanced debug output will show:
- Bytecode loading details
- Specific error messages and analysis
- Kernel configuration information
- Permissions and capability checks
- Map and program verification
## Recovering from Previous eBPF Sessions
If you've had a successful eBPF session that later stopped working:
1. Check for kernel updates that may have changed eBPF behavior
```bash
dpkg -l | grep linux-image
```
2. Verify if any system configuration changed
```bash
mount | grep debugfs
ls -la /sys/kernel/debug/tracing
```
3. Run the setup script again
```bash
sudo ./setup/setup_tracefs_permissions.sh
sudo ./setup/setup_tracefs_permissions.sh --debug
```
4. Reboot your system to reset any stuck eBPF state
```bash
sudo reboot
```
## Regression Debugging
If eBPF was working previously but stopped working:
1. Check the diagnostic tool output first
```bash
cargo run --bin ebpf_diag --features ebpf
```
2. Look for kernel log messages related to BPF
```bash
dmesg | grep -i bpf
```
3. Verify if the binary still has the correct capabilities
```bash
getcap $(which denet)
```
4. Test with the simplest eBPF program
```bash
cargo run --bin ebpf_test --features ebpf
```
5. Check if another process might be using conflicting BPF resources
```bash
sudo bpftool prog list
```
## Detailed Error Analysis
The enhanced debugging now automatically analyzes error types:
1. **Permission denied**: Indicates capability or privilege issues
2. **Not found**: Typically means tracefs or debugfs is not properly mounted
3. **Invalid argument**: Often caused by kernel version incompatibility
4. **Resource busy**: Another process may be using the same BPF resources
## Getting Help
If you've tried all these steps and still have issues:
1. Generate a complete diagnostic report:
```bash
cargo run --bin ebpf_diag --features ebpf --debug > ebpf_report.txt
```
2. Share the report along with:
- Your kernel version (`uname -r`)
- OS distribution (`lsb_release -a`)
- How you built denet (`cargo --version`)
- Steps to reproduce the issue