# Debugging
## Macros
PyO3's attributes (`#[pyclass]`, `#[pymodule]`, etc.) are [procedural macros](https://doc.rust-lang.org/reference/procedural-macros.html), which means that they rewrite the source of the annotated item. You can view the generated source with the following command, which also expands a few other things:
```bash
cargo rustc --profile=check -- -Z unstable-options --pretty=expanded > expanded.rs; rustfmt expanded.rs
```
(You might need to install [rustfmt](https://github.com/rust-lang-nursery/rustfmt) if you don't already have it.)
You can also debug classic `!`-macros by adding `-Z trace-macros`:
```bash
cargo rustc --profile=check -- -Z unstable-options --pretty=expanded -Z trace-macros > expanded.rs; rustfmt expanded.rs
```
Note that those commands require using the nightly build of rust and may occasionally have bugs. See [cargo expand](https://github.com/dtolnay/cargo-expand) for a more elaborate and stable version of those commands.
## Running with Valgrind
Valgrind is a tool to detect memory management bugs such as memory leaks.
You first need to install a debug build of Python, otherwise Valgrind won't produce usable results. In Ubuntu there's e.g. a `python3-dbg` package.
Activate an environment with the debug interpreter and recompile. If you're on Linux, use `ldd` with the name of your binary and check that you're linking e.g. `libpython3.7d.so.1.0` instead of `libpython3.7.so.1.0`.
[Download the suppressions file for CPython](https://raw.githubusercontent.com/python/cpython/master/Misc/valgrind-python.supp).
Run Valgrind with `valgrind --suppressions=valgrind-python.supp ./my-command --with-options`
## Getting a stacktrace
The best start to investigate a crash such as an segmentation fault is a backtrace. You can set `RUST_BACKTRACE=1` as an environment variable to get the stack trace on a `panic!`. Alternatively you can use a debugger such as `gdb` to explore the issue. Rust provides a wrapper, `rust-gdb`, which has pretty-printers for inspecting Rust variables. Since PyO3 uses `cdylib` for Python shared objects, it does not receive the pretty-print debug hooks in `rust-gdb` ([rust-lang/rust#96365](https://github.com/rust-lang/rust/issues/96365)). The mentioned issue contains a workaround for enabling pretty-printers in this case.
* Link against a debug build of python as described in the previous chapter
* Run `rust-gdb <my-binary>`
* Set a breakpoint (`b`) on `rust_panic` if you are investigating a `panic!`
* Enter `r` to run
* After the crash occurred, enter `bt` or `bt full` to print the stacktrace
Often it is helpful to run a small piece of Python code to exercise a section of Rust.
```console
rust-gdb --args python -c "import my_package; my_package.sum_to_string(1, 2)"
```
## Setting breakpoints in your Rust code
One of the preferred ways by developers to debug their code is by setting breakpoints. This can be achieved in PyO3 by using a debugger like `rust-gdb` or `rust-lldb` with your Python interpreter.
For more information about how to use both `lldb` and `gdb` you can read the [gdb to lldb command map](https://lldb.llvm.org/use/map.html) from the lldb documentation.
### Common setup
1. Compile your extension with debug symbols:
```bash
# Debug is the default for maturin, but you can explicitly ensure debug symbols with:
RUSTFLAGS="-g" maturin develop
# For setuptools-rust users:
pip install -e .
```
> **Note**: When using debuggers, make sure that `python` resolves to an actual Python binary or symlink and not a shim script. Some tools like pyenv use shim scripts which can interfere with debugging.
### Debugger specific setup
Depeding on your OS and your preferences you can use two different debuggers, `rust-gdb` or `rust-lldb`.
{{#tabs }}
{{#tab name="Using rust-gdb" }}
1. Launch rust-gdb with the Python interpreter:
```bash
rust-gdb --args python
```
2. Once in gdb, set a breakpoint in your Rust code:
```bash
(gdb) break your_module.rs:42
```
3. Run your Python script that imports and uses your Rust extension:
```bash
# Option 1: Run an inline Python command
(gdb) run -c "import your_module; your_module.your_function()"
# Option 2: Run a Python script
(gdb) run your_script.py
# Option 3: Run pytest tests
(gdb) run -m pytest tests/test_something.py::TestName
```
{{#endtab }}
{{#tab name="Using rust-lldb (for macOS users)" }}
1. Start rust-lldb with Python:
```bash
rust-lldb -- python
```
2. Set breakpoints in your Rust code:
```bash
(lldb) breakpoint set --file your_module.rs --line 42
```
3. Run your Python script:
```bash
# Option 1: Run an inline Python command
(lldb) run -c "import your_module; your_module.your_function()"
# Option 2: Run a Python script
(lldb) run your_script.py
# Option 3: Run pytest tests
(lldb) run -m pytest tests/test_something.py::TestName
```
{{#endtab }}
{{#endtabs }}
### Using VS Code
VS Code with the Rust and Python extensions provides an integrated debugging experience:
1. First, install the necessary VS Code extensions:
* [Rust Analyzer](https://marketplace.visualstudio.com/items?itemName=rust-lang.rust-analyzer)
* [CodeLLDB](https://marketplace.visualstudio.com/items?itemName=vadimcn.vscode-lldb)
* [Python](https://marketplace.visualstudio.com/items?itemName=ms-python.python)
2. Create a `.vscode/launch.json` file with a configuration that uses the LLDB Debug Launcher:
```json
{
"version": "0.2.0",
"configurations": [
{
"name": "Debug PyO3",
"type": "lldb",
"request": "attach",
"program": "${workspaceFolder}/.venv/bin/python",
"pid": "${command:pickProcess}",
"sourceLanguages": [
"rust"
]
},
{
"name": "Launch Python with PyO3",
"type": "lldb",
"request": "launch",
"program": "${workspaceFolder}/.venv/bin/python",
"args": ["${file}"],
"cwd": "${workspaceFolder}",
"sourceLanguages": ["rust"]
},
{
"name": "Debug PyO3 with Args",
"type": "lldb",
"request": "launch",
"program": "${workspaceFolder}/.venv/bin/python",
"args": ["path/to/your/script.py", "arg1", "arg2"],
"cwd": "${workspaceFolder}",
"sourceLanguages": ["rust"]
},
{
"name": "Debug PyO3 Tests",
"type": "lldb",
"request": "launch",
"program": "${workspaceFolder}/.venv/bin/python",
"args": ["-m", "pytest", "tests/your_test.py::test_function", "-v"],
"cwd": "${workspaceFolder}",
"sourceLanguages": ["rust"]
}
]
}
```
This configuration supports multiple debugging scenarios:
* Attaching to a running Python process
* Launching the currently open Python file
* Running a specific script with command-line arguments
* Running pytest tests
3. Set breakpoints in your Rust code by clicking in the gutter next to line numbers.
4. Start debugging:
* For attaching to a running Python process: First start the process, then select the "Debug PyO3" configuration and click Start Debugging (F5). You'll be prompted to select the Python process to attach to.
* For launching a Python script: Open your Python script, select the "Launch Python with PyO3" configuration and click Start Debugging (F5).
* For running with arguments: Select "Debug PyO3 with Args" (remember to edit the configuration with your actual script path and arguments).
* For running tests: Select "Debug PyO3 Tests" (edit the test path as needed).
5. When debugging PyO3 code:
* You can inspect Rust variables and data structures
* Use the debug console to evaluate expressions
* Step through Rust code line by line using the step controls
* Set conditional breakpoints for more complex debugging scenarios
### Advanced Debugging Configurations
For advanced debugging scenarios, you might want to add environment variables or enable specific Rust debug flags:
```json
{
"name": "Debug PyO3 with Environment",
"type": "lldb",
"request": "launch",
"program": "${workspaceFolder}/.venv/bin/python",
"args": ["${file}"],
"env": {
"RUST_BACKTRACE": "1",
"PYTHONPATH": "${workspaceFolder}"
},
"sourceLanguages": ["rust"]
}
```
### Debugging from Jupyter Notebooks
For Jupyter Notebooks run from VS Code, you can use the following helper functions to automate the launch configuration:
```python
from pathlib import Path
import os
import json
import sys
def update_launch_json(vscode_config_file_path=None):
"""Update VSCode launch.json with the correct Jupyter kernel PID.
Args:
vscode_config_file_path (str, optional): Path to the .vscode/launch.json file.
If not provided, will use the current working directory.
"""
pid = get_jupyter_kernel_pid()
if not pid:
print("Could not determine Jupyter kernel PID.")
return
# Determine launch.json path
if vscode_config_file_path:
launch_json_path = vscode_config_file_path
else:
launch_json_path = os.path.join(Path(os.getcwd()), ".vscode", "launch.json")
# Get Python interpreter path
python_path = sys.executable
# Default debugger config
debug_config = {
"version": "0.2.0",
"configurations": [
{
"name": "Debug PyO3 (Jupyter)",
"type": "lldb",
"request": "attach",
"program": python_path,
"pid": pid,
"sourceLanguages": ["rust"],
},
{
"name": "Launch Python with PyO3",
"type": "lldb",
"request": "launch",
"program": python_path,
"args": ["${file}"],
"cwd": "${workspaceFolder}",
"sourceLanguages": ["rust"]
}
],
}
# Create .vscode directory if it doesn't exist
try:
os.makedirs(os.path.dirname(launch_json_path), exist_ok=True)
# If launch.json already exists, try to update it instead of overwriting
if os.path.exists(launch_json_path):
try:
with open(launch_json_path, "r") as f:
existing_config = json.load(f)
# Check if our configuration already exists
config_exists = False
for config in existing_config.get("configurations", []):
if config.get("name") == "Debug PyO3 (Jupyter)":
config["pid"] = pid
config["program"] = python_path
config_exists = True
if not config_exists:
existing_config.setdefault("configurations", []).append(debug_config["configurations"][0])
debug_config = existing_config
except Exception:
# If reading fails, we'll just overwrite with our new configuration
pass
with open(launch_json_path, "w") as f:
json.dump(debug_config, f, indent=4)
print(f"Updated launch.json with PID: {pid} at {launch_json_path}")
except Exception as e:
print(f"Error updating launch.json: {e}")
def get_jupyter_kernel_pid():
"""Find the process ID (PID) of the running Jupyter kernel.
Returns:
int: The process ID of the Jupyter kernel, or None if not found.
"""
# Check if we're running in a Jupyter environment
if 'ipykernel' in sys.modules:
pid = os.getpid()
print(f"Jupyter kernel PID: {pid}")
return pid
else:
print("Not running in a Jupyter environment.")
return None
```
To use these functions:
1. Run the cell containing these functions in your Jupyter notebook
2. Run `update_launch_json()` in a cell
3. In VS Code, select the "Debug PyO3 (Jupyter)" configuration and start debugging