# IDB Utils
A command-line toolkit for inspecting, validating, and manipulating InnoDB database files. Written in Rust for performance and reliability, the `idb` binary operates directly on `.ibd` tablespace files, redo logs, and system tablespaces — no running MySQL instance required.
## What It Does
IDB Utils gives you low-level visibility into InnoDB's on-disk structures. Use it to:
- **Inspect tablespace files** — parse page headers, examine B+Tree index structures, walk undo logs, inspect BLOB/LOB chains
- **Validate data integrity** — verify CRC-32C and legacy InnoDB checksums, detect LSN mismatches between headers and trailers
- **Extract metadata** — read MySQL 8.0+ SDI (Serialized Dictionary Information) directly from `.ibd` files without a running server
- **Analyze redo logs** — parse log file headers, checkpoints, and data blocks from both legacy and MySQL 8.0.30+ formats
- **Test recovery scenarios** — intentionally corrupt specific pages or byte ranges to exercise InnoDB recovery mechanisms
- **Audit a data directory** — search for pages across files, list tablespace IDs, compare LSNs between ibdata1 and redo logs
## Installation
### From Source
Requires Rust 1.70+.
```bash
git clone https://github.com/ringo380/idb-utils.git
cd idb-utils
cargo build --release
```
The binary will be at `target/release/idb`. Copy it to a directory in your `$PATH`:
```bash
cp target/release/idb /usr/local/bin/
```
### With MySQL Query Support
To enable live MySQL instance queries via `idb info`:
```bash
cargo build --release --features mysql
```
This adds the `mysql_async` and `tokio` dependencies for connecting to a running MySQL server.
### Verify Installation
```bash
idb --help
idb --version
```
## Quick Start
```bash
# Parse a tablespace and show a page type summary
idb parse -f /var/lib/mysql/mydb/users.ibd
# Validate every page checksum in a tablespace
idb checksum -f users.ibd
# Get detailed INDEX page structure (record counts, B+Tree levels)
idb pages -f users.ibd -t INDEX
# Hex dump page 3
idb dump -f users.ibd -p 3
# Extract table/column definitions from MySQL 8.0+ SDI
idb sdi -f users.ibd --pretty
# Analyze redo log checkpoints
idb log -f /var/lib/mysql/ib_logfile0
# Search an entire data directory for page number 42
idb find -d /var/lib/mysql -p 42
# Check if ibdata1 and redo log LSNs are in sync
idb info --lsn-check -d /var/lib/mysql
```
Every subcommand supports `--json` for machine-readable output and `--help` for full option details.
## Subcommands
| [`parse`](#idb-parse) | Parse .ibd file and display page headers with type summary |
| [`pages`](#idb-pages) | Detailed page structure analysis (INDEX, UNDO, LOB, SDI) |
| [`dump`](#idb-dump) | Hex dump of raw page bytes |
| [`checksum`](#idb-checksum) | Validate page checksums (CRC-32C and legacy InnoDB) |
| [`corrupt`](#idb-corrupt) | Intentionally corrupt pages for recovery testing |
| [`find`](#idb-find) | Search for pages across a MySQL data directory |
| [`tsid`](#idb-tsid) | List or find tablespace IDs |
| [`sdi`](#idb-sdi) | Extract SDI metadata from MySQL 8.0+ tablespaces |
| [`log`](#idb-log) | Analyze InnoDB redo log files |
| [`info`](#idb-info) | Inspect ibdata1 headers, compare LSNs, query MySQL |
---
## Subcommand Reference
### idb parse
Parse an `.ibd` file and display page-level information. Shows FIL headers, FSP metadata, checksums, and a page type summary across the entire tablespace.
```
idb parse -f <file> [-p <page>] [-v] [-e] [--json] [--page-size <size>]
```
| `-f, --file` | Path to .ibd file (required) |
| `-p, --page` | Show a single page instead of all pages |
| `-v, --verbose` | Include checksum validation, record counts, LSN details |
| `-e, --no-empty` | Skip pages with zero checksum and Allocated type |
| `--json` | Output as JSON array |
| `--page-size` | Override page size in bytes (default: auto-detect from FSP flags) |
**All-pages mode** (default) iterates every page and prints a summary table showing how many pages of each type exist in the tablespace. **Single-page mode** (`-p N`) displays the full FIL header for that page including checksum, page type, LSN, prev/next page chain, and space ID.
```bash
# Overview of a tablespace
idb parse -f users.ibd
# Detailed look at the FSP header page
idb parse -f users.ibd -p 0 -v
# Machine-readable output for scripting
---
### idb pages
Detailed structural analysis of page internals. Goes deeper than `parse` by decoding type-specific headers: INDEX page B+Tree metadata, UNDO segment state, LOB/BLOB chain headers, and SDI content.
```
idb pages -f <file> [-p <page>] [-v] [-e] [-l] [-t <type>] [--json] [--page-size <size>]
```
| `-f, --file` | Path to .ibd file (required) |
| `-p, --page` | Analyze a single page |
| `-v, --verbose` | Show additional structural details |
| `-e, --show-empty` | Include empty/Allocated pages (hidden by default) |
| `-l, --list` | Compact one-line-per-page listing |
| `-t, --type` | Filter by page type name |
| `--json` | Output as JSON |
| `--page-size` | Override page size |
**Type filter** accepts exact type names and aliases:
| `INDEX` | B+Tree index pages |
| `UNDO` | Undo log pages |
| `BLOB` | Old-style BLOB, ZBlob, ZBlob2 |
| `LOB` | MySQL 8.0+ LobIndex, LobData, LobFirst |
| `SDI` | SDI and SDI BLOB pages |
| `COMPRESSED` or `COMP` | Compressed and CompressedEncrypted |
| `ENCRYPTED` or `ENC` | Encrypted, CompressedEncrypted, EncryptedRtree |
**What it shows per page type:**
- **INDEX pages** — index header (n_recs, level, index_id, heap_top, direction), FSEG headers (leaf/non-leaf segment pointers), system records (infimum/supremum next pointers), compact vs redundant format flag
- **UNDO pages** — undo page header (type: INSERT or UPDATE, log start/free offsets), undo segment header (state: Active/Cached/ToFree/ToPurge/Prepared)
- **BLOB pages** — part_len (bytes stored on this page), next_page_no in chain
- **LOB_FIRST pages** — version, flags, total data_len, trx_id
- **FSP_HDR pages** — space_id, tablespace size, free limit, compression algorithm, encryption flag, first unused segment ID
```bash
# List all INDEX pages with their B+Tree levels
idb pages -f users.ibd -t INDEX -l
# Deep inspection of a specific undo page
idb pages -f undo_001.ibu -p 5 -v
# Find all BLOB pages in a tablespace
idb pages -f users.ibd -t BLOB -l
```
---
### idb dump
Hex dump of raw page bytes. Output follows the standard `offset | hex bytes | ASCII` format with 16 bytes per line.
```
idb dump -f <file> [-p <page>] [--offset <byte>] [-l <length>] [--raw] [--page-size <size>]
```
| `-f, --file` | Path to file (required) |
| `-p, --page` | Page number to dump (default: 0) |
| `--offset` | Absolute byte offset into the file (bypasses page mode) |
| `-l, --length` | Number of bytes to dump (default: full page or 256 in offset mode) |
| `--raw` | Output raw binary bytes to stdout (no hex formatting) |
| `--page-size` | Override page size |
```bash
# Dump the FIL header of page 0 (first 38 bytes)
idb dump -f users.ibd -p 0 -l 38
# Dump raw bytes at a specific file offset
idb dump -f users.ibd --offset 16384 -l 64
# Extract a raw page to a file
idb dump -f users.ibd -p 3 --raw > page3.bin
```
---
### idb checksum
Validate page checksums across an entire tablespace. Supports both CRC-32C (MySQL 5.7.7+ default) and legacy InnoDB checksum algorithms. Also validates LSN consistency between the FIL header and trailer of each page.
```
idb checksum -f <file> [-v] [--json] [--page-size <size>]
```
| `-f, --file` | Path to .ibd file (required) |
| `-v, --verbose` | Show per-page checksum status |
| `--json` | Output as JSON with summary statistics |
| `--page-size` | Override page size |
**Validation logic:**
1. Pages with the magic checksum `0xDEADBEEF` are skipped (no checksum set)
2. All-zero pages are treated as valid (freshly allocated)
3. CRC-32C is tried first (covers bytes 4-25 and 38 to page_size-8)
4. Falls back to legacy InnoDB algorithm (`ut_fold_binary`) if CRC-32C doesn't match
5. LSN low 32 bits from the FIL header (offset 16) are compared against the trailer (last 4 bytes)
**Exit code:** Returns `1` if any invalid checksums are found.
```bash
# Quick validation
idb checksum -f users.ibd
# Detailed per-page report
idb checksum -f users.ibd -v
# JSON output for monitoring
idb checksum -f users.ibd --json
```
**JSON output structure:**
```json
{
"file": "users.ibd",
"page_size": 16384,
"total_pages": 100,
"empty_pages": 10,
"valid_pages": 88,
"invalid_pages": 2,
"lsn_mismatches": 0,
"pages": [...]
}
```
---
### idb corrupt
Intentionally corrupt page bytes for testing InnoDB crash recovery, backup validation, or checksum detection. Writes random bytes to specified locations in the file.
**Warning:** This command modifies files. Always work on copies, not production data.
```
idb corrupt -f <file> [-p <page>] [-b <bytes>] [-k] [-r] [--offset <byte>] [--json] [--page-size <size>]
```
| `-f, --file` | Path to file to corrupt (required) |
| `-p, --page` | Page number to corrupt (random page if omitted) |
| `-b, --bytes` | Number of bytes to overwrite with random data (default: 1) |
| `-k, --header` | Target the FIL header area (first 38 bytes of the page) |
| `-r, --records` | Target the user data area (bytes 120 through page_size-8) |
| `--offset` | Absolute byte offset (bypasses page calculation entirely) |
| `--json` | Output corruption details as JSON |
| `--page-size` | Override page size |
Works on any InnoDB file type: `.ibd` tablespaces, `ibdata1`, undo tablespaces, redo logs.
```bash
# Corrupt 4 bytes in the record area of page 5
idb corrupt -f users_copy.ibd -p 5 -b 4 -r
# Corrupt the FIL header of a random page
idb corrupt -f users_copy.ibd -k
# Corrupt at a specific file offset
idb corrupt -f ibdata1_copy -b 8 --offset 65536
# Get corruption details as JSON for test automation
idb corrupt -f users_copy.ibd -p 3 -b 2 --json
```
---
### idb find
Recursively search a MySQL data directory for `.ibd` files containing a specific page number. Useful for locating which tablespace owns a particular page when debugging InnoDB errors that reference page numbers.
```
idb find -d <datadir> -p <page> [-c <checksum>] [-s <space_id>] [--first] [--json]
```
| `-d, --datadir` | MySQL data directory path (required) |
| `-p, --page` | Page number to search for (required) |
| `-c, --checksum` | Also match this checksum value |
| `-s, --space-id` | Also match this space ID |
| `--first` | Stop after the first match |
| `--json` | Output as JSON |
```bash
# Find which tablespace contains page 42
idb find -d /var/lib/mysql -p 42
# Find page 0 with a specific space ID
idb find -d /var/lib/mysql -p 0 -s 15
# Stop at first match for speed
idb find -d /var/lib/mysql -p 100 --first
```
---
### idb tsid
List tablespace IDs from all `.ibd` and `.ibu` files in a data directory. Reads the space ID from the FSP header on page 0 of each file.
```
idb tsid -d <datadir> [-l] [-t <tsid>] [--json]
```
| `-d, --datadir` | MySQL data directory path (required) |
| `-l, --list` | List all tablespace IDs found |
| `-t, --tsid` | Find the file with this specific tablespace ID |
| `--json` | Output as JSON |
```bash
# List all tablespace IDs
idb tsid -d /var/lib/mysql -l
# Find which file has tablespace ID 42
idb tsid -d /var/lib/mysql -t 42
```
---
### idb sdi
Extract SDI (Serialized Dictionary Information) metadata from MySQL 8.0+ tablespaces. SDI records contain the full table and tablespace definitions (column types, indexes, partitions, etc.) stored as compressed JSON directly in the `.ibd` file.
```
idb sdi -f <file> [--pretty] [--page-size <size>]
```
| `-f, --file` | Path to .ibd file (required) |
| `--pretty` | Pretty-print the JSON output |
| `--page-size` | Override page size |
SDI records are zlib-compressed and may span multiple linked pages. The tool handles multi-page reassembly automatically by following the page chain.
Each record has a type (1 = Table definition, 2 = Tablespace definition) and contains the full JSON representation of the object as MySQL's data dictionary sees it.
```bash
# Extract and pretty-print SDI
idb sdi -f users.ibd --pretty
# Pipe to jq for specific fields
---
### idb log
Analyze InnoDB redo log files. Supports both legacy format (`ib_logfile0`, `ib_logfile1`) and the MySQL 8.0.30+ directory format (`#innodb_redo/#ib_redo*`).
Redo logs are structured as a sequence of 512-byte blocks. The first 4 blocks are reserved: block 0 is the file header, blocks 1 and 3 are checkpoints, and data blocks start at block 4.
```
idb log -f <file> [-b <blocks>] [--no-empty] [-v] [--json]
```
| `-f, --file` | Path to redo log file (required) |
| `-b, --blocks` | Limit output to the first N data blocks |
| `--no-empty` | Skip blocks with no data |
| `-v, --verbose` | Show MLOG record type breakdown per block |
| `--json` | Output as JSON |
**Verbose mode** scans each data block for MLOG record type bytes and reports counts. This gives a rough breakdown of the types of operations recorded in each block (inserts, updates, page creates, file operations, etc.).
```bash
# Show log file header and checkpoints
idb log -f /var/lib/mysql/ib_logfile0
# Analyze first 100 data blocks with record type breakdown
idb log -f /var/lib/mysql/ib_logfile0 -b 100 -v
# Skip empty blocks
idb log -f /var/lib/mysql/ib_logfile0 --no-empty
```
---
### idb info
Inspect InnoDB system files and optionally query a live MySQL instance. Has three modes of operation.
```
idb info [--ibdata] [--lsn-check] [-d <datadir>]
[-D <database> -t <table>] [--host <host>] [--port <port>]
[--user <user>] [--password <pass>] [--defaults-file <path>]
[--json]
```
**Mode 1: ibdata1 inspection (`--ibdata`)**
Reads page 0 of `ibdata1` and displays the FIL header fields (checksum, page type, LSN, flush LSN, space ID). Also reads the redo log checkpoint LSN for reference.
```bash
idb info --ibdata -d /var/lib/mysql
```
**Mode 2: LSN consistency check (`--lsn-check`)**
Compares the LSN from `ibdata1` page 0 with the redo log checkpoint LSN. Reports whether they are `IN SYNC` or `OUT OF SYNC`. Returns exit code `1` if out of sync.
Tries MySQL 8.0.30+ format (`#innodb_redo/`) first, falls back to legacy `ib_logfile0`.
```bash
idb info --lsn-check -d /var/lib/mysql
```
**Mode 3: MySQL query mode (`-D`, `-t`)**
Requires the `mysql` feature. Connects to a running MySQL instance and queries `information_schema` for table metadata: space_id, table_id, indexes (name, index_id, root_page), and current LSN from `SHOW ENGINE INNODB STATUS`.
```bash
# Using credentials from ~/.my.cnf
idb info -D mydb -t users
# Explicit connection parameters
idb info -D mydb -t users --host 127.0.0.1 --user root --password secret
# Using a specific defaults file
idb info -D mydb -t users --defaults-file /etc/mysql/my.cnf
```
| `--ibdata` | Inspect ibdata1 page 0 header |
| `--lsn-check` | Compare ibdata1 and redo log checkpoint LSNs |
| `-d, --datadir` | MySQL data directory (default: `/var/lib/mysql`) |
| `-D, --database` | Database name (MySQL query mode) |
| `-t, --table` | Table name (MySQL query mode) |
| `--host` | MySQL host |
| `--port` | MySQL port |
| `--user` | MySQL user |
| `--password` | MySQL password |
| `--defaults-file` | Path to `.my.cnf` defaults file |
| `--json` | Output as JSON |
MySQL credentials are resolved in order: CLI flags, then `--defaults-file`, then `~/.my.cnf`, then `/etc/my.cnf`.
---
## Supported MySQL Versions
| MySQL 5.7 | `.ibd` | `ib_logfile*` | N/A | `innodb_sys_tables` |
| MySQL 8.0 | `.ibd` | `ib_logfile*` | Yes | `innodb_tables` / `innodb_indexes` |
| MySQL 8.0.30+ | `.ibd` | `#innodb_redo/` | Yes | `innodb_tables` / `innodb_indexes` |
| MySQL 8.4 / 9.x | `.ibd` | `#innodb_redo/` | Yes | `innodb_tables` / `innodb_indexes` |
**Page sizes:** 4K, 8K, 16K (default), 32K, and 64K are all supported and auto-detected from FSP flags.
---
## InnoDB File Format Primer
Understanding the on-disk format helps when interpreting `idb` output.
### Tablespace Files (.ibd)
Each `.ibd` file is a tablespace containing one or more tables. The file is divided into fixed-size **pages** (default 16,384 bytes). Every page has the same basic structure:
```
+------------------+ byte 0
| Page Body | varies by page type
| |
+------------------+ byte (page_size - 8)
| FIL Trailer | 8 bytes — old checksum, LSN low 32 bits
+------------------+
```
**Page 0** is always an `FSP_HDR` (File Space Header) page containing tablespace-level metadata: total size, space ID, flags encoding page size and compression/encryption settings, and extent descriptors.
### Page Types
InnoDB uses ~26 page types. The most common ones you'll encounter:
| `INDEX` | 17855 | B+Tree leaf and non-leaf nodes (the bulk of any tablespace) |
| `FSP_HDR` | 8 | File space header (always page 0) |
| `INODE` | 3 | File segment inode (segment metadata) |
| `XDES` | 9 | Extent descriptor (tracks extent allocation) |
| `ALLOCATED` | 0 | Freshly allocated, not yet used |
| `UNDO_LOG` | 2 | Undo log records for MVCC |
| `BLOB` | 10 | Externally stored column data |
| `SDI` | 17853 | Serialized Dictionary Information (MySQL 8.0+) |
| `TRX_SYS` | 7 | Transaction system header (ibdata1 page 5) |
### Checksums
Each page stores a checksum in the first 4 bytes. MySQL supports two algorithms:
- **CRC-32C** (default since MySQL 5.7.7) — hardware-accelerated on modern CPUs
- **Legacy InnoDB** — a custom folding hash from the original InnoDB codebase
Both are computed over the same byte ranges (excluding the checksum field itself and the FIL trailer checksum). The `idb checksum` command tries CRC-32C first and falls back to legacy.
### LSN (Log Sequence Number)
The LSN is a monotonically increasing counter tracking the position in the redo log. Each page records its last-modified LSN in both the FIL header (8 bytes at offset 16) and the FIL trailer (low 32 bits). A mismatch between these two values indicates a torn page write.
### Redo Log Structure
Redo logs are sequences of 512-byte blocks:
| 0 | File header (group_id, start_lsn, file_no, creator string) |
| 1 | Checkpoint 1 (number, lsn, offset, buf_size) |
| 2 | Reserved |
| 3 | Checkpoint 2 |
| 4+ | Data blocks (14-byte header, up to 494 bytes of log records, 4-byte checksum) |
Each data block is checksummed with CRC-32C over the first 508 bytes.
---
## JSON Output
Every subcommand supports `--json` for structured output suitable for scripting, monitoring, and integration with other tools.
```bash
# Pipe to jq for filtering
# Checksum validation in CI
exit 1
fi
# Extract column names from SDI
# Count pages by type
---
## Building and Testing
### Build
```bash
# Standard build
cargo build --release
# With MySQL query support
cargo build --release --features mysql
```
### Test
The test suite includes 55 unit tests covering InnoDB parsing logic (checksums, page types, headers, compression, encryption, SDI, redo logs, records, undo) and 14 integration tests that build synthetic `.ibd` files and run CLI commands against them.
```bash
# Run all tests
cargo test
# Run a specific test
cargo test test_crc32c_checksum_valid_pages
```
### Lint
The project enforces zero clippy warnings:
```bash
cargo clippy -- -D warnings
```
### CI
GitHub Actions runs on every push and pull request:
- `cargo test` on Ubuntu
- `cargo clippy -- -D warnings`
- `cargo build --release` on Ubuntu and macOS
- `cargo build --release --features mysql`
---
## Project Structure
```
src/
main.rs CLI entry point and subcommand dispatch (clap derive)
lib.rs IdbError type, module re-exports
cli/
parse.rs idb parse
pages.rs idb pages
dump.rs idb dump
checksum.rs idb checksum
corrupt.rs idb corrupt
find.rs idb find
tsid.rs idb tsid
sdi.rs idb sdi
log.rs idb log
info.rs idb info
innodb/
constants.rs InnoDB on-disk format constants (offsets, sizes, flags)
page.rs FIL header, FIL trailer, FSP header parsing
page_types.rs PageType enum (26 variants) with names and descriptions
tablespace.rs Tablespace file I/O abstraction (open, read_page, iterate)
checksum.rs CRC-32C and legacy InnoDB checksum validation
index.rs INDEX page header, FSEG headers, system records
record.rs Compact record header parsing, record chain walking
undo.rs Undo page/segment headers, rollback segment arrays
lob.rs BLOB page headers, LOB first page, chain walking
sdi.rs SDI page detection, record extraction, zlib decompression
log.rs Redo log headers, checkpoints, block parsing, MLOG types
compression.rs Compression detection (zlib/lz4), decompression
encryption.rs Encryption detection (AES flag in FSP flags)
util/
hex.rs Hex dump formatting, offset/value formatters
mysql.rs MySQL connection, .my.cnf parsing (feature-gated)
tests/
integration_test.rs Integration tests with synthetic .ibd files
```
Each CLI subcommand follows the same pattern: an `Options` struct with clap derive attributes and an `execute()` function returning `Result<(), IdbError>`.
All InnoDB constants use `UPPERCASE_WITH_UNDERSCORES` and match the names from MySQL/InnoDB source code (`fil0fil.h`, `page0page.h`, `fsp0fsp.h`).
## License
MIT