file-identify
File type identification for Rust: a faithful port of Python's
identify, the library
pre-commit uses to decide which files a hook runs on
(types, types_or, exclude_types).
Given a path, it returns a set of tags such as file, non-executable, python, text,
derived from the file type, its permissions, its name or extension, its shebang and
its first kilobyte.
- Same answers as identify. The lookup tables are generated from identify
2.6.20 (
IDENTIFY_VERSION);ALL_TAGSis identify's exact tag set (323 tags). Shebangs are split with a port of Python'sshlex, including/usr/bin/env -Sandnix-shellshebangs. Thousands of parity vectors produced by running identify itself are checked in CI on Linux, macOS and Windows. - Cheap.
Tagis 2 bytes andTagSetis a 48-byteCopybitset; lookups never allocate. Identifying a path costs onelstatand at most one bounded read; with a known git index mode, known names cost no system call at all. - Safe on any input.
#![forbid(unsafe_code)], no panics, FIFOs and devices are never opened, shebang reads are bounded.
Quick start
[]
= "0.5"
use ;
// Name only, no I/O.
assert_eq!;
assert_eq!;
// Full identification: one lstat, at most one small read.
let tags = tags_from_path.expect;
assert!;
assert!;
// pre-commit's filter: types / types_or / exclude_types.
let types: TagSet = "file".parse.unwrap;
let types_or: TagSet = "rust, toml".parse.unwrap;
let exclude: TagSet = "binary".parse.unwrap;
assert!;
Tags and tag sets
use ;
let python = from_name.expect; // hash lookup
assert_eq!;
assert!; // what pre-commit rejects
let set = from_names.unwrap; // the error lists unknown names
assert_eq!; // Display / FromStr round-trip
assert_eq!; // |, &, -, ^, !
assert_eq!;
println!;
Iteration, Display and Debug list tags in name order. TagSet::ALL,
tags::TYPE_TAGS, tags::MODE_TAGS and tags::ENCODING_TAGS are constants.
Identification API
| Function | identify equivalent | I/O |
|---|---|---|
tags_from_path(path) |
tags_from_path |
1 lstat, ≤ 1 open + 1 KiB read |
tags_from_path_with(path, ExecPolicy) |
— | same |
tags_from_metadata(path, &fs::Metadata, ExecPolicy) |
— (reuse your own lstat) |
≤ 1 open + 1 KiB read |
tags_from_meta(path, Meta) |
— (kind and mode already known) | ≤ 1 open |
classify(Meta, name) → Done / NeedsContent(Pending) |
— | none |
Pending::finish(reader), Pending::finish_bytes(&[u8]) |
— | caller-provided |
tags_from_filename(name) |
tags_from_filename |
none |
tags_from_interpreter(name) |
tags_from_interpreter |
none |
file_is_text(path), is_text(reader), is_text_bytes(&[u8]) |
file_is_text, is_text |
≤ 1 KiB |
parse_shebang_from_file(path), parse_shebang(reader), parse_shebang_bytes(&[u8]) |
parse_shebang_from_file, parse_shebang |
first line(s), bounded |
parse_shebang_from_file_with(path, ExecPolicy) |
— | same |
shlex_split(&str) |
shlex.split |
none |
extensions::{EXTENSIONS, EXTENSIONS_NEED_BINARY_CHECK, NAMES} and
interpreters::INTERPRETERS expose the raw tables (TagTable: sorted iteration,
allocation-free get).
Git index fast path
When you already know a file's mode from git ls-files -s / git diff --raw, skip the
lstat (on Unix with core.fileMode=true and core.symlinks=true; see
Differences from identify for symlinks on Windows):
use ;
let meta = from_git_mode.expect;
// Known extension: no system call at all.
let tags = tags_from_meta.unwrap;
assert_eq!;
When you have already called fs::symlink_metadata (for example to check that the
file exists, as pre-commit does with os.path.lexists), pass the result to
tags_from_metadata instead of calling tags_from_path: same tags, no second lstat,
and a file smaller than 1 KiB is read with a single read.
classify is the fully I/O-free core: it returns the final tags, or a Pending that
tells you whether the content is needed for the shebang and/or text/binary, and
finishes from a reader or a byte slice (virtual file systems, archives, tests).
Executable files
ExecPolicy::PythonCompat (the default) reproduces os.access(path, os.X_OK):
on Unix it evaluates access(2) rules from the mode, owner and group against the
process's real uid/gid/groups (ACLs and noexec mounts are not consulted); on
Windows every file is executable, exactly as identify reports it (so unknown-name
files are checked for a shebang there too). ExecPolicy::ModeBits uses git's notion:
the owner x bit only (mode & 0o100), exactly what git add records as 100755, so
it agrees with Meta::from_git_mode on the staged mode (a 0o655 file is not
executable for either).
Differences from identify
Deliberate, to stay safe on hostile input:
- FIFOs and devices are never opened (identify blocks forever on a FIFO). They are
tagged
file, the mode tag and their name tags, plusbinarywhen the name does not decide the encoding. - The shebang parser reads at most
MAX_SHEBANG_BYTES(64 KiB); a longer first line counts as "no shebang" (identify reads the whole line, e.g. 50 MB of minified code). os.accessis evaluated from metadata, as described above.
Platform limitations:
- On Windows a directory junction (or another name-surrogate reparse point) is tagged
symlink: Rust's standard library reports it as a symlink and does not expose the reparse tag. identify (os.lstat) reportsdirectoryfor a junction. True symlinks agree.
The git index fast path describes the index, not the work tree, so it agrees with identify only where the two agree:
120000givessymlink, but withcore.symlinks=false(usual on Windows) git checks the link out as a regular file containing the target path, which identify tagsfile, …,text(andexecutableon Windows). On Windows, or whenevercore.symlinks=false, do not trust120000:lstatthose entries (tags_from_path/tags_from_metadata).100755/100644reflect the work tree only withcore.fileMode=true.
Keeping up with identify
src/generated.rs is produced by an xtask that parses identify's extensions.py and
interpreters.py as data (a small Python-literal evaluator; no Python code is
imported or executed):
src="" # wheel from PyPI, sha256-checked
The parity vectors in tests/data/identify-vectors.json are regenerated by running
identify itself on a trusted copy (a source checkout, or the wheel unpacked by
fetch-identify.sh):
python3 -I xtask/parity/gen_vectors.py --identify "$(dirname "$src")" --out tests/data/identify-vectors.json.
A weekly workflow (.github/workflows/identify-sync.yml) does both against the latest
identify on PyPI and, when the tables changed, runs the tests and opens (or updates) a
pull request on the identify-sync branch for review.
Fuzzing
fuzz/ holds cargo-fuzz targets: shebang
(parse_shebang / parse_shebang_bytes, also split into arbitrary reads), filename
(tags_from_filename, tags_from_interpreter), shlex and classify (classify +
Pending, bytes vs. reader, encoding tag invariants, is_text).
CI builds them with stable rustc (the coverage flags cargo fuzz build --sanitizer none
uses) and runs each target for 30 seconds.
CLI
# ["file", "non-executable", "python", "text"]
# ["cargo", "text", "toml"]
Output matches Python's identify-cli (json.dumps(sorted(tags))); exit status 1 when
the path does not exist (message on stderr) or no tag was found.
Performance
Median per call, cargo bench --bench identify style harness, Intel Core Ultra 7 155H,
Linux, warm page cache; 0.4.0 vs 0.5.0 in the same process:
| operation | 0.4.0 | 0.5.0 | speed-up |
|---|---|---|---|
tags_from_filename, 10 mixed names (per name) |
299 ns | 43 ns | 6.9x |
tags_from_interpreter("/usr/bin/python3.12.1") |
165 ns | 36 ns | 4.6x |
| types / types_or / exclude_types filter | 61 ns | 8 ns | 7.6x |
tags_from_path, known extension |
1427 ns | 649 ns | 2.2x |
tags_from_path, unknown extension |
2503 ns | 2060 ns | 1.2x |
tags_from_path, executable script without extension |
5161 ns | 2304 ns | 2.2x |
known extension with a git index mode (tags_from_meta) |
1422 ns | 44 ns | 32x |
System calls per file (strace): an executable script now costs statx, openat, read, close (0.4.0: 2× statx, 2× openat, 2× read, 2× close). A 50 MB executable
without a newline after #! took 6400 reads and 53 MB of memory in 0.4.0; 0.5.0 stops
after 64 KiB (16 system calls, no extra memory).
Migrating from 0.4
| 0.4 | 0.5 |
|---|---|
TagSet = HashSet<&'static str> |
TagSet bitset (Copy); contains("python") and contains(Tag::TEXT) both work; iterate with iter() (yields Tag) or names() (yields &'static str) |
tags::FILE etc. (&str) |
still available; typed constants are Tag::FILE, Tag::TEXT, … |
tags::ALL_TAGS: LazyLock<HashSet<&str>> |
ALL_TAGS: [&str; 323] (sorted) and TagSet::ALL; check names with Tag::from_name |
tags::TYPE_TAGS / MODE_TAGS / ENCODING_TAGS (LazyLock) |
same names, now const TagSet |
tags::tags_from_array |
TagSet::from_names / iter().collect() |
extensions::EXTENSION_TAGS, EXTENSIONS_NEED_BINARY_CHECK_TAGS, NAME_TAGS, interpreters::INTERPRETER_TAGS (phf::Map) |
identify's names: EXTENSIONS, EXTENSIONS_NEED_BINARY_CHECK, NAMES, INTERPRETERS (TagTable); the get_*_tags helpers remain |
tags_from_filename(&str) |
tags_from_filename(impl AsRef<[u8]>) — strips directories like identify; pass path.as_os_str().as_encoded_bytes() for paths |
parse_shebang(R) -> Result<ShebangTuple> |
parse_shebang(R) -> io::Result<Vec<String>> (first(), len(), indexing as on any Vec) |
parse_shebang_from_file(P) -> Result<ShebangTuple> |
-> Result<Vec<String>> |
is_text(R) -> Result<bool> |
-> io::Result<bool> (IdentifyError: From<io::Error> keeps ? working) |
FileIdentifier builder |
removed: use classify + Pending (skip content by not finishing), or tags_from_path_with |
FileInfo / tags_from_info |
kept (FileKind::Other added); classify + Pending::finish_bytes is the general form |
FileKind { Regular, Directory, Symlink, Socket } |
#[non_exhaustive], adds Other (FIFOs, devices) |
IdentifyError::{PathNotFound{path: String}, IoError, InvalidPath, InvalidUtf8} |
#[non_exhaustive]; PathNotFound { path: PathBuf, source }, IoError { source }; dead variants removed |
Windows: .exe/.bat/.cmd are executable |
every file is executable (identify parity), or ExecPolicy::ModeBits |
Behaviour fixes that change results: tables now equal identify 2.6.20 (31 missing tags
added, 18 invented names such as yarn.lock → yaml and .eslintrc → json removed);
quoted shebangs, env -S and nix-shell shebangs; directory prefixes in
tags_from_filename; ..foo has no extension; vertical tab / form feed allowed in
shebangs; is_text reads a full KiB even across short reads.
Minimum supported Rust version
Rust 1.85 (edition 2024), tested in CI with cargo +1.85.0 test.
License
MIT