Skip to main content

Crate file_identify

Crate file_identify 

Source
Expand description

§file-identify

File identification for Rust: a faithful port of Python’s identify, the library pre-commit uses to decide which files a hook runs on (types, types_or, exclude_types).

Given a path, it returns a set of tags such as file, non-executable, python, text, derived from the file type, its permissions, its name or extension, its shebang and its first kilobyte. The lookup tables are generated from identify IDENTIFY_VERSION — ALL_TAGS is exactly identify’s tag set.

§Quick start

use file_identify::{Tag, tags_from_filename, tags_from_path};

// Name only, no I/O.
let tags = tags_from_filename("Dockerfile.xenial");
assert_eq!(tags.to_string(), "dockerfile, text");

// Full identification (one lstat, at most one small read).
let tags = tags_from_path(&path)?;
assert!(tags.contains(Tag::FILE));
assert!(tags.contains("rust"));

§Tags and tag sets

Tag is a 2-byte handle to one of the TAG_COUNT known tags; TagSet is a Copy bitset with allocation-free set operations and pre-commit’s filter in TagSet::matches_types. Parse user-supplied names with Tag::from_name / TagSet::from_names; an unknown name is exactly what pre-commit reports as Type tag '…' is not recognized.

§Identification API

Functionidentify equivalentI/O
tags_from_path, tags_from_path_withtags_from_path1 lstat, ≤ 1 open + 1 KiB read
tags_from_metadata— (lstat metadata already at hand)≤ 1 open + 1 KiB read
tags_from_meta— (known Meta, e.g. from a git index mode)≤ 1 open
classify() + Pending—none / caller-provided content
tags_from_filenametags_from_filenamenone
tags_from_interpretertags_from_interpreternone
file_is_text, is_text, is_text_bytesfile_is_text, is_text≤ 1 KiB
parse_shebang_from_file, parse_shebang_from_file_with, parse_shebang, parse_shebang_bytesparse_shebang_from_file, parse_shebangfirst line(s)

extensions and interpreters expose the raw tables.

§Differences from identify

Behaviour matches identify, including its quirks (Python shlex shebang splitting, nix-shell shebangs, os.path.splitext rules, Windows treating every file as executable). The deliberate exceptions keep the library safe on hostile input:

  • FIFOs and devices are never opened (identify blocks on a FIFO); they are tagged file + mode + name tags, and binary if the name does not decide the encoding.
  • The shebang parser reads at most MAX_SHEBANG_BYTES; a longer first line counts as “no shebang”.
  • On Unix, os.access(X_OK) is evaluated from the file’s mode, owner and group (see ExecPolicy), ignoring ACLs and noexec mounts.

Platform limitations (not deliberate):

  • On Windows a directory junction (and any other name-surrogate reparse point) is tagged symlink: the standard library reports it as a symlink and does not expose the reparse tag. identify (os.lstat) reports directory for a junction; true symlinks agree.

The git index fast path (Meta::from_git_mode, tags_from_meta) describes the index, not the work tree. It matches identify only when the two agree: in particular a 120000 entry is a symlink only if git created one. With core.symlinks=false (the usual case on Windows) git checks the link out as a regular file holding the target path, which identify tags file, …, text. On Windows, or whenever core.symlinks=false, do not trust 120000: lstat the file (tags_from_path, tags_from_metadata) instead.

Re-exports§

pub use tags::IntoTag;
pub use tags::Tag;
pub use tags::TagSet;
pub use tags::UnknownTagError;

Modules§

extensions
identify’s extensions module: the extension and file-name tables.
interpreters
identify’s interpreters module: the shebang interpreter table.
tags
Tags, tag sets and identify’s tag-group constants.

Structs§

FileInfo
Pre-loaded file information for I/O-free identification (kept for compatibility; classify() + Pending::finish_bytes is the more flexible equivalent).
Meta
What identification needs to know about a path besides its name: its FileKind and whether it is executable.
Pending
A classification waiting for file content (see classify()).
TagTable
One of identify’s lookup tables (EXTENSIONS, NAMES, …): a static map from &str keys to TagSets.

Enums§

Classification
Outcome of classify().
ExecPolicy
How to decide whether a file is executable.
FileKind
The kind of a file-system entry, as seen by lstat (symlinks are not followed).
IdentifyError
Errors from identifying a path.
ShlexError
Error from shlex_split, mirroring the ValueError messages of Python’s shlex.

Constants§

IDENTIFY_VERSION
Version of the Python identify release the tables in this crate were generated from.
MAX_SHEBANG_BYTES
Upper bound on the bytes parse_shebang consumes (the #! line plus any nix-shell continuation lines).
TAG_COUNT
Number of tags known to identify (len(identify.ALL_TAGS)).
TEXT_PROBE_LEN
Number of leading bytes identify inspects to decide text vs binary.

Statics§

ALL_TAGS
Every tag known to identify (identify.ALL_TAGS), sorted by name.

Functions§

classify
The I/O-free part of identify.tags_from_path: everything that follows from the entry’s Meta and name.
file_is_text
Whether the file at path looks like text, like identify.file_is_text (reads at most TEXT_PROBE_LEN bytes).
is_text
Read up to TEXT_PROBE_LEN bytes from reader and report whether they look like text, exactly like identify.is_text.
is_text_bytes
Whether the first TEXT_PROBE_LEN bytes of content look like text, exactly like identify.is_text(io.BytesIO(content)).
parse_shebang
Parse a shebang from the start of reader, exactly like identify.parse_shebang, returning the command tokens (empty when there is no usable shebang).
parse_shebang_bytes
parse_shebang for in-memory content (infallible).
parse_shebang_from_file
Parse the shebang of the file at path, like identify.parse_shebang_from_file: empty unless the file is executable (ExecPolicy::PythonCompat: always on Windows), otherwise parse_shebang on its content.
parse_shebang_from_file_with
parse_shebang_from_file with an explicit ExecPolicy deciding whether the file is executable (and therefore read at all).
shlex_split
Python’s shlex.split(s) (POSIX mode, whitespace_split=True, no comment characters).
tags_from_filename
Identify a file from its name alone (no I/O), exactly like identify.tags_from_filename.
tags_from_info
Identify a file from FileInfo without any I/O.
tags_from_interpreter
Tags for a shebang interpreter, exactly like identify.tags_from_interpreter.
tags_from_meta
Identify a path whose Meta is already known (e.g. from Meta::from_git_mode), without lstat.
tags_from_metadata
Identify a path whose lstat metadata (fs::symlink_metadata) the caller already holds, e.g. from an existence check (pre-commit filters files with os.path.lexists before identifying them).
tags_from_path
Identify a path, exactly like identify.tags_from_path, with ExecPolicy::PythonCompat.
tags_from_path_with
tags_from_path with an explicit ExecPolicy.

Type Aliases§

Result
Result type of the path-based functions.