Crate sphinx_inv

Expand description

A library to parse (and maybe one day write) Sphinx inventory files for referencing other documentation pages that use Sphinx.

In contraty to Sphinx itself this library parses the data using a combinator parser, instead of a regex, which has better performance and better error reporting.

This library was originally made for use in snakedown but an effort has been made to make it more generally useful.

§Disclaimer

The Sphinx inventory format doesn’t have a formal specification. What follows are just the rules that we (and others) have inferred from files we’ve seen in the wild. We try to be as correct as possible. That said, we can’t be guaranteed to be correct. If you find any errors, or have a valid file we can’t parse please open an issue!

Currently only v2 of the Sphinx Inventory file format is supported.

§Usage

The main entry points of this create are the the InventoryHeader and SphinxReference data structs and the SphinxInventoryReader and SphinxInventoryWriter structs to handle with them.

The SphinxInventoryReader and SphinxInventoryWriter can work with any struct that implements std::io::Read and std::io::Write respectively. These are internally buffered so you do not have to wrap them yourself.

In previous version we had separate structs for the zlib and plain-text versions, but these have now all been combined into one reader and one writer. The reader will detect which is necessary based on the description in the header. For example if the header metnions:

# The remainder of this file is compressed using zlib

it will automatically decompress the body, whereas if it says

# The remainder of this file is compressed using plain-text

it will use plain text reading. The writer will write the correct header along with the body in the correct format based on the WriteFormat provided at finalisation.

In the following examples we will use the plain text versions and the std::io::Cursor to make it easier to display the results, but the code should work basically unchanged by switching to a std::fs::File.

§Examples

let header = InventoryHeader::new("Sphinx Inv", "0.2.0", "plain-text");
let join_reference = SphinxReference::new(
    "str.join",
    SphinxType::Python(PyRole::Method),
    SphinxPriority::Standard,
    "library/stdtypes.html#$",
    "-",
);
let lower_reference = SphinxReference::new(
    "str.lower",
    SphinxType::Python(PyRole::Method),
    SphinxPriority::Standard,
    "library/stdtypes.html#$",
    "-",
);

let mut buffer = Vec::new();

let mut cursor = Cursor::new(buffer);

// the capacity is just to preallocate the internal buffer, it can be anything
let mut writer = SphinxInventoryWriter::from_header(header.clone(), 2);

// add the references to the writer
writer.add_reference(join_reference.clone());
writer.add_reference(lower_reference.clone());

// add_reference on it's own only adds it to the internal buffer
// nothing actually happens until you call [`SphinxInventoryWriter::finalize`]
writer
    .finalize(&mut cursor, &WriteFormat::Plain, true)
    .unwrap();

let written = String::from_utf8(cursor.into_inner()).unwrap();

assert_eq!(
    &written,
    "# Sphinx inventory version 2
# Project: Sphinx Inv
# Version: 0.2.0
# The remainder of this file is compressed using plain-text.
str.join py:method 1 library/stdtypes.html#$ -
str.lower py:method 1 library/stdtypes.html#$ -
"
);

let mut cursor = Cursor::new(written);

let mut reader = SphinxInventoryReader::from_reader(cursor).unwrap();

assert_eq!(&header, reader.header());

assert_eq!(reader.next().unwrap().unwrap(), join_reference);
assert_eq!(reader.next().unwrap().unwrap(), lower_reference);

§Format Description

As noted by Skinn et al. currently, a inventory file (in the v2 format) has 2 parts:

the header
the body

§Header description

The header needs to be of the following format:

# Sphinx inventory version 2
# Project: <project name>
# Version: <full version number>
# The remainder of this file is compressed using zlib.

§Caveats:

The first line has to match exactly
version number should not contain a leading v
currently zlib is the only compression method that Sphinx supports.
Though it is not specified, it is expected that the text mentioned above is in ascii. The project name can contain unicode, but the text in the example must match exactly¹.
While Sphinx itself allows for userdefinable domains and roles, this is not possible for this library due to being complied. However we have made an attempt to include as many domains and roles we found out in the wild. If you are missing any, please submit a feature request or pull request to add it!

For more indepth explanation of the format, please see spobjinv

§Body format

The remaining body of the file after the header must be compressed with zlib. In the decompressed data each line should have the following format:

{name} {domain}:{role} {priority} {uri} {dispname}

Specifically it must match this regex: (.+?)\s+(\S+)\s+(-?\d+)\s+?(\S*)\s+(.*)

For example:

str.join py:method 1 library/stdtypes.html#$ -

technically it matter doesn’t as long as the byte offsets are the same since the Sphinx implementation just skips a known amount of bytes, but this is a impl detail so we recommend following the format. ↩

Structs§

InventoryHeader: Struct for handling the metadata of an inventory such as project name and version
SphinxInventoryReader: The main entrypoint to this crate, used to read and parse sphinx reference data
SphinxInventoryWriter: The main entrypoint to this crate, used to write and format sphinx reference data
SphinxParseError: Error type when parsing either the header or a record fails. An error type to show parsing errors when reading an inventory file this clones the underlying data so that it is not tied to the lifetime of the buffer. This can be used to format errors by users for example by using [annotate-snippets]
SphinxReference: The main data struct of this crate with the necessary information to link to external A reference to something (can be either internal or external) has all the info to be serialized or deserialised from a sphinx inventory file.

Enums§

CRole: The various known domains and roles used in sphinx inventory files Describes a C role that has been observed in the wild, i.e. one of the known inventory file declared at least one line with the type c:{role} if you would like one added please open a feature request
CmakeRole: The various known domains and roles used in sphinx inventory files
CppRole: The various known domains and roles used in sphinx inventory files Describes a C++ role that has been observed in the wild, i.e. one of the known inventory file declared at least one line with the type cpp:{role} if you would like one added please open a feature request
JsRole: The various known domains and roles used in sphinx inventory files Describes a JavaScript role that has been observed in the wild, i.e. one of the known inventory file declared at least one line with the type js:{role} if you would like one added please open a feature request
MathRole: The various known domains and roles used in sphinx inventory files Describes a Mathematics role that has been observed in the wild, i.e. one of the known inventory file declared at least one line with the type math:{role} if you would like one added please open a feature request
MissingHeaderComponent: Error type when there is not enough input from the underlying reader to properly parse the header This error occurs when the underlying reader does not provide enough input to parse a header correctly Note that this is purely based on how many lines have been read from the reader and the order in which they were, and does not occur when said lines cannot be parsed correctly, for that see SphinxInvError::ParseError
PyRole: The various known domains and roles used in sphinx inventory files Describes a Python role that has been observed in the wild, i.e. one of the known inventory file declared at least one line with the type py:{role} if you would like one added please open a feature request
RstRole: The various known domains and roles used in sphinx inventory files Describes a RST role that has been observed in the wild, i.e. one of the known inventory file declared at least one line with the type rst:{role} if you would like one added please open a feature request
SipRole: The various known domains and roles used in sphinx inventory files SIP is a tool for automatically generating Python bindings for C and C++ libraries. see more
SphinxInvError: The main error type returned by this crate
SphinxPriority: The search priority of the associated object used by Sphinx The search priority of the associated object used by Sphinx This is here mostly because it is a required part of the inventory format it is not used anywhere in [sphinx_inv]
SphinxType: type used to parse {domain}:{roles} information provided by Sphinx used to disembguate between object types and names between different languages a sphinx type consists of a domain, and a role written as {domain}:{role} (e.g. c:struct) this enum defines all the domains and roles that we know. in shpinx this is used to avoid name conflicts but for us it is not really used beyond parsing it.
StdRole: The various known domains and roles used in sphinx inventory files
WriteFormat: The main entrypoint to this crate, used to write and format sphinx reference data

Crate sphinx_inv

Crate sphinx_inv Copy item path

§Disclaimer

§Usage

§Examples

§Format Description

§Header description

§Caveats:

§Body format

Structs§

Enums§

Crate sphinx_inv