spex/

lib.rs

/*!
Simple(ish) parser and extractor of XML.

This package provides an `XmlReader` which can automatically determine the character encoding
of UTF-8 and UTF-16 (big endian and little endian byte order) XML byte streams, and parse the
XML into an immutable `Element` tree held within an `XmlDocument`. It's also possible to use a
custom byte stream decoder to read XML in other character encodings.

The aim of this package is to support as closely as possible the W3C specifications
[Extensible Markup Language (XML) 1.0](http://www.w3.org/TR/xml/) and
[Namespaces in XML 1.0](http://www.w3.org/TR/xml-names/) for **well-formed** XML. This package
does **not** aim to support validation of XML, and consequently DTD (document type definition)
is deliberately not supported.

Namespace support is always enabled, so the colon character is not permitted within the names of
elements nor attributes.

# XML concepts already supported

* Elements
* Attributes
* Default namespaces `xmlns="namespace.com"`
* Prefixed namespaces `xmlns:prefix="namespace.com"`
* Processing instructions
* Comments (skipped and thus not retrievable)
* CDATA sections
* Element language `xml:lang` and filtering by language
* White space indication `xml:space`
* Automatic detection and decoding of UTF-8 and UTF-16 XML streams.
* Support for custom encodings where the encoding is known **before** parsing, and where the
client supplies a custom decoder to handle the byte-to-character conversion.

# Examples

## Reading an XML file

Suppose you want to read and extract XML from a file you know to be either UTF-8 or UTF-16
encoded. You can use `XmlReader::parse_auto` to read, parse, and extract the XML from the file
and return either an `XmlDocument` or an `std::io::Error`.

```
# use std::fs::File;
# use spex::parsing::XmlReader;
# fn main() -> Result<(), std::io::Error> {
let xml_file = File::open("test_resources/xml_utf8_BOM.xml")?;
let xml_doc = XmlReader::parse_auto(xml_file)?;
# Ok(())
# }
```

## Traversing an `XmlDocument`

Once you have an `XmlDocument` you can grab an immutable reference to the root `Element` and
then traverse through the element tree using the `req` (required child element) and `opt`
(optional child element) methods to target the **first** child element with the specified name.
And once we're pointing at the desired target, we can use `element()` or `text()` to attempt to
grab the element or text-only content of the target element.

For example, let's define a simple XML structure where required elements have a name starting
with "r_" and optional elements have a name starting with "o_".

```
# use std::fs::File;
# use spex::{parsing::XmlReader, xml::XmlDocument, common::XmlError};
# fn main() {
#    // Panic if either step generates an error.
#    let xml_doc = read_xml().unwrap();
#    extract_xml(xml_doc).unwrap();
# }
#
# // The XML parsing methods might throw an std::io::Error, so they go into their own method.
# fn read_xml() -> Result<XmlDocument, std::io::Error> {
# let xml = "
<root>
    <r_Widget>
        <r_Name>Helix</r_Name>
        <o_AdditionalInfo>
            <r_ReleaseDate>2021-05-12</r_ReleaseDate>
            <r_CurrentVersion>23.10</r_CurrentVersion>
            <o_TopContributors>
                <r_Name>archseer</r_Name>
                <r_Name>the-mikedavis</r_Name>
                <r_Name>sudormrfbin</r_Name>
                <r_Name>pascalkuthe</r_Name>
                <r_Name>dsseng</r_Name>
                <r_Name>pickfire</r_Name>
            </o_TopContributors>
        </o_AdditionalInfo>
    </r_Widget>
</root>
# ";
# let xml_doc = XmlReader::parse_auto(xml.as_bytes());
#    xml_doc
# }

// Once the above XML is turned into an XmlDocument, it gets passed to this method.
fn extract_xml(xml_doc: XmlDocument) -> Result<(), XmlError> {
    // Let's start by grabbing a reference to the widget element. Because we use req to indicate
    // that it should be considered an error if this required element is missing, the element()
    // method will return a Result<&Element, XmlError>. So we use the `?` operator to throw the
    // XmlError if it occurs.
    let widget = xml_doc.root().req("r_Widget").element()?;

    // The name is required, so we just use req again. We also expect the name to contain only
    // simple text content (not mixed with other elements or processing instructions) so we
    // call text() followed by the `?` operator to throw the XmlError that will be generated if
    // either the name element is not found, or if it contains non-simple content.
    let widget_name = widget.req("r_Name").text()?;
    # assert_eq!(widget_name, "Helix");

    // The info and top contributor elements are optional (may or may not appear in this type of
    // XML document) so we can use the opt method to indicate that it is not an error if either
    // element is not found. Instead of a Result<&Element, XmlError> this entirely optional
    // chain will cause element() to give us an Option<&Element> instead, so we use `if let`
    // to take action only if the given optional chain elements all exist.
    if let Some(top_contrib_list) = widget
        .opt("o_AdditionalInfo")
        .opt("o_TopContributors")
        .element() {
        println!("Found top {} contributors!",
            top_contrib_list.elements().filter(|e| e.is_named("r_Name")).count());
        # assert_eq!(top_contrib_list.elements().filter(|e| e.is_named("r_Name")).count(), 6);
    }

    // If we want the release date, that's a required element within an optional element. In
    // other words, it's not an error if "o_AdditionalInfo" is missing, but if it *is* found
    // then we consider it an error if it does not contain "r_ReleaseDate". This is a mixed
    // chain, involving both required and optional, which means that element() will return a
    // Result<Option<&Element>, XmlError>, an Option wrapped in a Result. So we use `if let` and
    // the `?` operator together.
    if let Some(release_date) = widget.opt("o_AdditionalInfo").req("r_ReleaseDate").element()? {
        println!("Release date: {}", release_date.text()?);
        # assert_eq!(release_date.text().unwrap(), "2021-05-12");
    }

    Ok(())
}
```

Note that the return type of the `element()` and `text()` methods varies depending on whether
the method chain involves `req` or `opt` or both. This table summarizes the scenarios.

<table>
    <thead>
        <tr>
            <th scope="col">Chain involves</th>
            <th scope="col"><code>element()</code> returns</th>
            <th scope="col"><code>text()</code> returns</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <th scope="row">only <code>req</code></th>
            <td><code>Result&lt;&amp;Element, XmlError&gt;</code></td>
            <td><code>Result&lt;&amp;str, XmlError&gt;</code></td>
        </tr>
        <tr>
            <th scope="row">only <code>opt</code></th>
            <td><code>Option&lt;&amp;Element&gt;</code></td>
            <td><code>Result&lt;Option&lt;&amp;str&gt;, XmlError&gt;</code></td>
        </tr>
        <tr>
            <th scope="row">both <code>req</code> and <code>opt</code></th>
            <td><code>Result&lt;Option&lt;&amp;Element&gt;, XmlError&gt;</code></td>
            <td><code>Result&lt;Option&lt;&amp;str&gt;, XmlError&gt;</code></td>
        </tr>
    </tbody>
</table>

Similarly, the return types of `att_req` and `att_opt` methods also vary depending on the method
chain.

<table>
    <thead>
        <tr>
            <th scope="col">Chain involves</th>
            <th scope="col"><code>att_req(name)</code> returns</th>
            <th scope="col"><code>att_opt(name)</code> returns</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <th scope="row">only <code>req</code></th>
            <td><code>Result&lt;&amp;str, XmlError&gt;</code></td>
            <td><code>Result&lt;Option&lt;&amp;str&gt;, XmlError&gt;</code></td>
        </tr>
        <tr>
            <th scope="row">only <code>opt</code></th>
            <td><code>Result&lt;Option&lt;&amp;str&gt;, XmlError&gt;</code></td>
            <td><code>Option&lt;&amp;str&gt;</code></td>
        </tr>
        <tr>
            <th scope="row">both <code>req</code> and <code>opt</code></th>
            <td><code>Result&lt;Option&lt;&amp;str&gt;, XmlError&gt;</code></td>
            <td><code>Result&lt;Option&lt;&amp;str&gt;, XmlError&gt;</code></td>
        </tr>
    </tbody>
</table>

It's easier to remember this as the following: `req`/`att_req` will generate an error if the
element or attribute does not exist, so their use means that the return type must involve a
`Result<_, XmlError>` of some sort. And `opt`/`att_opt` may or may not return a value, so their
use means that the return type must involve an `Option<_>` of some sort. And mixing the two
(required and optional) means that the return type must involve a `Result<Option<_>, XmlError>`
of some sort. And `text()` generates an error if the target element does not have simple content
(no child elements and no processing instructions) so its use also means that the return type
must involve a `Result` of some sort.

## More complex traversal using `XmlPath`

The methods `req` and `opt` always turn their attention to the **first** child element with the
given name. It's not possible to use them to target a sibling, say the second "Widget" within a
list of "Widget" elements. To target siblings, and/or to iterate multiple elements, you instead
use `XmlPath`. (Don't confuse this with [XPath](https://www.w3.org/TR/xpath-3/) which has a
similar purpose but very different implementation.)

For example, if you have XML which contains a list of employees, and you want to iterate the
employees' tasks' deadlines, you could use `XmlPath` like this:


```
# use std::fs::File;
# use spex::{parsing::XmlReader, xml::XmlDocument, common::XmlError};
# fn main() {
#    // Panic if either step generates an error.
#    let xml_doc = read_xml().unwrap();
#    extract_xml(xml_doc).unwrap();
# }
#
# // The XML parsing methods might throw an std::io::Error, so they go into their own method.
# fn read_xml() -> Result<XmlDocument, std::io::Error> {
# let xml = "
<roster>
    <employee>
        <name>Angelica</name>
        <department>Finance</department>
        <task-list>
            <task>
                <name>Payroll</name>
                <deadline>tomorrow</deadline>
            </task>
            <task>
                <name>Reconciliation</name>
                <deadline>Friday</deadline>
            </task>
        </task-list>
    </employee>
    <employee>
        <name>Byron</name>
        <department>Sales</department>
        <task-list>
            <task>
                <name>Close the big deal</name>
                <deadline>Saturday night</deadline>
            </task>
        </task-list>
    </employee>
    <employee>
        <name>Cat</name>
        <department>Software</department>
        <task-list>
            <task>
                <name>Fix that bug</name>
                <deadline>Maybe later this month</deadline>
            </task>
            <task>
                <name>Add that new feature</name>
                <deadline>Possibly this year</deadline>
            </task>
            <task>
                <name>Make that customer happy</name>
                <deadline>Good luck with that</deadline>
            </task>
        </task-list>
    </employee>
</roster>
# ";
# let xml_doc = XmlReader::parse_auto(xml.as_bytes());
#    xml_doc
# }

// Once the above XML is turned into an XmlDocument, it gets passed to this method.
fn extract_xml(xml_doc: XmlDocument) -> Result<(), XmlError> {
    for deadline in xml_doc.root()
        .all("employee")
        .first("task-list")
        .all("task")
        .first("deadline")
        .iter() {
        println!("Found task deadline: {}", deadline.text()?);
    }
    Ok(())
}
```

This creates and iterates an `XmlPath` which represents "the first deadline element within
every task within the first task-list within every employee". Based on the example XML above,
this will print out all the text content of all six "deadline" elements.

Note that we could use `first("employee")` if we only wanted the first employee. Or we could
use `nth("employee", 1)` if we only want the second employee (zero would point to the first).
Or we could use `last("employee")` if we only want the last employee. Similarly, we could use
`first("task")` if we only wanted to consider the first task in each employee's list.

## Filtering elements within an `XmlPath`

An `XmlPath` not only lets you specify which child element names are of interest, but also lets
you specify which xml:lang patterns are of interest, and lets you specify a required attribute
name-value pair which must be found within a child element in order to include it in the
iterator.

```
# use spex::{parsing::XmlReader, xml::XmlDocument, common::XmlError,
# extraction::ExtendedLanguageRange};
# fn main() {
#    // Panic if either step generates an error.
#    let xml_doc = read_xml().unwrap();
#    extract_xml(xml_doc).unwrap();
# }
#
# // The XML parsing methods might throw an std::io::Error, so they go into their own method.
# fn read_xml() -> Result<XmlDocument, std::io::Error> {
# let xml = "
<inventory>
    <box type='games'>
        <item>
            <name xml:lang='en'>Command &amp; Conquer: Tiberian Dawn</name>
            <name xml:lang='en-US'>Command &amp; Conquer</name>
            <name xml:lang='de'>Command &amp; Conquer: Teil 1 - Der Tiberiumkonflikt</name>
        </item>
        <item>
            <name xml:lang='en'>Doom</name>
            <name xml:lang='sr'>Zla kob</name>
            <name xml:lang='ja'>ドゥーム</name>
        </item>
        <item>
            <name xml:lang='en'>Half-Life</name>
            <name xml:lang='sr'>Polu-život</name>
        </item>
    </box>
    <box type='movies'>
        <item>
            <name xml:lang='en'>Aliens</name>
            <name xml:lang='sv-SE'>Aliens - Återkomsten</name>
            <name xml:lang='vi'>Quái Vật Không Gian 2</name>
        </item>
        <item>
            <name xml:lang='en'>The Cabin In The Woods</name>
            <name xml:lang='bg'>Хижа в гората</name>
            <name xml:lang='fr'>La cabane dans les bois</name>
        </item>
    </box>
</inventory>
# ";
# let xml_doc = XmlReader::parse_auto(xml.as_bytes());
#    xml_doc
# }

// Once the above XML is turned into an XmlDocument, it gets passed to this method.
fn extract_xml(xml_doc: XmlDocument) -> Result<(), XmlError> {
    let english = ExtendedLanguageRange::new("en")?;

    for movie in xml_doc.root()
        .all("box")
        .with_attribute("type", "games")
        .all("item")
        .all("name")
        .filter_lang_range(&english)
        .iter() {
        println!("Found movie title in English: {}", movie.text()?);
    }
    # assert_eq!(xml_doc.root()
    #    .all("box")
    #    .with_attribute("type", "games")
    #    .all("item")
    #    .all("name")
    #    .filter_lang_range(&english).iter().count(), 4);
    Ok(())
}
```

This will print out the names of all four English-language titles for the three games. It will
skip all of the movies, and all names which are rejected by the "en" language filter. Note
that this "en" filter will match both `xml:lang="en"` and `xml:lang="en-US"` so you'll get two
matching name elements for the first game.

## Attribute extraction

Getting the value of an attribute is done with the methods `att_req` (generate an error if the
attribute is missing) and `att_opt` (no error if the attribute is missing).

For example, given this simple XML document, we can grab the attribute values easily.

```
# use spex::{parsing::XmlReader, xml::XmlDocument, common::XmlError,
# extraction::ExtendedLanguageRange};
# fn main() {
#    // Panic if either step generates an error.
#    let xml_doc = read_xml().unwrap();
#    extract_xml(xml_doc).unwrap();
# }
#
# // The XML parsing methods might throw an std::io::Error, so they go into their own method.
# fn read_xml() -> Result<XmlDocument, std::io::Error> {
# let xml = "
<root generationDate='2023-02-09T18:10:00Z'>
    <record id='35517'>
        <temp locationId='23'>40.5</temp>
    </record>
    <record id='35518'>
        <temp locationId='36'>38.9</temp>
    </record>
</root>
# ";
# let xml_doc = XmlReader::parse_auto(xml.as_bytes());
#    xml_doc
# }

// Once the above XML is turned into an XmlDocument, it gets passed to this method.
fn extract_xml(xml_doc: XmlDocument) -> Result<(), XmlError> {
    // Iterate the records using an XmlPath.
    for record in xml_doc.root().all("record").iter() {
        // The record@id attribute is required (we consider it an error if it is missing).
        // So use att_req and then the `?` syntax to throw any XmlError generated.
        let record_id = record.att_req("id")?;

        let temp = record.req("temp").element()?;
        let temp_value = temp.text()?;

        // The temp@locationId attribute is optional (we don't consider it an error if it's not
        // found within this element). So use att_opt and then `if let` to check for it.
        if let Some(loc_id) = temp.att_opt("locationId") {
            println!("Found temperature recording {} at location ID {}", temp_value, loc_id);
        } else {
            println!("Found temperature recording {} at unspecified location.", temp_value);
        }
    }
    Ok(())
}
```

**Note:** the `xml:lang` and `xml:space` values cannot be read from as attribute values from an
`Element`, because these are "special attributes" whose values are inherited by child elements
(and the language is inherited by an element's attributes too). To get the effective value of
these language and space properties, see the methods `language_tag` and `white_space_handling`
instead.

## Namespace handling

All of the examples so far have used XML without any namespace declarations, which means that
the element and attribute names are not within any namespace (or put another way, they have a
namespace which has no value). Specifying the target name of an element or attribute can be
done with a string slice `&str` when the namespace has no value. But when the target name has
a namespace value, you must specify the namespace in order to target the desired element.

The most direct way of doing this is to use a `(&str, &str)` tuple which contains the local
part and then **namespace** (not the prefix) of the element name. But you can also call the
`pre_ns` (preset or predefined namespace) method to let a cursor or XmlPath know that it should
assume the given namespace value if you don't use a tuple to directly specify the namespace for
each element and attribute within the method chain. An example is probably be the easiest way to
explain this.

```
# use spex::{parsing::XmlReader, xml::XmlDocument, common::XmlError,
# extraction::ExtendedLanguageRange};
# fn main() {
#    // Panic if either step generates an error.
#    let xml_doc = read_xml().unwrap();
#    extract_xml(xml_doc).unwrap();
# }
#
# // The XML parsing methods might throw an std::io::Error, so they go into their own method.
# fn read_xml() -> Result<XmlDocument, std::io::Error> {
# let xml = "
<!-- The root element declares that the default namespace for it and its descendants should
be the given URI. It also declares that any element/attribute using prefix 'pfx' belongs to a
namespace with a different URI. -->
<root xmlns='example.com/DefaultNamespace' xmlns:pfx='example.com/OtherNamespace'>
    <one>This child element has no prefix, so it inherits the default namespace.</one>
    <pfx:two>This child element has prefix pfx, so inherits the other namespace.</pfx:two>
    <pfx:three pfx:key='value'>Attribute names can be prefixed too.</pfx:three>
    <four key2='value2'>Unprefixed attribute names do *not* inherit namespaces.</four>
    <five xmlns='' key3='value3'>The default namespace can be cleared too.</five>
</root>
# ";
# let xml_doc = XmlReader::parse_auto(xml.as_bytes());
#    xml_doc
# }

// Once the above XML is turned into an XmlDocument, it gets passed to this method.
fn extract_xml(xml_doc: XmlDocument) -> Result<(), XmlError> {
    let root = xml_doc.root();
    // You can use a tuple to specify the local part and namespace of the targeted element.
    let one = root.req(("one", "example.com/DefaultNamespace")).element()?;

    // Or you can call pre_ns before a chain of req/opt/first/all/nth/last method calls.
    let two = root.pre_ns("example.com/OtherNamespace").req("two").element()?;

    // The effect of pre_ns continues until you call element() or text(), so you can keep
    // assuming the same namespace for child elements or attributes.
    let three_key = root.pre_ns("example.com/OtherNamespace").req("three").att_req("key")?;
    # assert_eq!(three_key, "value");

    // Be careful if the namespace changes (or is cleared) when moving down through child
    // elements and attributes. If that happens, you can call pre_ns again, or you can use a
    // tuple to explicitly state the different namespace.
    let four_key = root
        .pre_ns("example.com/DefaultNamespace")
        .req("four")
        .pre_ns("")
        .att_req("key2")?;
    # assert_eq!(four_key, "value2");

    // When no namespace applies to a method or attribute name, you don't need to specify any
    // namespace to target it, so you don't need to use pre_ns nor a tuple. But you can anyway
    // if you want to make it more explicit that there is no namespace.
    let five_key = root.req(("five", "")).att_req(("key3", ""))?;
    # assert_eq!(five_key, "value3");

    Ok(())
}
```

It's important to note that once you call `element()` the effect of pre_ns vanishes. So don't
forget that you if you do call `element()` in the middle of a method chain, you need to call
`pre_ns` again in order to specify the preset namespace from that point forward.

```
# use spex::{parsing::XmlReader, xml::XmlDocument, common::XmlError,
# extraction::ExtendedLanguageRange};
# fn main() {
#    // Panic if either step generates an error.
#    let xml_doc = read_xml().unwrap();
#    extract_xml(xml_doc).unwrap();
# }
#
# // The XML parsing methods might throw an std::io::Error, so they go into their own method.
# fn read_xml() -> Result<XmlDocument, std::io::Error> {
# let xml = "
<root xmlns='example.com/DefaultNamespace'>
    <topLevel>
        <innerLevel>
            <list>
                <item>something</item>
                <item>whatever</item>
                <item>more</item>
                <item>and so on</item>
            </list>
        </innerLevel>
    </topLevel>
</root>
# ";
# let xml_doc = XmlReader::parse_auto(xml.as_bytes());
#    xml_doc
# }

// Defining a static constant makes it quicker to type namespaces, and easier to read the code.
const NS_DEF: &str = "example.com/DefaultNamespace";

// Once the above XML is turned into an XmlDocument, it gets passed to this method.
fn extract_xml(xml_doc: XmlDocument) -> Result<(), XmlError> {
    // Use a chain of req calls to get to the required list, then use an XmlPath to iterate
    // however many items are found within the list and count them.

    // This first attempt will actually give us the wrong number, because once we call
    // element()? we receive an `&Element` reference, and the preset namespace effect is lost.
    // So the XmlPath we chain on straight after that will be searching the empty namespace and
    // won't find any matching elements and will report a count of zero.
    let mistake = xml_doc
        .root()
        .pre_ns(NS_DEF)
        .req("topLevel")
        .req("innerLevel")
        .req("list")
        .element()?
        .all("item")
        .iter()
        .count();
    # assert_eq!(mistake, 0);

    // You can fix the problem by either using an explicit name tuple `("item", NS_DEF)` or by
    // calling pre_ns again after element() so that the XmlPath knows which namespace should be
    // used when searching for items.
    let correct = xml_doc
        .root()
        .pre_ns(NS_DEF)
        .req("topLevel")
        .req("innerLevel")
        .req("list")
        .element()?
        .pre_ns(NS_DEF)
        .all("item")
        .iter()
        .count();
    # assert_eq!(correct, 4);

    // However, to avoid confusion, it's recommended to avoid including `element()` between
    // two different method chains, and to instead assign it to a variable name for clarity.
    let list = xml_doc
        .root()
        .pre_ns(NS_DEF)
        .req("topLevel")
        .req("innerLevel")
        .req("list")
        .element()?;

    let cleanest = list.all(("item", NS_DEF)).iter().count();
    # assert_eq!(cleanest, 4);

    Ok(())
}
```

## Error handling

The examples above have simplified the code snippets for brevity, but in a real
application you will need to handle the different error types returned by the different steps
of reading/parsing and extracting from XML. Here is a compact example which shows the error
handling needed for each step.

```
# use std::fs::File;
# use spex::{parsing::XmlReader, xml::XmlDocument, common::XmlError};
fn main() {
    // Decide what to do if either step returns an error.
    // For simplicity, we'll simply panic in this example, but in a real application you may
    // want to remap the error to the type used by your application, or trigger some recovery
    // logic instead.
    let xml_doc = match read_xml() {
        Ok(d) => d,
        Err(e) => panic!("XML reading or parsing failed!"),
    };
    match extract_xml(xml_doc) {
        Ok(()) => println!("Finished extracting from XML document without errors!"),
        Err(e) => panic!("XML extraction failed!"),
    }
}

// The XML parsing methods might throw an std::io::Error, so they go into their own method.
fn read_xml() -> Result<XmlDocument, std::io::Error> {
    let xml = "<root><child/></root>";
    let xml_doc = XmlReader::parse_auto(xml.as_bytes());
    xml_doc
}

// The extraction methods might throw an XmlError, so they go into their own method.
fn extract_xml(xml_doc: XmlDocument) -> Result<(), XmlError> {
    let child = xml_doc.root().req("child").element()?;
    # assert_eq!(child.text().unwrap(), "");
    Ok(())
}
```

# Release notes

## 0.1.0
Initial release.

## 0.2.0
Moved to [sipp 0.2.0](https://docs.rs/sipp/0.2.0/sipp/).

*/

#![forbid(unsafe_code)]
#![warn(missing_docs)]
pub mod common;
pub mod extraction;
pub mod parsing;
pub mod xml;