Crate xapi_rs

Source
Expand description

This project is an attempt at implementing a conformant xAPI 2.0.0 LRS.

It consists of three main modules that roughly map to (a) a data layer that defines the Rust bindings for the xAPI types, (b) a storage layer that takes care of persisting and fetching Data Access Objects representing the structures defined in the data layer, and finally (c) a Web server to handle the LRS calls proper.

§Third-party crates

This project depends on few best-of-breed crates to achieve correct compliance w/ other IETF and ISO standards referenced in xAPI.

Here’s a list of the most important ones:

  1. Deserialization and Serialization:

    • serde: for the basic serialization + deserialization capabilities.
    • serde_json: for the JSON format bindings.
    • serde_with: for custom helpers.

  2. IRL1, IRI2, URI3 and URL4:

    • iri-string: for IRIs and URIs incl. support for serde
    • url: for Uniform Resource Locators.

  3. UUID5:

    • uuid: for handling generating, parsing and formatting UUIDs.

  4. Date, Time and Durations:

    • chrono: for timezone-aware date and time handling.
    • speedate: for fast and simple duration6 parsing.

  5. Language Tags and MIME types:

    • language-tags: for parsing , formatting and comparing language tags as specified in BCP 47.
    • mime: for support of MIME types (a.k.a. Media Types) when dealing w/ Attachments.

  6. Email Address:

  7. Semantic Version:

  8. Case Insensitive Strings:

    • unicase: for comparing strings when case is not important (using Unicode Case-folding).

§data - Data structures

This module consists of Rust bindings for the IEEE Std 9274.1.1, IEEE Standard for Learning Technology— JavaScript Object Notation (JSON) Data Model Format and Representational State Transfer (RESTful) Web Service for Learner Experience Data Tracking and Access.

The standard describes a JSON7 data model format and a RESTful8 Web Service API9 for communication between Activities experienced by an individual, group, or other entity and an LRS10. The LRS is a system that exposes the RESTful Web Service API for the purpose of tracking and accessing experiential data, especially in learning and human performance.

In this project when i mention “xAPI” i mean the collection of published documents found here.

§The Validate trait

Types defined in this project rely on serde and a couple of other libraries to deserialize xAPI data. As they stand + Unit and Integration Tests for these types use the published examples to partly ensure their correctness in at least they will consume the input stream and will produce instances of those types that can be later manipulated.

I said partly b/c not all the rules specified by the specifications can or are encoded in the techniques used for unmarshelling the input data stream.

For example when xAPI specifies that a property must be an IRL the corresponding field is defined as an IriString. But an IRL is not just an IRI! As xAPI (3. Definitions, acronyms, and abbreviations) states…

…an IRL is an IRI that when translated into a URI (per the IRI to URI rules), is a URL.

Unfortunately the iri-string library does not offer out-of-the-box support for IRLs.

Another example of the limitations of solely relying on serde for instantiating correct types is the email address (mbox) property of Agents and Groups. xAPI (4.2.2.1 Actor) states that an mbox is a…

mailto IRI. The required format is “mailto:email address”.

For those reasons a Validate trait is defined and implemented to ensure that an instance of a type that implements this trait is valid, in that it satisfies the xAPI constraints, if + when it passes the validate() call.

If a validate() call returns None when it shouldn’t it’s a bug.

§Equality and Equivalence - The Fingerprint trait

There’s the classical Equality concept ubiquitous in software that deals w/ object Equality. That concept affects (in Rust) the Hash, PartialEq and Eq Traits. Ensuring that our xAPI Data Types implement those Traits mean they can be used as Keys in HashMaps and distinct elements in HashSets.

The xAPI describes (and relies) on a concept of Equivalence that determines if two instances of the same Data Type (say Statements) are equal. That Equivalence is different from Equality introduced earlier. So it’s possible to have two Statements that have different hash11 values yet are equivalent. Note though that if two Statements (and more generally two instances of the same Data Type) are equal then they’re also equivalent. In other words, Equality implies Equivalence but not the other way around.

To satisfy Equivalence between instances of a Data Type i intoroduce a Trait named Fingerprint. The required function of this Trait; i.e. fingerprint(), is used to test for Equivalence between two instances of the same Data Type.

For most xAPI Data Types, both the hash and fingerprint functions yield the same result. When they differ the Equivalence only considers properties the xAPI standard qualifies as preserving immutability requirements, for example for Statements those are…

  • Actor, except the ordering of Group members,
  • Verb, except for display property,
  • Object,
  • Duration, excluding precisions beyond 0.01 second.

Note though that even when they yield the same result, the implementations take into consideration the following constraints –and hence apply the required conversions before the test for equality…

  • Case-insensitive string comparisons when the property can be safely compared this way such as the mbox (email address) of an Actor but not the name of an Account.
  • IRI Normalization by first splitting it into Absolute and Fragment parts then normalizing the Absolute part before hashing the two in sequence.

§The Canonical trait

xAPI requires LRS implementations to sometimes produce results in canonical form –See Language Filtering Requirements for Canonical Format Statements for example.

This trait defines a method implemented by types required to produce such format.

§Getters, Builders and Setters

Once a type is instantiated, access to any of its fields –sometimes referred to in the documentation as properties using the camel case form mostly used in xAPI– is done through methods that mirror the Rust field names of the structures representing said types.

For example the homePage property of an Account is obtained by calling the method home_page() of an Account instance which returns a reference to the IRI string as &IriStr.

Sometimes however it is convenient to access the field as another type. Using the same example as above, the Account implementation offers a home_page_as_str() which returns a reference to the same field as &str.

This pattern is generalized thoughtout the project.

The project so far, except for rare use-cases, does NOT offer setters for any type field. Creating new instances of types by hand –as opposed to deserializing (from the wire)– is done by (a) instantiating a Builder for a type, (b) calling the Builder setters (using the same field names as those of the to-be built type) to set the desired values, and when ready, (c) calling the build() method.

Builders signal the occurence of errors by returning a Result w/ the error part being a DataError instance (a variant of MyError). Here’s an example…

    let act = Account::builder()
        .home_page("https://inter.net/login")?
        .name("example")?
        .build()?;
    // ...
    assert_eq!(act.home_page_as_str(), "https://inter.net/login");
    assert_eq!(act.name(), "example");

§Naming

Naming properties in xAPI Objects is inconsistent. Sometimes the singular form is used to refer to a collection of items; e.g. member instead of members when referring to a Group’s list of Agents. In other places the plural form is correctly used; e.g. attachments in a SubStatement, or extensions everywhere it’s referenced.

I tried to be consistent in naming the fields of the corresponding types while ensuring that their serialization to, and deserialization from, streams respect the label assigned to them in xAPI and backed by the accompanying examples. So to access a Group’s Agents one would call members(). To add an Agent to a Group one would call member() on a GroupBuilder.

§db - Persistent Storage

This module deals w/ storing, mutating and retrieving database records representing the data types.

This project does not hide the database engine and SQL dialect it uses for achieving its purpose. PostgreSQL is the relational database engine used. When this page was last updated the PosgreSQL version in use was 16.3.

§A note about how Agents, Groups and Actors are stored in the database

xAPI describes an Actor as “…Agent or Identified Group object (JSON)”. An Agent has the following properties:

PropertyTypeDescriptionRequired
nameStringFull name of the Agent.No
mboxmailto IRIEmail address.[13]
mbox_sha1sumStringThe hex-encoded SHA1 hash of mbox.[13]
openidURIAn openID that uniquely identifies the Agent.[13]
accountObjectA user account and username pair.[13]

While an Identified Group has the following properties:

PropertyTypeDescriptionRequired
nameStringName of the Group.No
mboxmailto IRIEmail address.[13]
mbox_sha1sumStringThe hex-encoded SHA1 hash of mbox.[13]
openidURIAn openID that uniquely identifies the Group.[13]
accountObjectA user account and username pair.[13]
memberArray of Agent objectsUnordered list of group’s membersNo

Those mbox, mbox_sha1sum, openid, and account properties are also referred to as Inverse Functional Identifier (IFI for short). The Kind of IFI is encoded as an integer:

  • 0 -> email address (or mbox in xAPI parlance). Note we only store the email address proper w/o the mailto scheme.
  • 1 -> hex-encoded SHA1 hash of a mailto IRI; i.e. 40-character string.
  • 2 -> OpenID URI identifying the owner.
  • 3 -> account on an existing system stored as a single string by catenating the home_page URL, a ‘~’ symbol followed by a name (the username of the account holder).

We store this information in two tables: one for the IFI data proper, and another for the association of Actors to their IFIs.

§A note about the relation between an Agent and a Person

Worth noting also that one of the REST endpoints of the LRS (see section 4.1.6.3 Agents resource) is expected, given an Agent instance, to retrieve a special Person object with combined information about an Agent derived from an outside service?!.

This Person object is very similar to an Agent, but instead of each attribute having a single value, it has an array of them. Also it’s OK to include multiple identifying properties. Here’s a table of those Person’s properties:

PropertyTypeDescription
nameArray of StringsList of names.
mboxArray of IRIsList of Email addresses.
mbox_sha1sumArray of StringsList of hashes.
openidArray of URIsList of openIDs
accountArray of ObjectsList of Accounts.

It’s important to note here that while xAPI expects the LRS to access an external source of imformation to collect an Agent’s possible multiple names and IFIs –in order to aggragte them to build a Person– it is silent as to how a same Agent being identified by its single IFI ends up having multiple ones of the same or different Kinds. In addition if the single IFI that identifies an Agent w/ respect to the REST Resources is not enough to uniquely identify it, how are multiple Agent persona 14 connected? do they share a primary key? which Authority assigns such key? and how is that information recorded / accessed by the LRS?

Until those points are resolved, this LRS considers a Person to be an Agent and vice-versa.

§lrs - LRS Web Server

This module, nicknamed LaRS, consists of the Learning Record Store (LRS) —a web server– that allows access from Learning Record Providers (LRPs), or Consumers (LRCs).

§Concurrency

Concurrency control makes certain that a PUT, POST or DELETE does not perform operations based on stale data.

§Details

In accordance w/ xAPI, LaRS uses HTTP 1.1 entity tags (ETags) to implement optimistic concurrency control in the following resources, where PUT, POST or DELETE are allowed to overwrite or remove existing data:

  • State Resource
  • Agent Profile Resource
  • Activity Profile Resource

§Requirements

  • LaRS responding to a GET request adds an ETag HTTP header to the response.
  • LaRS responding to a PUT, POST, or DELETE request handles the If-Match header as described in RFC2616, HTTP 1.1 in order to detect modifications made after the document was last fetched.

If the preconditions in the request fails, LaRS:

  • returns HTTP status 412 Precondition Failed.
  • does not modify the resource.

If a PUT request is received without either header for a resource that already exists, LaRS:

  • returns HTTP status 409 Conflict.
  • does not modify the resource.

§Detecting the Lost Update problem using unreserved checkout

When a conflict is detected, either because a precondition fails or a HEAD request indicated a resource already exists, some implementations offer the user two choices:

  • download the latest revision from the server so that the user can merge using some independent mechanism; or
  • override the existing version on the server with the client’s copy.

If the user wants to override the existing revision on the server, a 2nd PUT request is issued. Depending on whether the document initially was known to exist or not, the client may either…

  • If known to exist, issue a new PUT request which includes an If-None-Match header field with the same etag as was used in the If-match header field in the first PUT request, or
  • If not known, issue a new PUT request which includes an If-Match with the etag of the existing resource on the server (this etag being the one recieved in the response to the initial HEAD request). This could also have been achieved by resubmitting the PUT request without a precondition. However, the advantage of using the precondition is that the server can block all PUT requests without any preconditions as such requests are guaranteed to come from old clients without knowledge of etags and preconditions.

  1. IRL: Internationalized Resource Locator. ↩

  2. IRI: Internationalized Resource Identifier. ↩

  3. URI: Uniform Resource Identifier. ↩

  4. URL: Uniform Resource Locator. ↩

  5. UUID: Universally Unique Identifier –see https://en.wikipedia.org/wiki/Universally_unique_identifier. ↩

  6. Durations in ISO 8601:2004(E) sections 4.4.3.2 and 4.4.3.3. ↩

  7. JSON: JavaScript Object Notation. ↩

  8. REST: Representational State Transfer. ↩

  9. API: Application Programming Interface. ↩

  10. LRS: Learning Record Store. ↩

  11. Just to be clear, hash here means the result of computing a message digest over the non-null values of an object’s field(s). ↩

  12. Durations in ISO 8601:2004(E) sections 4.4.3.2 and 4.4.3.3. ↩

  13. Exactly One of mbox, mbox_sha1sum, openid, account is required. ↩

  14. Person in that same section 4.1.6.3 is being used to indicate a person-centric view of the LRS Agent data, but Agents just refer to one persona (a person in one context). ↩

Modules§

Macros§

  • add_language
    Given $map (a LanguageMap dictionary) insert $label keyed by $tag creating the collection in the process if it was None.
  • emit_db_error
    Macro for logging and wrapping database errors before returning them as ours.
  • emit_error
    Log $err at level error before returning it.
  • emit_response
    Given $resource of type $type that is serde Serializable and $headers (an instance of a type that handles HTTP request headers)…
  • eval_preconditions
    Given an $etag (Entity Tag) value and $headers (an instance of a type that handles HTTP request headers), check that the If-XXX pre- conditions when present, pass.
  • handle_db_error
    Macro for logging and handling errors with a custom return value to use when the database raises a RowNotFound error.
  • merge_maps
    Given dst and src as two BTreeMaps wrapped in Option, replace or augment dst’ entries w/ src’s.
  • runtime_error
    Generate a message (in the style of format! macro), log it at level error and raise a runtime error.
  • set_email
    Both Agent and Group have an mbox property which captures an email address. This macro eliminates duplication of the logic involved in (a) parsing an argument $val into a valid EmailAddress, (b) raising a DataError if an error occurs, (b) assigning the result when successful to the appropriate field of the given $builder instance, and (c) resetting the other three IFI (Inverse Functional Identifier) fields to None.

Structs§

Enums§

Constants§

Traits§

  • Canonical
    Trait implemented by types that can produce a canonical form of their instances.
  • Fingerprint
    To assert Equivalence of two instances of an xAPI Data Type we rely on this Trait to help us compute a fingerprint for each instance. The computation of this fingerprint uses a recursive descent mechanism not unlike what the standard Hash does.
  • MultiLingual
    xAPI refers to a Language Map as a dictionary of words or sentences keyed by the RFC 5646: “Tags for Identifying Languages”.
  • Validate
    xAPI mandates certain constraints on the values of some properties of types it defines. Our API binding structures however limit the Rust type of almost all fields to be Strings or derivative types based on Strings. This is to allow deserializing all types from the wire even when their values violate those constraints.

Functions§

  • adl_verb
    Return a Verb identified by its Vocabulary variant.
  • build
    Entry point for constructing a Local Rocket and use it for either testing or not. When testing is TRUE a mock DB is injected otherwise it’s the real McKoy.
  • config
    This LRS server configuration Singleton.