Crate xapi_rs

Expand description

HTTP Server implementation of xAPI 2.0.0 LRS.

It consists of three main modules that roughly map to (a) a data layer that defines the Rust bindings for the xAPI types, (b) a storage layer that takes care of persisting and fetching Data Access Objects representing the structures defined in the data layer, and finally (c) a Web server to handle the LRS calls proper.

§Third-party crates

This project depends on few best-of-breed crates to achieve correct compliance w/ other IETF and ISO standards referenced in xAPI.

Here’s a list of the most important ones:

Deserialization and Serialization:
- serde: for the basic serialization + deserialization capabilities.
- serde_json: for the JSON format bindings.
- serde_with: for custom helpers.
IRL¹, IRI², URI³ and URL⁴:
- iri-string: for IRIs and URIs incl. support for serde
- url: for Uniform Resource Locators.
UUID⁵:
- uuid: for handling generating, parsing and formatting UUIDs.
Date, Time and Durations:
- chrono: for timezone-aware date and time handling.
- speedate: for fast and simple duration⁶ parsing.
Language Tags and MIME types:
- language-tags: for parsing , formatting and comparing language tags as specified in BCP 47.
- mime: for support of MIME types (a.k.a. Media Types) when dealing w/ Attachments.
Email Address:
- email_address: for parsing and validating email addresses.
Semantic Version:
- semver: for semantic version parsing and generation as per Semantic Versioning 2.0.0.
Case Insensitive Strings:
- unicase: for comparing strings when case is not important (using Unicode Case-folding).
JWS signatures:
- josekit: for creating + validating JWS signed Statements.
- openssl: for handling X.509 certificates when included in JWS Headers.

§`data` - Data structures

This module consists of Rust bindings for the IEEE Std 9274.1.1, IEEE Standard for Learning Technology— JavaScript Object Notation (JSON) Data Model Format and Representational State Transfer (RESTful) Web Service for Learner Experience Data Tracking and Access.

The standard describes a JSON⁷ data model format and a RESTful⁸ Web Service API⁹ for communication between Activities experienced by an individual, group, or other entity and an LRS¹⁰. The LRS is a system that exposes the RESTful Web Service API for the purpose of tracking and accessing experiential data, especially in learning and human performance.

In this project when i mention “xAPI” i mean the collection of published documents found here.

§The `Validate` trait

Types defined in this project rely on serde and a couple of other libraries to deserialize xAPI data. As they stand + Unit and Integration Tests for these types use the published examples to partly ensure their correctness in at least they will consume the input stream and will produce instances of those types that can be later manipulated.

I said partly b/c not all the rules specified by the specifications can or are encoded in the techniques used for unmarshelling the input data stream.

For example when xAPI specifies that a property must be an IRL the corresponding field is defined as an IriString. But an IRL is not just an IRI! As xAPI (3. Definitions, acronyms, and abbreviations) states…

…an IRL is an IRI that when translated into a URI (per the IRI to URI rules), is a URL.

Unfortunately the iri-string library does not offer out-of-the-box support for IRLs.

Another example of the limitations of solely relying on serde for instantiating correct types is the email address (mbox) property of Agents and Groups. xAPI (4.2.2.1 Actor) states that an mbox is a…

mailto IRI. The required format is “mailto:email address”.

For those reasons a Validate trait is defined and implemented to ensure that an instance of a type that implements this trait is valid, in that it satisfies the xAPI constraints, if + when it passes the validate() call.

If a validate() call returns None when it shouldn’t it’s a bug.

§Equality and Equivalence - The `Fingerprint` trait

There’s the classical Equality concept ubiquitous in software that deals w/ object Equality. That concept affects (in Rust) the Hash, PartialEq and Eq Traits. Ensuring that our xAPI Data Types implement those Traits mean they can be used as Keys in HashMaps and distinct elements in HashSets.

The xAPI describes (and relies) on a concept of Equivalence that determines if two instances of the same Data Type (say Statements) are equal. That Equivalence is different from Equality introduced earlier. So it’s possible to have two Statements that have different hash¹¹ values yet are equivalent. Note though that if two Statements (and more generally two instances of the same Data Type) are equal then they’re also equivalent. In other words, Equality implies Equivalence but not the other way around.

To satisfy Equivalence between instances of a Data Type i introduce a Trait named Fingerprint. The required function of this Trait; i.e. fingerprint(), is used to test for Equivalence between two instances of the same Data Type.

For most xAPI Data Types, both the hash and fingerprint functions yield the same result. When they differ the Equivalence only considers properties the xAPI standard qualifies as preserving immutability requirements, for example for Statements those are…

Actor, except the ordering of Group members,
Verb, except for display property,
Object,
Duration, excluding precision beyond 0.01 second.

Note though that even when they yield the same result, the implementations take into consideration the following constraints –and hence apply the required conversions before the test for equality…

Case-insensitive string comparisons when the property can be safely compared this way such as the mbox (email address) of an Actor but not the name of an Account.
IRI Normalization by first splitting it into Absolute and Fragment parts then normalizing the Absolute part before hashing the two in sequence.

§The `Canonical` trait

xAPI requires LRS implementations to sometimes produce results in canonical form –See Language Filtering Requirements for Canonical Format Statements for example.

This trait defines a method implemented by types required to produce such format.

§Getters, Builders and Setters

Once a type is instantiated, access to any of its fields –sometimes referred to in the documentation as properties using the camel case form mostly used in xAPI– is done through methods that mirror the Rust field names of the structures representing said types.

For example the homePage property of an Account is obtained by calling the method home_page() of an Account instance which returns a reference to the IRI string as &IriStr.

Sometimes however it is convenient to access the field as another type. Using the same example as above, the Account implementation offers a home_page_as_str() which returns a reference to the same field as &str.

This pattern is generalized thoughtout the project.

The project so far, except for rare use-cases, does NOT offer setters for any type field. Creating new instances of types by hand –as opposed to deserializing (from the wire)– is done by (a) instantiating a Builder for a type, (b) calling the Builder setters (using the same field names as those of the to-be built type) to set the desired values, and when ready, (c) calling the build() method.

Builders signal the occurrence of errors by returning a Result w/ the error part being a DataError instance (a variant of MyError). Here’s an example…

    let act = Account::builder()
        .home_page("https://inter.net/login")?
        .name("example")?
        .build()?;
    // ...
    assert_eq!(act.home_page_as_str(), "https://inter.net/login");
    assert_eq!(act.name(), "example");

§Naming

Naming properties in xAPI Objects is inconsistent. Sometimes the singular form is used to refer to a collection of items; e.g. member instead of members when referring to a Group’s list of Agents. In other places the plural form is correctly used; e.g. attachments in a SubStatement, or extensions everywhere it’s referenced.

I tried to be consistent in naming the fields of the corresponding types while ensuring that their serialization to, and deserialization from, streams respect the label assigned to them in xAPI and backed by the accompanying examples. So to access a Group’s Agents one would call members(). To add an Agent to a Group one would call member() on a GroupBuilder.

§`db` - Persistent Storage

This module deals w/ storing, mutating and retrieving database records representing the data types.

This project does not hide the database engine and SQL dialect it uses for achieving its purpose. PostgreSQL is the relational database engine used. When this page was last updated the PosgreSQL version in use was 16.3.

§A note about how Agents, Groups and Actors are stored in the database

xAPI describes an Actor as “…Agent or Identified Group object (JSON)”. An Agent has the following properties:

Property	Type	Description	Required
`name`	String	Full name of the Agent.	No
`mbox`	mailto IRI	Email address.	[¹³]
`mbox_sha1sum`	String	The hex-encoded SHA1 hash of mbox.	[¹³]
`openid`	URI	An openID that uniquely identifies the Agent.	[¹³]
`account`	Object	A user account and username pair.	[¹³]

While an Identified Group has the following properties:

Property	Type	Description	Required
`name`	String	Name of the Group.	No
`mbox`	mailto IRI	Email address.	[¹³]
`mbox_sha1sum`	String	The hex-encoded SHA1 hash of mbox.	[¹³]
`openid`	URI	An openID that uniquely identifies the Group.	[¹³]
`account`	Object	A user account and username pair.	[¹³]
`member`	Array of Agent objects	Unordered list of group’s members	No

Those mbox, mbox_sha1sum, openid, and account properties are also referred to as Inverse Functional Identifier (IFI for short). The Kind of IFI is encoded as an integer:

0 -> email address (or mbox in xAPI parlance). Note we only store the email address proper w/o the mailto scheme.
1 -> hex-encoded SHA1 hash of a mailto IRI; i.e. 40-character string.
2 -> OpenID URI identifying the owner.
3 -> account on an existing system stored as a single string by catenating the home_page URL, a ‘~’ symbol followed by a name (the username of the account holder).

We store this information in two tables: one for the IFI data proper, and another for the association of Actors to their IFIs.

§A note about the relation between an Agent and a Person

Worth noting also that one of the REST endpoints of the LRS (see section 4.1.6.3 Agents resource) is expected, given an Agent instance, to retrieve a special Person object with combined information about an Agent derived from an outside service?!.

This Person object is very similar to an Agent, but instead of each attribute having a single value, it has an array of them. Also it’s OK to include multiple identifying properties. Here’s a table of those Person’s properties:

Property	Type	Description
`name`	Array of Strings	List of names.
`mbox`	Array of IRIs	List of Email addresses.
`mbox_sha1sum`	Array of Strings	List of hashes.
`openid`	Array of URIs	List of openIDs
`account`	Array of Objects	List of Accounts.

It’s important to note here that while xAPI expects the LRS to access an external source of imformation to collect an Agent’s possible multiple names and IFIs –in order to aggragte them to build a Person– it is silent as to how a same Agent being identified by its single IFI ends up having multiple ones of the same or different Kinds. In addition if the single IFI that identifies an Agent w/ respect to the REST Resources is not enough to uniquely identify it, how are multiple Agent persona ¹⁴ connected? do they share a primary key? which Authority assigns such key? and how is that information recorded / accessed by the LRS?

Until those points are resolved, this LRS considers a Person to be an Agent and vice-versa.

§`lrs` - LRS Web Server

This module, nicknamed LaRS, consists of the Learning Record Store (LRS) —a web server– that allows access from Learning Record Providers (LRPs), or Consumers (LRCs).

§Concurrency

Concurrency control makes certain that a PUT, POST or DELETE does not perform operations based on stale data.

§Details

In accordance w/ xAPI, LaRS uses HTTP 1.1 entity tags (ETags) to implement optimistic concurrency control in the following resources, where PUT, POST or DELETE are allowed to overwrite or remove existing data:

State Resource
Agent Profile Resource
Activity Profile Resource

§Requirements

LaRS responding to a GET request adds an ETag HTTP header to the response.
LaRS responding to a PUT, POST, or DELETE request handles the If-Match header as described in RFC2616, HTTP 1.1 in order to detect modifications made after the document was last fetched.

If the preconditions in the request fails, LaRS:

returns HTTP status 412 Precondition Failed.
does not modify the resource.

If a PUT request is received without either header for a resource that already exists, LaRS:

returns HTTP status 409 Conflict.
does not modify the resource.

§Detecting the Lost Update problem using unreserved checkout

When a conflict is detected, either because a precondition fails or a HEAD request indicated a resource already exists, some implementations offer the user two choices:

download the latest revision from the server so that the user can merge using some independent mechanism; or
override the existing version on the server with the client’s copy.

If the user wants to override the existing revision on the server, a 2nd PUT request is issued. Depending on whether the document initially was known to exist or not, the client may either…

If known to exist, issue a new PUT request which includes an If-None-Match header field with the same etag as was used in the If-match header field in the first PUT request, or
If not known, issue a new PUT request which includes an If-Match with the etag of the existing resource on the server (this etag being the one recieved in the response to the initial HEAD request). This could also have been achieved by resubmitting the PUT request without a precondition. However, the advantage of using the precondition is that the server can block all PUT requests without any preconditions as such requests are guaranteed to come from old clients without knowledge of etags and preconditions.

IRL: Internationalized Resource Locator. ↩
IRI: Internationalized Resource Identifier. ↩
URI: Uniform Resource Identifier. ↩
URL: Uniform Resource Locator. ↩
UUID: Universally Unique Identifier –see https://en.wikipedia.org/wiki/Universally_unique_identifier. ↩
Durations in ISO 8601:2004(E) sections 4.4.3.2 and 4.4.3.3. ↩
JSON: JavaScript Object Notation. ↩
REST: Representational State Transfer. ↩
API: Application Programming Interface. ↩
LRS: Learning Record Store. ↩
Just to be clear, hash here means the result of computing a message digest over the non-null values of an object’s field(s). ↩
Durations in ISO 8601:2004(E) sections 4.4.3.2 and 4.4.3.3. ↩
Exactly One of mbox, mbox_sha1sum, openid, account is required. ↩
Person in that same section 4.1.6.3 is being used to indicate a person-centric view of the LRS Agent data, but Agents just refer to one persona (a person in one context). ↩

Modules§

resources: Endpoints grouped by Resource.

Macros§

add_language: Given $map (a LanguageMap dictionary) insert $label keyed by $tag creating the collection in the process if it was None.
constraint_violation_error: Generate a message (in the style of format! macro), log it at level error and raise a data constraint violation error.
emit_db_error: Macro for logging and wrapping database errors before returning them as ours.
emit_error: Log $err at level error before returning it.
emit_response: Given $resource of type $type that is serde Serializable and $headers (an instance of a type that handles HTTP request headers)…
eval_preconditions: Given an $etag (Entity Tag) value and $headers (an instance of a type that handles HTTP request headers), check that the If-XXX pre- conditions when present, pass.
handle_db_error: Macro for logging and handling errors with a custom return value to use when the database raises a RowNotFound error.
merge_maps: Given dst and src as two BTreeMaps wrapped in Option, replace or augment dst’ entries w/ src’s.
runtime_error: Generate a message (in the style of format! macro), log it at level error and raise a runtime error.
set_email: Both Agent and Group have an mbox property which captures an email address. This macro eliminates duplication of the logic involved in (a) parsing an argument $val into a valid EmailAddress, (b) raising a DataError if an error occurs, (b) assigning the result when successful to the appropriate field of the given $builder instance, and (c) resetting the other three IFI (Inverse Functional Identifier) fields to None.

Structs§

About: Structure containing information about an LRS, including supported extensions and xAPI version(s).
Account: Structure sometimes used by Agents and Groups to identify them.
AccountBuilder: A Type that knows how to construct an Account.
Activity: Structure making up “this” in “I did this”; it is something with which an Actor interacted. It can be a unit of instruction, experience, or performance that is to be tracked in meaningful combination with a Verb.
ActivityBuilder: A Type that knows how to construct an Activity.
ActivityDefinition: Structure that provides additional information (metadata) related to an Activity.
ActivityDefinitionBuilder: A Type that knows how to construct an ActivityDefinition
Agent: Structure that provides combined information about an individual derived from an outside service, such as a Directory Service.
AgentBuilder: A Type that knows how to construct an Agent.
Aggregates: Structure used when computing SQL Aggregates suitable for use by a client’s pagination mechanism.
Attachment: Structure representing an important piece of data that is part of a Learning Record. Could be an essay, a video, etc…
AttachmentBuilder: A Type that knows how to construct an Attachment.
CIString: A Type that effectively wraps a UniCase type to allow + facilitate using case-insensitive strings including serializing and deserializing them to/from JSON.
Config: A structure that provides the current configuration settings.
Context: Structure that gives a Statement more meaning like a team the Actor is working with, or the altitude at which a scenario was attempted in a flight simulator exercise.
ContextActivities: Map of types of learning activity context that a Statement is related to, represented as a structure (rather than the usual map).
ContextActivitiesBuilder: A Type that knows how to construct a ContextActivities.
ContextAgent: Structure for capturing a relationship between a Statement and one or more Agent(s) –besides the Actor– in order to properly describe an experience.
ContextAgentBuilder: A Type that knows how to construct a ContextAgent.
ContextBuilder: A Type that knows how to construct a Context.
ContextGroup: Similar to ContextAgent this structure captures a relationship between a Statement and one or more Group(s) –besides the Actor– in order to properly describe an experience.
ContextGroupBuilder: A Type that knows how to construct a ContextGroup.
Extensions: Extensions are available as part of Activity Definitions, as part of a Statement’s context or result properties. In each case, they’re intended to provide a natural way to extend those properties for some specialized use.
Format: Structure that combines a Statement resource GET request format parameter along w/ the request’s Accept-Language, potentially empty, list of user-preferred language-tags, in descending order of preference. This is provided to facilitate reducing types to their canonical form when required by higher layer APIs.
Group: Structure that represents a group of Agents.
GroupBuilder: A Type that knows how to construct a Group.
InteractionComponent: Depending on the value of the interactionType property of an ActivityDefinition, an Activity can provide additional properties, each potentially being a list of InteractionComponents.
InteractionComponentBuilder: A Type that knows how to construct an InteractionComponent.
LanguageMap: A dictionary where the key is an RFC 5646 Language Tag, and the value is a string in the language indicated by the tag. This map is supposed to be populated as fully as possible.
MyDuration: Implementation of time duration that wraps Duration to better satisfy the requirements of the xAPI specifications.
MyEmailAddress: Implementation of Email-Address that wraps EmailAddress to better satisfy the requirements of xAPI while reducing the verbosity making the mandatory mailto: scheme prefix optional.
MyLanguageTag: A wrapper around LanguageTag to use it when Option<T>, serialization and deserialization are needed.
MyTimestamp: Own structure to enforce xAPI requirements for timestamps.
MyVersion: Type for serializing/deserializing xAPI Version strings w/ relaxed parsing rules to allow missing ‘patch’ or even ‘minor’ numbers.
Person: Structure used in response to a GET` Agents Resource request. It provides aggregated information about one Agent.
PersonBuilder: A Type that knows how to construct a Person.
Score: Structure capturing the outcome of a graded Activity achieved by an Actor.
ScoreBuilder: A Type that knows how to construct a Score.
Statement: Structure showing evidence of any sort of experience or event to be tracked in xAPI as a Learning Record.
StatementBuilder: A Type that knows how to construct Statement.
StatementIDs: A structure consisting of an array of Statement Identifiers (UUIDs) an LRS may return as part of a response to certain requests.
StatementRef: Structure containing the UUID (Universally Unique Identifier) of a Statement referenced as the object of another.
StatementRefBuilder: A Type that knows how to construct a StatementRef.
StatementResult: Structure that contains zero, one, or more Statements.
SubStatement: Alternative representation of a Statement when referenced as the object of another.
SubStatementBuilder: A Type that knows how to construct a SubStatement.
User: Representation of a user that is subject to authentication and authorization.
Verb: Structure consisting of an IRI (Internationalized Resource Identifier) and a set of labels corresponding to multiple languages or dialects which provide human-readable meanings of the Verb.
VerbBuilder: A Builder that knows how to construct a Verb.
VerbUI: Simplified Verb representation.
XResult: Structure capturing a quantifiable xAPI outcome.
XResultBuilder: A Type that knows how to construct an xAPI Result.

Enums§

Actor: Representation of an individual (Agent) or group (Group) (a) referenced in a Statement involved in an action within an Activity or (b) is the authority asserting the truthfulness of Statements.
DataError: Enumeration of different error types raised by methods in the data module.
FormatParam: Possible variants for format used to represent the StatementResult desired response to a GET Statement resource.
InteractionType: Enumeration used in ActivityDefinitions.
Mode: Modes of operations of this LRS.
MyError: Enumeration of different error types raised by this crate.
ObjectType: Objects (Activity, Agent, Group, StatementRef, etc…) in xAPI vary widely in use and can share similar properties making it hard, and sometimes impossible, to know or decide which type is meant except for the presence of a variant of this enumeration in those objects’ JSON representation.
Role: Authorization role variants.
StatementObject: Enumeration representing the subject (or target) of an action (a Verb) carried out by an Actor (an Agent or a Group) captured in a Statement.
SubStatementObject: Enumeration for a potential Object of a Statement itself being the Object of another; i.e. the designated variant here is the Object of the Statement referenced in a sub-statement variant.
ValidationError: An error that denotes a validation constraint violation.
Vocabulary: Enumeration of ADL Verbs referenced in xAPI.

Constants§

CONSISTENT_THRU_HDR: The xAPI specific X-Experience-API-Consistent-Through HTTP header name.
CONTENT_TRANSFER_ENCODING_HDR: The Content-Transfer-Encoding HTTP header name.
EMPTY_EXTENSIONS: The empty Extensions singleton.
EMPTY_LANGUAGE_MAP: The empty LanguageMap singleton.
EXT_STATS: Statistics/Metrics Extension IRI
EXT_USERS: User Management Extension IRI
EXT_VERBS: Verbs Extension IRI
HASH_HDR: The xAPI specific X-Experience-API-Hash HTTP header name.
SIGNATURE_CT: Mandated ‘contentType’ to use when an Attachment is a JWS signature.
SIGNATURE_UT: Mandated ‘usageTpe’ to use when an Attachment is a JWS signature.
STATS_EXT_BASE: Statistics Extension base URI.
TEST_USER_PLAIN_TOKEN: The pre base-64 encoded input for generating test user credentials and populating HTTP Authorization header.
USERS_EXT_BASE: Users Extension base URI.
V200: The xAPI version this project supports by default.
VERBS_EXT_BASE: Vebrs Extension base URI.
VERSION_HDR: The xAPI specific X-Experience-API-Version HTTP header name.

Traits§

Canonical: Trait implemented by types that can produce a canonical form of their instances.
Fingerprint: To assert Equivalence of two instances of an xAPI Data Type we rely on this Trait to help us compute a fingerprint for each instance. The computation of this fingerprint uses a recursive descent mechanism not unlike what the standard Hash does.
MultiLingual: xAPI refers to a Language Map as a dictionary of words or sentences keyed by the RFC 5646: “Tags for Identifying Languages”.
Validate: xAPI mandates certain constraints on the values of some properties of types it defines. Our API binding structures however limit the Rust type of almost all fields to be Strings or derivative types based on Strings. This is to allow deserializing all types from the wire even when their values violate those constraints.

Functions§

adl_verb: Return a Verb identified by its Vocabulary variant.
build: Entry point for constructing a Local Rocket and use it for either testing or not. When testing is TRUE a mock DB is injected otherwise it’s the real McKoy.
config: This LRS server configuration Singleton.

Crate xapi_rsCopy item path