Crate jsonld

Expand description

This crate is a Rust implementation of the JSON-LD data interchange format.

Linked Data (LD) is a World Wide Web Consortium (W3C) initiative built upon standard Web technologies to create an interrelated network of datasets across the Web. The JavaScript Object Notation (JSON) is a widely used, simple, unstructured data serialization format to describe data objects in a human readable way. JSON-LD brings these two technologies together, adding semantics to JSON to create a lightweight data serialization format that can organize data and help Web applications to inter-operate at a large scale.

§Status

Experimental. This crate is a fork of json-ld by Timothée Haudebourg, retargeted onto a different dependency stack (rdfx, jstrict, iri-rs) and extended for the needs of an NGSI-LD context broker: deriving expanded JSON-LD straight from Rust types (Expandable), compile-time vocabularies (vocab!), context memoization, and explicit conversions to and from serde_json / sonic-rs at the HTTP boundary.

The W3C JSON-LD API test suite covers the four algorithms implemented here, namely expansion, compaction, flattening and RDF serialization, and 1,152 of its 1,156 tests pass. The other four are skipped against open upstream issues; the repository README breaks the numbers down per manifest. fromRdf, HTML script extraction and HTTP content negotiation are not implemented. What is not settled is the API: names, error types and trait shapes are still moving, and no compatibility promise is made before 1.0. Pin an exact version.

The performance work behind this fork was aimed at that broker workload, and a changed performance profile is not the same as a faster one. Memoizing processed contexts and interning term keys spend memory and bookkeeping that only pay off when the same contexts and terms recur, so a workload that looks nothing like NGSI-LD can come out behind. Measure your own corpus rather than assuming this fork is the faster choice for it.

Two operational notes for untrusted input: the default hasher is fast but not hash-flooding resistant (see Hashing for alternatives), and processed-context caches are unbounded, so a stream of distinct contexts grows memory without bound.

§Usage

The entry point for this library is the JsonLdProcessor trait that provides an access to all the JSON-LD transformation algorithms (context processing, expansion, compaction, etc.). If you want to explore and/or transform ExpandedDocuments, you may also want to check out the Object type representing a JSON object.

§Expansion

If you want to expand a JSON-LD document, first describe the document to be expanded using either RemoteDocument or RemoteDocumentReference:

RemoteDocument wraps the JSON representation of the document alongside its remote URL.
RemoteDocumentReference may represent only an URL, letting some loader fetching the remote document by dereferencing the URL.

After that, you can simply use the JsonLdProcessor::expand function on the remote document.

§Example

use iri_rs::{iri, IriBuf};
use jsonld::{JsonLdProcessor, Options, RemoteDocument, syntax::{Value, Parse}};

// Create a "remote" document by parsing a file manually.
let input = RemoteDocument::new(
  // We use `IriBuf` as IRI type.
  Some(IriBuf::from(iri!("https://example.com/sample.jsonld"))),

  // Optional content type.
  Some("application/ld+json".parse().unwrap()),
   
  // Parse the file.
  Value::parse_str(r#"
    {
      "@context": {
        "name": "http://xmlns.com/foaf/0.1/name"
      },
      "@id": "https://www.rust-lang.org",
      "name": "Rust Programming Language"
    }"#).expect("unable to parse file").0
);

// Use `NoLoader` as we won't need to load any remote document.
let mut loader = jsonld::NoLoader;

// Expand the "remote" document.
let expanded = input
  .expand(&mut loader)
  .await
  .expect("expansion failed");

for object in expanded {
  if let Some(id) = object.id() {
    let name = object.as_node().unwrap()
      .get_any(&iri!("http://xmlns.com/foaf/0.1/name")).unwrap()
      .as_str().unwrap();

    println!("id: {id}");
    println!("name: {name}");
  }
}

Here is another example using RemoteDocumentReference.

use iri_rs::{iri, IriBuf};
use jsonld::{JsonLdProcessor, Options, RemoteDocumentReference};

let input = RemoteDocumentReference::iri(IriBuf::from(iri!("https://example.com/sample.jsonld")));

// Use `FsLoader` to redirect any URL starting with `https://example.com/` to
// the local `example` directory. No HTTP query.
let mut loader = jsonld::FsLoader::default();
loader.mount(IriBuf::from(iri!("https://example.com/")), "examples");

let expanded = input.expand(&mut loader)
  .await
  .expect("expansion failed");

Lastly, the same example replacing IriBuf with the lightweight rdfx::vocabulary::IriIndex type.

use rdfx::vocabulary::{IriVocabularyMut, IndexVocabulary};
use jsonld::Id;
use contextual::WithContext;
// Creates the vocabulary that will map each `rdfx::vocabulary::IriIndex`
// to an actual `IriBuf`.
let mut vocabulary: IndexVocabulary = IndexVocabulary::new();

let iri_index = vocabulary.insert(iri!("https://example.com/sample.jsonld"));
let input = RemoteDocumentReference::iri(iri_index);

// Use `FsLoader` to redirect any URL starting with `https://example.com/` to
// the local `example` directory. No HTTP query.
let mut loader = jsonld::FsLoader::default();
loader.mount(IriBuf::from(iri!("https://example.com/")), "examples");

let expanded = input
  .expand_with(&mut vocabulary, &mut loader)
  .await
  .expect("expansion failed");

// `foaf:name` property identifier.
let name_id = Id::iri(vocabulary.insert(iri!("http://xmlns.com/foaf/0.1/name")));

for object in expanded {
  if let Some(id) = object.id() {
    let name = object.as_node().unwrap()
      .get_any(&name_id).unwrap()
      .as_value().unwrap()
      .as_str().unwrap();

    println!("id: {}", id.with(&vocabulary));
    println!("name: {name}");
  }
}

§Compaction

The JSON-LD Compaction is a transformation that consists in applying a context to a given JSON-LD document reducing its size. There are two ways to get a compact JSON-LD document with this library depending on your starting point:

If you want to get a compact representation for an arbitrary remote document, simply use the JsonLdProcessor::compact (or JsonLdProcessor::compact_with) method.
Otherwise to compact an ExpandedDocument you can use the Compact::compact method.

§Example

Here is an example compacting an arbitrary RemoteDocumentReference using JsonLdProcessor::compact.

use iri_rs::{iri, IriBuf};
use jsonld::{JsonLdProcessor, Options, RemoteDocumentReference, RemoteContextReference, syntax::Print};

let input = RemoteDocumentReference::iri(IriBuf::from(iri!("https://example.com/sample.jsonld")));

let context = RemoteContextReference::iri(IriBuf::from(iri!("https://example.com/context.jsonld")));

// Use `FsLoader` to redirect any URL starting with `https://example.com/` to
// the local `example` directory. No HTTP query.
let mut loader = jsonld::FsLoader::default();
loader.mount(IriBuf::from(iri!("https://example.com/")), "examples");

let compact = input
  .compact(context, &mut loader)
  .await
  .expect("compaction failed");

println!("output: {}", compact.pretty_print());

§Flattening

The JSON-LD Flattening is a transformation that consists in moving nested nodes out. The result is a list of all the nodes declared in the document. There are two ways to flatten JSON-LD document with this library depending on your starting point:

If you want to get a compact representation for an arbitrary remote document, simply use the JsonLdProcessor::flatten (or JsonLdProcessor::flatten_with) method. This will return a JSON-LD document.
Otherwise to compact an ExpandedDocument you can use the Flatten::flatten (or Flatten::flatten_with) method. This will return the list of nodes as a FlattenedDocument.

Flattening requires assigning an identifier to nested anonymous nodes, which is why the flattening functions take an rdfx::LocalGenerator as parameter. This generator is in charge of creating new fresh identifiers (with their metadata). The most common generator is rdfx::generator::Blank that creates blank node identifiers.

§Example

Here is an example flattening an arbitrary RemoteDocumentReference using JsonLdProcessor::flatten.

use iri_rs::{iri, IriBuf};
use jsonld::{JsonLdProcessor, Options, RemoteDocumentReference, syntax::Print};

let input = RemoteDocumentReference::iri(IriBuf::from(iri!("https://example.com/sample.jsonld")));

// Use `FsLoader` to redirect any URL starting with `https://example.com/` to
// the local `example` directory. No HTTP query.
let mut loader = jsonld::FsLoader::default();
loader.mount(IriBuf::from(iri!("https://example.com/")), "examples");

let mut generator = rdfx::generator::Blank::new();

let nodes = input
  .flatten(&mut generator, &mut loader)
  .await
  .expect("flattening failed");

println!("output: {}", nodes.pretty_print());

§Interop with other JSON crates

This crate represents JSON values as jstrict::Value, whose object type preserves entry order and tolerates duplicate keys. The JSON-LD algorithms depend on both. Two feature flags convert to and from the common alternatives without manual unpacking:

Feature	Type it bridges to
`serde-json`	`serde_json::Value`
`sonic-rs`	`sonic_rs::Value`

§Input

RemoteDocument::from_serde_json and RemoteDocument::from_sonic_rs consume the respective value type directly. RemoteDocument::from_value is more general and accepts anything implementing Into<jstrict::Value>.

§Output

JsonLdProcessor::compact and JsonLdProcessor::flatten return a jstrict::Value; call its inherent into_serde_json / into_sonic_rs method.
ExpandedDocument gains into_serde_json_with / into_sonic_rs_with, and into_serde_json / into_sonic_rs for the default no-vocabulary case.

Note that a round trip through either type is lossy in one respect: neither serde_json::Map nor sonic_rs::Object keeps duplicate keys, and sonic_rs::Object does not preserve entry order.

§Example

use iri_rs::{iri, IriBuf};
use jsonld::{JsonLdProcessor, RemoteDocument};

let value = serde_json::json!({
    "@context": {"name": "http://xmlns.com/foaf/0.1/name"},
    "@id": "https://www.rust-lang.org",
    "name": "Rust Programming Language"
});

let input = RemoteDocument::from_serde_json(
    Some(IriBuf::from(iri!("https://example.com/sample.jsonld"))),
    Some("application/ld+json".parse().unwrap()),
    value,
);

let mut loader = jsonld::NoLoader;
let expanded = input.expand(&mut loader).await.expect("expansion failed");

let _: serde_json::Value = expanded.into_serde_json();

§Expanding your own types with `#[derive(Expandable)]`

The algorithms above all start from a JSON document. If the data starts out as Rust structs instead, the expandable feature lets a type render directly into expanded JSON-LD, skipping both the intermediate compact JSON and the expansion algorithm.

Annotate the struct with the IRIs its fields map to and derive Expandable:

use jsonld::Expandable;

#[derive(Expandable)]
#[jsonld(
    type = "https://schema.org/Person",
    prefix(schema = "https://schema.org/")
)]
struct Person {
    #[jsonld(id)]
    id: String,

    #[jsonld(property = "schema:name")]
    name: String,

    #[jsonld(property = "schema:age")]
    age: Option<u32>,
}

let person = Person {
    id: "https://example.com/#me".to_owned(),
    name: "Ada".to_owned(),
    age: Some(36),
};

let json: serde_json::Value = person.expand();

The derive is backend-agnostic: Expandable::expand is generic over the JsonValue trait, so the same type can render into serde_json::Value, sonic_rs::Value, or jstrict::Value depending on which backend feature is enabled, and expandable-chrono lets chrono temporal types take part without a newtype wrapper. Field attributes cover @type coercion, language maps, nesting, flattening and more; see Expandable for the full attribute reference. Generic structs are not supported yet.

§Compile-time vocabularies with `vocab!`

Repeatedly writing full IRIs is verbose and easy to get wrong. With the vocab feature, the vocab! macro reads one or more JSON-LD context files at compile time and emits a constant for every term they define, so a typo becomes a compile error rather than a silently wrong IRI:

mod vocab {
    jsonld::vocab! {
        contexts: ["contexts/schema.jsonld"]
    }
}

// An `iri_rs::Iri<&'static str>` resolved at compile time. Terms are split
// into `classes` (TitleCase names) and `properties` (lowerCase names).
assert_eq!(vocab::expanded::properties::NAME.as_str(), "https://schema.org/name");

The macro also emits the compact-to-expanded term mapping, letting you go from a term to its IRI without a runtime context lookup. See vocab! for the full syntax.

The split into classes and properties is a naming convention, the one NGSI-LD and most RDF vocabularies follow, rather than anything the JSON-LD specification mandates: a term whose first character is uppercase becomes a class, everything else becomes a property. Contexts that name things some other way still get a constant per term and complete lookup tables, but the module names stop describing their contents. Casing matters more than that in one place: constant names are the term in SHOUTY_SNAKE_CASE, so createdAt and created_at both want CREATED_AT and collide, which is an error rather than generated code. A context consistent about one style is fine, one that mixes both for the same concept is not. See vocab! for the details.

The constants are iri-rs types built by that crate’s iri! macro, which resolves iri-rs against your crate’s manifest as it expands. So vocab! needs iri-rs among your own dependencies, and a re-exported path cannot stand in for it. A plain iri-rs = "3" is enough: the vocab feature turns on the static feature that iri! lives behind.

§Feature flags

No feature other than bytewise-iri is enabled by default.

§IRI comparison

Feature	Effect
`bytewise-iri` (default)	Forwarded to `iri-rs`, where it makes `Iri`/`IriRef` compare, hash and order byte-wise rather than by RFC 3987 normalization.

This matches how JSON-LD produces IRIs. The IRI Expansion algorithm resolves against the base IRI using only the basic algorithm of RFC 3986 §5.2, and states that “neither Syntax-Based Normalization nor Scheme-Based Normalization are performed”, so IRIs reach the algorithms in whatever form the document wrote them, and comparing them bytewise compares exactly what the specification produced. Byte comparison is also considerably cheaper.

Turning the feature off buys normalization-aware equality, where two spellings of the same IRI compare equal, at the cost of normalizing on every comparison. The W3C test suites pass either way.

§Hashing

The collections used internally are hashed with foldhash by default (via hashbrown), which is fast but not resistant to hash-flooding from untrusted input. Two features swap that out:

Feature	Effect
`ahash`	Use `ahash` as the default hasher.
`gxhash`	Use `gxhash` as the default hasher. Requires a CPU with AES intrinsics and fails to build without them.

§JSON interop

See Interop with other JSON crates.

Feature	Effect
`serde`	Implement `serde::Serialize` / `Deserialize` for the syntax and document types.
`serde-json`	Convert between `jstrict::Value` and `serde_json::Value`.
`sonic-rs`	Convert between `jstrict::Value` and `sonic_rs::Value`.

§Loading remote documents

Feature	Effect
`reqwest`	Provide `loader::ReqwestLoader`, an HTTP loader for remote contexts and documents. Requires a `tokio` runtime.

§Deriving expansion

Feature	Effect
`expandable`	Provide the `Expandable` derive macro and its `JsonValue` abstraction. Enables no JSON backend on its own.
`expandable-serde-json`	`expandable` plus a `JsonValue` implementation for `serde_json::Value`.
`expandable-jstrict`	`expandable` plus a `JsonValue` implementation for `jstrict::Value`.
`expandable-sonic-rs`	`expandable` plus a `JsonValue` implementation for `sonic_rs::Value`. Implies `sonic-rs`.
`expandable-chrono`	`expandable` plus rendering of `chrono` date and time types as their XSD lexical forms.

§Vocabularies

Feature	Effect
`vocab`	Provide the `vocab!` macro, which turns JSON-LD context files into compile-time IRI constants.

§Fast IRIs and Blank Node Identifiers

This library gives you the opportunity to use any datatype you want to represent IRIs an Blank Node Identifiers. Most types have them parameterized. To avoid unnecessary allocations and expensive comparisons, it is highly recommended to use a cheap, lightweight datatype such as rdfx::vocabulary::IriIndex. This type will represent each distinct IRI/blank node identifier with a unique index. In this case a rdfx::vocabulary::IndexVocabulary that maps each index back to its original IRI or blank node identifier can be passed to every function.

You can also use your own index type, with your own rdfx::vocabulary::Vocabulary implementation.

§Displaying vocabulary-dependent values

Since using vocabularies separates IRIs and Blank ids from their textual representation, it complicates displaying data using them. Fortunately many types defined by this crate implement the contextual::DisplayWithContext trait that allow displaying value with a “context”, which here would be the vocabulary. By importing the contextual::WithContext which provides the with method you can display such value like this:

use iri_rs::{iri, IriBuf};
use rdfx::vocabulary::{IriVocabularyMut, IndexVocabulary};
use contextual::WithContext;

let mut vocabulary: IndexVocabulary = IndexVocabulary::new();
let i = vocabulary.insert(iri!("https://docs.rs/contextual"));
let value = jsonld::Id::iri(i);

println!("{}", value.with(&vocabulary))

Re-exports§

pub use jsonld_compaction as compaction;
pub use jsonld_context_processing as context_processing;
pub use jsonld_expansion as expansion;
pub use jsonld_serialization as ser;
pub use jsonld_syntax as syntax;
pub use self as expandable_core;expandable
pub use self;

Modules§

context: Context processing algorithm and related types.
expanded: Expanded documents.
flattened: Flattened documents.
flattening: Flattening algorithm and related types.
id: Node identifiers, valid or not.
loader: Document loaders.
object: Nodes, lists and values.
print: Pretty-printing of JSON-LD documents. Pretty-printing of expanded documents through a vocabulary.
quad: RDF quads of an expanded document.
rdf: Conversion of expanded JSON-LD documents to RDF, and the well-known IRI constants the conversion needs.
utils: Utility functions.
warning: Warnings raised by the algorithms.

Macros§

vocabvocab: Generate JSON-LD vocabulary constants from one or more local context files.

Structs§

BlankId: Blank node identifier.
BlankIdBuf: Owned blank node identifier.
ChainLoader: ChainLoader: loads document from the first loader, otherwise falls back to the second one.
Context: Active context: everything the algorithms need to expand or compact against the current scope.
Document: JSON-LD document in both compact and expanded form.
Environment: Vocabulary, loader and warning handler an algorithm runs against.
ExpandedDocument: Result of the document expansion algorithm: a set of (indexed) objects.
FsLoader: File-system loader.
Indexed: Indexed objects.
InvalidIri
InvalidLangString: Raised when something tried to build a language string without language tag or direction.
Iri
IriRef
LangStr: Language string reference.
LangString: Language string.
LenientLangTag: Language tag that may not be well-formed.
LenientLangTagBuf: Owned language tag that may not be well-formed.
LoadError: Loading error: wraps the loader-specific error with the IRI we tried to load. Error raised while loading a remote document, with the URL that failed.
MissingLangQualifier: Raised when a LangString mutation would leave it without both a language tag and a direction.
NoLoader: Loader that never loads anything.
Node: Node object.
NodeParts: Every member of a node, owned.
Nodes: Iterator through indexed nodes.
Objects: Iterator through indexed objects.
Options: JSON-LD Processor options.
RemoteDocument: Remote document.
ReqwestLoaderreqwest: reqwest-based loader.
ToRdf: Expanded document paired with the vocabulary and generator used to turn it into RDF quads.
UnknownProcessingMode: Error raised when parsing an unknown processing-mode name.

Enums§

CompactError: Error that can be raised by the JsonLdProcessor::compact function.
Container: Container mapping of a term: the combination of @container values it was defined with.
ContainerKind: A single @container value.
ContextLoadError: Error raised while loading a remote context.
ContextRef: Either a borrowed context or an owned one held behind a box.
Direction: Internationalized string direction.
ExpandError: Error that can be raised by the JsonLdProcessor::expand function.
ExpandedJsonTextError: Error reading an expanded document from its JSON text.
ExtractContextError: Context extraction error.
FlattenError: Error that can be raised by the JsonLdProcessor::flatten function.
GeneratedIdError: Error returned by generator_next_id when the underlying generator produced a term that cannot be turned into a ValidId.
Id: Node identifier.
JsonLdError: Umbrella error covering every JSON-LD processing failure.
Nullable: Value that can be null.
Object: Object.
ProcessingMode: Processing mode.
Profile: Value of the profile parameter of the application/ld+json media type.
Ref: Id to a reference.
RemoteDocumentReference: Remote document, loaded or not.
SerializationError
StandardProfile: Standard profile parameter values defined for the application/ld+json.
Term: Identifier, keyword or @null.
ToRdfError: Error that can be raised by the JsonLdProcessor::to_rdf function.
Type: Object type.
ValidId: Valid node identifier (IRI or blank node).
Value: Value object.

Constants§

JSON_MEDIA_TYPE: application/json media type.
LD_JSON_MEDIA_TYPE: application/ld+json media type.

Traits§

Compact: Whole documents that can be compacted against a processed @context.
Dereference: Dereferencing a document through a Loader on sufficient native stack.
Expand: Document expansion.
Expandableexpandable: Convert &self into the expanded JSON-LD form for the chosen backend V.
ExpandableLanguageMapexpandable: Renders a field tagged #[jsonld(container = "language")].
ExpandableTypeValueexpandable: Field marker for dynamic @type arrays.
ExtractContext: Documents from which a JSON-LD context can be extracted.
Flatten: Documents that can be flattened into node objects.
IdentifyAll: Fragments whose anonymous nodes can all be given identifiers.
JsonLdProcessor: Application Programming Interface.
JsonValueexpandable: Minimal vocabulary for any JSON value type the derive can construct.
LdQuads: Documents whose node objects can be read as quads.
Loader: Document loader.
Print: Print methods.
Process: Contexts that can be processed into an active context.
RdfQuads: Documents that can be serialized as RDF quads.
Relabel: Fragments whose blank node identifiers can be renamed.
ToJsonValueexpandable: Convert a leaf field value to the chosen JsonValue backend.
TryFromJson: Try to convert from a JSON value directly into an expanded JSON-LD document without going through the expansion algorithm.

Functions§

generator_next_id: Produces a ValidId from the next term yielded by generator, interning it in vocabulary.

Type Aliases§

CompactResult: Result of the JsonLdProcessor::compact function.
CompareResult: Result of the JsonLdProcessor::compare function.
DefaultBuildHashergxhash: Hasher used by the collections of this crate.
DocumentParts: Remote document paired with its expanded form.
ExpandResult: Result returned by the JsonLdProcessor::expand function.
FlattenResult: Result of the JsonLdProcessor::flatten function.
FlattenedDocument: Result of the document flattening algorithm.
HashMap: Hash map using DefaultBuildHasher.
HashSet: Hash set using DefaultBuildHasher.
IndexMap: Insertion-ordered map using DefaultBuildHasher.
IndexSet: Insertion-ordered set using DefaultBuildHasher.
IndexedNode: Node object carrying an optional @index.
IndexedObject: Indexed object.
IntoDocumentResult: Result returned by the JsonLdProcessor::into_document function.
IriBuf
IriRefBuf
LoadingResult: Result of loading a remote document.
RemoteContext: Remote document holding a context.
RemoteContextReference: Reference to a remote context, either inline or by URL.
ToRdfResult: Error that can be raised by the JsonLdProcessor::to_rdf function.
ValidVocabularyId: Valid identifier drawn from the terms of a vocabulary.
VocabularyId: Identifier drawn from the terms of a vocabulary, valid or not.

Derive Macros§

Expandableexpandable: Derive Expandable to generate an expand::<V>() impl producing expanded JSON-LD for any V: JsonValue backend.

Crate jsonld

Crate jsonld Copy item path

§Status

§Usage

§Expansion

§Example

§Compaction

§Example

§Flattening

§Example

§Interop with other JSON crates

§Input

§Output

§Example

§Expanding your own types with #[derive(Expandable)]

§Compile-time vocabularies with vocab!

§Feature flags

§IRI comparison

§Hashing

§JSON interop

§Loading remote documents

§Deriving expansion

§Vocabularies

§Fast IRIs and Blank Node Identifiers

§Displaying vocabulary-dependent values

Re-exports§

Modules§

Macros§

Structs§

Enums§

Constants§

Traits§

Functions§

Type Aliases§

Derive Macros§

Crate jsonld

§Expanding your own types with `#[derive(Expandable)]`

§Compile-time vocabularies with `vocab!`