Crate savefile_abi

Source

Expand description

This is the documentation for savefile-abi

§Welcome to savefile-abi!

Savefile-abi is a crate that is primarily meant to help building binary plugins using Rust.

Note! Savefile-abi now supports methods returning boxed futures! See the chapter ‘async’ below.

§Example

Let’s say we have a crate that defines this trait for adding u32s:

InterfaceCrate

extern crate savefile_derive;
use savefile_derive::savefile_abi_exportable;

#[savefile_abi_exportable(version=0)]
pub trait AdderInterface {
    fn add(&self, x: u32, y: u32) -> u32;
}

Now, we want to implement addition in a different crate, compile it to a shared library (.dll or .so), and use it in the first crate (or some other crate):

ImplementationCrate

use savefile_derive::{savefile_abi_export};
#[derive(Default)]
struct MyAdder { }

impl AdderInterface for MyAdder {
    fn add(&self, x: u32, y: u32) -> u32 {
        x + y
    }
}

/// Export this implementation as the default-implementation for
/// the interface 'AdderInterface', for the current library.
savefile_abi_export!(MyAdder, AdderInterface);

We add the following to Cargo.toml in our implementation crate:

[lib]
crate-type = ["cdylib"]

Now, in our application, we add a dependency to InterfaceCrate, but not to ImplementationCrate.

We then load the implementation dynamically at runtime:

ApplicationCrate

use adder_interface::AdderInterface;
use savefile_abi::AbiConnection;


// Load the implementation of `dyn AdderInterface` that was published
// using the `savefile_abi_export!` above.
let connection = AbiConnection::<dyn AdderInterface>
        ::load_shared_library("ImplementationCrate.so").unwrap();

// The type `AbiConnection::<dyn AdderInterface>` implements
// the `AdderInterface`-trait, so we can use it to call its methods.
assert_eq!(connection.add(1, 2), 3);

§More advanced examples

Interface containing closure arguments:

#[savefile_abi_exportable(version=0)]
pub trait CallMeBack {
    fn call_me(&self, x: &dyn Fn(u32) -> u32) -> u32;
    fn call_me_mut(&self, x: &mut dyn FnMut(u32) -> u32) -> u32;
}

Interface containing more complex types:

#[savefile_abi_exportable(version=0)]
pub trait Processor {
    fn process(&self, x: &HashMap<String,String>, parameters: f32) -> BinaryHeap<u32>;
}

Interface containing user defined types:


#[derive(Savefile)]
pub struct MyCustomType {
    pub name: String,
    pub age: u8,
    pub length: f32,
}

#[savefile_abi_exportable(version=0)]
pub trait Processor {
    fn insert(&self, x: &MyCustomType) -> Result<u32, String>;
}

§Versioning

Let’s say the last example from the previous chapter needed to be evolved. The type now needs a ‘city’ field.

We can add this while retaining compatibility with clients expecting the old API:

extern crate savefile_derive;


#[derive(Savefile)]
pub struct MyCustomType {
    pub name: String,
    pub age: u8,
    pub length: f32,
    #[savefile_versions="1.."]
    pub city: String,
}

#[savefile_abi_exportable(version=1)]
pub trait Processor {
    fn insert(&self, x: &MyCustomType) -> Result<u32, String>;
}

#[cfg(test)]
{
    #[test]
    pub fn test_backward_compatibility() {
       // Automatically verify backward compatibility isn't broken.
       // Schemas for each version are stored in directory 'schemas',
       // and consulted on next run to ensure no change.
       // You should check the schemas in to source control.
       // If check fails for an unreleased version, just remove the schema file from
       // within 'schemas' directory.
       verify_compatiblity::<dyn Processor>("schemas").unwrap()
    }
}

Older clients, not aware of the ‘city’ field, can still call newer implementations. The ‘city’ field will receive an empty string (Default::default()). Newer clients, calling older implementations, will simply, automatically, omit the ‘city’ field.

§Background

Savefile-abi is a crate that is primarily meant to help building binary plugins using Rust.

The primary usecase is that a binary rust program is to be shipped to some customer, who should then be able to load various binary modules into the program at runtime. Savefile-abi defines ABI-stable rust-to-rust FFI for calling between a program and a runtime-loaded shared library.

For now, both the main program and the plugins need to be written in rust. They can, however, be written using different versions of the rust compiler, and the API may be allowed to evolve. That is, data structures can be modified, and methods can be added (or removed).

The reason savefile-abi is needed, is that rust does not have a stable ‘ABI’. This means that if shared libraries are built using rust, all libraries must be compiled by the same version of rust, using the exact same source code. This means that rust cannot, ‘out of the box’, support a binary plugin system, without something like savefile-abi. This restriction may be lifted in the future, which would make this crate (savefile-abi) mostly redundant.

Savefile-abi does not solve the general ‘stable ABI’-problem. Rather, it defines a limited set of features, which allows useful calls between shared libraries, without allowing any and all rust construct.

§Why another stable ABI-crate for Rust?

There are other crates also providing ABI-stability. Savefile-abi has the following properties:

It is able to completely specify the protocol used over the FFI-boundary. I.e, it can isolate two shared libraries completely, making minimal assumptions about data type memory layouts.
When it cannot prove that memory layouts are identical, it falls back to (fast) serialization. This has a performance penalty, and may require heap allocation.
It tries to require a minimum of configuration needed by the user, while still being safe.
It supports versioning of data structures (with a performance penalty).
It supports trait objects as arguments, including FnMut() and Fn().
Boxed trait objects, including Fn-traits, can be transferred across FFI-boundaries, passing ownership, safely. No unsafe code is needed by the user.
It requires enums to be #[repr(uX)] in order to pass them by reference. Other enums will still work correctly, but will be serialized under the hood at a performance penalty.
It places severe restrictions on types of arguments, since they must be serializable using the Savefile-crate for serialization. Basically, arguments must be ‘simple’, in that they must own all their contents, and be free of cycles. I.e, the type of the arguments must have lifetime &'static. Note, arguments may still be references, and the contents of the argument types may include Box, Vec etc, so this does not mean that only primitive types are supported.

Arguments cannot be mutable, since if serialization is needed, it would be impractical to detect and handle updates to arguments made by the callee. This said, arguments can still have types such as HashMap, IndexMap, Vec, String and custom defined structs or enums.

§How it all works

The basic principle is that savefile-abi makes sure to send function parameters in a way that is certain to be understood by the code on the other end of the FFI-boundary. It analyses if memory layouts of reference-parameters are the same on both sides of the FFI-boundary, and if they are, the references are simply copied. In all other cases, including all non-reference parameters, the data is simply serialized and sent as a binary buffer.

The callee cannot rely on any particular lifetimes of arguments, since if the arguments were serialized, the arguments the callee sees will only have a lifetime of a single call, regardless of the original lifetime. Savefile-abi inspects all lifetimes and ensures that reference parameters don’t have non-default lifetimes. Argument types must have static lifetimes (otherwise they can’t be serialized). The only exception is that the argument can be reference types, but the type referenced must itself be &'static.

§About Safety

Savefile-Abi uses copious amounts of unsafe code. It has a test suite, and the test suite passes with miri.

One thing to be aware of is that, at present, the AbiConnection::load_shared_library-method is not marked as unsafe. However, if the .so-file given as argument is corrupt, using this method can cause any amount of UB. Thus, it could be argued that it should be marked unsafe.

However, the same is true for any shared library used by a rust program, including the system C-library. It is also true that rust programs rely on the rust compiler being implemented correctly. Thus, it has been judged that the issue of corrupt binary files is beyond the scope of safety for Savefile-Abi.

As long as the shared library is a real Savefile-Abi shared library, it should be sound to use, even if it contains code that is completely incompatible. This will be detected at runtime, and either AbiConnection::load_shared_library will panic, or any calls made after will panic.

§About Vec and String references

Savefile-Abi allows passing references containing Vec and/or String across the FFI-boundary. This is not normally guaranteed to be sound. However, Savefile-Abi uses heuristics to determine the actual memory layout of both Vec and String, and verifies that the two libraries agree on the layout. If they do not, the data is serialized instead. Also, since parameters can never be mutable in Savefile-abi (except for closures), we know the callee is not going to be freeing something allocated by the caller. Parameters called by value are always serialized.

§Async

Savefile-abi now supports methods returning futures:


use savefile_derive::savefile_abi_exportable;
use std::pin::Pin;
use std::future::Future;
use std::time::Duration;

#[savefile_abi_exportable(version = 0)]
pub trait BoxedAsyncInterface {
    fn add_async(&mut self, x: u32, y: u32) -> Pin<Box<dyn Future<Output=String>>>;

}

struct SimpleImpl;

impl BoxedAsyncInterface for SimpleImpl {
    fn add_async(&mut self, x: u32, y: u32) -> Pin<Box<dyn Future<Output=String>>> {
        Box::pin(
            async move {
                /* any async code, using .await */
                format!("{}",x+y)
            }
        )
    }
}

It also supports the #[async_trait] proc macro crate. Use it like this:

use async_trait::async_trait;
use savefile_derive::savefile_abi_exportable;
use std::time::Duration;

#[async_trait]
#[savefile_abi_exportable(version = 0)]
pub trait SimpleAsyncInterface {
    async fn add_async(&mut self, x: u32, y: u32) -> u32;
}

struct SimpleImpl;

#[async_trait]
impl SimpleAsyncInterface for SimpleImpl {
    async fn add_async(&mut self, x: u32, y: u32) -> u32 {
        /* any async code, using .await */
        x + y
    }
}

Structs§

AbiConnection
Information about an ABI-connection.
AbiConnectionMethod
Describes a method in a trait
AbiConnectionTemplate
Information about an entry point and the trait it corresponds to.
AbiErrorMsg
Helper struct carrying a pointer and length to an utf8 message. We use this instead of &str, to guard against the hypothetical possibility that the layout of &str would ever change.
PackagedTraitObject
A trait object together with its entry point
TraitObject
Type erased carrier of a dyn trait fat pointer

Enums§

AbiProtocol
This struct carries all information between different libraries. I.e, it is the sole carrier of information accross an FFI-boundary.
Owning
Helper to determine if something is owned, or not
RawAbiCallResult
The result of calling a method in a foreign library.

Traits§

AbiExportable
This trait is meant to be exported for a ‘dyn SomeTrait’. It can be automatically implemented by using the macro #[savefile_abi_exportable(version=0)] on a trait that is to be exportable.
AbiExportableImplementation
Trait that is to be implemented for the implementation of a trait whose dyn trait type implements AbiExportable.

Functions§

abi_entry^⚠
Helper implementation of ABI entry point. The actual low level extern "C" functions call into this. This is an entry point meant to be used by the derive macro.
abi_entry_light^⚠
Helper implementation of ABI entry point. The actual low level extern "C" functions call into this. This is an entry point meant to be used by the derive macro.
parse_return_boxed_trait
Parse an RawAbiCallResult instance into a Result<Box<dyn T>, SavefileError> .
parse_return_value_impl
Parse the given RawAbiCallResult. If it concerns a success, then deserialize a return value using the given closure.
verify_compatiblity
Verify compatibility with old versions.

Crate savefile_abiCopy item path