<div align="center">
<a href="https://metacall.io" target="_blank"><img src="https://raw.githubusercontent.com/metacall/core/master/deploy/images/logo.png" alt="M E T A C A L L" style="max-width:100%; margin: 0 auto;" width="80" height="80">
<p><b>M E T A C A L L</b></p></a>
<p>A library for providing inter-language foreign function interface calls</p>
</div>
# Abstract
**METACALL** is a library that allows calling functions, methods or procedures between programming languages. With **METACALL** you can transparently execute code from/to any programming language, e.g. by calling a Python function from NodeJS:
`sum.py`
```python
def sum(a, b):
return a + b
```
`main.js`
```javascript
const { sum } = require("sum.py");
sum(3, 4); // 7
```
Use the [installer](https://github.com/metacall/install) and try [some examples](#43-examples).
<div align="center">
<a href="https://medium.com/@metacall/call-functions-methods-or-procedures-between-programming-languages-with-metacall-58cfece35d7" target="_blank"><img src="https://raw.githubusercontent.com/metacall/core/master/deploy/images/overview.png" alt="M E T A C A L L" style="max-width:100%; margin: 0 auto;" width="350" height="auto"></a>
</div>
# Table of Contents
<!-- TOC -->
- [Abstract](#abstract)
- [Table of Contents](#table-of-contents)
- [1. Motivation](#1-motivation)
- [2. Language Support](#2-language-support)
- [2.1 Loaders (Backends)](#21-loaders-backends)
- [2.2 Ports (Frontends)](#22-ports-frontends)
- [3. Use Cases](#3-use-cases)
- [3.1 Known Projects Using MetaCall](#31-known-projects-using-metacall)
- [4. Usage](#4-usage)
- [4.1 Installation](#41-installation)
- [4.2 Environment Variables](#42-environment-variables)
- [4.3 Examples](#43-examples)
- [5. Architecture](#5-architecture)
- [5.1 Overview](#51-overview)
- [5.1.1 Design Decisions](#511-design-decisions)
- [5.1.2 Modules](#512-modules)
- [5.2 Reflect](#52-reflect)
- [5.2.1 Type System](#521-type-system)
- [5.2.2 Values](#522-values)
- [5.2.3 Functions](#523-functions)
- [5.3 Plugins](#53-plugins)
- [5.3.1 Loaders](#531-loaders)
- [5.3.1.1 Python](#5311-python)
- [5.3.1.2 NodeJS](#5312-nodejs)
- [5.3.1.3 JavaScript](#5313-javascript)
- [5.3.1.4 C#](#5314-c)
- [5.3.1.5 Ruby](#5315-ruby)
- [5.3.1.6 Mock](#5316-mock)
- [5.3.1.7 File](#5317-file)
- [5.3.2 Serials](#532-serials)
- [5.3.2.1 MetaCall](#5321-metacall)
- [5.3.2.2 RapidJSON](#5322-rapidjson)
- [5.3.3 Detours](#533-detours)
- [5.3.3.1 PLTHook](#5331-plthook)
- [5.4 Ports](#54-ports)
- [5.4.1 Responsibilities](#541-responsibilities)
- [5.4.2 Port Implementations](#542-port-implementations)
- [5.4.3 Representative Examples](#543-representative-examples)
- [5.4.4 Lifecycle](#544-lifecycle)
- [5.4.5 Limitations and Future Work](#545-limitations-and-future-work)
- [5.5 Serialization](#55-serialization)
- [5.6 Memory Layout](#56-memory-layout)
- [5.7 Fork Model](#57-fork-model)
- [5.8 Threading Model](#58-threading-model)
- [6. Application Programming Interface (API)](#6-application-programming-interface-api)
- [7. Build System](#7-build-system)
- [7.1 Build Options](#71-build-options)
- [7.2 Coverage](#72-coverage)
- [7.3 Debugging](#73-debugging)
- [7.4 Build using Scripts](#74-build-using-scripts)
- [7.5 Build on Cloud - Gitpod](#75-build-on-cloud---gitpod)
- [8. Platform Support](#8-platform-support)
- [8.1 Docker Support](#81-docker-support)
- [8.1.1 Docker Development](#811-docker-development)
- [8.1.2 Docker Testing](#812-docker-testing)
- [8.1.3 Docker MSan Development](#813-docker-msan-development)
- [9. Benchmarks](#9-benchmarks)
- [10. License](#10-license)
<!-- /TOC -->
## 1. Motivation
The **METACALL** project started a long time ago when I was coding a [Game Engine for an MMORPG](https://bitbucket.org/parrastudios/argentum-online-c). My idea was to provide an interface to allow other programmers to extend the Game Engine easily. By that time, I was finishing university so I decided to do my [Final Thesis](https://bitbucket.org/parrastudios/argentum-online-c/raw/e6e78fef80c6adc541640d68d422721ef735184f/common/doc/Plugin/plugin-framework-paper.pdf) and [Presentation](https://bitbucket.org/parrastudios/argentum-online-c/raw/e6e78fef80c6adc541640d68d422721ef735184f/common/doc/Plugin/plugin-framework-presentation.pdf) based on the plug-in system for my Game Engine. The Plugin Architecture designed for the Game Engine has similarities with **METACALL** although the architecture has been redefined and the code has been rewritten from scratch. After some refinement of the system, I came up with **METACALL** and other use cases for the tool. Currently we are using **METACALL** to build a cutting edge FaaS (Function as a Service) **[https://metacall.io](https://metacall.io/)** based on this technique to provide high scalability of the functions among multiple cores and **[Function Mesh](https://medium.com/@metacall/function-mesh-architecture-c0304ba4bad0)** pattern, a new technique I have developed to transparently interconnect functions in a distributed system based on this library.
## 2. Language Support
This section describes all programming languages that **METACALL** supports. **METACALL** is offered through a C API. This means you can use it as a library to embed different runtimes into C. The **[Loaders](#21-loaders-backends)** are the ones that allow to call different functions from C. They are plugins (libraries) that **METACALL** loads and they have a common interface. They usually implement JITs, VMs or interpreters. On the other hand we have the **[Ports](#22-ports-frontends)** which are wrappers to the **METACALL** C API that expose the API to other languages. With the Python Loader we can execute calls to Python from C. With the Python Port we can install **METACALL** via `pip` and use it to call other languages from Python. The combination of both provides the opportunity for complete interoperability between virtually any two languages.
### 2.1 Loaders (Backends)
This section describes all programming languages that **METACALL** allows to load and invoke from C language, in other words all languages that **METACALL** can embed. If you are interested in design and implementation details of the loaders, please go to [loaders section](#531-loaders).
- Currently supported languages and run-times:
| Language | Runtime | Version | Tag |
| ------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------- | :-------------------------------------: | :--: |
| [Python](https://www.python.org/) | [Python C API](https://docs.python.org/3/c-api/intro.html) | **>= 3.2 <= 3.9** | py |
| [NodeJS](https://nodejs.org/) | [N API](https://nodejs.org/api/n-api.html) | **>= 10.22.0 <= 17.x.x** | node |
| [TypeScript](https://www.typescriptlang.org/) | [TypeScript Language Service API](https://github.com/microsoft/TypeScript/wiki/Using-the-Language-Service-API) | **4.2.3** | ts |
| [JavaScript](https://developer.mozilla.org/bm/docs/Web/JavaScript) | [V8](https://v8.dev/) | **5.1.117** | js |
| [C#](https://dotnet.microsoft.com/) | [NetCore](https://github.com/dotnet/docs/blob/master/docs/core/tutorials/netcore-hosting.md) | **>= 1.0.0-preview2 <= 7.0.4** | cs |
| [Ruby](https://ruby-lang.org/) | [Ruby C API](https://silverhammermba.github.io/emberb/c/) | **>= 2.1 <= 2.7** | rb |
| [Cobol](https://sourceforge.net/projects/open-cobol/) | [GNU/Cobol](https://open-cobol.sourceforge.io/doxygen/gnucobol-2/libcob_8h.html) | **>= 1.1.0** | cob |
| [File](/source/loaders/file_loader) | **∅** | **0.1.0** | file |
| [Mock](/source/loaders/mock_loader) | **∅** | **0.1.0** | mock |
| [RPC](https://en.wikipedia.org/wiki/Remote_procedure_call) | [cURL](https://curl.haxx.se/) | **>=7.64.0** | rpc |
| [Java](https://www.java.com) | [JVM](https://en.wikipedia.org/wiki/Java_virtual_machine) | **>=11** | java |
| [WebAssembly](https://webassembly.org/) | [Wasmtime](https://github.com/bytecodealliance/wasmtime) | **>= 0.27 <= 8.0.1** | wasm |
| [C](<https://en.wikipedia.org/wiki/C_(programming_language)>) | [libclang](https://clang.llvm.org/doxygen/group__CINDEX.html) - [Tiny C Compiler](https://bellard.org/tcc/) - [libffi](http://sourceware.org/libffi/) | **>=12** - **>=2021-10-30** - **>=3.2** | c |
| [Rust](https://www.rust-lang.org/) | [rustc](https://doc.rust-lang.org/rustc/what-is-rustc.html) - [libffi](http://sourceware.org/libffi/) | **nightly-2026-01-15** | rs |
- Languages and run-times under construction:
| Language | Runtime | Tag |
| ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ | :--: |
| [C++](http://www.cplusplus.com/) | [Clang](https://clang.llvm.org/) - [LLVM](https://llvm.org/) | cpp |
| [PHP](https://php.net/) | [Zend](https://www.php.net/manual/en/internals2.ze1.zendapi.php) | php |
| [Go](https://golang.org/) | Go Runtime | go |
| [Haskell](https://www.haskell.org/) | [Haskell FFI](https://wiki.haskell.org/GHC/Using_the_FFI) | hs |
| [Crystal](https://crystal-lang.org/) | [Crystal Compiler Internals](https://github.com/crystal-lang/crystal/wiki/Compiler-internals) | cr |
| [JavaScript](https://developer.mozilla.org/bm/docs/Web/JavaScript) | [SpiderMonkey](https://spidermonkey.dev/) | jsm |
| [Dart](https://dart.dev/) | [Dart VM](https://dart.dev/tools/dart-vm) | dart |
| [LuaJIT](https://luajit.org/) | [LuaJIT2](https://github.com/openresty/luajit2) | lua |
| [LLVM IR](https://www.llvm.org/devmtg/2017-06/1-Davis-Chisnall-LLVM-2017.pdf) | [LLVM](https://llvm.org/) | llvm |
| [Julia](https://julialang.org/) | [Julia Runtime](https://docs.julialang.org/en/v1/devdocs/init/) | jl |
### 2.2 Ports (Frontends)
Ports are the frontends to the **METACALL C API** from other languages. They allow to use **METACALL** from different languages. If you are interested in design and implementation details of the ports, please go to [ports section](#54-ports).
- Currently supported languages and run-times:
| Language | Runtime | Version |
| ------------------------------------------------------------------ | ---------------------------------------------------------- | :-------------------: |
| [Python](https://www.python.org/) | [Python C API](https://docs.python.org/3/c-api/intro.html) | **3.x** |
| [NodeJS](https://nodejs.org/) | [N API](https://nodejs.org/api/n-api.html) | **>= 8.11.1** |
| [JavaScript](https://developer.mozilla.org/bm/docs/Web/JavaScript) | [D8 (V8)](https://v8.dev/docs/d8) | **5.1.117** |
| [C#](https://dotnet.microsoft.com/) | [NetCore](https://github.com/dotnet/core) | **>= 1.0.0-preview2** |
| [Ruby](https://ruby-lang.org/) | [Ruby C API](https://silverhammermba.github.io/emberb/c/) | **2.x** |
| [Go](https://golang.org/) | [CGO](https://golang.org/cmd/cgo/) | **1.x** |
| [D](https://dlang.org/) | [DMD](https://wiki.dlang.org/DMD) | **2.x** |
| [Rust](https://www.rust-lang.org/) | **∅** | **>= 1.47.0** |
| [Scala](https://www.scala-lang.org/) | [JVM](https://en.wikipedia.org/wiki/Java_virtual_machine) | **>= 2.13.x** |
| [Nim](https://nim-lang.org/) | **∅** | **>= 1.4.2** |
## 3. Use Cases
**METACALL** can be used in the following cases:
- Interconnect different technologies in the same project. It allows heterogeneous teams of developers to work on the same project in an isolated way and use different programming languages at the same time.
- Embedding programming languages in existing software. Game Engines and 3D Editors like [Blender](https://www.blender.org/), among others can take benefit of **METACALL** and extend the core functionality with higher level programming languages (aka scripting).
- Function as a Service. **METACALL** can be used to implement efficient FaaS architectures. We are using it to implement our own FaaS (Function as a Service) **[https://metacall.io](https://metacall.io/)** based on **[Function Mesh](https://medium.com/@metacall/function-mesh-architecture-c0304ba4bad0)** pattern and high performance function scalability thanks to this library.
- Source code migrations. **METACALL** can wrap large and legacy codebases, and provide an agnostic way to work with the codebase in a new programming language. Eventually the code can be migrated in parts, without needing to create a new project or stop the production environment. Incremental changes can be made, solving the migration easily and with less time and effort.
- Porting low level libraries to high level languages transparently. With **METACALL** you can get rid of extension APIs like Python C API or NodeJS N-API. You can call low level libraries directly from your high level languages without making a wrapper in C or C++ for it.
As you can see, there are plenty of uses. **METACALL** introduces a new model of programming which allows high interoperability between technologies. If you find any other use case just let us know about it with a Pull Request and we will add it to the list.
## 3.1 Known Projects Using MetaCall
- **[Acid Cam](https://www.facebook.com/AcidCam/)**: A software for video manipulation that distorts videos for generating art by means of OpenCV. [Acid Cam CLI](https://github.com/lostjared/acidcam-cli) uses **METACALL** to allow custom filters written in Python and easily embed Python programming language into its plugin system.
- **[Pragma](https://pragmalang.com/)**: Pragma is a language for building beautiful and extensible GraphQL APIs in no time. Within a single file, you can define your data models and authorization rules (roles and permissions), and import serverless functions for data validation, transformation, authorization, or any custom logic. [Pragma](https://github.com/pragmalang/pragma) uses **METACALL** to import and execute functions into the language so it can be extended with other programming languages.
## 4. Usage
## 4.1 Installation
Before trying any of the examples, you must have **METACALL** installed in your system. To install **METACALL** you have the following options:
- [Install precompiled tarball via shell script (downloads the tarball generated by Guix)](https://github.com/metacall/install).
- [Build and install it manually](#6-build-system).
- [Pull it from DockerHub](https://hub.docker.com/r/metacall/core).
- [Install via Guix package manager](https://github.com/metacall/distributable/blob/master/source/metacall.scm) (needs to fix the commit of [Guix channels](https://github.com/metacall/distributable/blob/master/channels/channels.scm)).
- [Download the precompiled tarball from Guix via Distributable Releases Assets](https://github.com/metacall/distributable/releases).
- [ArchLinux AUR](https://github.com/metacall/aur).
- [Homebrew](https://github.com/metacall/homebrew).
- [Download precompiled tarball (.tar.gz) or Debian (.deb) / RPM (.rpm) installers via Core Releases Assets](https://github.com/metacall/core/releases).
### 4.2 Environment Variables
The environment variables are optional, in case you want to modify default paths of **METACALL**.
| Name | Description | Default Value |
| :-----------------------: | ---------------------------------------------------------------- | :------------------------------: |
| **`DETOUR_LIBRARY_PATH`** | Directory where detour plugins to be loaded are located | **`detours`** |
| **`SERIAL_LIBRARY_PATH`** | Directory where serial plugins to be loaded are located | **`serials`** |
| **`CONFIGURATION_PATH`** | File path where the **METACALL** global configuration is located | **`configurations/global.json`** |
| **`LOADER_LIBRARY_PATH`** | Directory where loader plugins to be loaded are located | **`loaders`** |
| **`LOADER_SCRIPT_PATH`** | Directory where scripts to be loaded are located | **`${execution_path}`** ¹ |
¹ **`${execution_path}`** defines the path where the program is executed, **`.`** in Linux.
### 4.3 Examples
- [BeautifulSoup from Express](https://github.com/metacall/beautifulsoup-express-example): This example shows how to use [**METACALL** CLI](/source/cli/metacallcli) for building a **Polyglot Scraping API** that mixes NodeJS with Python.
- [Higher Order Functions with Python & NodeJS](https://github.com/metacall/fn.py-javascript-example): An example of using [Fn.py](https://github.com/kachayev/fn.py) (Python) from JavaScript (NodeJS).
- [Embedding NodeJS](https://github.com/metacall/embedding-nodejs-example): Example application for embedding NodeJS code into C/C++ using CMake as a build system.
- [Embedding Python](https://github.com/metacall/embedding-python-example): Example application for embedding Python code into C/C++ using CMake as a build system.
- [Embedding Ruby](https://github.com/metacall/embedding-ruby-example): Example application for embedding Ruby code into C/C++ using CMake as a build system.
- [Mixing Go and TypeScript](https://github.com/metacall/golang-typescript-example): This example shows how to embed TypeScript into Go using METACALL. In other words, calling TypeScript functions from Go.
- [Using `matplotlib` from C/C++](https://github.com/metacall/embedding-matplotlib-example): Example application for using Python `matplotlib` library into C/C++ using `gcc` for compiling it and installing **METACALL** by compiling it by hand.
- [Polyglot Redis Module](https://github.com/metacall/redis-module): Extend Redis DataBase modules with TypeScript, JavaScript, Python, C#, Ruby...
- [Rotulin](https://github.com/metacall/rotulin): Example of a multi-language application built with **METACALL**. This application embeds a Django server with a Ruby DataBase and C# business layer based on ImageMagick.
## 5. Architecture
### 5.1 Overview
#### 5.1.1 Design Decisions
- To provide a high level API with a simple UX and to be easy to understand.
- To work in high performance environments.
- To be as cross-platform as possible.
- To avoid modifying run-times directly or using the code inside **METACALL** to avoid maintaining them, or propagating security flaws or licenses into **METACALL**.
- To provide support for any embeddable programming language and to provide support for **METACALL** to be used from any programming language.
- All external code used in **METACALL** must be introduced by inversion of control in the plugin system so that the core must not remain aware of what software is used.
- All code developed in **METACALL** must be implemented in standalone libraries that can work by themselves in an isolated way (aka modules).
#### 5.1.2 Modules
- [`adt`](/source/adt) provides a base for Abstract Data Types and algorithms used in **METACALL**. Implementation must be done in an efficient and generic way. Some of the data structures implemented are vector, set, hash, comparable or trie.
- [`detour`](/source/detour) provides an interface to hook into functions. Detours are used by the [fork model](#57-fork-model) to intercept fork calls.
- [`detours`](/source/detours) implement the [`detour`](/source/detour) interface by using a plugin architecture. The current list of available detour plugins is the following one.
- [`plthook_detour`](/source/detours/plthook_detour) implemented by means of PLTHook library.
- [`dynlink`](/source/dynlink) implements a cross-platform method to dynamically load libraries. It is used to dynamically load plugins into **METACALL**.
- [`environment`](/source/environment) implements a standard way to deal with environment variables. **METACALL** uses environment variables to define custom paths for plugins and scripts.
- [`examples`](/source/examples) ...
- [`filesystem`](/source/filesystem) provides an abstraction for operative system file system.
- [`format`](/source/format) provides a standard way for printing to standard input output for old C versions that does not support newest constructions.
- [`loader`](/source/loader) ...
- [`loaders`](/source/loaders)
- [`log`](/source/log)
- [`memory`](/source/memory)
- [`metacall`](/source/metacall)
- [`ports`](/source/ports)
- [`preprocessor`](/source/preprocessor)
- [`reflect`](/source/reflect)
- [`scripts`](/source/scripts)
- [`serial`](/source/serial)
- [`serials`](/source/serials)
- [`tests`](/source/tests)
- [`version`](/source/version)
### 5.2 Reflect
The module that holds the representation of types, values and functions is called [`reflect`](/source/reflect) and it handles the abstraction of code loaded into **METACALL**.
**METACALL** uses reflection and introspection techniques to inspect the code loaded by the [`loaders`](/source/loaders) in order to interpret it and provide a higher abstraction of it. With this higher abstraction **METACALL** can easily inter-operate between languages transparently.
#### 5.2.1 Type System
**METACALL** implements an abstract type system which is a binary representation of the types supported by it. This means that **METACALL** can convert any type of a language to its own type system and back. Each loader is responsible of doing these conversions.
**METACALL** maintains most of the types of the languages but not all are supported. If new types are added they have to be implemented in the [`reflect`](/source/reflect) module and also in the [`loaders`](/source/loaders) and [`serials`](/source/serials) to fully support it.
| Type | Value |
| :------: | ------------------------------------------------------------------------------ |
| Boolean | `true` or `false` |
| Char | `-128` to `127` |
| Short | `-32,768` to `32,767` |
| Int | `-2,147,483,648` to `2,147,483,647` |
| Long | `–9,223,372,036,854,775,808` to `9,223,372,036,854,775,807` |
| Float | `1.2E-38` to `3.4E+38` |
| Double | `2.3E-308` to `1.7E+308` |
| String | NULL terminated list of characters |
| Buffer | Blob of memory representing a binary data |
| Array | Arrangement of values of any type |
| Map | List of elements formed by a key (String) value (Any) pair (Array) |
| Pointer | Low level representation of a memory reference |
| Null | Representation of NULL value type |
| Future | Promise in Node Loader, and any other type equivalent in other languages. |
| Function | Block of code that takes inputs (Arguments) and produces output (Return value) |
| Class | Defines properties and methods that are common to all objects |
| Object | An instance of Class |
- Boolean is mostly represented by an integer value. There are languages that does not support it so it gets converted to an integer value in the memory layout.
- Integer and Floating Point values provide a complete abstraction to numerical types. Type sizes are preserved and the correct type is used when using any number. This depends on the internal implementation of the value by the run-time. Although there can be problems related to this. A `bignum` type from Ruby may overflow if it is too big when trying to convert it to a `float` type in C#.
- String is represented by ASCII encoding currently. Future versions will implement multiple encodings to be interoperable between other language encodings.
- Buffer represents a blob of raw memory (i.e. an array of bytes). This can be used to represent files as images or any other resources into memory.
- Array is implemented using an array of values, which might lead one to ponder that it should be called _list_ instead. But as the memory layout is stored into a contiguous memory block of references to values, it is considered an array.
- Map implements an associative key value pair container. A map is implemented with an array of two sized elements array. Each element of the map is an array of size two, where the first element of it is always a String and the second element is a value of any type.
- Pointer is an opaque value representing a raw reference to a memory block. Some languages allow to use references to memory and some others do not. This type is opaque because **METACALL** does not know what kind of concrete value represents it. The representation may be a complex type handled by the developer source code inside the run-time.
- Null type implements a null value. This type has only been implemented in order to support null value from multiple run-times. It represents a null value and it does not have data size on the value allocated.
#### 5.2.2 Values
Values represent the instances of the **METACALL** type system.
The memory layout guarantees to fit at least the same size of the types into memory. This means if a boolean type can be represented with one bit inside a value of one byte size, maybe this value is stored in a bigger memory block and this fact is architecture and platform dependant.
When converting values between different types, if any potential number overflow or invalid conversion between types is done, **METACALL** will warn about it. If any conversion of types can be handled by **METACALL**, it will automatically cast or transform the values into the target type automatically in order to avoid errors in the call.
The value model is implemented by means of object pool. Each value is a reference to a memory block allocated from a memory pool (which can be injected into **METACALL**). The references can be passed by value, this means **METACALL** copies the reference value instead of the data which this reference is pointing to, like most run-times do when managing their own values.
Each created value must be destroyed manually, otherwise it will lead to a memory leak. This only occurs when dealing with **METACALL** at C level. If **METACALL** is being used in a higher language through [`ports`](/source/ports), the developer does not have to care about memory management.
The value memory layout is described in the following form.
| Memory Offset | `0` to `sizeof(data) - 1` | `sizeof(data)` to `sizeof(data) + sizeof(type_id) - 1` |
| :-----------: | :-----------------------: | :----------------------------------------------------: |
| **Content** | **DATA** | **TYPE ID** |
This layout is used for the following reasons:
- Data is located at the first position of the memory block, so it can be used as a normal low level value. This allows to treat **METACALL** values as normal C values. Therefore you can use **METACALL** with normal pointers to existing variables, literal values as shown in the previous examples or **METACALL** values.
- Data can be accessed faster as it is located at first position of the memory block. There is not extra calculation of an offset when trying to access the pointer.
- Data and type id are contiguously allocated in order to treat it as the same memory block so it can be freed with one operation.
#### 5.2.3 Functions
Functions are abstract callable representations of functions, methods or procedures loaded by [`loaders`](/source/loaders). A function is like a template which is linked to a loader run-time and allows to do a foreign function call.
A function is composed of a name and a signature. The signature defines the arguments names, types, and return type, if any. When a function is loaded, **METACALL** tries to inspect the signature and records the types, if any. It stores the arguments names and sizes and also a concrete type that will be used later by the loader to implement the call to the run-time.
The function interface must be implemented by the [`loaders`](/source/loaders) and it has the following form.
```c
typedef struct function_interface_type
{
function_impl_interface_create create;
function_impl_interface_invoke invoke;
function_impl_interface_await await;
function_impl_interface_destroy destroy;
} * function_interface;
```
- `create` instantiates the function concrete data related to the run-time.
- `invoke` transforms arguments from [`reflect`](/source/reflect) abstract types to run-time concrete types, executes the call in the run-time, and converts the result of the call from run-time concrete type to [`reflect`](/source/reflect) abstract type.
- `await` idem to invoke but awaiting the promise that is expected to be returned by the function.
- `destroy` clears all data previously instantiated in `create`.
The type deduction can be done at different levels. For example, it is possible to guess function types from the loaded code.
```python
def multiply_type(a: int, b: int) -> int:
return a * b
```
If this code is loaded, **METACALL** will be able to inspect the types and define the signature. Signature includes the names of the arguments, the types of those arguments, if any, and the return type, if any.
It may be possible that the function loaded into **METACALL** is duck typed. This means it does not have information about what types it supports and therefore they cannot be inspected statically.
```python
def multiply_duck(a, b):
return a * b
```
At low level **METACALL** must always know the types to do the call. This types can be inferred statically or dynamically and this has implications over the call model.
In the first example, we can simply call the function without specifying the types.
```c
metacall("multiply_type", 3, 4); // 12
```
As the signature is already know the literal values `3` and `4` can be converted into **METACALL** values automatically. Note that in this case, as literal values are provided, if we pass a double floating point, the memory representation of the value will be corrupted as there is no possible way to detect input values and cast them to the correct target values.
In the second example, the values are not known. If we use the same API to call the function, **METACALL** will not be able to call correctly the function as its types are not known. To allow calls to duck typed functions the developer must specify the value types he is passing to the function.
```c
const enum metacall_value_id multiply_types[] =
{
METACALL_INT, METACALL_INT
};
metacallt("multiply_duck", multiply_types, 3, 4); // 12
```
This method allows to pass different value types to the same function. The following call would be valid too.
```c
const enum metacall_value_id multiply_types[] =
{
METACALL_DOUBLE, METACALL_DOUBLE
};
metacallt("multiply_duck", multiply_types, 3.0, 4.0); // 12.0
```
### 5.3 Plugins
**METACALL** has a plugin architecture implemented at multiple levels.
- Loaders implement a layer of plugins related to the run-times.
- Serials implement a layer of (de)serializers in order to transform input (arguments) or output (return value) of the calls into a generic format.
- Detours is another layer of plugins focused on low level function interception (hooks).
Each plugin is a piece of software that can be dynamically loaded into the **METACALL** core, used and unloaded when it is not needed anymore.
#### 5.3.1 Loaders
Loaders are responsible for embedding run-times into **METACALL**. For a step-by-step guide to implementing a new loader, see [Creating a New Loader](CREATING_A_LOADER.md). Each loader has the following interface.
```c
typedef struct loader_impl_interface_type
{
loader_impl_interface_initialize initialize;
loader_impl_interface_execution_path execution_path;
loader_impl_interface_load_from_file load_from_file;
loader_impl_interface_load_from_memory load_from_memory;
loader_impl_interface_load_from_package load_from_package;
loader_impl_interface_clear clear;
loader_impl_interface_discover discover;
loader_impl_interface_destroy destroy;
} * loader_impl_interface;
```
A loader must implement it to be considered a valid loader.
- `initialize` starts up the run-time for that loader instance.
- `execution_path` defines a new import path to the run-time.
- `load_from_file` loads code from one or more files into the run-time and returns a handle which represents that loaded unit.
- `load_from_memory` loads code from memory into the run-time and returns a handle which represents that loaded unit.
- `load_from_package` loads code from a compiled library or package into the run-time and returns a handle which represents it. The exact meaning of a package is loader specific.
- `discover` inspects a handle previously loaded and populates the **METACALL** reflection context with the exported symbols, signatures, classes, and other metadata required for interoperation. In the common flow, this step is triggered by the core immediately after a successful `load_from_*`.
- `clear` unloads a handle from the run-time. This is a per-handle operation and invalidates that handle and the symbols associated with it.
- `destroy` shutdowns the run-time. This is the final per-loader operation and tears down the loader instance itself.
The common lifecycle is `initialize` -> optional `execution_path` -> `load_from_file`, `load_from_memory`, or `load_from_package` -> `discover` -> call or inspect through the **METACALL** API -> `clear` -> `destroy`. After a successful load, the returned handle becomes managed by **METACALL** and remains valid until it is cleared or the loader is destroyed during shutdown. Depending on the loader and loading mode, a single handle may represent multiple files loaded together, while symbols loaded without a private handle may be propagated to the loader scope instead.
##### 5.3.1.1 Python
##### 5.3.1.2 NodeJS
##### 5.3.1.3 JavaScript
##### 5.3.1.4 C#
##### 5.3.1.5 Ruby
##### 5.3.1.6 Mock
##### 5.3.1.7 File
#### 5.3.2 Serials
##### 5.3.2.1 MetaCall
##### 5.3.2.2 RapidJSON
#### 5.3.3 Detours
##### 5.3.3.1 PLTHook
### 5.4 Ports
Ports are the frontends to the **METACALL C API**. If [`loaders`](#531-loaders) allow **METACALL** to embed a foreign run-time into C, ports allow a host language to embed **METACALL** into itself. Both layers are complementary and in most real applications they are used together.
```
Host Language -> Port -> MetaCall C API -> MetaCall Core -> Loader -> Foreign Run-time
```
For example, when Python uses the Python Port to call Ruby code, the Python Port translates Python values and control flow into **METACALL** API calls, and the Ruby Loader is the component that actually embeds the Ruby run-time and executes the target function.
Unlike loaders, ports do not implement a single formal plugin interface inside the core. They are language specific wrappers built around the same **METACALL C API**, so what is shared between ports is a common lifecycle and conceptual model rather than a single ABI or vtable shape.
In the common case, ports are used explicitly through their API. The host language initializes **METACALL**, loads foreign code with helpers such as `metacall_load_from_file`, `metacall_load_from_memory`, or `metacall_load_from_package`, and then invokes exported symbols with `metacall` or the equivalent call helpers provided by that port. This explicit workflow is the default interoperability model and it is the one every port can provide consistently.
As a convenience, some ports also support direct import patterns such as `require('./sum.py')` or `import script.js`. This is optional sugar layered on top of the explicit API, not a separate loading mechanism. In order to support it, a port must integrate with the host language import or module system, map file extensions or prefixes to the appropriate loader tags, and rely on an existing loader for the target language. That integration may be implemented by monkey patching, custom import hooks, native module loader extension points, or any other mechanism offered by the host language. If any of those pieces are missing, the explicit port API remains the standard way to call foreign code. In most implementations, the imported value is not a native module loaded directly by the host runtime, but a host language object or module synthesized by the port from a **METACALL** handle and the exported symbols discovered by the loader. For example, the Python Port implements this behavior in [`py_port/metacall/api.py`](/source/ports/py_port/metacall/api.py): `__metacall_import__` overrides the import flow, `file_extensions_to_tag` and `package_extensions_to_tag` resolve the correct loader tag, and the actual loading is delegated to `module.metacall_load_from_file_export(...)` or `module.metacall_load_from_package_export(...)` before a Python module object is generated from the returned handle.
#### 5.4.1 Responsibilities
Although the concrete implementation varies between languages, every port has the same general responsibilities:
- Initialize and destroy the **METACALL** core from the host language.
- Expose the loading API (`metacall_load_from_file`, `metacall_load_from_memory`, `metacall_load_from_package`, and configuration based loading when available).
- Expose the invocation API (`metacall`, handle based calls, inspection, futures or await helpers when supported).
- Convert host language values to [`reflect`](#52-reflect) values and convert returned values back to native host language types.
- Hide the manual memory management required by the C API.
- Adapt callbacks, exceptions, promises or futures, and thread affinity to the semantics of the host language.
- Optionally integrate with the module or import system of the host language for a more idiomatic user experience.
Compiled host languages still need this boundary. Being compiled does not provide a common reflection model, calling convention, ownership model, or asynchronous interface. For languages such as Go or Rust, the port is mainly responsible for bridging the ABI and the type system, not for embedding an interpreter.
#### 5.4.2 Port Implementations
The current source tree contains two families of ports:
- Standalone or native ports, implemented directly with the host language tooling. At the moment this includes [`cxx_port`](/source/ports/cxx_port), [`node_port`](/source/ports/node_port), [`py_port`](/source/ports/py_port), [`go_port`](/source/ports/go_port), and [`rs_port`](/source/ports/rs_port).
- SWIG based ports, kept mostly for compatibility or bootstrap purposes. The current build system still includes ports such as [`cs_port`](/source/ports/cs_port), [`d_port`](/source/ports/d_port), [`java_port`](/source/ports/java_port), [`js_port`](/source/ports/js_port), [`lua_port`](/source/ports/lua_port), [`php_port`](/source/ports/php_port), [`pl_port`](/source/ports/pl_port), [`r_port`](/source/ports/r_port), and [`rb_port`](/source/ports/rb_port).
The project is gradually moving away from SWIG. The current build scripts already document this as a transitional architecture and the long term intention is to unify ports and loaders more cleanly.
#### 5.4.3 Representative Examples
The following examples show that ports are not identical wrappers; each one adapts **METACALL** to the expectations of the host ecosystem.
- The NodeJS Port combines a native addon with a JavaScript layer. Besides exposing explicit functions such as `metacall_load_from_file`, it monkey patches `require` so expressions like `require('./script.py')` can be resolved through the correct loader transparently.
- The Python Port loads a platform specific extension module and monkey patches `__import__`. This allows patterns such as `import script.js` or `from metacall.node import ramda` while presenting the loaded handle as a normal Python module.
- The Go Port is implemented over `cgo`. It wraps the C API, translates Go values and callbacks, and serializes work through a goroutine locked to an operating system thread in order to adapt to thread sensitive loader behavior.
- The Rust Port uses generated bindings together with a safe wrapper API. It exposes typed `Result` based calls, loader helpers, and higher level abstractions over the raw C interface.
- The C++ Port is a native library layer over the C API intended to provide a more idiomatic C++ integration without changing the core architecture.
#### 5.4.4 Lifecycle
In general, the lifecycle of a port is the following one:
1. Initialize **METACALL** from the host language.
2. Load one or more scripts or packages by specifying the loader tag.
3. Invoke functions, inspect handles, or await asynchronous results through the port API.
4. Destroy **METACALL** when the host process is finished with the runtime.
This lifecycle is intentionally similar across ports so that the operational model remains constant even if the syntax of each language changes.
#### 5.4.5 Limitations and Future Work
Ports do not replace loaders; they depend on them. A port can only call into run-times for which a loader exists and has been correctly configured. In the same way, ports inherit many constraints from the core, such as the threading and fork safety limitations described in the following sections.
Current limitations in the port layer include:
- Different maturity levels between ports.
- Remaining SWIG based wrappers.
- Host language specific deadlocks or callback limitations when multiple run-times are mixed together.
- Duplication of conversion logic between ports.
The long term goal of the ports architecture is to provide the same conceptual API and behavior in every host language while reducing wrapper specific code and converging on native implementations where possible.
### 5.5 Serialization
### 5.6 Memory Layout
### 5.7 Fork Model
**METACALL** implements a fork safe model. This means if **METACALL** is running in any program instance, the process which is running can be forked safely at any moment of the execution. This fact has many implications at design, implementation and use levels. But the whole **METACALL** architecture tries to remove all responsibility from the developer and make this transparent.
To understand the **METACALL** fork model, first of all we have to understand the implications of the forking model in operative systems and the difference between [fork-one and fork-all models](https://docs.oracle.com/cd/E37838_01/html/E61057/gen-1.html).
The main difference between fork-one and fork-all is that in fork-one only the thread which called the fork is preserved after the fork (i.e. gets cloned). In fork-all model, all threads are preserved after cloning. POSIX uses fork-one model, meanwhile Oracle Solaris use the fork-all model.
Because of fork-one model, forking a running run-time like NodeJS (which has a thread pool) implies that in the child process the thread pool will be almost dead except the thread which did the fork call. So NodeJS run-time cannot continue the execution anymore and the event-loop enters into a deadlock state.
When a fork is done, the status of the execution is lost by the moment. **METACALL** is not able to preserve the state when a fork is done. Some run-times do not allow to preserve the internal state. For example, the bad design<sup>[[0]](https://github.com/nodejs/node/issues/23265)[[1]](https://github.com/nodejs/node/issues/23265#issuecomment-452690239)[[2]](https://github.com/nodejs/node/issues/23265#issuecomment-496873739)[[3]](https://github.com/nodejs/node/issues/23265#issuecomment-496878712)[[4]](https://github.com/nodejs/node/issues/23265#issuecomment-496910654)[[5]](https://github.com/nodejs/node/issues/23265#issuecomment-496918901)</sup> of NodeJS does not allow to manage the thread pool from outside, so it cannot be preserved after a fork.
Because of these restrictions, **METACALL** cannot preserve the status of the run-times. In the future this model will be improved to maintain consistency and preserve the execution state of the run-times making **METACALL** more robust.
Although the state is not preserved, fork safety is. The mechanism **METACALL** uses to allow fork safety is described in the following enumeration.
1. Intercept fork call done by the program where **METACALL** is running.
2. Shutdown all run-times by means of unloading all loaders.
3. Execute the real fork function.
4. Restore all run-times by means of reloading all loaders.
5. Execute user defined fork callback if any.
To achieve this, **METACALL** hooks fork primitives depending on the platform.
- `fork` on POSIX systems.
- `RtlCloneUserProcess` on Windows systems.
If you use `clone` instead of `fork` to spawn a new process in a POSIX system, **METACALL** won't catch it.
Whenever you call a to a cloning primitive **METACALL** intercepts it by means of [**`detour`**](/source/detour). Detours is a way to intercept functions at low level by editing the memory and introducing a jump over your own function preserving the address of the old one. **METACALL** uses this method instead of POSIX `pthread_atfork` for three main reasons.
- The first one is that `pthread_atfork` is only supported by POSIX systems. So it is not a good solution because of the philosophy of **METACALL** is to be as cross-platform as possible.
- The second is that `pthread_atfork` has a [bug in the design of the standard](https://stackoverflow.com/a/6605487). It was designed to solve a problem which cannot be solved with `pthread_atfork` itself. This means that even having the control of NodeJS thread pool, it will not be possible to restore the [mutexes](https://github.com/nodejs/node/blob/v8.x/src/node_platform.cc) in the child process. The only possibility is to re-implement the thread pool of NodeJS with async safe primitives like a semaphore. Async safe primitives will be able to work in the child process handler. But this is not possible as it enters in conflict with the design decision of to not modify the run-times.
- The third one is that the mechanism of `pthread_atfork` also [will be deprecated](http://pubs.opengroup.org/onlinepubs/9699919799/functions/pthread_atfork.html) because of second reason.
> The `pthread_atfork()` function may be formally deprecated (for example, by shading it OB) in a future version of this standard.
Detours model is not safe. It is platform dependant and implies that the program modifies the memory of itself during the execution which is not safe at all and can induce bugs or security flaws if it is not done correctly. But because of limitations of run-times, there is not another alternative to solve the problem of fork safety.
Usually the developer is the same who does the fork, but it may be possible that **METACALL** is embedded into a larger application and the developer is in the middle between the application code and **METACALL** so it is impossible to control when a fork is done. Because of this the developer can register a callback by means of [**`metacall_fork`**](/source/metacall/include/metacall/metacall_fork.h) to know when a fork is executed to do the actions needed after the fork, for example, re-loading all previous code and restore the state of the run-times. This gives a partial solution to the problem of losing the state when doing a fork.
### 5.8 Threading Model
The threading model is still experimental. We are discovering the best ways of designing and implementing it, so it may vary over time. At the moment of writing (check the commit history), there are some concerns that are already known and parts of the design already achieved thanks to the NodeJS event loop nature.
The Node Loader is designed in a way in which the V8 instance is created in a new thread, and from there the event loop "blocks" that thread until the execution. Recent versions of N-API (since NodeJS 14.x) allow you to have control and reimplement your own event loop thanks to the new embedder API. But when this project started and NodeJS loader was implemented, only NodeJS 8.x exist. So the only option (without reimplementing part of NodeJS, because it goes against one design decisions of the project) was to use `node::Start`, a call that blocks your thread while executing the event loop. This also produces a lot of problems, because of lack of control over NodeJS, but they are not directly related to the thread model.
To overcome the blocking nature of `node::Start`, the event loop is launched in a separated thread, and all calls to the loader are executed via submission to the event loop in that thread. In the first implementation, it was done using `uv_async_t`, but in the current implementation (since NodeJS 10.x), with thread safe mechanisms that allow you to enqueue safely into the event loop thanks to the new additions to the N-API. The current thread where the call is done waits with a condition `uv_cond_t` upon termination of the submission and resolution of the call.
This solution of waiting to the call with the condition, introduces new problems. For completely async calls, there is no problem at all, but for synchronous calls, it can deadlock. For example, when calling recursively to the same synchronous function via **METACALL**, in the second call it will try to block twice and deadlock the thread. So in order to solve this an atomic variable was added in addition to a variable storing the thread id of the V8 thread. With this, recursive calls can be detected, and instead of blocking and enqueueing them, it is possible to call directly and safely to the function because we are already in the V8 thread when the second iteration is done.
This solves all (known) issues related to NodeJS threading model **if and only if** you use **METACALL** from C/C++ or Rust as a library, and you don't mix languages. This means, you use directly the low level API directly, and you do not use any `Port` or you mix this with other languages, doing calls in between. You can still have a chance to generate deadlocks if your software uses incorrectly the API. For example, you use one condition which gets released in an async callback (a lambda in the argument of the call to `metacall_await`) and your JS code never resolves properly that promise.
If you use the CLI instead, and your host language is Python or any other (which does not allow to use you the low level API), and you want to load scripts from other languages, you have to use **METACALL** through `Ports`. Ports provide a high abstraction of the low level API and allow you to load and call functions of other languages. Here is where the fun begins.
There are few considerations we must take into account. In order to explain this we are going to use a simple example first, using Python and NodeJS. Depending on the runtime, there are different mechanisms to handle threads and thread safety:
- Python:
1. Python uses a Global Interpreter Lock (GIL), which can be acquired from different threads in order to do thread safe calls. This can be problematic due to deadlocks.
2. Python event loop can be decoupled from Python interpreter thread by using Python Thread API (work in progress: https://github.com/metacall/core/pull/64). This fact simplifies the design.
3. Python can run multiple interpreter instances, starting from newer versions (not implemented yet).
- NodeJS:
1. NodeJS uses a submission queue and does not suffer from a global mutex like Python.
2. NodeJS V8 thread is coupled to the event loop (at least with the current version used in **METACALL**, and it is difficult to have control over it).
3. NodeJS can execute multiple V8 threads with the multi-isolate library from the latest versions of V8 (not implemented yet).
Once these concerns are clear, now we can go further and inspect some cases where we can find deadlocks or problems related to them:
1. **NodeJS is the host language**, and it launches the Python interprer in the V8 thread:
This model is relatively safe because Node Loader is completely reentrant, and Python GIL too. This means you can do recursive calls safely, and all those calls will always happen in V8. Even if we do callbacks, all of them will happen in the same thread, so there aren't potential deadlocks. This means we can safely use a functional library from NodeJS, and it won't deadlock. For example: [Using Fn.py from NodeJS](https://github.com/metacall/fn.py-javascript-example).
But there is a problem when we try to destroy the loaders. Python interpreter does not allow to be destroyed from a different thread where it was launched. This means, if we destroy the Node Loader first, then it will be impossible to destroy the Python Loader, because the V8 thread has been finished. We must destroy the Loaders in order and in the correct thread. This means if we try to destroy Node Loader, during its destruction in the V8 thread, we must destroy Python Loader and any other loader that has been initialized in that thread.
As a result, each loader must use the following instructions:
- When the loader has finished the initialization, it must register its initialization order. It will record internally the current thread id too.
```c
loader_initialization_register(impl);
```
- When the loader is going to be destroyed, but before destroy starts, the children must be destroyed in a recursive way, so the whole tree can be iterated properly in order.
```c
loader_unload_children();
```
The result of the current destruction model is that: **`metacall_initialize` and `metacall_destroy` must be done from the same thread**. This should not be a problem for developers using the CLI. But embedders must take this into account.
2. **Python is the host language**, and it launches NodeJS in a new (V8) thread:
[TODO: Explain why callbacks deadlock in this context]
In order to end this section, here's a list of ideas that are not completely implemented yet, but they are in progress:
- Lock free data structures for holding the functions.
- Asynchronous non-deadlocking, non-stack growing callbacks between runtimes (running multiple event loops between languages). This will solve the second case where Python is the host language and deadlocks because of NodeJS event loop nature.
- Support for multi-isolate and multiple interpreters instances.
## 6. Application Programming Interface (API)
## 7. Build System
Follow these steps to build and install **METACALL** manually.
```sh
git clone https://github.com/metacall/core.git
mkdir core/build && cd core/build
cmake ..
# Unix (Linux and MacOs)
sudo HOME="$HOME" cmake --build . --target install
# Windows (or when installing to a path with permissions)
cmake --build . --target install
```
### 7.1 Build Options
These options can be set using **`-D`** prefix when configuring CMake. For example, the following configuration enables the build of Python and Ruby loaders.
```sh
cmake -DOPTION_BUILD_LOADERS_PY=On -DOPTION_BUILD_LOADERS_RB=On ..
```
Available build options are the following ones.
| Build Option | Description | Default Value |
| :-------------------------: | ------------------------------------------------------ | :-----------: |
| **BUILD_SHARED_LIBS** | Build shared instead of static libraries. | ON |
| **OPTION_SELF_CONTAINED** | Create a self-contained install with all dependencies. | OFF |
| **OPTION_BUILD_TESTS** | Build tests. | ON |
| **OPTION_BUILD_BENCHMARKS** | Build benchmarks. | OFF |
| **OPTION_BUILD_DOCS** | Build documentation. | OFF |
| **OPTION_BUILD_EXAMPLES** | Build examples. | ON |
| **OPTION_BUILD_LOADERS** | Build loaders. | ON |
| **OPTION_BUILD_SCRIPTS** | Build scripts. | ON |
| **OPTION_BUILD_SERIALS** | Build serials. | ON |
| **OPTION_BUILD_DETOURS** | Build detours. | ON |
| **OPTION_BUILD_PORTS** | Build ports. | OFF |
| **OPTION_FORK_SAFE** | Enable fork safety. | OFF |
| **OPTION_THREAD_SAFE** | Enable thread safety. | OFF |
| **OPTION_COVERAGE** | Enable coverage. | OFF |
| **CMAKE_BUILD_TYPE** | Define the type of build. | Release |
It is possible to enable or disable concrete loaders, script, ports, serials or detours. For building use the following options.
| Build Option Prefix | Build Option Suffix |
| :-----------------------: | --------------------------------------------------------------------- |
| **OPTION_BUILD_LOADERS_*** | `C` `JS` `CS` `MOCK` `PY` `JSM` `NODE` `RB` `FILE` |
| **OPTION_BUILD_SCRIPTS_*** | `C` `CS` `JS` `NODE` `PY` `RB` `JAVA` |
| **OPTION_BUILD_SERIALS_*** | `METACALL` `RAPID_JSON` |
| **OPTION_BUILD_DETOURS_*** | `PLTHOOK` |
| **OPTION_BUILD_PORTS_*** | `CS` `CXX` `D` `GO` `JAVA` `JS` `LUA` `NODE` `PHP` `PL` `PY` `R` `RB` |
To format the entire C/C++ codebase use:
```sh
cmake --build build --target clang-format
```
Be aware that this target won't exist if clang-format was not installed when cmake was last run.
### 7.2 Coverage
In order to run code coverage and obtain html reports use the following commands (assuming you just cloned the repository):
```sh
git clone https://github.com/metacall/core.git
mkdir core/build && cd core/build
cmake -DCMAKE_BUILD_TYPE=Debug -DOPTION_COVERAGE=On ..
make -j$(NPROC)
ctest
ctest -T Coverage
gcovr -r ../source/ . --html-details coverage.html
```
The output reports will be generated in `${CMAKE_BINARY_DIR}/coverage.html` in html format.
### 7.3 Debugging
For debugging memory leaks, undefined behaviors and other related problems, the following compile options are provided:
| Build Option | Description | Default Value |
| :--------------------------------: | --------------------------------------------------------- | :-----------: |
| **OPTION_TEST_MEMORYCHECK** | Enable Valgrind with memcheck tool for the tests. | OFF |
| **OPTION_BUILD_ADDRESS_SANITIZER** | Build with AddressSanitizer family (GCC, Clang and MSVC). | OFF |
| **OPTION_BUILD_THREAD_SANITIZER** | Build with ThreadSanitizer family (GCC, Clang and MSVC). | OFF |
| **OPTION_BUILD_MEMORY_SANITIZER** | Build with MemorySanitizer family (Clang and MSVC). | OFF |
All options are mutually exclusive. Valgrind is not compatible with AddressSanitizer and AddressSanitizer is not compatible with ThreadSanitizer and AddressSanitizer with MemorySanitizer. Some run-times may fail if they are not compiled with AddressSanitizer too, for example NetCore. Due to this, tests implying may fail with signal 11. The same problem happens with Valgrind, due to that, some tests are excluded of the memcheck target.
For running all tests with Valgrind, enable the `OPTION_TEST_MEMORYCHECK` flag and then run:
```sh
make memcheck
```
For running a test (or all) with AddressSanitizer or ThreadSanitizer, enable the `OPTION_BUILD_ADDRESS_SANITIZER` or `OPTION_BUILD_THREAD_SANITIZER` flags respectively and then run:
```sh
# Run one test
make py_loader rb_loader node_loader metacall-node-port-test # Build required dependencies and a test
ctest -VV -R metacall-node-port-test # Run one test (verbose)
# Run all
make
ctest
```
For running other Valgrind's tools like helgrind or similar, I recommend running them manually. Just run one test with `ctest -VV -R metacall-node-port-test`, copy the environment variables, and configure the flags by yourself.
## 7.4 Build using scripts
MetaCall is a multi-language runtime (C, Python, Node, Rust, etc).
Because of that, installation is complicated. So the project created scripts for installing dependencies, configuring and build the build system. MetaCall provides helper scripts in the [tools](/tools) directory to automatically install dependencies, configure the project, and compile it.
Using these scripts is the recommended way to build MetaCall instead of manually installing packages or running CMake directly, because they ensure consistent environments across Linux, macOS, and Windows.
### Why use the scripts?
The tools scripts:
- Support Linux, macOS, and Windows CI environments.
- Prevent toolchain mismatches.
- Ensure all language loaders compile correctly.
- The script detects the platform and installs the required dependencies.
- The script provides a simple interface such as:
```sh
./metacall-environment.sh [options]
```
- Different contributors may use different Linux distributions, compilers, or package versions. The script ensures everyone builds MetaCall using the same configuration.
### Prepare the Environment
From the repository root:
```sh
./tools/metacall-environment.sh
```
Installs system dependencies required by MetaCall (compilers, build tools, and language runtimes). This script prepares the machine so the project can be configured and compiled. It handles platform-specific packages automatically.
Usage: `metacall-environment.sh` list of components, example:
```sh
./tools/metacall-environment.sh base python nodejs
```
**Core**
| Components | Description |
|:---------------------------------------------------|-------------------------------------------------------------------------------------------------------:|
| base | Compiler and base requirements for compiling MetaCall. |
| rapidjson | Installs RapidJSON serialization for serializing MetaCall data with JSON. |
| pack | Bundle scripts into one deployable runtime. |
| sandbox | Provides isolated execution contexts for safer script execution. |
| backtrace | Generates detailed stack traces and improves error diagnostics across languages. |
**Languages**
| Components | Description |
|:---------------------------------------------------|-------------------------------------------------------------------------------------------------------:|
| python | Enables execution and invocation of Python scripts and functions. |
| ruby | Enables execution of Ruby programs. |
| nodejs | Integrates the Node.js runtime and allows calling CommonJS/ES modules and npm packages. |
| typescript | Executes TypeScript sources via the Node.js runtime. |
| v8 | Embeds the V8 JavaScript engine for executing standalone JavaScript without Node.js APIs. |
| v8rep51 / v8rep54 / v8rep57 / v8rep58 | Compatibility loaders for specific V8 engine versions used in embedded environments. |
| c | Allows MetaCall to load and invoke native C shared libraries and functions. |
| java | Integrates the Java Virtual Machine (JVM) and allows calling Java classes and methods. |
| go | Allows using MetaCall from Go. |
| cobol | Enables interoperability with COBOL. |
| netcore | Support for .NET Core 1.x runtime. |
| netcore2 | Support for .NET Core 2.x runtime. |
| netcore5 | Support for .NET Core 5 runtime. |
| netcore7 | Support for .NET Core 7 runtime. |
| netcore8 | Support for .NET Core 8 runtime. |
| wasm | Executes WebAssembly (WASM) modules through the MetaCall runtime. |
| rpc | Enables remote procedure calls so MetaCall functions can be invoked across network boundaries. |
| rust | Allows calling Rust libraries and functions. |
| zig | Allows loading Zig libraries through C ABI compatibility. |
**Development and Tooling**
| Components | Description |
|:---------------------------------------------------|-------------------------------------------------------------------------------------------------------:|
| coverage | Enables generation of code coverage reports during testing. |
| clangformat | Adds automatic C/C++ source formatting according to project style rules. |
### Configure the project
```sh
./tools/metacall-configure.sh
```
Creates the build directory. Runs CMake with the correct options and also detects available loaders and runtimes.
Usage: `metacall-configure.sh` list of options, example:
```sh
metacall-configure.sh relwithdebinfo python tests
```
**Build Type: Control Debugging vs Speed**
| Options | Description |
|:---------------------------------------------------|-------------------------------------------------------------------------------------------------------:|
| debug | Builds MetaCall with debug symbols and runtime assertions. Slower execution. |
| release | Optimized production build. No debug symbols. Fastest execution but harder to debug. |
| relwithdebinfo | Optimized build with debug information. |
**Languages**
The languages here are the same as in the `metacall-environment.sh` script. You can reuse them.
**Diagnostics and Code Quality**
| Options | Description |
|:---------------------------------------------------|-------------------------------------------------------------------------------------------------------:|
| coverage | Enables generation of code coverage reports during testing. |
| address-sanitizer | Detects invalid memory access. |
| thread-sanitizer | Detects race conditions and thread synchronization issues. |
| memory-sanitizer | Detects usage of uninitialized memory. |
| memcheck | Set up the project for using valgrind to detect memory issues. |
**Development Targets**
| Options | Description |
|----------------------------------------------------|--------------------------------------------------------------------------------------------------------|
| scripts | Builds all MetaCall example scripts, required for tests. |
| examples | Builds example applications demonstrating MetaCall usage. |
| tests | Builds and executes the full test suite. |
| benchmarks | Builds and runs performance benchmarks. |
### Build MetaCall
```sh
./tools/metacall-build.sh
```
This compiles MetaCall and all enabled loaders.
Usage: `metacall-build.sh` list of options, example:
```sh
./tools/metacall-build.sh relwithdebinfo python tests
```
| Options | Description |
|:---------------------------------------------------|-------------------------------------------------------------------------------------------------------:|
| debug | Builds MetaCall with debug symbols and runtime assertions. Slower execution. |
| release | Optimized production build. No debug symbols. Fastest execution but harder to debug. |
| relwithdebinfo | Optimized build with debug information. |
| tests | Build and run all tests. |
| coverage | Build coverage reports. |
| install | Installs compiled MetaCall libraries into the system after a successful build. |
### Verify the Installation
After building, you can run the CLI:
```sh
cd build
./metacallcli
```
If the CLI starts successfully, MetaCall is correctly built.
### 7.5 Build on Cloud - Gitpod
Instead of configuring a local setup, you can also use [Gitpod](https://www.gitpod.io/), an automated cloud dev environment.
Click the button below. A workspace with all required environments will be created.
[](https://gitpod.io/#https://github.com/metacall/core)
> To use it on your forked repo, edit the 'Open in Gitpod' button url to `https://gitpod.io/#https://github.com/<your-github-username>/core`
## 8 Platform Support
The following platforms and architectures have been tested and are known to work correctly with all plugins of **METACALL**.
| Operative System | Architecture | Compiler |
| :--------------: | :---------------------------------------------------------------------------------------------------------------------------------: | :---------: |
| **`ubuntu`** | **`amd64`** | **`gcc`** |
| **`debian`** | **`amd64`** **`amd64/v2`** **`amd64/v3`** **`386`** **`arm64`** **`riscv64`** **`ppc64le`** **`arm/v7`** **`arm/v6`** **`loong64`** | **`gcc`** |
| **`macos`** | **`amd64`** **`arm64`** | **`clang`** |
| **`windows`** | **`x86`** **`x64`** | **`msvc`** |
| **`freebsd`** | **`amd64`** **`arm64`** | **`clang`** |
## 8.1 Docker Support
To provide a reproducible environment **METACALL** is also distributed under Docker on [DockerHub](https://hub.docker.com/r/metacall/core). Current images are based on `debian:bookworm-slim` for `amd64` architecture.
For pulling the **METACALL** `latest` image containing the runtime, use:
```sh
docker pull metacall/core
```
For pulling a specific image depending on the tag, use:
- **METACALL** `deps` image. Includes all dependencies for development:
```sh
docker pull metacall/core:deps
```
- **METACALL** `dev` image. Includes all dependencies, headers and libraries for development:
```sh
docker pull metacall/core:dev
```
- **METACALL** `runtime` image. Includes all dependencies and libraries for runtime:
```sh
docker pull metacall/core:runtime
```
- **METACALL** `cli` image. Includes all dependencies and libraries for runtime and the CLI as entry point (equivalent to `latest`):
```sh
docker pull metacall/core:cli
```
### 8.1.1 Docker Development
It is possible to develop **METACALL** itself or applications using **METACALL** as standalone library with Docker. The `dev` image can be used for development. It contains all dependencies with all run-times installed with the code, allowing debugging too.
Use the following commands to start developing with **METACALL**:
```sh
mkdir -p $HOME/metacall
code $HOME/metacall
```
We are going to run a docker container with a mounted volume. This volume will connect the `LOADER_SCRIPT_PATH` inside the container, and your development path in the host. We are using `$HOME/metacall`, where we have our editor opened.
```sh
docker pull metacall/core:dev
docker run -e LOADER_SCRIPT_PATH=/metacall -v $HOME/metacall:/metacall -w /metacall -it metacall/core:dev /bin/bash
```
Inside docker terminal you can run `python` or `ruby` command to test what you are developing. You can also run `metacallcli` to test (load, clear, inspect and call).
### 8.1.2 Docker Testing
An alternative for testing is to use a reduced image that includes the runtime and also the CLI. This alternative allows fast prototyping and CLI management in order to test and inspect your own scripts.
Use the following commands to start testing with **METACALL**:
```sh
mkdir -p $HOME/metacall
code $HOME/metacall
```
We are going to run a docker container with a mounted volume. This volume will connect the `LOADER_SCRIPT_PATH` inside the container, and your development path in the host. We are using `$HOME/metacall`, where we have our editor opened.
```sh
docker pull metacall/core:cli
docker run -e LOADER_SCRIPT_PATH=/metacall -v $HOME/metacall:/metacall -w /metacall -it metacall/core:cli
```
After the container is up, it is possible to load any script contained in host folder `$HOME/metacall`. If we have a `script.js` inside the folder, we can just load it (each line beginning with `>` is the input command):
`script.js`
```js
function sum(left, right) {
return left + right;
}
module.exports = {
sum,
};
```
`Command Line Interface`
```sh
> load node script.js
Script (script.js) loaded correctly
> inspect
runtime node {
module script {
function sum(left, right)
}
}
runtime __metacall_host__
> call sum(3, 5)
8.0
> exit
```
Where `script.js` is a script contained in host folder `$HOME/metacall` that will be loaded on the CLI after starting up the container. Type `help` to see all available CLI commands.
### 8.1.3 Docker MSan Development
This method is used for iterating on **Memory Sanitizer** (MSan) fixes locally without waiting for CI on every change. The idea is to build the Docker image **once** (with a trick to allow build errors), then mount your local `source/` and `cmake/` directories so code changes are reflected instantly inside the running container.
Prerequisites:
- At least **20GB** of free disk space
- Close all heavy applications before building this maxes out CPU and RAM
1. Clean Docker completely
```sh
docker system prune --all -f
docker buildx prune --all -f
```
2. Patch the Dockerfile to allow build errors
In `tools/docker/Dockerfile` near line 85, add `|| true` at the end of the build command (same line):
```dockerfile
RUN cd $METACALL_PATH/build \
&& $METACALL_PATH/tools/metacall-build.sh ${METACALL_BUILD_TYPE} ${METACALL_BUILD_OPTIONS} || true
```
This prevents Docker from aborting the image build if compilation has errors — since MSan builds often have expected failures during development.
3. Build the Docker image (once)
In one terminal:
```sh
./docker-compose.sh test-memory-sanitizer &> output.txt
```
In a second terminal, watch progress live:
```sh
tail -f output.txt
```
> For a lighter build without MSan (useful for contributors with less disk space):
> ```sh
> # Completely disable sanitizers:
> ./docker-compose.sh test &> output.txt
> # Or with address sanitizer:
> ./docker-compose.sh test-address-sanitizer &> output.txt
> ```
4. Run the container with shared volumes
```sh
docker run --rm \
-v `pwd`/cmake:/usr/local/metacall/cmake \
-v `pwd`/source:/usr/local/metacall/source \
-it metacall/core:dev
```
Edit source files in VSCode on the host changes are reflected instantly inside the container.
5. Build and test inside the container
```sh
cd /usr/local/metacall/build
cmake .. # required after any ignorelist or CMake changes
make -j8 <target> # build a specific target
```
Example targets:
```sh
make -j8 metacall-py-call-bench
make -j8 metacall-node-call-bench
make -j8 metacall-node-bench
```
Run a specific test:
```sh
ctest -VV -R <test-name>
```
Run all benchmarks and save full log:
```sh
ctest -VV -R bench &> log.txt
```
Quick triage show only MSan error summaries:
```sh
grep "SUMMARY:" log.txt
```
Full stack trace for a specific error:
```sh
grep -B5 -A 30 "use-of-uninitialized-value\|WARNING: MemorySanitizer" log.txt
```
> **Note:** The `msan.ignorelist` file is compile-time only. After editing it, always run `cmake ..` followed by a rebuild otherwise changes are silently ignored since `make` reuses cached object files compiled under the old rules.
## 9. Benchmarks
**METACALL** provides benchmarks for multiple operative systems in order to improve performance iteratively, those can be found in GitHub Pages:
| Operative System | URL |
| :-----------------: | :--------------------------------------------------: |
| **`ubuntu-latest`** | https://metacall.github.io/core/bench/ubuntu-latest/ |
| **`macos-latest`** | https://metacall.github.io/core/bench/macos-latest/ |
| **`windows-2022`** | https://metacall.github.io/core/bench/windows-2022/ |
| **`windows-2025`** | https://metacall.github.io/core/bench/windows-2025/ |
## 10. License
**METACALL** is licensed under **[Apache License Version 2.0](/LICENSE)**.
> Copyright (C) 2016 - 2026 Vicente Eduardo Ferrer Garcia <<vic798@gmail.com>>
>
> Licensed under the Apache License, Version 2.0 (the "License");
> you may not use this file except in compliance with the License.
> You may obtain a copy of the License at
>
> http://www.apache.org/licenses/LICENSE-2.0
>
> Unless required by applicable law or agreed to in writing, software
> distributed under the License is distributed on an "AS IS" BASIS,
> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> See the License for the specific language governing permissions and
> limitations under the License.