serde_luaq  

[!NOTE] This library is still a work in progress, and there are no API stability guarantees.

serde_luaq is a library for deserialising (and eventually, serialising) simple, JSON-equivalent data structures from Lua 5.4 source code, without requiring Lua itself (unlike mlua).

The goal is to be able to read state from software (mostly games) which is serialised using Lua %q formatting (and similar techniques) without requiring arbitrary code execution.

This library consists of four parts:

• A LuaValue enum, which describes Lua 5.4's basic data types (nil, boolean, string, number, table).

• A peg-based parser for parsing a &[u8] (containing Lua) into a LuaValue.

• A Serde-based Deserialize implementation for converting a LuaValue into your own data types.

• Optional lossy converter to and from serde_json's Value type.

Goal

For example, you could have a Lua script like this:

a = 1
b = {1, 2, 3}
c = {
    ["foo"] = "bar",
}

And define some a schema using Serde traits:

#[derive(Deserialize, PartialEq, Debug)]
struct ComplexType {
    foo: String,
}

#[derive(Deserialize, PartialEq, Debug)]
struct Test {
    a: u32,
    b: Vec<u32>,
    c: ComplexType,
}

Then deserialise it with:

use serde_luaq::{from_slice, LuaFormat};

let parsed: Test = serde_luaq::from_slice(
  input,
  LuaFormat::Script,
  /* maximum table depth */ 16,
).unwrap();

assert_eq!(parsed, Test {
  a: true,
  b: vec![1, 2, 3],
  c: ComplexType { foo: "bar".to_string() },
});

Parser features

• Input formats
  • Bare Lua value expression, similar to JSON ({["hello"] = "world"})
  • Lua return statement (return {["hello"] = "world"})
  • Script with identifier assignments only (hello = "world")
• Serde (partial)
  • Deserialising
  • Serialising
• Lossy serde_json interoperability
  • LuaValue -> serde_json::Value
  • serde_json::Value -> LuaValue

Lua language features

This library aims to implement a subset of Lua 5.4 that is equivalent to the subset of JavaScript that a JSON parser would implement:

• nil
• Booleans (true, false)
• Numbers
  • Integers (i64 only)
    • Decimal integers (123)
      • Coercion to float for decimal numbers < i64::MIN or > i64::MAX (Lua 5.4)
    • Hexadecimal integers (0xFF)
      • Wrapping large hexadecimal numbers to i64
  • Floats (f64 only)
    • Decimal floats with decimal point and optional exponent (3.14, 0.314e1)
    • Decimal floats with mandatory exponent (3e14)
    • Hexadecimal floating points (0x.ABCDEFp+24) (not supported on WASM before v0.2.1)
    • Positive and negative infinity (1e9999, -1e9999)
    • NaN ((0/0))
• Strings
  • Strings in single quotes (')
  • Strings in double quotes (")
  • Strings in long brackets ([[string]], [==[string]==]) (up to 5 = deep)
  • Arbitrary 8-bit binary data inside strings (like [u8])
  • Escapes in quoted strings:
    • C-like backslash-escapes (abfnrtv\"')
    • Escaped line breaks (\\\n, \\\r, \\\r\n, \\\n\r)
    • \z whitespace span escapes (str\z ing == string)
    • Decimal escape sequences (\1, \01, \001)
    • Hexadecimal escape sequences (\x01)
    • UTF-8 escape sequences (\u{1F4A9})
      • Sequences allowed by RFC 2279 but not RFC 3629 (\u{D800}, \u{7FFFFFFF})
• Tables
  • Key-values / expression keys ({["foo"] = "bar"}, {[1234]="bar"})
  • Name-values / identifier keys ({foo = "bar"})
    • Identifier validation (Lua 5.4-style)
  • Values / implicit keys ({"bar"})
  • Mixed key types
  • Recursion depth limits

This library is not designed to replace Lua, nor execute arbitrary Lua code, so these Lua features are intentionally unsupported:

• Arithmetic operators (+, -, *, /...)
• Bitwise operators (<<, >>, &, |, ~...)
• Blocks and control structures (if, break, do, end, for, goto, repeat, until, while...)
• Comments
• Function calls
• Function definitions
• Length operator (#)
• Locale-specific behaviour (3,14159)
• Logical operators (and, or, not)
• Newline character normalisation in strings (\r\n => \n on UNIX, \n => \r\n on Windows)
• Parentheses, except for (0/0) (NaN)
• Pointers (light userdata)
• Referencing other variables (a = 10; b = a)
• Relational operators (==, ~=, <, >...)
• String concatenation ("hello" .. " world")
• Threads and coroutines
• Updating other variables (a = {}; a.b = 'foo')
• Userdata
• Vararg assignments and destructuring (a, b = 1, 2)
• Variable attributes and visibility modifiers (local <const> a = 10)

If you want to use these language features or otherwise need to run arbitrary Lua code, look at something like mlua, which links to liblua, and also provides serde bindings.

Known users of Lua serialisation

• Programming in Lua has a function that dumps a Lua table into a file as a script.

• SaveData: Love2D library, emits a return statement which is loaded by evaluating the string.

• Balatro (engine/string_packer.lua) is a modified version of SaveData that also compresses with deflate for save games and settings (.jkr files).

• World of Warcraft: addon state, written to WTF/{Account,SavedVariables}/**/*.lua as scripts that set variables.

License

This project is dual-licensed under the terms of the Apache-2.0 and MIT licenses.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this project by you, as defined in the Apache-2.0 license, shall be dual licensed under the terms of the Apache-2.0 and MIT licenses, without any additional terms or conditions.

serde_luaq does not include any copy of Lua itself. To ensure maximal compatibility with Lua's syntax, some of its test cases are derived from Lua's test suite.