//! `expry` is a fast schemaless binary format for data, with the ability for filtering and selecting parts of the information using expressions over the stored binary data. Expressions are compiled and optimized to bytecode, and during evaluation this bytecode is interpreted.
//! JSON can be read into these data structures (using `expry_parse_json`). Because `expry`
//! supports binary strings, not all `expry` values can be exported to JSON.
//!
//! The following data types are supported:
//! - `null`;
//! - `bool`;
//! - `int64`;
//! - `float`;
//! - `double`;
//! - `string` (can contain binary data);
//! - `array`;
//! - `object`.
//!
//!
//! # Expressions
//!
//! Expressions over binary objects are compiled to byte code, and can be transmitted over the network. This bytecode can be quickly evaluated, as the bytecode is optimized on the way the binary objects are stored on a low level. Keys inside objects are stored in an ordered way. If keys are not needed any more for evaluation of the current expression, they are permanently skipped.
//!
//! - Constants: `null`, `false`, `true`, numbers (including: `0x...`, `0b...`, `...f`), strings with `'string'` or `"string"` (including raw strings using `r#"..."#` with variable number of `#`, and hex strings with `x'hex'` and `x"hex"` and ).
//! - Conditionals: if-then-else is support with the conditional (ternary) operator: `cond ? then-part : else-part`.
//! - Object expression with `{` fields? `}`, with optional fields, individual fields separated by `,` (last entry can have a trailing `,`):
//! - fields can be `name: expr` with expr an expression or a primary data type (see above);
//! - fields can be `"name": expr` with expr an expression or a primary data type (see above);
//! - fields can be `field` (or `field???`), which copies the field from the `this` object;
//! - fields can be `(expr): expr` to dynamically generate a field name (must be a string), e.g. `{ (somefield): 42, }` (results in `{x: 42}` if `somefield` is `"x"`).
//! - Array expressions: `[` expressions? `]`, with the optional `expressions` which are one or more expressions delimited by `,` (last entry can have a trailing `,`). Example: `[ 42+42, 37+37, ]`.
//! - Arithmetic operators: `+`, `-`, `*`, `/`, `**` (pow), `%` (mod).
//! - Bitwise operators: `|`, `&`, `^` (xor), `<<`, `>>`
//! - Logical operators: `&&`, `||`, `a ? b : c`, `==`, `!=`, `>`, `>=`, `<`, `<=`.
//! - Special `this` field, to signify current object.
//! - Array subscription: `a[b]` (array subscription) is supported to access indiviual elements in arrays
//! - Length operator:
//! - `field.len()` which result in the length of the field if it is a string (number of bytes), array (number of elements), or object (number of key-values).
//! - String operators:
//! - `a .. b`: concatenate strings `a` and `b`;
//! - `a *= b`: true if string `a` contains with string `b`
//! - `a $= b`: true if string `a` ends with string `b`
//! - `a ^= b`: true if string `a` starts with string `b`
//! - Field operators:
//! - `a.b.c.d` is supported to access subfields;
//! - `a.get(b).get(c).get(d)` is supported for dynamic subfields;
//! - Error operators:
//! - try operator: `a ??? b`: if there is an error during evaluation of `a` (like an undefined field, or division by zero). The shorthand `a ???` is equivalent to `a ??? null`. Alternative syntax is `try(a, b)` and `try(a)`.
//! - not-null-else operator: `a ?? b`: if `a` is null, return `b`.
//! - `defined(x)` and `undefined(x)`: checks if value is (un)defined (such as a field lookup).
//! - String methods:
//! - `.trim()`
//! - `.lower()`
//! - `.upper()`
//! - `.hex()`
//! - `.htmlescape()`
//! - `.urlescape()`
//! - `.sub(int[, int])` (third argument is length of resulting string, default the max integer value)
//! - `.basename()`
//! - `.dirname()`
//! - `.splitn(max, split_on)` (split_on is the splitting string, max is the max number of elements in the resulting array), results in an array of strings
//! - Type methods:
//! - `.tostring()`
//! - `.toint()`
//! - `.tofloat()`
//! - `.todouble()`
//! - Methods taking a lambda function as argument. Lambda functions are specified as `|x| expr`
//! with `x` the argument to the lambda function.
//! - `array.filter(lambda)` filters array to only contain elements `e` where `lambda(e)` returns true (note that lambda should always return a boolean, otherwise the filter will fail)
//! - `array.map(lambda)` maps array of elements `e` to an array of elements `lambda(e)`
//! - `array.sort_by_key(lambda)` sorts array of elements `e` based on key `lambda(e)`. Sorting
//! of values is based on this type order: null, bool, int, float, double, string, array, object. If the type matches,
//! the sorting is performed on the contents itself. Note that sorting of NaN floats will yield
//! a dynamic error as this is not defined.
//! - `array.to_object(lamba)` converts the array to an object, in which the key is the first
//! value of the array returned by the lambda. The value is the second value of the array
//! returned by the lambda.
//! - `array.to_map(lamba)` creates a key - array of values object. The lambda returns [key,
//! sort, value].
//!
//!
//! # User defined functions
//!
//! During evaluation, there is also the possibility to add user defined functions. These user defined functions are only known at evaluation time, therefore there is no static type checking in place. Only runtime errors are generated if the number of arguments or type of arguments do not match.
//!
//!
//! # Easily convert custom data types to values
//!
//! Using the `From` trait, custom data types can be easily converted to values, even if they are
//! contained in data structures like `Vec`.
//!
//! ```
//! use expry::*;
//!
//! struct Foo {
//! foo: u32,
//! bar: bool,
//! }
//!
//! impl<'a> From<&'a Foo> for DecodedValue<'a> {
//! fn from(v: &'a Foo) -> Self {
//! value!({
//! "foo": v.foo as i64,
//! "bar": v.bar,
//! })
//! }
//! }
//!
//! let foos = vec![
//! Foo{foo:1,bar:true},
//! Foo{foo:2,bar:false},
//! ];
//! let encoded_value = value!({
//! "foo": Foo{foo:1,bar:true},
//! "foos": foos,
//! }).encode_to_vec();
//! ```
#![allow(dead_code)]
#![cfg_attr(not(feature = "std"), no_std)]
// useful to check binary size of parser
// cargo bloat -n 20 --bin bo-expr-parse --filter expry --profile minsize --features "std bin mini"
//
//
// Todo:
// - arr.find/position(|x| x.id == 12) to find a position in an array, returning null if not found
// - obj.map(|k,v| {id:k, ...v}) results in an (unsorted) array
// - arr.take(n) to take up to n items. If n is negative, take up to n items from the back of the
// list (reversing the order, so result[0] is the last object)
// - combine a to_array() + filter() + sort_by_key() + take() into one operation
extern crate alloc;
use alloc::format;
use alloc::vec;
use alloc::vec::Vec;
use alloc::string::String;
use alloc::boxed::Box;
use core::cmp::min;
use core::cmp::Ordering;
use core::fmt;
use core::convert::TryFrom;
use core::str::Chars;
use core::str::Utf8Error;
use std::collections::BTreeMap;
use std::fmt::Write;
pub mod memorypool;
pub mod raw_utils;
pub mod parser;
pub mod termcolors;
pub mod macros;
pub mod stringparser;
pub use crate::raw_utils::EncodingError;
pub use crate::memorypool::*;
pub use crate::raw_utils::*;
pub use crate::termcolors::*;
pub use crate::parser::*;
#[allow(unused_imports)]
pub use crate::stringparser::*;
const WIDTH_OF_JSON_TYPE_MASK: usize = 3;
const JSON_TYPE_MASK: usize = (1 << WIDTH_OF_JSON_TYPE_MASK) - 1;
const KEY_HASH_SIZE: usize = 1;
const MAX_DYNAMIC_VALUES: u8 = 64; // encoded as u8 in bytecode, so not easily changed
const MAX_RECURSION: u8 = 64;
const MAX_AST_DEPTH: u8 = 64;
const MAX_PARSER_DEPTH: usize = 128;
const MAX_FUNCTION_ARGS: usize = 16;
/// Described the different types of value `expry` supports.
#[derive(PartialEq, Eq)]
pub enum ValueType {
Null = 0, // length = 0
String = 1, // length = 0+
BoolFalse = 2, // length = 0
BoolTrue = 3, // length = 0
Float = 4, // length = 4 or 8 bytes of IEEE 754, length = 0 is used by the CreateObjectDiff algorithm to encode 'key is removed'.
Int = 5, // length = [0..]
Object = 6, // length = [0..]
Array = 7, // length = [0..]
}
pub const EXPRY_EMPTY_OBJECT: &[u8] = b"\x06";
pub const EXPRY_NULL: &[u8] = b"\x00";
impl ValueType {
pub fn type_string(&self) -> &'static str {
match self {
ValueType::Null => "null",
ValueType::BoolFalse => "bool-false",
ValueType::BoolTrue => "bool-true",
ValueType::Int => "int",
ValueType::Float => "float/double",
ValueType::String => "string",
ValueType::Object => "object",
ValueType::Array => "array",
}
}
}
/// Keys in Binary uses a simple 8-bit hash to speed up look ups.
pub type KeyHash = u8;
/// The keys in encoded expry objects.
// Keep this a type definition, not a struct. Struct makes tests::twitter_binary_parse_lazy
// increase in time from 175ms to 288ms in a tight loop. Seems to only affect amd64?
pub type Key<'a> = (KeyHash, &'a [u8]);
/// Returns empty key.
pub const fn key_empty() -> Key<'static> {
(0, b"")
}
/// The central data type of expry. It is modelled similar to JSON, however, contains a couple of
/// noteworthy differences.
///
/// The differences with JSON:
/// - There is an explicit integer type;
/// - Floats are either 32-bits or 64-bits;
/// - All strings are u8: they can contain both text or binary data.
///
/// To convert DecodedValue to JSON, one can use the formatting options like:
/// ```
/// use expry::*;
/// let value = value!({"foo": 42});
/// let output = format!("JSON output: {}", value);
/// assert_eq!(output, r#"JSON output: {"foo":42}"#);
/// ```
#[derive(PartialOrd, PartialEq, Clone, Default)]
pub enum DecodedValue<'a> {
#[default]
Null,
Bool(bool),
/// Signed 64-bits integer (that is packed into smaller value in the wire format if possible).
Int(i64),
/// 32-bits float type.
Float(f32),
/// 64-bits float type.
Double(f64),
/// Can be both binary data as regular strings. So not necessary UTF-8 (like Rust strings).
String(&'a [u8]), // not necessary UTF-8 (so in Rust `str` can not be used)
Array(DecodedArray<'a>),
Object(DecodedObject<'a>),
}
impl<'a> DecodedValue<'a> {
pub fn is_valid_json(&self) -> bool {
match self {
DecodedValue::Null => true,
DecodedValue::Bool(_) => true,
DecodedValue::Int(_) => true,
DecodedValue::Float(_) => true,
DecodedValue::Double(_) => true,
DecodedValue::String(s) => core::str::from_utf8(s).is_ok(),
DecodedValue::Object(m) => {
for ((_,k),v) in m {
if core::str::from_utf8(k).is_err() || !v.is_valid_json() {
return false;
}
}
true
},
DecodedValue::Array(a) => {
for v in a {
if !v.is_valid_json() {
return false;
}
}
true
},
}
}
}
impl<'a> From<&DecodedValue<'a>> for DecodedValue<'a> {
fn from(v: &DecodedValue<'a>) -> Self {
v.clone()
}
}
impl<'a, T: 'a> From<&'a Option<T>> for DecodedValue<'a>
where
DecodedValue<'a>: From<&'a T>,
{
fn from(v: &'a Option<T>) -> Self {
match v {
Some(v) => v.into(),
None => DecodedValue::Null,
}
}
}
impl<'a> From<&bool> for DecodedValue<'a> {
fn from(v: &bool) -> Self {
Self::Bool(*v)
}
}
impl<'a> From<&i64> for DecodedValue<'a> {
fn from(v: &i64) -> Self {
Self::Int(*v)
}
}
impl<'a> From<&f32> for DecodedValue<'a> {
fn from(v: &f32) -> Self {
Self::Float(*v)
}
}
impl<'a> From<&f64> for DecodedValue<'a> {
fn from(v: &f64) -> Self {
Self::Double(*v)
}
}
impl<'a, const N: usize> From<&'a [u8; N]> for DecodedValue<'a> {
fn from(v: &'a [u8; N]) -> Self {
Self::String(v)
}
}
impl<'a> From<&'a [u8]> for DecodedValue<'a> {
fn from(v: &'a [u8]) -> Self {
Self::String(v)
}
}
impl<'a> From<&'a mut [u8]> for DecodedValue<'a> {
fn from(v: &'a mut [u8]) -> Self {
Self::String(v)
}
}
impl<'a> From<&'a Vec<u8>> for DecodedValue<'a> {
fn from(v: &'a Vec<u8>) -> Self {
Self::String(v)
}
}
impl<'a> From<&'a str> for DecodedValue<'a> {
fn from(v: &'a str) -> Self {
Self::String(v.as_bytes())
}
}
impl<'a> From<&'a mut str> for DecodedValue<'a> {
fn from(v: &'a mut str) -> Self {
Self::String(v.as_bytes())
}
}
impl<'a, 'b> From<&'b &'a str> for DecodedValue<'a> {
fn from(v: &'b &'a str) -> Self {
Self::String(v.as_bytes())
}
}
impl<'a> From<&'a String> for DecodedValue<'a> {
fn from(v: &'a String) -> Self {
Self::String(v.as_bytes())
}
}
impl<'a,'b> From<&'b &'a String> for DecodedValue<'a> {
fn from(v: &'b &'a String) -> Self {
Self::String(v.as_bytes())
}
}
impl<'a> From<&'a BytecodeVec> for DecodedValue<'a> {
fn from(v: &'a BytecodeVec) -> Self {
Self::String(v.get())
}
}
impl<'a> From<&'_ BytecodeRef<'a>> for DecodedValue<'a> {
fn from(v: &'_ BytecodeRef<'a>) -> Self {
Self::String(v.get())
}
}
impl<'a, T: 'a, const N: usize> From<&'a &'a [T; N]> for DecodedValue<'a>
where
DecodedValue<'a>: From<&'a T>,
T: Clone,
{
fn from(v: &'a &'a [T; N]) -> Self {
let mut retval = DecodedArray::new();
for item in *v {
retval.push(item.into());
}
Self::Array(retval)
}
}
impl<'a, T> From<&'a &'a [T]> for DecodedValue<'a>
where
DecodedValue<'a>: From<&'a T>,
{
fn from(v: &'a &'a [T]) -> Self {
Self::Array(v.iter().map(|x| x.into()).collect())
}
}
impl<'a, T> From<&'a Vec<T>> for DecodedValue<'a>
where
DecodedValue<'a>: From<&'a T>,
{
fn from(v: &'a Vec<T>) -> Self {
Self::Array(v.iter().map(|x| x.into()).collect())
}
}
pub type DecodedObject<'a> = BTreeMap<Key<'a>, DecodedValue<'a>>;
pub type DecodedArray<'a> = Vec<DecodedValue<'a>>;
pub trait CloneInMemoryScope<'c, T> {
fn clone_in(&self, scope: &mut MemoryScope<'c>) -> T;
}
impl<'a, 'b, 'c> CloneInMemoryScope<'c, DecodedObject<'b>> for DecodedObject<'a> where 'c: 'b {
fn clone_in(&self, scope: &mut MemoryScope<'c>) -> DecodedObject<'b> {
let mut retval = DecodedObject::new();
for ((hash, k),v) in self {
retval.insert((*hash, scope.copy_u8(k)), v.clone_in(scope));
}
retval
}
}
impl<'a, 'b, 'c> CloneInMemoryScope<'c, DecodedArray<'b>> for DecodedArray<'a> where 'c: 'b {
fn clone_in(&self, scope: &mut MemoryScope<'c>) -> DecodedArray<'b> {
let mut retval = DecodedArray::new();
for v in self {
retval.push(v.clone_in(scope));
}
retval
}
}
impl<'a> fmt::Debug for DecodedValue<'a> {
fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
match self {
DecodedValue::Null => write!(f, "null"),
DecodedValue::Bool(false) => write!(f, "false"),
DecodedValue::Bool(true) => write!(f, "true"),
DecodedValue::Int(i) => write!(f, "{}", i),
DecodedValue::Float(d) => write!(f, "{}f", d),
DecodedValue::Double(d) => write!(f, "{}", d),
DecodedValue::String(s) => {
if let Ok(s) = core::str::from_utf8(s) {
write!(f, r#""{}""#, s)
} else {
write!(f, r#"{:?}"#, s)
}
}
DecodedValue::Object(obj) => {
write!(f, "{{")?;
let mut first = true;
for ((_,k),v) in obj {
if !first {
write!(f, ", ")?;
}
if let Ok(key) = core::str::from_utf8(k) {
write!(f, "{}: {:?}", key, v)?;
} else {
write!(f, "{:?}: {:?}", k, v)?;
}
first = false;
}
write!(f, "}}")
},
DecodedValue::Array(arr) => write!(f, "{:?}", arr),
}
}
}
fn json_escaped_write(f: &mut core::fmt::Formatter<'_>, src: &str) -> Result<(),core::fmt::Error> {
let mut utf16_buf = [0u16; 2];
for c in src.chars() {
match c {
'\x08' => write!(f, "\\b")?,
'\x0c' => write!(f, "\\f")?,
'\n' => write!(f, "\\n")?,
'\r' => write!(f, "\\r")?,
'\t' => write!(f, "\\t")?,
'"' => write!(f, "\\\"")?,
'\\' => write!(f, "\\\\")?,
' ' => write!(f, " ")?,
c if (c as u32) < 0x20 => {
let encoded = c.encode_utf16(&mut utf16_buf);
for utf16 in encoded {
write!(f, "\\u{:04X}", utf16)?;
}
},
c => write!(f, "{}", c)?,
}
}
Ok(())
}
impl<'a> core::fmt::Display for DecodedValue<'a> {
/// Silently ignores non-UTF8 data in strings. If an error is needed, use
/// [`DecodedValue::print_json`].
fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
match self {
DecodedValue::Null => write!(f, "null"),
DecodedValue::Bool(false) => write!(f, "false"),
DecodedValue::Bool(true) => write!(f, "true"),
DecodedValue::Int(i) => write!(f, "{}", i),
DecodedValue::Float(d) => write!(f, "{}", d),
DecodedValue::Double(d) => write!(f, "{}", d),
DecodedValue::String(s) => {
// because format! does no error handling, throwing an error on utf8
// problems will cause panics in format! with a decoded value. We want to avoid
// this, so we have to ignore non utf8 chars
let s = core::str::from_utf8(s);
if let Ok(s) = s {
write!(f, "\"")?;
json_escaped_write(f, s)?;
write!(f, "\"")
} else {
write!(f, "null")
}
}
DecodedValue::Object(obj) => {
write!(f, "{{")?;
let mut first = true;
for ((_,k),v) in obj {
// because format! does no error handling, throwing an error on utf8
// problems will cause panics in format! with a decoded value. We want to avoid
// this, so we have to ignore non utf8 chars
let key = core::str::from_utf8(k);
if let Ok(key) = key {
if !first {
write!(f, ",")?;
}
write!(f, "\"")?;
json_escaped_write(f, key)?;
write!(f, "\":{}", v)?;
first = false;
} else {
write!(f, "null")?;
}
}
write!(f, "}}")
},
DecodedValue::Array(arr) => {
write!(f, "[")?;
let mut first = true;
for v in arr {
if !first {
write!(f, ",")?;
}
write!(f, "{}", v)?;
first = false;
}
write!(f, "]")
},
}
}
}
/// Returns the hash for a name.
fn key_stable_hash(s: &[u8]) -> KeyHash {
let mut state = 0;
for (i,b) in s.iter().enumerate() {
state ^= b.wrapping_add(i as u8);
}
state
}
impl<'a> DecodedValue<'a> {
pub const fn type_string(&self) -> &'static str {
match self {
DecodedValue::Null => "null",
DecodedValue::Bool(_) => "bool",
DecodedValue::Int(_) => "int",
DecodedValue::Float(_) => "float",
DecodedValue::Double(_) => "double",
DecodedValue::String(_) => "string",
DecodedValue::Object(_) => "object",
DecodedValue::Array(_) => "array",
}
}
/// Returns the size needed if we serialize the Binary to a binary wire-format.
/// See [`DecodedValue::print_binary()`] for explanation of the `compact` argument.
pub fn size_of_binary(&self) -> usize {
let mut length_collector = RawWriterLength::new();
self.print_binary(&mut length_collector).unwrap_infallible();
length_collector.length()
}
/// Converts a Binary to a wire representation of that Binary. This wire representation can be sent
/// to other implementations as it is stable and 'endian safe'.
///
/// For objects we have a compact
/// representation and one that inserts hints how to process the wire-representation faster when
/// using an expression to evaluate the Binary to another value. This is controlled with the
/// `compact` argument.
///
/// This function expects that the `writer` has enough room. The needed room can be queried
/// beforehand by using the [`DecodedValue::size_of_binary`].
pub fn print_binary<E, Out: RawOutput<E>>(&self, writer: &mut Out) -> Result<(),E> {
match self {
DecodedValue::Null => {
writer.write_u8(ValueType::Null as u8)?;
Ok(())
}
DecodedValue::Bool(b) => {
if *b {
writer.write_u8(ValueType::BoolTrue as u8)?;
} else {
writer.write_u8(ValueType::BoolFalse as u8)?;
}
Ok(())
}
DecodedValue::Int(i) => {
if *i == 0 {
writer.write_u8(ValueType::Int as u8)?;
return Ok(());
}
let size = size_of_i64(*i);
debug_assert!(size <= 8);
writer.write_u8((size << WIDTH_OF_JSON_TYPE_MASK) as u8 | ValueType::Int as u8)?;
writer.write_i64(*i, size)?;
Ok(())
}
DecodedValue::Float(f) => {
writer.write_u8((4 << WIDTH_OF_JSON_TYPE_MASK) as u8 | ValueType::Float as u8)?;
writer.write_f32(*f)?;
Ok(())
}
DecodedValue::Double(d) => {
writer.write_u8((8 << WIDTH_OF_JSON_TYPE_MASK) as u8 | ValueType::Float as u8)?;
writer.write_f64(*d)?;
Ok(())
}
DecodedValue::String(s) => {
writer.write_var_u64((s.len() << WIDTH_OF_JSON_TYPE_MASK) as u64 | ValueType::String as u64)?;
writer.write_bytes(s)?;
Ok(())
}
DecodedValue::Array(arr) => {
if arr.is_empty() {
writer.write_u8(ValueType::Array as u8)?;
return Ok(());
}
write_with_header(writer, |writer,total| {
writer.write_var_u64(total << WIDTH_OF_JSON_TYPE_MASK | ValueType::Array as u64)
}, |writer| {
writer.write_var_u64((arr.len() as u64) << 1)?;
for e in arr {
e.print_binary(writer)?;
}
Ok(())
})
}
DecodedValue::Object(obj) => {
write_binary_object(writer, &mut obj.iter(), |writer, x: &DecodedValue| x.print_binary(writer))?;
Ok(())
}
}
}
/// Converts a Binary to a wire representation of that Binary, in a self contained [`EncodedValueVec`] unsing [`DecodedValue::print_binary`].
pub fn encode_to_vec(&self) -> EncodedValueVec {
if cfg!(feature = "mini") {
let mut writer = RawString::new();
self.print_binary(&mut writer).expect("expry: problem during actual write");
EncodedValueVec(writer.data)
} else {
let len = self.size_of_binary();
let mut retval : Vec<u8> = vec![0u8; len];
let mut writer = RawWriter::with(&mut retval[..]);
self.print_binary(&mut writer).expect("expry: calculated size differs from actual size");
debug_assert!(writer.left() == 0);
EncodedValueVec(retval)
}
}
/// Converts a Binary to a wire representation of that Binary, in a self contained [`EncodedValueVec`] unsing [`DecodedValue::print_binary`].
pub fn encode_to_scope<'c>(&self, scope: &mut MemoryScope<'c>) -> EncodedValueRef<'c> {
if cfg!(feature = "mini") {
// FIXME: convert to delayed deallocation function of MemoryPool (to avoid move/copy)
let raw = self.encode_to_vec().0;
let raw = scope.move_object(raw);
EncodedValueRef(raw)
} else {
let len = self.size_of_binary();
let retval = scope.alloc(len);
let mut writer = RawWriter::with_uninit(retval);
self.print_binary(&mut writer).expect("expry: calculated size differs from actual size");
debug_assert!(writer.left() == 0);
EncodedValueRef(writer.build())
}
}
/// Convert a Binary value to a JSON output.
///
/// Note that some Binary values can not be encoded, because not all strings can be encoded as UTF-8. If this is the case, this method will result in a [`Utf8Error`]. If non-UTF8 data needs to be ignored, use `format!("{}", decoded_value)` or similar methods.
pub fn print_json(&self, writer: &mut String) -> Result<(),Utf8Error> {
// not using a helper to calculate size, as this forces numbers to be converted twice.
match self {
DecodedValue::Null => {
writer.push_str("null");
}
DecodedValue::Bool(b) => {
if *b {
writer.push_str("true");
} else {
writer.push_str("false");
}
}
DecodedValue::Int(i) => {
write!(writer, "{}", i).unwrap();
}
DecodedValue::Float(f) => {
write!(writer, "{:.99}", f).unwrap();
}
DecodedValue::Double(d) => {
write!(writer, "{:.99}", d).unwrap();
}
DecodedValue::String(s) => {
writer.push('\"');
json_escape(core::str::from_utf8(s)?, writer);
writer.push('\"');
}
DecodedValue::Array(arr) => {
writer.push('[');
let mut comma = false;
for e in arr {
if comma {
writer.push(',');
}
e.print_json(writer)?;
comma = true;
}
writer.push(']');
}
DecodedValue::Object(obj) => {
writer.push('{');
let mut comma = false;
for ((_,k),v) in obj {
if comma {
writer.push(',');
}
writer.push('\"');
json_escape(core::str::from_utf8(k)?, writer);
writer.push_str("\":");
v.print_json(writer)?;
comma = true;
}
writer.push('}');
}
};
Ok(())
}
fn lookup_field<'k>(&self, key: Key<'k>) -> Option<&DecodedValue<'a>> where 'k: 'a {
if let DecodedValue::Object(values) = self {
values.get(&key)
} else {
None
}
}
pub fn easy(&'a self) -> Easy<'a> {
Easy(Some(self))
}
}
// from https://stackoverflow.com/questions/27859822/is-it-possible-to-have-stack-allocated-arrays-with-the-size-determined-at-runtim
enum SmallVector<T, const N: usize> {
Inline(usize, [T; N]),
Dynamic(Vec<T>),
}
impl<T: Copy + Clone, const N: usize> SmallVector<T, N> {
fn new(v: T, n: usize) -> Self {
if n <= N {
Self::Inline(n, [v; N])
} else {
Self::Dynamic(vec![v; n])
}
}
}
impl<T, const N: usize> SmallVector<T, N> {
fn as_slice(&self) -> &[T] {
match self {
Self::Inline(n, array) => &array[0..*n],
Self::Dynamic(vec) => vec,
}
}
fn as_mut_slice(&mut self) -> &mut [T] {
match self {
Self::Inline(n, array) => &mut array[0..*n],
Self::Dynamic(vec) => vec,
}
}
}
use std::ops::{Deref, DerefMut};
impl<T, const N: usize> Deref for SmallVector<T, N> {
type Target = [T];
fn deref(&self) -> &Self::Target {
self.as_slice()
}
}
impl<T, const N: usize> DerefMut for SmallVector<T, N> {
fn deref_mut(&mut self) -> &mut Self::Target {
self.as_mut_slice()
}
}
/// WARNING: iterator should be sorted by Key
fn write_binary_object<'a, 'b, T, E, Iter: ExactSizeIterator + Iterator<Item=(&'b Key<'a>,&'b T)>, Out: RawOutput<E>, B>(writer: &mut Out, sorted_fields: &mut Iter, to_binary: B) -> Result<(),E>
where
'a: 'b,
T: 'b,
B: Fn(&mut Out, &T) -> Result<(),E>,
{
let count = sorted_fields.len();
if count == 0 {
writer.write_u8(ValueType::Object as u8)?;
return Ok(());
}
if !cfg!(feature = "mini") && count > 4 {
write_with_header(writer, |writer,total| {
writer.write_var_u64(total << WIDTH_OF_JSON_TYPE_MASK | ValueType::Object as u64)
}, |writer| {
// two seperate vectors to avoid internal padding (currently: usize + u8 = 9 bytes -> 7 bytes padding)
let mut positions = SmallVector::<_,32>::new(0, count); // start_pos in binary
let mut hashes = SmallVector::<_,32>::new(0, count);
for (i,((hash,key), value)) in sorted_fields.enumerate() {
positions[i] = writer.pos();
hashes[i] = *hash;
let key_size = KEY_HASH_SIZE + key.len();
writer.write_var_u64((key_size as u64) << 1)?;
writer.write_u8(*hash)?;
writer.write_bytes(key)?;
to_binary(writer, value)?;
}
// in reverse order pop jumps and insert headers with the new construct
for i in (0..count).rev() {
for j in (0..i.trailing_zeros()).rev() {
let to_index = i + (2 << j);
if to_index == 0 || to_index >= count {
continue;
}
// checks if the entry before has the same hash, if so, jumping should no be done, so skip this jump
if hashes[to_index-1] == hashes[to_index] {
continue;
}
let from_pos = positions[i];
let to_pos = positions[to_index];
let jump = to_pos - from_pos;
let current = writer.pos();
writer.write_var_u64(((jump as u64) << 1) | 1)?;
writer.write_u8(hashes[to_index])?;
let extra = writer.pos() - current;
writer.swap_range(from_pos, current)?;
// HINT: can be made a bit faster if we also store the from_index (part of the loop can be skipped)
for i in 1..count { // 1 because the position of the first element is never changed nor used
let pos = &mut positions[i];
if *pos > from_pos { // using '>': a jump points to new jump entries (so the next jump can quickly be taken)
// using '>=': a jump points to an actual entry
*pos += extra;
}
}
}
}
Ok(())
})?;
return Ok(());
}
write_with_header(writer, |writer,total| {
writer.write_var_u64(total << WIDTH_OF_JSON_TYPE_MASK | ValueType::Object as u64)
}, |writer| {
for ((hash,key), value) in sorted_fields {
let key_size = KEY_HASH_SIZE + key.len();
writer.write_var_u64((key_size as u64) << 1)?;
writer.write_u8(*hash)?;
writer.write_bytes(key)?;
to_binary(writer, value)?;
}
Ok(())
})
}
#[derive(PartialEq)]
pub struct Easy<'a>(pub Option<&'a DecodedValue<'a>>);
impl<'a> Easy<'a> {
pub fn lookup_field<'k>(&self, key: Key<'k>) -> Easy<'a> where 'k: 'a {
if let Some(DecodedValue::Object(values)) = self.0 {
let values : &'a _ = values;
Easy(values.get(&key))
} else {
Easy(None)
}
}
pub fn as_string(&self) -> Option<&'a str> {
if let Some(DecodedValue::String(v)) = self.0 {
core::str::from_utf8(v).ok()
} else {
None
}
}
pub fn as_array(&self) -> Option<&'a [DecodedValue<'a>]> {
if let Some(DecodedValue::Array(v)) = self.0 {
Some(v)
} else {
None
}
}
pub fn iter(&self) -> std::slice::Iter<'a, DecodedValue<'a>> {
if let Some(DecodedValue::Array(v)) = self.0 {
v.iter()
} else {
[].iter()
}
}
}
/*
pub trait EasyInterfaceDecodedValue<'a> {
fn lookup_field<'k>(&self, key: Key<'k>) -> Option<&DecodedValue<'a>> where 'k: 'a;
fn as_string(&self) -> Option<&'a str>;
fn as_array(&self) -> Option<&'a [DecodedValue<'a>]>;
}
impl<'a> EasyInterfaceDecodedValue<'a> for DecodedValue<'a> {
fn lookup_field<'k>(&self, key: Key<'k>) -> Option<&DecodedValue<'a>> where 'k: 'a {
if let DecodedValue::Object(values) = self {
values.get(&key)
} else {
None
}
}
fn as_string(&self) -> Option<&'a str> {
if let DecodedValue::String(v) = self {
core::str::from_utf8(v).ok()
} else {
None
}
}
fn as_array(&self) -> Option<&'a [DecodedValue<'a>]> {
if let DecodedValue::Array(v) = self {
Some(v)
} else {
None
}
}
}
impl<'a> EasyInterfaceDecodedValue<'a> for Option<&DecodedValue<'a>> {
fn lookup_field<'k>(&self, key: Key<'k>) -> Option<&DecodedValue<'a>> where 'k: 'a {
self.map_or(None, |v| v.lookup_field(key))
}
fn as_string(&self) -> Option<&'a str> {
self.map_or(None, |v| v.as_string())
}
fn as_array(&self) -> Option<&'a [DecodedValue<'a>]> {
self.map_or(None, |v| v.as_array())
}
}
*/
impl<'a, 'b, 'c> CloneInMemoryScope<'c, DecodedValue<'b>> for DecodedValue<'a> where 'c: 'b {
fn clone_in(&self, scope: &mut MemoryScope<'c>) -> DecodedValue<'b> {
match self {
DecodedValue::Null => DecodedValue::Null,
DecodedValue::Bool(v) => DecodedValue::Bool(*v),
DecodedValue::Int(v) => DecodedValue::Int(*v),
DecodedValue::Float(v) => DecodedValue::Float(*v),
DecodedValue::Double(v) => DecodedValue::Double(*v),
DecodedValue::String(s) => DecodedValue::String(scope.copy_u8(s)),
DecodedValue::Object(values) => {
let mut retval = DecodedObject::new();
for ((hash, k),v) in values {
retval.insert((*hash, scope.copy_u8(k)), v.clone_in(scope));
}
DecodedValue::Object(retval)
},
DecodedValue::Array(values) => {
let mut retval = DecodedArray::new();
for v in values {
retval.push(v.clone_in(scope));
}
DecodedValue::Array(retval)
}
}
}
}
fn json_escape(src: &str, output: &mut String) {
for c in src.chars() {
match c {
'\x08' => output.push_str("\\b"),
'\x0c' => output.push_str("\\f"),
'\n' => output.push_str("\\n"),
'\r' => output.push_str("\\r"),
'\t' => output.push_str("\\t"),
'"' => output.push_str("\\\""),
'\\' => output.push_str("\\\\"),
' ' => output.push(' '),
// c.is_ascii_graphic()
c if (c as u32) < 0x20 => {
let mut utf16_buf = [0u16; 2];
let encoded = c.encode_utf16(&mut utf16_buf);
for utf16 in encoded {
write!(output, "\\u{:04X}", utf16).unwrap();
}
},
c => {
let mut utf8_buf = [0u8; 4];
output.push_str(c.encode_utf8(&mut utf8_buf));
},
}
}
}
/// Calculates the key as used in this crate from a str. There is also a `[u8]` version.
pub fn key_str(key: &str) -> Key<'_> {
let hash = key_stable_hash(key.as_bytes());
(hash, key.as_bytes())
}
/// Calculates the key as used in this crate from an u8 slice. There is also a `str` version.
pub fn key_u8(key: &[u8]) -> Key<'_> {
let hash = key_stable_hash(key);
(hash, key)
}
impl TryFrom<usize> for ValueType {
type Error = EncodingError;
fn try_from(v: usize) -> Result<Self, Self::Error> {
match v {
x if x == ValueType::Null as usize => Ok(ValueType::Null),
x if x == ValueType::String as usize => Ok(ValueType::String),
x if x == ValueType::BoolFalse as usize => Ok(ValueType::BoolFalse),
x if x == ValueType::BoolTrue as usize => Ok(ValueType::BoolTrue),
x if x == ValueType::Float as usize => Ok(ValueType::Float),
x if x == ValueType::Int as usize => Ok(ValueType::Int),
x if x == ValueType::Object as usize => Ok(ValueType::Object),
x if x == ValueType::Array as usize => Ok(ValueType::Array),
_ => Err(EncodingError { line_nr: line!() }),
}
}
}
impl<'a> DecodedValue<'a> {
pub fn decoded_type_of(type_and_length: u64) -> Result<ValueType, EncodingError> {
((type_and_length as usize) & JSON_TYPE_MASK).try_into()
}
/// Based on a variable length integer read for the wire, return the [`ValueType`] raw number.
pub const fn type_of_binary(type_and_length: u64) -> u8 {
((type_and_length as usize) & JSON_TYPE_MASK) as u8
}
/// Based on a variable length integer read for the wire, return the length of the current
/// entry.
pub const fn length_of_binary(type_and_length: u64) -> usize {
(type_and_length >> WIDTH_OF_JSON_TYPE_MASK) as usize
}
pub fn decoded_type_and_length(type_and_length: u64) -> Result<(ValueType, usize), EncodingError> {
Ok((Self::decoded_type_of(type_and_length)?, Self::length_of_binary(type_and_length)))
}
/// Parse the wire-format.
pub fn decode<'b>(reader: &mut RawReader<'a>) -> Result<DecodedValue<'b>, EncodingError> where 'a: 'b {
let type_and_length = reader.read_var_u64()?;
let (t, len) = Self::decoded_type_and_length(type_and_length)?;
match t {
ValueType::Null => {
if len == 0 {
Ok(DecodedValue::Null)
} else {
Err(EncodingError { line_nr: line!() })
}
}
ValueType::String => {
let value = reader.read_bytes(len)?;
Ok(DecodedValue::String(value))
},
ValueType::BoolFalse => {
if len == 0 {
Ok(DecodedValue::Bool(false))
} else {
Err(EncodingError { line_nr: line!() })
}
}
ValueType::BoolTrue => {
if len == 0 {
Ok(DecodedValue::Bool(true))
} else {
Err(EncodingError { line_nr: line!() })
}
}
ValueType::Float => {
if len == 4 {
let value = reader.read_f32()?;
Ok(DecodedValue::Float(value))
} else if len == 8 {
let value = reader.read_f64()?;
Ok(DecodedValue::Double(value))
} else {
Err(EncodingError { line_nr: line!() })
}
}
ValueType::Int => {
let v = reader.read_i64(len)?;
Ok(DecodedValue::Int(v))
}
ValueType::Array => {
if len == 0 {
return Ok(DecodedValue::Array(vec![]));
}
let mut subreader = RawReader::with(reader.read_bytes(len)?);
let mut count = subreader.read_var_u64()?;
count >>= 1;
let mut retval : Vec<DecodedValue> = Vec::with_capacity(min(16384, count as usize)); // limit the max allocation to prevent rouge input data leading to out of memory problems
for _ in 0 .. count {
let v = DecodedValue::decode(&mut subreader)?;
retval.push(v);
}
Ok(DecodedValue::Array(retval))
},
ValueType::Object => {
if len == 0 {
return Ok(DecodedValue::Object(DecodedObject::new()));
}
let mut last : Key = (0, b"");
let mut retval = DecodedObject::new();
let mut subreader = RawReader::with(reader.read_bytes(len)?);
while !subreader.is_empty() {
let mut key_length = subreader.read_var_u64()?;
if key_length == 0 {
return Err(EncodingError{ line_nr: line!() });
}
if key_length & 0x1 != 0 {
subreader.read_u8()?;
continue;
}
key_length = (key_length >> 1) - 1; // - 1 to ignore hash (we handle it differently)
let hash = subreader.read_u8()?;
let key = (hash, subreader.read_bytes(key_length as usize)?);
//let hash = binary_object_stable_hash(&key);
let v = DecodedValue::decode(&mut subreader)?;
retval.insert(key, v);
if key < last {
// object not sorted, that is an encoding error
return Err(EncodingError{line_nr: line!(), });
}
last = key;
}
debug_assert!(subreader.is_empty());
Ok(DecodedValue::Object(retval))
},
}
}
}
#[derive(PartialEq, Copy, Clone)]
pub enum LazyDecodedValue<'a> {
Null,
Bool(bool),
Int(i64),
Float(f32),
Double(f64),
String(&'a [u8]), // ugly in Rust, because a String is forced to be Unicode
Object(LazyDecodedObject<'a>),
Array(LazyDecodedArray<'a>),
}
#[derive(PartialEq, Eq, Copy, Clone)]
pub struct LazyDecodedObject<'a> {
reader: RawReader<'a>,
}
impl<'a> LazyDecodedObject<'a> {
fn _read_skip_list(reader: &mut RawReader<'a>) -> Result<(), EncodingError> {
while !reader.is_empty() && reader.clone().read_var_u64()? & 0x1 != 0 {
reader.read_var_u64()?;
reader.read_u8()?;
}
Ok(())
}
fn _read_value(reader: &mut RawReader<'a>) -> Result<&'a [u8],EncodingError> {
let mut reader_type = *reader;
let type_and_length = reader_type.read_var_u64()?;
let len = DecodedValue::length_of_binary(type_and_length);
reader_type.skip(len)?;
let v = reader.read_bytes(reader.len() - reader_type.len())?;
Ok(v)
}
fn _read_key(reader: &mut RawReader<'a>) -> Result<Key<'a>,EncodingError> {
Self::_read_skip_list(reader)?;
let mut key_length = reader.read_var_u64()?;
if key_length == 0 || key_length & 0x1 != 0 {
// skip lists should have been read away by the the read_skip_list()
return Err(EncodingError{ line_nr: line!() });
}
key_length = (key_length >> 1) - 1; // - 1 to ignore hash (we handle it differently)
let hash = reader.read_u8()?;
let key = reader.read_bytes(key_length as usize)?;
Ok((hash, key))
}
fn _read(reader: &mut RawReader<'a>) -> Result<(Key<'a>, &'a [u8]),EncodingError> {
Ok((Self::_read_key(reader)?, Self::_read_value(reader)?))
}
pub fn lookup_binary(&self, key: Key<'_>) -> Result<Option<EncodedValueRef<'a>>,EncodingError> {
let mut value_reader = self.reader;
match evaluate_seek(key.0, key.1, &mut value_reader) {
Ok(_) => {},
Err(EvalError::FieldNotFound(_)) => return Ok(None),
Err(_) => return Err(EncodingError{ line_nr: line!(), }),
}
let binary_value = Self::_read(&mut value_reader)?.1;
Ok(Some(EncodedValueRef(binary_value)))
}
pub fn lookup_value(&self, key: Key<'_>) -> Result<Option<LazyDecodedValue<'a>>,EncodingError> {
let binary_value = self.lookup_binary(key)?;
if let Some(binary_value) = binary_value {
Ok(Some(expry_decode_lazy(binary_value.get())?))
} else {
Ok(None)
}
}
pub fn lookup_decoded_value(&self, key: Key<'_>) -> Result<Option<DecodedValue<'a>>,EncodingError> {
let binary_value = self.lookup_binary(key)?;
if let Some(binary_value) = binary_value {
Ok(Some(expry_decode(binary_value.get())?))
} else {
Ok(None)
}
}
pub fn get(&mut self) -> Result<(Key<'a>,LazyDecodedValue<'a>),EncodingError> {
let (key, value) = LazyDecodedObject::<'a>::_read(&mut self.reader)?;
Ok((key, expry_decode_lazy(value)?))
}
pub fn get_raw(&mut self) -> Result<(Key<'a>,EncodedValueRef<'a>),EncodingError> {
let (key, value) = LazyDecodedObject::<'a>::_read(&mut self.reader)?;
Ok((key, EncodedValueRef(value)))
}
pub fn is_empty(&mut self) -> Result<bool,EncodingError> {
Self::_read_skip_list(&mut self.reader)?;
Ok(self.reader.is_empty())
}
pub fn new(reader: RawReader<'a>) -> Self {
Self {
reader,
}
}
}
#[derive(PartialEq, Eq, Copy, Clone)]
pub struct LazyDecodedArray<'a> {
reader: RawReader<'a>,
count: u64,
}
impl<'a> LazyDecodedArray<'a> {
pub fn get(&mut self) -> Result<LazyDecodedValue<'a>,EncodingError> {
if self.count == 0 {
return Err(EncodingError{ line_nr: line!() });
}
debug_assert!(!self.reader.is_empty());
self.count -= 1;
LazyDecodedValue::decode(&mut self.reader)
}
pub fn get_raw(&mut self) -> Result<EncodedValueRef<'a>,EncodingError> {
if self.count == 0 {
return Err(EncodingError{line_nr: line!()});
}
self.count -= 1;
let type_and_length = self.reader.clone().read_var_u64()?;
let len = DecodedValue::length_of_binary(type_and_length);
Ok(EncodedValueRef(self.reader.read_bytes(len + size_of_var_u64(type_and_length))?))
}
pub fn skip(&mut self, count: usize) -> Result<(),EncodingError> {
for _ in 0..count {
self.get_raw()?;
}
Ok(())
}
pub fn remaining(&self) -> u64 {
self.count
}
pub fn is_empty(&self) -> bool {
self.count == 0
}
}
impl<'a> LazyDecodedValue<'a> {
pub fn type_string(&self) -> &'static str {
match self {
LazyDecodedValue::Null => "null",
LazyDecodedValue::Bool(_) => "bool",
LazyDecodedValue::Int(_) => "int",
LazyDecodedValue::Float(_) => "float",
LazyDecodedValue::Double(_) => "double",
LazyDecodedValue::String(_) => "string",
LazyDecodedValue::Object(_) => "object",
LazyDecodedValue::Array(_) => "array",
}
}
pub fn decode(reader: &mut RawReader<'a>) -> Result<LazyDecodedValue<'a>, EncodingError> {
let type_and_length = reader.read_var_u64()?;
let (t, len) = DecodedValue::decoded_type_and_length(type_and_length)?;
match t {
ValueType::Null => {
if len == 0 {
Ok(LazyDecodedValue::Null)
} else {
Err(EncodingError { line_nr: line!() })
}
}
ValueType::String => {
let value = reader.read_bytes(len)?;
Ok(LazyDecodedValue::String(value))
},
ValueType::BoolFalse => {
if len == 0 {
Ok(LazyDecodedValue::Bool(false))
} else {
Err(EncodingError { line_nr: line!() })
}
}
ValueType::BoolTrue => {
if len == 0 {
Ok(LazyDecodedValue::Bool(true))
} else {
Err(EncodingError { line_nr: line!() })
}
}
ValueType::Float => {
if len == 4 {
let value = reader.read_f32()?;
Ok(LazyDecodedValue::Float(value))
} else if len == 8 {
let value = reader.read_f64()?;
Ok(LazyDecodedValue::Double(value))
} else {
Err(EncodingError { line_nr: line!() })
}
}
ValueType::Int => {
let v = reader.read_i64(len)?;
Ok(LazyDecodedValue::Int(v))
}
ValueType::Array => {
if len == 0 {
return Ok(LazyDecodedValue::Array(LazyDecodedArray { count: 0, reader: RawReader::with(b""), }));
}
let mut subreader = RawReader::with(reader.read_bytes(len)?);
let mut count = subreader.read_var_u64()?;
count >>= 1;
let retval = LazyDecodedArray {
count,
reader: subreader,
};
Ok(LazyDecodedValue::Array(retval))
},
ValueType::Object => {
if len == 0 {
return Ok(LazyDecodedValue::Object(LazyDecodedObject { reader: RawReader::with(b""), }));
}
Ok(LazyDecodedValue::Object(LazyDecodedObject::new(RawReader::with(reader.read_bytes(len)?))))
},
}
}
}
/// Parses a JSON input to a decoded value.
pub fn expry_parse_json<'a,'b,'c>(reader: &'a str, scope: &'_ mut MemoryScope<'c>) -> Result<DecodedValue<'b>, CompileError<'b>> where 'a: 'b, 'c: 'b {
DecodedValue::parse_json(reader, scope)
}
/// Parses an expry expression input to a decoded value. This involves compiling and evaluating the input. For more flexibility and re-use of compiled expression, see [`expry_compile`] and [`expry_eval`].
pub fn expry_parse_expression<'a,'b,'c>(reader: &'a str, scope: &'_ mut MemoryScope<'c>) -> Result<DecodedValue<'b>, CompileError<'b>> where 'a: 'b, 'c: 'b {
DecodedValue::parse_expression_to_value(reader, scope)
}
/// Parses an encoded value to a decoded value.
pub fn expry_decode<'a,'b>(reader: &'a [u8]) -> Result<DecodedValue<'b>, EncodingError> where 'a: 'b {
let mut reader = RawReader::with(reader);
let retval = DecodedValue::decode(&mut reader);
if retval.is_ok() {
debug_assert_eq!(0, reader.len());
}
retval
}
/// Parses an encoded value to a value that can be easily decoded on the go.
pub fn expry_decode_lazy<'a,'b>(reader: &'a [u8]) -> Result<LazyDecodedValue<'b>, EncodingError> where 'a: 'b {
let mut reader = RawReader::with(reader);
let retval = LazyDecodedValue::decode(&mut reader);
if retval.is_ok() {
debug_assert_eq!(0, reader.len());
}
retval
}
/// Reference to a expry value.
#[derive(PartialEq, Eq, Copy, Clone, Debug)]
pub struct EncodedValueRef<'a>(pub &'a [u8]);
impl<'a> EncodedValueRef<'a> {
fn lookup_field(&self, key: Key) -> Result<Option<DecodedValue<'a>>,EncodingError> {
if self.get().is_empty() {
return Ok(None);
}
let mut value_reader = RawReader::with(self.get());
if DecodedValue::type_of_binary(value_reader.read_var_u64()?) != ValueType::Object as u8 {
return Err(EncodingError{line_nr: line!()});
}
match evaluate_seek(key.0, key.1, &mut value_reader) {
Ok(_) => {},
Err(EvalError::FieldNotFound(_)) => return Ok(None),
Err(_) => return Err(EncodingError{line_nr: line!()}),
}
let key_length = value_reader.read_var_u64()?;
if key_length == 0 || (key_length & 0x1) != 0 {
return Err(EncodingError{line_nr: line!()});
}
let _key_name = &value_reader.read_bytes(key_length as usize >> 1)?[1..];
Ok(Some(DecodedValue::decode(&mut value_reader)?))
}
pub fn get(&self) -> &'a [u8] {
self.0
}
pub fn is_empty(&self) -> bool {
self.0.is_empty()
}
pub fn new() -> Self {
Self(b"")
}
pub fn to_vec(&self) -> EncodedValueVec {
EncodedValueVec(self.0.to_vec())
}
}
impl<'a> Default for EncodedValueRef<'a> {
fn default() -> Self {
Self::new()
}
}
impl<'a> std::ops::Deref for EncodedValueRef<'a> {
type Target = [u8];
fn deref(&self) -> &'a Self::Target {
self.0
}
}
/// Self-contained expry value (in encoded form).
#[derive(PartialEq, Eq, PartialOrd, Ord, Clone, Debug)]
pub struct EncodedValueVec(pub Vec<u8>);
impl EncodedValueVec {
pub fn new() -> Self {
Self(Vec::new())
}
pub fn len(&self) -> usize {
self.0.len()
}
pub fn to_ref(&self) -> EncodedValueRef {
EncodedValueRef(&self.0)
}
pub fn get(&self) -> &[u8] {
&self.0
}
pub fn is_empty(&self) -> bool {
self.0.is_empty()
}
}
impl Default for EncodedValueVec {
fn default() -> Self {
Self::new()
}
}
impl std::ops::Deref for EncodedValueVec {
type Target = [u8];
fn deref(&self) -> &Self::Target {
&self.0[..]
}
}
impl<'a> From<&'a EncodedValueVec> for EncodedValueRef<'a> {
fn from(v: &'a EncodedValueVec) -> Self {
EncodedValueRef(&v.0)
}
}
/// Reference to expry expression bytecode.
#[derive(PartialEq, Eq, Copy, Clone, Debug)]
pub struct BytecodeRef<'a>(pub &'a [u8]);
impl<'a> BytecodeRef<'a> {
pub fn get(&self) -> &'a [u8] {
self.0
}
pub fn is_empty(&self) -> bool {
self.0.is_empty()
}
pub fn new() -> Self {
Self(b"")
}
pub fn to_vec(&self) -> BytecodeVec {
BytecodeVec(self.0.to_vec())
}
}
impl<'a> Default for BytecodeRef<'a> {
fn default() -> Self {
Self::new()
}
}
impl<'a> std::ops::Deref for BytecodeRef<'a> {
type Target = [u8];
fn deref(&self) -> &'a Self::Target {
self.0
}
}
/// Self-contained expry expression bytecode.
#[derive(PartialEq, Eq, Clone, Debug)]
pub struct BytecodeVec(pub Vec<u8>);
impl BytecodeVec {
pub fn new() -> Self {
Self(Vec::new())
}
pub fn len(&self) -> usize {
self.0.len()
}
pub fn to_ref(&self) -> BytecodeRef {
BytecodeRef(&self.0)
}
pub fn get(&self) -> &[u8] {
&self.0
}
pub fn is_empty(&self) -> bool {
self.0.is_empty()
}
}
impl Default for BytecodeVec {
fn default() -> Self {
Self::new()
}
}
impl std::ops::Deref for BytecodeVec {
type Target = [u8];
fn deref(&self) -> &Self::Target {
&self.0[..]
}
}
// we can use the .into() method to convert (but this is an explicit action)
impl Into<EncodedValueVec> for Vec<u8> {
fn into(self) -> EncodedValueVec {
EncodedValueVec(self)
}
}
impl Into<BytecodeVec> for Vec<u8> {
fn into(self) -> BytecodeVec {
BytecodeVec(self)
}
}
// safe of misuse to convert from value/bytecode to vec
impl Into<Vec<u8>> for EncodedValueVec {
fn into(self) -> Vec<u8> {
self.0
}
}
impl Into<Vec<u8>> for BytecodeVec {
fn into(self) -> Vec<u8> {
self.0
}
}
// reads away skip lists
// returns current key, entry length, and the payload type+length
fn read_key_and_entry<'a>(reader: &'_ mut RawReader<'a>) -> Result<(Key<'a>,&'a [u8],usize),EncodingError> {
while !reader.is_empty() {
let mut key_length = reader.read_var_u64()?;
if key_length == 0 {
return Err(EncodingError{ line_nr: line!() });
}
if (key_length & 0x1) != 0 {
let _hash_of_next_entry = reader.read_u8()?; // read hash of forward reference
continue;
}
key_length = (key_length >> 1) - KEY_HASH_SIZE as u64;
let hash = reader.read_u8()?;
let key = reader.read_bytes(key_length as usize)?;
let type_and_length = reader.clone().read_var_u64()?;
let payload = reader.read_bytes(size_of_var_u64(type_and_length) + DecodedValue::length_of_binary(type_and_length))?;
return Ok(((hash, key), payload, type_and_length as usize));
}
Ok((Key::default(), b"", 0))
}
/// Merges two encoded objects. Left hand side take preference (so the left hand side value of keys occuring in
/// both is taken).
pub fn merge_objects_to_scope<'b, 'c>(lhs: EncodedValueRef, rhs: EncodedValueRef, allocator: &mut MemoryScope<'c>) -> Result<EncodedValueRef<'b>, EncodingError> where 'c: 'b {
let mut lhs_reader = RawReader::with(lhs.get());
let mut rhs_reader = RawReader::with(rhs.get());
/*
// Simple version with higher memory usage
let lhs = DecodedValue::parse(&mut lhs_reader)?;
let rhs = DecodedValue::parse(&mut rhs_reader)?;
if let (DecodedValue::Object(mut lhs),DecodedValue::Object(rhs)) = (lhs, rhs) {
for (k,v) in rhs {
lhs.entry(k).or_insert(v);
}
Ok(DecodedValue::Object(lhs).to_scope(allocator, true))
} else {
Err(EncodingError{ line_nr: line!() })
}
*/
let lhs_length_and_type = lhs_reader.read_var_u64()?;
if DecodedValue::type_of_binary(lhs_length_and_type) != ValueType::Object as u8 {
return Err(EncodingError{ line_nr: line!() });
}
if DecodedValue::length_of_binary(lhs_length_and_type) != lhs_reader.len() {
return Err(EncodingError{ line_nr: line!() });
}
let rhs_length_and_type = rhs_reader.read_var_u64()?;
if DecodedValue::type_of_binary(rhs_length_and_type) != ValueType::Object as u8 {
return Err(EncodingError{ line_nr: line!() });
}
if DecodedValue::length_of_binary(rhs_length_and_type) != rhs_reader.len() {
return Err(EncodingError{ line_nr: line!() });
}
let mut chunks : ScopedArrayBuilder<'_, 'c, _> = ScopedArrayBuilder::new(allocator);
let (mut lhs_key,mut lhs_value,mut lhs_value_type_and_length) : (Key,&[u8],usize) = read_key_and_entry(&mut lhs_reader)?;
let (mut rhs_key,mut rhs_value,_) : (Key,&[u8],usize) = read_key_and_entry(&mut rhs_reader)?;
while !lhs_value.is_empty() || !rhs_value.is_empty() {
if !lhs_value.is_empty() && lhs_key <= rhs_key || rhs_value.is_empty() {
// special 'remove' marker as generated by CreateObjectDiff
if lhs_value_type_and_length != ValueType::Float as usize { // float of 0 bytes implies a size of 0
chunks.push((lhs_key, lhs_value));
}
// if key is in both, skip the rhs one
if lhs_key == rhs_key {
(rhs_key, rhs_value, _) = read_key_and_entry(&mut rhs_reader)?;
}
(lhs_key, lhs_value, lhs_value_type_and_length) = read_key_and_entry(&mut lhs_reader)?;
} else {
chunks.push((rhs_key, rhs_value));
(rhs_key, rhs_value, _) = read_key_and_entry(&mut rhs_reader)?;
}
}
let sorted_values = chunks.build();
let mut writer = RawScopedArrayBuilder::new(allocator);
write_binary_object(&mut writer, &mut sorted_values.iter().map(|(x,y)| (x,y)), |writer: &mut _, x: &&[u8]| writer.write_bytes(x)).unwrap_infallible();
Ok(EncodedValueRef(writer.build()))
}
#[derive(Eq, PartialEq, Copy, Clone, Debug)]
pub enum SimpleType {
Bool,
Int,
Float,
Double,
String,
}
#[derive(PartialEq, Copy, Clone, Debug)]
enum Number {
Float(f32),
Double(f64),
Int(i64),
}
#[derive(PartialEq, Copy, Clone, Debug)]
enum JSONToken<'a> {
End(),
Bool(bool),
String(&'a [u8]),
PositiveNumber(Number),
NegativeNumber(Number), // needed for parsing 2-1 so that it is recognized as 2 - 1
Comma(),
ArrayOpen(),
ArrayClose(),
ObjectOpen(),
Colon(),
ObjectClose(),
NullObject(),
Type(SimpleType),
Comparison(ExpryComparison),
Arithmetic(Arithmetic),
Logical(Logical),
Bitwise(Bitwise),
Open(),
Close(),
As(),
Not(),
Conditional(),
NullOperator(),
TryOperator(),
Concat(),
Dot(),
Spread(),
Field(&'a str),
FunctionCall(&'a str),
}
#[derive(Debug, Copy, Clone, PartialEq, Eq)]
pub struct LexerError {
line_nr: u32,
}
#[derive(Debug, Copy, Clone, PartialEq, Eq)]
pub enum CompileErrorDescription<'a> {
NoToken,
ParserTooLong,
UnknownField(&'a str),
Lexer(LexerError),
Parser(&'a str),
Optimizer(EvalError<'a>),
Types(TypeError<'a>),
}
#[derive(Clone, Debug)]
pub struct CompileError<'a> {
pub expr: &'a str,
pub error: CompileErrorDescription<'a>,
pub start: usize, // counted from the back of the input string
pub end: usize, // counted from the back of the input string
pub extra: Option<(usize, usize)>,
}
impl<'a> CompileError<'a> {
pub fn error_start(&self) -> usize {
self.start
}
pub fn error_end(&self) -> usize {
self.end
}
pub fn error(&self) -> CompileErrorDescription<'a> {
self.error
}
pub fn extra(&self) -> Option<(usize, usize)> {
self.extra
}
}
impl<'a> From<TypeError<'a>> for CompileErrorDescription<'a> {
fn from(t: TypeError<'a>) -> Self {
CompileErrorDescription::Types(t)
}
}
impl<'a> core::fmt::Display for CompileErrorDescription<'a> {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
CompileErrorDescription::NoToken => write!(f, "could not tokenize remaining value"),
CompileErrorDescription::Lexer(err) => write!(f, "{:?} (in lexer)", err),
CompileErrorDescription::Parser(msg) => write!(f, "{}", msg),
CompileErrorDescription::Optimizer(msg) => write!(f, "{} (during optimizing)", msg),
CompileErrorDescription::Types(msg) => write!(f, "{} (during optimizing)", msg),
CompileErrorDescription::UnknownField(msg) => write!(f, "unknown field '{}'", msg),
CompileErrorDescription::ParserTooLong => write!(f, "parser recursion too high"),
}
}
}
fn get_codepoint_from_hex(chars: &mut Chars) -> Result<u32,LexerError> {
let mut values = [None; 4];
values.fill_with(|| chars.next().and_then(|x| from_hex(x).ok()));
if let [Some(first), Some(second), Some(third), Some(fourth)] = values {
let unicode = (first as u32) << 12 | (second as u32) << 8 | (third as u32) << 4 | (fourth as u32);
Ok(unicode)
} else {
Err(LexerError{ line_nr: line!() })
}
}
fn json_unescape<'b>(str: &'_ str, allocator: &'_ mut MemoryScope<'b>, extended: bool) -> Result<&'b [u8], (LexerError,usize)> {
// see spec http://www.ietf.org/rfc/rfc4627.txt; section 2.5:
let mut retval = ScopedStringBuilder::with_capacity(allocator, str.len());
let mut chars = str.chars();
let mut err_pos = 0;
while let Some(c) = chars.next() {
if c != '\\' {
err_pos += 1;
retval.push_char(c);
continue;
}
err_pos += 2;
let c = match chars.next() {
Some('b') => '\x08',
Some('f') => '\x0c',
Some('n') => '\n',
Some('r') => '\r',
Some('t') => '\t',
Some('\\') => '\\',
Some('"') => '"',
Some('\'') if extended => '\'',
Some('u') => {
let unicode = get_codepoint_from_hex(&mut chars).map_err(|x| (x,err_pos))?;
err_pos += 4;
if unicode <= 0x7F {
(unicode & 0x7F) as u8 as char
} else if unicode <= 0x7FF {
char::from_u32(unicode).ok_or((LexerError{ line_nr: line!()}, err_pos))?
} else if (0xD800..=0xDBFF).contains(&unicode) {
// surrogate pair
if let (Some('\\'), Some('u')) = (chars.next(), chars.next()) {
} else {
return Err((LexerError{ line_nr: line!()}, err_pos));
}
let unicode2 = get_codepoint_from_hex(&mut chars).map_err(|x| (x, err_pos))?;
err_pos += 4;
let u = ((unicode & 0x3FF) << 10 | (unicode2 & 0x3FF)) + 0x10000;
char::from_u32(u).ok_or((LexerError{ line_nr: line!()}, err_pos))?
} else {
char::from_u32(unicode).ok_or((LexerError{ line_nr: line!()}, err_pos))?
}
},
_ => return Err((LexerError{ line_nr: line!()}, err_pos)),
};
retval.push_char(c);
}
Ok(retval.build().as_bytes())
}
fn json_tokenize<'b,'c>(reader: &mut &'b str, context: &mut JSONParserContext<'_,'b,'c>) -> Result<(JSONToken<'b>,usize), (CompileErrorDescription<'b>,usize,usize)>
where 'c: 'b
{
// might be faster using https://github.com/maciejhirsz/logos
*reader = reader.trim_start();
let remaining = reader.len();
if reader.is_empty() {
return Ok((JSONToken::End(), remaining));
}
if reader.accept("null") {
return Ok((JSONToken::NullObject(), remaining));
}
if reader.accept("true") {
return Ok((JSONToken::Bool(true), remaining));
}
if reader.accept("false") {
return Ok((JSONToken::Bool(false), remaining));
}
if reader.accept(",") {
return Ok((JSONToken::Comma(), remaining));
}
if reader.accept("[") {
return Ok((JSONToken::ArrayOpen(), remaining));
}
if reader.accept("]") {
return Ok((JSONToken::ArrayClose(), remaining));
}
if reader.accept("{") {
return Ok((JSONToken::ObjectOpen(), remaining));
}
if reader.accept(":") {
return Ok((JSONToken::Colon(), remaining));
}
if reader.accept("}") {
return Ok((JSONToken::ObjectClose(), remaining));
}
let mut string_matcher = StringLiteralSpanner::new('"');
if let Some(value) = string_matcher.span(reader) {
debug_assert!(value.len() >= 2);
if string_matcher.unescape_needed {
let err = |x,offset| (CompileErrorDescription::Lexer(x), remaining-1-offset,reader.len());
let unescaped = json_unescape(&value[1..value.len()-1], context.allocator, false).map_err(|(x,offset)| err(x, offset))?;
return Ok((JSONToken::String(unescaped), remaining));
}
return Ok((JSONToken::String(value[1..value.len()-1].as_bytes()), remaining));
}
// FIXME:
// 0 is a special case; numbers can not start with 0 execpt if it is the number zero or a real number with only digits after the comma
// FIXME: -0000000.1 is not accounted for (is not allowed)
// -0 is allowed!
let mut number_matcher = NumberSpanner::new();
if let Some(value) = number_matcher.span(reader) {
let err = || (CompileErrorDescription::Lexer(LexerError{ line_nr: line!() }), remaining,reader.len());
if !number_matcher.float {
// given the number spanner, we know that value is UTF8
let number : i64 = value.parse().map_err(|_| err())?;
if number < 0 {
return Ok((JSONToken::NegativeNumber(Number::Int(number)), remaining));
} else {
return Ok((JSONToken::PositiveNumber(Number::Int(number)), remaining));
}
}
// given the number spanner, we know that value is UTF8
let number : f64 = value.parse().map_err(|_| err())?;
if number < 0.0 {
return Ok((JSONToken::NegativeNumber(Number::Double(number)), remaining));
} else {
return Ok((JSONToken::PositiveNumber(Number::Double(number)), remaining));
}
}
Err((CompileErrorDescription::NoToken, remaining,reader.len()))
}
struct JSONParserContext<'a,'b,'c> {
allocator: &'a mut MemoryScope<'c>,
static_value: &'a DecodedObject<'b>,
allow_unbound_variables: Option<u8>,
scopes: Vec<&'a str>, // names of variables
depth_left: usize,
}
type JSONParserState<'b,'a,'c> = ParserState<'b,JSONToken<'b>,CompileErrorDescription<'b>, JSONParserContext<'a,'b,'c>>;
type JSONParserResult<'b,T> = ParserResult<T,CompileErrorDescription<'b>>;
fn json_expr<'b>(parser: &mut JSONParserState<'b,'_,'_>) -> JSONParserResult<'b,DecodedValue<'b>> {
match parser.get()? {
(JSONToken::NullObject(),_) => Ok(DecodedValue::Null),
(JSONToken::Bool(b),_) => Ok(DecodedValue::Bool(b)),
(JSONToken::PositiveNumber(n),_) | (JSONToken::NegativeNumber(n),_) => {
match n {
Number::Int(n) => Ok(DecodedValue::Int(n)),
// strictly not part of the JSON spec, so depends of the lexer produces a float. This is used in the predicate parsing
Number::Float(n) => Ok(DecodedValue::Float(n)),
Number::Double(n) => Ok(DecodedValue::Double(n)),
}
},
(JSONToken::String(b),_) => Ok(DecodedValue::String(b)),
(JSONToken::ObjectOpen(), open) => {
let mut retval = DecodedObject::new();
parser.opt(|parser| {
json_object_entry(parser, &mut retval)?;
parser.repeat(|parser| {
parser.accept(JSONToken::Comma(), None, || CompileErrorDescription::Parser(""))?;
json_object_entry(parser, &mut retval)?;
Ok(true)
})
})?;
parser.accept(JSONToken::ObjectClose(), Some(&open), || CompileErrorDescription::Parser("expected '}' to close object"))?;
Ok(DecodedValue::Object(retval))
},
(JSONToken::ArrayOpen(), open) => {
let mut retval : Vec<DecodedValue<'b>> = Vec::new();
parser.opt(|parser| {
json_array_entry(parser, &mut retval)?;
parser.repeat(|parser| {
parser.accept(JSONToken::Comma(), None, || CompileErrorDescription::Parser(""))?;
json_array_entry(parser, &mut retval)?;
Ok(true)
})
})?;
parser.accept(JSONToken::ArrayClose(), Some(&open), || CompileErrorDescription::Parser("expected ']' to close array"))?;
Ok(DecodedValue::Array(retval))
}
(token,info) => Err(parser.error_token(token, info, |_| CompileErrorDescription::Parser("expected primary"))),
}
}
fn json_object_entry<'b>(parser: &mut JSONParserState<'b,'_,'_>, value: &mut DecodedObject<'b>) -> JSONParserResult<'b,()> {
match parser.get()? {
(JSONToken::String(k),_) => {
parser.accept(JSONToken::Colon(), None, || CompileErrorDescription::Parser("expected ':' as field delimeter for an object field"))?;
let v = json_expr(parser)?;
value.insert(key_u8(k), v);
Ok(())
},
(token,info) => Err(parser.error_token(token, info, |_| CompileErrorDescription::Parser("expected string as object field"))),
}
}
fn json_array_entry<'b>(parser: &mut JSONParserState<'b,'_,'_>, value: &mut Vec<DecodedValue<'b>>) -> JSONParserResult<'b,()> {
let v = json_expr(parser)?;
value.push(v);
Ok(())
}
fn json_top_level<'b>(parser: &mut JSONParserState<'b,'_,'_>) -> JSONParserResult<'b,DecodedValue<'b>> {
let retval = json_expr(parser)?;
parser.accept(JSONToken::End(), None, || CompileErrorDescription::Parser("expected end of input (possibly an operator was missing, or a '{'/'[')"))?;
Ok(retval)
}
impl<'a> DecodedValue<'a> {
pub fn parse_json<'b,'c>(reader: &'a str, allocator: &'_ mut MemoryScope<'c>) -> Result<DecodedValue<'b>,CompileError<'b>> where 'c: 'b, 'a: 'b {
let static_value = DecodedObject::new();
let mut parser = JSONParserState::new_with(reader, json_tokenize, JSONParserContext{allocator, static_value: &static_value, scopes: vec![], allow_unbound_variables: None, depth_left: MAX_PARSER_DEPTH });
parser.parse(json_top_level, CompileErrorDescription::Parser("unexpected token"), CompileErrorDescription::Parser("too much parser recursion")).map_err(|(error, start, end, extra)| CompileError{ expr: reader, error, start, end, extra })
}
pub fn parse_expression_to_value<'c>(reader: &'a str, scope: &'_ mut MemoryScope<'c>) -> Result<DecodedValue<'a>,CompileError<'a>> where 'c: 'a {
let (bytecode,_) = expry_compile_expr(reader, None, None, &[], scope)?;
expry_eval(bytecode, &mut Vec::new(), scope).map_err(|err| err.into())
}
}
#[derive(PartialEq, Eq, Copy, Clone, Debug)]
enum ExpryComparison {
Equal,
NotEqual,
LessEqual,
GreaterEqual,
Less,
Greater,
Contains,
StartsWith,
EndsWith,
}
impl ExpryComparison {
fn reverse(&self) -> Option<ExpryComparison> {
match self {
ExpryComparison::LessEqual => Some(ExpryComparison::GreaterEqual),
ExpryComparison::Less => Some(ExpryComparison::Greater),
ExpryComparison::GreaterEqual => Some(ExpryComparison::LessEqual),
ExpryComparison::Greater => Some(ExpryComparison::Less),
ExpryComparison::Equal => Some(ExpryComparison::Equal),
ExpryComparison::NotEqual => Some(ExpryComparison::NotEqual),
_ => None,
}
}
}
#[derive(PartialEq, Eq, Copy, Clone, Debug)]
enum Arithmetic {
Add,
Sub,
Mul,
Div,
Mod,
#[cfg(feature = "power")]
Pow,
}
impl Arithmetic {
fn to_bytecode(&self) -> ExpryBytecode {
match self {
Arithmetic::Add => ExpryBytecode::ArithmeticPlus,
Arithmetic::Sub => ExpryBytecode::ArithmeticMin,
Arithmetic::Mul => ExpryBytecode::ArithmeticMul,
Arithmetic::Div => ExpryBytecode::ArithmeticDiv,
Arithmetic::Mod => ExpryBytecode::ArithmeticMod,
#[cfg(feature = "power")]
Arithmetic::Pow => ExpryBytecode::ArithmeticPower,
}
}
}
#[derive(PartialEq, Eq, Copy, Clone, Debug)]
enum Logical {
And,
Or,
}
#[derive(PartialEq, Eq, Copy, Clone, Debug)]
enum Bitwise {
And,
Or,
Xor,
ShiftLeft,
ShiftRight,
}
impl Bitwise {
fn to_bytecode(&self) -> ExpryBytecode {
match self {
Bitwise::And => ExpryBytecode::ArithmeticBitwiseAnd,
Bitwise::Or => ExpryBytecode::ArithmeticBitwiseOr,
Bitwise::Xor => ExpryBytecode::ArithmeticBitwiseXor,
Bitwise::ShiftLeft => ExpryBytecode::ArithmeticShiftLeft,
Bitwise::ShiftRight => ExpryBytecode::ArithmeticShiftRight,
}
}
}
fn expry_tokenize<'b,'c>(reader: &mut &'b str, context: &mut JSONParserContext<'_,'b,'c>) -> Result<(JSONToken<'b>,usize), (CompileErrorDescription<'b>,usize,usize)> where 'c: 'b,
{
// might be faster using https://github.com/maciejhirsz/logos
loop {
*reader = reader.trim_start();
let start_of_comment = reader.len();
if reader.accept("/*") {
if let Some((_comment, tail)) = reader.split_once("*/") {
*reader = tail;
} else {
return Err((CompileErrorDescription::Lexer(LexerError{ line_nr: line!()}), start_of_comment,reader.len()))
}
continue;
}
if reader.accept("//") {
if let Some((_comment, tail)) = reader.split_once('\n') {
*reader = tail;
} else {
*reader = "";
}
continue;
}
break;
}
let remaining = reader.len();
if reader.is_empty() {
return Ok((JSONToken::End(), remaining));
}
// parse hex values 0x
if reader.accept("0x") {
if let Some(value) = reader.span_fn(&mut |x| from_hex(x).is_ok()) {
let err = || (CompileErrorDescription::Lexer(LexerError{ line_nr: line!()}), remaining,reader.len());
return Ok((JSONToken::PositiveNumber(Number::Int(i64::from_str_radix(value, 16).map_err(|_| err())?)), remaining));
} else {
return Err((CompileErrorDescription::Lexer(LexerError{ line_nr: line!()}), remaining,reader.len()));
}
}
// parse binary values 0b
if reader.accept("0b") {
if let Some(value) = reader.span_fn(&mut |x| x == '0' || x == '1') {
let err = || (CompileErrorDescription::Lexer(LexerError{ line_nr: line!()}), remaining,reader.len());
return Ok((JSONToken::PositiveNumber(Number::Int(i64::from_str_radix(value, 2).map_err(|_| err())?)), remaining));
} else {
return Err((CompileErrorDescription::Lexer(LexerError{ line_nr: line!()}), remaining,reader.len()));
}
}
// parse single precision floats which have the suffix 'f' (this is not part of the JSON spec)
let mut reader_copy = *reader;
let mut number_matcher = NumberSpanner::new();
if let Some(value) = number_matcher.span(&mut reader_copy) {
if reader_copy.accept("f") {
*reader = reader_copy;
let err = || (CompileErrorDescription::Lexer(LexerError{ line_nr: line!()}), remaining,reader.len());
// given the number spanner, we know that value is UTF8
let number : f32 = value.parse().map_err(|_| err())?;
if number < 0.0 {
return Ok((JSONToken::NegativeNumber(Number::Float(number)), remaining));
} else {
return Ok((JSONToken::PositiveNumber(Number::Float(number)), remaining));
}
}
}
if reader.accept("bool") {
return Ok((JSONToken::Type(SimpleType::Bool), remaining));
}
if reader.accept("int") {
return Ok((JSONToken::Type(SimpleType::Int), remaining));
}
if reader.accept("float") {
return Ok((JSONToken::Type(SimpleType::Float), remaining));
}
if reader.accept("double") {
return Ok((JSONToken::Type(SimpleType::Double), remaining));
}
if reader.accept("string") {
return Ok((JSONToken::Type(SimpleType::String), remaining));
}
// call regular JSON lexer
match json_tokenize(reader, context) {
Ok(v) => return Ok(v),
Err((CompileErrorDescription::NoToken,_,_)) => {},
Err(err) => return Err(err),
}
// parse single quoted strings
let mut string_matcher = StringLiteralSpanner::new('\'');
if let Some(value) = string_matcher.span(reader) {
debug_assert!(value.len() >= 2);
if string_matcher.unescape_needed {
let err = || (CompileErrorDescription::Lexer(LexerError{ line_nr: line!()}), remaining,reader.len());
return Ok((JSONToken::String(json_unescape(&value[1..value.len()-1], context.allocator, true).map_err(|_| err())?),remaining));
}
return Ok((JSONToken::String(value[1..value.len()-1].as_bytes()),remaining));
}
// parse strings starting with 'x"'
let mut alt = false;
if reader.accept("x\"") || (alt = true, reader.accept("x'")).1 {
if let Some((hex,tail)) = reader.split_once(if alt { "'" } else { "\"" }) {
*reader = tail;
let err = || (CompileErrorDescription::Lexer(LexerError{ line_nr: line!()}), remaining,reader.len());
return Ok((JSONToken::String(context.allocator.copy_unhex(hex.as_bytes()).map_err(|_| err())?),remaining));
} else {
return Err((CompileErrorDescription::Lexer(LexerError{ line_nr: line!()}), remaining,reader.len()));
}
}
// parse strings starting with `r##"..."##`
if reader.starts_with('r') && reader.len() >= 2 {
let count = reader[1..].chars().take_while(|x| *x == '#').count();
if let Some(x) = reader.chars().nth(1+count) {
if x == '"' || x == '\'' {
let delim : String = format!("{}{}", x, &reader[1..1+count]);
*reader = &reader[2+count..];
if let Some((string,tail)) = reader.split_once(&delim) {
*reader = tail;
return Ok((JSONToken::String(string.as_bytes()), remaining));
} else {
return Err((CompileErrorDescription::Lexer(LexerError{ line_nr: line!()}), remaining,reader.len()));
}
}
}
}
if reader.accept("$=") {
return Ok((JSONToken::Comparison(ExpryComparison::EndsWith), remaining));
}
// parse functions with '(' after (with a variable string spanner)
// parse fields (with a variable string spanner) (handle `this` differently)
let mut matcher = |c: char| c == '$' || c == '_' || c == '\\' || c.is_ascii_alphanumeric();
if let Some(value) = reader.span_fn(&mut matcher) {
// field name or function call (peek at next)
if !reader.is_empty() && reader.starts_with('(') {
return Ok((JSONToken::FunctionCall(value), remaining));
} else {
return Ok((JSONToken::Field(value), remaining));
}
}
if reader.accept("<=") {
return Ok((JSONToken::Comparison(ExpryComparison::LessEqual), remaining));
}
if reader.accept("<<") {
return Ok((JSONToken::Bitwise(Bitwise::ShiftLeft), remaining));
}
if reader.accept("<") {
return Ok((JSONToken::Comparison(ExpryComparison::Less), remaining));
}
if reader.accept(">=") {
return Ok((JSONToken::Comparison(ExpryComparison::GreaterEqual), remaining));
}
if reader.accept(">>") {
return Ok((JSONToken::Bitwise(Bitwise::ShiftRight), remaining));
}
if reader.accept(">") {
return Ok((JSONToken::Comparison(ExpryComparison::Greater), remaining));
}
if reader.accept("==") {
return Ok((JSONToken::Comparison(ExpryComparison::Equal), remaining));
}
if reader.accept("!=") {
return Ok((JSONToken::Comparison(ExpryComparison::NotEqual), remaining));
}
if reader.accept("!") {
return Ok((JSONToken::Not(), remaining));
}
#[cfg(feature = "power")]
if reader.accept("**") {
return Ok((JSONToken::Arithmetic(Arithmetic::Pow), remaining));
}
if reader.accept("*=") {
return Ok((JSONToken::Comparison(ExpryComparison::Contains), remaining));
}
if reader.accept("^=") {
return Ok((JSONToken::Comparison(ExpryComparison::StartsWith), remaining));
}
if reader.accept("&&") {
return Ok((JSONToken::Logical(Logical::And), remaining));
}
if reader.accept("||") {
return Ok((JSONToken::Logical(Logical::Or), remaining));
}
if reader.accept("&") {
return Ok((JSONToken::Bitwise(Bitwise::And), remaining));
}
if reader.accept("|") {
return Ok((JSONToken::Bitwise(Bitwise::Or), remaining));
}
if reader.accept("^") {
return Ok((JSONToken::Bitwise(Bitwise::Xor), remaining));
}
if reader.accept("(") {
return Ok((JSONToken::Open(), remaining));
}
if reader.accept(")") {
return Ok((JSONToken::Close(), remaining));
}
if reader.accept("???") {
return Ok((JSONToken::TryOperator(), remaining));
}
if reader.accept("??") {
return Ok((JSONToken::NullOperator(), remaining));
}
if reader.accept("?") {
return Ok((JSONToken::Conditional(), remaining));
}
if reader.accept("...") {
return Ok((JSONToken::Spread(), remaining));
}
if reader.accept("..") {
return Ok((JSONToken::Concat(), remaining));
}
if reader.accept(".") {
return Ok((JSONToken::Dot(), remaining));
}
if reader.accept("+") {
return Ok((JSONToken::Arithmetic(Arithmetic::Add), remaining));
}
if reader.accept("-") {
return Ok((JSONToken::Arithmetic(Arithmetic::Sub), remaining));
}
if reader.accept("*") {
return Ok((JSONToken::Arithmetic(Arithmetic::Mul), remaining));
}
if reader.accept("/") {
return Ok((JSONToken::Arithmetic(Arithmetic::Div), remaining));
}
if reader.accept("%") {
return Ok((JSONToken::Arithmetic(Arithmetic::Mod), remaining));
}
Err((CompileErrorDescription::NoToken, remaining,reader.len()))
}
type ExpryParserState<'a,'b,'c> = JSONParserState<'a,'b,'c>;
type ExpryParserResult<'b, T> = ParserResult<T,CompileErrorDescription<'b>>;
#[derive(Debug, Copy, Clone)]
enum UnaryOperation {
Not,
Negate,
}
#[derive(Eq, PartialEq, Debug, Copy, Clone)]
enum ValueSource {
Computed, // can be either one of the others, but that is only dynamically known
Value(u8), // u8 signifies which input
Expression,
}
#[derive(Debug, Copy, Clone)]
struct ASTProperties {
needs_dynamic_eval: bool,
#[cfg(not(feature = "mini"))]
source: ValueSource,
}
impl ASTProperties {
fn constant() -> Self {
Self {
needs_dynamic_eval: false,
#[cfg(not(feature = "mini"))]
source: ValueSource::Expression,
}
}
#[allow(unused_variables)] // for feature = "mini"
fn input(value_source: u8) -> Self {
Self {
needs_dynamic_eval: true,
#[cfg(not(feature = "mini"))]
source: ValueSource::Value(value_source),
}
}
}
#[derive(Debug)]
enum ExpryAST<'b> {
Constant(DecodedValue<'b>),
All(u8),
ObjectAccessWithExpr(&'b mut ExpryAST<'b>, &'b mut ExpryAST<'b>, ASTProperties),
ArrayAccessWithExpr(&'b mut ExpryAST<'b>, &'b mut ExpryAST<'b>, ASTProperties),
Comparison(&'b mut ExpryAST<'b>, ExpryComparison, &'b mut ExpryAST<'b>, ASTProperties),
Logical(&'b mut ExpryAST<'b>, Logical, &'b mut ExpryAST<'b>, ASTProperties),
Function(&'b str, &'b mut [ExpryAST<'b>], ASTProperties),
Bytecode(ExpryBytecode, &'b mut [ExpryAST<'b>], ASTProperties),
BytecodeWrap(ExpryBytecode, &'b mut [ExpryAST<'b>], ASTProperties),
Object(&'b mut [(&'b mut ExpryAST<'b>, &'b mut ExpryAST<'b>)], ASTProperties),
MergeObjects(&'b mut ExpryAST<'b>, &'b mut ExpryAST<'b>, ASTProperties),
Array(&'b mut [ExpryAST<'b>], ASTProperties),
Lambda(&'b mut ExpryAST<'b>, u8, ASTProperties), // usize is number of arguments expected
}
impl<'b> Default for ExpryAST<'b> {
fn default() -> Self {
Self::Constant(DecodedValue::Null)
}
}
fn fast_path(ast: &ExpryAST) -> bool {
match ast {
ExpryAST::Constant(_) => true,
ExpryAST::All(_) => true,
ExpryAST::ObjectAccessWithExpr(lhs, _, _) => fast_path(lhs),
ExpryAST::ArrayAccessWithExpr(lhs, _, _) => fast_path(lhs),
_ => false,
}
}
fn ast_properties<'b>(ast: &ExpryAST<'b>) -> ASTProperties {
match ast {
ExpryAST::Constant(_) => ASTProperties::constant(),
ExpryAST::All(value_index) => ASTProperties::input(*value_index),
ExpryAST::ObjectAccessWithExpr(_, _, prop) => *prop,
ExpryAST::ArrayAccessWithExpr(_, _, prop) => *prop,
ExpryAST::Comparison(_, _, _, prop) => *prop,
ExpryAST::Logical(_, _, _, prop) => *prop,
ExpryAST::Function(_, _, prop) => *prop,
ExpryAST::Bytecode(_, _, prop) => *prop,
ExpryAST::BytecodeWrap(_, _, prop) => *prop,
ExpryAST::Object(_, prop) => *prop,
ExpryAST::Array(_, prop) => *prop,
ExpryAST::MergeObjects(_, _, prop) => *prop,
ExpryAST::Lambda(_, _, prop) => *prop,
}
}
fn ast_properties_plus(x: &ASTProperties, y: &ASTProperties) -> ASTProperties {
ASTProperties {
needs_dynamic_eval: x.needs_dynamic_eval || y.needs_dynamic_eval,
#[cfg(not(feature = "mini"))]
source: ValueSource::Computed,
}
}
fn ast_properties_plus_ast<'a>(x: &ExpryAST<'a>, y: &ExpryAST<'a>) -> ASTProperties {
ast_properties_plus(&ast_properties(x), &ast_properties(y))
}
fn ast_properties_combine<'a>(args: &[ExpryAST<'a>]) -> ASTProperties {
args.iter().fold(ASTProperties::constant(), |x,y| {
let y = ast_properties(y);
ast_properties_plus(&x,&y)
})
}
fn ast_properties_iter<'a,'b, Iter>(args: Iter) -> ASTProperties
where Iter: Iterator<Item=&'b &'b ExpryAST<'a>>, 'a: 'b
{
args.fold(ASTProperties::constant(), |x,y| {
let y = ast_properties(y);
ASTProperties {
needs_dynamic_eval: x.needs_dynamic_eval || y.needs_dynamic_eval,
#[cfg(not(feature = "mini"))]
source: ValueSource::Computed,
}
})
}
fn expry_parse_primary<'b,'c>(parser: &mut ExpryParserState<'b,'_,'c>) -> ExpryParserResult<'b, ExpryAST<'b>> where 'c: 'b {
match parser.get()? {
(JSONToken::Not(), _info) => {
let lhs = parser.call(expry_parse_primary_object_access)?;
let lhs_prop = ast_properties(&lhs);
let args = parser.context().allocator.slice_from_iter([lhs].into_iter());
Ok(ExpryAST::Bytecode(ExpryBytecode::Not, args, lhs_prop))
},
(JSONToken::Arithmetic(Arithmetic::Sub), _info) => {
let info = parser.token_info()?;
let lhs = parser.call(expry_parse_primary_object_access)?;
match lhs {
ExpryAST::Constant(DecodedValue::Double(number)) => {
return Ok(ExpryAST::Constant(DecodedValue::Double(-number)));
},
ExpryAST::Constant(DecodedValue::Float(number)) => {
return Ok(ExpryAST::Constant(DecodedValue::Float(-number)));
},
ExpryAST::Constant(DecodedValue::Int(number)) => {
let value = number.checked_neg();
if let Some(value) = value {
return Ok(ExpryAST::Constant(DecodedValue::Int(value)));
}
return Err(parser.error_other(&info, CompileErrorDescription::Parser("could not negate the int number")));
},
_ => {},
}
let lhs_prop = ast_properties(&lhs);
let args = parser.context().allocator.slice_from_iter([lhs].into_iter());
Ok(ExpryAST::Bytecode(ExpryBytecode::ArithmeticUnaryMin, args, lhs_prop))
},
(JSONToken::FunctionCall(fname),_) => {
let open = parser.accept(JSONToken::Open(), None, || CompileErrorDescription::Parser("expected '(' to open function call"))?;
let mut args : [_; MAX_FUNCTION_ARGS] = core::array::from_fn(|_| ExpryAST::All(0));
let mut arg_count = 0;
parser.opt(|parser| {
args[arg_count] = expry_parse_expr(parser)?;
arg_count += 1;
parser.repeat(|parser| {
parser.accept(JSONToken::Comma(), None, || CompileErrorDescription::Parser(""))?;
args[arg_count] = expry_parse_expr(parser)?;
arg_count += 1;
Ok(arg_count < args.len())
})?;
Ok(())
})?;
parser.accept(JSONToken::Close(), Some(&open), || CompileErrorDescription::Parser("expected ')' to close function call"))?;
#[allow(unused_mut)]
let args = parser.context().allocator.slice_from_array(args, 0..arg_count);
let mut prop = ast_properties_combine(args);
prop.needs_dynamic_eval = internal_function(fname, arg_count).is_none();
Ok(ExpryAST::Function(fname, args, prop))
},
(JSONToken::Field(x),info) => {
resolve(x, parser).map_err(|x| parser.error_other(&info, x))
},
(JSONToken::Open(),info) => {
let value = expry_parse_expr(parser)?;
parser.accept(JSONToken::Close(), Some(&info), || CompileErrorDescription::Parser("expected ')' to close '('"))?;
Ok(value)
},
(JSONToken::ArrayOpen(),open) => {
let mut retval : Vec<ExpryAST> = Vec::new();
parser.repeat(|parser| {
if let Some(value) = parser.opt(expry_parse_expr)? {
retval.push(value);
Ok(parser.opt(|parser| {
parser.accept(JSONToken::Comma(), None, || CompileErrorDescription::Parser(""))
})?.is_some())
} else {
Ok(false)
}
})?;
parser.accept(JSONToken::ArrayClose(), Some(&open), || CompileErrorDescription::Parser("expected ']' to close array"))?;
let prop = ast_properties_combine(&retval);
// FIXME: eliminate Vec of retval
let retval = parser.context().allocator.move_object(retval);
Ok(ExpryAST::Array(retval, prop))
},
(JSONToken::ObjectOpen(),open) => {
// previous operations such as SliceObject and MergeObjects
let mut lhs = None;
// lazy accumulator of current SliceObject
let mut accumulator = Vec::new();
parser.repeat(|parser| {
let (keyvalue,spread) = expry_parse_slice(parser)?;
process_slice(&mut lhs, &mut accumulator, keyvalue, spread, parser.context().allocator);
Ok(parser.accept(JSONToken::Comma(), None, || CompileErrorDescription::Parser("")).is_ok())
})?;
parser.accept(JSONToken::ObjectClose(), Some(&open), || CompileErrorDescription::Parser("expected a '}' to close this object"))?;
Ok(combine(lhs, &mut accumulator, parser.context().allocator))
},
(token,info) => {
parser.undo_get(token, info);
let result = json_expr(parser).map(ExpryAST::Constant);
match result {
Ok(_) => result,
// if not progress is made, we can not match anything, so backtrack
Err(ParserStatus::LookaheadBacktrack()) => result,
Err(_) => Err(parser.error_other(&info, CompileErrorDescription::Parser("expected expr primary (possibly mismatched closing '}'/']')"))),
}
},
}
}
fn resolve<'b,'c>(x: &'b str, parser: &mut ExpryParserState<'b,'_,'c>) -> Result<ExpryAST<'b>,CompileErrorDescription<'b>> where 'c: 'b {
if x == "$this" {
return Ok(ExpryAST::Constant(DecodedValue::Object(parser.context.static_value.clone())));
}
if let Some(position) = parser.context.scopes.iter().rposition(|&name| name == x) {
return Ok(ExpryAST::All(position as u8));
}
if let Some(value) = parser.context.static_value.get(&key_str(x)) {
return Ok(ExpryAST::Constant(value.clone()));
}
if let Some(position) = parser.context.allow_unbound_variables {
let lhs = ExpryAST::All(position as u8);
Ok(field_access(lhs, x, parser.context().allocator))
} else {
Err(CompileErrorDescription::UnknownField(x))
}
}
fn expry_parse_primary_object_access<'b,'c>(parser: &mut ExpryParserState<'b,'_,'c>) -> ExpryParserResult<'b, ExpryAST<'b>> where 'c: 'b {
let mut retval = parser.call(expry_parse_primary)?;
// implement static (with .) and dynamic (with .get()) object access
parser.repeat(|parser| {
match parser.get()? {
(JSONToken::Dot(), _) => {
match parser.get()? {
(JSONToken::Field(name),_) => {
let lhs = core::mem::take(&mut retval);
retval = field_access(lhs, name, parser.context().allocator);
Ok(true)
},
(JSONToken::FunctionCall(name), open) => {
parser.accept(JSONToken::Open(), None, || CompileErrorDescription::Parser(""))?;
let mut args : [_; MAX_FUNCTION_ARGS] = core::array::from_fn(|_| ExpryAST::All(0));
let mut arg_count = 1;
parser.repeat(|parser| {
let value = expry_parse_expr_or_lambda(parser)?;
args[arg_count] = value;
arg_count += 1;
Ok(parser.accept(JSONToken::Comma(), None, || CompileErrorDescription::Parser("")).is_ok() && arg_count < args.len())
})?;
parser.accept(JSONToken::Close(), Some(&open), || CompileErrorDescription::Parser("expected a ')' to close this '('."))?;
args[0] = core::mem::take(&mut retval);
let prop = ast_properties_combine(&args);
// special handling to decide on compute or binary path to take during
// bytecode evaluation
if name == "get" {
if arg_count != 2 { // because of the reserved first spot
return Err(parser.error_other(&open, internal_method_expects(name, 1)));
}
let lhs = std::mem::take(&mut args[0]);
let rhs = std::mem::take(&mut args[1]);
let lhs : &'b mut ExpryAST<'b> = parser.context().allocator.move_object(lhs);
let rhs : &'b mut ExpryAST<'b> = parser.context().allocator.move_object(rhs);
retval = ExpryAST::ObjectAccessWithExpr(lhs, rhs, prop);
return Ok(true);
}
let opcode = internal_method(name, &args[1..arg_count]).map_err(|err| parser.error_other(&open, err))?;
let args = parser.context().allocator.slice_from_array(args, 0..arg_count);
retval = ExpryAST::Bytecode(opcode, args, prop);
Ok(true)
},
(token,info) => Err(parser.error_token(token, info, |_| CompileErrorDescription::Parser("expected field name to access object"))),
}
},
(JSONToken::ArrayOpen(), open) => {
let rhs = expry_parse_expr(parser)?;
parser.accept(JSONToken::ArrayClose(), Some(&open), || CompileErrorDescription::Parser("expected ']' to close this array subscripting operator"))?;
let lhs = core::mem::take(&mut retval);
let prop = ast_properties_iter([&lhs, &rhs].iter());
let lhs : &'b mut ExpryAST<'b> = parser.context().allocator.move_object(lhs);
let rhs : &'b mut ExpryAST<'b> = parser.context().allocator.move_object(rhs);
retval = ExpryAST::ArrayAccessWithExpr(lhs, rhs, prop);
Ok(true)
},
(token,info) => Err(parser.error_token(token, info, |_| CompileErrorDescription::Parser("expected field name to access object"))),
}
})?;
Ok(retval)
}
fn field_access<'a,'c>(lhs: ExpryAST<'a>, name: &'a str, scope: &mut MemoryScope<'c>) -> ExpryAST<'a> where 'c: 'a {
let rhs = ExpryAST::Constant(DecodedValue::String(name.as_bytes()));
let prop = ast_properties_plus(&ast_properties(&lhs),&ast_properties(&rhs));
let lhs : &'a mut ExpryAST<'a> = scope.move_object(lhs);
let rhs : &'a mut ExpryAST<'a> = scope.move_object(rhs);
ExpryAST::ObjectAccessWithExpr(lhs, rhs, prop)
}
fn expry_parse_exponentation<'b,'c>(parser: &mut ExpryParserState<'b,'_,'c>) -> ExpryParserResult<'b, ExpryAST<'b>> where 'c: 'b {
#[allow(unused_mut)]
let mut retval = expry_parse_primary_object_access(parser)?;
#[cfg(feature = "power")]
parser.repeat(|parser| {
match parser.get()? {
(JSONToken::Arithmetic(op),_) if matches!(op, Arithmetic::Pow) => {
let lhs = core::mem::take(&mut retval);
let rhs = expry_parse_primary_object_access(parser)?;
let prop = ast_properties_iter([&lhs, &rhs].iter());
let args = parser.context().allocator.slice_from_iter([lhs, rhs].into_iter());
retval = ExpryAST::Bytecode(op.to_bytecode(), args, prop);
Ok(true)
}
(token,info) => Err(parser.error_token(token, info, |_| CompileErrorDescription::Parser("expected primary as rhs for exponentation"))),
}
})?;
Ok(retval)
}
fn expry_parse_multiplication<'b,'c>(parser: &mut ExpryParserState<'b,'_,'c>) -> ExpryParserResult<'b, ExpryAST<'b>> where 'c: 'b {
let mut retval = expry_parse_exponentation(parser)?;
parser.repeat(|parser| {
match parser.get()? {
(JSONToken::Arithmetic(x),_) if matches!(x, Arithmetic::Mul | Arithmetic::Div | Arithmetic::Mod) => {
let lhs = core::mem::take(&mut retval);
let rhs = expry_parse_exponentation(parser)?;
let prop = ast_properties_iter([&lhs, &rhs].iter());
let args = parser.context().allocator.slice_from_iter([lhs, rhs].into_iter());
retval = ExpryAST::Bytecode(x.to_bytecode(), args, prop);
Ok(true)
}
(token, info) => Err(parser.error_token(token, info, |_| CompileErrorDescription::Parser("expected exponentation as rhs for multiplication"))),
}
})?;
Ok(retval)
}
fn expry_parse_addition<'b,'c>(parser: &mut ExpryParserState<'b,'_,'c>) -> ExpryParserResult<'b, ExpryAST<'b>> where 'c: 'b {
let mut retval = expry_parse_multiplication(parser)?;
parser.repeat(|parser| {
match parser.get()? {
(JSONToken::NegativeNumber(n),_) => {
let lhs = core::mem::take(&mut retval);
let rhs = ExpryAST::Constant(match n {
Number::Float(n) => DecodedValue::Float(n),
Number::Double(n) => DecodedValue::Double(n),
Number::Int(n) => DecodedValue::Int(n),
});
let prop = ast_properties_iter([&lhs, &rhs].iter());
let args = parser.context().allocator.slice_from_iter([lhs, rhs].into_iter());
retval = ExpryAST::Bytecode(ExpryBytecode::ArithmeticPlus, args, prop);
Ok(true)
},
(JSONToken::Arithmetic(op),_) if matches!(op, Arithmetic::Add | Arithmetic::Sub) => {
let lhs = core::mem::take(&mut retval);
let rhs = expry_parse_multiplication(parser)?;
let prop = ast_properties_iter([&lhs, &rhs].iter());
let args = parser.context().allocator.slice_from_iter([lhs, rhs].into_iter());
retval = ExpryAST::Bytecode(op.to_bytecode(), args, prop);
Ok(true)
}
(token, info) => Err(parser.error_token(token, info, |_| CompileErrorDescription::Parser("expected multiplication as rhs for an addition"))),
}
})?;
Ok(retval)
}
fn expry_parse_bitwise_shift<'b,'c>(parser: &mut ExpryParserState<'b,'_,'c>) -> ExpryParserResult<'b, ExpryAST<'b>> where 'c: 'b {
let mut retval = expry_parse_addition(parser)?;
parser.repeat(|parser| {
match parser.get()? {
(JSONToken::Bitwise(op),_) if matches!(op, Bitwise::ShiftLeft | Bitwise::ShiftRight) => {
let lhs = core::mem::take(&mut retval);
let rhs = expry_parse_addition(parser)?;
let prop = ast_properties_iter([&lhs, &rhs].iter());
let args = parser.context().allocator.slice_from_iter([lhs, rhs].into_iter());
retval = ExpryAST::Bytecode(op.to_bytecode(), args, prop);
Ok(true)
}
(token, info) => Err(parser.error_token(token, info, |_| CompileErrorDescription::Parser("expected addition as rhs for an bitwise shift"))),
}
})?;
Ok(retval)
}
fn expry_parse_bitwise_logical<'b,'c>(parser: &mut ExpryParserState<'b,'_,'c>) -> ExpryParserResult<'b, ExpryAST<'b>> where 'c: 'b {
let mut retval = expry_parse_bitwise_shift(parser)?;
parser.repeat(|parser| {
match parser.get()? {
(JSONToken::Bitwise(op),_) if matches!(op, Bitwise::And | Bitwise::Or | Bitwise::Xor) => {
let lhs = core::mem::take(&mut retval);
let rhs = expry_parse_bitwise_shift(parser)?;
let prop = ast_properties_iter([&lhs, &rhs].iter());
let args = parser.context().allocator.slice_from_iter([lhs, rhs].into_iter());
retval = ExpryAST::Bytecode(op.to_bytecode(), args, prop);
Ok(true)
}
(token, info) => Err(parser.error_token(token, info, |_| CompileErrorDescription::Parser("expected bitwise-shift as rhs for an bitwise-logical"))),
}
})?;
Ok(retval)
}
fn expry_parse_concat<'b,'c>(parser: &mut ExpryParserState<'b,'_,'c>) -> ExpryParserResult<'b, ExpryAST<'b>> where 'c: 'b {
let mut retval = expry_parse_bitwise_logical(parser)?;
parser.repeat(|parser| {
match parser.get()? {
(JSONToken::Concat(),_) => {
let lhs = core::mem::take(&mut retval);
let rhs = expry_parse_bitwise_logical(parser)?;
let prop = ast_properties_iter([&lhs, &rhs].iter());
let args = parser.context().allocator.slice_from_iter([lhs, rhs].into_iter());
retval = ExpryAST::Bytecode(ExpryBytecode::StringConcat, args, prop);
Ok(true)
}
(token,info) => Err(parser.error_token(token, info, |_| CompileErrorDescription::Parser("expected bitwise-logical as rhs for a concat"))),
}
})?;
Ok(retval)
}
fn expry_parse_comparison<'b,'c>(parser: &mut ExpryParserState<'b,'_,'c>) -> ExpryParserResult<'b, ExpryAST<'b>> where 'c: 'b {
let mut retval = expry_parse_concat(parser)?;
parser.repeat(|parser| {
match parser.get()? {
(JSONToken::Comparison(c),_) => {
let lhs = core::mem::take(&mut retval);
let rhs = expry_parse_concat(parser)?;
let prop = ast_properties_iter([&lhs, &rhs].iter());
let lhs : &'b mut ExpryAST<'b> = parser.context().allocator.move_object(lhs);
let rhs : &'b mut ExpryAST<'b> = parser.context().allocator.move_object(rhs);
retval = ExpryAST::Comparison(lhs, c, rhs, prop);
Ok(true)
}
(token, info) => Err(parser.error_token(token, info, |_| CompileErrorDescription::Parser("expected concat as rhs for a comparison"))),
}
})?;
Ok(retval)
}
fn expry_parse_logical<'b,'c>(parser: &mut ExpryParserState<'b,'_,'c>) -> ExpryParserResult<'b, ExpryAST<'b>> where 'c: 'b {
let mut retval = expry_parse_comparison(parser)?;
parser.repeat(|parser| {
match parser.get()? {
(JSONToken::Logical(c),_) => {
let lhs = core::mem::take(&mut retval);
let rhs = expry_parse_comparison(parser)?;
let prop = ast_properties_iter([&lhs, &rhs].iter());
let lhs : &'b mut ExpryAST<'b> = parser.context().allocator.move_object(lhs);
let rhs : &'b mut ExpryAST<'b> = parser.context().allocator.move_object(rhs);
retval = ExpryAST::Logical(lhs, c, rhs, prop);
Ok(true)
}
(token,info) => Err(parser.error_token(token, info, |_| CompileErrorDescription::Parser("expected concat as rhs for a comparison"))),
}
})?;
Ok(retval)
}
fn expry_parse_try_catch<'b,'c>(parser: &mut ExpryParserState<'b,'_,'c>) -> ExpryParserResult<'b, ExpryAST<'b>> where 'c: 'b {
let mut retval = expry_parse_logical(parser)?;
parser.repeat(|parser| {
match parser.get()? {
(JSONToken::TryOperator(),_) => {
// changes here should also be applied to expry_parse_slice_key
let rhs = parser.opt(expry_parse_logical)?;
let lhs = core::mem::take(&mut retval);
if let Some(rhs) = rhs {
let prop = ast_properties_iter([&lhs, &rhs].iter());
let args = parser.context().allocator.slice_from_iter([lhs,rhs].into_iter());
retval = ExpryAST::BytecodeWrap(ExpryBytecode::Try, args, prop);
} else {
let prop = ast_properties(&lhs);
let args = parser.context().allocator.slice_from_iter([lhs,ExpryAST::Constant(DecodedValue::Null)].into_iter());
retval = ExpryAST::BytecodeWrap(ExpryBytecode::Try, args, prop);
}
Ok(true)
},
(JSONToken::NullOperator(),_) => {
let lhs = core::mem::take(&mut retval);
let rhs = expry_parse_logical(parser)?;
let prop = ast_properties_iter([&lhs, &rhs].iter());
let args = parser.context().allocator.slice_from_iter([lhs,rhs].into_iter());
retval = ExpryAST::BytecodeWrap(ExpryBytecode::NotNullElse, args, prop);
Ok(true)
},
(token, info) => Err(parser.error_token(token, info, |_| CompileErrorDescription::Parser(""))),
}
})?;
Ok(retval)
}
fn expry_parse_conditional<'b,'c>(parser: &mut ExpryParserState<'b,'_,'c>) -> ExpryParserResult<'b, ExpryAST<'b>> where 'c: 'b {
let value = expry_parse_try_catch(parser)?;
if let Some(info) = parser.opt(|parser| {
parser.accept(JSONToken::Conditional(), None, || CompileErrorDescription::Parser(""))
})? {
let value_then = expry_parse_try_catch(parser)?;
parser.accept(JSONToken::Colon(), Some(&info), || CompileErrorDescription::Parser(""))?;
let value_else = expry_parse_conditional(parser)?;
let prop = ast_properties_iter([&value, &value_then, &value_else].iter());
let args = parser.context().allocator.slice_from_iter([value,value_then,value_else].into_iter());
return Ok(ExpryAST::BytecodeWrap(ExpryBytecode::Conditional, args, prop))
}
Ok(value)
}
fn expry_parse_expr<'b,'c>(parser: &mut ExpryParserState<'b,'_,'c>) -> ExpryParserResult<'b, ExpryAST<'b>> where 'c: 'b {
parser.call(expry_parse_conditional)
}
fn expry_parse_lambda<'b,'c>(parser: &mut ExpryParserState<'b,'_,'c>) -> ExpryParserResult<'b, ExpryAST<'b>>
where
'c: 'b,
{
let err_info = parser.accept(JSONToken::Bitwise(Bitwise::Or), None, || CompileErrorDescription::Parser(""))?;
let mut names : Vec<&str> = Vec::new();
parser.repeat(|parser| {
match parser.get()? {
(JSONToken::Field(variable_name),_) => {
names.push(variable_name);
},
(token,info) => return Err(parser.error_token(token, info, |_| CompileErrorDescription::Parser("expected variable name for lambda"))),
};
Ok(parser.accept(JSONToken::Comma(), None, || CompileErrorDescription::Parser("")).is_ok())
})?;
let names_count : u8 = names.len().try_into().map_err(|_| parser.error_other(&err_info, CompileErrorDescription::Parser("too many lambda arguments")))?;
parser.accept(JSONToken::Bitwise(Bitwise::Or), None, || CompileErrorDescription::Parser(""))?;
let scopes_before = parser.context().scopes.len();
parser.context().scopes.extend_from_slice(&names);
let value_expr = expry_parse_expr(parser);
parser.context().scopes.drain(scopes_before..);
let value_expr = value_expr?;
let prop = ast_properties(&value_expr);
let value_expr : &'b mut ExpryAST<'b> = parser.context().allocator.move_object(value_expr);
Ok(ExpryAST::Lambda(value_expr, names_count, prop))
}
fn expry_parse_expr_or_lambda<'b,'c>(parser: &mut ExpryParserState<'b,'_,'c>) -> ExpryParserResult<'b, ExpryAST<'b>> where 'c: 'b {
parser.choose(&[
expry_parse_lambda,
expry_parse_expr,
][..],
|| CompileErrorDescription::Parser("expected expression or lambda"))
}
// spread syntax returns (None, expr)
// 'key: value' syntax returns (Some(expr), expr)
fn expry_parse_slice<'b, 'c>(parser: &mut ExpryParserState<'b,'_,'c>) -> ExpryParserResult<'b, (Option<&'b mut ExpryAST<'b>>, &'b mut ExpryAST<'b>)> where 'c: 'b {
if let Some(value_expr) = parser.opt(|parser| {
parser.accept(JSONToken::Spread(), None, || CompileErrorDescription::Parser(""))?;
let value_expr = expry_parse_expr(parser)?;
let value_expr = parser.context().allocator.move_object(value_expr);
Ok(value_expr)
})? {
return Ok((None, value_expr));
}
// dynamic key (with expr)
if let Some((key_expr,value_expr)) = parser.opt(|parser| {
let open = parser.accept(JSONToken::Open(), None, || CompileErrorDescription::Parser(""))?;
let key_expr = expry_parse_expr(parser)?;
parser.accept(JSONToken::Close(), Some(&open), || CompileErrorDescription::Parser("expected ')' to close this dynamic field specifier"))?;
parser.accept(JSONToken::Colon(), None, || CompileErrorDescription::Parser("expected ':' as field delimeter for an object field"))?;
let value_expr = expry_parse_expr(parser)?;
Ok((key_expr, value_expr))
})? {
let key_expr = parser.context().allocator.move_object(key_expr);
let value_expr = parser.context().allocator.move_object(value_expr);
return Ok((Some(key_expr), value_expr));
}
let name = match parser.get()? {
(JSONToken::Field(mut name),_) => {
match resolve(name, parser) {
Ok(mut value_expr) => {
if parser.accept(JSONToken::Colon(), None, || CompileErrorDescription::Parser("")).is_ok() {
let value_expr = expry_parse_expr(parser)?;
let con = ExpryAST::Constant(DecodedValue::String(name.as_bytes()));
let con = parser.context().allocator.move_object(con);
let value_expr = parser.context().allocator.move_object(value_expr);
return Ok((Some(con), value_expr));
}
parser.repeat(|parser: &mut ExpryParserState<'b,'_,'_>| {
match parser.get()? {
(JSONToken::Dot(), _) => {
match parser.get()? {
(JSONToken::Field(fieldname),_) => {
let lhs = core::mem::take(&mut value_expr);
value_expr = field_access(lhs, fieldname, parser.context().allocator);
name = fieldname;
Ok(true)
},
(token,info) => Err(parser.error_token(token, info, |_| CompileErrorDescription::Parser("expected field access syntax"))),
}
},
(token,info) => Err(parser.error_token(token, info, |_| CompileErrorDescription::Parser("expected field access syntax"))),
}
})?;
parser.opt(|parser| {
match parser.get()? {
(JSONToken::TryOperator(),_) => {
let rhs = parser.opt(expry_parse_logical)?;
let lhs = core::mem::take(&mut value_expr);
if let Some(rhs) = rhs {
let prop = ast_properties_iter([&lhs, &rhs].iter());
let args = parser.context().allocator.slice_from_iter([lhs,rhs].into_iter());
value_expr = ExpryAST::BytecodeWrap(ExpryBytecode::Try, args, prop);
} else {
let prop = ast_properties(&lhs);
let args = parser.context().allocator.slice_from_iter([lhs,ExpryAST::Constant(DecodedValue::Null)].into_iter());
value_expr = ExpryAST::BytecodeWrap(ExpryBytecode::Try, args, prop);
}
Ok(())
},
(token, info) => Err(parser.error_token(token, info, |_| CompileErrorDescription::Parser(""))),
}
})?;
let con = ExpryAST::Constant(DecodedValue::String(name.as_bytes()));
let con = parser.context().allocator.move_object(con);
let value_expr = parser.context().allocator.move_object(value_expr);
return Ok((Some(con), value_expr));
},
Err(_) => {
name.as_bytes()
},
}
},
(JSONToken::String(name),_) => name,
(token,info) => return Err(parser.error_token(token, info, |_| CompileErrorDescription::Parser(""))),
};
parser.accept(JSONToken::Colon(), None, || CompileErrorDescription::Parser("a value for this field is necessary (use different syntax to copy over fields)"))?;
let value_expr = expry_parse_expr(parser)?;
let con = ExpryAST::Constant(DecodedValue::String(name));
let con = parser.context().allocator.move_object(con);
let value_expr = parser.context().allocator.move_object(value_expr);
Ok((Some(con), value_expr))
}
// combine lhs and accumulator into one value and return that value
fn combine<'a,'b,'c>(lhs: Option<&'a mut ExpryAST<'a>>, accumulator: &'b mut Vec<(&'a mut ExpryAST<'a>, &'a mut ExpryAST<'a>)>, scope: &mut MemoryScope<'c>) -> ExpryAST<'a> where 'c: 'a, 'a: 'b {
if let Some(lhs) = lhs {
if accumulator.is_empty() {
// ends with spread syntax
return std::mem::take(lhs);
}
// earlier a spread syntax occured (which should be in in lhs)
let accumulator = scope.move_object(std::mem::take(accumulator));
let current = make_object(accumulator);
let current = scope.move_object(current);
let prop = ast_properties_plus_ast(&lhs, ¤t);
ExpryAST::MergeObjects(lhs, current, prop)
} else if accumulator.is_empty() {
// nothing, so empty object
ExpryAST::Constant(DecodedValue::Object(DecodedObject::new()))
} else {
// only fields, no spread syntax
let accumulator = scope.move_object(std::mem::take(accumulator));
make_object(accumulator)
}
}
fn process_slice<'a,'b,'c>(lhs: &mut Option<&'a mut ExpryAST<'a>>, accumulator: &'b mut Vec<(&'a mut ExpryAST<'a>, &'a mut ExpryAST<'a>)>, key: Option<&'a mut ExpryAST<'a>>, value: &'a mut ExpryAST<'a>, scope: &mut MemoryScope<'c>) where 'c: 'a, 'a: 'b {
if let Some(key) = key {
accumulator.push((key,value));
return;
}
let l = lhs.take();
// check if there is something to combine
if l.is_none() && accumulator.is_empty() {
*lhs = Some(value);
return;
}
let l = combine(l, accumulator, scope);
let prop = ast_properties_plus_ast(&l, &value);
let l = scope.move_object(l);
*lhs = Some(scope.move_object(ExpryAST::MergeObjects(l, value, prop)));
}
fn make_object<'a>(retval: &'a mut [(&'a mut ExpryAST<'a>, &'a mut ExpryAST<'a>)]) -> ExpryAST<'a> {
let prop = retval.iter().map(|(x,y)| ast_properties_plus_ast(x, y)).fold(ASTProperties::constant(), |x,y| {
ast_properties_plus(&x,&y)
});
ExpryAST::Object(retval, prop)
}
fn expry_parse_expr_top_level<'b,'c>(parser: &mut ExpryParserState<'b,'_,'c>) -> ExpryParserResult<'b, ExpryAST<'b>> where 'c: 'b {
let retval = expry_parse_expr(parser)?;
parser.accept(JSONToken::End(), None, || CompileErrorDescription::Parser("expected end of input (possibly an operator was missing)"))?;
Ok(retval)
}
fn expry_parse_slice_top_level<'b,'c>(parser: &mut ExpryParserState<'b,'_,'c>) -> ExpryParserResult<'b, ExpryAST<'b>> where 'c: 'b {
let mut lhs : Option<_> = None;
// lazy accumulator of current SliceObject
let mut accumulator : Vec<(_, _)> = Vec::new();
parser.repeat(|parser| {
let (keyvalue,spread) = expry_parse_slice(parser)?;
process_slice(&mut lhs, &mut accumulator, keyvalue, spread, parser.context().allocator);
Ok(parser.accept(JSONToken::Comma(), None, || CompileErrorDescription::Parser("")).is_ok())
})?;
parser.accept(JSONToken::End(), None, || CompileErrorDescription::Parser("expected end of input (possibly an operator was missing)"))?;
Ok(combine(lhs, &mut accumulator, parser.context().allocator))
}
fn expry_object_type_spec<'b,'c>(parser: &mut ExpryParserState<'b,'_,'c>) -> ExpryParserResult<'b, TypeObject> where 'c: 'b {
let mut types = TypeObject::new();
parser.repeat(|parser| {
let label = match parser.get()? {
(JSONToken::Field(label),_) => label,
(token, info) => return Err(parser.error_token(token, info, |_| CompileErrorDescription::Parser("expected label (without quotes) for the type-spec"))),
};
let optional = parser.opt(|parser| parser.accept(JSONToken::Conditional(), None, || CompileErrorDescription::NoToken))?;
parser.accept(JSONToken::Colon(), None, || CompileErrorDescription::Parser("expected ':' as field delimeter for an object type-spec"))?;
let value = expry_type_spec(parser)?;
types.insert(label.as_bytes().to_vec(), (optional.is_none(), value));
Ok(parser.accept(JSONToken::Comma(), None, || CompileErrorDescription::Parser("")).is_ok())
})?;
Ok(types)
}
fn expry_type_spec<'b,'c>(parser: &mut ExpryParserState<'b,'_,'c>) -> ExpryParserResult<'b, ExpryType> where 'c: 'b {
let mut retval = match parser.get()? {
(JSONToken::Arithmetic(Arithmetic::Mul),_) => ExpryType::Any,
(JSONToken::NullObject(),_) => ExpryType::Null,
(JSONToken::Type(t),_) => {
match t {
SimpleType::Bool => ExpryType::Bool,
SimpleType::Int => ExpryType::Int,
SimpleType::Float => ExpryType::Float,
SimpleType::Double => ExpryType::Double,
SimpleType::String => ExpryType::String,
}
},
(JSONToken::ArrayOpen(),open) => {
let retval = expry_type_spec(parser)?;
parser.accept(JSONToken::ArrayClose(), Some(&open), || CompileErrorDescription::Parser("expected ']' to close array"))?;
ExpryType::Array(Box::new(retval))
},
(JSONToken::ObjectOpen(),open) => {
if let Some(retval) = parser.opt(|parser| {
parser.accept(JSONToken::Arithmetic(Arithmetic::Mul), None, || CompileErrorDescription::Parser(""))?;
parser.accept(JSONToken::Colon(), None, || CompileErrorDescription::Parser("expected ':' as field delimeter for an uniform object type-spec"))?;
let value = expry_type_spec(parser)?;
parser.accept(JSONToken::ObjectClose(), Some(&open), || CompileErrorDescription::Parser("expected a '}' to close this object"))?;
Ok(value)
})? {
ExpryType::UniformObject(Box::new(retval))
} else {
let types = expry_object_type_spec(parser)?;
parser.accept(JSONToken::ObjectClose(), Some(&open), || CompileErrorDescription::Parser("expected a '}' to close this object"))?;
ExpryType::Object(types)
}
},
(token, info) => return Err(parser.error_token(token, info, |_| CompileErrorDescription::Parser("unexpected token for type-spec"))),
};
parser.opt(|parser| {
parser.accept(JSONToken::Conditional(), None, || CompileErrorDescription::Parser(""))?;
let t = core::mem::take(&mut retval);
retval = ExpryType::Nullable(Box::new(t));
Ok(())
})?;
Ok(retval)
}
fn expry_object_type_spec_top_level<'b,'c>(parser: &mut ExpryParserState<'b,'_,'c>) -> ExpryParserResult<'b, TypeObject> where 'c: 'b {
let retval = expry_object_type_spec(parser)?;
parser.accept(JSONToken::End(), None, || CompileErrorDescription::Parser("expected end of input (possibly an operator was missing)"))?;
Ok(retval)
}
fn expry_type_spec_top_level<'b,'c>(parser: &mut ExpryParserState<'b,'_,'c>) -> ExpryParserResult<'b, ExpryType> where 'c: 'b {
let retval = expry_type_spec(parser)?;
parser.accept(JSONToken::End(), None, || CompileErrorDescription::Parser("expected end of input (possibly an operator was missing)"))?;
Ok(retval)
}
/// Parse expressions that can be executed with an optional binary value as argument. These
/// expressions can be compiled to bytecode and executed very fast so it is suitable as a query
/// language, performing selection and modification of values on the server.
fn parse_expry_expr<'a,'b,'c>(slice: bool, reader: &'b str, static_value: &DecodedObject<'b>, allow_unbound_variables: Option<u8>, dynamic_value_names: &[&str], allocator: &'a mut MemoryScope<'c>) -> Result<ExpryAST<'b>,(CompileErrorDescription<'b>,usize,usize,Option<(usize,usize)>)> where 'c: 'b, 'b: 'a {
let mut parser = ExpryParserState::new_with(reader, expry_tokenize, JSONParserContext{ allocator, static_value, allow_unbound_variables, scopes: dynamic_value_names.to_vec(), depth_left: MAX_PARSER_DEPTH });
parser.parse(if slice { expry_parse_slice_top_level } else { expry_parse_expr_top_level }, CompileErrorDescription::Parser("unexpected token"), CompileErrorDescription::ParserTooLong)
}
pub fn expry_parse_type<'a,'b,'c>(reader: &'b str, allocator: &'a mut MemoryScope<'c>) -> Result<ExpryType,CompileError<'b>> where 'c: 'b, 'b: 'a {
let static_value = &DecodedObject::new();
let mut parser = ExpryParserState::new_with(reader, expry_tokenize, JSONParserContext{ allocator, static_value, allow_unbound_variables: None, scopes: Vec::new(), depth_left: MAX_PARSER_DEPTH });
parser.parse(expry_type_spec_top_level, CompileErrorDescription::Parser("unexpected token"), CompileErrorDescription::ParserTooLong).map_err(|(error,start,end,extra)| CompileError { expr: reader, error, start, end, extra })
}
pub fn expry_parse_object_type<'a,'b,'c>(reader: &'b str, allocator: &'a mut MemoryScope<'c>) -> Result<TypeObject,CompileError<'b>> where 'c: 'b, 'b: 'a {
let static_value = &DecodedObject::new();
let mut parser = ExpryParserState::new_with(reader, expry_tokenize, JSONParserContext{ allocator, static_value, allow_unbound_variables: None, scopes: Vec::new(), depth_left: MAX_PARSER_DEPTH });
parser.parse(expry_object_type_spec_top_level, CompileErrorDescription::Parser("unexpected token"), CompileErrorDescription::ParserTooLong).map_err(|(error,start,end,extra)| CompileError { expr: reader, error, start, end, extra })
}
#[derive(Debug,Copy,Clone)]
#[repr(u8)]
enum ExpryBytecode {
Invalid = 0,
SliceObjectStatic = 2,
SliceArray = 3,
Conditional = 4,
Field = 6, All = 7,
MergeObjects = 8,
SliceObjectDynamic = 9,
Concrete = 10,
FieldAccessBinary = 11,
FieldAccessDecoded = 12,
ArrayAccessBinary = 13,
ArrayAccessDecoded = 14,
Call = 16,
ArithmeticPlus = 20, ArithmeticMin, ArithmeticMul, ArithmeticDiv, ArithmeticMod, ArithmeticShiftLeft, ArithmeticShiftRight, ArithmeticUnaryMin, ArithmeticBitwiseAnd, ArithmeticBitwiseOr, ArithmeticBitwiseXor,
Not = 32, And, Or,
Equal = 35, NotEqual,
ParseEqual = 40, ParseNotEqual,
NumericLess = 45, NumericGreaterEqual, NumericGreater, NumericLessEqual,
StringContains = 55, StringStartsWith, StringEndsWith,
StringConcat = 65,
OpenObject = 70,
NotNullElse = 76,
Try = 77,
Defined = 78,
Undefined = 79,
ArithmeticSin = 80, ArithmeticTan, ArithmeticCos, ArithmeticSqrt, ArithmeticAbs, ArithmeticLog, ArithmeticCeil, ArithmeticFloor, ArithmeticTrunc, ArithmeticRound, ArithmeticLog10, ArithmeticLog2,
#[cfg(feature = "power")]
ArithmeticPower,
StringUpper = 100, ReservedForMagicHeader = 101, StringLower = 102, StringTrim, StringHex, StringXxhash32, StringXxhash64, StringSha1, StringSha512, StringHtmlescape, StringUrlescape,
StringDirname = 111, StringBasename, StringSubstringFromTo, StringCount, StringSplit, StringSubstringFrom,
ToInteger = 124, ToFloat, ToDouble, ToString,
TypeIsNull = 128, TypeIsBool, TypeIsNumber, TypeIsInteger, TypeIsFloat, TypeIsDouble, TypeIsString, TypeIsArray, TypeIsObject,
Length = 140, ArrayReverse = 141,
Lambda = 192,
LambdaFilter = 193, LambdaArrayMap = 194, LambdaSortByKey = 195, LambdaArrayToObject = 196, LambdaArrayToMap = 197, LambdaObjectToArray = 198,
}
fn expry_cmp<'a>(lhs: Key<'a>, rhs: Key<'a>) -> Ordering {
if lhs.1.is_empty() {
return Ordering::Greater;
}
if rhs.1.is_empty() {
return Ordering::Less;
}
lhs.cmp(&rhs)
}
fn expry_min<'a>(lhs: Key<'a>, rhs: Key<'a>) -> Key<'a> {
if lhs.1.is_empty() {
return rhs;
}
if rhs.1.is_empty() {
return lhs;
}
core::cmp::min(lhs, rhs)
}
#[cfg(not(feature = "mini"))]
fn expry_ast_optimize<'b,'c>(ast: &'_ mut ExpryAST<'b>) -> Result<(),EvalError<'b>> where 'c: 'b
{
match ast {
ExpryAST::Constant(_) => {},
ExpryAST::All(_) => {},
ExpryAST::ObjectAccessWithExpr(lhs, rhs, _) => {
expry_ast_optimize(lhs)?;
expry_ast_optimize(rhs)?;
},
ExpryAST::ArrayAccessWithExpr(lhs, rhs, _) => {
expry_ast_optimize(lhs)?;
expry_ast_optimize(rhs)?
},
ExpryAST::Comparison(lhs, _, rhs, _) => {
expry_ast_optimize(lhs)?;
expry_ast_optimize(rhs)?;
},
ExpryAST::Logical(lhs, _, rhs, _) => {
expry_ast_optimize(lhs)?;
expry_ast_optimize(rhs)?;
},
ExpryAST::Function(_, args, _) => {
for v in args.iter_mut() {
expry_ast_optimize(v)?;
}
},
ExpryAST::Bytecode(_, args, _) | ExpryAST::BytecodeWrap(_, args, _) => {
for v in args.iter_mut() {
expry_ast_optimize(v)?;
}
},
ExpryAST::Object(fields, _) => {
for (k,v) in fields.iter_mut() {
expry_ast_optimize(k)?;
expry_ast_optimize(v)?;
}
},
ExpryAST::MergeObjects(lhs, rhs, _) => {
expry_ast_optimize(lhs)?;
expry_ast_optimize(rhs)?;
},
ExpryAST::Array(arr, _) => {
for v in arr.iter_mut() {
expry_ast_optimize(v)?;
}
},
ExpryAST::Lambda(expr, _, _) => {
expry_ast_optimize(expr)?;
},
};
Ok(())
}
#[cfg(not(feature = "mini"))]
fn expry_ast_fold<'b,'c>(ast: &'_ mut ExpryAST<'b>, scope: &mut MemoryScope<'c>) -> Result<(),EvalError<'b>> where 'c: 'b
{
// if this is a 'static' subtree, replace with a constant
if !ast_properties(ast).needs_dynamic_eval {
// storing bytecode in allocator, as evaluating might create a reference to the bytecode
let bytecode = expry_ast_to_bytecode(scope, ast, 0).map_err(EvalError::Expression)?;
*ast = ExpryAST::Constant(expry_eval_func(bytecode, &mut Vec::new(), scope, &NoCustomFuncs{})?);
return Ok(());
}
match ast {
ExpryAST::Constant(_) => {},
ExpryAST::All(_) => {},
ExpryAST::ObjectAccessWithExpr(lhs, rhs, _) => {
expry_ast_fold(lhs, scope)?;
expry_ast_fold(rhs, scope)?;
},
ExpryAST::ArrayAccessWithExpr(lhs, rhs, _) => {
expry_ast_fold(lhs, scope)?;
expry_ast_fold(rhs, scope)?;
},
ExpryAST::Comparison(lhs, _, rhs, _) => {
expry_ast_fold(lhs, scope)?;
expry_ast_fold(rhs, scope)?;
},
ExpryAST::Logical(lhs, _, rhs, _) => {
expry_ast_fold(lhs, scope)?;
expry_ast_fold(rhs, scope)?;
},
ExpryAST::Function(_, args, _) => {
for v in args.iter_mut() {
expry_ast_fold(v, scope)?;
}
},
ExpryAST::Bytecode(_, args, _) | ExpryAST::BytecodeWrap(_, args, _) => {
for v in args.iter_mut() {
expry_ast_fold(v, scope)?;
}
},
ExpryAST::Object(fields, _) => {
for (k,v) in fields.iter_mut() {
expry_ast_fold(k, scope)?;
expry_ast_fold(v, scope)?;
}
},
ExpryAST::Array(arr, _) => {
for v in arr.iter_mut() {
expry_ast_fold(v, scope)?;
}
},
ExpryAST::Lambda(expr, _, _) => {
expry_ast_fold(expr, scope)?;
},
ExpryAST::MergeObjects(lhs, rhs, _) => {
expry_ast_fold(lhs, scope)?;
expry_ast_fold(rhs, scope)?;
},
};
Ok(())
}
fn internal_method_expects(_name: &str, _args: usize) -> CompileErrorDescription<'static> {
CompileErrorDescription::Parser("wrong number of arguments")
}
fn internal_method(name: &str, args:&[ExpryAST]) -> Result<ExpryBytecode,CompileErrorDescription<'static>> {
let args_count = args.len();
match name {
"len" => if args_count == 0 {
Ok(ExpryBytecode::Length)
} else {
Err(internal_method_expects(name, 0))
},
"rev" => if args_count == 0 {
Ok(ExpryBytecode::ArrayReverse)
} else {
Err(internal_method_expects(name, 0))
},
"sub" => if args_count == 1 {
Ok(ExpryBytecode::StringSubstringFrom)
} else if args_count == 2 {
Ok(ExpryBytecode::StringSubstringFromTo)
} else {
Err(CompileErrorDescription::Parser(".sub() expects 1 or 2 arguments"))
},
"upper" => if args_count == 0 {
Ok(ExpryBytecode::StringUpper)
} else {
Err(internal_method_expects(name, 0))
},
"lower" => if args_count == 0 {
Ok(ExpryBytecode::StringLower)
} else {
Err(internal_method_expects(name, 0))
},
"trim" => if args_count == 0 {
Ok(ExpryBytecode::StringTrim)
} else {
Err(internal_method_expects(name, 0))
},
"hex" => if args_count == 0 {
Ok(ExpryBytecode::StringHex)
} else {
Err(internal_method_expects(name, 0))
},
"urlescape" => if args_count == 0 {
Ok(ExpryBytecode::StringUrlescape)
} else {
Err(internal_method_expects(name, 0))
},
"htmlescape" => if args_count == 0 {
Ok(ExpryBytecode::StringHtmlescape)
} else {
Err(internal_method_expects(name, 0))
},
"dirname" => if args_count == 0 {
Ok(ExpryBytecode::StringDirname)
} else {
Err(internal_method_expects(name, 0))
},
"basename" => if args_count == 0 {
Ok(ExpryBytecode::StringBasename)
} else {
Err(internal_method_expects(name, 0))
},
"splitn" => if args_count == 2 {
Ok(ExpryBytecode::StringSplit)
} else {
Err(internal_method_expects(name, 2))
},
"tostring" => if args_count == 0 {
Ok(ExpryBytecode::ToString)
} else {
Err(internal_method_expects(name, 0))
},
"toint" => if args_count == 0 {
Ok(ExpryBytecode::ToInteger)
} else {
Err(internal_method_expects(name, 0))
},
"tofloat" => if args_count == 0 {
Ok(ExpryBytecode::ToFloat)
} else {
Err(internal_method_expects(name, 0))
},
"todouble" => if args_count == 0 {
Ok(ExpryBytecode::ToDouble)
} else {
Err(internal_method_expects(name, 0))
},
"filter" => if args_count == 1 {
if let Some(ExpryAST::Lambda(_, 1, _)) = args.first() {
Ok(ExpryBytecode::LambdaFilter)
} else {
Err(CompileErrorDescription::Parser("filter() expects a lambda function with one argument"))
}
} else {
Err(internal_method_expects(name, 1))
},
"map" => if args_count == 1 {
if let Some(ExpryAST::Lambda(_, 1, _)) = args.first() {
Ok(ExpryBytecode::LambdaArrayMap)
} else {
Err(CompileErrorDescription::Parser("map() expects a lambda function with one argument"))
}
} else {
Err(internal_method_expects(name, 1))
},
"sort_by_key" => if args_count == 1 {
if let Some(ExpryAST::Lambda(_, 1, _)) = args.first() {
Ok(ExpryBytecode::LambdaSortByKey)
} else {
Err(CompileErrorDescription::Parser("sort_by_key() expects a lambda function with one argument"))
}
} else {
Err(internal_method_expects(name, 1))
},
"to_object" => if args_count == 1 {
if let Some(ExpryAST::Lambda(_, 1, _)) = args.first() {
Ok(ExpryBytecode::LambdaArrayToObject)
} else {
Err(CompileErrorDescription::Parser("to_object() expects a lambda function with one argument"))
}
} else {
Err(internal_method_expects(name, 1))
},
"to_map" => if args_count == 1 {
if let Some(ExpryAST::Lambda(_, 1, _)) = args.first() {
Ok(ExpryBytecode::LambdaArrayToMap)
} else {
Err(CompileErrorDescription::Parser("to_map() expects a lambda function with one argument"))
}
} else {
Err(internal_method_expects(name, 1))
},
"to_array" => if args_count == 1 {
if let Some(ExpryAST::Lambda(_, 2, _)) = args.first() {
Ok(ExpryBytecode::LambdaObjectToArray)
} else {
Err(CompileErrorDescription::Parser("to_array() expects a lambda function with two arguments (key, value)"))
}
} else {
Err(internal_method_expects(name, 1))
},
// do not call defined/undefined here, as it will mess up the interpreter due to early abort on error (so need to write a fixed length)
// on error
_ => Err(CompileErrorDescription::Parser("only built-in methods are supported")),
}
}
fn internal_function(_name: &str, _args: usize) -> Option<u8> {
None
}
const EXPRY_MAGIC : &[u8; 1] = b"e";
fn expry_ast_to_bytecode<'a,'c>(scope: &mut MemoryScope<'c>, ast: &'_ mut ExpryAST<'a>, dynamic_values: u8) -> Result<BytecodeRef<'c>,&'static str> {
if cfg!(feature = "mini") {
let mut writer = RawString::new();
writer.write_bytes(EXPRY_MAGIC).unwrap_infallible();
writer.write_u8(dynamic_values).unwrap_infallible();
if !_expry_ast_to_bytecode(ast, &mut writer, dynamic_values, MAX_AST_DEPTH).expect("binary encoding error") {
return Err("too complex AST to convert to bytecode");
}
// FIXME: use delayed deallocation function of MemoryScope here (to avoid move/copy)
Ok(BytecodeRef(scope.move_object(writer.data)))
} else {
let length;
{
let mut length_collector = RawWriterLength::new();
length_collector.write_bytes(EXPRY_MAGIC).unwrap_infallible();
length_collector.write_u8(dynamic_values).unwrap_infallible();
if !_expry_ast_to_bytecode(ast, &mut length_collector, dynamic_values, MAX_AST_DEPTH).unwrap_infallible() {
return Err("too complex AST to convert to bytecode");
}
length = length_collector.length();
}
let retval = scope.alloc(length);
let mut writer = RawWriter::with_uninit(retval);
writer.write_bytes(EXPRY_MAGIC).expect("expry: internal error");
writer.write_u8(dynamic_values).expect("expry: internal error");
if !_expry_ast_to_bytecode(ast, &mut writer, dynamic_values, MAX_AST_DEPTH).expect("expry: calculated size differs from actual size") {
return Err("too complex AST to convert to bytecode");
}
debug_assert!(writer.left() == 0);
Ok(BytecodeRef(writer.build()))
}
}
fn _expry_ast_to_bytecode<'a, 'b, E, Out>(ast: &'_ mut ExpryAST<'b>, writer: &mut Out, values: u8, depth: u8) -> Result<bool,E>
where
Out: RawOutput<E>,
'b: 'a,
{
if depth == 0 {
return Ok(false);
}
let mut retval = true;
match ast {
ExpryAST::Constant(v) => {
writer.write_u8(ExpryBytecode::Concrete as u8)?;
v.print_binary(writer)?;
},
ExpryAST::All(position) => {
writer.write_u8(ExpryBytecode::All as u8)?;
writer.write_u8(*position)?;
},
ExpryAST::ObjectAccessWithExpr(lhs, rhs, _) => {
if fast_path(lhs) {
writer.write_u8(ExpryBytecode::FieldAccessBinary as u8)?;
} else {
writer.write_u8(ExpryBytecode::FieldAccessDecoded as u8)?;
}
retval &= _expry_ast_to_bytecode(lhs, writer, values, depth-1)?;
retval &= _expry_ast_to_bytecode(rhs, writer, values, depth-1)?;
},
ExpryAST::ArrayAccessWithExpr(lhs, rhs, _) => {
if fast_path(lhs) {
writer.write_u8(ExpryBytecode::ArrayAccessBinary as u8)?;
} else {
writer.write_u8(ExpryBytecode::ArrayAccessDecoded as u8)?;
}
retval &= _expry_ast_to_bytecode(lhs, writer, values, depth-1)?;
retval &= _expry_ast_to_bytecode(rhs, writer, values, depth-1)?;
},
ExpryAST::Bytecode(opcode, args, _) => {
writer.write_u8(*opcode as u8)?;
for arg in args.iter_mut() {
retval &= _expry_ast_to_bytecode(arg, writer, values, depth-1)?;
}
},
ExpryAST::BytecodeWrap(opcode, args, _) => {
writer.write_u8(*opcode as u8)?;
for arg in args.iter_mut() {
retval &= write_with_header(writer, Out::write_var_u64, |writer| {
_expry_ast_to_bytecode(arg, writer, values, depth-1)
})?;
}
},
ExpryAST::Comparison(lhs, op, rhs, _) => {
// equal and unequal
#[allow(unused_mut)]
let mut op_bin : u8 = match op {
ExpryComparison::Equal => ExpryBytecode::ParseEqual as u8,
ExpryComparison::NotEqual => ExpryBytecode::ParseNotEqual as u8,
ExpryComparison::Less => ExpryBytecode::NumericLess as u8,
ExpryComparison::LessEqual => ExpryBytecode::NumericLessEqual as u8,
ExpryComparison::Greater => ExpryBytecode::NumericGreater as u8,
ExpryComparison::GreaterEqual => ExpryBytecode::NumericGreaterEqual as u8,
ExpryComparison::Contains => ExpryBytecode::StringContains as u8,
ExpryComparison::StartsWith => ExpryBytecode::StringStartsWith as u8,
ExpryComparison::EndsWith => ExpryBytecode::StringEndsWith as u8,
};
// if the source is the same, the encoding should be the same and we can use a raw compare
#[cfg(not(feature = "mini"))]
if ast_properties(lhs).source != ValueSource::Computed && ast_properties(lhs).source == ast_properties(rhs).source {
if op_bin == ExpryBytecode::ParseEqual as u8 {
op_bin = ExpryBytecode::Equal as u8;
} else if op_bin == ExpryBytecode::ParseNotEqual as u8 {
op_bin = ExpryBytecode::NotEqual as u8;
}
}
writer.write_u8(op_bin)?;
retval &= _expry_ast_to_bytecode(lhs, writer, values, depth-1)?;
retval &= _expry_ast_to_bytecode(rhs, writer, values, depth-1)?;
},
ExpryAST::Logical(lhs, op, rhs, _) => {
let op_bin : u8 = match op {
Logical::And => ExpryBytecode::And as u8,
Logical::Or => ExpryBytecode::Or as u8,
};
writer.write_u8(op_bin)?;
retval &= write_with_header(writer, Out::write_var_u64, |writer| {
_expry_ast_to_bytecode(lhs, writer, values, depth-1)?;
_expry_ast_to_bytecode(rhs, writer, values, depth-1)
})?;
},
ExpryAST::Function(name, args, _) => {
if *name == "undefined" && args.len() == 1 {
// special handling because length of expression is needed (because of early abort
// on error)
writer.write_u8(ExpryBytecode::Undefined as u8)?;
retval &= write_with_header(writer, Out::write_var_u64, |writer| {
_expry_ast_to_bytecode(&mut args[0], writer, values, depth-1)
})?;
} else if *name == "defined" && args.len() == 1 {
// special handling because length of expression is needed (because of early abort
// on error)
writer.write_u8(ExpryBytecode::Defined as u8)?;
retval &= write_with_header(writer, Out::write_var_u64, |writer| {
_expry_ast_to_bytecode(&mut args[0], writer, values, depth-1)
})?;
} else if let Some(opcode) = internal_function(name, args.len()) {
writer.write_u8(opcode)?;
for arg in args.iter_mut() {
retval &= _expry_ast_to_bytecode(arg, writer, values, depth-1)?;
}
} else {
writer.write_u8(ExpryBytecode::Call as u8)?;
writer.write_var_bytes(name.as_bytes())?;
writer.write_var_u64(args.len() as u64)?;
for i in 0..args.len() {
_expry_ast_to_bytecode(&mut args[i], writer, values, depth-1)?;
}
}
},
ExpryAST::Object(fields, _) => {
let slice_static = fields.iter().all(|(k,_)| matches!(k, ExpryAST::Constant(DecodedValue::String(_))));
if slice_static {
writer.write_u8(ExpryBytecode::SliceObjectStatic as u8)?;
// for static objects, we can do a static sort
fields.sort_unstable_by_key(|(x,_)| {
if let ExpryAST::Constant(DecodedValue::String(key)) = x {
(key_stable_hash(key), *key)
} else {
(0u8, &b""[..])
}
});
} else {
writer.write_u8(ExpryBytecode::SliceObjectDynamic as u8)?;
}
writer.write_var_u64(fields.len() as u64)?;
for i in 0..fields.len() {
if slice_static {
if let ExpryAST::Constant(DecodedValue::String(key)) = &fields[i].0 {
let hash = key_stable_hash(key);
writer.write_u8(hash)?;
writer.write_var_bytes(key)?;
} else {
unimplemented!();
}
} else {
retval &= _expry_ast_to_bytecode(&mut fields[i].0, writer, values, depth-1)?;
}
retval &= _expry_ast_to_bytecode(&mut fields[i].1, writer, values, depth-1)?;
}
},
ExpryAST::MergeObjects(lhs, rhs, _) => {
writer.write_u8(ExpryBytecode::MergeObjects as u8)?;
retval &= _expry_ast_to_bytecode(lhs, writer, values, depth-1)?;
retval &= _expry_ast_to_bytecode(rhs, writer, values, depth-1)?;
},
ExpryAST::Array(fields, _) => {
writer.write_u8(ExpryBytecode::SliceArray as u8)?;
writer.write_var_u64(fields.len() as u64)?;
for i in 0..fields.len() {
retval &= _expry_ast_to_bytecode(&mut fields[i], writer, values, depth-1)?;
}
},
ExpryAST::Lambda(expr, args, _) => {
// extra safeguard that we have the correct function (just like regular expry magic)
writer.write_u8(ExpryBytecode::Lambda as u8)?;
writer.write_u8(values)?;
writer.write_u8(*args)?;
retval &= write_with_header(writer, Out::write_var_u64, |writer| {
_expry_ast_to_bytecode(expr, writer, values+*args, depth+1)
})?;
},
}
Ok(retval)
}
#[derive(Copy, Clone, PartialEq, Eq)]
pub enum EvalError<'a> {
// errors in the _encoding_ of the expression bytecode
// errors in the static types of the expression (e.g. if a certain expression always produces a boolean, but a different type was expected this implies an error during generation of the expression)
Expression(&'a str),
// errors in the _encoding_ of the value
Value(&'a str),
// type errors, division by zero
// these errors depends on the data given
Dynamic(&'a str),
// errors if a field is not found
FieldNotFound(&'a str),
// other errors (if it is unclear what is the source of the error)
Other(),
EvalTooLong(),
}
impl<'a> core::fmt::Debug for EvalError<'a> {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "{}", self)
}
}
impl<'a> core::fmt::Display for EvalError<'a> {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
EvalError::Expression(msg) => write!(f, "error in decoding expression: {}", msg),
EvalError::Value(msg) => write!(f, "error in decoding value: {}", msg),
EvalError::Dynamic(msg) => write!(f, "dynamic error: {}", msg),
EvalError::FieldNotFound(msg) => write!(f, "field '{}' not found", msg),
EvalError::Other() => write!(f, "unknown error"),
EvalError::EvalTooLong() => write!(f, "stack overflow"),
}
}
}
impl<'a> From<EvalError<'a>> for CompileError<'a> {
fn from(err: EvalError<'a>) -> Self {
Self {
expr: "",
error: CompileErrorDescription::Optimizer(err),
start: 0,
end: 0,
extra: None,
}
}
}
/// Trait that can be used to pass a custom user-provided function handler to `expry` eval functions.
pub trait CustomFuncs<'a> {
fn call<'b,'c>(&'_ self, name: &'_ [u8], args: &'_ [DecodedValue<'b>], scope: &'_ mut MemoryScope<'c>) -> Result<DecodedValue<'b>,&'b str> where 'c: 'b, 'a: 'b;
fn types(&self) -> BTreeMap<Key<'static>,(Vec<ExpryType>,ExpryType)>;
}
/// An provided implementation of the custom function handler, that directly throws an error when
/// invoked.
pub struct NoCustomFuncs {
}
impl<'a> CustomFuncs<'a> for NoCustomFuncs {
fn call<'b,'c>(&'_ self, _name: &'_ [u8], _args: &'_ [DecodedValue<'b>], _scope: &'_ mut MemoryScope<'c>) -> Result<DecodedValue<'b>,&'b str> where 'c: 'b, 'a: 'b {
Err("no custom functions defined")
}
fn types(&self) -> BTreeMap<Key<'static>,(Vec<ExpryType>,ExpryType)> {
BTreeMap::new()
}
}
struct Context<'a,'b,'c> {
originals: &'a mut Vec<EncodedValueRef<'b>>, // needed for the `this` input
custom: &'a dyn CustomFuncs<'c>,
}
// does not throw an error if the object is not sorted, because we can not reliable detect all
// cases (if first item after previous call to evaluate_seek is less, we can not detect it without
// adding state).
fn evaluate_seek<'a,'b>(looking_for_hash: KeyHash, looking_for_key: &'b [u8], value: &mut RawReader<'a>) -> Result<(),EvalError<'b>> {
//eprintln!("seeking {}/{}", sorted_key.hash, String::from_utf8_lossy(sorted_key.key));
while !value.is_empty() {
let mut peeker = *value;
let mut key_length = peeker.read_var_u64().map_err(|_| EvalError::Value("Could not read additional field"))?;
//eprintln!("got key length {} -> {}", key_length, key_length & 0x1);
if (key_length & 0x1) != 0 {
let hash_of_next_entry = peeker.read_u8().map_err(|_| EvalError::Value("Invalid skip hash"))?;
if looking_for_hash >= hash_of_next_entry {
peeker.skip(key_length as usize >> 1).map_err(|_| EvalError::Value("seek yielded invalid index in value"))?;
}
*value = peeker;
continue;
}
key_length = (key_length >> 1) - 1;
let current_hash = peeker.read_u8().map_err(|_| EvalError::Value("Invalid hash"))?;
let current_key = peeker.read_bytes(key_length as usize).map_err(|_| EvalError::Value("Invalid key"))?;
//eprintln!("looking at {}/{} -> {:?}", hash_of_key, String::from_utf8_lossy(key.key), sorted_key.cmp(&key));
match (looking_for_hash,looking_for_key).cmp(&(current_hash, current_key)) {
Ordering::Less => break,
Ordering::Equal => return Ok(()),
Ordering::Greater => {},
}
let length_and_type = peeker.read_var_u64().map_err(|_| EvalError::Value("Could not read additional field"))?;
let length = DecodedValue::length_of_binary(length_and_type);
peeker.skip(length).map_err(|_| EvalError::Value("seek yielded invalid index in value"))?;
*value = peeker;
}
Err(EvalError::FieldNotFound(core::str::from_utf8(looking_for_key).map_err(|_| EvalError::Dynamic("invalid UTF8 for key"))?))
}
fn read_hashed_string<'a>(expression: &mut RawReader<'a>) -> Result<(KeyHash,&'a [u8]),EvalError<'a>> {
let hash = expression.read_u8().map_err(|_| EvalError::Expression("read hashed string resulted in error"))?;
let key = expression.read_var_string().map_err(|_| EvalError::Expression("read hashed string resulted in error"))?;
Ok((hash, key))
}
fn get_double<'a,'b>(v: &DecodedValue<'a>) -> Result<f64,EvalError<'b>> {
match v {
DecodedValue::Int(n) => Ok(*n as f64),
DecodedValue::Float(n) => Ok(*n as f64),
DecodedValue::Double(n) => Ok(*n),
_ => Err(EvalError::Dynamic("expected a number, found something else")),
}
}
fn compare<'a,'b,'c>(lhs: &DecodedValue<'a>, rhs: &DecodedValue<'a>, scope: &mut MemoryScope<'c>) -> Result<Ordering,EvalError<'b>> where 'c: 'b {
if let (DecodedValue::Int(l), DecodedValue::Int(r)) = (&lhs, &rhs) {
return Ok(l.cmp(r));
}
if let (DecodedValue::Float(l), DecodedValue::Float(r)) = (&lhs, &rhs) {
return match l.partial_cmp(r) {
None => Err(EvalError::Dynamic("uncomparable floats")),
Some(v) => Ok(v),
};
}
if let (DecodedValue::Double(l), DecodedValue::Double(r)) = (&lhs, &rhs) {
return match l.partial_cmp(r) {
None => Err(EvalError::Dynamic("uncomparable doubles")),
Some(v) => Ok(v),
};
}
Err(EvalError::Dynamic(write!(scope, "trying to compare two values that differ in type: {} and {}", lhs.type_string(), rhs.type_string())))
}
fn find_subsequence<T>(haystack: &[T], needle: &[T]) -> Option<usize>
where for<'a> &'a [T]: PartialEq
{
haystack.windows(needle.len()).position(|window| window == needle)
}
fn evaluate_to_decoded_value<'a,'b,'c,'d>(expression: &mut RawReader<'b>, context: &mut Context<'a,'b,'d>, allocator: &mut MemoryScope<'c>, depth: u8) -> Result<DecodedValue<'b>,EvalError<'b>>
where 'c: 'b, 'b: 'a, 'd: 'b
{
if depth == 0 {
return Err(EvalError::EvalTooLong());
}
let original = *expression;
let opcode = expression.read_u8().map_err(|_| EvalError::Expression("Expected more bytes in expression"))?;
if opcode == ExpryBytecode::And as u8 ||
opcode == ExpryBytecode::Or as u8 {
let mut sub = RawReader::with(expression.read_var_string().map_err(|_| EvalError::Expression("corrupted AND/OR"))?);
let default = opcode == ExpryBytecode::And as u8;
while !sub.is_empty() {
let value = evaluate_to_decoded_value(&mut sub, context, allocator, depth-1)?;
if let DecodedValue::Bool(v) = value {
if v != default {
return Ok(DecodedValue::Bool(!default));
}
} else {
return Err(EvalError::Dynamic(write!(allocator, "value for AND/OR was not a boolean, but {}", value.type_string())));
}
}
return Ok(DecodedValue::Bool(default));
}
if opcode == ExpryBytecode::Call as u8 {
let function_name = expression.read_var_string().map_err(|_| EvalError::Expression("invalid CALL"))?;
let arg_count = expression.read_var_u64().map_err(|_| EvalError::Expression("invalid CALL"))?;
if arg_count >= MAX_FUNCTION_ARGS as u64 {
return Err(EvalError::Expression("too many function call arguments"));
}
let args : &mut [DecodedValue] = ScopedArrayBuilder::with_size(allocator, arg_count as usize, DecodedValue::default()).build();
for i in 0..arg_count as usize {
let v = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
args[i] = v;
}
return context.custom.call(function_name, args, allocator).map_err(|err| EvalError::Dynamic(write!(allocator, "error during call to '{}(..)':\n{}", String::from_utf8_lossy(function_name), err)));
}
if opcode == ExpryBytecode::All as u8 {
let position = expression.read_u8().map_err(|_| EvalError::Expression("invalid ALL"))?;
if let Some(value) = context.originals.get(position as usize) {
let mut reader = RawReader::with(value.get());
return DecodedValue::decode(&mut reader).map_err(|_| EvalError::Value("Corrupted original input"));
}
return Err(EvalError::Expression("invalid input value specified (probably related to a lambda)"));
}
#[cfg(not(feature = "mini"))]
if opcode == ExpryBytecode::FieldAccessBinary as u8 {
let mut output = Vec::with_capacity(1);
evaluate_to_binary(expression, context, allocator, &mut output, depth-1)?;
// This op can be implemented faster because the value does not need to be decoded at all for the field access to work.
// fast_path should only be triggered for operations that return a whole value in output
// (such as Concrete, All, Field, FieldAccess*), so there is no additional copy involved
// (to concat the values).
debug_assert!(output.len() == 1);
let output : &[u8] = concat_output(&output, allocator);
let rhs = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
if let DecodedValue::String(key) = rhs {
let lhs = expry_decode_lazy(output).map_err(|_| EvalError::Value("Corrupted lhs for field access"))?;
if let LazyDecodedValue::Object(obj) = lhs {
if let Some(value) = obj.lookup_binary(key_u8(key)).map_err(|_| EvalError::Value("Corrupted lhs for field access"))? {
let retval = expry_decode(value.get()).map_err(|_| EvalError::Value("Corrupted field from value"))?;
return Ok(retval);
} else {
return Err(EvalError::FieldNotFound(core::str::from_utf8(key).map_err(|_| EvalError::Dynamic("invalid UTF8 for key"))?));
}
} else {
return Err(EvalError::Dynamic(write!(allocator, "field access on a non-object: {}", lhs.type_string())));
}
} else {
return Err(EvalError::Dynamic(write!(allocator, "field access with a non-string: {}", rhs.type_string())));
}
}
if opcode == ExpryBytecode::FieldAccessDecoded as u8 ||
(cfg!(feature = "mini") && opcode == ExpryBytecode::FieldAccessBinary as u8) {
let lhs = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
if let DecodedValue::Object(mut obj) = lhs {
let rhs = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
if let DecodedValue::String(key) = rhs {
if let Some(value) = obj.get_mut(&key_u8(key)) {
let value = core::mem::take(value);
return Ok(value);
}
return Err(EvalError::FieldNotFound(core::str::from_utf8(key).map_err(|_| EvalError::Dynamic("invalid UTF8 for key"))?));
} else {
return Err(EvalError::Dynamic(write!(allocator, "field access with a non-string: {}", rhs.type_string())));
}
} else {
return Err(EvalError::Dynamic(write!(allocator, "field access on a non-object: {}", lhs.type_string())));
}
}
if opcode == ExpryBytecode::ArrayAccessDecoded as u8 ||
(cfg!(feature = "mini") && opcode == ExpryBytecode::ArrayAccessBinary as u8) {
let lhs = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
if let DecodedValue::Array(mut arr) = lhs {
let rhs = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
if let DecodedValue::Int(index) = rhs {
if index >= 0 && (index as usize) < arr.len() {
let value = core::mem::take(&mut arr[index as usize]);
return Ok(value);
}
return Err(EvalError::Dynamic(write!(allocator, "array out of bounds (index {} of array of length {})", index, arr.len())));
} else {
return Err(EvalError::Dynamic(write!(allocator, "array access with a non-integer subscript: {}", rhs.type_string())));
}
} else {
return Err(EvalError::Dynamic(write!(allocator, "array access on a non-array: {}", lhs.type_string())));
}
}
if opcode == ExpryBytecode::Concrete as u8 {
return DecodedValue::decode(expression).map_err(|_| {
EvalError::Expression("Corrupted concrete value")
});
}
// can be disabled for smaller binary size (no tables to make operation faster are compiled in)
#[cfg(feature = "power")]
if opcode == ExpryBytecode::ArithmeticPower as u8 {
let lhs = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
let rhs = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
let l = get_double(&lhs)?;
let r = get_double(&rhs)?;
return Ok(DecodedValue::Double(l.powf(r)));
}
if opcode == ExpryBytecode::ArithmeticPlus as u8 ||
opcode == ExpryBytecode::ArithmeticMin as u8 ||
opcode == ExpryBytecode::ArithmeticMul as u8 ||
opcode == ExpryBytecode::ArithmeticDiv as u8 ||
opcode == ExpryBytecode::ArithmeticMod as u8 ||
opcode == ExpryBytecode::ArithmeticShiftLeft as u8 ||
opcode == ExpryBytecode::ArithmeticShiftRight as u8 ||
opcode == ExpryBytecode::ArithmeticBitwiseOr as u8 ||
opcode == ExpryBytecode::ArithmeticBitwiseXor as u8 ||
opcode == ExpryBytecode::ArithmeticBitwiseAnd as u8
{
let lhs = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
let rhs = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
if let (DecodedValue::Int(l), DecodedValue::Int(r)) = (&lhs, &rhs) {
if opcode == ExpryBytecode::ArithmeticPlus as u8 {
return Ok(DecodedValue::Int(
l.checked_add(*r).ok_or(EvalError::Dynamic("int add resulted in overflow"))?
));
}
if opcode == ExpryBytecode::ArithmeticMin as u8 {
return Ok(DecodedValue::Int(
l.checked_sub(*r).ok_or(EvalError::Dynamic("int sub resulted in overflow"))?
));
}
if opcode == ExpryBytecode::ArithmeticMul as u8 {
return Ok(DecodedValue::Int(
l.checked_mul(*r).ok_or(EvalError::Dynamic("int mul resulted in overflow"))?
));
}
if opcode == ExpryBytecode::ArithmeticDiv as u8 {
if *r == 0 {
return Err(EvalError::Dynamic("Division by zero"));
}
return Ok(DecodedValue::Int(
l.checked_div(*r).ok_or(EvalError::Dynamic("int div resulted in overflow"))?
));
}
if opcode == ExpryBytecode::ArithmeticMod as u8 {
if *r == 0 {
return Err(EvalError::Dynamic("Modulo zero"));
}
return Ok(DecodedValue::Int(l % r));
}
if opcode == ExpryBytecode::ArithmeticShiftLeft as u8 {
return Ok(DecodedValue::Int(
l.checked_shl((*r).try_into().map_err(|_| EvalError::Dynamic("shift-left resulted in overflow"))?).ok_or(EvalError::Dynamic("shift-left resulted in overflow"))?
));
}
if opcode == ExpryBytecode::ArithmeticShiftRight as u8 {
return Ok(DecodedValue::Int(
l.checked_shr((*r).try_into().map_err(|_| EvalError::Dynamic("shift-right resulted in overflow"))?).ok_or(EvalError::Dynamic("shift-right resulted in overflow"))?
));
}
if opcode == ExpryBytecode::ArithmeticBitwiseOr as u8 {
return Ok(DecodedValue::Int(l | r));
}
if opcode == ExpryBytecode::ArithmeticBitwiseXor as u8 {
return Ok(DecodedValue::Int(l ^ r));
}
if opcode == ExpryBytecode::ArithmeticBitwiseAnd as u8 {
return Ok(DecodedValue::Int(l & r));
}
}
let l = get_double(&lhs)?;
let r = get_double(&rhs)?;
if opcode == ExpryBytecode::ArithmeticPlus as u8 {
return Ok(DecodedValue::Double(l + r));
}
if opcode == ExpryBytecode::ArithmeticMin as u8 {
return Ok(DecodedValue::Double(l - r));
}
if opcode == ExpryBytecode::ArithmeticMul as u8 {
return Ok(DecodedValue::Double(l * r));
}
if opcode == ExpryBytecode::ArithmeticDiv as u8 {
if r == 0.0 {
return Err(EvalError::Dynamic("Division by zero"));
}
return Ok(DecodedValue::Double(l / r));
}
// disabled for smaller binary size
#[cfg(not(feature = "mini"))]
if opcode == ExpryBytecode::ArithmeticMod as u8 {
if r == 0.0 {
return Err(EvalError::Dynamic("Modulo zero"));
}
return Ok(DecodedValue::Double(l % r));
}
return Err(EvalError::Dynamic("Arithmetic operation without proper arguments (maybe a bitwise operation on a float or a comparison between a float and int?)"));
}
#[cfg(not(feature = "mini"))]
if opcode == ExpryBytecode::Equal as u8 ||
opcode == ExpryBytecode::NotEqual as u8
{
let mut output = Vec::with_capacity(16);
evaluate_to_binary(expression, context, allocator, &mut output, depth-1)?;
let divider = output.len();
evaluate_to_binary(expression, context, allocator, &mut output, depth-1)?;
let lhs_output = concat_output(&output[..divider], allocator);
let rhs_output = concat_output(&output[divider..], allocator);
if opcode == ExpryBytecode::NotEqual as u8 {
return Ok(DecodedValue::Bool(lhs_output != rhs_output));
}
return Ok(DecodedValue::Bool(lhs_output == rhs_output));
}
if opcode == ExpryBytecode::Not as u8 {
let lhs = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
if let DecodedValue::Bool(b) = &lhs {
return Ok(DecodedValue::Bool(!b));
}
return Err(EvalError::Dynamic("NOT expects a boolean"));
}
if opcode == ExpryBytecode::ParseEqual as u8 ||
opcode == ExpryBytecode::ParseNotEqual as u8 ||
opcode == ExpryBytecode::Equal as u8 ||
opcode == ExpryBytecode::NotEqual as u8
{
let lhs = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
let rhs = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
if opcode == ExpryBytecode::ParseNotEqual as u8 ||
opcode == ExpryBytecode::NotEqual as u8 {
return Ok(DecodedValue::Bool(lhs != rhs));
}
return Ok(DecodedValue::Bool(lhs == rhs));
}
if opcode == ExpryBytecode::NumericLess as u8 ||
opcode == ExpryBytecode::NumericLessEqual as u8 ||
opcode == ExpryBytecode::NumericGreater as u8 ||
opcode == ExpryBytecode::NumericGreaterEqual as u8
{
let lhs = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
let rhs = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
if opcode == ExpryBytecode::NumericLess as u8 {
return Ok(DecodedValue::Bool(compare(&lhs, &rhs, allocator)? == Ordering::Less));
}
if opcode == ExpryBytecode::NumericLessEqual as u8 {
return Ok(DecodedValue::Bool(compare(&lhs, &rhs, allocator)? != Ordering::Greater));
}
if opcode == ExpryBytecode::NumericGreater as u8 {
return Ok(DecodedValue::Bool(compare(&lhs, &rhs, allocator)? == Ordering::Greater));
}
if opcode == ExpryBytecode::NumericGreaterEqual as u8 {
return Ok(DecodedValue::Bool(compare(&lhs, &rhs, allocator)? != Ordering::Less));
}
unimplemented!();
}
if opcode == ExpryBytecode::StringContains as u8 ||
opcode == ExpryBytecode::StringStartsWith as u8 ||
opcode == ExpryBytecode::StringEndsWith as u8
{
let lhs = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
let rhs = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
match (lhs, opcode, rhs) {
(DecodedValue::String(lhs), op, DecodedValue::String(rhs)) if op == ExpryBytecode::StringContains as u8 => {
if rhs.is_empty() {
return Err(EvalError::Dynamic("string contains operation with zero-size right hand size"));
}
return Ok(DecodedValue::Bool(find_subsequence(lhs, rhs).is_some()));
},
(DecodedValue::String(lhs), op, DecodedValue::String(rhs)) if op == ExpryBytecode::StringStartsWith as u8 => {
return Ok(DecodedValue::Bool(lhs.starts_with(rhs)));
},
(DecodedValue::String(lhs), op, DecodedValue::String(rhs)) if op == ExpryBytecode::StringEndsWith as u8 => {
return Ok(DecodedValue::Bool(lhs.ends_with(rhs)));
},
_ => return Err(EvalError::Dynamic("string operation (contains, starts-with, ends-with) expects two string arguments")),
}
}
if opcode == ExpryBytecode::StringConcat as u8 {
let lhs = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
let rhs = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
match (lhs, rhs) {
(DecodedValue::String(lhs), DecodedValue::String(rhs)) => {
let value = allocator.concat_u8(&[lhs, rhs]);
return Ok(DecodedValue::String(value));
},
(lhs, rhs) => return Err(EvalError::Dynamic(write!(allocator, "string concat expects two string arguments, not {} and {}", lhs.type_string(), rhs.type_string()))),
}
}
if opcode == ExpryBytecode::Length as u8 {
let lhs = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
match lhs {
DecodedValue::String(s) => return Ok(DecodedValue::Int(s.len() as i64)),
DecodedValue::Array(arr) => return Ok(DecodedValue::Int(arr.len() as i64)),
DecodedValue::Object(obj) => return Ok(DecodedValue::Int(obj.len() as i64)),
_ => return Err(EvalError::Dynamic(write!(allocator, "taking len() from non supported value type: {}", lhs.type_string()))),
}
}
if opcode == ExpryBytecode::ArrayReverse as u8 {
let lhs = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
match lhs {
DecodedValue::Array(mut arr) => {
arr.reverse();
return Ok(DecodedValue::Array(arr));
},
_ => return Err(EvalError::Dynamic(write!(allocator, "rev() from non supported value type: {}", lhs.type_string()))),
}
}
if opcode == ExpryBytecode::StringDirname as u8 ||
opcode == ExpryBytecode::StringBasename as u8
{
let string = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
if let DecodedValue::String(string) = string {
let mut it = string.rsplitn(2, |x| *x == b'/');
if let (Some(basename), dirname) = (it.next(), it.next()) {
return Ok(DecodedValue::String(if opcode == ExpryBytecode::StringBasename as u8 {
basename
} else if let Some(dirname) = dirname {
dirname
} else {
b""
}));
}
} else {
return Err(EvalError::Dynamic("trying to apply basename()/dirname() to non supported value type"));
}
}
if opcode == ExpryBytecode::StringSplit as u8 {
let string = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
let max = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
let split_on = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
if let (DecodedValue::String(mut string), DecodedValue::Int(mut max), DecodedValue::String(split_on)) = (string, max, split_on) {
let mut retval = DecodedArray::new();
if max > 1 {
max -= 1;
while let Some(pos) = find_subsequence(string, split_on) {
let (begin, end) = string.split_at(pos);
retval.push(DecodedValue::String(begin));
string = &end[split_on.len()..];
if retval.len() >= max as usize {
break;
}
}
}
if !string.is_empty() {
retval.push(DecodedValue::String(string));
}
return Ok(DecodedValue::Array(retval));
} else {
return Err(EvalError::Dynamic("splitn() method expects to be called on a string, with a int and string argument"));
}
}
if opcode == ExpryBytecode::StringSubstringFromTo as u8 ||
opcode == ExpryBytecode::StringSubstringFrom as u8 {
let string = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
let start = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
let length = if opcode == ExpryBytecode::StringSubstringFromTo as u8 {
evaluate_to_decoded_value(expression, context, allocator, depth-1)?
} else {
DecodedValue::Int(i64::MAX)
};
if let DecodedValue::String(string) = string {
if let DecodedValue::Int(start) = start {
if let DecodedValue::Int(length) = length {
// u32 for 32-bits targets
let start = start.clamp(0, u32::MAX as i64) as usize;
let length = length.clamp(0, u32::MAX as i64) as usize;
let from = core::cmp::min(start, string.len());
let to = core::cmp::min((start).saturating_add(length), string.len());
return Ok(DecodedValue::String(&string[from..to]));
} else {
return Err(EvalError::Dynamic("substring needs an int as third argument"));
}
} else {
return Err(EvalError::Dynamic("substring needs an int as second argument"));
}
} else {
return Err(EvalError::Dynamic("substring from string needs a string as first argument"));
}
}
if opcode == ExpryBytecode::StringUpper as u8 ||
opcode == ExpryBytecode::StringLower as u8 ||
opcode == ExpryBytecode::StringTrim as u8 ||
opcode == ExpryBytecode::StringHex as u8 ||
opcode == ExpryBytecode::StringHtmlescape as u8 ||
opcode == ExpryBytecode::StringUrlescape as u8
{
let lhs = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
match lhs {
DecodedValue::String(s) if opcode == ExpryBytecode::StringUpper as u8 => {
// full utf-8 to_uppercase() makes the resulting binary a lot larger (~50 KiB)
let retval = allocator.copy_u8(s);
for c in retval.iter_mut() {
c.make_ascii_uppercase();
}
return Ok(DecodedValue::String(retval));
},
DecodedValue::String(s) if opcode == ExpryBytecode::StringLower as u8 => {
// full utf-8 to_lowercase() makes the resulting binary a lot larger (~50 KiB)
let retval = allocator.copy_u8(s);
for c in retval.iter_mut() {
c.make_ascii_lowercase();
}
return Ok(DecodedValue::String(retval));
},
DecodedValue::String(mut s) if opcode == ExpryBytecode::StringTrim as u8 => {
let from = s.iter().position(|x| *x != b' ');
if let Some(from) = from {
let to = s.iter().rev().position(|x| *x != b' ');
if let Some(to) = to {
s = &s[from..s.len()-to];
} else {
s = &s[from..];
}
} else {
s = b"";
}
return Ok(DecodedValue::String(s));
},
DecodedValue::String(s) if opcode == ExpryBytecode::StringHex as u8 => {
return Ok(DecodedValue::String(allocator.copy_hex(s).as_bytes()));
},
DecodedValue::String(s) if opcode == ExpryBytecode::StringHtmlescape as u8 => {
return Ok(DecodedValue::String(allocator.copy_with_replacement(s, html_escape_inside_attribute_u8)));
},
DecodedValue::String(s) if opcode == ExpryBytecode::StringUrlescape as u8 => {
return Ok(DecodedValue::String(allocator.copy_with_dynamic_replacement(s, url_escape_u8)));
},
_ => return Err(EvalError::Dynamic("string function expects a string argument"))
}
}
if opcode == ExpryBytecode::ToString as u8 ||
opcode == ExpryBytecode::ToInteger as u8 ||
opcode == ExpryBytecode::ToFloat as u8 ||
opcode == ExpryBytecode::ToDouble as u8
{
let lhs = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
if opcode == ExpryBytecode::ToString as u8 {
if !lhs.is_valid_json() {
return Err(EvalError::Dynamic("TO_STRING resulted in Utf8Error"));
}
return Ok(DecodedValue::String(write!(allocator, "{}", lhs).as_bytes()));
}
if opcode == ExpryBytecode::ToInteger as u8 {
return match lhs {
DecodedValue::Int(v) => Ok(DecodedValue::Int(v)),
DecodedValue::Float(v) => Ok(DecodedValue::Int(v as i64)),
DecodedValue::Double(v) => Ok(DecodedValue::Int(v as i64)),
DecodedValue::String(v) => Ok(DecodedValue::Int(core::str::from_utf8(v).map_err(|_| EvalError::Dynamic("invalid UTF8 for tointeger()"))?.parse().map_err(|_| EvalError::Dynamic("could not parse in tointeger()"))?)),
_ => Err(EvalError::Dynamic("tointeger() expects a number or a string")),
}
}
if opcode == ExpryBytecode::ToFloat as u8 {
return match lhs {
DecodedValue::Int(v) => Ok(DecodedValue::Float(v as f32)),
DecodedValue::Float(v) => Ok(DecodedValue::Float(v)),
DecodedValue::Double(v) => Ok(DecodedValue::Float(v as f32)),
DecodedValue::String(v) => Ok(DecodedValue::Float(core::str::from_utf8(v).map_err(|_| EvalError::Dynamic("invalid UTF8 for tofloat()"))?.parse().map_err(|_| EvalError::Dynamic("could not parse in tofloat()"))?)),
_ => Err(EvalError::Dynamic("tofloat() expects a number or a string")),
}
}
if opcode == ExpryBytecode::ToDouble as u8 {
return match lhs {
DecodedValue::Int(v) => Ok(DecodedValue::Double(v as f64)),
DecodedValue::Float(v) => Ok(DecodedValue::Double(v as f64)),
DecodedValue::Double(v) => Ok(DecodedValue::Double(v)),
DecodedValue::String(v) => Ok(DecodedValue::Double(core::str::from_utf8(v).map_err(|_| EvalError::Dynamic("invalid UTF8 for todouble()"))?.parse().map_err(|_| EvalError::Dynamic("could not parse in todouble()"))?)),
_ => Err(EvalError::Dynamic("todouble() expects a number or a string")),
}
}
unimplemented!();
}
if opcode == ExpryBytecode::ArithmeticUnaryMin as u8 {
let lhs = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
match lhs {
// signed overflow protection here (generating a dynamic error)
// signed int can overflow if -i64::MIN is calculated
DecodedValue::Int(i) => return Ok(DecodedValue::Int(i.checked_neg().ok_or(EvalError::Dynamic("UNARY_MIN resulted in overflow"))?)),
DecodedValue::Float(i) => return Ok(DecodedValue::Float(-i)),
DecodedValue::Double(i) => return Ok(DecodedValue::Double(-i)),
_ => return Err(EvalError::Dynamic("UNARY_MIN expects a number")),
}
}
// FIXME: is_type etc
// FIXME: math functions: ceil, floor, trunc, round, sqrt, abs
if opcode == ExpryBytecode::Defined as u8 ||
opcode == ExpryBytecode::Undefined as u8
{
let mut lhs_reader = RawReader::with(expression.read_var_string().map_err(|_| EvalError::Expression("error in (UN)DEFINED"))?);
let result = evaluate_to_decoded_value(&mut lhs_reader, context, allocator, depth-1);
if opcode == ExpryBytecode::Defined as u8 {
return Ok(DecodedValue::Bool(result.is_ok()));
}
return Ok(DecodedValue::Bool(result.is_err()));
}
if opcode == ExpryBytecode::Try as u8 ||
opcode == ExpryBytecode::NotNullElse as u8 {
let mut lhs_reader = RawReader::with(expression.read_var_string().map_err(|_| EvalError::Expression("error in TRY/NOTNULLELSE"))?);
let rhs_size = expression.read_var_u64().map_err(|_| EvalError::Expression("error in TRY/NOTNULLELSE"))?;
match evaluate_to_decoded_value(&mut lhs_reader, context, allocator, depth-1) {
Ok(DecodedValue::Null) if opcode == ExpryBytecode::NotNullElse as u8 => {
return evaluate_to_decoded_value(expression, context, allocator, depth-1);
},
Ok(v) => {
expression.skip(rhs_size as usize).map_err(|_| EvalError::Expression("error in TRY/NOTNULLELSE"))?;
return Ok(v);
}
Err(err) if opcode == ExpryBytecode::Try as u8 && matches!(err, EvalError::Dynamic(_) | EvalError::FieldNotFound(_)) => {
return evaluate_to_decoded_value(expression, context, allocator, depth-1);
}
Err(err) => {
return Err(err);
}
}
}
if opcode == ExpryBytecode::SliceObjectDynamic as u8 {
let count = expression.read_var_u64().map_err(|_| EvalError::Expression("error in SLICEOBJECT"))?;
let mut retval = DecodedObject::new();
for _ in 0..count {
match evaluate_to_decoded_value(expression, context, allocator, depth-1)? {
DecodedValue::String(key) => {
let hash = key_stable_hash(key);
retval.insert((hash, key), evaluate_to_decoded_value(expression, context, allocator, depth-1)?);
},
_ => {
return Err(EvalError::Dynamic("expected string as key for a SLICEOBJECT"));
}
}
}
return Ok(DecodedValue::Object(retval));
}
if opcode == ExpryBytecode::SliceObjectStatic as u8 {
let count = expression.read_var_u64().map_err(|_| EvalError::Expression("error in SLICEOBJECT"))?;
let mut retval = DecodedObject::new();
for _ in 0..count {
let hash_of_key = expression.read_u8().map_err(|_| EvalError::Expression("invalid hash in SLICEOBJECT"))?;
let key = expression.read_var_string().map_err(|_| EvalError::Expression("invalid key in SLICEOBJECT"))?;
retval.insert((hash_of_key, key), evaluate_to_decoded_value(expression, context, allocator, depth-1)?);
}
return Ok(DecodedValue::Object(retval));
}
if opcode == ExpryBytecode::MergeObjects as u8 {
let mut lhs = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
let mut rhs = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
if let (DecodedValue::Object(lhs), DecodedValue::Object(rhs)) = (&mut lhs, &mut rhs) {
for (k,v) in rhs {
let v = core::mem::take(v);
lhs.insert(*k, v);
}
let v = core::mem::take(lhs);
return Ok(DecodedValue::Object(v));
}
return Err(EvalError::Dynamic(write!(allocator, "merging object (with e.g. spread syntax) expects two objects as arguments: lhs={}, rhs={}", lhs.type_string(), rhs.type_string())));
}
if opcode == ExpryBytecode::SliceArray as u8 {
let count = expression.read_var_u64().map_err(|_| EvalError::Expression("error in SLICEARRAY"))?;
let mut retval = DecodedArray::new();
for _ in 0..count {
retval.push(evaluate_to_decoded_value(expression, context, allocator, depth-1)?);
}
return Ok(DecodedValue::Array(retval));
}
if opcode == ExpryBytecode::Conditional as u8 {
let mut c = RawReader::with(expression.read_var_string().map_err(|_| EvalError::Expression("error in CONDITIONAL condition"))?);
let t = RawReader::with(expression.read_var_string().map_err(|_| EvalError::Expression("error in CONDITIONAL then"))?);
let e = RawReader::with(expression.read_var_string().map_err(|_| EvalError::Expression("error in CONDITIONAL else"))?);
match evaluate_to_decoded_value(&mut c, context, allocator, depth-1) {
Ok(DecodedValue::Bool(c)) => {
return evaluate_to_decoded_value(&mut if c { t } else { e }, context, allocator, depth-1);
},
Ok(t) => {
return Err(EvalError::Dynamic(write!(allocator, "conditional expects a boolean for the condition, not a {}", t.type_string())));
}
Err(err) => {
return Err(err);
}
}
}
// catch for all the opcodes that should be evaluated in evaluate_to_binary as that
// implementation is straightforward and mostly used
if opcode == ExpryBytecode::LambdaFilter as u8 ||
opcode == ExpryBytecode::LambdaArrayMap as u8 ||
opcode == ExpryBytecode::LambdaSortByKey as u8 ||
opcode == ExpryBytecode::LambdaArrayToObject as u8 ||
opcode == ExpryBytecode::LambdaArrayToMap as u8 ||
opcode == ExpryBytecode::LambdaObjectToArray as u8 ||
opcode == ExpryBytecode::FieldAccessBinary as u8 ||
opcode == ExpryBytecode::ArrayAccessBinary as u8 {
*expression = original;
let mut output = Vec::with_capacity(16);
evaluate_to_binary(expression, context, allocator, &mut output, depth-1)?;
let output : &[u8] = concat_output(&output, allocator);
return expry_decode(output).map_err(|_| EvalError::Dynamic("after evaluate_to_binary could not decode value"));
}
Err(EvalError::Expression(write!(allocator, "Unrecognized opcode {}", opcode)))
}
fn to_var_length<'c>(scope: &mut MemoryScope<'c>, len: usize) -> &'c [u8] {
let header_size = size_of_var_u64(len as u64);
let header = scope.alloc(header_size);
let mut writer = RawWriter::with_uninit(header);
writer.write_var_u64(len as u64).unwrap();
debug_assert!(writer.left() == 0);
writer.build()
}
fn create_object_header<'c>(scope: &mut MemoryScope<'c>, hash_of_key: u8, key: &[u8]) -> &'c [u8] {
let prefix = scope.alloc(size_of_var_u64(((key.len() + KEY_HASH_SIZE) << 1) as u64) + KEY_HASH_SIZE + key.len());
let mut writer = RawWriter::with_uninit(prefix);
writer.write_var_u64(((key.len() + KEY_HASH_SIZE) << 1) as u64).unwrap();
writer.write_u8(hash_of_key).unwrap();
writer.write_bytes(key).unwrap();
debug_assert!(writer.left() == 0);
writer.build()
}
pub fn to_expry_array<'a,T>(values: &[T]) -> EncodedValueVec
where
T: std::ops::Deref<Target = [u8]>
{
let total : usize = size_of_var_u64((values.len() as u64) << 1) + values.iter().map(|v| v.len()).sum::<usize>();
let header = (total as u64) << WIDTH_OF_JSON_TYPE_MASK | ValueType::Array as u64;
let mut writer = RawString::with_capacity(total + size_of_var_u64(header));
writer.write_var_u64(header).unwrap_infallible();
writer.write_var_u64((values.len() as u64) << 1).unwrap_infallible();
for e in values {
writer.write_bytes(e.deref()).unwrap_infallible();
}
EncodedValueVec(writer.data)
}
pub fn to_expry_array_to_scope<'a,'c,T>(values: &[T], scope: &mut MemoryScope<'c>) -> EncodedValueRef<'c>
where
T: std::ops::Deref<Target = [u8]>
{
let total : usize = size_of_var_u64((values.len() as u64) << 1) + values.iter().map(|v| v.len()).sum::<usize>();
let header = (total as u64) << WIDTH_OF_JSON_TYPE_MASK | ValueType::Array as u64;
let mut writer = RawScopedArrayBuilder::with_capacity(scope, total + size_of_var_u64(header));
writer.write_var_u64(header).unwrap_infallible();
writer.write_var_u64((values.len() as u64) << 1).unwrap_infallible();
for e in values {
writer.write_bytes(e.deref()).unwrap_infallible();
}
EncodedValueRef(writer.data.build())
}
/// Creates an object with a single key-value pair inside. Expects `value` to be binary encoded.
pub fn expry_object_with_single_field(key: &[u8], value: EncodedValueRef) -> Result<EncodedValueVec,EncodingError> {
let key_size = key.len() + KEY_HASH_SIZE;
let entry = size_of_var_u64((key_size as u64) << 1) + key_size + value.len();
let object_size = size_of_var_u64((entry << WIDTH_OF_JSON_TYPE_MASK) as u64) + entry;
let mut retval : Vec<u8> = vec![0u8; object_size];
let mut writer = RawWriter::with(&mut retval[..]);
write_with_header(&mut writer, |writer,total| {
writer.write_var_u64(total << WIDTH_OF_JSON_TYPE_MASK | ValueType::Object as u64)
}, |writer| {
let hash = key_stable_hash(key);
let key_size = KEY_HASH_SIZE + key.len();
writer.write_var_u64((key_size as u64) << 1)?;
writer.write_u8(hash)?;
writer.write_bytes(key)?;
writer.write_bytes(&value)?;
Ok(())
})?;
assert_eq!(writer.left(), 0);
Ok(EncodedValueVec(retval))
}
/// Create an encoded object from individual _sorted_ fields.
pub fn expry_object_raw<'c>(scope: &mut MemoryScope<'c>, sorted_slice: &[(Key<'_>, &[u8])]) -> Result<&'c [u8],EncodingError> {
let mut writer = RawScopedArrayBuilder::new(scope);
write_binary_object(&mut writer, &mut sorted_slice.iter().map(|(x,y)| (x,y)), |writer: &mut _, x: &&[u8]| writer.write_bytes(x)).unwrap_infallible();
Ok(writer.build())
}
pub fn to_expry_object<Key,T>(values: &std::collections::HashMap<Key,T>) -> EncodedValueVec
where
Key: AsRef<str>,
T: std::ops::Deref<Target = [u8]>
{
let mut capacity = 1;
let mut sortable_values = Vec::new();
for (k,v) in values {
let key = key_str(k.as_ref());
let v = v.deref();
sortable_values.push(((key.0, key.1), v));
capacity += 1 + key.1.len() + v.len(); // hash + key + value length
}
sortable_values.sort_unstable();
let mut writer = RawString::with_capacity(capacity);
write_binary_object(&mut writer, &mut sortable_values.iter().map(|(x,y)| (x,y)), |writer: &mut RawString, x: &&[u8]| writer.write_bytes(x)).unwrap_infallible();
EncodedValueVec(writer.build())
}
pub fn to_expry_object_in_scope<'b,'c,Key,T>(values: &std::collections::HashMap<Key,T>, scope: &mut MemoryScope<'c>) -> EncodedValueRef<'b>
where
Key: AsRef<str>,
T: std::ops::Deref<Target = [u8]>,
'c: 'b,
{
let mut capacity = 1;
let mut sortable_values = ScopedArrayBuilder::with_capacity(scope, values.len());
for (k,v) in values {
let key = key_str(k.as_ref());
let v = v.deref();
sortable_values.push(((key.0, key.1), v));
capacity += 1 + key.1.len() + v.len(); // hash + key + value length
}
let sortable_values = sortable_values.build();
sortable_values.sort_unstable();
let mut writer = RawScopedArrayBuilder::with_capacity(scope, capacity);
write_binary_object(&mut writer, &mut sortable_values.iter().map(|(x,y)| (x,y)), |writer: &mut RawScopedArrayBuilder, x: &&[u8]| writer.write_bytes(x)).unwrap_infallible();
EncodedValueRef(writer.build())
}
// maybe return an array of [u8], to avoid copying. Only concat the value in the expry_slice_func
// function. Using this approach MergeObjects can also be efficiently implemented.
fn evaluate_to_binary<'a,'b,'c,'d>(expression: &mut RawReader<'b>, context: &mut Context<'a,'b,'d>, allocator: &mut MemoryScope<'c>, output: &mut Vec<&'b [u8]>, depth: u8) -> Result<(),EvalError<'b>>
where 'c: 'b, 'b: 'a, 'd: 'b
{
if depth == 0 {
return Err(EvalError::EvalTooLong());
}
let expression_original = *expression;
let opcode = expression.read_u8().map_err(|_| EvalError::Expression("Expected more bytes in expression"))?;
// there are no 'decoded' alternatives (in evaluate_to_decoded_value) for these opcodes, so
// implement here
if opcode == ExpryBytecode::LambdaFilter as u8 ||
opcode == ExpryBytecode::LambdaArrayMap as u8 ||
opcode == ExpryBytecode::LambdaSortByKey as u8 ||
opcode == ExpryBytecode::LambdaArrayToObject as u8 ||
opcode == ExpryBytecode::LambdaArrayToMap as u8 {
let mut lhs = Vec::new();
evaluate_to_binary(expression, context, allocator, &mut lhs, depth-1)?;
if Ok(ExpryBytecode::Lambda as u8) != expression.read_u8() {
return Err(EvalError::Expression("error in lambda array operation"));
}
if Ok(context.originals.len() as u8) != expression.read_u8() {
return Err(EvalError::Expression("error in lambda operation: lambda expected different stack size"));
}
if Ok(1u8) != expression.read_u8() {
return Err(EvalError::Expression("error in lambda array operation: lambda should have one argument"));
}
let lambda_reader = RawReader::with(expression.read_var_string().map_err(|_| EvalError::Expression("error in LambdaFilter"))?);
let lhs : &'b [u8] = concat_output(&lhs, allocator);
if let LazyDecodedValue::Array(mut arr) = expry_decode_lazy(lhs).map_err(|_| EvalError::Value("Corrupted lhs for lambda"))? {
let mut retval = Vec::new();
if opcode == ExpryBytecode::LambdaFilter as u8 {
while let Ok(v) = arr.get_raw() {
context.originals.push(v);
let result = evaluate_to_decoded_value(&mut lambda_reader.clone(), context, allocator, depth-1);
context.originals.pop();
if let DecodedValue::Bool(result) = result? {
if result {
retval.push(v);
}
} else {
return Err(EvalError::Dynamic("error in lambda array operation, lambda should return a boolean"));
}
}
} else if opcode == ExpryBytecode::LambdaArrayMap as u8 {
retval.reserve(arr.remaining() as usize);
while let Ok(v) = arr.get_raw() {
context.originals.push(v);
let mut output = Vec::new();
let result = evaluate_to_binary(&mut lambda_reader.clone(), context, allocator, &mut output, depth-1);
context.originals.pop();
result?;
// FIXME: might eliminate this concat operation by supporting
// to_expry_array_to_scope with a slice of slices argument
let output : &'b [u8] = concat_output(&output, allocator);
retval.push(EncodedValueRef(output));
}
} else if opcode == ExpryBytecode::LambdaSortByKey as u8 {
let mut sorted = Vec::with_capacity(arr.remaining() as usize);
while let Ok(v) = arr.get_raw() {
context.originals.push(v);
let result = evaluate_to_decoded_value(&mut lambda_reader.clone(), context, allocator, depth-1);
context.originals.pop();
sorted.push((result?, v));
}
let mut ok = true;
sorted.sort_unstable_by(|a,b| a.0.partial_cmp(&b.0).unwrap_or_else(|| { ok = false; Ordering::Less }));
if !ok {
return Err(EvalError::Dynamic("sort of NaN floats is not supported"));
}
retval = sorted.into_iter().map(|(_,v)| v).collect();
} else if opcode == ExpryBytecode::LambdaArrayToObject as u8 {
let mut map = DecodedObject::new();
while let Ok(v) = arr.get_raw() {
context.originals.push(v);
let result = evaluate_to_decoded_value(&mut lambda_reader.clone(), context, allocator, depth-1);
context.originals.pop();
if let Ok(DecodedValue::Array(mut arr)) = result {
if let [DecodedValue::String(key),value] = &mut arr[0..2] {
let value = core::mem::take(value);
map.insert(key_u8(key), value);
continue;
}
}
return Err(EvalError::Dynamic("lambda of to_object() should return an array of two elements: [key, value]"));
}
let retval = DecodedValue::Object(map);
output.push(retval.encode_to_scope(allocator).0);
return Ok(());
} else if opcode == ExpryBytecode::LambdaArrayToMap as u8 {
let mut map : BTreeMap<Key<'a>, Vec<(DecodedValue<'a>,DecodedValue<'a>)>> = BTreeMap::new();
while let Ok(v) = arr.get_raw() {
context.originals.push(v);
let result = evaluate_to_decoded_value(&mut lambda_reader.clone(), context, allocator, depth-1);
context.originals.pop();
if let Ok(DecodedValue::Array(mut arr)) = result {
if let [DecodedValue::String(key),sort,value] = &mut arr[0..3] {
let sort = core::mem::take(sort);
let value = core::mem::take(value);
let array = map.entry(key_u8(key)).or_default();
array.push((sort,value));
continue;
}
}
return Err(EvalError::Dynamic("lambda of to_map() should return an array of three elements: [key, sort, value]"));
}
let mut ok = true;
for arr in map.values_mut() {
arr.sort_unstable_by(|a,b| a.0.partial_cmp(&b.0).unwrap_or_else(|| { ok = false; Ordering::Less }));
}
let retval = DecodedValue::Object(map.into_iter().map(|(k,values)| (k, DecodedValue::Array(values.iter().map(|(_,v)| v).cloned().collect()))).collect());
output.push(retval.encode_to_scope(allocator).0);
return Ok(());
} else {
unimplemented!();
}
output.push(to_expry_array_to_scope(&retval, allocator).0);
return Ok(());
}
return Err(EvalError::Dynamic("error in lambda array operation, expected array"));
}
if opcode == ExpryBytecode::LambdaObjectToArray as u8 {
let mut lhs = Vec::new();
evaluate_to_binary(expression, context, allocator, &mut lhs, depth-1)?;
if Ok(ExpryBytecode::Lambda as u8) != expression.read_u8() {
return Err(EvalError::Expression("error in lambda object operation"));
}
if Ok(context.originals.len() as u8) != expression.read_u8() {
return Err(EvalError::Expression("error in lambda operation: lambda expected different stack size"));
}
if Ok(2u8) != expression.read_u8() {
return Err(EvalError::Expression("error in lambda object operation: lambda should have two arguments"));
}
let lambda_reader = RawReader::with(expression.read_var_string().map_err(|_| EvalError::Expression("error in LambdaFilter"))?);
let lhs : &'b [u8] = concat_output(&lhs, allocator);
let lhs = expry_decode_lazy(lhs).map_err(|_| EvalError::Value("Corrupted lhs for lambda"))?;
if let LazyDecodedValue::Object(mut obj) = lhs {
if opcode == ExpryBytecode::LambdaObjectToArray as u8 {
let mut arr = DecodedArray::new();
while let Ok(((_,k),v)) = obj.get_raw() {
context.originals.push(DecodedValue::String(k).encode_to_scope(allocator));
context.originals.push(v);
let result = evaluate_to_decoded_value(&mut lambda_reader.clone(), context, allocator, depth-1);
context.originals.pop();
context.originals.pop();
arr.push(result?);
}
let retval = DecodedValue::Array(arr);
output.push(retval.encode_to_scope(allocator).0);
} else {
unimplemented!();
}
return Ok(());
}
return Err(EvalError::Dynamic(write!(allocator, "error in lambda object operation, expected object, not a {}", lhs.type_string())));
}
if !cfg!(feature = "mini") {
if opcode == ExpryBytecode::All as u8 {
let position = expression.read_u8().map_err(|_| EvalError::Expression("invalid ALL"))?;
if let Some(value) = context.originals.get(position as usize) {
output.push(value.get());
return Ok(());
}
return Err(EvalError::Expression("invalid input value specified (probably related to a lambda)"));
}
if opcode == ExpryBytecode::Concrete as u8 {
let mut expression_copy = *expression;
let type_and_length = expression_copy.read_var_u64().map_err(|_| {
EvalError::Expression("Corrupted concrete value")
})?;
let (_, len) = DecodedValue::decoded_type_and_length(type_and_length).map_err(|_| {
EvalError::Expression("Corrupted concrete value")
})?;
output.push(expression.read_bytes(len + expression.len() - expression_copy.len()).map_err(|_| {
EvalError::Expression("Corrupted concrete value")
})?);
return Ok(());
}
if opcode == ExpryBytecode::FieldAccessBinary as u8 {
let mut lhs_output = Vec::new();
evaluate_to_binary(expression, context, allocator, &mut lhs_output, depth-1)?;
// binary version should only be triggered for operations that return a whole value in output
// (such as Concrete, All, Field, FieldAccess*), so lhs_output.len() == 1
debug_assert!(lhs_output.len() == 1);
let lhs : &'b [u8] = concat_output(&lhs_output, allocator);
let rhs = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
if let DecodedValue::String(key) = rhs {
let lhs = expry_decode_lazy(lhs).map_err(|_| EvalError::Value("Corrupted lhs for field access"))?;
if let LazyDecodedValue::Object(obj) = lhs {
if let Some(value) = obj.lookup_binary(key_u8(key)).map_err(|_| EvalError::Value("Corrupted lhs for field access"))? {
output.push(value.get());
return Ok(());
} else {
return Err(EvalError::FieldNotFound(core::str::from_utf8(key).map_err(|_| EvalError::Dynamic("invalid UTF8 for key"))?));
}
} else {
return Err(EvalError::Dynamic(write!(allocator, "field access on a non-object: {}", lhs.type_string())));
}
} else {
return Err(EvalError::Dynamic(write!(allocator, "field access with a non-string: {}", rhs.type_string())));
}
}
if opcode == ExpryBytecode::ArrayAccessBinary as u8 {
let mut lhs_output = Vec::new();
evaluate_to_binary(expression, context, allocator, &mut lhs_output, depth-1)?;
// binary version should only be triggered for operations that return a whole value in output
// (such as Concrete, All, Field, FieldAccess*), so lhs_output.len() == 1
debug_assert!(lhs_output.len() == 1);
let lhs : &'b [u8] = concat_output(&lhs_output, allocator);
let lhs = expry_decode_lazy(lhs).map_err(|_| EvalError::Value("Corrupted lhs for array access"))?;
if let LazyDecodedValue::Array(mut arr) = lhs {
let rhs = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
if let DecodedValue::Int(index) = rhs {
if index >= 0 && (index as u64) < arr.remaining() {
arr.skip(index as usize).map_err(|_| EvalError::Dynamic("error in array representation during array access"))?;
output.push(arr.get_raw().map_err(|_| EvalError::Dynamic("error in array representation during array access"))?.0);
return Ok(());
}
return Err(EvalError::Dynamic(write!(allocator, "array out of bounds (index {} of array of length {})", index, arr.remaining())));
} else {
return Err(EvalError::Dynamic(write!(allocator, "array access with a non-integer subscript: {}", rhs.type_string())));
}
} else {
return Err(EvalError::Dynamic(write!(allocator, "array access on a non-array: {}", lhs.type_string())));
}
}
// BELOW IS UNTESTED BOTH FOR CORRECTNESS AND PERFORMANCE
// if opcode == ExpryBytecode::SliceObjectStatic as u8 ||
// opcode == ExpryBytecode::SliceObjectDynamic as u8 {
// let count = expression.read_var_u64().map_err(|_| EvalError::Expression("error in SLICEOBJECT"))?;
// let index = output.len();
// output.push(&[]); // reserve space for object header
// // FIXME: we have a Vec of Vec's that is used to sort based on the key
// // However, this is not allocation efficient, so we might need a different approach.
// // The current approach is better than decoding the values though, so for now it is ok.
// let mut unsorted = Vec::new();
// if opcode == ExpryBytecode::SliceObjectStatic as u8 {
// for _ in 0..count {
// let hash_of_key = expression.read_u8().map_err(|_| EvalError::Expression("invalid hash in SLICEOBJECT"))?;
// let key = expression.read_var_string().map_err(|_| EvalError::Expression("invalid key in SLICEOBJECT"))?;
// let mut buffer = Vec::new();
// let prefix = create_object_header(allocator, hash_of_key, key);
// buffer.push(prefix);
// evaluate_to_binary(expression, context, allocator, &mut buffer)?;
// unsorted.push((hash_of_key, key, buffer));
// }
// } else {
// for _ in 0..count {
// match evaluate_to_decoded_value(expression, context, allocator)? {
// DecodedValue::String(key) => {
// let hash_of_key = key_stable_hash(key);
// let mut buffer = Vec::new();
// let prefix = create_object_header(allocator, hash_of_key, key);
// buffer.push(prefix);
// evaluate_to_binary(expression, context, allocator, &mut buffer)?;
// unsorted.push((hash_of_key, key, buffer));
// },
// _ => {
// return Err(EvalError::Dynamic("expected string as key for a SLICEOBJECT"));
// }
// }
// }
// }
// unsorted.sort_unstable();
// for (_, _, mut v) in unsorted {
// output.append(&mut v);
// }
// let size = output[index+1..].iter().fold(0usize, |sum,val| sum + val.len());
// output[index] = to_var_length(allocator, size << WIDTH_OF_JSON_TYPE_MASK | ValueType::Object as usize);
// return Ok(());
// }
if opcode == ExpryBytecode::SliceObjectStatic as u8 {
let count = expression.read_var_u64().map_err(|_| EvalError::Expression("error in SLICEOBJECT"))?;
// SliceObjectStatic should have sorted values
let mut sorted_values = Vec::new();
for _ in 0..count {
let hash_of_key = expression.read_u8().map_err(|_| EvalError::Expression("invalid hash in SLICEOBJECT"))?;
let key = expression.read_var_string().map_err(|_| EvalError::Expression("invalid key in SLICEOBJECT"))?;
// eprintln!("slicing {}/{}", hash_of_key, String::from_utf8_lossy(key));
let mut output = Vec::new();
evaluate_to_binary(expression, context, allocator, &mut output, depth-1)?;
sorted_values.push(((hash_of_key, key), &*allocator.concat_u8(&output)));
// let value = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
// sorted_values.push(((hash_of_key, key), value.encode_to_scope(allocator).get()));
}
let mut writer = RawScopedArrayBuilder::new(allocator);
write_binary_object(&mut writer, &mut sorted_values.iter().map(|(x,y)| (x,y)), |writer: &mut _, x: &&[u8]| writer.write_bytes(x)).unwrap_infallible();
output.push(writer.build());
return Ok(());
}
if opcode == ExpryBytecode::Try as u8 {
let mut lhs_reader = RawReader::with(expression.read_var_string().map_err(|_| EvalError::Expression("error in TRY"))?);
let rhs_size = expression.read_var_u64().map_err(|_| EvalError::Expression("error in TRY"))?;
let output_len = output.len();
match evaluate_to_binary(&mut lhs_reader, context, allocator, output, depth-1) {
Ok(()) => {
expression.skip(rhs_size as usize).map_err(|_| EvalError::Expression("error in TRY"))?;
return Ok(());
}
Err(err) if matches!(err, EvalError::Dynamic(_) | EvalError::FieldNotFound(_)) => {
output.truncate(output_len);
return evaluate_to_binary(expression, context, allocator, output, depth-1);
}
Err(err) => {
return Err(err);
}
}
}
// FIXME: add case for NotNullElse
// FIXME: add case for SliceArray
}
// use dynamic evaluation and convert to a binary value
*expression = expression_original;
let value = evaluate_to_decoded_value(expression, context, allocator, depth-1)?;
let len = value.size_of_binary();
let buffer = allocator.alloc(len);
let mut writer = RawWriter::with_uninit(buffer);
value.print_binary(&mut writer).map_err(|_| EvalError::Other())?;
output.push(writer.build());
Ok(())
}
// optimizes the case that lhs is a single value, don't copy if that is the case
fn concat_output<'b,'c>(lhs: &[&'b [u8]], allocator: &mut MemoryScope<'c>) -> &'b [u8] where 'c: 'b {
if lhs.len() == 1 {
return lhs[0];
}
allocator.concat_u8(lhs)
}
// FIXME: add properties to the result value (as a seperate struct):
// - needs dynamic evaluation;
// - is a simple slicer (only returns 1 part of the input value or bytecode)
// - is a slicer (only returns a composition of multiple values)
/// Compiles an expression to bytecode that can quickly be evaluated using [`expry_eval`] and
/// [`expry_slice`].
pub fn expry_compile_slice<'b,'c>(expr: &'b str, static_value: Option<&DecodedObject<'b>>, allow_unbound_variables: Option<u8>, dynamic_value_names: &[&str], scope: &mut MemoryScope<'c>) -> Result<(BytecodeRef<'b>,bool), CompileError<'b>> where 'c: 'b {
_expry_compile(true, expr, static_value, allow_unbound_variables, dynamic_value_names, scope)
}
pub fn expry_compile_expr<'b,'c>(expr: &'b str, static_value: Option<&DecodedObject<'b>>, allow_unbound_variables: Option<u8>, dynamic_value_names: &[&str], scope: &mut MemoryScope<'c>) -> Result<(BytecodeRef<'b>,bool), CompileError<'b>> where 'c: 'b {
_expry_compile(false, expr, static_value, allow_unbound_variables, dynamic_value_names, scope)
}
fn _expry_compile<'b,'c>(slice: bool, expr: &'b str, static_value: Option<&DecodedObject<'b>>, allow_unbound_variables: Option<u8>, dynamic_value_names: &[&str], scope: &mut MemoryScope<'c>) -> Result<(BytecodeRef<'b>,bool), CompileError<'b>> where 'c: 'b {
if expr.len() >= u32::MAX as usize {
return Err(CompileError{expr, error: CompileErrorDescription::Parser("input too large to parse (max 4 GiB)"), start: expr.len(), end: 0, extra: None});
}
if let Some(position) = allow_unbound_variables {
if position as usize >= dynamic_value_names.len() {
return Err(CompileError{expr, error: CompileErrorDescription::Parser(write!(scope, "unbound variables should be resolved to one of the dynamic values (position {} specified, {} dynamic values given)", position, dynamic_value_names.len())), start: expr.len(), end: 0, extra: None});
}
}
let default_value = DecodedObject::new();
let static_value = static_value.unwrap_or(&default_value);
#[allow(unused_mut)]
let mut ast = parse_expry_expr(slice, expr, static_value, allow_unbound_variables, dynamic_value_names, scope).map_err(|(error, start, end, extra)| CompileError{ expr, error, start, end, extra })?;
#[cfg(not(feature = "mini"))]
{
// FIXME: return source range where the error is triggered (in the fold/optimize)
expry_ast_fold(&mut ast, scope).map_err(|x| CompileError{expr, error: CompileErrorDescription::Optimizer(x), start: 0, end: 0, extra: None})?;
expry_ast_optimize(&mut ast).map_err(|x| CompileError{expr, error: CompileErrorDescription::Optimizer(x), start: 0, end: 0, extra: None})?;
}
let needs_dynamic_eval = ast_properties(&ast).needs_dynamic_eval;
let arg_count = if needs_dynamic_eval {
if dynamic_value_names.len() >= MAX_DYNAMIC_VALUES as usize {
return Err(CompileError{expr, error: CompileErrorDescription::Parser(write!(scope, "too many arguments: {} is above max of {}", dynamic_value_names.len(), MAX_DYNAMIC_VALUES)), start: expr.len(), end: 0, extra: None});
}
dynamic_value_names.len().try_into().unwrap()
} else {
0u8
};
let bytecode = expry_ast_to_bytecode(scope, &mut ast, arg_count).map_err(|x| CompileError{expr, error: CompileErrorDescription::Parser(x), start: 0, end: 0, extra: None})?;
Ok((bytecode, needs_dynamic_eval))
}
pub fn expry_compile_expr_typed<'b,'c>(expr: &'b str, static_value: Option<&DecodedObject<'b>>, allow_unbound_variables: Option<u8>, types: &[(&str,ExpryType)], custom_functions: &BTreeMap<Key,(Vec<ExpryType>,ExpryType)>, scope: &mut MemoryScope<'c>) -> Result<(BytecodeRef<'b>, bool, ExpryType, Vec<ExpryTypeWarning<'b>>), CompileError<'b>> where 'c: 'b {
let (bytecode, needs_dynamic_eval) = expry_compile_expr(expr, static_value, allow_unbound_variables, &types.iter().map(|(name,_)| *name).collect::<Vec<&str>>(), scope)?;
let (return_type, warnings) = expry_type_from_bytecode(bytecode, &types.iter().map(|(_,t)| t).collect::<Vec<&ExpryType>>(), custom_functions, scope).map_err(|x| CompileError{expr, error: CompileErrorDescription::Types(x), start: 0, end: 0, extra: None})?;
Ok((bytecode, needs_dynamic_eval, return_type, warnings))
}
/// Formats errors from compilation with a nice layout and underlines on the source lines, so that
/// users can easily understand what to do. Returns the line number of the main error, and a string
/// that can be displayed to the user.
pub fn expry_compile_error_format(e: &CompileError, extra_line_no: u32) -> (u32,String) {
let mut retval = String::new();
let line_context = LineContext::new(e.expr);
let (line_no, prefix, error_msg) = line_context.format_error_context(e.expr, e.start, e.end, extra_line_no).map_or((0u32, String::new(), String::new()), |x| x);
if line_no > 0 {
writeln!(&mut retval, "{}{}error at line {}:{} {}", prefix, TERM_BRIGHT_RED, line_no, TERM_RESET, e.error).ok();
retval.push_str(&error_msg);
}
if let Some((start, end)) = e.extra {
let (line_no, prefix, error_msg) = line_context.format_error_context(e.expr, start, end, extra_line_no).map_or((0u32, String::new(), String::new()), |x| x);
if line_no > 0 {
writeln!(&mut retval, "{}{}related at line {}:{} expected it here (or earlier)", prefix, TERM_DIM_CYAN, line_no, TERM_RESET).ok();
retval.push_str(&error_msg);
}
}
(line_no, retval)
}
pub fn expry_compile_error_format_short(e: &CompileError) -> String {
let mut retval = String::new();
let line_context = LineContext::new(e.expr);
let error_msg = line_context.format_error_context_short(e.expr, e.start, e.end).map_or(String::new(), |x| x);
writeln!(&mut retval, "error: {}", e.error).ok();
retval.push_str(&error_msg);
if let Some((start, end)) = e.extra {
let error_msg = line_context.format_error_context_short(e.expr, start, end).map_or(String::new(), |x| x);
retval.push_str("\nrelated: expected it here (or earlier)\n");
retval.push_str(&error_msg);
}
retval
}
pub fn expry_type_warnings_to_string(warnings: &[ExpryTypeWarning]) -> String {
let mut buffer = String::new();
write!(&mut buffer, "possibly non existant field used without error handling: ").unwrap();
for w in warnings {
match w {
ExpryTypeWarning::PossibleUnsetField(name) => { write!(&mut buffer, "{}, ", String::from_utf8_lossy(name)).unwrap(); },
}
}
buffer
}
/// Evaluate expression bytecode in the context of the given `value`, resulting in a **decoded**
/// value.
pub fn expry_eval<'b,'c>(bytecode: BytecodeRef<'b>, values: &mut Vec<EncodedValueRef<'b>>, allocator: &mut MemoryScope<'c>) -> Result<DecodedValue<'b>, EvalError<'b>> where 'c: 'b {
let funcs = NoCustomFuncs{};
expry_eval_func(bytecode, values, allocator, &funcs)
}
/// Evaluate expression bytecode in the context of the given `value`, resulting in a **decoded**
/// value. This version supports custom user defined functions.
pub fn expry_eval_func<'a,'b,'c,'d>(bytecode: BytecodeRef<'b>, values: &mut Vec<EncodedValueRef<'b>>, allocator: &mut MemoryScope<'c>, funcs: &'a dyn CustomFuncs<'d>) -> Result<DecodedValue<'b>, EvalError<'b>>
where 'c: 'b, 'b: 'a, 'd: 'b
{
let mut bytecode_reader = RawReader::with(bytecode.0);
let magic = bytecode_reader.read_bytes(EXPRY_MAGIC.len());
if magic != Ok(EXPRY_MAGIC) {
return Err(EvalError::Expression("not an expry expression"));
}
let count = bytecode_reader.read_u8().map_err(|_| EvalError::Expression("expry bytecode error"))?;
if count > 0 && count != values.len() as u8 {
return Err(EvalError::Expression(write!(allocator, "expry expression expected {} arguments instead of the given {} arguments", count, values.len())));
}
let value_count = values.len();
let mut context = Context { originals: values, custom: funcs};
let result = evaluate_to_decoded_value(&mut bytecode_reader, &mut context, allocator, MAX_RECURSION);
values.truncate(value_count);
result
}
/// Evaluate expression bytecode in the context of the given `value`, resulting in an **encoded**
/// value. This version supports custom user defined functions.
///
/// This object can be directly transfered over a wire.
///
/// This version is cheaper because values
/// can be included directly in encoded form, so a decoding step is skipped.
pub fn expry_slice_func<'a,'b,'c,'d>(bytecode: BytecodeRef<'b>, values: &mut Vec<EncodedValueRef<'b>>, allocator: &'a mut MemoryScope<'c>, funcs: &'a dyn CustomFuncs<'d>) -> Result<EncodedValueRef<'b>, EvalError<'b>>
where 'c: 'b, 'b: 'a, 'd: 'b
{
let mut bytecode_reader = RawReader::with(bytecode.0);
let magic = bytecode_reader.read_bytes(EXPRY_MAGIC.len());
if magic != Ok(EXPRY_MAGIC) {
return Err(EvalError::Expression("not an expry expression"));
}
let count = bytecode_reader.read_u8().map_err(|_| EvalError::Expression("not an expry expression"))?;
if count > 0 && count != values.len() as u8 {
return Err(EvalError::Expression(write!(allocator, "expry expression expected {} arguments instead of the given {} arguments", count, values.len())));
}
let value_count = values.len();
let mut context = Context { originals: values, custom: funcs};
let mut output = Vec::with_capacity(16);
let result = evaluate_to_binary(&mut bytecode_reader, &mut context, allocator, &mut output, MAX_RECURSION);
values.truncate(value_count);
result?;
debug_assert!(!output.is_empty());
let output = concat_output(&output, allocator);
if cfg!(debug_assertions) {
let mut reader = RawReader::with(output);
let len = reader.read_var_u64().unwrap();
debug_assert_eq!(DecodedValue::length_of_binary(len), reader.len());
debug_assert!(expry_decode(output).map_err(|err| eprintln!("error = {}", err)).is_ok());
}
Ok(EncodedValueRef(output))
}
/// Evaluate expression bytecode in the context of the given `value`, resulting in an **encoded**
/// value.
///
/// The encoded resulting value can be directly transfered over a wire.
///
/// This version is cheaper because values
/// can be included directly in encoded form, so a decoding step is skipped.
pub fn expry_slice<'b,'c>(bytecode: BytecodeRef<'b>, values: &mut Vec<EncodedValueRef<'b>>, allocator: &mut MemoryScope<'c>) -> Result<EncodedValueRef<'b>, EvalError<'b>> where 'c: 'b {
expry_slice_func(bytecode, values, allocator, &NoCustomFuncs{})
}
// helper data type to type check a binary expression
#[derive(PartialEq, Eq, Clone, Default)]
pub enum ExpryType {
#[default]
Any, // FIXME: Sometimes we can not type a expression, because it depends on a dynamic value (like object.get(name)). So we type such expressions as Any. This will be equal to any other type, as it might work. The goal of this type checker is to catch errors compile time, not exclude these kind of language statements. In the next version we force a more strict type, in that we enforce types with the to_x() methods. Only these methods accept the Any type (because they throw a runtime error if wrong type, and they enforce the type afterwards). Needs an ensure_object(_, typespec) with typespec a type spec.
Null,
Bool,
Int,
Float,
Double,
String,
Object(TypeObject),
UniformObject(Box<ExpryType>),
Array(Box<ExpryType>), // can be empty if it is an empty array
EmptyArray,
StaticArray(Vec<ExpryType>), // always the same number of elements
Nullable(Box<ExpryType>),
}
impl std::fmt::Debug for ExpryType {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
ExpryType::Any => write!(f, "*"),
ExpryType::Null => write!(f, "null"),
ExpryType::Bool => write!(f, "bool"),
ExpryType::Int => write!(f, "int"),
ExpryType::Float => write!(f, "float"),
ExpryType::Double => write!(f, "double"),
ExpryType::String => write!(f, "string"),
ExpryType::Object(v) => {
write!(f, "{{")?;
for (k,v) in v {
write!(f, "{}: {:?},", String::from_utf8_lossy(k), v)?;
}
write!(f, "}}")
},
ExpryType::UniformObject(v) => write!(f, "{{{:?}}}", v),
ExpryType::Array(v) => write!(f, "[{:?}]", v),
ExpryType::StaticArray(v) => {
write!(f, "[")?;
for v in v {
write!(f, "{:?},", v)?;
}
write!(f, "]")
},
ExpryType::EmptyArray => write!(f, "[]"),
ExpryType::Nullable(v) => write!(f, "{:?}?", v),
}
}
}
impl ExpryType {
pub fn used_as(&self, other: &Self) -> bool {
if *self == Self::Any {
return true;
}
if *other == Self::Any {
return true;
}
if let ExpryType::Nullable(rhs) = other {
if self == &ExpryType::Null {
return true;
}
if let ExpryType::Nullable(lhs) = self {
return (**lhs).used_as(&**rhs);
} else {
return self.used_as(&**rhs);
}
}
if let ExpryType::Nullable(_) = self {
if other == &ExpryType::Null {
return true;
}
}
if matches!((self, other),
(ExpryType::EmptyArray, ExpryType::Array(_))) {
return true;
}
if let (ExpryType::Array(lhs),ExpryType::Array(rhs)) = (self, other) {
return (**lhs).used_as(&**rhs);
}
if let (Self::UniformObject(lhs), Self::UniformObject(rhs)) = (self, other) {
return (**lhs).used_as(&**rhs);
}
if let (Self::Object(lhs), Self::UniformObject(rhs)) = (self, other) {
for (_lhs_required,lhs_v) in lhs.values() {
if !lhs_v.used_as(rhs) {
return false;
}
}
return true;
}
if let (Self::Object(lhs), Self::Object(rhs)) = (self, other) {
for (k,(rhs_required,rhs_v)) in rhs {
if let Some((lhs_required,lhs_v)) = lhs.get(k) {
if !*lhs_required && *rhs_required {
// we have an optional field, that can not be converted to a required field
return false;
}
// both are required,
// or both not required,
// or lhs is optional and can be converted to from a required field
if !lhs_v.used_as(rhs_v) {
return false;
}
} else if *rhs_required {
return false;
}
}
return true;
}
return self == other;
}
}
impl core::fmt::Display for ExpryType {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
ExpryType::Any => write!(f, "*"),
ExpryType::Null => write!(f, "null"),
ExpryType::Bool => write!(f, "bool"),
ExpryType::Int => write!(f, "int"),
ExpryType::Float => write!(f, "float"),
ExpryType::Double => write!(f, "double"),
ExpryType::String => write!(f, "string"),
ExpryType::Object(x) => {
write!(f, "{{")?;
for (k,(required,v)) in x {
write!(f, "{}{}:{},", String::from_utf8_lossy(k), if *required { "" } else { "?" }, v)?;
}
write!(f, "}}")
},
ExpryType::UniformObject(x) => write!(f, "{{*: {}}}", x),
ExpryType::Array(x) => write!(f, "[{}]", x),
ExpryType::StaticArray(x) => {
write!(f, "[")?;
for v in x {
write!(f, "{},", v)?;
}
write!(f, "]")
},
ExpryType::EmptyArray => write!(f, "[]"),
ExpryType::Nullable(x) => write!(f, "{}?", x),
}
}
}
pub type TypeObject = BTreeMap<Vec<u8>, (bool,ExpryType)>; // bool indcates if it is required
#[derive(Clone,Copy,Debug,Eq,PartialEq)]
pub enum TypeError<'a> {
Untypable(&'a str), // can work runtime, not sure
InconsistentTypes(&'a str), // should never happen
FieldNotFound(&'a [u8]), // should never happen
Bytecode,
UnsupportedOpcode(u8),
}
impl<'a> core::fmt::Display for TypeError<'a> {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
match self {
TypeError::Untypable(msg) => write!(f, "untypable: {}", msg),
TypeError::InconsistentTypes(msg) => write!(f, "inconsistent: {}", msg),
TypeError::FieldNotFound(msg) => write!(f, "field not found in type: {}", String::from_utf8_lossy(msg)),
TypeError::Bytecode => write!(f, "bytecode problem"),
TypeError::UnsupportedOpcode(opcode) => write!(f, "Unsupported opcode {} in type checker", opcode),
}
}
}
pub enum ExpryTypeWarning<'a> {
PossibleUnsetField(&'a [u8]),
}
pub fn expry_type_from_bytecode<'b,'c>(bytecode: BytecodeRef<'b>, types: &[&ExpryType], custom_functions: &BTreeMap<Key,(Vec<ExpryType>,ExpryType)>, allocator: &mut MemoryScope<'c>) -> Result<(ExpryType,Vec<ExpryTypeWarning<'b>>),TypeError<'b>> where 'c: 'b
{
let mut bytecode_reader = RawReader::with(bytecode.0);
bytecode_reader.read_bytes(EXPRY_MAGIC.len()).unwrap();
bytecode_reader.read_u8().unwrap();
let mut warnings = Vec::new();
let mut context = TypeContext {
custom_functions,
warnings: &mut warnings,
};
let return_type = get_type_from_bytecode(&mut bytecode_reader, types, &mut context, allocator)?;
Ok((return_type,warnings))
}
struct TypeContext<'a, 'b> {
custom_functions: &'a BTreeMap<Key<'a>,(Vec<ExpryType>,ExpryType)>,
warnings: &'a mut Vec<ExpryTypeWarning<'b>>,
}
fn get_type_from_bytecode<'b,'c>(expression: &mut RawReader<'b>, types: &[&ExpryType], context: &mut TypeContext<'_, 'b>, allocator: &mut MemoryScope<'c>) -> Result<ExpryType,TypeError<'b>> where 'c: 'b
{
let opcode = expression.read_u8().map_err(|_| TypeError::Bytecode)?;
if opcode == ExpryBytecode::And as u8 ||
opcode == ExpryBytecode::Or as u8 {
let mut sub = RawReader::with(expression.read_var_string().map_err(|_| TypeError::Bytecode)?);
while !sub.is_empty() {
let clause = get_type_from_bytecode(&mut sub, types, context, allocator)?;
if !clause.used_as(&ExpryType::Bool) {
return Err(TypeError::InconsistentTypes(write!(allocator, "value for AND/OR was not a boolean, but {}", clause)));
}
}
return Ok(ExpryType::Bool);
}
if opcode == ExpryBytecode::Call as u8 {
let function_name = expression.read_var_string().map_err(|_| TypeError::Bytecode)?;
let arg_count = expression.read_var_u64().map_err(|_| TypeError::Bytecode)?;
if arg_count >= MAX_FUNCTION_ARGS as u64 {
return Err(TypeError::Bytecode);
}
if let Some((args,return_type)) = context.custom_functions.get(&key_u8(function_name)) {
if arg_count != args.len() as u64 {
return Err(TypeError::InconsistentTypes(write!(allocator, "Wrong number of arguments for custom function '{}' ({} instead of {})", String::from_utf8_lossy(function_name), arg_count, args.len())));
}
for (i,t) in args.iter().enumerate() {
let actual_type = get_type_from_bytecode(expression, types, context, allocator)?;
if !actual_type.used_as(t) {
return Err(TypeError::InconsistentTypes(write!(allocator, "Argument {} for custom function '{}' expects {}, not {}", i, String::from_utf8_lossy(function_name), t, actual_type)));
}
}
return Ok(return_type.clone());
}
return Err(TypeError::InconsistentTypes(write!(allocator, "Unknown custom function called '{}'", String::from_utf8_lossy(function_name))));
}
if opcode == ExpryBytecode::All as u8 {
let position = expression.read_u8().map_err(|_| TypeError::Bytecode)?;
if let Some(value) = types.get(position as usize) {
return Ok((*value).clone());
}
return Err(TypeError::InconsistentTypes(write!(allocator, "invalid input value specified (probably related to a lambda, position {})", position)));
}
if opcode == ExpryBytecode::FieldAccessDecoded as u8 ||
opcode == ExpryBytecode::FieldAccessBinary as u8 {
let mut lhs = get_type_from_bytecode(expression, types, context, allocator)?;
if let ExpryType::Nullable(v) = lhs {
lhs = *v;
}
if let ExpryType::Object(obj) = lhs {
let mut expression_copy = *expression;
if expression_copy.read_u8().map_err(|_| TypeError::Bytecode)? == ExpryBytecode::Concrete as u8 {
*expression = expression_copy;
let key = DecodedValue::decode(expression).map_err(|_| {
TypeError::Bytecode
})?;
if let DecodedValue::String(key) = key {
if let Some((required,value)) = obj.get(key) {
// emit warning about error that can happen
if !required {
context.warnings.push(ExpryTypeWarning::PossibleUnsetField(key));
}
return Ok(value.clone());
} else {
return Err(TypeError::FieldNotFound(key));
}
}
}
let mut rhs = get_type_from_bytecode(expression, types, context, allocator)?;
if let ExpryType::Nullable(inner) = rhs {
rhs = *inner;
}
if rhs.used_as(&ExpryType::String) {
// FIXME: if we have an enforced type here we can return that, so something as
// FIXME: maybe add a strict mode, that disallows returning a type here, because typing errors can not be excluded.
// object.get<string>("foo") or the like.
return Ok(ExpryType::Any);
}
return Err(TypeError::InconsistentTypes(write!(allocator, "field access with a non-string: {}", rhs)));
} else if let ExpryType::UniformObject(fields_type) = lhs {
let mut rhs = get_type_from_bytecode(expression, types, context, allocator)?;
if let ExpryType::Nullable(inner) = rhs {
rhs = *inner;
}
if rhs.used_as(&ExpryType::String) {
return Ok(*fields_type);
}
return Err(TypeError::InconsistentTypes(write!(allocator, "field access with a non-string: {}", rhs)));
} else if let ExpryType::Any = lhs {
let mut rhs = get_type_from_bytecode(expression, types, context, allocator)?;
if let ExpryType::Nullable(inner) = rhs {
rhs = *inner;
}
if rhs.used_as(&ExpryType::String) {
return Ok(ExpryType::Any);
}
return Err(TypeError::InconsistentTypes(write!(allocator, "field access with a non-string: {}", rhs)));
} else {
return Err(TypeError::InconsistentTypes(write!(allocator, "field access on a non-object: {}", lhs)));
}
}
if opcode == ExpryBytecode::ArrayAccessDecoded as u8 ||
opcode == ExpryBytecode::ArrayAccessBinary as u8 {
let lhs = get_type_from_bytecode(expression, types, context, allocator)?;
let rhs = get_type_from_bytecode(expression, types, context, allocator)?;
if !rhs.used_as(&ExpryType::Int) {
return Err(TypeError::InconsistentTypes("array subscript should be an int"));
}
return match lhs {
ExpryType::Any => {
Ok(ExpryType::Any)
},
ExpryType::Array(x) => Ok(*x),
ExpryType::EmptyArray => Err(TypeError::InconsistentTypes("array subscript on empty array")),
_ => Err(TypeError::InconsistentTypes("array access on a non-object or non-uniform array")),
};
}
if opcode == ExpryBytecode::Concrete as u8 {
let value = DecodedValue::decode(expression).map_err(|_| {
TypeError::Bytecode
})?;
return Ok(expry_to_type(&value));
}
#[cfg(feature = "power")]
if opcode == ExpryBytecode::ArithmeticPower as u8 {
let lhs = get_type_from_bytecode(expression, types, context, allocator)?;
if !(lhs.used_as(&ExpryType::Int) || lhs.used_as(&ExpryType::Float) || lhs.used_as(&ExpryType::Double)) {
return Err(TypeError::InconsistentTypes("value for + should be a number"));
}
let rhs = get_type_from_bytecode(expression, types, context, allocator)?;
if !(rhs.used_as(&ExpryType::Int) || rhs.used_as(&ExpryType::Float) || rhs.used_as(&ExpryType::Double)) {
return Err(TypeError::InconsistentTypes("value for + should be a number"));
}
return Ok(ExpryType::Double);
}
if opcode == ExpryBytecode::ArithmeticPlus as u8 ||
opcode == ExpryBytecode::ArithmeticMin as u8 ||
opcode == ExpryBytecode::ArithmeticMul as u8 ||
opcode == ExpryBytecode::ArithmeticDiv as u8 ||
opcode == ExpryBytecode::ArithmeticMod as u8 ||
opcode == ExpryBytecode::ArithmeticShiftLeft as u8 ||
opcode == ExpryBytecode::ArithmeticShiftRight as u8 ||
opcode == ExpryBytecode::ArithmeticBitwiseOr as u8 ||
opcode == ExpryBytecode::ArithmeticBitwiseXor as u8 ||
opcode == ExpryBytecode::ArithmeticBitwiseAnd as u8 {
let lhs = get_type_from_bytecode(expression, types, context, allocator)?;
if !(lhs.used_as(&ExpryType::Int) || lhs.used_as(&ExpryType::Float) || lhs.used_as(&ExpryType::Double)) {
return Err(TypeError::InconsistentTypes("value for arithmetic operation should be a number"));
}
let rhs = get_type_from_bytecode(expression, types, context, allocator)?;
if !(rhs.used_as(&ExpryType::Int) || rhs.used_as(&ExpryType::Float) || rhs.used_as(&ExpryType::Double)) {
return Err(TypeError::InconsistentTypes("value for arithmetic operation should be a number"));
}
if lhs.used_as(&rhs) && rhs.used_as(&lhs) {
return Ok(lhs);
}
return Err(TypeError::InconsistentTypes(write!(allocator, "types for arithmetic operation should be the same: {} vs {}", lhs, rhs)));
}
if opcode == ExpryBytecode::ArithmeticShiftLeft as u8 ||
opcode == ExpryBytecode::ArithmeticShiftRight as u8 ||
opcode == ExpryBytecode::ArithmeticBitwiseOr as u8 ||
opcode == ExpryBytecode::ArithmeticBitwiseXor as u8 ||
opcode == ExpryBytecode::ArithmeticBitwiseAnd as u8 {
let lhs = get_type_from_bytecode(expression, types, context, allocator)?;
if !(lhs.used_as(&ExpryType::Int)) {
return Err(TypeError::InconsistentTypes("value for bitwise arithmetic operation should be an integer"));
}
let rhs = get_type_from_bytecode(expression, types, context, allocator)?;
if !(rhs.used_as(&ExpryType::Int)) {
return Err(TypeError::InconsistentTypes("value for bitwise arithmetic operation should be an integer"));
}
if lhs.used_as(&rhs) && rhs.used_as(&lhs) {
return Ok(lhs);
}
return Err(TypeError::InconsistentTypes("types for bitwise arithmetic operation should be the same"));
}
//////////
if opcode == ExpryBytecode::Not as u8 {
let lhs = get_type_from_bytecode(expression, types, context, allocator)?;
if lhs.used_as(&ExpryType::Bool) {
return Ok(ExpryType::Bool);
} else {
return Err(TypeError::InconsistentTypes(write!(allocator, "not (!) expects a boolean: {}", lhs)));
}
}
if opcode == ExpryBytecode::ParseEqual as u8 ||
opcode == ExpryBytecode::ParseNotEqual as u8 ||
opcode == ExpryBytecode::Equal as u8 ||
opcode == ExpryBytecode::NotEqual as u8
{
let lhs = get_type_from_bytecode(expression, types, context, allocator)?;
let rhs = get_type_from_bytecode(expression, types, context, allocator)?;
// even if object might not be a subtype of each other, the can consist of the same values:
// {a: ..} can at runtime also contain {a: .., b: ..}, so might compare with the likewise
// rhs with type {b: ..} but runtime {a: ..., b: ..}.
// FIXME: handle Nullable(Object) same as two objects (or neither)
if matches!(lhs, ExpryType::Object(_)) || matches!(rhs, ExpryType::Object(_)) || lhs.used_as(&rhs) || rhs.used_as(&lhs) {
return Ok(ExpryType::Bool);
} else {
return Err(TypeError::InconsistentTypes(write!(allocator, "comparison between different types is always false: {} and {}", lhs, rhs)));
}
}
if opcode == ExpryBytecode::NumericLess as u8 ||
opcode == ExpryBytecode::NumericLessEqual as u8 ||
opcode == ExpryBytecode::NumericGreater as u8 ||
opcode == ExpryBytecode::NumericGreaterEqual as u8
{
let lhs = get_type_from_bytecode(expression, types, context, allocator)?;
if !(lhs.used_as(&ExpryType::Int) || lhs.used_as(&ExpryType::Float) || lhs.used_as(&ExpryType::Double)) {
return Err(TypeError::InconsistentTypes("value for numeric comparison should be a number"));
}
let rhs = get_type_from_bytecode(expression, types, context, allocator)?;
if !(rhs.used_as(&ExpryType::Int) || rhs.used_as(&ExpryType::Float) || rhs.used_as(&ExpryType::Double)) {
return Err(TypeError::InconsistentTypes("value for numeric comparison should be a number"));
}
if lhs.used_as(&rhs) && rhs.used_as(&lhs) {
return Ok(ExpryType::Bool);
}
return Err(TypeError::InconsistentTypes("types for numeric comparison should be the same"));
}
if opcode == ExpryBytecode::StringContains as u8 ||
opcode == ExpryBytecode::StringStartsWith as u8 ||
opcode == ExpryBytecode::StringEndsWith as u8
{
let lhs = get_type_from_bytecode(expression, types, context, allocator)?;
let rhs = get_type_from_bytecode(expression, types, context, allocator)?;
if lhs.used_as(&ExpryType::String) && rhs.used_as(&ExpryType::String) {
return Ok(ExpryType::Bool);
}
return Err(TypeError::InconsistentTypes("string contains, starts-with, ends-with expects string arguments"));
}
if opcode == ExpryBytecode::StringConcat as u8 {
let lhs = get_type_from_bytecode(expression, types, context, allocator)?;
let rhs = get_type_from_bytecode(expression, types, context, allocator)?;
if lhs.used_as(&ExpryType::String) && rhs.used_as(&ExpryType::String) {
return Ok(ExpryType::String);
}
return Err(TypeError::InconsistentTypes(write!(allocator, "string concat (..) expects string arguments, not {} and {}", lhs, rhs)));
}
if opcode == ExpryBytecode::Length as u8 {
let lhs = get_type_from_bytecode(expression, types, context, allocator)?;
if lhs.used_as(&ExpryType::String) || matches!(lhs, ExpryType::Any | ExpryType::EmptyArray | ExpryType::Array(_) | ExpryType::StaticArray(_) | ExpryType::UniformObject(_) | ExpryType::Object(_)) {
return Ok(ExpryType::Int);
}
return Err(TypeError::InconsistentTypes(write!(allocator, "length() can only be taken of a string, array, or object: not from {}", lhs)));
}
if opcode == ExpryBytecode::ArrayReverse as u8 {
let lhs = get_type_from_bytecode(expression, types, context, allocator)?;
if matches!(lhs, ExpryType::Any | ExpryType::EmptyArray | ExpryType::Array(_)) {
return Ok(lhs);
}
if let ExpryType::StaticArray(mut arr) = lhs {
arr.reverse();
return Ok(ExpryType::StaticArray(arr));
}
return Err(TypeError::InconsistentTypes(write!(allocator, "reverse() can only be applied to an array: not {}", lhs)));
}
/*
if opcode == ExpryBytecode::StringSplit as u8 {
let string = evaluate_to_decoded_value(expression, value, context, allocator)?;
let max = evaluate_to_decoded_value(expression, value, context, allocator)?;
let split_on = evaluate_to_decoded_value(expression, value, context, allocator)?;
if let (DecodedValue::String(mut string), DecodedValue::Int(mut max), DecodedValue::String(split_on)) = (string, max, split_on) {
let mut retval = DecodedArray::new();
if max > 1 {
max -= 1;
while let Some(pos) = find_subsequence(string, split_on) {
let (begin, end) = string.split_at(pos);
retval.push(DecodedValue::String(begin));
string = &end[split_on.len()..];
if retval.len() >= max as usize {
break;
}
}
}
if !string.is_empty() {
retval.push(DecodedValue::String(string));
}
return Ok(DecodedValue::Array(retval));
} else {
return Err(EvalError::Dynamic("splitn() method expects to be called on a string, with a int and string argument"));
}
}
if opcode == ExpryBytecode::StringSubstringFromTo as u8 ||
opcode == ExpryBytecode::StringSubstringFrom as u8 {
let string = evaluate_to_decoded_value(expression, value, context, allocator)?;
let start = evaluate_to_decoded_value(expression, value, context, allocator)?;
let length = if opcode == ExpryBytecode::StringSubstringFromTo as u8 {
evaluate_to_decoded_value(expression, value, context, allocator)?
} else {
DecodedValue::Int(i64::MAX)
};
if let DecodedValue::String(string) = string {
if let DecodedValue::Int(start) = start {
if let DecodedValue::Int(length) = length {
// u32 for 32-bits targets
let start = start.clamp(0, u32::MAX as i64) as usize;
let length = length.clamp(0, u32::MAX as i64) as usize;
let from = core::cmp::min(start, string.len());
let to = core::cmp::min((start).saturating_add(length), string.len());
return Ok(DecodedValue::String(&string[from..to]));
} else {
return Err(EvalError::Dynamic("substring needs an int as third argument"));
}
} else {
return Err(EvalError::Dynamic("substring needs an int as second argument"));
}
} else {
return Err(EvalError::Dynamic("substring from string needs a string as first argument"));
}
}
*/
if opcode == ExpryBytecode::StringUpper as u8 ||
opcode == ExpryBytecode::StringLower as u8 ||
opcode == ExpryBytecode::StringTrim as u8 ||
opcode == ExpryBytecode::StringHex as u8 ||
opcode == ExpryBytecode::StringHtmlescape as u8 ||
opcode == ExpryBytecode::StringUrlescape as u8 ||
opcode == ExpryBytecode::StringBasename as u8 ||
opcode == ExpryBytecode::StringDirname as u8
{
let arg_type = get_type_from_bytecode(expression, types, context, allocator)?;
if !matches!(arg_type, ExpryType::String | ExpryType::Any) {
return Err(TypeError::InconsistentTypes("string functions expects a string as argument"));
}
return Ok(ExpryType::String);
}
if opcode == ExpryBytecode::ToString as u8 ||
opcode == ExpryBytecode::ToInteger as u8 ||
opcode == ExpryBytecode::ToFloat as u8 ||
opcode == ExpryBytecode::ToDouble as u8
{
let lhs = get_type_from_bytecode(expression, types, context, allocator)?;
if opcode == ExpryBytecode::ToString as u8 {
return Ok(ExpryType::String);
}
if opcode == ExpryBytecode::ToInteger as u8 {
if matches!(lhs, ExpryType::Any | ExpryType::Int | ExpryType::Float | ExpryType::Double | ExpryType::String) {
return Ok(ExpryType::Int);
} else {
return Err(TypeError::InconsistentTypes(write!(allocator, "toint() expects a int, float, double, or string, not a {}", lhs)));
}
}
if opcode == ExpryBytecode::ToFloat as u8 {
if matches!(lhs, ExpryType::Any | ExpryType::Int | ExpryType::Float | ExpryType::Double | ExpryType::String) {
return Ok(ExpryType::Float);
} else {
return Err(TypeError::InconsistentTypes(write!(allocator, "tofloat() expects a int, float, double, or string, not a {}", lhs)));
}
}
if opcode == ExpryBytecode::ToDouble as u8 {
if matches!(lhs, ExpryType::Any | ExpryType::Int | ExpryType::Float | ExpryType::Double | ExpryType::String) {
return Ok(ExpryType::Double);
} else {
return Err(TypeError::InconsistentTypes(write!(allocator, "todouble() expects a int, float, double, or string, not a {}", lhs)));
}
}
unimplemented!();
}
if opcode == ExpryBytecode::ArithmeticUnaryMin as u8 {
let lhs = get_type_from_bytecode(expression, types, context, allocator)?;
if matches!(lhs, ExpryType::Any | ExpryType::Int | ExpryType::Float | ExpryType::Double) {
return Ok(lhs);
} else {
return Err(TypeError::InconsistentTypes(write!(allocator, "-x expects x to be a number (int, float double), not a {}", lhs)));
}
}
/*
// FIXME: is_type etc
// FIXME: math functions: ceil, floor, trunc, round, sqrt, abs
if opcode == ExpryBytecode::Defined as u8 ||
opcode == ExpryBytecode::Undefined as u8
{
let mut lhs_reader = RawReader::with(expression.read_var_string().map_err(|_| EvalError::Expression("error in (UN)DEFINED"))?);
let result = evaluate_to_decoded_value(&mut lhs_reader, value, context, allocator);
if opcode == ExpryBytecode::Defined as u8 {
return Ok(DecodedValue::Bool(result.is_ok()));
}
return Ok(DecodedValue::Bool(result.is_err()));
}
*/
if opcode == ExpryBytecode::Try as u8 {
let warnings = context.warnings.len();
let mut lhs_reader = RawReader::with(expression.read_var_string().map_err(|_| TypeError::Bytecode)?);
let lhs = get_type_from_bytecode(&mut lhs_reader, types, context, allocator)?;
// ignore warnings emitted in lhs
context.warnings.truncate(warnings);
let mut rhs_reader = RawReader::with(expression.read_var_string().map_err(|_| TypeError::Bytecode)?);
let rhs = get_type_from_bytecode(&mut rhs_reader, types, context, allocator)?;
if matches!(rhs, ExpryType::Null) {
if matches!(lhs, ExpryType::Nullable(_) | ExpryType::Any) {
return Ok(lhs);
} else {
return Ok(ExpryType::Nullable(Box::new(lhs)));
}
}
if lhs.used_as(&rhs) {
return Ok(rhs);
}
if rhs.used_as(&lhs) {
return Ok(lhs);
}
return Err(TypeError::InconsistentTypes("try (??? operator) needs similar types on both sides"));
}
if opcode == ExpryBytecode::NotNullElse as u8 {
let mut lhs_reader = RawReader::with(expression.read_var_string().map_err(|_| TypeError::Bytecode)?);
let lhs = get_type_from_bytecode(&mut lhs_reader, types, context, allocator)?;
let mut rhs_reader = RawReader::with(expression.read_var_string().map_err(|_| TypeError::Bytecode)?);
let rhs = get_type_from_bytecode(&mut rhs_reader, types, context, allocator)?;
if let ExpryType::Null = lhs {
return Ok(rhs);
}
if let ExpryType::Nullable(lhs) = lhs {
if lhs.used_as(&rhs) {
return Ok(rhs);
}
if rhs.used_as(&lhs) {
return Ok(*lhs);
}
return Err(TypeError::InconsistentTypes(write!(allocator, "notnullelse (?? operator) needs similar types on both sides: {} and {}", *lhs, rhs)));
}
if lhs.used_as(&rhs) {
return Ok(rhs);
}
if rhs.used_as(&lhs) {
return Ok(lhs);
}
return Err(TypeError::InconsistentTypes("notnullelse (?? operator) needs similar types on both sides"));
}
/*
if opcode == ExpryBytecode::SliceObjectDynamic as u8 {
let count = expression.read_var_u64().map_err(|_| EvalError::Expression("error in SLICEOBJECT"))?;
let mut retval = DecodedObject::new();
for _ in 0..count {
match evaluate_to_decoded_value(expression, value, context, allocator)? {
DecodedValue::String(key) => {
let hash = key_stable_hash(key);
retval.insert((hash, key), evaluate_to_decoded_value(expression, value, context, allocator)?);
},
_ => {
return Err(EvalError::Dynamic("expected string as key for a SLICEOBJECT"));
}
}
}
return Ok(DecodedValue::Object(retval));
}
*/
if opcode == ExpryBytecode::SliceObjectStatic as u8 {
let count = expression.read_var_u64().map_err(|_| TypeError::Bytecode)?;
//eprintln!("slicing object of {} fields", count);
let mut retval = TypeObject::new();
for _i in 0..count {
let _hash_of_key = expression.read_u8().map_err(|_| TypeError::Bytecode)?;
//eprintln!("{}: hash {}", i, hash_of_key);
let key = expression.read_var_string().map_err(|_| TypeError::Bytecode)?;
let value = get_type_from_bytecode(expression, types, context, allocator)?;
//eprintln!("{}: {}={}", i, String::from_utf8_lossy(key), value);
retval.insert(key.to_vec(), (true, value));
}
return Ok(ExpryType::Object(retval));
}
if opcode == ExpryBytecode::MergeObjects as u8 {
let lhs = get_type_from_bytecode(expression, types, context, allocator)?;
let rhs = get_type_from_bytecode(expression, types, context, allocator)?;
if let (ExpryType::Object(mut lhs), ExpryType::Object(rhs)) = (lhs.clone(), rhs.clone()) {
for (k,(mut required,mut v)) in rhs {
if !required {
// if this item is not required, we should combine the type with the other type
// (from lhs), and generalize it. If the other type exists
if let Some((lhs_required, lhs_v)) = lhs.get(&k) {
required = *lhs_required;
if v.used_as(lhs_v) {
v = lhs_v.clone();
}
}
}
let v = core::mem::take(&mut v);
lhs.insert(k, (required,v));
}
return Ok(ExpryType::Object(lhs));
} else {
return Err(TypeError::InconsistentTypes(write!(allocator, "merging object (with e.g. spread syntax) expects two objects as arguments, not {} and {}", lhs, rhs)));
}
}
if opcode == ExpryBytecode::SliceArray as u8 {
let count = expression.read_var_u64().map_err(|_| TypeError::Bytecode)?;
if count == 0 {
return Ok(ExpryType::EmptyArray);
}
let mut retval = Vec::new();
for _ in 0..count {
let v = get_type_from_bytecode(expression, types, context, allocator)?;
retval.push(v);
}
return Ok(ExpryType::StaticArray(retval));
}
if opcode == ExpryBytecode::Conditional as u8 {
let mut c = RawReader::with(expression.read_var_string().map_err(|_| TypeError::Bytecode)?);
let c = get_type_from_bytecode(&mut c, types, context, allocator)?;
let mut t = RawReader::with(expression.read_var_string().map_err(|_| TypeError::Bytecode)?);
let t = get_type_from_bytecode(&mut t, types, context, allocator)?;
let mut e = RawReader::with(expression.read_var_string().map_err(|_| TypeError::Bytecode)?);
let e = get_type_from_bytecode(&mut e, types, context, allocator)?;
if !c.used_as(&ExpryType::Bool) {
return Err(TypeError::InconsistentTypes("condition for `a?b:c` needs a boolean"));
}
// FIXME: should implement proper combine_type, that for two objects combines all the fields (see MergeObjects a few lines above)
// avoid that something? and null is reduced to null
if t != ExpryType::Null && e.used_as(&t) {
return Ok(t);
}
if t.used_as(&e) {
return Ok(e);
}
return Err(TypeError::InconsistentTypes("both branches for `a?b:c` needs similar types"));
}
if opcode == ExpryBytecode::LambdaFilter as u8 ||
opcode == ExpryBytecode::LambdaSortByKey as u8 {
let retval = get_type_from_bytecode(expression, types, context, allocator)?;
if Ok(ExpryBytecode::Lambda as u8) != expression.read_u8() {
return Err(TypeError::Bytecode);
}
if Ok(types.len() as u8) != expression.read_u8() {
return Err(TypeError::Bytecode);
}
if Ok(1u8) != expression.read_u8() {
return Err(TypeError::Bytecode);
}
expression.read_var_string().map_err(|_| TypeError::Bytecode)?;
return Ok(retval);
}
if opcode == ExpryBytecode::LambdaObjectToArray as u8 {
let lhs = get_type_from_bytecode(expression, types, context, allocator)?;
if Ok(ExpryBytecode::Lambda as u8) != expression.read_u8() {
return Err(TypeError::Bytecode);
}
if Ok(types.len() as u8) != expression.read_u8() {
return Err(TypeError::Bytecode);
}
if Ok(2u8) != expression.read_u8() {
return Err(TypeError::Bytecode);
}
let mut lambda_reader = RawReader::with(expression.read_var_string().map_err(|_| TypeError::Bytecode)?);
if let ExpryType::UniformObject(value_type) = lhs {
let mut types = types.to_vec();
types.push(&ExpryType::String);
types.push(&value_type);
let retval = get_type_from_bytecode(&mut lambda_reader, &types, context, allocator);
types.pop();
types.pop();
return Ok(ExpryType::Array(Box::new(retval?)));
} else if matches!(lhs, ExpryType::Any) {
let mut types = types.to_vec();
types.push(&ExpryType::String);
types.push(&ExpryType::Any);
let retval = get_type_from_bytecode(&mut lambda_reader, &types, context, allocator);
types.pop();
types.pop();
return Ok(ExpryType::Array(Box::new(retval?)));
} else {
return Err(TypeError::Untypable(write!(allocator, "error in lambda object operation, only uniformed typed objects are supported for object.to_array(..), not {}", lhs)));
}
}
if opcode == ExpryBytecode::LambdaArrayMap as u8 {
let lhs = get_type_from_bytecode(expression, types, context, allocator)?;
if Ok(ExpryBytecode::Lambda as u8) != expression.read_u8() {
return Err(TypeError::Bytecode);
}
if Ok(types.len() as u8) != expression.read_u8() {
return Err(TypeError::Bytecode);
}
if Ok(1u8) != expression.read_u8() {
return Err(TypeError::Bytecode);
}
let mut lambda_reader = RawReader::with(expression.read_var_string().map_err(|_| TypeError::Bytecode)?);
if matches!(lhs, ExpryType::EmptyArray | ExpryType::Any) {
let mut types = types.to_vec();
types.push(&ExpryType::Any);
let retval = get_type_from_bytecode(&mut lambda_reader, &types, context, allocator);
types.pop();
return Ok(ExpryType::Array(Box::new(retval?)));
} else if let ExpryType::Array(value_type) = lhs {
let mut types = types.to_vec();
types.push(&value_type);
let retval = get_type_from_bytecode(&mut lambda_reader, &types, context, allocator);
types.pop();
return Ok(ExpryType::Array(Box::new(retval?)));
} else {
return Err(TypeError::Untypable(write!(allocator, "error in lambda array map() operation, only array types supported, not {}", lhs)));
}
}
if opcode == ExpryBytecode::LambdaArrayToMap as u8 {
let lhs = get_type_from_bytecode(expression, types, context, allocator)?;
if Ok(ExpryBytecode::Lambda as u8) != expression.read_u8() {
return Err(TypeError::Bytecode);
}
if Ok(types.len() as u8) != expression.read_u8() {
return Err(TypeError::Bytecode);
}
if Ok(1u8) != expression.read_u8() {
return Err(TypeError::Bytecode);
}
let mut lambda_reader = RawReader::with(expression.read_var_string().map_err(|_| TypeError::Bytecode)?);
if let ExpryType::Array(value_type) = lhs {
let mut types = types.to_vec();
types.push(&value_type);
let retval = get_type_from_bytecode(&mut lambda_reader, &types, context, allocator);
types.pop();
let mut retval = retval?;
if let ExpryType::StaticArray(ref mut arr) = retval {
if let [ExpryType::String,ExpryType::String,value] = &mut arr[0..3] {
let value = core::mem::take(value);
return Ok(ExpryType::UniformObject(Box::new(ExpryType::Array(Box::new(value)))));
}
}
return Err(TypeError::InconsistentTypes(write!(allocator, "error in lambda for array.to_map(..): should return [string,string,*], not {}", retval)));
} else {
return Err(TypeError::Untypable(write!(allocator, "error in lambda object operation, only arrays are supported for array.to_map(..), not {}", lhs)));
}
}
if opcode == ExpryBytecode::LambdaArrayToObject as u8 {
let lhs = get_type_from_bytecode(expression, types, context, allocator)?;
if Ok(ExpryBytecode::Lambda as u8) != expression.read_u8() {
return Err(TypeError::Bytecode);
}
if Ok(types.len() as u8) != expression.read_u8() {
return Err(TypeError::Bytecode);
}
if Ok(1u8) != expression.read_u8() {
return Err(TypeError::Bytecode);
}
let mut lambda_reader = RawReader::with(expression.read_var_string().map_err(|_| TypeError::Bytecode)?);
if let ExpryType::Array(value_type) = lhs {
let mut types = types.to_vec();
types.push(&value_type);
let retval = get_type_from_bytecode(&mut lambda_reader, &types, context, allocator);
types.pop();
let mut retval = retval?;
if let ExpryType::StaticArray(ref mut arr) = retval {
if let [ExpryType::String,value] = &mut arr[0..2] {
let value = core::mem::take(value);
return Ok(ExpryType::UniformObject(Box::new(value)));
}
}
return Err(TypeError::InconsistentTypes(write!(allocator, "error in lambda for array.to_object(..): should return [string,*], not {}", retval)));
} else {
return Err(TypeError::Untypable(write!(allocator, "error in lambda object operation, only arrays are supported for array.to_map(..), not {}", lhs)));
}
}
/*
// catch for all the opcodes that should be evaluated in evaluate_to_binary as that
// implementation is straightforward and mostly used
opcode == ExpryBytecode::LambdaSortByKey as u8 ||
opcode == ExpryBytecode::LambdaArrayToObject as u8 ||
opcode == ExpryBytecode::LambdaArrayToMap as u8 ||
opcode == ExpryBytecode::LambdaObjectToArray as u8 ||
opcode == ExpryBytecode::FieldAccessBinary as u8 ||
opcode == ExpryBytecode::ArrayAccessBinary as u8 {
*expression = original;
let mut output = Vec::new();
evaluate_to_binary(expression, value, context, allocator, &mut output)?;
let output : &[u8] = concat_output(&output, allocator);
return expry_decode(output).map_err(|_| EvalError::Dynamic("after evaluate_to_binary could not decode value"));
}
*/
Err(TypeError::UnsupportedOpcode(opcode))
}
pub fn expry_to_type(value: &DecodedValue<'_>) -> ExpryType {
match value {
DecodedValue::Null => ExpryType::Null,
DecodedValue::Bool(_) => ExpryType::Bool,
DecodedValue::Int(_) => ExpryType::Int,
DecodedValue::Float(_) => ExpryType::Float,
DecodedValue::Double(_) => ExpryType::Double,
DecodedValue::String(_) => ExpryType::String,
DecodedValue::Object(obj) => {
let mut retval = TypeObject::new();
for (k,v) in obj {
retval.insert(k.1.to_vec(), (true, expry_to_type(v)));
}
ExpryType::Object(retval)
},
DecodedValue::Array(arr) => {
if arr.is_empty() {
return ExpryType::EmptyArray;
}
let t = expry_to_type(&arr[0]);
for (i,v) in arr.iter().enumerate().skip(1) {
let this_type = expry_to_type(v);
if t != this_type {
// switch to non-uniform array typing
let mut types = Vec::new();
types.resize(i-1, t);
types.push(this_type);
for v in &arr[i+1..] {
types.push(expry_to_type(v));
}
return ExpryType::StaticArray(types);
}
}
ExpryType::Array(Box::new(t))
},
}
}
pub fn expry_to_type_lazy<'a,'b>(value: &LazyDecodedValue<'a>) -> Result<ExpryType,EncodingError> where 'a: 'b {
match value {
LazyDecodedValue::Null => Ok(ExpryType::Null),
LazyDecodedValue::Bool(_) => Ok(ExpryType::Bool),
LazyDecodedValue::Int(_) => Ok(ExpryType::Int),
LazyDecodedValue::Float(_) => Ok(ExpryType::Float),
LazyDecodedValue::Double(_) => Ok(ExpryType::Double),
LazyDecodedValue::String(_) => Ok(ExpryType::String),
LazyDecodedValue::Object(mut obj) => {
let mut retval = TypeObject::new();
while let Ok((k,v)) = obj.get() {
retval.insert(k.1.to_vec(), (true, expry_to_type_lazy(&v)?));
}
Ok(ExpryType::Object(retval))
},
LazyDecodedValue::Array(mut arr) => {
if arr.is_empty() {
return Ok(ExpryType::EmptyArray);
}
let t = expry_to_type_lazy(&arr.get()?)?;
while let Ok(v) = arr.get() {
if t != expry_to_type_lazy(&v)? {
return Err(EncodingError{ line_nr: line!() });
}
}
Ok(ExpryType::Array(Box::new(t)))
},
}
}
#[derive(Debug)]
pub struct ExpryUntilNewLine {
prev_was_escape: bool,
string: bool,
nested: u32,
len: usize,
}
impl Spanner for ExpryUntilNewLine {
fn next(&mut self, b: char) -> bool {
if self.nested == 0 && !self.string && b == '\n' {
return false;
}
if !self.prev_was_escape && !self.string && (b == '{' || b == '[') {
self.nested += 1;
}
if !self.prev_was_escape && !self.string && (b == '}' || b == ']') {
if self.nested == 0 {
return false;
}
self.nested -= 1;
}
if !self.prev_was_escape && b == '"' {
self.string = !self.string;
}
self.prev_was_escape = !self.prev_was_escape && b == '\\';
self.len += 1;
true
}
fn valid(&mut self, len: usize) -> bool {
len > 0 && self.nested == 0 && !self.string && !self.prev_was_escape
}
}
impl ExpryUntilNewLine {
pub fn new() -> Self {
Self {
prev_was_escape: false,
string: false,
nested: 0,
len: 0,
}
}
}
impl Default for ExpryUntilNewLine {
fn default() -> Self {
Self::new()
}
}
#[cfg(test)]
extern crate quickcheck;
#[cfg(test)]
#[macro_use(quickcheck)]
extern crate quickcheck_macros;
#[cfg(test)]
mod expry {
use core::time;
use std::{time::{Instant, Duration}, fs::File, io::{BufReader, Read}};
use crate::*;
fn convert_test(v: &DecodedValue) {
println!("testing {:?}", v);
let len = v.size_of_binary();
let mut buffer = vec![0u8; len];
let mut writer = RawWriter::with(&mut buffer[..]);
v.print_binary(&mut writer).unwrap();
assert_eq!(writer.left(), 0);
let mut reader = RawReader::with(&buffer[..]);
let v2 = DecodedValue::decode(&mut reader).expect("error");
assert_eq!(reader.len(), 0);
assert_eq!(*v, v2);
}
#[quickcheck]
fn quickcheck_binary_int(v: i64) -> bool {
let v = DecodedValue::Int(v);
println!("testing {:?}", v);
let len = v.size_of_binary();
let mut buffer = vec![0u8; len];
let mut writer = RawWriter::with(&mut buffer[..]);
v.print_binary(&mut writer).unwrap();
let writer_left = writer.left();
if buffer.len() - writer_left != len {
println!("wrong expected size of binary representation");
return false;
}
println!("binary representation: {:?}", buffer);
let mut reader = RawReader::with(&buffer[..]);
let v2 = DecodedValue::decode(&mut reader).expect("error");
println!("left reader: {}; left writer: {}", reader.len(), writer_left);
reader.len() == writer_left && v == v2
}
#[test]
fn basic() {
let mut foo = DecodedValue::Null;
assert_eq!(foo, foo);
println!("testing null");
foo = DecodedValue::Null;
assert_eq!(1, foo.size_of_binary());
convert_test(&foo);
println!("testing bool");
foo = DecodedValue::Bool(false);
assert_eq!(1, foo.size_of_binary());
convert_test(&foo);
foo = DecodedValue::Bool(true);
assert_eq!(1, foo.size_of_binary());
convert_test(&foo);
println!("testing float");
foo = DecodedValue::Float(0.0);
assert_eq!(5, foo.size_of_binary());
convert_test(&foo);
println!("testing double");
foo = DecodedValue::Double(0.0);
assert_eq!(9, foo.size_of_binary());
convert_test(&foo);
println!("testing string");
foo = DecodedValue::String("foobar".as_bytes());
assert_eq!(7, foo.size_of_binary());
convert_test(&foo);
println!("foo string = {:?}", foo);
println!("testing int");
foo = DecodedValue::Int(0);
assert_eq!(1, foo.size_of_binary());
convert_test(&foo);
foo = DecodedValue::Int(1);
assert_eq!(2, foo.size_of_binary());
convert_test(&foo);
foo = DecodedValue::Int(2);
assert_eq!(2, foo.size_of_binary());
convert_test(&foo);
foo = DecodedValue::Int(-1);
assert_eq!(2, foo.size_of_binary());
convert_test(&foo);
println!("testing array");
foo = DecodedValue::Array(vec![]);
assert_eq!(1, foo.size_of_binary());
convert_test(&foo);
let arr = vec![DecodedValue::Null];
foo = DecodedValue::Array(arr);
assert_eq!(3, foo.size_of_binary());
convert_test(&foo);
println!("testing object");
foo = value!({
"null": null,
"one": 1,
"foo": {
"null": null,
"one": 1,
},
});
assert_eq!(36, foo.size_of_binary());
convert_test(&foo);
}
pub fn throughput(dur: time::Duration, bytes: usize) -> u64 {
let mut megabytes_per_second = bytes as u64 / dur.as_micros() as u64;
// Round to two significant digits.
if megabytes_per_second > 100 {
if megabytes_per_second % 10 >= 5 {
megabytes_per_second += 10;
}
megabytes_per_second = megabytes_per_second / 10 * 10;
}
megabytes_per_second
}
#[test] #[ignore]
fn twitter_binary_parse_fat() {
// let _guard = sentry::init(("https://examplePublicKey@o0.ingest.sentry.io/0", sentry::ClientOptions {
// release: sentry::release_name!(),
// ..Default::default()
// }));
// panic!("Everything is on fire!");
let mut allocator = MemoryPool::new();
let mut scope = allocator.rewind();
let v = get_data(&mut scope);
let bytes = v.encode_to_vec();
let before = Instant::now();
let count = 256;
for _ in 0..count {
let mut reader = RawReader::with(bytes.get());
let _ = DecodedValue::decode(&mut reader).expect("error");
assert_eq!(0, reader.len());
}
let after = Instant::now();
let dur = after - before;
//println!("{}x in {} ms -> {} MB/s", count, dur.as_millis(), throughput(dur, count*bytes.len()));
println!("{}x in {} ms -> {} MB/s", count, dur.as_millis(), ((count*bytes.len()) as u128)/dur.as_micros());
let mut reader = RawReader::with(bytes.get());
let v = DecodedValue::decode(&mut reader).expect("error");
let mut buffer = String::new();
v.print_json(&mut buffer).unwrap();
// let mut f = File::create("twitter.json").unwrap();
// f.write_all(&buffer).unwrap();
// drop(f);
/*
let (v, pos) = parse(&bytes[..]).expect("error");
assert_eq!(pos, bytes.len());
let len = size_of_binary(&v, false);
let mut writer = vec![0u8; len];
let used_len = print(&v, &mut writer[..], false);
assert_eq!(used_len, len);
let (v2, pos) = parse(&writer[..]).expect("error");
assert_eq!(pos, writer.len());
assert_eq!(v, v2);
// println!("{:?}", v);
*/
assert!(false);
}
#[test] #[ignore]
fn twitter_binary_parse_lazy() {
let mut allocator = MemoryPool::new();
let mut scope = allocator.rewind();
let v = get_data(&mut scope);
let bytes = v.encode_to_vec();
let before = Instant::now();
let count = 256;
for _ in 0..count {
let mut reader = RawReader::with(bytes.get());
let _ = walk_lazy_binary_object(LazyDecodedValue::decode(&mut reader).expect("error")).expect("error2");
assert_eq!(0, reader.len());
}
let after = Instant::now();
let dur = after - before;
//println!("{}x in {} ms -> {} MB/s", count, dur.as_millis(), throughput(dur, count*bytes.len()));
println!("{}x in {} ms -> {} MB/s", count, dur.as_millis(), ((count*bytes.len()) as u128)/dur.as_micros());
assert!(false);
}
fn walk_lazy_binary_object(value: LazyDecodedValue) -> Result<(),EncodingError> {
match value {
LazyDecodedValue::Null => Ok(()),
LazyDecodedValue::Bool(_) => Ok(()),
LazyDecodedValue::Int(_) => Ok(()),
LazyDecodedValue::Float(_) => Ok(()),
LazyDecodedValue::Double(_) => Ok(()),
LazyDecodedValue::String(_) => Ok(()),
LazyDecodedValue::Object(mut obj) => {
while !obj.is_empty()? {
let (_,v) = obj.get()?;
walk_lazy_binary_object(v)?;
}
if !obj.reader.is_empty() {
return Err(EncodingError{ line_nr: line!(), });
}
Ok(())
},
LazyDecodedValue::Array(mut arr) => {
while !arr.is_empty() {
let v = arr.get()?;
walk_lazy_binary_object(v)?;
}
if !arr.reader.is_empty() {
return Err(EncodingError{ line_nr: line!(), });
}
Ok(())
},
}
}
#[test] #[ignore]
fn twitter_binary_print() {
let mut allocator = MemoryPool::new();
let mut scope = allocator.rewind();
let v = get_data(&mut scope);
let len = v.size_of_binary();
let count = 256;
let before = Instant::now();
for _ in 0..count {
let len = v.size_of_binary();
let mut buffer = vec![0u8; len];
let mut writer = RawWriter::with(&mut buffer[..]);
v.print_binary(&mut writer).unwrap();
assert!(!buffer.is_empty());
}
let after = Instant::now();
let dur = after - before;
//println!("{}x in {} ms -> {} MB/s", count, dur.as_millis(), throughput(dur, count*bytes.len()));
println!("{}x in {} ms -> {} MB/s", count, dur.as_millis(), ((count*len) as u128)/dur.as_micros());
assert!(false);
}
#[test] #[ignore]
fn twitter_json_print() {
let mut allocator = MemoryPool::new();
let mut scope = allocator.rewind();
let v = get_data(&mut scope);
let count = 256;
let len = 466908;
let before = Instant::now();
for _ in 0..count {
let mut buffer = String::with_capacity(len);
v.print_json(&mut buffer).unwrap();
assert!(buffer.len() > 0);
}
let after = Instant::now();
let dur = after - before;
//println!("{}x in {} ms -> {} MB/s", count, dur.as_millis(), throughput(dur, count*bytes.len()));
println!("{}x in {} ms -> {} MB/s", count, dur.as_millis(), ((count*len) as u128)/dur.as_micros());
assert!(false);
}
fn get_data<'a,'c>(scope: &'a mut MemoryScope<'c>) -> DecodedValue<'a> where 'c: 'a {
let f = File::open("twitter.json").unwrap();
let mut reader = BufReader::new(f);
let mut bytes = String::new();
reader.read_to_string(&mut bytes).unwrap();
let bytes = scope.copy_str(&bytes);
println!("read {} bytes", bytes.len());
DecodedValue::parse_json(bytes, scope).expect("error")
}
#[test] #[ignore]
fn twitter_json_parse() {
let f = File::open("twitter.json").unwrap();
let mut reader = BufReader::new(f);
let mut bytes = String::new();
reader.read_to_string(&mut bytes).unwrap();
println!("read {} bytes", bytes.len());
let before = Instant::now();
let count = 4096;
let mut min_loop : Duration = Duration::MAX;
for _ in 0..count {
let before_loop_body = Instant::now();
let mut allocator = MemoryPool::new();
let mut scope = allocator.rewind();
let parsed = DecodedValue::parse_json(&bytes, &mut scope);
let after_loop_body = Instant::now();
assert!(parsed.is_ok());
let dur = after_loop_body - before_loop_body;
min_loop = Duration::min(min_loop, dur);
}
let after = Instant::now();
let dur = after - before;
println!("{}x in {} ms -> {} MB/s", count, dur.as_millis(), ((count*bytes.len()) as u128)/dur.as_micros());
println!("fastest loop body in {} ms -> {} MB/s", min_loop.as_millis(), ((bytes.len()) as u128)/min_loop.as_micros());
assert!(false);
}
#[test] #[ignore]
fn twitter_json_eval() {
let f = File::open("twitter.json").unwrap();
let mut reader = BufReader::new(f);
let mut bytes = String::new();
reader.read_to_string(&mut bytes).unwrap();
println!("read {} bytes", bytes.len());
let before = Instant::now();
let count = 256;
let mut min_loop : Duration = Duration::MAX;
for _ in 0..count {
let before_loop_body = Instant::now();
let mut allocator = MemoryPool::new();
let mut scope = allocator.rewind();
let mut parsed = parse_expry_expr(false, &bytes, &DecodedObject::new(), None, &["this"], &mut scope);
if let Ok(parsed) = &mut parsed {
#[cfg(not(feature = "mini"))]
expry_ast_fold(parsed, &mut scope).unwrap();
#[cfg(not(feature = "mini"))]
expry_ast_optimize(parsed).unwrap();
let bytecode = expry_ast_to_bytecode(&mut scope, parsed, 1).expect("bytecode generator error");
assert!(!bytecode.is_empty());
} else {
assert!(false);
}
let after_loop_body = Instant::now();
assert!(parsed.is_ok());
let dur = after_loop_body - before_loop_body;
min_loop = Duration::min(min_loop, dur);
}
let after = Instant::now();
let dur = after - before;
println!("{}x in {} ms -> {} MB/s", count, dur.as_millis(), ((count*bytes.len()) as u128)/dur.as_micros());
println!("fastest loop body in {} ms -> {} MB/s", min_loop.as_millis(), ((bytes.len()) as u128)/min_loop.as_micros());
assert!(false);
}
#[test]
fn insert_jumps() {
for size in 0..24u32 {
let mut jumps = 0;
for i in 0..size {
for j in 0..i.trailing_zeros() {
let to = i + (2 << j);
if to > size {
break;
}
jumps += 1;
println!("jump from {} to {}", i, to);
}
}
println!("{} -> {}", size, jumps);
assert!(size >= jumps);
}
//assert!(false);
}
#[test]
fn parsing_json() {
//let test = br#"[PI,123,456,789,1e9,0.001,-42,"foobar",{"a":1,"b":2}]"#;
let test = r#"{"a":1,"b":2}"#;
let mut allocator = MemoryPool::new();
let mut scope = allocator.rewind();
let object = DecodedValue::parse_json(test, &mut scope).unwrap();
println!("Parsed object: {:?}", object);
}
fn expression_tester<'a>(v: &EncodedValueVec, expr: &str, expected: Result<DecodedValue<'a>,EvalError<'a>>, line_nr: u32) {
println!("testing expr {} on line {}, expecting {:?}", expr, line_nr, expected);
let mut allocator = MemoryPool::new();
let mut scope = allocator.rewind();
let bytecode = expry_compile_expr(expr, None, Some(0), &["this"], &mut scope);
if let Ok((bytecode,_)) = bytecode {
let result = expry_eval(bytecode, &mut vec![v.to_ref()], &mut scope);
assert_eq!(expected, result);
} else if let Err(error) = bytecode {
println!("{}", expry_compile_error_format(&error, 0).1);
assert!(false);
}
}
fn expression_tester_parser_error<'a>(expr: &str, expected: CompileErrorDescription<'a>) {
println!("testing expr {}, expecting {:?}", expr, expected);
let mut allocator = MemoryPool::new();
let mut scope = allocator.rewind();
let bytecode = expry_compile_expr(expr, None, Some(0), &["this"], &mut scope);
if let Err(CompileError{error: got_error, ..}) = bytecode {
assert_eq!(expected, got_error);
} else {
println!("instead got: {:?}", bytecode);
assert!(false);
}
}
#[test]
fn basic_expressions() {
let value = value!({
"null": null,
"zero": 0,
"one": 1,
"two": 2,
"two_f32": 2.25f32,
"two_f64": 2.55555555,
"three": 3,
"sub": {
"subnull": null,
"subzero": 0,
"subone": 1,
"subtwo": 2,
"subthree": 3,
},
"arr": [0,1,2,3,4,5],
"x": 1,
"y": 1,
"a": "a",
"b": "b",
});
let v = value.encode_to_vec();
let json = format!("json: {}", value);
assert_eq!(json.len(), 208);
expression_tester(&v, r#"2.14"#, Ok(DecodedValue::Double(2.14)), line!());
expression_tester(&v, r#"2.14f"#, Ok(DecodedValue::Float(2.14f32)), line!());
expression_tester(&v, r#"-42"#, Ok(DecodedValue::Int(-42)), line!());
expression_tester(&v, r#"-2.14"#, Ok(DecodedValue::Double(-2.14)), line!());
//expression_tester(&v, br#"-one"#, Ok(Binary::Int(-1), line!());
//expression_tester(&v, br#"-two_f32"#, Ok(Binary::Float(-2.25f32)), line!();
expression_tester(&v, r#"one + two"#, Ok(DecodedValue::Int(3)), line!());
expression_tester(&v, r#"one - two"#, Ok(DecodedValue::Int(-1)), line!());
expression_tester(&v, r#"two * two"#, Ok(DecodedValue::Int(4)), line!());
expression_tester(&v, r#"three / two"#, Ok(DecodedValue::Int(1)), line!());
expression_tester(&v, r#"three % two"#, Ok(DecodedValue::Int(1)), line!());
//expression_tester(&v, r#"three ** two"#, Ok(Binary::Double(9.0)), line!());
expression_tester(&v, r#"10 - 2"#, Ok(DecodedValue::Int(8)), line!());
// -2 is tokenized as number...
expression_tester(&v, r#"10-2"#, Ok(DecodedValue::Int(8)), line!());
expression_tester(&v, r#"10-2.1"#, Ok(DecodedValue::Double(7.9)), line!());
expression_tester(&v, r#""a" .. "b""#, Ok(DecodedValue::String(b"ab")), line!());
expression_tester(&v, r#""a" .. /* "c" .. */ "b""#, Ok(DecodedValue::String(b"ab")), line!());
expression_tester(&v, r#""a" .. "b" // .. "c""#, Ok(DecodedValue::String(b"ab")), line!());
expression_tester(&v, r#"this.get("a") .. this.get("b")"#, Ok(DecodedValue::String(b"ab")), line!());
expression_tester(&v, r##" r#"foo"# "##, Ok(DecodedValue::String(b"foo")), line!());
expression_tester(&v, r##" "\uD83D\uDE00" "##, Ok("😀".into()), line!());
expression_tester(&v, r##" '\uD83D\uDE00' "##, Ok("😀".into()), line!());
expression_tester(&v, r##" 'fo\'o' "##, Ok(b"fo'o".into()), line!());
expression_tester(&v, r##" 'fo\'o'.upper() "##, Ok(DecodedValue::String(b"FO'O")), line!());
expression_tester(&v, r##" 'fo\'o'.upper() "##, Ok(DecodedValue::String(b"FO'O")), line!());
expression_tester(&v, r##" '<html>'.htmlescape() "##, Ok(DecodedValue::String(b"<html>")), line!());
expression_tester(&v, r##" 'fo\'o'.sub(1, 2) "##, Ok(DecodedValue::String(b"o'")), line!());
expression_tester(&v, r##" 'foobar'.sub(0, 1000) "##, Ok(DecodedValue::String(b"foobar")), line!());
expression_tester(&v, r##" 'foobar'.sub(1001, 1000) "##, Ok(DecodedValue::String(b"")), line!());
expression_tester(&v, r##" 'foobar'.sub(3, -1) "##, Ok(DecodedValue::String(b"")), line!());
expression_tester(&v, r##" 'foobar'.sub(3) "##, Ok(DecodedValue::String(b"bar")), line!());
// in mini mode, the code is not folded, so this error is not triggered
#[cfg(not(feature = "mini"))]
{
expression_tester_parser_error(r##"!1"##, CompileErrorDescription::Optimizer(EvalError::Dynamic("NOT expects a boolean")));
expression_tester_parser_error(r##"1f>>3"##, CompileErrorDescription::Optimizer(EvalError::Dynamic("Arithmetic operation without proper arguments (maybe a bitwise operation on a float or a comparison between a float and int?)")));
}
expression_tester(&v, r##"1==!!!!faalse"##, Err(EvalError::FieldNotFound("faalse")), line!());
expression_tester(&v, r#"0???"#, Ok(DecodedValue::Int(0)), line!());
expression_tester_parser_error(r#"--9223372036854775808"#, CompileErrorDescription::Parser("could not negate the int number"));
expression_tester(&v, r#"foo(1)"#, Err(EvalError::Dynamic("error during call to 'foo(..)':\nno custom functions defined")), line!());
expression_tester(&v, r##" one < two "##, Ok(value!(true)), line!());
expression_tester(&v, r##" two > one "##, Ok(value!(true)), line!());
expression_tester(&v, r##" "foo".basename() "##, Ok(value!("foo")), line!());
expression_tester(&v, r##" "foo".dirname() "##, Ok(value!("")), line!());
expression_tester(&v, r##" "/foo".basename() "##, Ok(value!("foo")), line!());
expression_tester(&v, r##" "/foo".dirname() "##, Ok(value!("")), line!());
expression_tester(&v, r##" "x/foo".basename() "##, Ok(value!("foo")), line!());
expression_tester(&v, r##" "x/foo".dirname() "##, Ok(value!("x")), line!());
expression_tester(&v, r##" "x/y/foo".basename() "##, Ok(value!("foo")), line!());
expression_tester(&v, r##" "x/y/foo".dirname() "##, Ok(value!("x/y")), line!());
expression_tester(&v, r##" "x/y/foo".splitn(999, "/") "##, Ok(value!(["x", "y", "foo"])), line!());
expression_tester(&v, r##" {...this, x:2}.x "##, Ok(value!(2)), line!());
expression_tester(&v, r##" {x:2, ...this}.x "##, Ok(value!(1)), line!());
expression_tester(&v, r##" {x:1,y:2}.tostring() "##, Ok(value!(r#"{"x":1,"y":2}"#)), line!());
expression_tester(&v, r##" (42).toint() "##, Ok(value!(42)), line!());
expression_tester(&v, r##" (42.5).toint() "##, Ok(value!(42)), line!());
expression_tester(&v, r##" (42.5f).toint() "##, Ok(value!(42)), line!());
expression_tester(&v, r##" (42).tofloat() "##, Ok(value!(42f32)), line!());
expression_tester(&v, r##" (42.0).tofloat() "##, Ok(value!(42f32)), line!());
expression_tester(&v, r##" (42.0f).tofloat() "##, Ok(value!(42f32)), line!());
expression_tester(&v, r##" "42.5".tofloat() "##, Ok(value!(42.5f32)), line!());
expression_tester(&v, r##" (42).todouble() "##, Ok(value!(42f64)), line!());
expression_tester(&v, r##" (42.0).todouble() "##, Ok(value!(42f64)), line!());
expression_tester(&v, r##" (42.0f).todouble() "##, Ok(value!(42f64)), line!());
expression_tester(&v, r##" "42.5".todouble() "##, Ok(value!(42.5f64)), line!());
expression_tester(&v, r##" {("foo" .. (2).tostring()): 42} "##, Ok(value!({"foo2":42})), line!());
expression_tester(&v, r##" undefined(foobarfoobar)"##, Ok(value!(true)), line!());
expression_tester(&v, r##" undefined(null)"##, Ok(value!(false)), line!());
expression_tester(&v, r##" undefined(one)"##, Ok(value!(false)), line!());
expression_tester(&v, r##" defined(foobarfoobar)"##, Ok(value!(false)), line!());
expression_tester(&v, r##" defined(null)"##, Ok(value!(true)), line!());
expression_tester(&v, r##" defined(one)"##, Ok(value!(true)), line!());
expression_tester(&v, r##" ' foo '.trim() "##, Ok(value!("foo")), line!());
expression_tester(&v, r##" ' foo '.trim() == 'foo' "##, Ok(value!(true)), line!());
expression_tester(&v, r#"[]"#, Ok(value!([])), line!());
expression_tester(&v, r#"[1]"#, Ok(value!([1])), line!());
expression_tester(&v, r#"[1,]"#, Ok(value!([1,])), line!());
expression_tester(&v, r#"[1,2]"#, Ok(value!([1,2])), line!());
expression_tester(&v, r#"[1,2,]"#, Ok(value!([1,2,])), line!());
expression_tester(&v, r#"{}"#, Ok(value!({})), line!());
expression_tester(&v, r#"{a:1}"#, Ok(value!({"a":1})), line!());
expression_tester(&v, r#"{a:1,}"#, Ok(value!({"a":1,})), line!());
expression_tester(&v, r#"{a:1,b:2}"#, Ok(value!({"a":1,"b":2})), line!());
expression_tester(&v, r#"{a:1,b:2,}"#, Ok(value!({"a":1,"b":2,})), line!());
expression_tester(&v, r#"arr.filter(|x| x % 2 == zero)"#, Ok(value!([0, 2, 4])), line!());
expression_tester(&v, r#"[0,1,2,3,4,5].filter(|x| x % 2 == zero)"#, Ok(value!([0, 2, 4])), line!());
expression_tester(&v, r#"arr.map(|x| x * 2)"#, Ok(value!([0, 2, 4, 6, 8, 10])), line!());
expression_tester(&v, r#"[0,1,2,3,4,5].map(|x| x * 2)"#, Ok(value!([0, 2, 4, 6, 8, 10])), line!());
expression_tester(&v, r#"[0,1,2].map(|x| [x,2*x].map(|y| x+y))"#, Ok(value!([[0, 0], [2,3], [4,6]])), line!());
//expression_tester(&v, r#"{a:{v:1},b:{v:2}}.map_to_array(|k,v| {id:k, ...v})"#, Ok(value!([{"id":"a", "v":1}, {"id":"b", "v":2}])));
expression_tester(&v, r#"[2,1,0].sort_by_key(|x| x)"#, Ok(value!([0,1,2])), line!());
expression_tester(&v, r#"[2,1,0].sort_by_key(|x| 2-x)"#, Ok(value!([2,1,0])), line!());
expression_tester(&v, r#"[{a:1},{a:2},{a:3}].sort_by_key(|x| x.a)"#, Ok(value!([{"a":1},{"a":2},{"a":3}])), line!());
expression_tester(&v, r#"[{a:1},{a:2},{a:3}].sort_by_key(|x| ["a", x.a])"#, Ok(value!([{"a":1},{"a":2},{"a":3}])), line!());
expression_tester(&v, r#"["c","b","a"].sort_by_key(|x| x)"#, Ok(value!(["a","b","c"])), line!());
if cfg!(feature = "mini") {
expression_tester(&v, r#"[1,2,3][-1]"#, Err(EvalError::Dynamic("array out of bounds (index -1 of array of length 3)")), line!());
} else {
expression_tester_parser_error(r#"[1,2,3][-1]"#, CompileErrorDescription::Optimizer(EvalError::Dynamic("array out of bounds (index -1 of array of length 3)")));
}
expression_tester(&v, r#"[1,2,3][0]"#, Ok(value!(1)), line!());
expression_tester(&v, r#"[1,2,3][1]"#, Ok(value!(2)), line!());
expression_tester(&v, r#"[1,2,3][2]"#, Ok(value!(3)), line!());
if cfg!(feature = "mini") {
expression_tester(&v, r#"[1,2,3][3]"#, Err(EvalError::Dynamic("array out of bounds (index 3 of array of length 3)")), line!());
} else {
expression_tester_parser_error(r#"[1,2,3][3]"#, CompileErrorDescription::Optimizer(EvalError::Dynamic("array out of bounds (index 3 of array of length 3)")));
}
expression_tester(&v, r#"arr[-1]"#, Err(EvalError::Dynamic("array out of bounds (index -1 of array of length 6)")), line!());
expression_tester(&v, r#"arr[0]"#, Ok(value!(0)), line!());
expression_tester(&v, r#"arr[1]"#, Ok(value!(1)), line!());
expression_tester(&v, r#"arr[2]"#, Ok(value!(2)), line!());
// based on fuzzing some weird test cases
if cfg!(feature = "mini") {
expression_tester_parser_error(r#"0f.F**fa"#, CompileErrorDescription::Parser("expected expr primary (possibly mismatched closing '}'/']')"));
} else {
expression_tester_parser_error(r#"0f.F**fa"#, CompileErrorDescription::Optimizer(EvalError::Dynamic("field access on a non-object: float")));
}
// triggers a stack overflow without the depth protection that is build in
expression_tester_parser_error(r#"[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[["#, CompileErrorDescription::ParserTooLong);
expression_tester_parser_error(r#"!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!1"#, CompileErrorDescription::ParserTooLong);
expression_tester_parser_error(r#"{x'Ad'ld"#, CompileErrorDescription::Parser("a value for this field is necessary (use different syntax to copy over fields)"));
expression_tester_parser_error(r#"-2<-0<<3||23||2<-4<<3||3<-4<<3||324<<3||3<--0<<3||23||2<-2<-0<<3||23||2<-4<<2<-0<<3||23||2<-2<-0<<3||23||2<-4<<3||3<-4<<3||324<<3||3<--0<<3||2<-2<-0<<3||23||2<-4<<3||3<-4<<3||4<<3||3<-4<<3||324<<3||3<--0<<3||23||2<-2|23||2<-2<-0<<3||23||2<-4<<2<-0<<3||23||2<-2<-0<<3||23||2<-4<<3||3<-4<<3||324<<3||3<--0<<3||2<-2<-0<<3||23||2<-4<<3||3<-2<<3||324<<3||3<-4<<3||4<<3||3<-4<<3||324<<3||3<--0<<3||23||2<-2<-0<<3||23||2<-4<<2<-0<<3||23||2<-2<-0<<3||23||2<-4<<3||3<-4<<3||324<<3||3<--0<<3<-0<<3||23||2<-4<<2<-0<23||2<-4<<3||3<-4<<3||324<<3||3<--0<<3||23||2<-2<-0<<3||23||2<-4<<2<-0<<3||23||2<-2<-0<<3||23||2<-4<<3||3<-4<<3||324<<3||3<--0<<3||2<-2<-0<<3||23||2<-4<<3||3<-4<<3||4<<3||3<-4<<3||324<<3||3<--0<<3||23||2<-2|23||2<-2<-0<<3||23||2<-4<<2<-0<<3||23||2<-2<-0<<3||23||2<-4<<3||3<-4<<3||324<<3||3<--0<<3||2<-2<-0<<3||23||2<-4<<3||3<-2<<3||324<<3||3<<3||324<<3||3<--0<<3||2<-2<-0<<3||23||2<-4<<3||3<-4<<3||324<<3||3<-4<<3||34<<3|3||3<-434<<3|3||3<-4<<34<<3|23||2<-2<-0<<3||23||2||324<<3||3<-4<<3||34<<3||32|3||3<-4<<34<<3|23||2<-2<-0<<3||23||2||324<<3||3<-4<<3||34<<3||32"#, CompileErrorDescription::Optimizer(EvalError::Dynamic("value for AND/OR was not a boolean, but int")));
if !cfg!(feature = "mini") {
expression_tester_parser_error(r#"j>$this.t"#, CompileErrorDescription::Optimizer(EvalError::FieldNotFound("t")));
}
}
#[test]
fn merge() {
let s = String::from("foo");
let lhs_value = value!({
"null": null,
"zero": 0,
"one": 1,
"two_f32": 2.25f32,
"two_f64": 2.55555555,
"three": 3,
});
let lhs = lhs_value.encode_to_vec();
let rhs_value = value!({
"zero": 1, // this is not zero, so we can test the merge function
"two": 2,
"foo": s,
});
let rhs = rhs_value.encode_to_vec();
let result_value = value!({
"null": null,
"zero": 0,
"one": 1,
"two": 2,
"two_f32": 2.25f32,
"two_f64": 2.55555555,
"three": 3,
"foo": s,
});
let expected = result_value.encode_to_vec();
let mut allocator = MemoryPool::new();
let mut scope = allocator.rewind();
let result = merge_objects_to_scope(lhs.to_ref(), rhs.to_ref(), &mut scope).unwrap();
assert_eq!(expected.to_ref(), result);
}
#[test]
fn parsing_objects() {
let subthree_str = "subthree";
let sub = value!({
"subnull": null,
"subzero": 0,
"subone": 1,
"subtwo": 1+1,
subthree_str: 3,
});
let value = value!({
"null": null,
"zero": 0,
"one": 1,
"two": 2,
"three": 3,
"sub": sub,
});
let value_packed = value.encode_to_vec();
println!("Using value {:?}", value);
println!("Packed value {:?}", value_packed);
//let test = br#"1+2+3"#;
//let test = br#"f(1*2+3*4,5,6,"foo", "bar")"#;
//let test = br#"f("foo",field) >= foo && true"#;
//let test = br#"a ? b : c ? d : e"#;
//let test = br#"{name: 1+2*3+4, field2: 42, "field3": 37}"#;
//let test = br#"a.b???c??d"#;
//let test = br#"{a:1f,foo???}"#;
// let test = br#"c + (b + a)"#;
// println!("a = {:?}", key("a"));
// println!("b = {:?}", key("b"));
// println!("c = {:?}", key("c"));
//let test = br#"10.5 % 2.25"#;
//let test = br#"10.5 / 0 ??? 42"#;
//let test = br#"zero + one + [1,2,3].len()"#;
//let test = br#"zero + sub.subone + sub.subtwo"#;
let test_expr = r#"zero + sub.get("subone") + sub.subtwo"#;
//let test = br#"null ?? 42"#;
//let test = br#" "foo".len()"#;
//let test = br#"one == 0 ? "true" : "false""#;
//let test = br#"[1,2,3]"#;
//let test = br#"{"one":one,"two":two,"three":three}"#;
//let test = br#"{"one":1,"two":2,"three":3}"#;
let mut allocator = MemoryPool::new();
let mut scope2 = allocator.rewind();
{
let mut scope1 = scope2.rewind();
let mut bytecode = expry_compile_expr(test_expr, None, Some(0), &["this"], &mut scope1);
if let Ok((bytecode,_)) = &mut bytecode {
let mut scope = scope1.rewind();
println!("Parsed eval {:?}", bytecode);
let result = expry_eval(*bytecode, &mut vec![value_packed.to_ref()], &mut scope).unwrap();
println!("Result {:?}", result);
assert_eq!(result, value!(3));
} else {
println!("Parse error {:?}", bytecode);
assert!(false);
}
// will trigger a compiler warning: because expry_parse_ast_optimize will modify the ast in
// place and that will result in a dependency on the lifetime of `scope` (which is not
// available here anymore).
// let mut scope3 = scope1.scope();
// expry_parse_ast_optimize(&mut object.unwrap(), &mut scope3).unwrap();
}
}
/*
* value!() knows all the types, even if something is None the full type should be
* available (CHECK THIS). If this is the case, value!() can return a tuple, one with the
* concrete value, and one with the type info. This second tuple can then be used for
* checking types compile time on invocation sites. These needs a macro, e.g.
* expry_static_eval!("a+b", val); with let val = value!(..);
* This only works with static available expressions.
*
* The same can be done for hair templates if they are known statically.
* The type checker for hair templates can work with get_type(), because types
* are derived from binary expressions. And then assigned to loop variables and such.
*
* We can also have a function to dynamically type check an expression. But I don't know
* how useful that will be. It gives a bit more certainty, but is there a big advantage
* besides just running it the template/expry? There is a bit of a mismatch between one
* type instance or all possible types. But I can not imagine when to use that.
*/
#[test]
fn type_check() {
//FIXME: let expr = r#"a.len()+b+c.get("z" .. d).tostring().len()"#; // this one converts to json,
//so might not be ideal? check_string(x) is better, but... verbose syntax, so might be
//something like r#"a.len()+b+c.get<string>("z" .. d).len()"#;
//let expr = r#"a.len()+b+c.get("z" .. d).tostring().len()"#;
//let expr = r#"a == b"#;
//let expr = r#"a == []"#;
let expr = r#"a == [1]"#;
let mut allocator = MemoryPool::new();
let mut scope = allocator.rewind();
let value_names = &["this"];
let bytecode = expry_compile_expr(expr, None, Some(0), value_names, &mut scope);
if let Ok((bytecode,_)) = bytecode {
let type_string = r#"{a: [int], b: int, c: {z: string}, d: string}"#;
let types = expry_parse_type(type_string, &mut scope);
if let Err(err) = &types {
println!("{}", expry_compile_error_format(err, 0).1);
}
let types = types.unwrap();
//let result = expry_eval(bytecode, v.to_ref(), &mut scope);
// let mut subtypes = TypeObject::new();
// subtypes.insert(key_str("z"), TypeValue::String);
// let mut types = TypeObject::new();
// types.insert(key_str("a"), TypeValue::Array(Box::new(TypeValue::Int)));
// types.insert(key_str("b"), TypeValue::Int);
// types.insert(key_str("c"), TypeValue::Object(subtypes));
// types.insert(key_str("d"), TypeValue::String);
let mut bytecode_reader = RawReader::with(bytecode.get());
let magic = bytecode_reader.read_bytes(EXPRY_MAGIC.len());
assert!(magic == Ok(EXPRY_MAGIC));
let count = bytecode_reader.read_u8();
assert!(count == Ok(value_names.len() as u8));
let mut context = TypeContext {
custom_functions: &BTreeMap::new(),
warnings: &mut Vec::new(),
};
assert_eq!(Ok(ExpryType::Bool), get_type_from_bytecode(&mut bytecode_reader, &[&types], &mut context, &mut scope));
} else {
panic!();
}
}
}