hairy 0.1.2

Compiled text templates (not unlike Mustache and Handlebars), with support for expressions and custom functions inside such expressions.
Documentation
//! The `hairy` crate provides text templates, not unlike mustache and handlebars (and the earlier ctemplate original), but a bit different.
//! Scoping is different: variables must specify explicitly which scope they use (by using `a.b` syntax). This is to avoid implicit behaviour when variables are (accidentally) overwritten.
//! All errors are treated as hard errors and are reported.
//! So missing values and typing errors are not silently ignored.
//! Also, the templates support evaluation: `{{=expression}}`.
//! The supported expressions are from the [`expry`] crate, which allows executing expression on binary encoded JSON-like values (with support for defining custom functions).
//!
//! # Syntax
//! In short the features and syntax:
//! - Variables can be declared at top of an input file with `type key: expry` pairs (JSON is a valid expry). `type` can be either `inline` (value is replaced during compilation of the template) or `default` (which is a default value that is added to the evaluation time value if the key does not exists). The first non-parsable line signals the end of the variables.
//! - Values can be outputted with `{{=value}}` (supports expressions, see below). Escape mode can
//! be specified with `{{=value:escape_mode}}`. Values are processed by a user-definable escaper
//! that can take the escape mode into account. Normally only strings and numbers are displayed,
//! and the `null` value is considered a nop (without generating an error). Other values are considered an error, except when the
//! `sjs` (script-js) or `js` escape modes are used.
//! - Conditionals with `{{if value}}contents{{end}}`. Contents is displayed if value evaluates to `true`.
//! - Iterators with `{{for variable in name}}content{{end}}`, which can be used to iterate over (arrays of) any type. The contents of the array is available in the loop body under key `variable`;
//! - Template definition with `{{define name}}content{{end}}`. Templates can have optional default values (in an object) with `{{define name defaults object}}content{{end}}`. Defaults are resolved at template compile time, using the global context given to the compile function;
//! - Template instantiation with `{{call name}}` or `{{call name with value}}`. `name` can be an expression. If the name starts with a `*`, name can be an expression that resolves to a string (which is treated as a template name). If the name starts with `**`, name should be an expression that resolves to a binary form template code (as produced by the compile functions). If the `with` syntax is used, only the data specified in `value` is passed along (so the current context is not automatically transfered/merged, to do that use for value `{...this, key: value}`). This is done to avoid errors.
//! - Error handling is different from normal Mustache: missing fields is always considered an error. Errors can be suppressed with the `expr ??? alternative` try syntax (on error in `expr`, `alternative` will be executed). Shorthand for `expr ??? null` is `expr ???`, which can be used in loops with `{{for i in someField???}}`, which ignores errors if the field is not found. Same for conditionals.
//!
//! Some other template systems have the ability to iterate without variables, in effect making 
//! the object fields available in the current context. Although this is at first appealing,
//! this makes these templates error-prone. Types can differ, depending on the presence of fields
//! in the object iterated over (which can change for each iteration). One of the design goals of hairy
//! is error handling. To fall back to the value in the global context on error such as field not
//! found, the following expression can be used: `{{iteration_variable.foo???foo}}`. This makes the
//! whole expression explicit in its intent.
//!
//! ## Escaping
//!
//! Normally, all strings that are dynamically outputted (using an evaluating `{{=..}}`
//! statement) are automatically 'escaped' using the escaper as given to an argument to the
//! `hairy_eval` function. However, often you will want different escape modes depending on the
//! input context. For example in HTML, output may not normally contain `<` or `>`, but output
//! inside attributes in HTML tags are normally escaped like URLs. Although users can specify for
//! every output the escape mode by appending a `:html` or `:url` escaping mode to an expression,
//! this is error-prone. Auto escaping is therefore a safer alternative, by automatically deciding
//! the escape mode from the input. The general advise is to escape defensively. `hairy` functions
//! support an argument that is used to look up the escaping mode for a certain position in the input.
//!
//! # Easy interface
//! The 'easy' interface if you just want to quickly use an HTML template, with auto escaping of the
//! input, and returning a nicely formatted error that can be presented to the user. To evaluate, use [`hairy_eval_html`], so the proper escaper is used.
//!
//! Note that although [`hairy_compile_html`] and [`hairy_eval_html]` are easier to use, they are
//! somewhat slower. For top performance please use the [`hairy_compile`] and the [`hairy_eval`]
//! functions.
//!
//! # Example using the simple interface
//!
//! ```
//! use hairy::*;
//! use expry::*;
//! use std::io::Write;
//! let template = r#"foobar = {{=foovar .. barvar}}"#;
//! let result = hairy_compile_html(template, "test.tpl", None, 0);
//! match result {
//!   Ok(parsed) => {
//!     let value = value!({
//!       "foovar": "foo",
//!       "barvar": "bar",
//!     }).to_vec(false);
//!     match hairy_eval_html(parsed.to_ref(), value.to_ref()) {
//!       Ok(output) => { std::io::stdout().write_all(&output); },
//!       Err(err) => { eprintln!("{}", err); },
//!     }
//!   },
//!   Err(err) => {
//!     eprintln!("{}", err);
//!     assert!(false); // to check this example
//!   }
//! }
//! ```
//!
//! # Enclosing other template files
//!
//! The value can contain other template compiled with `hairy_compile_html`. So if value `body`
//! contains the bytecode of such a template, that template can be invoked from the 'main' template
//! by using the template expression `{{call **body with argument}}`.
//!
//! ```
//! use hairy::*;
//! use expry::*;
//! use std::io::Write;
//! let main_template = r#"<html><title>{{=title}}</title><body>{{call **body with {title,foobarvar}}}</body></html>"#;
//! let main = hairy_compile_html(main_template, "main.tpl", None, 0).unwrap();
//! let child_template = r#"<p>title of this page = {{=title}}</p><p>foobar = {{=foobarvar}}"#;
//! let child = hairy_compile_html(child_template, "child.tpl", None, 0).unwrap();
//! let value = value!({
//!   "body": child,
//!   "foobarvar": "foobar",
//!   "title": "my title",
//! }).to_vec(false);
//! match hairy_eval_html(main.to_ref(), value.to_ref()) {
//!   Ok(output) => { std::io::stderr().write_all(&output).unwrap(); },
//!   Err(err) => {
//!     eprintln!("{}", err);
//!     assert!(false); // to check this example
//!   },
//! }
//! ```
#![allow(dead_code)]

extern crate alloc;
use alloc::format;
use alloc::vec;
use alloc::vec::Vec;
use alloc::string::String;

use core::fmt;
use core::fmt::Write;

pub use expry::{BytecodeVec,BytecodeRef,ValueVec,ValueRef,NoCustomFuncs};
use expry::*;

const EMPTY_EXPRY_OBJECT: &[u8] = b"\x06";

/// Internal error of the Hairy lexer. Contains the source code line number.
#[derive(Debug, Copy, Clone)]
pub struct HairyLexerError {
    line_nr: u32,
}

/// The errors that can occur during Hairy template parsing. It can either be a lexer error (so a part of
/// the input can not be translated to a token), or a parser error (a different token expected).
/// As Hairy templates can contain expressions, these expressions can also trigger errors. These
/// can either be in the parsing of the expressions, or during optimizing (which causes
/// EvalErrors). Other errors are reporting using `Other`.
#[derive(Copy, Clone)]
pub enum HairyParserError<'a> {
    Lexer(HairyLexerError),
    Parser(&'a str),
    Expr(CompileErrorDescription<'a>),
    Eval(EvalError<'a>),
    Other(&'a str),
}

#[derive(Clone, Debug)]
pub struct HairyCompileError<'a> {
    error: HairyParserError<'a>,
    start: usize, // counted from the back of the input string
    end: usize, // counted from the back of the input string
    extra: Option<(usize, usize)>,
    line_context: LineContext,
}

impl<'a> core::fmt::Display for HairyCompileError<'a> {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        write!(f, "{} [{}-{}]", self.error, self.start, self.end)
    }
}

impl<'a> From<EncodingError> for HairyParserError<'a> {
    fn from(_: EncodingError) -> Self {
        HairyParserError::Other("internal encoding error")
    }
}
impl<'a> fmt::Debug for HairyParserError<'a> {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        write!(f, "{}", self)
    }
}
impl<'a> core::fmt::Display for HairyParserError<'a> {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        match self {
            HairyParserError::Lexer(err) => write!(f, "{:?} (in lexer)", err),
            HairyParserError::Expr(err) => write!(f, "{} (during parsing of the expression)", err),
            HairyParserError::Parser(s) => write!(f, "{}", s),
            HairyParserError::Other(s) => write!(f, "{}", s),
            HairyParserError::Eval(s) => write!(f, "{}", s),
        }
    }
}

type Expr<'a> = &'a str;
type Text<'a> = &'a str;
type Name<'a> = &'a str;
type Bytes<'a> = &'a [u8];

#[derive(PartialEq, Eq, Copy, Clone, Debug)]
enum HairyToken<'a> {
    EndOfInput(),
    Text(Text<'a>),
    Loop(Expr<'a>, usize, Name<'a>, Name<'a>), // expr, offset of expr variable, index variable, contents variable
    Conditional(Expr<'a>, usize), // expr
    Else(),
    BlockEnd(Name<'a>),
    Eval(Expr<'a>, usize, Bytes<'a>), // .1: offset of expression (for error messages)
    CallTemplate(Name<'a>, usize, Expr<'a>, usize), // .1, .3: offset of name/expression (for error messages)
    TemplateDef(Name<'a>, Expr<'a>, usize),  // name, defaults, offset_defaults
}

/// This spanner will try to recognize the content of a Hairy template (`{{..}}`). It counts the number of
/// nested `{` and `}`. It does have support for recognizing embedded strings, so `{{foo != "}"}}` will work. Raw strings or the like are not supported, instead you can use `{foo != r#" \" "#}`. Stops at first `}` encountered that is not started inside the expression.
pub struct TemplateTagContentSpanner {
    prev_was_escape: bool,
    string: bool,
    nested: u32,
}

impl Spanner for TemplateTagContentSpanner {
    fn next(&mut self, b: char) -> bool {
        if !self.prev_was_escape && !self.string && b == '{' {
            self.nested += 1;
        }
        if !self.prev_was_escape && !self.string && b == '}' {
            if self.nested == 0 {
                return false;
            }
            self.nested -= 1;
        }
        if !self.prev_was_escape && b == '"' {
            self.string = !self.string;
        }
        self.prev_was_escape = !self.prev_was_escape && b == '\\';
        true
    }
    fn valid(&mut self) -> bool {
        self.nested == 0 && !self.string && !self.prev_was_escape
    }
}
impl TemplateTagContentSpanner {
    pub fn new() -> Self {
        Self {
            prev_was_escape: false,
            string: false,
            nested: 0,
        }
    }
}

impl Default for TemplateTagContentSpanner {
    fn default() -> Self {
        Self::new()
    }
}

/// Recognizes expressions by counting the `{`, `}`, `[`, `]`, and `"`. Does not support advanced
/// string literals such as raw strings (e.g. `r##" " "##`). Stops at first `]` or `}` encountered
/// that is not started inside the expression.
pub struct ExprySpanner {
    prev_was_escape: bool,
    string: bool,
    nested: u32,
}

impl Spanner for ExprySpanner {
    fn next(&mut self, b: char) -> bool {
        if !self.prev_was_escape && !self.string && (b == '{' || b == '[') {
            self.nested += 1;
        }
        if !self.prev_was_escape && !self.string && (b == '}' || b == ']') {
            if self.nested == 0 {
                return false;
            }
            self.nested -= 1;
        }
        if !self.prev_was_escape && b == '"' {
            self.string = !self.string;
        }
        self.prev_was_escape = !self.prev_was_escape && b == '\\';
        true
    }

    fn valid(&mut self) -> bool {
        self.nested == 0 && !self.string && !self.prev_was_escape
    }
}

impl ExprySpanner {
    pub fn new() -> Self {
        Self {
            prev_was_escape: false,
            string: false,
            nested: 0,
        }
    }
}

impl Default for ExprySpanner {
    fn default() -> Self {
        Self::new()
    }
}

fn find_subsequence<T>(haystack: &[T], needle: &[T]) -> Option<usize> where for<'a> &'a [T]: PartialEq {
    haystack.windows(needle.len()).position(|window| window == needle)
}

fn hairy_tokenize<'a,'b,'c>(
    reader: &mut &'b str,
    context: &mut ParserContext<'a,'b,'c>,
) -> Result<(HairyToken<'b>,usize), (HairyParserError<'b>,usize,usize)> {
    const EXPECT_HAIRY_CLOSE : HairyParserError = HairyParserError::Parser("expecting closing `}}` after a non empty expression");
    fn read_enters(reader: &mut &str) -> usize {
        let mut retval = 0;
        if reader.accept("\r") {
            retval += 1;
        }
        if reader.accept("\n") {
            retval += 1;
        }
        retval
    }
    while reader.accept("{{!") {
        let mut embedded_spanner = TemplateTagContentSpanner::new();
        embedded_spanner.span(reader);
        if !reader.accept("}}") {
            return Err((EXPECT_HAIRY_CLOSE, reader.len(), reader.len()));
        }
        read_enters(reader);
    }
    let remaining = reader.len();
    if reader.is_empty() {
        return Ok((HairyToken::EndOfInput(), remaining));
    }
    //let err_pos = *pos;
    if reader.accept("{{for ") {
        let mut embedded_spanner = TemplateTagContentSpanner::new();
        let expr = embedded_spanner.span(reader).map_or("", |x| x);
        if expr.is_empty() || !reader.accept("}}") {
            return Err((EXPECT_HAIRY_CLOSE, reader.len(), reader.len()));
        }
        let content_end = reader.len() + 2;
        let content_end_offset = 2 + read_enters(reader);
        const IN_LEN : usize = 4;
        if let Some((variable, expr)) = expr.split_once(" in ") {
            let mut variable = variable.trim_start();
            let mut index_variable = "";
            let mut content_variable = "";
            let matcher = |c: char| c == '$' || c == '_' || c == '\\' || c.is_ascii_alphanumeric();
            if strparse(variable).accept('(').span(is_space, 0).span(matcher, 1).matched(&mut index_variable).span(is_space, 0).accept(",").span(is_space, 0).span(matcher, 1).matched(&mut content_variable).span(is_space, 0).accept(')').valid(&mut variable) {
                return Ok((HairyToken::Loop(expr, content_end_offset, index_variable, content_variable), remaining));
            }
            if variable.trim().contains(|x| !matcher(x)) {
                const EXPECT_HAIRY_VARIABLE : HairyParserError = HairyParserError::Parser("variable name may only consists of alphanumeric chars, and `$` and `_`");
                return Err((EXPECT_HAIRY_VARIABLE, variable.len() + IN_LEN + expr.len() + content_end, IN_LEN + expr.len() + content_end));
            }
            return Ok((HairyToken::Loop(expr, content_end_offset, "", variable), remaining));
        } else {
            const EXPECT_HAIRY_VARIABLE : HairyParserError = HairyParserError::Parser("expected variable name (indicated with `{{for variable in expr}}`)");
            return Err((EXPECT_HAIRY_VARIABLE, expr.len() + content_end, content_end));
        }
    }
    if reader.accept("{{if ") {
        let mut embedded_spanner = TemplateTagContentSpanner::new();
        let expr = embedded_spanner.span(reader).map_or("", |x| x);
        if expr.is_empty() || !reader.accept("}}") {
            return Err((EXPECT_HAIRY_CLOSE, reader.len(), reader.len()));
        }
        let content_end_offset = 2 + read_enters(reader);
        return Ok((HairyToken::Conditional(expr, content_end_offset),remaining));
    }
    if reader.accept("{{else}}") {
        read_enters(reader);
        return Ok((HairyToken::Else(),remaining));
    }
    if reader.accept("{{end") {
        let mut embedded_spanner = TemplateTagContentSpanner::new();
        let expr = embedded_spanner.span(reader).map_or("", |x| x);
        if !reader.accept("}}") {
            return Err((EXPECT_HAIRY_CLOSE, reader.len(), reader.len()));
        }
        read_enters(reader);
        return Ok((HairyToken::BlockEnd(expr),remaining));
    }
    if reader.accept("{{define ") {
        let mut embedded_spanner = TemplateTagContentSpanner::new();
        let expr = embedded_spanner.span(reader).map_or("", |x| x);
        if expr.is_empty() || !reader.accept("}}") {
            return Err((EXPECT_HAIRY_CLOSE, reader.len(), reader.len()));
        }
        let content_end = reader.len() + 2;
        let content_end_offset = 2 + read_enters(reader);
        const DEFAULTS_LEN : usize = 10;
        let check = |name: &str, extra_len: usize| -> Result<(),(HairyParserError, usize, usize)> {
            let matcher = |c: char| c == '$' || c == '_' || c == '\\' || c.is_ascii_alphanumeric();
            if name.trim().contains(|x| !matcher(x)) {
                const HAIRY_ERROR : HairyParserError = HairyParserError::Parser("template name may only consists of alphanumeric chars, and `$` and `_`");
                return Err((HAIRY_ERROR, name.len() + extra_len + content_end, extra_len + content_end));
            }
            Ok(())
        };
        if let Some((name, defaults)) = expr.split_once(" defaults ") {
            check(name, DEFAULTS_LEN + defaults.len())?;
            return Ok((HairyToken::TemplateDef(name.trim(), defaults, content_end_offset),remaining));
        }
        check(expr, 0)?;
        return Ok((HairyToken::TemplateDef(expr.trim(), "", content_end_offset),remaining));
    }
    if reader.accept("{{call ") {
        // FIXME: switch to different spanner, to handle {{call **(" with ") with 1}}
        // likewise for different commands
        let mut embedded_spanner = TemplateTagContentSpanner::new();
        let expr = embedded_spanner.span(reader).map_or("", |x| x);
        if expr.is_empty() || !reader.accept("}}") {
            return Err((EXPECT_HAIRY_CLOSE, reader.len(), reader.len()));
        }
        let content_end = reader.len() + 2;
        let content_end_offset = 2 + read_enters(reader);
        const WITH_LEN : usize = 6;
        let check = |name: &str, extra_len: usize| -> Result<(),(HairyParserError, usize, usize)> {
            let matcher = |c: char| c == '$' || c == '_' || c == '\\' || c.is_ascii_alphanumeric();
            if !name.starts_with('*') && name.trim().contains(|x| !matcher(x)) {
                const HAIRY_ERROR : HairyParserError = HairyParserError::Parser("template name may only consists of alphanumeric chars, and `$` and `_` (and start with `*` or `**`)");
                return Err((HAIRY_ERROR, name.len() + extra_len + content_end, extra_len + content_end));
            }
            Ok(())
        };
        if let Some((name, expr)) = expr.split_once(" with ") {
            //println!("call to template '{}' with expr '{}'", String::from_utf8_lossy(name), String::from_utf8_lossy(expr));
            check(name, WITH_LEN + expr.len())?;
            return Ok((HairyToken::CallTemplate(name.trim(), WITH_LEN + expr.len() + content_end_offset, expr, content_end_offset),remaining));
        }
        check(expr, 0)?;
        return Ok((HairyToken::CallTemplate(expr, content_end_offset, "", content_end_offset), remaining));
    }
    if reader.accept("{{=") {
        let mut embedded_spanner = TemplateTagContentSpanner::new();
        let expr = embedded_spanner.span(reader).map_or("", |x| x);
        if expr.is_empty() || !reader.accept("}}") {
            return Err((EXPECT_HAIRY_CLOSE, reader.len(), reader.len()));
        }
        let content_end_offset = 2;
        const ESCAPE_MODE_LEN : usize = 1;
        if let Some((expr, escape_mode)) = expr.rsplit_once(':') {
            if escape_mode.chars().all(|c| c.is_ascii_alphanumeric()) {
                return Ok((HairyToken::Eval(expr, escape_mode.len()+ESCAPE_MODE_LEN+content_end_offset, escape_mode.as_bytes()),remaining));
            }
        }
        return Ok((HairyToken::Eval(expr, content_end_offset, b""),remaining));
    }
    if reader.accept("{{") {
        const EXPECT_HAIRY_COMMAND : HairyParserError = HairyParserError::Parser("expected a command after an hairy opening tag: if, else, end, for, call, define, =");
        return Err((EXPECT_HAIRY_COMMAND, remaining, reader.len()));
    }

    let mut retval = ScopedStringBuilder::new(context.allocator);
    loop {
        // searching with reader.find("{{") is slower (?!?)
        let mut start = 0;
        let mut found;
        loop {
            found = reader[start..].find('{').map(|x| x+start);
            if let Some(pos) = found {
                if pos >= reader.len()-1 || &reader[pos..pos+2] != "{{" {
                    start = pos+1;
                    continue;
                }
            }
            break;
        }
        match found {
            None => {
                let extra = &reader[..];
                if !extra.is_empty() {
                    retval.write_str(extra).ok();
                }
                *reader = &reader[0..0];
                return Ok((HairyToken::Text(retval.build()),remaining));
            },
            Some(n) => {
                let extra = &reader[0..n];
                *reader = &reader[n..];
                if !extra.ends_with('\\') {
                    if !extra.is_empty() {
                        retval.write_str(extra).ok();
                    }
                    return Ok((HairyToken::Text(retval.build()),remaining));
                }
                let extra = &extra[..extra.len()-1];
                if !extra.is_empty() {
                    retval.write_str(extra).ok();
                }
                let reader_saved = *reader;
                *reader = &reader[2..]; // skip "{{"
                let mut embedded_spanner = ExprySpanner::new();
                if let Some(expr) = embedded_spanner.span(reader) {
                    let expr = &reader_saved[0..expr.len()+4]; // include the "{{" before and the "}}" after
                    retval.write_str(expr).ok();
                }
                if !reader.accept("}}") {
                    return Err((EXPECT_HAIRY_CLOSE, reader.len(), reader.len()));
                }
            },
        }
    }
}

#[allow(clippy::upper_case_acronyms)]
type AST<'a> = Vec<HairyCommand<'a>>;
type LineNo = u32;
type ColumnNo = u32;

type SourceContext = (LineNo, ColumnNo);

#[derive(PartialEq, Eq, Clone)]
enum ResolveTemplate<'a> {
    Static(Name<'a>),
    DynamicName(BytecodeRef<'a>),
    DynamicBody(BytecodeRef<'a>),
}

#[derive(PartialEq, Clone)]
enum HairyCommand<'a> {
    Text(Text<'a>),
    Loop(BytecodeRef<'a>, Name<'a>, Name<'a>, AST<'a>, SourceContext),
    Conditional(BytecodeRef<'a>, AST<'a>, AST<'a>, SourceContext),
    Eval(BytecodeRef<'a>, Bytes<'a>, SourceContext), // name is the escape mode
    CallTemplate(ResolveTemplate<'a>, BytecodeRef<'a>, SourceContext),
}

impl<'a> fmt::Debug for ResolveTemplate<'a> {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        match self {
            ResolveTemplate::Static(name) => write!(f, "{}", name),
            ResolveTemplate::DynamicName(_) => write!(f, "<dynamic-name>"),
            ResolveTemplate::DynamicBody(_) => write!(f, "<dynamic-body>"),
        }
    }
}
impl<'a> fmt::Debug for HairyCommand<'a> {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        match self {
            HairyCommand::Text(s) => write!(f, "\"{}\"", s),
            HairyCommand::Eval(expr, mode, _) => write!(f, "{{{{{:?}:{:?}}}}}", expr, mode),
            HairyCommand::Conditional(tag, body, else_body, _) => {
                write!(f, "{{{{if {:?}}}}}", tag)?;
                write!(f, "{:?}", body)?;
                write!(f, "{{{{else}}}}")?;
                write!(f, "{:?}", else_body)?;
                write!(f, "{{{{end}}}}")
            },
            HairyCommand::Loop(tag, index_variable, content_variable, body, _) => {
                write!(f, "{{{{for ({:?},{:?}) in {:?}}}}}", index_variable, content_variable, tag)?;
                write!(f, "{:?}", body)?;
                write!(f, "{{{{end}}}}")
            },
            HairyCommand::CallTemplate(name, defaults, _) => {
                write!(f, "{{{{{:?} <- {:?}}}}}", name, defaults)
            },
        }
    }
}

struct ParserContext<'a,'b,'c> where 'c: 'b {
    line_context: LineContext,
    allocator: &'a mut MemoryScope<'c>,
    templates: Vec<(&'b str, &'b str, ValueRef<'b>, AST<'b>)>,
    inlines: Option<&'a DecodedValue<'b>>,
    filename: &'b str,
    custom: &'a mut dyn CustomFuncs,
    input_len: usize,
    escaping: Option<&'a [(usize, &'b [u8])]>,
    extra_line_no: u32,
}
type HairyParserState<'a,'b,'c> = ParserState<'b,HairyToken<'b>, HairyParserError<'b>, ParserContext<'a,'b,'c>>;
type HairyParserResult<'b,T> = ParserResult<T, HairyParserError<'b>>;

fn hairy_expr<'b,'c>(parser: &'_ mut HairyParserState<'_,'b,'c>) -> HairyParserResult<'b,AST<'b>> where 'c: 'b {
    let mut retval: AST = Vec::new();
    parser.repeat(|parser: &mut _| {
        let to_source_context = |pos, parser: &mut HairyParserState| { let (a,b,_,_) = parser.context().line_context.remaining_to_line_info(pos as u32); (a+parser.context().extra_line_no,b) };
        match parser.get()? {
            (HairyToken::Text(text), _) => {
                retval.push(HairyCommand::Text(text));
                Ok(true)
            }
            (HairyToken::Eval(expr, expr_offset, mut escape_mode), info) => {
                let bytecode = expry_compile(expr, parser.context().inlines, parser.context().allocator).map_err(|e| parser.error_other(&info.bound(e.error_start()+expr_offset, e.error_end()+expr_offset), HairyParserError::Expr(e.error())))?;

                let source_context = to_source_context(info.start_to_end_of_input(), parser);
                if escape_mode.is_empty() {
                    if let Some(escaping) = parser.context().escaping {
                        escape_mode = lookup_escape_mode(escaping, parser.context().input_len - info.start_to_end_of_input());
                    }
                }
                retval.push(HairyCommand::Eval(bytecode, escape_mode, source_context));
                Ok(true)
            }
            (HairyToken::CallTemplate(name, offset_name, expr, offset_expr), info) => {
                let expr_bytecode = if expr.is_empty() { BytecodeRef::new() } else { expry_compile(expr, parser.context().inlines, parser.context().allocator).map_err(|e| parser.error_other(&info.bound(e.error_start()+offset_expr, e.error_end()+offset_expr), HairyParserError::Expr(e.error())))? };

                let source_context = to_source_context(info.start_to_end_of_input(), parser);
                let name = name.trim_start_matches(is_space);
                if let Some(name) = name.strip_prefix("**") {
                    let name_bytecode = expry_compile(name, parser.context().inlines, parser.context().allocator).map_err(|e| parser.error_other(&info.bound(e.error_start()+offset_name, e.error_end()+offset_name), HairyParserError::Expr(e.error())))?;
                    retval.push(HairyCommand::CallTemplate(ResolveTemplate::DynamicBody(name_bytecode), expr_bytecode, source_context));
                } else if let Some(name) = name.strip_prefix('*') {
                    let name_bytecode = expry_compile(name, parser.context().inlines, parser.context().allocator).map_err(|e| parser.error_other(&info.bound(e.error_start()+offset_name, e.error_end()+offset_name), HairyParserError::Expr(e.error())))?;
                    retval.push(HairyCommand::CallTemplate(ResolveTemplate::DynamicName(name_bytecode), expr_bytecode, source_context));
                } else {
                    retval.push(HairyCommand::CallTemplate(ResolveTemplate::Static(name), expr_bytecode, source_context));
                }
                Ok(true)
            }
            (HairyToken::Conditional(expr, offset_expr), info) => {
                let expr_bytecode = expry_compile(expr, parser.context().inlines, parser.context().allocator).map_err(|e| parser.error_other(&info.bound(e.error_start()+offset_expr, e.error_end()+offset_expr), HairyParserError::Expr(e.error())))?;

                let source_context = to_source_context(info.start_to_end_of_input(), parser);
                let body = hairy_expr(parser)?;
                match parser.get()? {
                    (HairyToken::Else(), _info) => {
                        let else_body = hairy_expr(parser)?;
                        match parser.get()? {
                            (HairyToken::BlockEnd(end_tag), info) => {
                                if !end_tag.is_empty() && "if" != end_tag.trim() {
                                    let err_msg = write!(parser.context().allocator, "start (if) and end ({}) tag should be the same", end_tag.trim());
                                    return Err(parser.error_other(&info, HairyParserError::Parser(err_msg)))
                                }
                                retval.push(HairyCommand::Conditional(expr_bytecode, body, else_body, source_context));
                                Ok(true)
                            }
                            (token, info) => Err(parser.error_token(token, info, |_| {
                                HairyParserError::Parser("expected 'end' of conditional")
                            })),
                        }
                    }
                    (HairyToken::BlockEnd(end_tag), info) => {
                        if !end_tag.is_empty() && "if" != end_tag.trim() {
                            let err_msg = write!(parser.context().allocator, "start (if) and end ({}) tag should be the same", end_tag.trim());
                            return Err(parser.error_other(&info, HairyParserError::Parser(err_msg)))
                        }
                        retval.push(HairyCommand::Conditional(expr_bytecode, body, Vec::new(), source_context));
                        Ok(true)
                    }
                    (token, info) => Err(parser.error_token(token, info, |_| {
                        HairyParserError::Parser("expected 'end' of conditional")
                    })),
                }
            }
            (HairyToken::Loop(expr, expr_offset, index_variable, content_variable), info) => {
                let expr_bytecode = expry_compile(expr, parser.context().inlines, parser.context().allocator).map_err(|e| parser.error_other(&info.bound(e.error_start()+expr_offset, e.error_end()+expr_offset), HairyParserError::Expr(e.error())))?;

                let source_context = to_source_context(info.start_to_end_of_input(), parser);
                let body = hairy_expr(parser)?;
                match parser.get()? {
                    (HairyToken::BlockEnd(end_tag), info) => {
                        if !end_tag.is_empty() && "for" != end_tag.trim() {
                            let err_msg = write!(parser.context().allocator, "start (for) and end ({}) tag should be the same", end_tag.trim());
                            return Err(parser.error_other(&info, HairyParserError::Parser(err_msg)))
                        }
                        retval.push(HairyCommand::Loop(expr_bytecode, index_variable, content_variable, body, source_context));
                        Ok(true)
                    }
                    (token, info) => Err(parser.error_token(token, info, |_| {
                        HairyParserError::Parser("expected 'end' of loop")
                    })),
                }
            }
            (HairyToken::TemplateDef(name, defaults, offset_defaults), info) => {
                let defaults_bytecode = if defaults.is_empty() {
                    ValueRef(b"")
                } else {
                    let bytecode = expry_compile(defaults, parser.context().inlines, parser.context().allocator).map_err(|e| parser.error_other(&info.bound(e.error_start()+offset_defaults, e.error_end()+offset_defaults), HairyParserError::Expr(e.error())))?;
                    let context = parser.context();
                    expry_slice_func(bytecode, ValueRef(b""), context.allocator, context.custom).map_err(|err| parser.error_other(&info, HairyParserError::Eval(err)))?
                };
                let body = hairy_expr(parser)?;
                match parser.get()? {
                    (HairyToken::BlockEnd(end_tag), info) => {
                        if !end_tag.is_empty() && "define" != end_tag {
                            let err_msg = write!(parser.context().allocator, "start (define) and end ({}) tag should be the same", end_tag);
                            return Err(parser.error_other(&info, HairyParserError::Parser(err_msg)))
                        }
                        let filename = parser.context().filename;
                        // check that we don't override a template
                        if parser.context().templates.iter().any(|(x,_,_,_)| *x == name) {
                            let err_msg = write!(parser.context().allocator, "template '{}' already defined", name);
                            return Err(parser.error_other(&info, HairyParserError::Parser(err_msg)));
                        }

                        parser.context().templates.push((name, filename, defaults_bytecode, body));
                        Ok(true)
                    }
                    (token, info) => Err(parser.error_token(token, info, |_| {
                        HairyParserError::Parser("expected 'end' of template definition")
                    })),
                }
            }
            (token, info) => Err(parser.error_token(token, info, |_| HairyParserError::Parser(""))),
        }
    })?;
    Ok(retval)
}
fn hairy_top_level<'b,'c>(parser: &'_ mut HairyParserState<'_,'b,'c>) -> HairyParserResult<'b,AST<'b>> where 'c: 'b {
    let retval = hairy_expr(parser)?;
    parser.accept(HairyToken::EndOfInput(), None, || HairyParserError::Parser("expected end of input"))?;
    Ok(retval)
}

#[derive(Copy, Clone, Debug, Eq, PartialEq)]
pub enum EscapeMode {
    Normal,
    InsideTag,
    InsideAttributeSingleQuote,
    InsideAttributeDoubleQuote,
    InsideScript,
    InsideStyle,
    InsideComment,
}
fn mode_to_bytes(mode: EscapeMode, tag: &str, attribute: &str) -> &'static [u8] {
    // see https://wonko.com/post/html-escaping/ for explanation
    match mode {
        EscapeMode::Normal => b"", // "html" is the default, so we don't need to store it
        EscapeMode::InsideTag => b"",
        EscapeMode::InsideAttributeDoubleQuote | EscapeMode::InsideAttributeSingleQuote if is_url(tag, attribute) => b"url",
        EscapeMode::InsideAttributeDoubleQuote | EscapeMode::InsideAttributeSingleQuote => b"+",
        EscapeMode::InsideScript => b"sjs", // script js
        EscapeMode::InsideStyle => b"",
        EscapeMode::InsideComment => b"",
    }
}

fn is_url(tag: &str, attribute: &str) -> bool {
    if attribute == "srcset" && tag == "source" {
        return true;
    }
    if attribute == "object" && tag == "data" {
        return true;
    }
    if attribute == "action" && tag == "form" {
        return true;
    }
    if attribute == "href" && (tag == "a" || tag == "area" || tag == "base" || tag == "link") {
        true
    } else {
        attribute == "src" && (tag == "audio" || tag == "embed" || tag == "iframe" || tag == "img" || tag == "input" || tag == "script" || tag == "source" || tag == "track" || tag == "video")
    }
}
fn parse_html<'a,'b, 'c>(input: &'b str, scope: &mut MemoryScope<'c>) -> Result<&'a [(usize,&'static [u8])], HairyCompileError<'b>> where 'c: 'a {
    let mut retval = ScopedArrayBuilder::new(scope);
    let mut inside = 0;
    let mut inside_string = false;

    let mut mode = EscapeMode::Normal;
    let mut tag = "";
    let mut attribute = ""; // can also contain the tag name
    let mut last_mode = mode;
    let mut last_mode_bytes = mode_to_bytes(mode, tag, attribute);
    retval.push((0, mode_to_bytes(mode, tag, attribute)));
    for (i,c) in input.char_indices() {
        // skip any chars inside hairy templates, because otherwise {{foo <- args}} triggers the
        // HTML open.
        if inside_string {
            inside_string = c != '"';
            continue;
        }
        match c {
            '{' => inside += 1,
            '}' => inside -= 1,
            '"' if inside > 0 => inside_string = true,
            _ => {},
        }
        if inside > 0 {
            continue;
        }
        match mode {
            EscapeMode::Normal => {
                if c == '<' {
                    if input[i..].starts_with("<!--") {
                        mode = EscapeMode::InsideComment;
                        tag = "!--";
                    } else if let Some((prefix, _)) = input[i+1..].split_once(|x: char| !x.is_ascii_alphanumeric() && x != '/') {
                        if !prefix.is_empty() {
                            mode = EscapeMode::InsideTag;
                            tag = prefix;
                        }
                    }
                    attribute = "";
                }
            },
            EscapeMode::InsideTag => {
                match c {
                    '=' => {
                        if !input[i..].starts_with("=\"") && !input[i..].starts_with("=\'") {
                            return Err(HairyCompileError{error: HairyParserError::Parser("for auto escaping to work, attributes of HTML should have quotes around the values"), start: input.len()-i, end: input.len()-i, extra: None, line_context: LineContext::new(input), });

                        }
                    }
                    '"' => {
                        mode = EscapeMode::InsideAttributeDoubleQuote;
                    },
                    '\'' => {
                        mode = EscapeMode::InsideAttributeSingleQuote;
                    },
                    '>' => {
                        mode = if tag == "script" { EscapeMode::InsideScript } else if tag == "style" { EscapeMode::InsideStyle } else { EscapeMode::Normal };
                    },
                    '<' => {
                        return Err(HairyCompileError{error: HairyParserError::Parser("unexpected character encounted inside tag, likely to break auto escaping."), start: input.len()-i, end: input.len()-i, extra: None, line_context: LineContext::new(input), });
                    }
                    c if attribute.is_empty() && c.is_ascii_alphanumeric() => {
                        if let Some((prefix,_)) = input[i..].split_once(|x: char| !x.is_ascii_alphanumeric() && x != '_') {
                            attribute = prefix;
                        }
                    },
                    ' ' => {
                        attribute = "";
                    },
                    _ => {
                    },
                };
            },
            EscapeMode::InsideAttributeDoubleQuote => {
                if c == '"' {
                    mode = EscapeMode::InsideTag;
                    attribute = "";
                } else if c == '<' || c == '>' {
                    return Err(HairyCompileError{error: HairyParserError::Parser("unexpected character encounted inside attribute, likely to break auto escaping."), start: input.len()-i, end: input.len()-i, extra: None, line_context: LineContext::new(input), });
                }
            },
            EscapeMode::InsideAttributeSingleQuote => {
                if c == '\'' {
                    mode = EscapeMode::InsideTag;
                    attribute = "";
                } else if c == '<' || c == '>' {
                    return Err(HairyCompileError{error: HairyParserError::Parser("unexpected character encounted inside attribute, likely to break auto escaping."), start: input.len()-i, end: input.len()-i, extra: None, line_context: LineContext::new(input), });
                }
            },
            EscapeMode::InsideScript => {
                if c == '<' {
                    if input[i..].starts_with("</script>") {
                        mode = EscapeMode::InsideTag;
                        tag = "/script";
                    } else {
                        // valid inside script tag:
                        // document.innerHTML = '<button>Open in App</button>';
                    }
                }
            },
            EscapeMode::InsideStyle => {
                if c == '<' {
                    if input[i..].starts_with("</style>") {
                        mode = EscapeMode::InsideTag;
                        tag = "/style";
                    } else {
                        return Err(HairyCompileError{error: HairyParserError::Parser("expected end of style tag"), start: input.len()-i, end: input.len()-i, extra: None, line_context: LineContext::new(input), });
                    }
                }
            },
            EscapeMode::InsideComment => {
                if input[i..].starts_with("-->") {
                    mode = EscapeMode::Normal;
                }
            }
        }
        if mode != last_mode {
            if mode_to_bytes(mode, tag, attribute) != last_mode_bytes {
                last_mode_bytes = mode_to_bytes(mode, tag, attribute);
                retval.push((i+1, last_mode_bytes));
            }
            last_mode = mode;
        }
    }
    if mode != EscapeMode::Normal {
        // FIXME: remember when the last tag/attribute was opened, that is probably a more useful error
        // location for this error
        let message = match mode {
            EscapeMode::Normal => "",
            EscapeMode::InsideTag => "expected to end with correctly closed tag, now 'inside tag' mode",
            EscapeMode::InsideAttributeDoubleQuote => "expected to end with correctly closed tag, now 'inside attribute with doublin quotes' mode",
            EscapeMode::InsideAttributeSingleQuote => "expected to end with correctly closed tag, now 'inside attribute with single quotes' mode",
            EscapeMode::InsideScript => "expected to end with correctly closed tag, now 'inside script' mode",
            EscapeMode::InsideStyle => "expected to end with correctly closed tag, now 'inside style' mode",
            EscapeMode::InsideComment => "expected to end with correctly closed HTML comment, now 'inside HTML comment' mode",
        };
        return Err(HairyCompileError{error: HairyParserError::Parser(message), start: 0, end: 0, extra: None, line_context: LineContext::new(input), });
    }
    Ok(retval.build())
}

fn lookup_escape_mode<'a>(escaping: &'_ [(usize, &'a [u8])], pos: usize) -> &'a [u8] {
    let pp = escaping.partition_point(|x| x.0 <= pos);
    if pp > 0 {
        return escaping[pp-1].1;
    }
    b""
}

/// Converts a Hairy template (in `reader`) to bytecode that can be evaluation later. To help debugging during later evaluation, a
/// `filename` can be added to the bytecode. A MemoryScope is used to reduce the number of
/// allocated strings. The result stays valid as long as the MemoryScope is in the scope (forced by
/// the lifetime system of Rust). The globals and custom functions are used when evaluation the
/// defaults (for template calls and such).
///
/// For an easy to version of this function, see the [`hairy_compile_html`] function (which provide
/// auto escaping automatically).
///
/// ## Escaping
///
/// Normally, all strings that are dynamically outputted (between an evaluating `{{..}}`
/// statement) are automatically 'escaped' using the escaper as given to an argument to the
/// `hairy_eval` function. However, often you will want different escape modes depending on the
/// input context. For example in HTML, output may not normally contain `<` or `>`, but output
/// inside attributes in HTML tags are normally escaped using url-escaping. Although users can specify for
/// every output the escape mode by appending a `:html` or `:url` escaping mode to an expression,
/// this is error prone. Auto escaping is therefore a safer alternative, by automatically dedcing
/// the escape mode from the input. That is where the `escaping` argument comes in to play. That
/// argument is used to lookup the escaping mode for a certain position in the input.
///
/// # Example
///
/// ```
/// use hairy::*;
/// use expry::memorypool::MemoryPool;
/// use std::io::Write;
/// let mut allocator = MemoryPool::new();
/// let mut scope = allocator.rewind();
/// let template = r#"foo {{bar}}"#;
/// let result = hairy_compile(template, "test.tpl", None, ValueRef::new(), &mut scope, None, None, 0);
/// match result {
///   Ok(parsed) => {
///     let value = ValueRef(b"");
///     let output = hairy_eval(parsed.to_ref(), value, &mut scope, None, &[], None);
///     if let Ok(output) = output {
///       std::io::stdout().write_all(&output.concat());
///       // OR, more efficient (avoiding allocations):
///       for v in output {
///         std::io::stdout().write_all(v);
///       }
///     }
///   },
///   Err(err) => {
///     eprintln!("{}", hairy_compile_error_format(template, &err, 0).1);
///   }
/// }
/// ```
pub fn hairy_compile<'a,'b,'c>(reader: &'b str, filename: &'b str, inlines: Option<&'a DecodedValue<'b>>, defaults: ValueRef<'b>, scope: &mut MemoryScope<'c>, mut option_custom: Option<&mut dyn CustomFuncs>, escaping: Option<&'_ [(usize, &'b [u8])]>, extra_line_no: u32) -> Result<BytecodeVec,HairyCompileError<'b>> where 'c: 'b, 'b: 'a {
    let mut custom : &mut dyn CustomFuncs = &mut NoCustomFuncs{};
    if let Some(f) = &mut option_custom {
        custom = *f;
    }
    if reader.len() >= u32::MAX as usize {
        return Err(HairyCompileError{error: HairyParserError::Other("input too large to parse (max 4 GiB)"), start: reader.len(), end: 0usize, extra: None, line_context: LineContext::empty()});
    }
    let context = ParserContext {
        allocator: scope,
        line_context: LineContext::new(reader),
        templates: Vec::new(),
        inlines,
        filename,
        custom,
        escaping,
        input_len: reader.len(),
        extra_line_no,
    };
    let mut parser = HairyParserState::new_with(reader, hairy_tokenize, context);
    let ast = match parser.parse(hairy_top_level, HairyParserError::Parser("unexpected token"), HairyParserError::Parser("max recursion depth of parser reached")) {
        Ok(ast) => ast,
        Err((e,start,end,extra)) => return Err(HairyCompileError{error: e, start, end, extra, line_context: parser.context.line_context}),
    };
    //println!("{:?}", ast);

    (|| {
        let bytecode_length = ast_to_binary_size(&ast);
        let templates_length = templates_to_binary_size(&parser.context().templates);
        let total_length = 4 // from the magic code 'hair'
            + size_of_var_bytes(filename.as_bytes())
            + size_of_var_u64(bytecode_length as u64) + bytecode_length
            + size_of_var_u64(templates_length as u64) + templates_length
            + size_of_var_bytes(defaults.get());
        let mut bytecode: Vec<u8> = vec![0; total_length];
        let mut writer = RawWriter::with(&mut bytecode);
        writer.write_bytes(b"hair")?;
        writer.write_var_bytes(filename.as_bytes())?;
        writer.write_var_u64(bytecode_length as u64)?;
        ast_to_binary(&ast, &mut writer).map_err(|err| { println!("{:?}", err); HairyParserError::Other("error generating bytecode")})?;
        writer.write_var_u64(templates_length as u64)?;
        templates_to_binary(&parser.context().templates, &mut writer).map_err(|err| { println!("{:?}", err); HairyParserError::Other("error generating bytecode")})?;
        writer.write_var_bytes(defaults.get())?;
        Ok(BytecodeVec(bytecode))
    })().map_err(|x| HairyCompileError{error: x, start:0, end:0, extra: None, line_context: parser.context.line_context})
}

pub fn hairy_extract_subtemplates<'a>(template_bytecode: BytecodeRef<'a>) -> Result<&'a [u8],HairyError> {
    let (_template_name, _template_bytecode, new_templates, _defaults) = decode_hairy(template_bytecode.get())?;
    Ok(new_templates)
}

pub fn hairy_compile_error_format(expr: &str, e: &HairyCompileError, extra_line_no: u32) -> (u32, String) {
    let mut retval = String::new();
    let (line_no, prefix, error_msg) = e.line_context.format_error_context(expr, e.start, e.end, extra_line_no).map_or((0u32, String::new(), String::new()), |x| x);
    if line_no > 0 {
        write!(retval, "{}{}error at line {}:{} {}\n{}", prefix, TERM_BRIGHT_RED, line_no, TERM_RESET, e.error, error_msg).ok();
    } else {
        write!(retval, "unknown error, error during hairy_compile_error_format").ok();
    }
    if let Some((start, end)) = e.extra {
        let (line_no, prefix, error_msg) = e.line_context.format_error_context(expr, start, end, extra_line_no).map_or((0u32, String::new(), String::new()), |x| x);
        if line_no > 0 {
            write!(retval, "{}{}related at line {}:{} expected it here (or earlier)\n{}", prefix, TERM_DIM_CYAN, line_no, TERM_RESET, error_msg).ok();
        }
    }
    (line_no, retval)
}

#[derive(Debug)]
pub struct BinExprUntilNewLine {
    prev_was_escape: bool,
    string: bool,
    nested: u32,
    len: usize,
}

impl Spanner for BinExprUntilNewLine {
    fn next(&mut self, b: char) -> bool {
        if self.nested == 0 && !self.string && b == '\n' {
            return false;
        }
        if !self.prev_was_escape && !self.string && (b == '{' || b == '[') {
            self.nested += 1;
        }
        if !self.prev_was_escape && !self.string && (b == '}' || b == ']') {
            if self.nested == 0 {
                return false;
            }
            self.nested -= 1;
        }
        if !self.prev_was_escape && b == '"' {
            self.string = !self.string;
        }
        self.prev_was_escape = !self.prev_was_escape && b == '\\';
        self.len += 1;
        true
    }
    fn valid(&mut self) -> bool {
        self.nested == 0 && !self.string && !self.prev_was_escape
    }
}
impl BinExprUntilNewLine {
    pub fn new() -> Self {
        Self {
            prev_was_escape: false,
            string: false,
            nested: 0,
            len: 0,
        }
    }
}

impl Default for BinExprUntilNewLine {
    fn default() -> Self {
        Self::new()
    }
}


/// The 'easy' interface if you just want to quickly use a HTML template, with auto escaping of the
/// input, and returning a nicely formatted error that can be presented to the user. For more
/// options, see the [`hairy_compile`] function.
///
/// To evaluate, use [`hairy_eval_html`], so the proper escaper is used.
///
/// Note that although `hairy_compile_html` and `hairy_eval_html` are easier to use, they are
/// somewhat slower. For top performance please use other functions.
///
/// The `inline_overrides` argument overrides the inline key-value pairs that can be specified on the top
/// of a template. These keys can contain alphanumeric characters, `_`, and `$`. The values should be
/// valid expry expressions (JSON is a valid expry). These values are compiled using the
/// `inline_overrides` as value, so the inlines and defaults can depend on a (sub)value of `compiled` (so derived inlines and defaults can be generated).
///
/// # Example
///
/// ```
/// use hairy::*;
/// use expry::*;
/// use std::io::Write;
/// let template = r#"foobar = {{=foovar .. barvar}}"#;
/// let result = hairy_compile_html(template, "test.tpl", None, 0);
/// match result {
///   Ok(parsed) => {
///     let value = value!({
///       "foovar": "foo",
///       "barvar": "bar",
///     }).to_vec(false);
///     match hairy_eval_html(parsed.to_ref(), value.to_ref()) {
///       Ok(output) => { std::io::stdout().write_all(&output); },
///       Err(err) => { eprintln!("{}", err); },
///     }
///   },
///   Err(err) => {
///     eprintln!("{}", err);
///   }
/// }
/// ```
pub fn hairy_compile_html<'b>(reader: &'b str, filename: &'b str, inline_overrides: Option<&DecodedValue<'b>>, mut extra_line_no: u32) -> Result<BytecodeVec,String> {
    let mut allocator = MemoryPool::new();
    let mut scope = allocator.rewind();
    let mut custom = NoCustomFuncs{};

    let mut defaults = DecodedObject::new();
    let mut inlines = DecodedObject::new();
    if let Some(DecodedValue::Object(object)) = inline_overrides {
        inlines = object.clone();
    }

    let mut key = "";
    let mut value = "";
    let mut current = reader;
    let mut type_of = "";
    loop {
        let mut embedded_spanner = BinExprUntilNewLine::new();
        if StringParser::new(current).accepts(&["default ", "inline "]).matched(&mut type_of).span(|c: char| -> bool { char::is_ascii_alphanumeric(&c) || c == '$' || c == '_' }, 1).matched(&mut key).accept(':').spanner(&mut embedded_spanner).matched(&mut value).accept('\n').valid(&mut current) {
            extra_line_no += 1;
            let inlines_clone = DecodedValue::Object(inlines.clone());
            let container = if type_of == "default " { &mut defaults } else { &mut inlines };
            let key = key_str(key);
            if container.contains_key(&key) {
                continue;
            }
            let bytecode = expry_compile(value, Some(&inlines_clone), &mut scope);
            match bytecode {
                Ok(bytecode) => {
                    let result = expry_eval(bytecode, ValueRef::new(), &mut scope);
                    match result {
                        Ok(value) => { container.insert(key, value); },
                        Err(error) => {
                            return Err(hairy_compile_error_format(reader, &HairyCompileError{
                                error: HairyParserError::Eval(error),
                                start: current.len() + 1,
                                end: current.len() + 1,
                                extra: None,
                                line_context: LineContext::new(reader),
                            }, 0).1);
                        },
                    }
                },
                Err(error) => {
                    return Err(hairy_compile_error_format(reader, &HairyCompileError{
                        error: HairyParserError::Expr(error.error),
                        start: current.len() + 1 + error.start,
                        end: current.len() + 1 + error.end,
                        extra: error.extra.map(|(start,end)| (current.len() + 1 + start, current.len() + 1 + end)),
                        line_context: LineContext::new(reader),
                    }, 0).1);
                },
            }
        } else if strparse(current).accept("//").split(&mut value, '\n').valid(&mut current) {
            extra_line_no += 1;
        } else {
            break;
        }
    }
    let inlines = DecodedValue::Object(inlines);
    let defaults = if defaults.is_empty() { ValueRef::new() } else { DecodedValue::Object(defaults).to_scope(&mut scope, false) };
    let escaping = parse_html(current, &mut scope).map_err(|err| hairy_compile_error_format(current, &err, extra_line_no).1)?;
    hairy_compile(current, filename, Some(&inlines), defaults, &mut scope, Some(&mut custom), Some(escaping), extra_line_no).map_err(|err| hairy_compile_error_format(current, &err, extra_line_no).1)
}

fn templates_to_binary<E, Out: RawOutput<E>>(templates: &[(&str, &str, ValueRef, Vec<HairyCommand>)], writer: &mut Out) -> Result<(), E> {
    for (name, filename, defaults, body) in templates {
        writer.write_var_bytes(name.as_bytes())?;
        writer.write_var_bytes(filename.as_bytes())?;
        writer.write_var_bytes(defaults.get())?;
        write_with_header(writer, |writer,body_length| {
            writer.write_var_u64(body_length as u64)
        }, |writer| {
            ast_to_binary(body, writer)
        })?;
    }
    Ok(())
}

fn templates_to_binary_size(templates: &[(&str, &str, ValueRef, Vec<HairyCommand>)]) -> usize {
    let mut length_collector = RawWriterLength::new();
    templates_to_binary(templates, &mut length_collector).unwrap_infallible();
    length_collector.length()
}

fn ast_to_binary_size(ast: &[HairyCommand]) -> usize {
    let mut length_collector = RawWriterLength::new();
    ast_to_binary(ast, &mut length_collector).unwrap_infallible();
    length_collector.length()
}

enum HairyBytecode {
    Return = 0,
    Text = 1,
    Evaluate = 2,
    Loop = 4,
    Conditional = 5,
    CallStatic = 6,
    CallDynamicName = 7,
    CallDynamicBody = 8,
}

fn ast_to_binary<E, Out: RawOutput<E>>(ast: &[HairyCommand], writer: &mut Out) -> Result<(), E> {
    for cmd in ast {
        match cmd {
            HairyCommand::Text(text) => {
                writer.write_u8(HairyBytecode::Text as u8)?;
                writer.write_var_bytes(text.as_bytes())?;
            },
            HairyCommand::Conditional(expr, body, else_body, source_context) => {
                writer.write_u8(HairyBytecode::Conditional as u8)?;
                writer.write_var_u64(source_context.0 as u64)?;
                writer.write_var_u64(source_context.1 as u64)?;
                writer.write_var_bytes(expr.get())?;
                write_with_header(writer, |writer,length| writer.write_var_u64(length), |writer| ast_to_binary(body, writer))?;
                write_with_header(writer, |writer,length| writer.write_var_u64(length), |writer| ast_to_binary(else_body, writer))?;
            },
            HairyCommand::Loop(expr, index_variable, content_variable, body, source_context) => {
                writer.write_u8(HairyBytecode::Loop as u8)?;
                writer.write_var_u64(source_context.0 as u64)?;
                writer.write_var_u64(source_context.1 as u64)?;
                writer.write_var_bytes(expr.get())?;
                writer.write_var_bytes(index_variable.as_bytes())?;
                writer.write_var_bytes(content_variable.as_bytes())?;
                write_with_header(writer, |writer,length| writer.write_var_u64(length), |writer| ast_to_binary(body, writer))?;
            },
            HairyCommand::Eval(expr, escape_mode, source_context) => {
                writer.write_u8(HairyBytecode::Evaluate as u8)?;
                writer.write_var_u64(source_context.0 as u64)?;
                writer.write_var_u64(source_context.1 as u64)?;
                writer.write_var_bytes(expr.get())?;
                writer.write_var_bytes(escape_mode)?;
            },
            HairyCommand::CallTemplate(name, expr, source_context) => {
                match name {
                    ResolveTemplate::Static(_) => writer.write_u8(HairyBytecode::CallStatic as u8)?,
                    ResolveTemplate::DynamicName(_) => writer.write_u8(HairyBytecode::CallDynamicName as u8)?,
                    ResolveTemplate::DynamicBody(_) => writer.write_u8(HairyBytecode::CallDynamicBody as u8)?,
                }
                writer.write_var_u64(source_context.0 as u64)?;
                writer.write_var_u64(source_context.1 as u64)?;
                match name {
                    ResolveTemplate::Static(name) => writer.write_var_bytes(name.as_bytes())?,
                    ResolveTemplate::DynamicName(expr) => writer.write_var_bytes(expr.get())?,
                    ResolveTemplate::DynamicBody(expr) => writer.write_var_bytes(expr.get())?,
                }
                writer.write_var_bytes(expr.get())?;
            },
        }
    }
    Ok(())
}

#[derive(Debug)]
pub enum HairyEvalError<'a> {
    Bytecode(&'a str),
    Evaluate(&'a str),
    Error(&'a str),
    EvalTooLong(), // takes too long, probably recusion depth hit
    TemplateNotFound(&'a [u8]),
}

impl<'a> From<EncodingError> for HairyEvalError<'a> {
    fn from(_: EncodingError) -> Self {
        HairyEvalError::Error("hairy template bytecode error")
    }
}
struct Context<'a,'b> {
    custom: &'a mut dyn CustomFuncs,
    templates: Vec<RawReader<'b>>,
    template_name: &'b [u8],
    escaper: &'a mut dyn Escaper<'b>,
}

pub trait Escaper<'e> {
    #[allow(clippy::ptr_arg)]
    fn append_to_output<'b,'c,'d>(&mut self, value: DecodedValue<'b>, escape_mode: &'b [u8], allocator: &mut MemoryScope<'c>, output: &mut Vec<&'d [u8]>) -> Result<(),HairyEvalError<'d>> where 'c: 'd, 'b: 'd, 'e: 'b;
}

/// Default escaper. Defaults to escaping in `html` mode. Currently supports `html`, `url`, and `none`
/// (throws error on other escape modes).
pub struct DefaultEscaper<'e> {
    default_escape_mode: &'e [u8],
}

impl<'e> DefaultEscaper<'e> {
    pub fn new(default_escape_mode: &'e [u8]) -> Self { Self { default_escape_mode } }

    pub fn default() -> Self { Self { default_escape_mode: b"",  } }
}

impl<'e> Escaper<'e> for DefaultEscaper<'e> {
    fn append_to_output<'b,'c,'d>(&mut self, value: DecodedValue<'b>, mut escape_mode: &'b [u8], allocator: &mut MemoryScope<'c>, output: &mut Vec<&'d [u8]>) -> Result<(),HairyEvalError<'d>> where 'c: 'd, 'b: 'd, 'e: 'b {
        if escape_mode.is_empty() {
            escape_mode = self.default_escape_mode;
        }
        if escape_mode == b"sjs" || escape_mode == b"js" {
            if !value.is_valid_json() {
                return Err(HairyEvalError::Error("UTF-8 problem in js mode: maybe non-text is outputted in a key or a string?"));
            }
            let mut json = write!(allocator, "{}", value).as_bytes();
            if escape_mode == b"sjs" {
                json = allocator.copy_with_replacement(json, html_escape_outside_attribute_u8)?;
            }
            output.push(json);
            return Ok(());
        }
        match value {
            DecodedValue::Null() => Ok(()),
            DecodedValue::Int(i) => {
                output.push(write!(allocator, "{}", i).as_bytes());
                Ok(())
            },
            DecodedValue::Float(f) => {
                // The output is improved for humans. It forces to convert 0.99999998f to 1.0
                let mut out = if escape_mode == b"unrounded" { write!(allocator, "{}", f) } else { write!(allocator, "{:.6}", f) };
                // strip of zero's of the end
                out = &out[0..out.rfind(|x| x != '0').map_or(out.len(), |x| if out.as_bytes()[x] == b'.' { x+2 } else { x+1 })];
                output.push(out.as_bytes());
                Ok(())
            },
            DecodedValue::Double(d) => {
                // The output is improved for humans. It forces to convert 0.999999999999996 to 1.0
                let mut out = if escape_mode == b"unrounded" { write!(allocator, "{}", d) } else { write!(allocator, "{:.14}", d) };
                // strip of zero's of the end
                out = &out[0..out.rfind(|x| x != '0').map_or(out.len(), |x| if out.as_bytes