razeri_plugin_contract/

sid_spelling.rs

// Copyright (C) 2026 Gabriel Shea
// Licensed under either of the Apache License, Version 2.0 or the MIT
// license, at your option.

//! **How this engine spells a surrogate key** — the one thing about the key a
//! dialect decides.
//!
//! Razeri derives every surrogate key one way:
//!
//! ```text
//! guard( hash( normalize(part₁) ‖ '|' ‖ … ‖ normalize(partₙ) ) )
//! ```
//!
//! That derivation is Razeri's and is composed in one place. What varies
//! between engines is only the **vocabulary** — what this engine calls `LOWER`,
//! what it calls `TRIM`, how it joins strings, how it casts a value to text,
//! and how it spells a 64-bit hash. [`SidSpelling`] is that vocabulary, and a
//! dialect states it by overriding
//! [`SqlRenderer::sid_spelling`](crate::SqlRenderer::sid_spelling).
//!
//! Two expressions built from this struct differ **only** in these fields,
//! which is what makes *"provably the same key on every engine"* a proof
//! rather than an assumption.
//!
//! # If your engine has no 64-bit hash
//!
//! Say so rather than reaching for the nearest-looking function. A hash that is
//! deterministic *within* one warehouse and equal to nothing outside it is the
//! exact lock-in a portable key exists to end — a warehouse keyed with it
//! reconciles against nothing. The honest answers are an artifact your dialect
//! installs (name it in [`SidSpelling::hash`] and emit its definition from your
//! prerequisites), or a refusal that says what the reader should do instead.

/// The [`SidSpelling::platform`] of a dialect that has not stated its spelling.
///
/// The default [`SidSpelling`] is ANSI's vocabulary, which is a **guess** and
/// not a claim about any platform. It has to exist — a dialect must render
/// *something* — and it has to be distinguishable from the agnostic renderer's
/// answer, which is ANSI on purpose. Otherwise an unspelled platform gets ANSI
/// everywhere, every producer agrees, and Razeri's cross-checks pass over a
/// platform nobody has spelled: a check that cannot see its own blind spot.
///
/// **A dialect that leaves this value in place is telling Razeri it has not
/// decided**, and the conformance kit reports it as an unanswered obligation.
pub const UNKNOWN_PLATFORM: &str = "unknown";

/// How a dialect spells the canonical lowercase.
///
/// The canonical lowercase is fixed as **locale-independent simple case
/// mapping, one code point to one code point, with no combining mark introduced
/// by the operation**, and it renders as the engine's own `LOWER` over a value
/// whose two divergent code points have already been substituted away:
///
/// ```text
/// <function>(<replace>(<replace>(x, 'İ', 'I'), 'Σ', 'σ'))
/// ```
///
/// # Why the substitution is unconditional
///
/// Whether an engine's `LOWER` is the simple mapping or the **full** one is not
/// a property of the engine. It is a property of the engine **and its collation
/// provider**, and the provider is chosen when a database is created — so the
/// same server answers both ways. Measured on one `PostgreSQL` 17.9 instance:
/// under a libc collation `lower('İ')` is a bare `i`, and in a database created
/// with the ICU provider it is `i` + U+0307 and `ΟΔΟΣ` lowercases with a final
/// `ς`. Classifying the engine therefore classifies the wrong thing, and the
/// engine that is classified wrong keys one value to two different numbers.
///
/// So the substitution is applied **everywhere**, which costs nothing because
/// it is a no-op wherever the lowercase is already simple: `'İ'` → `'I'`
/// lowercases to `i` either way, and `'Σ'` → `'σ'` is already lowercase and is
/// left alone. That makes the composed key **collation-independent** rather
/// than correct-on-the-configurations-somebody-tried.
///
/// # The two code points, and why they are two facts
///
/// - **`İ` (U+0130) — the unconditional divergence.** Its full lowercase adds a
///   combining mark the canonical mapping excludes by name (`i` followed by
///   U+0307), and it is the **only** code point whose simple and full mappings
///   differ with no context at all. Replacing it with `I` first makes the
///   composition exactly the simple mapping;
/// - **`Σ` (U+03A3) — the conditional divergence.** Its full lowercase is `σ`
///   *except* word-finally, where Unicode's `Final_Sigma` condition makes it
///   `ς` (U+03C2). No enumeration of unconditional mappings can see this: a
///   different quantifier hides a different letter. Replacing every `Σ` with an
///   already-lowercase `σ` first removes the context the rule reads, and
///   `LOWER` then leaves it alone.
///
/// Note the direction: `ς` is never folded into `σ`. Its own simple lowercase
/// is itself, so a substitution that merged the two would be a *third*
/// normalization rather than a repair.
///
/// # What a dialect has to answer
///
/// Two function names, and — where the engine matches a replacement pattern by
/// something other than the code point — how to pin that match. An engine whose
/// replacement is collation-sensitive is the one to watch: under a
/// case-insensitive collation `REPLACE(x, 'Σ', 'σ')` matches `ς` as well, and
/// silently performs the third normalization. Measured on SQL Server 2022 with
/// its default collation, cured by [`Lower::match_collation`].
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Lower {
    /// The platform's lowercase function, in its own casing — `LOWER` on most
    /// engines, `lowerUTF8` on one.
    pub function: &'static str,
    /// The platform's three-argument string-replacement function — `REPLACE` on
    /// most engines, `replaceAll` on one.
    pub replace: &'static str,
    /// **How this engine spells a Unicode string literal** — what goes in front
    /// of the opening quote, where that differs from an ordinary literal. Empty
    /// on every engine whose ordinary literal is already Unicode; `"N"` on
    /// T-SQL, where a bare literal is narrowed to the database's code page and
    /// the two Greek letters do not survive it.
    ///
    /// The substitution's two patterns are the reason the field exists, and they
    /// are not the only literal that has to survive: the equality gate
    /// substitutes probe values where a column reference stands, and a bare
    /// `'ΟΔΟΣ'` reaches a T-SQL hash as `???S`. Anything that writes one of the
    /// corpus's values into this engine as a literal reads this field.
    pub pattern_literal_prefix: &'static str,
    /// The collation the substitution's subject is pinned to, where this engine
    /// matches a replacement pattern **under the subject's collation** rather
    /// than by code point.
    ///
    /// `None` on every engine whose replacement is already by code point.
    /// Name a **binary** collation where it is not: under an accent- or
    /// case-insensitive one, replacing `Σ` also replaces `ς`, which moves a
    /// Greek key that was already correct.
    pub match_collation: Option<&'static str>,
}

/// How a dialect casts one call-site argument to text.
///
/// The hash is a hash **of a string** and a call-site argument is a raw column
/// of its own declared type, so the cast is not optional.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum TextCast {
    /// `CAST(x AS <type>)` — the cast propagates NULL, so Razeri's NULL
    /// sentinel is reached with the statement intact.
    Direct(&'static str),
    /// `assumeNotNull(CAST(x AS Nullable(<type>)))` — `ClickHouse`, whose
    /// `CAST` to a non-`Nullable` type **raises** instead of propagating
    /// (*"Cannot convert NULL to a non-nullable type"*).
    ///
    /// `ClickHouse` evaluates both arms of a `CASE`, so the sentinel cannot
    /// save the statement: the emitted key expression failed outright on any
    /// NULL natural-key part rather than returning the sentinel. That is not a
    /// wrong key — it is a dimension load that does not run. The wrapper makes
    /// the unreachable arm total; which value it would produce is irrelevant,
    /// because the sentinel has already answered.
    ClickHouseNullTolerant(&'static str),
}

/// How a dialect spells `TRIM`.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Trim {
    /// `TRIM(x)`, in the platform's own casing.
    Function(&'static str),
    /// `LTRIM(RTRIM(x))` — T-SQL, whose `TRIM` requires SQL Server 2017. The
    /// pair removes the same leading and trailing spaces.
    LtrimRtrim,
    /// `TRIM(x, ' ')` — the engines whose **bare** `TRIM` removes every
    /// whitespace character rather than the ASCII space alone.
    ///
    /// The canonical trim is the ASCII space (`U+0020`) and nothing else, which
    /// is why a tab-padded and a newline-padded probe are in the corpus and are
    /// *not* expected to normalize away. `BigQuery` is the measured member:
    /// `LENGTH(TRIM('\tBoston\t'))` is 6 there — the tab gone and the key moved
    /// — while `LENGTH(TRIM('\tBoston\t', ' '))` is 8, and a non-breaking space
    /// survives the second form and not the first.
    ///
    /// The explicit set is not a repair of the engine; it is the canonical
    /// clause written in an engine whose default set is wider. `TRIM(x, ' ')`
    /// and `TRIM(x)` are the same function on every platform whose bare form
    /// already strips the space alone, so the variant states a difference
    /// rather than introducing one.
    ///
    /// **Only a tab-padded probe on your engine tells you which variant is
    /// yours** — run the corpus on the engine itself. An emulator can give the
    /// opposite answer: the community `BigQuery` emulator strips the space
    /// alone.
    FunctionWithSpaceSet(&'static str),
}

/// How a dialect joins the normalized parts of a composite key.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Concat {
    /// `CONCAT(a, '|', b)`, in the platform's own casing.
    Function(&'static str),
    /// `CONCAT(CONCAT(a, '|'), b)` — the engines whose `CONCAT` takes
    /// **exactly two arguments**, in the platform's own casing.
    ///
    /// Rendered by nesting pairwise, left to right, one call per pair: three
    /// parts read `CONCAT(CONCAT(a, b), c)` where a variadic engine reads
    /// `CONCAT(a, b, c)`. Db2 is the engine that needs it, and older Oracle
    /// releases are the same shape.
    ///
    /// The distinction is not cosmetic. On such an engine the variadic
    /// spelling is not a wider call to the same function, it is a call to a
    /// function that is not there, so the first composite key stops the load
    /// rather than reading oddly in review.
    ///
    /// The nesting joins the same parts with the same separator in the same
    /// order, so the string that reaches the hash — and therefore the key — is
    /// the one every other engine hashes. What differs is only how many calls
    /// it takes to spell it.
    FunctionBinary(&'static str),
    /// `a + '|' + b` — T-SQL's string operator. Equal to `CONCAT` for non-NULL
    /// operands, which Razeri's NULL sentinel is what guarantees: `+`
    /// propagates NULL where `CONCAT` swallows it, and a swallowed NULL
    /// silently collapses `('boston', NULL)` and `('boston', '')` into one
    /// member.
    Plus,
    /// `a || '|' || b` — the engines that join strings with an operator, in
    /// the operator's own spelling. `||` is the standard's, and the one an
    /// engine with no `CONCAT` function usually takes.
    ///
    /// Rendered left to right, the operator between each part and the
    /// separator: `Operator("||")` over three parts reads
    /// `a || '|' || b || '|' || c`. One expression, whatever the number of
    /// parts, and no function the warehouse has to hold.
    ///
    /// Like [`Plus`](Self::Plus), the operator propagates NULL where `CONCAT`
    /// swallows it, and Razeri's NULL sentinel is what makes that safe: a key
    /// with a NULL part is answered before the join is reached.
    Operator(&'static str),
}

/// **How a column's `clean:` step changes the case of a letter** — on this
/// engine, for the twenty-six ASCII letters and nothing else.
///
/// A fold is never the engine's `LOWER` or `UPPER`: what they do to a letter
/// outside `a` to `z` belongs to the database's collation, so the same value
/// would be stored two ways on two databases. The default spells the fold in
/// plain SQL that no collation reads, twenty-six nested replacements. An engine
/// that holds a function doing exactly that fold names it here, and the step is
/// one call.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum AsciiFold {
    /// `REPLACE(… REPLACE(x, 'a', 'A') …, 'z', 'Z')`: one replacement per
    /// letter, written with [`Lower::replace`] over the value pinned to
    /// [`Lower::match_collation`]. Every engine with a three-argument replace
    /// can run it, which is why it is the default.
    ///
    /// The nesting is twenty-six calls deep, and a parser can refuse that depth
    /// inside some statements. `SQLite` 3.45 is the measured case: a
    /// `parser stack overflow` inside an `UPDATE` or an `INSERT … SELECT`,
    /// while the same expression runs in a plain `SELECT`.
    Replacements,
    /// One call to a function the engine holds: `to_upper(x)` changes `a` to
    /// `z` into `A` to `Z`, `to_lower(x)` changes `A` to `Z` into `a` to `z`,
    /// and both leave every other character as it is.
    ///
    /// The function is the dialect's to install, from its prerequisites or an
    /// extension it ships. It must answer exactly what the replacements
    /// answer for every value, including a letter outside ASCII, which it
    /// leaves alone; measure it against them before you ship it.
    Functions {
        /// The function that folds to capitals.
        to_upper: &'static str,
        /// The function that folds to small letters.
        to_lower: &'static str,
    },
}

/// One platform's vocabulary for the surrogate-key derivation — the only thing
/// that varies between engines.
#[derive(Debug, Clone, Copy)]
pub struct SidSpelling {
    /// The name this dialect answers under. [`UNKNOWN_PLATFORM`] means the
    /// dialect has not stated a spelling and is taking ANSI's guess.
    pub platform: &'static str,
    /// The canonical lowercase — this engine's own `LOWER` and the
    /// substitution that runs in front of it.
    pub lower: Lower,
    /// The `TRIM` function, or the pair that stands in for it.
    pub trim: Trim,
    /// The join used for composite keys.
    pub concat: Concat,
    /// The platform's hash of an already-normalized text expression, as the
    /// text before and after it. `("CAST(xxHash64(", ") AS Int64)")` reads
    /// `CAST(xxHash64(x) AS Int64)`.
    pub hash: (&'static str, &'static str),
    /// The CAST that turns one argument into text. The hash is a hash OF A
    /// STRING, and at the inline call sites the argument is a raw column of its
    /// own declared type — `TRIM(DATE)` is a binder error, not a value.
    pub cast_type: TextCast,
    /// Whether the load path reaches the hash through Razeri's emitted helper
    /// function instead of inlining it. Inlining is forced where a UDF cannot
    /// call another UDF (`DuckDB` macros, `ClickHouse` lambdas) and chosen
    /// where the platform's own function is already the derivation.
    pub through_helper: bool,
    /// How a column's `clean:` step changes the case of a letter here — the
    /// twenty-six replacements, or one function the engine holds.
    pub ascii_fold: AsciiFold,
}

/// **ANSI's vocabulary** — the spelling a dialect inherits until it states its
/// own, and the agnostic renderer's answer on purpose.
///
/// A dialect building its own arm should start from this and change only the
/// fields its engine actually spells differently:
///
/// ```
/// use razeri_plugin_contract::{ANSI_SID_SPELLING, SidSpelling, TextCast};
///
/// const MY_ENGINE: SidSpelling = SidSpelling {
///     platform: "my-engine",
///     hash: ("my_xxhash64(", ")"),
///     cast_type: TextCast::Direct("STRING"),
///     through_helper: false,
///     ..ANSI_SID_SPELLING
/// };
/// ```
pub const ANSI_SID_SPELLING: SidSpelling = SidSpelling {
    platform: "ansi",
    lower: Lower {
        function: "LOWER",
        replace: "REPLACE",
        pattern_literal_prefix: "",
        match_collation: None,
    },
    trim: Trim::Function("TRIM"),
    concat: Concat::Function("CONCAT"),
    hash: ("xxhash64(", ")"),
    cast_type: TextCast::Direct("TEXT"),
    through_helper: true,
    ascii_fold: AsciiFold::Replacements,
};

/// **The language a surrogate-key helper's body is written in**, where that
/// body is not SQL.
///
/// The two named languages are the ones Razeri's function renderers already
/// spell a `LANGUAGE` clause for. Anything else is [`Named`](Self::Named),
/// which carries the keyword through unchanged.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum SidHelperLanguage {
    /// JavaScript.
    JavaScript,
    /// Python.
    Python,
    /// A language keyword this contract does not name, spelled exactly as the
    /// engine's `LANGUAGE` clause takes it.
    Named(&'static str),
}

/// **The whole body of the surrogate-key helper, supplied by the dialect.**
///
/// Razeri normally builds the helper's body itself: it is the one derivation,
/// composed from this engine's [`SidSpelling`] in this engine's own SQL. A
/// dialect answers with this type only when that is impossible — the engine has
/// no 64-bit hash in any SQL spelling, so there is no vocabulary to compose
/// from and the definition has to be written in another language.
///
/// Answering here does **not** change the key. The body must compute XXH64 with
/// seed 0 over the UTF-8 bytes of its single already-normalized argument, and
/// return it as a signed 64-bit integer — the same number every other engine
/// returns for the same input. A body that computes something else keys a
/// warehouse to numbers nothing else reproduces, which is the lock-in a
/// portable key exists to end. Measure it against the canonical number before
/// you ship it.
///
/// The body is the function's inner text only: Razeri writes the `CREATE
/// FUNCTION` envelope, the parameter, the return type and the `LANGUAGE`
/// clause around it.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct SidHelperBody {
    /// What the body is written in.
    pub language: SidHelperLanguage,
    /// The body itself.
    pub body: &'static str,
}

/// The spelling a dialect that has decided nothing renders — ANSI's vocabulary
/// under the name that says it is a guess.
pub const UNSPELLED_SID: SidSpelling =
    SidSpelling { platform: UNKNOWN_PLATFORM, ..ANSI_SID_SPELLING };

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn the_unspelled_default_differs_from_ansi_only_in_its_name() {
        assert_eq!(UNSPELLED_SID.platform, UNKNOWN_PLATFORM);
        assert_eq!(UNSPELLED_SID.hash, ANSI_SID_SPELLING.hash);
        assert_eq!(UNSPELLED_SID.lower, ANSI_SID_SPELLING.lower);
        assert_eq!(UNSPELLED_SID.trim, ANSI_SID_SPELLING.trim);
        assert_eq!(UNSPELLED_SID.concat, ANSI_SID_SPELLING.concat);
        assert_eq!(UNSPELLED_SID.cast_type, ANSI_SID_SPELLING.cast_type);
        assert_eq!(UNSPELLED_SID.through_helper, ANSI_SID_SPELLING.through_helper);
        assert_eq!(UNSPELLED_SID.ascii_fold, ANSI_SID_SPELLING.ascii_fold);
    }
}