Skip to main content

evalhub_query/
typecheck.rs

1//! The schema-derived path table and the type check.
2//!
3//! [`PathTable`] is built once per process from the committed JSON Schema in
4//! `evalhub_schema` plus the currently `applied` `ext_schemas` from the
5//! registry. It maps each legal query path to where its value lives, what
6//! type it has, and which operators the storage can actually serve:
7//!
8//! | Path form                          | Type                 | Served by                | Operators |
9//! | ---------------------------------- | -------------------- | ------------------------ | --------- |
10//! | `{facet}.{key}` with a column      | from the schema leaf | generated column         | all that suit the type |
11//! | `{facet}.{key}` without one        | from the schema leaf | GIN index on `body`      | `eq`, `exists` |
12//! | `fingerprint.{facet}`              | string               | `fingerprints` table     | `eq`, `ne`, `in`, `exists` |
13//! | `results`, `relations`, `attachments` | array of object   | own tables               | `any` |
14//! | `results[{metric}].value` (sort)   | number               | `results` table          | sort only |
15//! | `ext.{ns}.{key}` registered        | from the ext_schema  | expression index         | all that suit the type |
16//! | `ext.{ns}.{key}` unregistered      | unknown              | GIN index on `body`      | `eq`, `exists` |
17//! | `title`, `producer.name`, …        | string               | generated column         | all that suit the type |
18//!
19//! The row that is easy to get wrong is the second one. Only sixteen facet
20//! keys have a generated column (the migration lists them, and
21//! [`GENERATED_COLUMNS`] mirrors that list with a test that fails if the two
22//! disagree). Every other key of the record is readable and equality-
23//! searchable through the GIN index over `body`, but ordering or ranging on
24//! it would be a sequential scan, so the check refuses with `not_indexed`
25//! and names the registry. That is the same answer an unregistered `ext`
26//! path gets, and for the same reason.
27//!
28//! `runs` is deliberately absent: an Eval's runs are rows of their own, not
29//! a key of the header's body, so a record query cannot reach them. A
30//! question about runs is asked of the run projection, which has its own
31//! table (below).
32//!
33//! The check walks the parsed tree and, for each leaf, looks the path up,
34//! confirms the operator is legal for the type, confirms the literal is of
35//! that type, and confirms the operator is served by the path's index.
36//! Failures are collected into `422 { errors[] }` with codes
37//! `unknown_path`, `type_mismatch`, `not_indexed`, each carrying a JSON
38//! pointer into the submitted request.
39//!
40//! The table is rebuilt when an `ext_schema` transitions to `applied`; the
41//! server holds it behind an `ArcSwap`-style handle so a query in flight
42//! sees a consistent table.
43//!
44//! # The run projection's table
45//!
46//! `GET /evals/{ns}/{name}/runs` reads one row per run of one Eval, joined
47//! with the `run_results` of the Cards the request names. Its table is
48//! [`PathTable::for_runs`], built by hand rather than from a record
49//! schema, because a row of the projection is not a record: it is a run's
50//! columns, its side tables, and a join.
51//!
52//! | Path form                              | Type                 | Resolves to                     |
53//! | -------------------------------------- | -------------------- | ------------------------------- |
54//! | `run_id`, `status`, `error.kind`       | string               | a column of `runs`              |
55//! | `started_at`, `ended_at`               | string (RFC 3339)    | a column of `runs`              |
56//! | `{facet}.{key}` (`model.id`, …)        | from the schema leaf | the run's body                  |
57//! | `fingerprint.{facet}`                  | string               | `run_fingerprints`              |
58//! | `metrics[{ns}/{name}]`                 | number               | `run_metrics`                   |
59//! | `results[{card}][{metric}].value`      | number               | the Card's `run_results`        |
60//! | `results[{card}][{metric}].label`      | string               | the Card's `run_results`        |
61//! | `ext.{ns}.{key}`                       | as for records       | the run's body                  |
62//!
63//! - The facet keys are the Eval header's: the leaves of the six Eval
64//!   facets in `RecordKind::Eval.schema()` (a run carries the same facet
65//!   types, `evalhub_schema::run`). There is no `grading`.
66//! - `metrics[…]` takes any well-formed metric id (`{ns}/{name}`, the
67//!   registry id rule); the registry is not consulted, because a run's
68//!   metrics are accepted whether or not the registry knows them, and so
69//!   they must be searchable the same way. A malformed id is
70//!   `unknown_path`.
71//! - `results[{card}]…` exists only for the Cards named in the request
72//!   ([`CardRef`], `{ns}/{name}`). A Card the request did not name is
73//!   `unknown_path`, with a hint to name it: joining a Card the caller did
74//!   not ask for would change the projection behind the caller's back.
75//!   Which Cards a caller may name at all is the server's decision, made
76//!   before the table is built.
77//! - Parameters use the bracket form the record table's
78//!   `results[{metric}].value` sort key uses. The dotted forms
79//!   (`metrics.core/tokens_out`, `results.alice/g.core/pass.value`) are
80//!   refused with a hint naming the bracket form: a slug may contain `.`,
81//!   so a dotted path cannot say where a parameter ends. The bracket form
82//!   needs no escaping because ids and Card references are made of
83//!   `[a-z0-9._-]` and one `/`, never `[` or `]`.
84//! - `ext.*` comes through the same [`PathTable::with_ext`] as for records;
85//!   in the projection its paths are below the run's own `ext`.
86//! - There are no array paths and so no `any`: a run's `attachments[]` and
87//!   `artifacts[]` are not searchable here, and a Card's judgements are
88//!   addressed by Card and metric rather than ranged over.
89//!
90//! **Index awareness is different here.** Every path of the projection is
91//! served in full (every operator its type allows, and sortable), facet
92//! keys included, although the run's facets have no generated column. The
93//! record table refuses ranges on such keys because a record query spans
94//! every record of the hub, and a range the index cannot answer grows with
95//! the hub. The projection is always scoped to one Eval (`runs` is keyed
96//! by `(record, run_id)`), so the worst case is a scan of one Eval's runs,
97//! which the page limit and the `runs:batch` limits bound. An `ext` key
98//! with no registered schema still answers only `eq` and `exists`, not for
99//! the index but for the type: nothing declares what it holds, so a range
100//! on it has no meaning.
101
102use std::collections::BTreeMap;
103
104use evalhub_core::validate::is_valid_id;
105use evalhub_schema::RecordKind;
106use evalhub_schema::error::{ErrorCode, ErrorEntry};
107use evalhub_schema::query::{self as envelope, QueryRequest};
108use serde_json::Value;
109
110use crate::grammar::{self, Filter, MatchTerm, ScalarOp};
111use crate::ir::{self, CardRef};
112
113/// The facets, in the order the record types declare them. `grading`
114/// belongs to a Card only, which [`RecordKind`] decides.
115const CARD_FACETS: [&str; 7] = [
116    "model",
117    "task",
118    "harness",
119    "generation",
120    "trial",
121    "grading",
122    "env",
123];
124
125/// The generated columns of `versions`, as JSON path and column name.
126///
127/// This mirrors `crates/evalhub-store/migrations/0001_init.sql`. It is the
128/// one list in this crate that is not projected from the schema, because
129/// the schema does not know which keys the migration chose to materialise.
130/// A test parses the migration and fails if the two ever disagree, so the
131/// duplication cannot rot silently.
132pub const GENERATED_COLUMNS: &[(&[&str], &str)] = &[
133    (&["title"], "title"),
134    (&["producer", "name"], "producer_name"),
135    (&["model", "id"], "model_id"),
136    (&["model", "provider"], "model_provider"),
137    (&["model", "revision"], "model_revision"),
138    (&["task", "id"], "task_id"),
139    (&["task", "version"], "task_version"),
140    (&["task", "split"], "task_split"),
141    (&["harness", "name"], "harness_name"),
142    (&["harness", "version"], "harness_version"),
143    (&["generation", "temperature"], "gen_temperature"),
144    (&["generation", "top_p"], "gen_top_p"),
145    (&["generation", "max_tokens"], "gen_max_tokens"),
146    (&["trial", "k"], "trial_k"),
147    (&["env", "git", "commit"], "env_git_commit"),
148    (&["env", "git", "dirty"], "env_git_dirty"),
149];
150
151/// How much of the operator set a path's storage can serve.
152#[derive(Debug, Clone, Copy, PartialEq, Eq)]
153pub enum Indexed {
154    /// A dedicated index: every operator the type allows.
155    Full,
156    /// The GIN index over `body`, which answers containment and
157    /// existence and nothing else. The path's type is known from the
158    /// schema, so a literal of the wrong type is still refused.
159    EqExistsOnly,
160    /// The GIN index over `body`, for a path whose type nobody has
161    /// declared: an `ext` key with no registered schema.
162    ///
163    /// Containment answers equality on any JSON scalar, and the hub has
164    /// no ground to call one of them wrong here — there is no schema
165    /// saying what the key holds. Registering an `ext_schema` is what
166    /// turns the guess into a declaration, and with it both the type
167    /// check and the ranges.
168    EqExistsUntyped,
169}
170
171/// What a legal query path denotes.
172///
173/// The directive's `{ column, ty, indexed }` shape covers scalars; array
174/// paths carry no column of their own, because an `any` condition ranges
175/// over a side table whose columns are named by its terms. Keeping the two
176/// apart is what lets the check reject `{"path": "results", "op": "eq"}`
177/// with a type error rather than a confusing column error.
178#[derive(Debug, Clone, PartialEq)]
179pub enum PathInfo {
180    /// A single value.
181    Scalar {
182        /// Where it lives.
183        column: ir::Column,
184        /// What it is.
185        ty: ir::ValueType,
186        /// What the storage can do with it.
187        indexed: Indexed,
188    },
189    /// An array of objects, searchable with `any`.
190    Array {
191        /// The side table to range over.
192        table: ir::ArrayTable,
193    },
194}
195
196/// A registered extension schema, as the registry holds it.
197#[derive(Debug, Clone, PartialEq)]
198pub struct ExtSchema {
199    /// The namespace that owns it, `{ns}/{name}`.
200    pub ns: String,
201    /// Typed paths below `ext.{ns}`, each as its key segments.
202    pub paths: Vec<(Vec<String>, ir::ValueType)>,
203    /// Whether the store has finished building the expression indexes.
204    /// Until it has, the paths behave as unregistered.
205    pub applied: bool,
206}
207
208/// The set of legal query paths with their types and index status,
209/// projected from the record schema and the applied `ext_schemas`, or
210/// built for the run projection by [`PathTable::for_runs`].
211#[derive(Debug, Default, Clone)]
212pub struct PathTable {
213    paths: BTreeMap<String, PathInfo>,
214    target: Target,
215}
216
217/// What a table's paths address.
218#[derive(Debug, Default, Clone, PartialEq, Eq)]
219enum Target {
220    /// Versions of a record kind; [`compile`] reads it.
221    #[default]
222    Record,
223    /// The runs of one Eval, joined with these Cards; [`compile_runs`]
224    /// reads it.
225    Runs {
226        /// The Cards named in the request, in request order, deduplicated.
227        cards: Vec<CardRef>,
228    },
229}
230
231impl PathTable {
232    /// Build the table for one record kind from its committed schema.
233    pub fn from_schema(kind: RecordKind) -> Self {
234        let mut paths = BTreeMap::new();
235        let schema = kind.schema().to_value();
236
237        // Root scalars that are not facets.
238        let root_props = schema
239            .get("properties")
240            .and_then(Value::as_object)
241            .cloned()
242            .unwrap_or_default();
243        let defs = schema.get("$defs").cloned().unwrap_or(Value::Null);
244
245        for (name, prop) in &root_props {
246            match name.as_str() {
247                // Facets and other objects are walked below; the open `ext`
248                // map is contributed by the registry, not by the schema.
249                "ext" => {}
250                "results" => {
251                    paths.insert(
252                        name.clone(),
253                        PathInfo::Array {
254                            table: ir::ArrayTable::Results,
255                        },
256                    );
257                }
258                "relations" => {
259                    paths.insert(
260                        name.clone(),
261                        PathInfo::Array {
262                            table: ir::ArrayTable::Relations,
263                        },
264                    );
265                }
266                "attachments" => {
267                    paths.insert(
268                        name.clone(),
269                        PathInfo::Array {
270                            table: ir::ArrayTable::Attachments,
271                        },
272                    );
273                }
274                _ => {
275                    let mut leaves = Vec::new();
276                    collect_leaves(&mut leaves, &defs, prop, std::slice::from_ref(name));
277                    for (segments, ty) in leaves {
278                        let column = GENERATED_COLUMNS
279                            .iter()
280                            .find(|(json_path, _)| json_path == &segments.as_slice())
281                            .map(|(_, name)| ir::Column::Generated((*name).to_string()));
282                        let (column, indexed) = match column {
283                            Some(column) => (column, Indexed::Full),
284                            None => (ir::Column::Body(segments.clone()), Indexed::EqExistsOnly),
285                        };
286                        paths.insert(
287                            segments.join("."),
288                            PathInfo::Scalar {
289                                column,
290                                ty,
291                                indexed,
292                            },
293                        );
294                    }
295                }
296            }
297        }
298
299        // Fingerprints are the hub's own facts, not a key of the record.
300        for facet in facets(kind) {
301            paths.insert(
302                format!("fingerprint.{facet}"),
303                PathInfo::Scalar {
304                    column: ir::Column::Fingerprint((*facet).to_string()),
305                    ty: ir::ValueType::String,
306                    indexed: Indexed::Full,
307                },
308            );
309        }
310
311        Self {
312            paths,
313            target: Target::Record,
314        }
315    }
316
317    /// Build the table of the run projection (`GET /evals/{ns}/{name}/runs`)
318    /// for a request that names `cards`.
319    ///
320    /// The fixed paths are the run's columns, the leaves of the six Eval
321    /// facets (read from `RecordKind::Eval.schema()`, the header's schema,
322    /// whose facet types a run shares) and `fingerprint.{facet}`. The
323    /// parametric paths, `metrics[{ns}/{name}]` and
324    /// `results[{card}][{metric}].value` / `.label`, are resolved by
325    /// [`PathTable::lookup`] rather than listed, since metric ids are an
326    /// open set; `results` resolves only for a Card in `cards`. Fold the
327    /// registry in with [`PathTable::with_ext`], as for records. See the
328    /// module doc for the table and why every path is served in full.
329    ///
330    /// A Card named twice is kept once, at its first position. The table
331    /// is cheap to build (a few dozen entries) and is meant to be built per
332    /// request, since `cards` is per request.
333    pub fn for_runs(cards: &[CardRef]) -> Self {
334        use ir::{RunColumn as R, ValueType as V};
335
336        let mut paths = BTreeMap::new();
337        let mut scalar = |path: &str, column: R, ty: V| {
338            paths.insert(
339                path.to_string(),
340                PathInfo::Scalar {
341                    column: ir::Column::Run(column),
342                    ty,
343                    indexed: Indexed::Full,
344                },
345            );
346        };
347        scalar("run_id", R::RunId, V::String);
348        scalar("status", R::Status, V::String);
349        scalar("error.kind", R::ErrorKind, V::String);
350        scalar("started_at", R::StartedAt, V::String);
351        scalar("ended_at", R::EndedAt, V::String);
352
353        let facets = facets(RecordKind::Eval);
354        let schema = RecordKind::Eval.schema().to_value();
355        let defs = schema.get("$defs").cloned().unwrap_or(Value::Null);
356        let root_props = schema
357            .get("properties")
358            .and_then(Value::as_object)
359            .cloned()
360            .unwrap_or_default();
361        for facet in facets {
362            if let Some(prop) = root_props.get(*facet) {
363                let mut leaves = Vec::new();
364                collect_leaves(&mut leaves, &defs, prop, &[(*facet).to_string()]);
365                for (segments, ty) in leaves {
366                    let path = segments.join(".");
367                    scalar(&path, R::Facet(segments), ty);
368                }
369            }
370            scalar(
371                &format!("fingerprint.{facet}"),
372                R::Fingerprint((*facet).to_string()),
373                V::String,
374            );
375        }
376
377        let mut named: Vec<CardRef> = Vec::with_capacity(cards.len());
378        for card in cards {
379            if !named.contains(card) {
380                named.push(card.clone());
381            }
382        }
383        Self {
384            paths,
385            target: Target::Runs { cards: named },
386        }
387    }
388
389    /// The Cards a run projection table was built for, in request order;
390    /// empty for a record table.
391    pub fn cards(&self) -> &[CardRef] {
392        match &self.target {
393            Target::Record => &[],
394            Target::Runs { cards } => cards,
395        }
396    }
397
398    /// Whether this is the run projection's table ([`PathTable::for_runs`])
399    /// rather than a record kind's.
400    pub fn is_runs(&self) -> bool {
401        matches!(self.target, Target::Runs { .. })
402    }
403
404    /// The same table with the registry's extension schemas folded in.
405    ///
406    /// An entry that is not yet `applied` is skipped, so its paths keep
407    /// behaving as unregistered until the indexes exist.
408    pub fn with_ext(&self, entries: &[ExtSchema]) -> Self {
409        let mut paths = self.paths.clone();
410        for entry in entries.iter().filter(|e| e.applied) {
411            for (segments, ty) in &entry.paths {
412                let path = format!("ext.{}.{}", entry.ns, segments.join("."));
413                let mut json_path = vec!["ext".to_string(), entry.ns.clone()];
414                json_path.extend(segments.iter().cloned());
415                paths.insert(
416                    path,
417                    PathInfo::Scalar {
418                        column: ir::Column::Ext {
419                            path: json_path,
420                            ty: *ty,
421                        },
422                        ty: *ty,
423                        indexed: Indexed::Full,
424                    },
425                );
426            }
427        }
428        Self {
429            paths,
430            target: self.target.clone(),
431        }
432    }
433
434    /// Look a path up.
435    ///
436    /// An `ext.` path the registry does not know still resolves — to the
437    /// GIN index, with `eq` and `exists` only — because a producer may
438    /// search its own extension before registering a schema for it.
439    ///
440    /// In a run projection table this also resolves the parametric paths
441    /// `metrics[{ns}/{name}]` and `results[{card}][{metric}].value` /
442    /// `.label` (see [`PathTable::for_runs`]).
443    pub fn lookup(&self, path: &str) -> Option<PathInfo> {
444        self.resolve(path).ok()
445    }
446
447    /// [`PathTable::lookup`], with the reason a path does not resolve: the
448    /// hint of the `unknown_path` entry. The reason is specific where a
449    /// near miss is likely (a dotted parameter, a Card the request did not
450    /// name, a malformed metric id).
451    fn resolve(&self, path: &str) -> Result<PathInfo, String> {
452        if let Some(info) = self.paths.get(path) {
453            return Ok(info.clone());
454        }
455        if let Target::Runs { cards } = &self.target
456            && let Some(result) = resolve_run_param(path, cards)
457        {
458            return result;
459        }
460        if let Some(rest) = path.strip_prefix("ext.") {
461            // `ext.{ns}/{name}.{key…}`: the namespace carries a slash but
462            // never a dot, so the first segment is the namespace.
463            if let Some((ns, key)) = rest.split_once('.')
464                && !ns.is_empty()
465                && !key.is_empty()
466            {
467                let mut json_path = vec!["ext".to_string(), ns.to_string()];
468                json_path.extend(key.split('.').map(str::to_string));
469                return Ok(PathInfo::Scalar {
470                    column: ir::Column::Body(json_path),
471                    // A placeholder: nothing has declared this key's type, so
472                    // `Indexed::EqExistsUntyped` tells the check to ignore it.
473                    ty: ir::ValueType::String,
474                    indexed: Indexed::EqExistsUntyped,
475                });
476            }
477        }
478        Err(match self.target {
479            Target::Record => format!("`{path}` is not a queryable path of this record"),
480            Target::Runs { .. } => format!("`{path}` is not a queryable path of a run"),
481        })
482    }
483
484    /// Every path the table knows, for diagnostics and for the UI's
485    /// completion list.
486    pub fn paths(&self) -> impl Iterator<Item = (&str, &PathInfo)> {
487        self.paths.iter().map(|(k, v)| (k.as_str(), v))
488    }
489}
490
491/// Resolve the parametric paths of the run projection: `None` when `path`
492/// is not one of their forms (and the ordinary lookup should go on),
493/// otherwise the column or the reason it is not one.
494fn resolve_run_param(path: &str, cards: &[CardRef]) -> Option<Result<PathInfo, String>> {
495    use ir::{ResultField, RunColumn as R, ValueType as V};
496
497    let full = |column: R, ty: V| PathInfo::Scalar {
498        column: ir::Column::Run(column),
499        ty,
500        indexed: Indexed::Full,
501    };
502
503    if let Some(rest) = path.strip_prefix("metrics[") {
504        // `metrics[{ns}/{name}]`, and nothing after the bracket: a metric
505        // is a number, not an object.
506        let Some(metric) = rest.strip_suffix(']') else {
507            return Some(Err(format!(
508                "a run's metric is `metrics[{{ns}}/{{name}}]`, a number with no fields; \
509                 `{path}` does not end at the closing bracket"
510            )));
511        };
512        if !is_valid_id(metric) {
513            return Some(Err(format!(
514                "`{metric}` is not a metric id of the form `{{ns}}/{{name}}`"
515            )));
516        }
517        return Some(Ok(full(R::Metric(metric.to_string()), V::Number)));
518    }
519    if path == "metrics" || path.starts_with("metrics.") {
520        return Some(Err(format!(
521            "a run's metrics are addressed with brackets, `metrics[{{ns}}/{{name}}]`, \
522             because a metric id may contain `.`; `{path}` is not a path"
523        )));
524    }
525
526    if let Some(rest) = path.strip_prefix("results[") {
527        const FORM: &str = "`results[{card}][{metric}].value` or `.label`";
528        let Some((card, rest)) = rest.split_once("][") else {
529            return Some(Err(format!(
530                "a judgement is addressed as {FORM}; `{path}` is not of that form"
531            )));
532        };
533        let Some((metric, field)) = rest.split_once("].") else {
534            return Some(Err(format!(
535                "a judgement is addressed as {FORM}; `{path}` is not of that form"
536            )));
537        };
538        let Some(card) = CardRef::parse(card) else {
539            return Some(Err(format!(
540                "`{card}` is not a Card reference of the form `{{ns}}/{{name}}`"
541            )));
542        };
543        if !cards.contains(&card) {
544            return Some(Err(format!(
545                "the Card `{card}` is not named in this request; name it in `cards` to join \
546                 its judgements"
547            )));
548        }
549        if !is_valid_id(metric) {
550            return Some(Err(format!(
551                "`{metric}` is not a metric id of the form `{{ns}}/{{name}}`"
552            )));
553        }
554        let (field, ty) = match field {
555            "value" => (ResultField::Value, V::Number),
556            "label" => (ResultField::Label, V::String),
557            _ => {
558                return Some(Err(format!(
559                    "a judgement has `value` and `label`; `{field}` is neither"
560                )));
561            }
562        };
563        return Some(Ok(full(
564            R::Result {
565                card,
566                metric: metric.to_string(),
567                field,
568            },
569            ty,
570        )));
571    }
572    if path == "results" || path.starts_with("results.") {
573        return Some(Err(format!(
574            "a Card's judgements are addressed with brackets, \
575             `results[{{card}}][{{metric}}].value` or `.label`, because a Card name and a \
576             metric id may contain `.`; `{path}` is not a path"
577        )));
578    }
579    None
580}
581
582/// The facets a record kind has. An Eval has no `grading`.
583fn facets(kind: RecordKind) -> &'static [&'static str] {
584    match kind {
585        RecordKind::Card => &CARD_FACETS,
586        RecordKind::Eval => &["model", "task", "harness", "generation", "trial", "env"],
587    }
588}
589
590/// Walk one property of the schema, collecting every scalar leaf below it
591/// as its path segments and type.
592fn collect_leaves(
593    out: &mut Vec<(Vec<String>, ir::ValueType)>,
594    defs: &Value,
595    prop: &Value,
596    segments: &[String],
597) {
598    let resolved = resolve(defs, prop);
599    let Some(resolved) = resolved else { return };
600
601    if let Some(properties) = resolved.get("properties").and_then(Value::as_object) {
602        for (name, child) in properties {
603            // Each facet's `ext` is the registry's business, and a facet
604            // object itself is not a value to compare against.
605            if name == "ext" {
606                continue;
607            }
608            let mut next = segments.to_vec();
609            next.push(name.clone());
610            collect_leaves(out, defs, child, &next);
611        }
612        return;
613    }
614
615    let Some(ty) = scalar_type(&resolved) else {
616        // Arrays of objects below a facet (`grading.graders`,
617        // `env.packages`) have no side table, so they are not queryable;
618        // the same reasoning as `runs`.
619        return;
620    };
621    out.push((segments.to_vec(), ty));
622}
623
624/// Follow a `$ref` (and the `anyOf [T, null]` an `Option<T>` generates)
625/// to the object that carries the properties or the type.
626fn resolve(defs: &Value, prop: &Value) -> Option<Value> {
627    if let Some(reference) = prop.get("$ref").and_then(Value::as_str) {
628        let name = reference.rsplit('/').next()?;
629        return defs.get(name).cloned();
630    }
631    if let Some(any_of) = prop.get("anyOf").and_then(Value::as_array) {
632        // `Option<T>` is `anyOf: [T, null]`; the non-null branch is the one
633        // that carries the shape.
634        for branch in any_of {
635            if branch.get("type").and_then(Value::as_str) == Some("null") {
636                continue;
637            }
638            if let Some(resolved) = resolve(defs, branch) {
639                return Some(resolved);
640            }
641        }
642        return None;
643    }
644    Some(prop.clone())
645}
646
647/// The scalar type of a leaf, or `None` when it is not a scalar.
648fn scalar_type(schema: &Value) -> Option<ir::ValueType> {
649    let ty = schema.get("type")?;
650    let names: Vec<&str> = match ty {
651        Value::String(s) => vec![s.as_str()],
652        Value::Array(items) => items.iter().filter_map(Value::as_str).collect(),
653        _ => return None,
654    };
655    for name in names {
656        match name {
657            "string" => return Some(ir::ValueType::String),
658            "number" | "integer" => return Some(ir::ValueType::Number),
659            "boolean" => return Some(ir::ValueType::Boolean),
660            _ => continue,
661        }
662    }
663    None
664}
665
666/// The columns an `any` condition may compare, per side table.
667fn array_column(table: ir::ArrayTable, key: &str) -> Option<(ir::ArrayColumn, ir::ValueType)> {
668    use ir::{ArrayColumn as C, ArrayTable as T, ValueType as V};
669    match (table, key) {
670        (T::Results, "metric") => Some((C::ResultMetric, V::String)),
671        (T::Results, "aggregation") => Some((C::ResultAggregation, V::String)),
672        (T::Results, "value") => Some((C::ResultValue, V::Number)),
673        (T::Results, "n") => Some((C::ResultN, V::Number)),
674        (T::Relations, "type") => Some((C::RelationType, V::String)),
675        (T::Relations, "to") => Some((C::RelationTo, V::String)),
676        (T::Attachments, "path") => Some((C::AttachmentPath, V::String)),
677        (T::Attachments, "sha256") => Some((C::AttachmentSha256, V::String)),
678        _ => None,
679    }
680}
681
682/// The keys each side table accepts, for the hint on an unknown one.
683fn array_keys(table: ir::ArrayTable) -> &'static [&'static str] {
684    match table {
685        ir::ArrayTable::Results => &["metric", "aggregation", "value", "n"],
686        ir::ArrayTable::Relations => &["type", "to"],
687        ir::ArrayTable::Attachments => &["path", "sha256"],
688    }
689}
690
691fn entry(path: &str, code: ErrorCode, hint: impl Into<String>) -> ErrorEntry {
692    ErrorEntry {
693        path: path.to_string(),
694        code,
695        hint: Some(hint.into()),
696    }
697}
698
699/// Whether an operator suits a type, per the table in the crate doc.
700fn op_suits(op: ScalarOp, ty: ir::ValueType) -> bool {
701    use ir::ValueType as V;
702    match op {
703        ScalarOp::Eq | ScalarOp::Ne | ScalarOp::In => true,
704        ScalarOp::Gt | ScalarOp::Gte | ScalarOp::Lt | ScalarOp::Lte => {
705            matches!(ty, V::Number | V::String)
706        }
707        ScalarOp::Prefix | ScalarOp::Contains => matches!(ty, V::String),
708    }
709}
710
711/// Whether the storage behind a path can serve an operator.
712fn index_serves(indexed: Indexed, op: ScalarOp) -> bool {
713    match indexed {
714        Indexed::Full => true,
715        Indexed::EqExistsOnly | Indexed::EqExistsUntyped => matches!(op, ScalarOp::Eq),
716    }
717}
718
719/// Whether a literal is of a type.
720fn literal_matches(value: &Value, ty: ir::ValueType) -> bool {
721    match ty {
722        ir::ValueType::Number => value.is_number(),
723        ir::ValueType::String => value.is_string(),
724        ir::ValueType::Boolean => value.is_boolean(),
725    }
726}
727
728fn type_name(ty: ir::ValueType) -> &'static str {
729    match ty {
730        ir::ValueType::Number => "a number",
731        ir::ValueType::String => "a string",
732        ir::ValueType::Boolean => "a boolean",
733    }
734}
735
736/// Check one comparison against a path's type and index, producing the IR
737/// node or the reasons it cannot be served.
738fn check_cmp(
739    column: ir::Column,
740    ty: ir::ValueType,
741    indexed: Indexed,
742    op: ScalarOp,
743    value: &Value,
744    pointer: &str,
745    errors: &mut Vec<ErrorEntry>,
746) -> Option<ir::Cmp> {
747    let mut ok = true;
748    // A path with no declared type has nothing to check a literal
749    // against; containment matches whatever scalar is given.
750    let typed = indexed != Indexed::EqExistsUntyped;
751    if typed && !op_suits(op, ty) {
752        errors.push(entry(
753            &format!("{pointer}/op"),
754            ErrorCode::TypeMismatch,
755            format!("`{op}` does not apply to {}", type_name(ty)),
756        ));
757        ok = false;
758    } else if !index_serves(indexed, op) {
759        errors.push(entry(
760            &format!("{pointer}/op"),
761            ErrorCode::NotIndexed,
762            format!(
763                "`{op}` needs an index this path does not have; it answers `eq` and `exists` \
764                 from the index over the whole record. Register an `ext_schema` for it in the \
765                 registry (`PUT /registry/ext_schemas/{{ns}}/{{id}}@{{version}}`) to make it \
766                 sortable and rangeable."
767            ),
768        ));
769        ok = false;
770    }
771
772    // `in` takes an array of the path's type; every other operator a scalar.
773    if op == ScalarOp::In {
774        match value.as_array() {
775            None => {
776                errors.push(entry(
777                    &format!("{pointer}/value"),
778                    ErrorCode::TypeMismatch,
779                    "`in` takes an array",
780                ));
781                ok = false;
782            }
783            Some(items) => {
784                if let Some(bad) = items.iter().position(|v| typed && !literal_matches(v, ty)) {
785                    errors.push(entry(
786                        &format!("{pointer}/value/{bad}"),
787                        ErrorCode::TypeMismatch,
788                        format!("this path holds {}", type_name(ty)),
789                    ));
790                    ok = false;
791                }
792            }
793        }
794    } else if typed && !literal_matches(value, ty) {
795        errors.push(entry(
796            &format!("{pointer}/value"),
797            ErrorCode::TypeMismatch,
798            format!("this path holds {}", type_name(ty)),
799        ));
800        ok = false;
801    }
802
803    ok.then(|| ir::Cmp {
804        column,
805        op: to_ir_op(op),
806        value: value.clone(),
807    })
808}
809
810fn to_ir_op(op: ScalarOp) -> ir::Op {
811    match op {
812        ScalarOp::Eq => ir::Op::Eq,
813        ScalarOp::Ne => ir::Op::Ne,
814        ScalarOp::Gt => ir::Op::Gt,
815        ScalarOp::Gte => ir::Op::Gte,
816        ScalarOp::Lt => ir::Op::Lt,
817        ScalarOp::Lte => ir::Op::Lte,
818        ScalarOp::In => ir::Op::In,
819        ScalarOp::Prefix => ir::Op::Prefix,
820        ScalarOp::Contains => ir::Op::Contains,
821    }
822}
823
824/// Type-check a parsed filter against a path table.
825///
826/// Every problem is collected; the returned filter is the IR the store
827/// compiles. The `pointer` is where this filter sits in the request, so
828/// that an error can say `/where/and/2/value` rather than "somewhere".
829pub fn check(ast: &Filter, table: &PathTable) -> Result<ir::Filter, Vec<ErrorEntry>> {
830    let mut errors = Vec::new();
831    let filter = check_at(ast, table, "/where", &mut errors);
832    match (filter, errors.is_empty()) {
833        (Some(filter), true) => Ok(filter),
834        _ => Err(errors),
835    }
836}
837
838fn check_at(
839    ast: &Filter,
840    table: &PathTable,
841    pointer: &str,
842    errors: &mut Vec<ErrorEntry>,
843) -> Option<ir::Filter> {
844    match ast {
845        Filter::And { children, .. } | Filter::Or { children, .. } => {
846            let key = grammar::wire_key(ast);
847            let checked: Vec<ir::Filter> = children
848                .iter()
849                .enumerate()
850                .filter_map(|(i, child)| {
851                    check_at(child, table, &format!("{pointer}/{key}/{i}"), errors)
852                })
853                .collect();
854            if checked.len() != children.len() {
855                return None;
856            }
857            Some(match ast {
858                Filter::And { .. } => ir::Filter::And(checked),
859                _ => ir::Filter::Or(checked),
860            })
861        }
862        Filter::Not { child, .. } => {
863            let inner = check_at(child, table, &format!("{pointer}/not"), errors)?;
864            Some(ir::Filter::Not(Box::new(inner)))
865        }
866        Filter::Compare {
867            path, op, value, ..
868        } => {
869            let info = lookup_or_report(table, path, pointer, errors)?;
870            match info {
871                PathInfo::Scalar {
872                    column,
873                    ty,
874                    indexed,
875                } => {
876                    check_cmp(column, ty, indexed, *op, value, pointer, errors).map(ir::Filter::Cmp)
877                }
878                PathInfo::Array { .. } => {
879                    errors.push(entry(
880                        &format!("{pointer}/op"),
881                        ErrorCode::TypeMismatch,
882                        format!("`{path}` is an array; search it with `any` and a `match`"),
883                    ));
884                    None
885                }
886            }
887        }
888        Filter::Exists { path, .. } => {
889            let info = lookup_or_report(table, path, pointer, errors)?;
890            match info {
891                PathInfo::Scalar { column, .. } => Some(ir::Filter::Exists { column }),
892                PathInfo::Array { .. } => {
893                    errors.push(entry(
894                        &format!("{pointer}/path"),
895                        ErrorCode::TypeMismatch,
896                        format!("`{path}` is an array; search it with `any` and a `match`"),
897                    ));
898                    None
899                }
900            }
901        }
902        Filter::Any { path, terms, .. } => {
903            let info = lookup_or_report(table, path, pointer, errors)?;
904            let PathInfo::Array { table: side } = info else {
905                errors.push(entry(
906                    &format!("{pointer}/op"),
907                    ErrorCode::TypeMismatch,
908                    format!("`{path}` is not an array; `any` ranges over one"),
909                ));
910                return None;
911            };
912            check_terms(side, terms, pointer, errors).map(|conditions| ir::Filter::Any {
913                table: side,
914                conditions,
915            })
916        }
917    }
918}
919
920fn check_terms(
921    side: ir::ArrayTable,
922    terms: &BTreeMap<String, MatchTerm>,
923    pointer: &str,
924    errors: &mut Vec<ErrorEntry>,
925) -> Option<Vec<ir::Cmp>> {
926    if terms.is_empty() {
927        errors.push(entry(
928            &format!("{pointer}/match"),
929            ErrorCode::Schema,
930            format!(
931                "`match` needs at least one of {}",
932                array_keys(side)
933                    .iter()
934                    .map(|k| format!("`{k}`"))
935                    .collect::<Vec<_>>()
936                    .join(", ")
937            ),
938        ));
939        return None;
940    }
941    let mut conditions = Vec::new();
942    let mut ok = true;
943    for (key, term) in terms {
944        let term_pointer = format!("{pointer}/match/{key}");
945        let Some((column, ty)) = array_column(side, key) else {
946            errors.push(entry(
947                &term_pointer,
948                ErrorCode::UnknownPath,
949                format!(
950                    "unknown key `{key}`; this array has {}",
951                    array_keys(side)
952                        .iter()
953                        .map(|k| format!("`{k}`"))
954                        .collect::<Vec<_>>()
955                        .join(", ")
956                ),
957            ));
958            ok = false;
959            continue;
960        };
961        match check_cmp(
962            ir::Column::Array(column),
963            ty,
964            Indexed::Full,
965            term.op,
966            &term.value,
967            &term_pointer,
968            errors,
969        ) {
970            Some(cmp) => conditions.push(cmp),
971            None => ok = false,
972        }
973    }
974    ok.then_some(conditions)
975}
976
977fn lookup_or_report(
978    table: &PathTable,
979    path: &str,
980    pointer: &str,
981    errors: &mut Vec<ErrorEntry>,
982) -> Option<PathInfo> {
983    match table.resolve(path) {
984        Ok(info) => Some(info),
985        Err(hint) => {
986            errors.push(entry(
987                &format!("{pointer}/path"),
988                ErrorCode::UnknownPath,
989                hint,
990            ));
991            None
992        }
993    }
994}
995
996/// Parse and check the `where` of a request, collecting its problems.
997fn compile_filter(
998    request: &QueryRequest,
999    table: &PathTable,
1000    errors: &mut Vec<ErrorEntry>,
1001) -> Option<ir::Filter> {
1002    match &request.where_ {
1003        None => None,
1004        Some(value) => match grammar::parse(value) {
1005            Ok(ast) => match check(&ast, table) {
1006                Ok(filter) => Some(filter),
1007                Err(mut e) => {
1008                    errors.append(&mut e);
1009                    None
1010                }
1011            },
1012            Err(mut e) => {
1013                errors.append(&mut e);
1014                None
1015            }
1016        },
1017    }
1018}
1019
1020fn to_ir_dir(dir: envelope::Dir) -> ir::Dir {
1021    match dir {
1022        envelope::Dir::Asc => ir::Dir::Asc,
1023        envelope::Dir::Desc => ir::Dir::Desc,
1024    }
1025}
1026
1027/// Parse, check and assemble a whole request into the IR the store runs.
1028///
1029/// This is the one function the server calls for a record query. `table`
1030/// is the record kind's ([`PathTable::from_schema`]), never the run
1031/// projection's. `limit` is the clamped page size, and `cursor` the
1032/// decoded keyset the server verified: neither is this crate's business to
1033/// decide.
1034pub fn compile(
1035    request: &QueryRequest,
1036    kind: RecordKind,
1037    table: &PathTable,
1038    limit: u32,
1039    cursor: Option<ir::Cursor>,
1040) -> Result<ir::Query, Vec<ErrorEntry>> {
1041    debug_assert!(!table.is_runs(), "a record query needs a record table");
1042    let mut errors = Vec::new();
1043
1044    let filter = compile_filter(request, table, &mut errors);
1045
1046    let mut sort = Vec::new();
1047    for (i, key) in request.sort.iter().enumerate() {
1048        let pointer = format!("/sort/{i}");
1049        if let Some(checked) = check_sort(&key.path, table, &pointer, &mut errors) {
1050            sort.push(ir::Sort {
1051                key: checked,
1052                dir: to_ir_dir(key.dir),
1053            });
1054        }
1055    }
1056
1057    if let Some(cursor) = &cursor
1058        && cursor.keys.len() != sort.len()
1059    {
1060        errors.push(entry(
1061            "/cursor",
1062            ErrorCode::Schema,
1063            "the cursor does not match this sort; start the page sequence again",
1064        ));
1065    }
1066
1067    if !errors.is_empty() {
1068        return Err(errors);
1069    }
1070
1071    Ok(ir::Query {
1072        record_type: match kind {
1073            RecordKind::Card => ir::RecordType::Card,
1074            RecordKind::Eval => ir::RecordType::Eval,
1075        },
1076        filter,
1077        sort,
1078        limit,
1079        cursor,
1080        expand: request
1081            .expand
1082            .iter()
1083            .map(|e| match e {
1084                envelope::Expand::Relations => ir::Expand::Relations,
1085                envelope::Expand::Badges => ir::Expand::Badges,
1086                envelope::Expand::Fingerprints => ir::Expand::Fingerprints,
1087                envelope::Expand::Changed => ir::Expand::Changed,
1088            })
1089            .collect(),
1090        version: match request.version {
1091            None | Some(envelope::VersionSelector::Latest) => ir::VersionSelector::Latest,
1092            Some(envelope::VersionSelector::All) => ir::VersionSelector::All,
1093        },
1094    })
1095}
1096
1097/// Resolve one sort path. Sorting needs an order, which the GIN index
1098/// cannot give, so an `eq`-only path is `not_indexed` here too.
1099fn check_sort(
1100    path: &str,
1101    table: &PathTable,
1102    pointer: &str,
1103    errors: &mut Vec<ErrorEntry>,
1104) -> Option<ir::SortKey> {
1105    if path == "created_at" {
1106        return Some(ir::SortKey::CreatedAt);
1107    }
1108    // `results[{metric}].value` is the one indexed path with a parameter.
1109    if let Some(rest) = path.strip_prefix("results[")
1110        && let Some((metric, tail)) = rest.split_once(']')
1111    {
1112        if tail != ".value" {
1113            errors.push(entry(
1114                &format!("{pointer}/path"),
1115                ErrorCode::UnknownPath,
1116                "a metric sorts by its `value`: `results[{metric}].value`",
1117            ));
1118            return None;
1119        }
1120        if !metric.contains('/') {
1121            errors.push(entry(
1122                &format!("{pointer}/path"),
1123                ErrorCode::UnknownPath,
1124                format!("`{metric}` is not a metric id of the form `{{ns}}/{{name}}`"),
1125            ));
1126            return None;
1127        }
1128        return Some(ir::SortKey::Metric(metric.to_string()));
1129    }
1130
1131    match table.lookup(path) {
1132        Some(PathInfo::Scalar {
1133            column,
1134            indexed: Indexed::Full,
1135            ..
1136        }) => Some(ir::SortKey::Column(column)),
1137        Some(PathInfo::Scalar { .. }) => {
1138            errors.push(entry(
1139                &format!("{pointer}/path"),
1140                ErrorCode::NotIndexed,
1141                format!(
1142                    "`{path}` has no ordered index, so it cannot be sorted on. Register an \
1143                     `ext_schema` for it in the registry to make it sortable."
1144                ),
1145            ));
1146            None
1147        }
1148        Some(PathInfo::Array { .. }) => {
1149            errors.push(entry(
1150                &format!("{pointer}/path"),
1151                ErrorCode::TypeMismatch,
1152                format!("`{path}` is an array; sort by `results[{{metric}}].value` instead"),
1153            ));
1154            None
1155        }
1156        None => {
1157            errors.push(entry(
1158                &format!("{pointer}/path"),
1159                ErrorCode::UnknownPath,
1160                format!("`{path}` is not a queryable path of this record"),
1161            ));
1162            None
1163        }
1164    }
1165}
1166
1167/// Parse, check and assemble a request for the run projection
1168/// (`GET /evals/{ns}/{name}/runs`) into the IR the store runs.
1169///
1170/// The counterpart of [`compile`] for the second target. `table` is
1171/// [`PathTable::for_runs`] for the Cards the request names (with the
1172/// registry folded in), and those Cards are carried into
1173/// [`ir::RunQuery::cards`]. `limit` and `cursor` are, as for [`compile`],
1174/// the server's: the clamped page size and the verified, decoded
1175/// [`ir::RunCursor`].
1176///
1177/// The grammar is the record query's. What differs:
1178///
1179/// - paths resolve against the run table (module doc), so a record path
1180///   such as `title` is `unknown_path`, and `any` has nothing to range
1181///   over;
1182/// - `sort` takes any path of the table except an unregistered `ext` key;
1183///   there is no `created_at` (sort on `started_at` / `ended_at`), and
1184///   `run_id` is the implicit final key;
1185/// - `expand` and `version` are record concepts and are not read; the
1186///   projection has nothing to expand and no versions.
1187///
1188/// Which Eval is read, whether archived and deleted runs are included,
1189/// and which Cards the caller may name are server concerns, decided around
1190/// this call.
1191pub fn compile_runs(
1192    request: &QueryRequest,
1193    table: &PathTable,
1194    limit: u32,
1195    cursor: Option<ir::RunCursor>,
1196) -> Result<ir::RunQuery, Vec<ErrorEntry>> {
1197    debug_assert!(table.is_runs(), "a run query needs PathTable::for_runs");
1198    let mut errors = Vec::new();
1199
1200    let filter = compile_filter(request, table, &mut errors);
1201
1202    let mut sort = Vec::new();
1203    for (i, key) in request.sort.iter().enumerate() {
1204        let pointer = format!("/sort/{i}");
1205        if let Some(checked) = check_run_sort(&key.path, table, &pointer, &mut errors) {
1206            sort.push(ir::Sort {
1207                key: checked,
1208                dir: to_ir_dir(key.dir),
1209            });
1210        }
1211    }
1212
1213    if let Some(cursor) = &cursor
1214        && cursor.keys.len() != sort.len()
1215    {
1216        errors.push(entry(
1217            "/cursor",
1218            ErrorCode::Schema,
1219            "the cursor does not match this sort; start the page sequence again",
1220        ));
1221    }
1222
1223    if !errors.is_empty() {
1224        return Err(errors);
1225    }
1226
1227    Ok(ir::RunQuery {
1228        cards: table.cards().to_vec(),
1229        filter,
1230        sort,
1231        limit,
1232        cursor,
1233    })
1234}
1235
1236/// Resolve one sort path of the run projection. Every run column sorts;
1237/// a registered `ext` path sorts as a [`ir::Column::Ext`] of the run; an
1238/// unregistered one has no declared type and so no order.
1239fn check_run_sort(
1240    path: &str,
1241    table: &PathTable,
1242    pointer: &str,
1243    errors: &mut Vec<ErrorEntry>,
1244) -> Option<ir::SortKey> {
1245    match table.resolve(path) {
1246        Ok(PathInfo::Scalar {
1247            column: ir::Column::Run(column),
1248            ..
1249        }) => Some(ir::SortKey::Run(column)),
1250        Ok(PathInfo::Scalar {
1251            column,
1252            indexed: Indexed::Full,
1253            ..
1254        }) => Some(ir::SortKey::Column(column)),
1255        Ok(PathInfo::Scalar { .. }) => {
1256            errors.push(entry(
1257                &format!("{pointer}/path"),
1258                ErrorCode::NotIndexed,
1259                format!(
1260                    "`{path}` has no declared type, so it has no order. Register an \
1261                     `ext_schema` for it in the registry to make it sortable."
1262                ),
1263            ));
1264            None
1265        }
1266        Ok(PathInfo::Array { .. }) => {
1267            errors.push(entry(
1268                &format!("{pointer}/path"),
1269                ErrorCode::TypeMismatch,
1270                format!("`{path}` is an array and cannot be sorted on"),
1271            ));
1272            None
1273        }
1274        Err(hint) => {
1275            errors.push(entry(
1276                &format!("{pointer}/path"),
1277                ErrorCode::UnknownPath,
1278                hint,
1279            ));
1280            None
1281        }
1282    }
1283}