# Build the CSV host
The CSV host turns files into arrays of records and makes them available to
Zirql by name. Zirql reads and evaluates the query; the host renders the result
as JSON. This tutorial follows that implementation so you can adapt it to your
own application. The [CSV guide](csv.md) shows what its users can query.
The complete implementation is [examples/csv.rs](../examples/csv.rs). Code
blocks below are excerpts from that file, with the surrounding responsibilities
explained at each step. Use the complete file for a runnable application; the
excerpts omit imports and some error and budget handling.
## 1. Start with a runnable application
Use Rust 1.88 or newer. From a Zirql `v0.1.1` checkout or extracted
`zirql-0.1.1` crate, create a sibling application and copy the example and its
fixtures:
```sh
cargo new --bin --edition 2024 ../csv-query
cp examples/csv.rs ../csv-query/src/main.rs
mkdir -p ../csv-query/examples
cp -R examples/csv ../csv-query/examples/csv
cd ../csv-query
cargo add zirql@0.1.1 csv@1.4 serde_json@1
```
In the copied `src/main.rs`, change the usage message in `main` to name your
application:
```rust
The original message names the repository's Cargo example. The copied test
module also uses the fixture paths under `examples/csv`, so keep that directory
if you keep the tests.
The application's dependencies are:
```toml
[dependencies]
zirql = "0.1.1"
csv = "1.4"
serde_json = "1"
```
In Zirql's own manifest, `csv` is a development dependency because it serves an
example. Your application needs it as an ordinary dependency. `csv` decodes
quoted records; `serde_json` escapes JSON scalars and strings. Zirql reads and
evaluates the query language.
Run a selection through your new application:
```sh
cargo run -- '
adapt this working application to another input source.
## 2. Choose the value representation
A CSV table fits an array of records. Each record stores header/value pairs in
header order, and each cell is a string:
```rust
use std::sync::Arc;
use zirql::ConstructedValue as V;
struct Table {
headers: Vec<String>,
rows: Arc<V>,
}
```
For example, the first row of `people.csv` becomes a record with `id = "2"`,
`name = "Grace Hopper"`, `team = "compiler"`, `active = "yes"` and
`note = "Said \"hello\""`. Empty cells become `V::String(String::new())`.
Numeric-looking text keeps its exact spelling. The host retains the immutable
table in an `Arc`, which lets the evaluator share its ownership without copying
all rows when it first reads a binding. Each invocation owns its own tables.
This representation gives Zirql access to ordinary fields and collection
operations. CSV rows do not need native handles, custom predicates or a native
pipeline evaluator. Choose native values when queries must retain objects from
an existing model or navigate domain relationships; see
[choosing a data model](implementation-reference.md#choose-a-data-model).
## 3. Load and declare each table
The [load_table and decode_table functions](../examples/csv.rs#L268)
read bounded chunks, enforce an aggregate 64 KiB limit across files, then decode
the bytes. The CSV decoder handles quoting, embedded newlines and UTF-8. The
loader requires a nonempty header, unique nonempty labels and the same field
count in each record. It does not trim or infer cell types.
The inner decoding loop constructs each record after charging for its fields:
```rust
budget.charge(size, location.clone())?;
fields.push((header.clone(), V::String(cell.to_owned())));
```
The row and table containers are then built with `V::Record(fields)` and
`V::Array(rows)`. The complete loop also checks row counts, charges result
allocations and checks the field-size sum before allocating.
Attach the header declaration to the table's stored value:
```rust
let rows = declare_constructed(
V::Array(rows),
DeclaredType::Array(Box::new(DeclaredType::Record(declaration))),
budget,
header_location.clone(),
)?;
budget.charge(1, header_location.clone())?;
budget.charge_results(1, header_location)?;
Ok(Table {
headers,
rows: Arc::new(rows),
})
```
Here `declaration` contains one `(header, DeclaredType::String)` pair per column.
Keep the declaration even for a header-only file. A table without data still
has known columns. Charge the shared owner allocation before wrapping the
declared value in `Arc::new`.
The evaluator also needs type facts before it reads binding values. The
[Table::types method](../examples/csv.rs#L249) derives those facts from headers:
```rust
impl Table {
fn types(
&self,
budget: &mut Budget,
location: Location,
) -> Result<Vec<ValueType>, QueryError> {
budget.charge(self.headers.len() + 3, location.clone())?;
let mut fields = Vec::new();
let mut declaration = Vec::new();
for header in &self.headers {
budget.charge(header.len() * 2 + 2, location.clone())?;
fields.push((header.clone(), ValueType::String));
declaration.push((header.clone(), DeclaredType::String));
}
let row = ValueType::Declared {
actual: Box::new(ValueType::Record(fields)),
declaration: DeclaredType::Record(declaration),
};
Ok(vec![ValueType::Array(ArrayType::Elements(vec![row]))])
}
}
```
The header schema makes `p.missing` an error even if a comprehension has no
rows to visit. Both the stored declaration and the type facts describe the same
array of string records. For another data source, change them together.
## 4. Expose names to the reader
The [CsvHost](../examples/csv.rs#L395) retains loaded tables, their input names
and the names introduced by parsed lets:
```rust
struct CsvHost<'a> {
tables: Vec<Table>,
inputs: &'a [InputSpec],
let_names: Vec<String>,
}
```
`InputSpec` stores an optional name and a path. `admit_inputs` checks names and
rejects duplicates or a mixture of named and unnamed inputs. With one table,
`implicit` returns it; with several, it returns a diagnostic asking the user to
choose a named table.
Implement `ValueReaderHost` so the shared reader recognizes these bindings:
```rust
fn field_pipeline_source(&self) -> zirql::FieldPipelineSource {
zirql::FieldPipelineSource::Constructed
}
fn is_binding(&self, name: &str) -> bool {
name == "rows"
|| self
.inputs
.iter()
.any(|input| input.name.as_deref() == Some(name))
|| self.let_names.iter().any(|binding| binding == name)
}
fn is_constructed_binding(&self, name: &str) -> bool {
self.is_binding(name)
}
```
`FieldPipelineSource::Constructed` lets a field such as `g.members` feed
shared collection stages, as in `g.members | sort`. The same reader owns field
access, comparisons, predicates, comprehensions and sorting.
The remaining [reader methods](../examples/csv.rs#L428) declare the host's
unsupported native operations. `Stage`, `SetOperator`, `Emitter` and `Predicate`
are `()`. `is_stage` returns `false`, `set_operator` returns `None`, and the
native stage and predicate parsers return located errors. The required native
`filter` and `projection` constructors have empty bodies because this reader
never admits those native forms. Shared constructed filters and projections
continue to work.
## 5. Supply values and current input
Implement [ValueQueryHost](../examples/csv.rs#L472) for
`ReadExpression<Self>`, with `type Error = QueryError`. This trait connects
binding names to their data and defines the optional implicit input.
The `binding` helper resolves `rows` through `implicit`; other names select the
corresponding table in `inputs`. The shared hook returns a reference-counted
owner of that table:
```rust
fn read_shared_binding(
&mut self,
name: &str,
range: &Range<usize>,
budget: &mut Budget,
) -> Result<Option<Arc<V>>, QueryError> {
let table = self.binding(name, range)?;
// Share the retained owner; Zirql still admits the full table value.
budget.charge(1, Location::Query(range.clone()))?;
Ok(Some(Arc::clone(&table.rows)))
}
```
This path serves named bindings such as `people` and the implicit table alias
`rows`. Cloning the `Arc` shares the table; it does not copy its contents.
Zirql still performs full runtime admission when it first reads the binding in
this evaluation, checking the table under the request's active budget. Sharing
removes the initial owned copy while retaining those checks.
Keep `read_binding` as the owned fallback. It uses the same `binding` helper and
returns `copy_constructed_value(&table.rows, budget, location)`, charging for
the copy. Hosts without a retained immutable owner can leave
`read_shared_binding` at its default `Ok(None)` to use that owned path.
Zirql resolves saved `let` bindings internally; these hooks supply only table
data.
The two current-input hooks use the same table and schema:
```rust
fn current_input_types(
&mut self,
range: &Range<usize>,
budget: &mut Budget,
) -> Result<Vec<ValueType>, QueryError> {
self.implicit(range)?
.types(budget, Location::Query(range.clone()))
}
fn read_current_input(
&mut self,
range: &Range<usize>,
budget: &mut Budget,
) -> Result<V, QueryError> {
copy_constructed_value(
&self.implicit(range)?.rows,
budget,
Location::Query(range.clone()),
)
}
```
A shorthand query such as `count` reads current input through the owned
`read_current_input` hook. The existing API has no shared current-input hook.
Write `rows | count` to use the shared binding path. Both forms return the same
count and keep copies or sharing within the same request allowance.
The other hooks have small, explicit roles:
| `validate_name` | Reject ambiguous `rows` access before execution |
| `query_types` | Return an error for native query leaves |
| `evaluate_query` | Return an error for native query execution |
## 6. Read lets and a final expression
Follow [read_program](../examples/csv.rs#L549). Start an
`ArgumentReader::new(query, 0)` so diagnostics retain byte offsets in the user's
original query. Repeatedly:
1. Call `read_pattern_let` to read a let pattern, or stop when there is none.
2. Call `read_value_expression` for its initializer.
3. Use `pattern_names` to add the newly bound names to `host.let_names` after
reading the initializer. Later expressions can then reference them.
4. Require `;` and retain the `(pattern, value)` pair.
Read one final expression with `read_value_expression` and reject trailing text.
When there are bindings, wrap them and the final value in one located block:
```rust
Located::new(
K::Block {
bindings,
value: Box::new(value),
},
0..query.len(),
)
```
The source imports `ValueExpressionKind as K`. With no bindings, return the
final expression directly. Pass the same budget and nesting allowance to every
reader call. On parsing failures, release retained patterns with
`dispose_pattern` and expression trees with `dispose_native_value`, as the
complete function does.
The host reads the small program envelope. Zirql evaluates binding scope,
destructuring and each initializer once, including saved selections used by
nested comprehensions. There is no separate CSV expression evaluator. This
frontend admits ordered lets and one expression; named function declarations
need the [larger program interfaces](embedding-programs.md).
## 7. Evaluate under one budget
The [run function](../examples/csv.rs#L93) shows the complete request order:
```text
admit input arguments
→ load tables and their header declarations
→ build binding type facts
→ read the query
→ evaluate the value
→ render into a buffer
→ return the completed bytes
```
Create one budget at the beginning:
```rust
let mut budget = Budget::for_program(limits);
```
Use it for input loading, type facts, parsing, owner allocation and sharing,
value copies, evaluation and rendering. In `run`, the `types` map contains each
named table and `rows`. With one input, `rows` has that table's schema. With several inputs, it has
`ValueType::Unestablished`; the host refuses actual implicit access through
`validate_name` and the current-input hooks.
The evaluation and cleanup sequence is:
```rust
let expression =
read_program(query, &mut host, &mut budget, limits.nesting)
.map_err(query_error)?;
let result = evaluate_query_value(&expression, &types, &mut host, &mut budget);
dispose_native_value(expression);
let result = result.map_err(query_error)?;
```
Here `query_error` adds the submitted query and table paths to a `QueryError`.
Disposing the expression before propagating the evaluation error handles both
success and failure. The returned constructed value belongs to the caller.
The example chooses these limits:
```rust
const LIMITS: ProgramLimits = ProgramLimits {
work: 1_000_000,
results: 100_000,
nesting: 64,
stream_items: 1_000,
};
```
Choose limits that fit your application. `stream_items` also limits the loader
to 1,000 rows per table and 1,000 columns per header. The separate
`MAX_INPUT_BYTES` constant caps source bytes at 64 KiB across all files, bounding
decoder buffers before Zirql values exist.
The core charges its own operations; the host charges its file reads, metadata,
allocations, copies and output.
[Budget and diagnostic APIs](runtime-limits.md) describes that accounting.
## 8. Render only a completed result
[write_json](../examples/csv.rs#L644) walks `value.data()`. Arrays, tuples and
sets become JSON arrays. Records become JSON objects by visiting their stored
fields in order. Absence becomes `null`; scalar encoding uses `serde_json`.
Nominal members and keyed arrays return an error because plain JSON would lose
their identity or shape.
The writer uses `budget.within_evaluation` for recursive containers. `append`
charges output bytes before extending the buffer, and `JsonBuffer` applies the
same accounting to bytes written by `serde_json`. String escaping therefore
counts toward the output budget too.
`run` builds all output in a `Vec<u8>`, including the final newline. `main`
writes that buffer only after the request succeeds. Parsing, evaluation and
rendering failures leave stdout empty; diagnostics go to stderr. An I/O error
while writing the completed buffer is still possible.
CSV diagnostics retain a decoder record, line and byte position where available.
Query diagnostics use `Location::Query` ranges from the shared reader. Keep
these input coordinates when adapting the example so errors can identify the
offending query text or input record.
## Adapt it to your own input
To query another CSV file with the same rules, pass its path; headers become
fields automatically. No Rust changes are needed. For a different source of
tabular data, replace `InputSpec`, `load_table` and `decode_table`, then update
`Table::types` and the stored declaration to match your fields. Keep the reader
and value-query interfaces while your data fits
constructed arrays and records. If your application already has typed data in
memory, start at the value construction step and omit CSV decoding.
For typed CSV columns, make conversion a loader decision. For example, parse an
`amount` column as an integer and store `V::Integer`. Use `DeclaredType::Integer`
for that column in both `decode_table` and `Table::types`; the latter also needs
`ValueType::Integer` for its value facts. Report a bad cell at its CSV record
position. Decide how empty cells behave before changing their representation;
the example preserves them as empty strings.
Decide which names users can reference, whether one source can be implicit and
how callers receive the final value. A library API can return the constructed
value directly. A command-line tool can keep the buffered JSON renderer or
choose another output format.
Reuse Zirql's grammar and evaluator for filters, saved values, correlated
iteration and collection operations. Add native interfaces when your domain
needs retained handles, identity or navigation. The
[host integration guide](host-integration.md) describes those obligations.