Crate ytsaurus_job 

Write YTsaurus MapReduce jobs in Rust.

A YTsaurus job is an ordinary executable. The cluster runs it once per chunk of input, feeds it rows on fd 0, and collects output tables from fds 1, 4, 7… The wire format is YSON, normally binary. This crate handles that protocol so a job can be written as a loop over rows.

A complete mapper

use serde::{Deserialize, Serialize};
use ytsaurus_job::{Event, JobReader, JobWriter};

#[derive(Deserialize)]
struct Input<'a> {
    #[serde(borrow)]
    url: &'a str,
    size: i64,
}

#[derive(Serialize)]
struct Output<'a> {
    host: &'a str,
    size: i64,
}

fn main() {
    ytsaurus_job::run(|| {
        let mut reader = JobReader::from_stdin();
        let mut writer = JobWriter::descriptors(1)?;

        while let Some(event) = reader.next_event()? {
            let Event::Row(row) = event else { continue };
            let input: Input = row.parse()?;
            let host = input.url.split('/').next().unwrap_or("");
            writer.write(0, &Output { host, size: input.size })?;
        }

        writer.finish()
    })
}

Build it for the cluster with scripts/build-worker.sh, then launch it with the yt CLI — see docs/writing-a-job.md.

Memory

The input stream is usually much larger than the job’s memory limit. JobReader never accumulates it: it holds one buffer (1 MiB by default) and hands out rows that borrow from it. A row is only copied if you ask for an owned type when decoding it.

Control records

When the operation enables them, YTsaurus interleaves control records with the data: <table_index=N>#, <row_index=N>#, <range_index=N># and <key_switch=%true>#. JobReader consumes the first three and reflects them on each Row; key_switch surfaces as Event::KeySwitch, or is turned into per-key iterators by JobReader::groups.

Re-exports

pub use crate::error::JobError;

pub use crate::error::Result;

pub use crate::reader::Event;

pub use crate::reader::Group;

pub use crate::reader::GroupKey;

pub use crate::reader::Groups;

pub use crate::reader::JobReader;

pub use crate::reader::Row;

pub use crate::skiff::SkiffJobReader;

pub use crate::skiff::SkiffRow;

pub use crate::skiff_writer::SkiffJobWriter;

pub use crate::statistics::JobStatistics;

pub use crate::worker::WorkerEvent;

pub use crate::worker::WorkerReader;

pub use crate::worker::WorkerRow;

pub use crate::worker::WorkerWriter;

pub use crate::writer::JobWriter;

pub use crate::writer::TableId;

pub use crate::writer::table_descriptor;

pub use ytsaurus_yson as yson;

Modules

error

Errors a job can fail with. Errors a job can fail with.

reader

Reading the input stream. Incremental reading of a job’s input stream.

skiff

Reading Skiff input streams. Streaming Skiff job input.

skiff_writer

Writing Skiff output streams. Writing schema-described Skiff output tables.

statistics

Custom job statistics. Custom job statistics: numbers a job reports for the operation to aggregate.

worker

Shared-format worker reader and writer. One worker API selected by ytsaurus_format::DataFormat.

writer

Writing output tables. Writing a job’s output tables.

Enums

DataFormat

A supported YTsaurus data format.

Functions

install_panic_hook

Installs a panic hook that reports panics in a form a human can act on.

is_inside_job

Whether this process is running as a job on a cluster.

job_cookie

Which job of its task this is, counting from zero.

job_id

The job’s ID, when running inside one.

run

Runs a job body, reporting failures the way YTsaurus expects.

run_if_inside_job

Runs job if this process is a job, and returns otherwise.