Crate kopitiam_tensor 

Kopitiam Runtime: tensor.

The CPU tensor type and the inference-time op set (matmul, broadcasting arithmetic, softmax, reductions, normalization, activations, embedding gather) that every layer of a transformer forward pass is built from, plus decoders for the f16/bf16/GGUF-quantized formats model weights ship in.

Scope: Phase 1, inference only

This crate has no autograd, no gradient tape, and no training ops. That is a scope decision, not an oversight: the Kopitiam Runtime’s stated purpose (see docs/ai-decisions/) is local inference, and every op here is chosen because a forward pass needs it. Concretely, this means:

• Every general-purpose op (arithmetic, softmax, reductions, normalization, activations, Tensor::matmul) works on f32 and rejects other dtypes with Error::DTypeMismatch (see Tensor::require_dtype). The one exception is Tensor::quantized_matmul, which computes directly on Q4_0/Q8_0 weight blocks — see that method’s docs for why a fused quantized matmul earns an exception to “everything is f32” (memory: a dequantized 7B Q4_0 model is ~28GB of f32 vs ~4GB quantized) and for the correctness gate that keeps it honest against the plain to_dtype(F32) + Tensor::matmul reference path, which remains this crate’s default and stays permanently, both as the general fallback for every other op and as that oracle.
• There is no backward pass, so there is nothing here resembling a requires_grad flag or a computation graph — Tensor is a plain value type.
• Tensor::gather_rows implements embedding lookup specifically (indices select whole rows), not PyTorch’s general per-element gather; see that method’s docs for why the general form is out of scope.
• There is still no general f32 -> quantized encoder (no Tensor::to_dtype(DType::Q4_0)) — see Tensor::to_dtype’s docs for why requantizing weights is a model-export concern out of this crate’s scope. The quantized module’s quantize_row_q8_0 is a narrower thing: an activation-only Q8_0 encoder, private to Tensor::quantized_matmul’s implementation, not a public Tensor -> Tensor conversion.

Layering

This crate depends only on kopitiam-core for the shared vocabulary (DType, Shape, Device, Error) and re-exports it so downstream crates (kopitiam-loader, kopitiam-runtime) need not add a direct kopitiam-core dependency just to name a dtype:

kopitiam-runtime -> { kopitiam-loader, kopitiam-tokenizer } -> kopitiam-tensor -> kopitiam-core

Module map

• Tensor — the shape + strided-view + shared-storage tensor type, and every op, split across tensor/*.rs by concern (see that module’s docs for why the split is by Rust module rather than by flat file, and how that lets internals stay module-private instead of crate-wide pub(crate)).
• Storage — the owned CPU buffer Tensor is a view into.
• half — f16/bf16 <-> f32 conversion (re-exported as free functions; useful on their own, e.g. for a loader converting a whole weight file up front).
• quant (private) — GGUF block-quantized format decoders, used only through Tensor::to_dtype.

Structs

Shape

The dimensions of a tensor, outermost first.

Tensor

A CPU tensor: a Shape plus a strided view into a shared, ref-counted Storage buffer.

Enums

DType

The element type of a tensor.

Device

Where a tensor’s storage lives and where kernels run on it.

Error

Every way a Kopitiam Runtime operation can fail.

Storage

An owned CPU buffer of tensor elements.

Functions

bf16_to_f32

Decodes a bfloat16 bit pattern to f32.

f16_to_f32

Decodes an IEEE 754 binary16 bit pattern to f32.

f32_to_bf16

Encodes an f32 as the nearest bfloat16 value, rounding ties to even.

f32_to_f16

Encodes an f32 as the nearest IEEE 754 binary16 value, rounding ties to even (the IEEE default), with overflow saturating to infinity and underflow flushing to a signed zero or subnormal.

has_fused_matmul_kernel

Whether Tensor::quantized_matmul has a fused, dequantize-free kernel for dtype in this build — the single source of truth every layer that must choose “compute on the quantized bytes directly” vs. “dequantize to f32 first” shares, so the two decisions can never drift apart.

Type Aliases

Result

The runtime’s result alias.