Struct CpuBackend 

pub struct CpuBackend { /* private fields */ }

The CPU backend, which owns its threads.

Implementations

impl CpuBackend

pub fn new(threads: usize) -> Self

A backend on threads threads, the calling thread included.

pub fn with_int8(self, on: bool) -> Self

The same backend with the GEMMs over token rows in INT8 when on: weights rounded per output channel at upload, activations per row as they are read, sums in i32. See qgemm.

pub fn int8(&self) -> bool

Whether the GEMMs over token rows run in INT8.

pub fn threads(&self) -> usize

Threads.

Trait Implementations

impl Backend for CpuBackend

type Weights = Weights

Weights in the backend’s own layout.

type Plan = CpuPlan

A graph lowered for one bucket, with its arena.

fn caps(&self) -> Caps

What the backend can do.

fn weight_bytes(&self, w: &Weights) -> usize

Bytes the weights take on the device.

fn plan_bytes(&self, p: &CpuPlan) -> usize

Bytes a plan takes on the device, its arena and staging buffers.

fn upload(&self, tensors: &[HostTensor<'_>], graph: &Graph) -> Result<Weights>

Converts a checkpoint’s tensors, indexed as the graph’s weights are.

fn lower(&self, w: &Weights, graph: &Graph, bucket: Bucket) -> Result<CpuPlan>

Lowers a graph for one bucket. This is where memory is allocated.

fn run( &self, plan: &mut CpuPlan, batch: &Batch<'_>, out: &mut Outputs, ) -> Result<()>

Runs a batch that fits the plan’s bucket and has been checked.

impl Debug for CpuBackend

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.