Module gather 

Copying the items of a slice into a buffer by position: repeating each, keeping those a mask selects, taking them at indices, walking an n-dimensional pattern of offsets over them, and writing them at offsets. positions writes the positions that a mask selects. Each applies a conversion to each item it writes, so a caller can widen items as it gathers them. Most are generic and #[inline], so they compile in the caller, with the caller’s types and conversion. walk and scatter_walk instead compile their walk once for each item type, and run any other conversion as a separate contiguous pass. Their fixed-width loops move Units, which compile once for each size of item. Booleans and integers convert into f64 through fixed-width loops of their own as they write (Item::cells_from).

Structs

Same

A Convert for a source and result of one type, which keeps each item as it is. A walk or scatter with it moves items straight between its source and result, with no conversion pass.

Table

Explicit offsets along one axis, with Steps::FILL where a fill goes: a list of offsets, or index values that each give one. start and len choose the offsets in use, so that slicing a table copies none of them.

Enums

Steps

The source offsets of one result axis. Regular patterns need no position table; a missing offset reads fill.

Writes

Where a scatter writes: at flat offsets, or at the positions that steps reach from base, in the order walk reads them. A Steps::FILL offset, or a fill position, writes nothing. A walk writes without listing its positions.

Traits

Convert

How walk and scatter_walk turn a source item into a result item. Every Fn(S) -> D converts item by item. Same keeps each item of a source and result of one type as it is.

IndexSource

Index values that a table reads as positions along an axis, converting each one as it reads it, so that the values aren’t first copied into a list of offsets. Table::indices takes them.

IndexValue

A value that names a position along an axis: a Boolean or an integer. A float index value names the whole number it equals within the comparison tolerance, which crate::structure::float_position reads before any table holds it.

Item

An item type that walk and scatter_walk move as a Unit of its size and alignment: each element type of compact buffers. Their fixed-width loops then compile once for each size of item, whatever the item’s type or its conversion. A few conversions into f64 write through fixed-width loops of their own instead (Item::cells_from).

Unit

A type through which walks move items: an unsigned integer, or an array of two, with the size and alignment of the item types that use it.

Functions

counts_up

Whether xs is 0, 1, 2 and so on: offsets that read a source in order, or an axis order that a transpose leaves unchanged.

interleave

Appends rows rows to out, each holding the next lane of each part in turn, as arrays joined along an axis lay out their lanes. Lane r of a part whose lanes hold n items is its items from r × n up to (r + 1) × n. Rows go straight into out‘s spare capacity, so each result item is written once. Where two parts’ lanes hold one length of at most RUN items, each lane moves as one block of units. Other lanes copy item by item.

positions

Appends position(j) for each position j whose count in mask is 1, in order. Every count is 0 or 1, and total of them are 1. Long runs cost little: a block of 64 counts that are all 0 is skipped, and one whose counts are all 1 is written in one loop. In a block that mixes both, selected computes only the selected items. Each count is read by its low bit, which writes every slot whatever the counts hold.

repeat

Appends item(j) for each j below len, as many times as its count. counts holds one count for all the items, or one for each item. The counts are nonnegative and sum to total. An item with a count of at most 4 writes four copies where its run starts. The items after it overwrite the copies beyond its count. A larger count fills its run. With a count for each item, a group of 4 items whose counts are all at most 4 writes its copies after one check for the whole group. With a check and a taken branch for each item, the loop’s cost varied by about a third with where its code was placed.

scatter

Replaces the item of out at each offset with the matching one of items, converted by conv, or with the only one when items holds one. A later offset wins. An out-of-range offset, including Steps::FILL, writes nothing.

scatter_walk

Replaces the items of out that steps reach from base, in the order that walk reads them, with the matching ones of items, converted by conv, or with the only one when items holds one. items holds one item for each position the steps reach, or one item. A fill position writes nothing. A conversion other than Same converts a block of items at a time, whole rows of the first axis up to about CHUNK items, then writes that block while it’s still in cache. Where D has a fixed-width loop that converts from S (Item::cells_from), short cells along a stride convert as they write instead, with no separate pass. The walk’s fixed-width loops then move items as Units, so they compile once for each size of item, whatever the types and conversion. Rows longer than RUN write along a slice each. Short cells write through one table of their offsets, as walk reads them. Cells that a table of starts picks, each reaching items a stride apart, as columns’ do, must be disjoint or identical. They write in blocks of 8 cells, row by row, so each block’s writes share the lines they touch, as walk’s reads do. A later identical cell wins. Other steps write in walk’s order.

take

Appends the items of items at indices, each converted by f. Each index is in -n..n for n items, and a negative index counts back from the end.

walk

Appends the items of s that steps reach from base, each converted by conv: one Steps for each axis, with the last varying fastest. A None offset gives fill, converted. conv is any Fn(S) -> D, or Same for a source and result of one type. The walk’s fixed-width loops move items as Units, so they compile once for each size of item. With Same, the walk appends straight onto out, splitting across threads from [SPLIT] items. Any other conversion walks CHUNK items at a time into a buffer of source items, which it converts onto out while they’re still in cache.