Module rasters::chunking

Process rasters in memory-efficient chunks.

It is often inefficient to load a large rasters completely into memory while processing it. This module provides iterators to load data in smaller chunks.

Raster Memory Layout

Large rasters are typically sub-divided internally into rectangular blocks of a specific size. For instance, each band of a GDAL raster may be configured with a block size and the total dimension is split into consecutive blocks of the specified size.

The individual blocks support random access while data within a block may require reading the entire block (eg. if the blocks are compressed). While the GDAL API supports reading an arbitary window of data, the underlying driver implements this by reading all the necessary blocks and copying the necessary data into the buffer. Thus, it is more efficient to read along block boundaries.

Memory Efficient Iteration

In order to process with a small memory footprint, the algorithm must satisfy a locality constraint: to process data at pixel (x, y), it is sufficient to access a small window (say 5x5) centered around the pixel. In particular, the chunks supported by this module have the following properties:

• Full Width. Each chunk spans the full width of the raster. This simplifies the iteration logic, and is currently the only supported mode.

• Fixed Padding. Each chunk may additionally use a fixed number of rows above and below it.

Structs

ChunkConfig

Builder to configure chunking. Supports configuring the following paramaters.

Type Definitions

ChunkWindow

The type of item produced by the iterations. Consists of: