Module preprocess 

Image -> pixel_values: the content path between a decoded file and the vision tower.

Steps 3 and 4 deliberately fed the tower the reference’s own pixel_values, so that a tensor mismatch could be attributed to the tower rather than to a resize. This module is the brick they isolated out, and it gets its own gate for the same reason.

The spec, pinned from preprocessor_config.json

step	value
convert to RGB	do_convert_rgb: true
resize longest edge	2048
resample filter	1 = LANCZOS
split into tiles	max_image_size.longest_edge: 512
plus a global thumbnail	-> 17 tiles for a square image
rescale	1/255
normalize	mean 0.5, std 0.5 -> [-1, 1]

That arithmetic is what produces 17 tiles: the image is scaled UP to 2048x2048, cut into sixteen 512x512 tiles, and a thumbnail is appended.

Why Lanczos is written out rather than substituted

The tree already has carmenta::image::resize_bilinear and a Catmull-Rom resize_bicubic, each chosen for a measured reason in its own path. Neither is this one. Substituting a different filter changes every pixel_values and therefore every tensor downstream — silently, because the output is still a plausible image. So the filter the reference declares is the filter implemented, and it is gated against the reference’s own tensor.

The convention matters as much as the kernel. PIL’s resampler, which is what resample: 1 means here:

• maps output pixel i to input centre (i + 0.5) * scale, where scale = in / out — a HALF-PIXEL-CENTRED mapping. Using i * scale instead shifts the whole image by half a pixel and is the classic off-by-half that survives visual inspection;
• widens the kernel when DOWNSCALING (filter_scale = max(1, scale)) so the filter low-passes rather than aliases, and leaves it at 1 when upscaling;
• normalises the weights to sum to 1, so brightness is preserved;
• resamples horizontally then vertically — separable, and it is what makes the cost O(w*h*support) instead of O(w*h*support^2).

Structs

Preprocessed

One image, preprocessed into what the tower and the prompt both need.

Functions

fit_longest_edge

Longest-edge resize target, preserving aspect ratio.

lanczos3

Lanczos-3 kernel: sinc(x) * sinc(x/3), zero outside |x| < 3.

normalize_u8

Rescale to [0,1] then normalize to [-1,1], matching rescale_factor 1/255 with mean and std 0.5.

preprocess_rgb8

Decoded RGB8 -> pixel_values, the whole Idefics3 content path.

preprocess_rgb8_opts

The same path, with tile splitting optional.

resize_lanczos

Separable Lanczos resize of an interleaved channels-plane image.

resize_lanczos_u8

PIL-faithful Lanczos resize on u8 samples.

resized_size

The size an image is resized to before tiling — step 2 of the content path.

tile_geometry

The tile geometry an image WOULD get, without touching a pixel.

tile_grid

The tile grid a given image size produces, before the thumbnail.

vision_encoder_size

The vision encoder wants both dimensions to be exact multiples of the tile size, so the split is a clean grid rather than a grid plus an odd remainder.