Flint

Flint is all you need to light the Torch.

Flint is a RustScript-native AI inference framework. It exposes Torch tensor operations, llama.cpp model primitives, stable-diffusion.cpp image generation, GGML backend discovery, tokenization, and safetensors I/O as composable host functions. Model architecture, sampling loops, and inference workflows remain in RustScript, while the Rust library manages native resources through koharu and executes scripts with the RustScript VM.

How it works

Flint compiles a RustScript program and runs it with ScriptRunner, which binds the flint::* host functions listed below. ScriptRunner::new() executes native GGML, llama.cpp, and stable-diffusion.cpp programs without initializing LibTorch. ScriptRunner::with_device(device).await initializes LibTorch and executes tensor programs on the selected CPU, CUDA, MPS, or Vulkan device.

Tensor and pair values cross the VM boundary as opaque integer handles. A RustScript program uses those handles to compose a graph, then publishes its result with flint::runtime::set_output or flint::runtime::set_text_output. Torch safetensors weights are loaded directly onto the selected device and reused by handle during the run. Native model resources retain their upstream formats, including quantized GGUF tensors.

A typical integration follows this flow:

1. Compile a .rss source file with RustScript.
2. Create ScriptRunner::new() for native programs or ScriptRunner::with_device(device).await for Torch programs.
3. Pass the compiled program and string arguments to run_text.
4. Read the published text from ScriptTextOutput.

Installation

Install the CLI from crates.io:

cargo install rs-flint

Add Flint as a library dependency with cargo add rs-flint; the Rust crate is imported as flint_ai.

cargo install installs the runner binary. The commands below assume a Flint repository checkout so the referenced files under scripts/ are available; --script also accepts an absolute path to an RSS program.

Release Archives

Release assets are zip archives containing the runner and its native Koharu shims. LibTorch, llama.cpp, and stable-diffusion.cpp are fetched by koharu-runtime on first use and stored next to the executable under store. The macos-arm64 runner uses Apple Silicon LibTorch with its MPS backend, which uses Metal; run Torch scripts with --device mps. The windows-x86_64 runner is built against CUDA 13 LibTorch; with a compatible NVIDIA driver, Torch scripts can use --device cuda or --device cuda:N.

CLI

The flint binary has explicit modes:

flint --llm --device cuda:0 --script scripts/lfm2.rss --model <weights> --tokenizer <tokenizer.json> --system-prompt <system> --prompt <text> [--n-predict 128] [--ignore-eos]
flint --llm --device cuda:0 --script scripts/lfm2_5.rss --model <weights> --tokenizer <tokenizer.json> --system-prompt <system> --prompt <text> [--n-predict 128] [--model-kind 0|1] [--image <image>]
flint --llama --script scripts/llama_quantized.rss --model <model.gguf> --prompt <text> [llama options]
flint --llm --script scripts/xlm_roberta_ner_japanese.rss --model <weights> --tokenizer <tokenizer.json> --text <text> [--ctx-size 128] [--labels <csv>]
flint --llm --script scripts/flux_klein_encode_prompt.rss --model <qwen3.safetensors> --tokenizer <tokenizer.json> --prompt <text> --output <prompt.safetensors>
flint --lama --weights model.safetensors --image input.png --mask mask.png --output output.png [--device cuda:0]
flint --sd --script scripts/flux_klein.rss --diffusion-model <model> --vae <vae> --llm <llm> --prompt <text> --output <output.png> [SD options]
flint --sd --script scripts/ggml_devices.rss [--backend cpu|cuda|vulkan]
flint --sd --script scripts/ggml_devices_from_path.rss --path <runtime-directory>

--llm runs Torch-based RustScript model programs. --llama runs llama.cpp programs, and --sd runs stable-diffusion.cpp or GGML programs through the native runner. --lama is the dedicated LAMA image inpainting command and is unrelated to llama.cpp.

For --llm and --lama, omitting --device selects cuda:0 when CUDA is available and otherwise falls back to CPU. Passing --device accepts cpu, cuda, cuda:N, mps, or vulkan. Model-specific arguments after --script are registered by the RSS program and parsed by flint::cli host functions.

RSS command-line arguments

CLI-facing programs use the host-backed flint::cli API, modeled after Rust's argparse crate. A program creates an argument parser, attaches typed references with refer<T>, registers Store, StoreOption, StoreTrue, or StoreFalse actions, and calls parse_args once. Named options accept --name value, --name=value, and aliases. Required values, positional arguments, generated help, integers, floats, strings, and boolean flags are handled by the host parser.

The embedded stdlib::rss::cli module supplies generic option<T>, required_option<T>, and get<T> helpers while leaving registration and parsing in the host. Scripts therefore retain RustScript type information without duplicating parsing logic.

Flint embeds the complete RustScript RSS standard library at compile time and registers it as module source overrides. compile_script_file applies the same module setup for library users, so RSS programs may import modules such as stdlib::rss::parse without a RustScript source checkout at runtime.

--llama runs native llama.cpp RSS programs without loading LibTorch. The scripts/llama_quantized.rss program loads GGUF quantized weights directly, builds the prompt with the model chat template, runs batched prompt decode, and performs the token generation loop in RustScript. Its --backend argument accepts auto, cpu, cuda, vulkan, or a compatible llama.cpp runtime directory.

At koharu commit 3a832b5, the bundled llama.cpp b9938 context layout is newer than the header used by koharu-llama-sys. Until those upstream pieces are aligned, pass a compatible runtime directory such as b8665 as the backend argument.

scripts/flux_klein.rss builds a FLUX.2 Klein text-to-image run from the low-level flint::diffusion::* API backed by koharu-diffusion. The native stable-diffusion.cpp runtime is packaged by koharu-runtime; pass the FLUX.2 Klein diffusion model, VAE, and Qwen3 text encoder paths explicitly.

scripts/flux_klein_encode_prompt.rss runs a Qwen3 text encoder with RustScript torch operations and writes a koharu-ml-compatible prompt embedding file. The safetensors file contains one tensor named prompt_embeds; for Qwen3-4B FLUX.2 Klein this is expected to be [1, 512, 7680]. Torch scripts can reload that tensor with flint::tensor::load_safetensors; the current flux_klein.rss SD path still supplies llm_path to stable-diffusion.cpp.

scripts/xlm_roberta_ner_japanese.rss is a token classification example for tsmatz/xlm-roberta-ner-japanese. It implements the XLM-RoBERTa encoder and classifier from torch host functions in RustScript; the host only provides generic tensor, tokenizer, and formatting helpers.

scripts/lfm2.rss implements LFM2-350M text generation and translation from Torch host functions. scripts/lfm2_5.rss supports both LFM2.5-230M text generation and LFM2.5-VL-450M image input; --model-kind 0 selects text and --model-kind 1 --image <path> selects VL.

Host functions

All functions are registered under the flint namespace.

CLI

Host-backed command-line parsing modeled after argparse::ArgumentParser and its typed references:

flint::cli::argument_parser
flint::cli::set_description
flint::cli::refer
flint::cli::add_option
flint::cli::add_argument
flint::cli::required
flint::cli::metavar
flint::cli::parse_args
flint::cli::get

refer and get preserve the reference value type through the generic stdlib::rss::cli facade. add_option accepts an array of aliases and one of the argparse action names Store, StoreOption, StoreTrue, or StoreFalse.

Runtime

Runtime arguments, host inputs, outputs, and tensor lifetime control:

flint::runtime::args
flint::runtime::arg
flint::runtime::arg_or
flint::runtime::arg_int
flint::runtime::arg_int_or
flint::runtime::arg_float_or
flint::runtime::input
flint::runtime::set_output
flint::runtime::set_text_output
flint::runtime::start_timer
flint::runtime::start_decode_timer
flint::runtime::set_token_count
flint::runtime::set_decode_token_count
flint::runtime::compact2

args returns all script arguments as a string array. Individual arguments can also be addressed by zero-based index. set_output publishes a tensor handle, while set_text_output publishes generated text. Timer and token-count functions populate the timing and count fields returned by ScriptTextOutput; the CLI uses them to print throughput. The compact helper keeps long-running scripts from retaining unused temporary tensors.

GGML

GGML backend discovery helpers:

flint::ggml::load_backends
flint::ggml::list_devices
flint::ggml::stable_diffusion_package_dir
flint::ggml::load_stable_diffusion_backends
flint::ggml::list_stable_diffusion_devices

load_backends and list_devices accept a directory containing ggml.dll or libggml.so, or a file inside that directory. The stable-diffusion helpers select the packaged ggml runtime with the same backend strings accepted by the SD host functions.

Diffusion

Low-level stable-diffusion.cpp host functions:

flint::diffusion::ctx_params_init
flint::diffusion::ctx_params_set_paths
flint::diffusion::ctx_params_set_backend
flint::diffusion::ctx_params_set_wtype
flint::diffusion::ctx_params_set_vae_format
flint::diffusion::ctx_params_set_flags
flint::diffusion::new_sd_ctx
flint::diffusion::free_sd_ctx
flint::diffusion::img_gen_params_init
flint::diffusion::img_gen_params_set_prompt
flint::diffusion::img_gen_params_set_size
flint::diffusion::img_gen_params_set_sample
flint::diffusion::img_gen_params_set_sampler
flint::diffusion::str_to_sample_method
flint::diffusion::str_to_scheduler
flint::diffusion::sample_method_name
flint::diffusion::scheduler_name
flint::diffusion::get_default_sample_method
flint::diffusion::get_default_scheduler
flint::diffusion::generate_image
flint::diffusion::images_save
flint::diffusion::free_sd_images

The flint::diffusion::* functions expose C API-shaped resource handles backed by the owning context, parameter, and image types from koharu-diffusion. Optional backend strings follow sd.cpp names such as cpu, cuda0, or assignment specs like te=cpu,vae=cpu,diffusion=cuda0. Passing cpu, cuda*, or vulkan* also selects the matching packaged stable-diffusion.cpp runtime; auto keeps koharu-runtime's automatic choice.

scripts/flux_klein.rss accepts --sampling-method and --scheduler. Use auto to keep stable-diffusion.cpp defaults, or pass upstream names such as euler, euler_a, dpm++2m, dpm++2m_sde, flux2, simple, karras, or beta.

Llama

flint::llama::* exposes the koharu-llama inference objects as independent handles. GGUF quantization is retained by llama.cpp during model loading and execution.

flint::llama::backend_init
flint::llama::backend_supports_gpu_offload
flint::llama::backend_list_devices
flint::llama::backend_free
flint::llama::model_params_init
flint::llama::model_params_set_gpu_layers
flint::llama::model_params_set_main_gpu
flint::llama::model_params_set_memory
flint::llama::model_load
flint::llama::model_free
flint::llama::model_n_ctx_train
flint::llama::model_n_vocab
flint::llama::model_tokenize
flint::llama::model_is_eog
flint::llama::chat_template
flint::llama::chat_messages_init
flint::llama::chat_messages_add
flint::llama::apply_chat_template
flint::llama::chat_free
flint::llama::tokens_len
flint::llama::tokens_get
flint::llama::tokens_free
flint::llama::context_params_init
flint::llama::context_params_set_sizes
flint::llama::context_params_set_threads
flint::llama::context_new
flint::llama::context_n_ctx
flint::llama::context_decode
flint::llama::context_free
flint::llama::batch_init
flint::llama::batch_add
flint::llama::batch_add_sequence
flint::llama::batch_clear
flint::llama::batch_free
flint::llama::sampler_chain_init
flint::llama::sampler_add_top_k
flint::llama::sampler_add_top_p
flint::llama::sampler_add_min_p
flint::llama::sampler_add_temp
flint::llama::sampler_add_dist
flint::llama::sampler_add_greedy
flint::llama::sampler_chain_build
flint::llama::sampler_sample
flint::llama::sampler_accept
flint::llama::sampler_free
flint::llama::decoder_init
flint::llama::decoder_push
flint::llama::decoder_free

Cache

Named tensor storage for state shared across steps within one execution:

flint::cache::clear
flint::cache::has
flint::cache::get
flint::cache::set

Tokenizer

Tokenizer loading, chat encoding, incremental token collection, decoding, and end-of-sequence checks:

flint::tokenizer::load
flint::tokenizer::encode_chat
flint::tokenizer::encode_vl_chat
flint::tokenizer::encode_padded
flint::tokenizer::format_token_labels
flint::tokenizer::decode_generated
flint::tokenizer::append_token
flint::tokenizer::append_token_tensor
flint::tokenizer::clear_generated_tokens
flint::tokenizer::push_generated_token_tensor
flint::tokenizer::decode_generated_tokens
flint::tokenizer::single_token
flint::tokenizer::is_eos

Load a tokenizer before encoding or decoding. Token tensors remain native tensors and are passed through the VM as handles. encode_padded returns a pair: local is padded input ids and global is the attention mask.

Weights

Safetensors loading and lookup:

flint::weights::load
flint::weights::get
flint::weights::get_indexed
flint::weights::get_or
flint::weights::get_optional

load reads a safetensors file, or every .safetensors file in a directory, onto the runner device. get resolves a tensor by key, get_indexed formats layer names from prefix/index/suffix, and get_or supports alternative keys for model formats that use different parameter names. get_optional returns handle 0 when a key is absent.

Pairs

Two-handle return values used by fused operations and local/global branches:

flint::pair::new
flint::pair::local
flint::pair::global

Tensor operations

Shape inspection, casting, construction, arithmetic, indexing, activation, layout, complex tensors, FFT, and pooling:

flint::tensor::size
flint::tensor::shape
flint::tensor::save_safetensors
flint::tensor::load_safetensors
flint::tensor::to_float
flint::tensor::to_bfloat16
flint::tensor::ones_like
flint::tensor::zeros_like
flint::tensor::zeros_like_int
flint::tensor::arange
flint::tensor::arange_start
flint::tensor::causal_mask
flint::tensor::causal_padding_mask
flint::tensor::padding_mask
flint::tensor::rope_cos
flint::tensor::rope_sin
flint::tensor::rope_cos_at
flint::tensor::rope_sin_at
flint::tensor::add
flint::tensor::sub
flint::tensor::mul
flint::tensor::add_scalar
flint::tensor::mul_scalar
flint::tensor::div_scalar
flint::tensor::pow_scalar
flint::tensor::mean_dim
flint::tensor::rsqrt
flint::tensor::neg
flint::tensor::cos
flint::tensor::sin
flint::tensor::matmul
flint::tensor::softmax
flint::tensor::masked_fill
flint::tensor::cat2
flint::tensor::stack2
flint::tensor::chunk
flint::tensor::narrow
flint::tensor::tail
flint::tensor::transpose
flint::tensor::unsqueeze
flint::tensor::repeat_interleave
flint::tensor::argmax
flint::tensor::argmax_int
flint::tensor::argmax_token
flint::tensor::pad_reflect2d
flint::tensor::relu
flint::tensor::sigmoid
flint::tensor::silu
flint::tensor::gelu
flint::tensor::swiglu
flint::tensor::contiguous
flint::tensor::permute3
flint::tensor::permute4
flint::tensor::permute5
flint::tensor::view2
flint::tensor::view3
flint::tensor::view4
flint::tensor::view5
flint::tensor::select
flint::tensor::real
flint::tensor::imag
flint::tensor::complex
flint::tensor::fft_rfftn2
flint::tensor::fft_irfftn2
flint::tensor::avg_pool2d_2

Tensor operations accept opaque tensor handles and return a new handle unless the function name indicates a scalar result, such as size or argmax_int. Rank-specific functions such as view3 and permute4 take that number of dimensions explicitly. The safetensors helpers read and write one named tensor; use the name prompt_embeds for FLUX prompt embedding files.

Neural network operations

Common model layers and fused inference operations:

flint::nn::embedding
flint::nn::linear
flint::nn::layer_norm
flint::nn::swiglu_linear
flint::nn::rms_norm
flint::nn::add_rms_norm
flint::nn::apply_rope
flint::nn::apply_rope_pair
flint::nn::scaled_dot_product_attention
flint::nn::scaled_dot_product_attention_masked
flint::nn::conv1d
flint::nn::conv1d_step
flint::nn::conv2d
flint::nn::conv_transpose2d
flint::nn::batch_norm2d

Fused functions may return a pair handle. Use flint::pair::local and flint::pair::global to access each tensor result. For optional tensor arguments such as a linear bias, handle 0 represents no tensor.

Vision operations

Image preprocessing and multimodal tensor layout helpers used by LFM2.5-VL:

flint::image::lfm2_vl_patches
flint::vl::siglip2_position_embedding
flint::vl::pixel_unshuffle2
flint::vl::scatter_image_embeddings

The image helper reads an image path and returns model-ready patches. The VL helpers keep the SigLIP2 position, pixel layout, and image-token scatter logic inside generic tensor operations rather than a model-sized host function.

Configuration

• KOHARU_TORCH_WEIGHT_KIND selects the floating-point kind used while loading weights: native, half, bf16, or float. The default is float; use native to preserve the source tensor kind.
• KOHARU_TORCH_LINEAR_MV=0 disables the single-token matrix-vector fast path, which is enabled by default.
• KOHARU_TORCH_PROFILE_OPS=1 prints aggregate Torch host-operation timings after execution.

Links

• Flint repository
• Examples
• RustScript model programs
• RustScript
• Koharu