Crate flodl_hf 

flodl-hf

HuggingFace integration for flodl: safetensors I/O, hub downloads, tokenizers, and pre-built transformer architectures.

This crate is a sibling to flodl and depends on it for tensor, module, and named-parameter primitives. Transformer building blocks come from flodl::nn.

Scope

• safetensors_io — load/save named tensor dicts from safetensors files.
• hub — download models from the HuggingFace Hub with local caching.
• tokenizer — wrappers over the HuggingFace tokenizers crate.
• models — pre-built architectures (BERT first, LLaMA next).
• path — dotted-path builder for HF-compatible module naming.
• task_heads — shared task_heads::Answer / task_heads::TokenPrediction output types + internal helpers reused by every *For* task head.
• export — write a HF-compatible directory (model.safetensors + config.json) from a trained flodl graph.

Modules

export

HuggingFace-compatible export: write a trained flodl model as a directory HF Python can load with AutoModel.from_pretrained.

hub

HuggingFace Hub download and local cache integration.

models

Pre-built transformer architectures compatible with HuggingFace checkpoints.

path

Path-building helpers for HuggingFace-compatible module naming.

safetensors_io

Safetensors format I/O and load-time validation.

task_heads

Shared task-head types and helpers.

tokenizer

Wrappers over the HuggingFace tokenizers crate for flodl pipelines.