Crate hermes_mal 

Model Architecture Language (MAL) for Hermes LLM

A composable DSL for defining LLM model architectures using pest parser.

Example MAL - Minimal inline style

model tiny {
    vocab_size: 32000
    max_seq_len: 2048
    hidden_size: 128
    num_layers: 4
    block: {
        attention: { num_heads: 4 }
        ffn: { hidden_dim: 512 }
    }
}

Optional continuum memory

A block can replace its ordinary ffn with an ordered, fast-to-slow residual memory chain. Reserve experts are fixed-capacity architecture; their active masks live in checkpoints.

memory cms {
    tier fast {
        ffn: fast_ffn
        reserve_experts { capacity: 2 rank: 32 top_k: 1 }
    }
    tier slow {
        ffn: slow_ffn
        reserve_experts { capacity: 8 rank: 32 top_k: 1 }
        residual_init: zero
    }
}

block remembered {
    attention: gqa
    memory: cms
}

Example MAL - Composable style

# Define attention mechanism
attention gqa {
    num_heads: 32
    num_kv_heads: 8
    head_dim: 128
    position_encoding: rope { theta: 10000.0 }
}

# Define FFN
ffn swiglu_mlp {
    hidden_dim: 14336
    activation: swiglu
    bias: false
}

# Define transformer block
block llama_block {
    attention: gqa
    ffn: swiglu_mlp
    norm: rmsnorm { eps: 1e-5 }
    norm_position: pre
}

# Define model using the block
model llama_7b {
    vocab_size: 32000
    max_seq_len: 4096
    hidden_size: 4096
    block: llama_block
    num_layers: 32
}

Structs

AttentionDef

Attention mechanism definition

BlockDef

Transformer block definition

EmbeddingsConfig

Embeddings configuration

FfnDef

Feed-forward network definition

MalFile

Complete parsed MAL file with all definitions

MalParser

MemoryDef

Ordered fast-to-slow FFN/MoE memory levels following a sequence mixer.

MemoryTierDef

One level in a fast-to-slow continuum memory chain.

ModelDef

Parsed model definition from MAL

MoeDef

Configurable dropless token-choice mixture of experts.

NormConfig

Normalization configuration

OutputConfig

Output head configuration

ReserveExpertsDef

Fixed-capacity low-rank experts available for sleep-time consolidation.

SsmDef

Selective state-space (Mamba) mixer definition

Enums

Activation

Activation function

MemoryTierInit

Initial behavior of a memory tier’s ordinary FFN branch.

NormPosition

Normalization position in block

NormType

Normalization type

PositionEncoding

Position encoding type

Rule

Functions

get_builtin_model

Get a well-known model definition by name

get_wellknown_mal

Get the raw MAL content for a well-known model

list_wellknown_models

List all well-known model names (auto-discovered from embedded files)

parse_mal

Parse a MAL string containing exactly one model definition.

parse_mal_file

Parse MAL from a file

parse_mal_full

Parse complete MAL file with all definitions