Crate pidgin_lang

Expand description

§Pidgin — A Compact Agent Handoff Protocol & Runtime

Pidgin is a small, fast, local-first protocol and runtime for compact, structured handoffs between AI agents, between an agent and a human operator, and between an orchestrator and the tools/executors it drives.

Take a compact Pidgin packet (9 lines of key=value text)
→ parse it
→ validate it against a schema and registries
→ run it through a safety gate
→ resolve short references into real paths/IDs
→ expand it into a fully-specified, executable packet
→ estimate token cost
→ recommend a route
→ log every step

Pidgin is not a model, not an agent framework, not a replacement for MCP/A2A/ACP. It is a narrow waist — a deliberately minimal layer that sits between whatever produces a task and whatever executes it.

§Background

Every serious 2025–2026 study of multi-agent token cost reaches the same conclusion: the dominant cost and failure surface in multi-agent LLM systems is the communication layer itself, not the reasoning layer. Replace verbose natural-language handoffs with a small, typed, validated wire format, and you cut both token use and error rate without touching the models.

Pidgin formalizes that conclusion into a runtime instead of a one-off convention.

§Quick Start

cargo install pidgin-lang

# Scaffold a host configuration
pgn init

# Parse a packet
pgn parse examples/basic/generic_task.pgn

# Full pipeline
pgn run examples/basic/generic_task.pgn

# Check (validate + safety + resolve)
pgn check examples/basic/generic_task.pgn

# Token-cost estimation
pgn measure examples/basic/generic_task.pgn

# Validate host configuration
pgn doctor

§Architecture

The runtime is a linear pipeline — each stage is a pure function that transforms or inspects the packet:

Pidgin packet (.pgn text)
   │
   ▼
┌─────────────┐
│   Lexer     │  winnow tokenizer
└─────────────┘
   │
   ▼
┌─────────────┐
│   Parser    │  builds typed AST (PgnPacket)
└─────────────┘
   │
   ▼
┌──────────────────┐
│ Syntax Validator  │  required fields present? types correct?
└──────────────────┘
   │
   ▼
┌──────────────────┐
│ Schema Validator  │  workflow/mode/risk values legal against registries
└──────────────────┘
   │
   ▼
┌──────────────────┐
│   Safety Gate     │  contradiction check, deny precedence, human-required
└──────────────────┘
   │
   ▼
┌──────────────────┐
│ Reference Resolver │ short refs → real paths/IDs with confidence scores
└──────────────────┘
   │
   ▼
┌──────────────────┐
│  Packet Expander   │  builds fully-specified executable packet (YAML)
└──────────────────┘
   │
   ▼
┌──────────────────┐
│  Context Planner   │  decides what to retrieve and how
└──────────────────┘
   │
   ▼
┌──────────────────┐
│  Token Estimator    │  estimates packet + context token cost
└──────────────────┘
   │
   ▼
┌──────────────────┐
│  Router Planner     │  recommends an executor
└──────────────────┘
   │
   ▼
┌──────────────────┐
│  Logger / Metrics    │  every step writes a structured log row
└──────────────────┘
   │
   ▼
Expanded packet, ready for: dry-run report | execution handoff | human approval queue

§Packet Grammar

A Pidgin packet is plain text — a header line followed by key=value fields:

@run task.example
wf=generic_review
mode=draft
in=[primary_subject,source_refs]
out=[review_notes]
do=[draft,review]
deny=[publish,send,delete,secrets]
risk=med
human=yes

Directives: @run, @result, @approval, @context

Reference syntax (inside lists and the route field):

Namespace ref: namespace:id — e.g. file:src/main.rs, ep:UNIT012
Bare alias: resolved through REFERENCE_ALIASES.yaml

§Safety Gate

The safety gate enforces 9 numbered rules (SG-1 through SG-9):

Rule	Description
SG-1	Action in both `do` and `deny` → blocked
SG-2	Human-gated action without `human=yes` → blocked
SG-3	High/critical risk forces `human=yes`, cannot override
SG-4	References resolving to private paths → blocked
SG-5	Unknown workflow → blocked
SG-6	Invalid mode → blocked
SG-7	Free-text `note` field is never parsed for instructions
SG-8	Unresolved required inputs → expansion blocked
SG-9	Critical risk requires an `@approval` packet

Safety-first principle: If the runtime is ever uncertain, it fails closed (blocks, asks for human approval, or refuses to expand) rather than fail open.

§Multi-Agent Integration

Pidgin is the handoff format between agents, not an orchestrator itself. The integration pattern is:

Agent A (orchestrator) ──produces .pgn──→ Pidgin (validate→safety→resolve→expand)
                                                                               │
                                                                          result .pgn
                                                                               │
                                                                          Pidgin (validate→log)
                                                                               │
Agent A ◀──────────────────────── reads result ────────────────────────────────┘

§Integration with Orchestrator Frameworks

LangGraph: A Pidgin node parses/validates the packet before routing to the next agent. Expanded packets become structured messages in the graph state.
CrewAI: Each agent’s task output is a .pgn packet; Pidgin validates inter-agent handoffs.
A2A (Agent2Agent): Pidgin’s expanded Run Packet is the payload inside an A2A Task when crossing trust boundaries.
MCP (Model Context Protocol): Pidgin runs as an MCP server, exposing parse, validate, expand as MCP tools.

§Python SDK

A Python SDK (scaffolded in python/) wraps the pgn binary as a subprocess or via PyO3 bindings, providing typed Pydantic models so orchestrators get Python objects instead of raw CLI output.

§Hooking into the Pipeline

Each stage in the pipeline is a public function you can call from your own code:

use pidgin_lang::parser::parse_packet;
use pidgin_lang::safety::check_safety;
use pidgin_lang::expander::expand_to_run_packet;

let packet = parse_packet("@run my.task\nwf=generic_review\nmode=draft")
    .expect("valid packet");

You can hook into any individual stage, skip stages, or compose them in custom orders depending on your use case.

§Host Configuration

Pidgin reads its configuration from .pidgin/ in the host directory. Use pgn init to scaffold the default config:

pgn init

File	Purpose
`WORKFLOW_REGISTRY.yaml`	Workflow definitions with modes, inputs, executors
`ACTION_REGISTRY.yaml`	Action tiers: `safe`, `controlled`, `human_gated`
`SAFETY_RULES.yaml`	Default denies, private paths, human approval rules
`REFERENCE_ALIASES.yaml`	Short-name aliases for frequently-used references
`PIDGIN_RUNTIME_CONFIG.yaml`	Runtime settings (paths, modes, defaults)

§CLI Reference

Command	Description
`pgn init [--host .] [--force]`	Scaffold default `.pidgin/` config
`pgn parse <file>`	Parse a packet and print the AST
`pgn validate <files> --host .`	Validate syntax + schema against registries
`pgn check <file> --host .`	Parse → validate → safety → resolve (end-to-end)
`pgn resolve <file> --host .`	Resolve all short references
`pgn expand <file> --host . [--out file]`	Expand into executable YAML
`pgn context-plan <file> --host .`	Build a context retrieval plan
`pgn measure <file>`	Estimate token cost
`pgn compare --pgn <file> --verbose <file>`	Compare Pidgin vs verbose token cost
`pgn run <file> --host . [--out file]`	Full pipeline end-to-end
`pgn doctor --host .`	Check host configuration

§Modules

The library is organized into the following modules, each corresponding to a pipeline stage or supporting infrastructure:

Modules§

ast: Typed AST types — PgnPacket, FieldValue, PacketDirective, and related data structures that represent a parsed Pidgin packet in memory.
context: Context planner — decides what information to retrieve given a packet’s workflow and resolved references, producing a structured retrieval plan.
errors: Error types — ParseError, ValidationError, SafetyError, and other error enums with typed variants and formatted messages.
expander: Packet expander — takes a parsed, validated, safety-checked, and resolved packet and produces a fully-specified executable YAML packet (RunPacket, ResultPacket, ApprovalPacket, etc.).
lexer: Lexer/tokenizer — winnow-based tokenizer that converts raw .pgn text into tokens (headers, fields, scalars, lists, comments).
logging: Structured logging — append-only CSV logging for every pipeline stage (parse, validate, safety, resolve, expand, run) with sanitization.
metrics: Token estimation and cost metrics — estimates token cost of raw text and structured packets, and compares Pidgin format against verbose alternatives.
parser: Packet parser — winnow-based grammar parser that converts tokenized input into a typed PgnPacket AST, with size limits and input validation.
registry: Registry loader — deserializes YAML configuration files (WorkflowRegistry, ActionRegistry, SafetyRules) from the host’s .pidgin/ directory.
resolver: Reference resolver — resolves short references (namespace:id, bare aliases) into real filesystem paths or IDs, with containment checks and symlink traversal protection.
router: Route planner — recommends an executor for a packet based on the workflow’s recommended and fallback executors and the safety result.
safety: Safety gate — enforces 9 safety rules (SG-1 through SG-9) including contradiction detection, human-gated actions, private path protection, and post-resolution safety checks.
validator: Validator — syntax validation (structural completeness) and schema validation (registry-checked legality of values) for parsed packets.