Crate pidgin_lang

Expand description

§Pidgin — A Compact Agent Handoff Protocol & Runtime

Pidgin is a minimal, local-first protocol runtime for structured, validated handoffs between AI agents. It parses a compact nine-line text packet, runs it through a deterministic pipeline (validate → safety check → resolve → expand), and produces an executable YAML specification — all without calling a single LLM or opening a single network socket.

.pgn file (9 lines of key=value text)
    │
    ▼
┌─────────────┐
│   Lexer     │  winnow tokenizer, input limits
└─────────────┘
    │
    ▼
┌─────────────┐
│   Parser    │  typed AST (PgnPacket)
└─────────────┘
    │
    ▼
┌──────────────────┐
│ Syntax Validator  │  required fields present? types correct?
└──────────────────┘
    │
    ▼
┌──────────────────┐
│ Schema Validator  │  workflow/mode/risk legal against registries?
└──────────────────┘
    │
    ▼
┌──────────────────┐
│   Safety Gate     │  9 rules (SG-1 through SG-9), fail closed
└──────────────────┘
    │
    ▼
┌──────────────────┐
│ Reference Resolver │ short refs → canonical paths/IDs
└──────────────────┘
    │
    ▼
┌──────────────────┐
│  Packet Expander   │  fully-specified executable YAML
└──────────────────┘
    │
    ▼
┌──────────────────┐
│  Context Planner   │  what to retrieve and how
└──────────────────┘
    │
    ▼
┌──────────────────┐
│  Token Estimator    │  packet + context token cost
└──────────────────┘
    │
    ▼
┌──────────────────┐
│  Router Planner     │  recommended executor
└──────────────────┘
    │
    ▼
┌──────────────────┐
│  Logger              │  every step writes structured log
└──────────────────┘
    │
    ▼
Expanded packet, ready for: dry-run report | execution | human approval queue

§Why This Exists

Every serious study of multi-agent token cost in 2025–2026 converges on the same finding: the communication layer is the dominant cost and failure surface, not the reasoning layer. Agents pass verbose natural-language messages to each other — paragraphs of implicit instructions, unclearly scoped tasks, unvalidated assumptions. Each handoff costs hundreds or thousands of tokens. Worse, there’s no validation: Agent B can interpret Agent A’s message differently from what Agent A intended, and nobody audits it until something breaks.

Pidgin formalizes the narrow waist between agents into a typed, validated, safety-checked wire format. The packet is nine lines. The entire runtime pipeline — lex, parse, validate, safety, resolve, expand, log — runs in single-digit milliseconds on a laptop. You can audit every handoff because every handoff is logged to a structured file.

Pidgin is not an orchestrator. It does not run agents, it does not make decisions, it does not call models. It sits between whatever produces a task (a LangGraph node, a CrewAI agent, a human typing at a terminal) and whatever executes it (Claude, Codex, a shell script), and ensures the handoff is parseable, safe, and logged.

§Quick Start

cargo install pidgin-lang
pgn init                   # scaffold .pidgin/ config
pgn parse examples/basic/generic_task.pgn
pgn run examples/basic/generic_task.pgn
pgn check examples/basic/unsafe_contradiction.pgn
pgn measure examples/basic/generic_task.pgn
pgn doctor

§Packet Grammar

A Pidgin packet is plain text with a strict, unambiguous grammar. There is no nesting, no optional syntax, no inline markup — every byte is either a header, a field, a list element, or a comment.

@<directive> <run_id>

Four directives:

Directive	Purpose
`@run`	A task for an agent to execute
`@result`	The outcome of a completed task
`@approval`	Human sign-off for a critical-risk task
`@context`	A request for additional information

run_id is a dotted identifier — task.example, docs.review.2026-06-01 — that uniquely identifies the handoff across the system.

§Fields

Every field is key=value on its own line. Whitespace around = is not allowed, which eliminates an entire class of lexer ambiguities.

wf=generic_review          # bare scalar
risk=med                   # bare scalar (allowed values depend on workflow)
in=[primary_subject,refs]  # list — comma-separated, no spaces
note="Draft only"          # quoted string (for free text)

§Scalar fields

Field	Type	Appears On
`wf`	bare word	all directives
`mode`	bare word	`@run`, `@context`
`risk`	`low │ med │ high │ crit`	`@run`, `@approval`
`human`	`yes │ no`	`@run`, `@approval`
`status`	`ok │ fail │ partial`	`@result`, `@approval`
`ttl`	integer	all directives
`note`	quoted string	all directives

§List fields

Field	Appears On	Contents
`in`	`@run`, `@context`	Input references
`out`	`@run`, `@result`	Output references
`do`	`@run`	Actions to perform
`deny`	`@run`	Actions to explicitly deny
`produced`	`@result`	References to produced artifacts

§Reference syntax

Inside list fields, references can be either:

Namespaced: namespace:id — e.g. file:src/main.rs, ep:UNIT012
Bare alias: primary_subject — resolved through REFERENCE_ALIASES.yaml

Built-in namespaces:

Namespace	What It References
`ep`	Entity pointer — a content item, document, or record
`rb`	Rollback target — a point-in-time snapshot
`ledger`	Ledger entry — an audit record
`claim`	Claim — a config key, channel, or property
`policy`	Policy — a rule file or constraints document
`skill`	Skill — a capability or tool definition
`wf`	Workflow — a workflow definition
`file`	File path (checked against safety rules)
`folder`	Directory path (checked against safety rules)
`dash`	Dashboard — a view or report
`queue`	Queue — a named message queue

§Safety Gate (SG-1 through SG-9)

The safety gate is the core safety mechanism. It is a separate pipeline stage — it runs after parsing and validation but before reference resolution and expansion. It cannot be disabled, skipped, or overridden by packet fields.

Every rule fails closed: if the gate cannot determine whether a rule applies, it treats the rule as violated.

§SG-1 — Contradiction

An action cannot appear in both do and deny. If a packet says both “do publish” and “deny publish”, something is wrong. The runtime refuses to guess which intent is correct.

§SG-2 — Missing Human Approval

If a packet requests a human-gated action (publishing, deleting, sending credentials, etc.) without human=yes, the gate blocks. Human-gated actions are defined in ACTION_REGISTRY.yaml as the human_gated tier plus human_required_actions in SAFETY_RULES.yaml.

§SG-3 — Forced Human Approval

High- or critical-risk packets cannot opt out of human review. Even if the packet declares human=no, the gate overrides it. This prevents a risky packet from bypassing human oversight by lying about its own human field.

§SG-4 — Private Path Access

Any file: or folder: reference that resolves to a path matching a private path pattern (.env, .ssh/, *.pem, secrets/, etc.) is blocked. Paths are canonicalized before matching, so symlink tricks and traversal sequences (../../etc/passwd, %2e%2e) are caught.

§SG-5 — Unknown Workflow

The wf field must match a workflow defined in WORKFLOW_REGISTRY.yaml. An unknown workflow is not a no-op — it is a bug or an attack.

§SG-6 — Invalid Mode

The mode field must be in the workflow’s allowed_modes list. A generic_review workflow should never be executed in production mode.

§SG-7 — Note Isolation

The note field stores free text. No pipeline stage reads or interprets it. This is structural: the note is an opaque string throughout the entire pipeline. If you want an agent to read a note, the expanded packet makes it available, but the runtime never acts on its contents. This closes the most obvious prompt-injection surface.

§SG-8 — Unresolved Required Input

If a required input reference fails to resolve (the alias is not in REFERENCE_ALIASES.yaml and the namespace:id doesn’t exist), expansion is blocked. Running with missing inputs produces silent failures.

§SG-9 — Critical Risk Requires Approval Packet

A packet with risk=crit requires a separate @approval packet with status=ok before it can be expanded. A single human=yes on the same packet is not enough — the two-packet pattern ensures separation of concerns.

§Multi-Agent Integration

Pidgin is the handoff format, not the orchestrator. Here is how it fits into various agent architectures:

                                         expanded .pgn
Orchestrator ─── .pgn ──→ Pidgin ──────────────────────────→ Executor Agent
(LangGraph,    ←──────  (validate, safety,    ←──────────  (Claude, Codex,
 CrewAI,       result    resolve, expand,      result       shell, tool)
 A2A, MCP)               log)

§LangGraph

Add a Pidgin node between any two agent nodes. The Pidgin node parses and validates the .pgn packet produced by the source agent, routes to the destination agent based on the expanded packet, and logs the handoff to the shared graph state. The flow remains typed and auditable.

§CrewAI

Each CrewAI agent produces a .pgn packet as its task output. A custom CrewAI tool wraps pgn run (or the library) to validate inter-agent handoffs. Invalid or unsafe handoffs block before reaching the next agent.

§A2A (Agent2Agent)

Pidgin’s expanded Run Packet maps naturally into an A2A Task. The .pgn format becomes the wire representation inside trust boundaries; the expanded A2A Task is the representation crossing them. Pidgin’s safety gate provides the guardrails that A2A intentionally leaves to implementors.

§MCP (Model Context Protocol)

Pidgin can run as an MCP server, exposing parse, validate, check, and expand as MCP tools. Agents connected through MCP call these tools as part of their workflow, getting structured, validated handoffs without leaving the MCP protocol.

§Python SDK

The Python SDK (scaffolded in python/) wraps the pgn binary via subprocess, giving Python-based orchestrators (LangChain, CrewAI, custom scripts) typed Pydantic models for packets, safety results, and expanded output. PyO3 bindings are on the roadmap.

§Host Configuration

Every Pidgin host provides a .pidgin/ directory with five YAML config files. Run pgn init to scaffold default versions.

File	What It Defines
`PIDGIN_RUNTIME_CONFIG.yaml`	Host name, log directory, default deny list, input limits (1 MB max packet, 100 max fields, 10K max field length, 10 MB max config)
`WORKFLOW_REGISTRY.yaml`	Workflow definitions — each with risk default, allowed modes, required inputs, expected outputs, recommended executor, and human-approval requirement
`ACTION_REGISTRY.yaml`	Three tiers: `safe` (always allowed), `controlled` (allowed with validation), `human_gated` (requires `human=yes`)
`SAFETY_RULES.yaml`	Default deny list, gitignore-style private path patterns, human-required actions and risk levels
`REFERENCE_ALIASES.yaml`	Short-name aliases mapping bare identifiers to full `namespace:id` references

Config files are loaded at startup from the host root (. or $PIDGIN_ROOT_DIR). Each file is validated for structure and key presence.

§CLI

The pgn binary exposes every pipeline stage as a subcommand plus infrastructure commands:

Command	Action
`init`	Scaffold `.pidgin/` directory
`parse <path>`	Lex and parse, print AST
`validate <path>`	Syntax + schema validation
`check <path>`	Full guard: validate → safety → resolve
`resolve <path>`	Expand short references
`expand <path>`	Full pipeline → executable YAML
`run <path>`	Full pipeline + structured logging
`measure <path>`	Estimate token cost
`compare <path>`	Compare vs verbose token cost
`context-plan <path>`	Build retrieval plan
`doctor`	Check host configuration health
`docs`	Print full protocol documentation

Exit codes:

0 success
1 validation error (syntax or schema)
2 safety gate blocked
3 unresolved required reference
4 configuration error
5 internal error (file a bug)

§Using as a Library

Add to Cargo.toml:

[dependencies]
pidgin-lang = "0.1"

Then compose your own pipeline:

use pidgin_lang::parser::parse_packet;
use pidgin_lang::safety::check_safety;
use pidgin_lang::expander::expand_to_run_packet;

let packet = parse_packet("@run my.task\nwf=generic_review\nmode=draft")
    .expect("valid packet");

Every stage function is public and well-typed. You can:

Call only the parser (if you just need an AST)
Call validate + safety without resolving (quick check)
Skip the safety gate in test/development only
Add custom stages before or after any built-in stage
Implement your own logger by implementing the Logger trait

§Safety Properties

Beyond the safety gate, several cross-cutting properties hold:

No network calls in core. The runtime never opens a socket or makes an HTTP request. There is no network attack surface.
No LLM in the hot path. Every stage is deterministic. No model is called at any point. The safety gate is algorithmic, not probabilistic.
Fail closed. Every uncertain decision blocks. Unknown workflow → blocked. Missing required input → blocked. Ambiguous reference → blocked.
Path containment. All file references are canonicalized and checked against the host root. No traversal, no symlink escape, no encoded bypass.
Input limits. Packets are bounded (1 MB, 100 fields, 10K per field). Config files are bounded (10 MB). The lexer rejects oversized input before any heavy processing.
Sanitized logging. User values in logs are filtered to printable ASCII with length caps. No log injection, no CSV injection.
Append-only logs. Every log write completes (flush + close) before the next pipeline stage starts. No log loss on crash.

§Modules

The library is organized into modules corresponding to pipeline stages and supporting infrastructure:

Modules§

ast: Typed AST types — PgnPacket, FieldValue, PacketDirective, and related data structures representing a parsed Pidgin packet in memory.
context: Context planner — decides what information to retrieve based on a packet’s workflow and resolved references, producing a structured retrieval plan.
errors: Error types — ParseError, ValidationError, SafetyError, and other error enums with typed variants, source chaining, and formatted messages.
expander: Packet expander — takes a validated, safety-checked, resolved packet and produces a fully-specified executable YAML packet ready for consumption by executors (RunPacket, ResultPacket, ApprovalPacket, ContextPacket).
lexer: Lexer/tokenizer — winnow-based tokenizer that converts raw .pgn text into structured tokens: header, fields, scalars, lists, comments. Enforces input size limits (1 MB max, 100 fields max, 10,000 chars per field).
logging: Structured logging — append-only CSV logging for every pipeline stage, with user-value sanitization (printable ASCII only, length-capped, newline-escaped) to prevent log injection and CSV injection.
metrics: Token estimation and cost metrics — estimates token cost of raw text and structured packets using configurable models. Also supports comparing the same handoff expressed as Pidgin vs verbose natural language.
parser: Packet parser — winnow-based grammar parser that converts lexed tokens into a typed PgnPacket AST with full field validation, directive-specific required-field checks, and human-readable parse errors.
registry: Registry loader — deserializes YAML configuration files from .pidgin/ (WorkflowRegistry, ActionRegistry, SafetyRules, ReferenceAliases, PidginRuntimeConfig) with validation of structure and required keys.
resolver: Reference resolver — maps short references (namespace:id and bare aliases) to real filesystem paths or identifiers, with canonicalization, host-root containment checks, and symlink-traversal protection.
router: Route planner — recommends an executor for a packet based on workflow configuration and safety results, with fallback chains and logging.
safety: Safety gate — enforces 9 safety rules (SG-1 through SG-9) including contradiction detection, human-approval requirements, private-path protection, workflow validation, mode validation, note isolation, required-ref checking, and critical-risk dual-packet approval.
validator: Validator — two-stage validation: syntax validation checks structural completeness (required fields present, types correct) and schema validation checks semantic legality (workflow in registry, mode allowed, risk level valid).