Module streaming 

Streaming Memory Ingestion for Implicit Learning

Enables continuous memory formation from streaming data without explicit remember() calls. Designed for LLM agents, robotics, drones, and other autonomous systems that need to learn implicitly from their environment.

Architecture

WebSocket /api/stream
    │
    ▼
┌─────────────────────────────────────────────────────────────────┐
│  StreamingMemoryExtractor                                       │
│  ┌──────────────┐  ┌──────────────┐  ┌────────────────────────┐ │
│  │ Buffer       │→ │ Triggers     │→ │ Extraction Pipeline    │ │
│  │ (messages)   │  │ (time/event) │  │ (NER + dedup + store)  │ │
│  └──────────────┘  └──────────────┘  └────────────────────────┘ │
└─────────────────────────────────────────────────────────────────┘
    │
    ▼
Response stream: { memories_created, entities_detected, dedupe_skipped }

Supported Stream Modes

• Conversation: Agent dialogue with user (high semantic content)
• Sensor: IoT/robotics sensor readings (continuous, needs aggregation)
• Event: Discrete system events (logs, errors, state changes)

Extraction Triggers

• Time-based: Flush every N milliseconds (configurable checkpoint_interval_ms)
• Event-based: Flush on important events (errors, decisions, discoveries)
• Content-based: Flush when buffer semantic density exceeds threshold
• Manual: Explicit flush via { type: "flush" } message

Structs

BufferedMessage

Buffered message awaiting extraction

DetectedEntity

Entity detected during extraction

ExtractionConfig

Configuration for automatic memory extraction from streams

RelevanceBreakdown

Breakdown of composite relevance score components

SessionStats

Session statistics

StreamHandshake

WebSocket handshake message - sent by client to initialize stream

StreamSession

Per-session streaming state

StreamingMemoryExtractor

Streaming Memory Extractor - core processing engine

SurfacedStreamMemory

Memory surfaced during streaming context injection

Enums

ExtractionResult

Result of memory extraction - sent back to client

StreamMessage

Incoming stream message types

StreamMode

Stream processing modes - determines extraction behavior

Functions

contains_ignore_ascii_case

Case-insensitive substring search without allocation. Returns true if haystack contains needle (ASCII case-insensitive). needle MUST be lowercase ASCII for correct results.

content_hash

Quick hash for exact deduplication (public for benchmarking). Uses case-insensitive byte-level hashing with batched writes to minimize hasher call overhead.