newt-inference

Newt-Agent inference layer: backend trait + local Ollama/vLLM + provider-plugin host.

• backend::InferenceBackend is the trait every backend implements.
• local::LocalOllamaBackend and local::LocalVllmBackend are built-in HTTP backends.
• embedded::EmbeddedBackend runs a local quantized Qwen2 model in process with the embedded feature, enabled by the workspace CLI by default.
• provider_plugin::ProviderPluginBackend spawns a subprocess speaking the Newt-Provider JSON-RPC protocol — how OpenAI, Anthropic, etc. join via opt-in plugin installs.

Also provides the BackendRegistry and retry/backoff helpers used by the rest of the workspace.

Embedded generation admits one worker per process; busy requests fail without queuing another model load. complete_with_timeout bounds the async request, including loading and generation. Timeout or caller cancellation signals the worker cooperatively, which retains its slot until it exits. A synchronous load or forward may still finish after the caller returns. Single-token prefill gives the pinned Candle Qwen2 engine cancellation checkpoints without an incompatible multi-token cache mask, trading batch throughput for responsiveness.

smart_harness::build creates an auxiliary callback and a credential-free manifest for both CLI and TUI hosts. The default requires an installed embedded model and pins its generation device through EmbeddedBackend::new_cpu. That constructor captures immutable weights and tokenizer bytes; the manifest records their raw CIDs and generation reads those same bytes even if the original paths change. The immutable model bytes remain resident while the backend lives. The embedded auxiliary remains CPU-only. An external override requires an explicit endpoint, model, protocol, and nonempty placement label (cpu, cuda, ...). Its manifest records placement_evidence = "operator-declared"; the client does not verify the server's hardware. The override may reuse the primary endpoint and model, with shares_primary_origin recording whether their URL origins match. Calls use the configured timeout and output-token limit, with transport retries disabled. Separate auxiliary budgets do not guarantee physical independence or prevent contention on shared servers or hardware. An unavailable auxiliary returns an error without silently selecting another backend. Optional adjudication.system_instruction precedes the task prompt as a system-role message. It defaults to empty and applies to both navigation and classification; task-specific contracts remain in each user prompt. The host counts both messages against its input-byte budget.

The classifier comparison runs the production auxiliary prompt/parser against curated fixtures and the deterministic baseline, recording failures and timings without a quality claim. The external model's 8/8 result reported in PR #2263 covers eight curated fixtures and does not establish general classification quality or superiority. Smart classification remains experimental.

Part of Newt-Agent, a free, friendly, local agentic coder.

License

Apache-2.0