Expand description
§Dynamo LLM
The dynamo.llm crate is a Rust library that provides a set of traits and types for building
distributed LLM inference solutions.
Modules§
- backend
- Backend
- common
- disagg_
router - discovery
- endpoint_
type - engines
- entrypoint
- The entrypoint module provides tools to build a Dynamo runner.
- gguf
- grpc
- http
- hub
- kv_
router - local_
model - migration
- mocker
- model_
card - Model Deployment Card
- model_
type - namespace
- perf
- Performance recording and analysis for streaming LLM responses
- preprocessor
- The Preprocessor consists of the following modules
- protocols
- Dynamo LLM Protocols
- recorder
- request_
template - tokenizers
- tokens
- Types and utilities for handling sequences of tokens, including block creation and hashing.
- types
- utils
Functions§
- file_
json_ field - Reads a JSON file, extracts a specific field, and deserializes it into type T.
- log_
json_ err - Pretty-print the part of JSON that has an error.