Skip to main content

Crate tsift_local_model

Crate tsift_local_model 

Source

Structs§

GpuLeaseAcquisition
GpuLeaseConflict
GpuLeaseReap
GpuLeaseRecord
One held lease on a profile, written to the cooperative registry file.
GpuLeaseRegistry
File-backed cooperative registry: { version, leases: { profile_id: [record, ...] } }.
GpuLeaseRelease
GpuLeaseRenew
GpuProbe
GpuProcess
LocalModelLease
LocalModelLifecycleReport
LocalModelStatusReport
LocalModelSwapReport
ModelProfile
PreparedUnloadRequest
Pure projection of a ProviderUnloadAction into the request that will be sent. Separated from execute_unload_request so the model-tag override and body-rewrite logic is fully testable without a live HTTP server.
ProfileResolution
Result of resolving a ProfilePreference against the live GPU probe.
ProfileSelection
ProviderUnloadAction
UnloadActionResult
Outcome of dispatching a single ProviderUnloadAction. The planner owns execution (#kgunloadpost) so any caller — tsift kg unload, a future lease-drop hook, or the lifecycle swap path — gets the same POST behavior without re-implementing the HTTP fallback chain.
VramCleanupEvaluation

Enums§

ConcurrencyClass
GpuLeaseAcquisitionStatus
GpuLeaseReleaseOutcome
GpuLeaseRenewOutcome
LeaseMode
ModelRole
ProfilePreference
Caller-supplied preference for which local model profile a single call should use.
ProfileResolutionSource
How a resolved profile was chosen.
ProviderKind
SwapStatus
UnloadActionKind
UnloadStrategy
VramCleanupStatus

Constants§

DEFAULT_DESKTOP_RUNTIME_MARGIN_MIB
DEFAULT_IDLE_TTL_SECONDS
DEFAULT_LEASE_TTL_SECONDS
Default idle TTL (0 = no TTL-based staleness, only pid-dead pruning).
DEFAULT_LLAMA_CPP_ENDPOINT
Default llama.cpp router unload endpoint. Also the value llama-server listens on by default. Override via --provider-endpoint or the TSIFT_LLAMA_CPP_ENDPOINT env var when this port is taken (e.g. by a local WordPress instance at 8080).
DEFAULT_OLLAMA_ENDPOINT
Default Ollama generate endpoint. Override via --provider-endpoint or the TSIFT_OLLAMA_ENDPOINT env var.
DEFAULT_VLLM_ENDPOINT
Default vLLM sleep endpoint. Override via --provider-endpoint or the TSIFT_VLLM_ENDPOINT env var.
DEFAULT_VRAM_CLEANUP_TOLERANCE_MIB
LEASE_FILE_ENV_VAR
Environment variable override for the lease registry file path.
LEASE_REGISTRY_VERSION
Cooperative GPU lease registry file format version.
LLAMA_CPP_ENDPOINT_ENV_VAR
Env var override for the llama.cpp router unload endpoint.
OLLAMA_ENDPOINT_ENV_VAR
Env var override for the Ollama generate endpoint.
VLLM_ENDPOINT_ENV_VAR
Env var override for the vLLM sleep endpoint.

Functions§

acquire_lease
High-level acquire: read file, prune stale, apply, write file.
apply_acquire
Apply an acquire to the registry in place.
apply_release
Apply a release to the registry in place.
apply_renew
Apply a heartbeat renewal in place: slide a live holder’s heartbeat (acquired_at_unix_seconds) forward to now so its TTL window restarts.
build_lifecycle_report
build_local_model_lease
build_status_report
build_status_report_with_probe
build_swap_report
Build a combined swap report: source unload lifecycle + target resolution.
build_unload_actions
current_unix_seconds
default_model_profiles
evaluate_vram_cleanup
execute_unload_actions
Plan + execute a full unload action chain in one call. The default path for callers that don’t need to inspect the prepared request. Skips non-API actions (noop, process-exit) by reporting them as not executed.
execute_unload_request
Dispatch a single prepared request: POST the body, fall back to the subprocess command if the HTTP path fails. Impure — exercised end-to-end via the tsift kg unload smoke; unit tests cover prepare_unload_request and rewrite_unload_body_model instead.
format_lease_show_human
Human-readable summary of the lease registry.
format_lifecycle_human
format_status_human
is_pid_alive
Best-effort pid-liveness check via kill -0.
lease_mode_for_profile
normalize_unload_url
Normalize an unload endpoint into the canonical /api/generate path used by Ollama’s keep_alive:0 contract. Tolerates callers that supply the base host (http://host:11434) or the full generate URL.
prepare_unload_request
Project a planned action into a concrete HTTP request, rewriting the body with the resolved model tag. Returns None for non-API actions.
probe_nvidia_smi
profile_by_id
prune_stale_leases
Prune stale holders from the registry in place.
rank_profiles_for_role
read_lease_registry
Read the lease registry, returning an empty default when the file is missing.
reap_leases
Sweep stale holders (crashed pids or expired TTL) out of the registry and report which profiles dropped to zero live holders.
registry_lock_path
Sidecar advisory-lock path for a registry file (<registry>.lock).
release_lease
High-level release: read file, prune, drop this holder, write file.
renew_lease
High-level heartbeat: read file, slide this holder’s heartbeat, write file.
resolve_lease_file
Resolve the cooperative lease registry file path.
resolve_profile_preference
Resolve a caller preference to a concrete profile for a given role.
resolve_provider_endpoint
Resolve a provider endpoint for a given unload strategy.
rewrite_unload_body_model
Rewrite the model field of an unload action body to honor an explicit --model override. build_unload_actions formats the body with the profile’s model_ref, so without this rewrite the override would be silently ignored. Falls back to the original body if JSON parsing fails (defensive — the planner’s body templates are always valid JSON).
show_registry
Read the registry and return the pruned view. include_stale skips the pruning pass so the caller can inspect raw state for diagnostics.
unload_model_at
Fire-and-forget unload for callers that don’t have a full action plan — used by tsift kg smoke --unload and any future lease-drop hook that just needs to push a single model out of VRAM. Builds a minimal Ollama keep_alive:0 request and dispatches it.
write_lease_registry
Atomically write the lease registry (temp file + rename).