Memory-pressure protection

The metal-agent watches host memory and surfaces conditions on every workload, but only stops a process when LLMKube itself is the cause of the pressure.

Why this exists

Apple Silicon and bare-metal Linux hosts don't expose a Kubernetes-managed memory model for native LLM processes. Without protection, an oversized model or a runaway warmup can swap-thrash the entire box and pull unrelated workloads down with it. LLMKube's metal-agent runs a watchdog that observes host memory, surfaces a MemoryPressure condition on every managed InferenceService, and (when enabled) evicts the lowest-priority workload before the kernel does.

Three llama.cpp services on a kind cluster, one with evictionProtection: true. Watch the watchdog hit Critical, surface conditions on every workload, and explicitly hold fire.

Real-time recording on a kind cluster, idle waits compressed.

What the cast shows

The recording layers three behaviors on top of each other:

Pressure detection. A 5-second-tick watchdog samples host memory and per-process RSS. When the available fraction drops below the configured threshold, it transitions to Warning or Critical and updates the llmkube_metal_agent_memory_pressure_level gauge.
Visibility on the workload. Every managed InferenceService receives a MemoryPressure status condition, including services that opted out of eviction. Operators can read it from kubectl get isvc instead of needing Prometheus.
Friendly-fire guard. Even at Critical, the watchdog refuses to evict if LLMKube isn't the dominant memory consumer (default: it must own >50% of system RSS). The decision is recorded in llmkube_metal_agent_evictions_skipped_total{reason="below_guard"} so you can alert on "the system is in distress, but it's not us."

Eviction selector rules

When the watchdog does decide to evict, the selector applies these rules in order:

Safety floor. If only one managed process exists, refuse to evict. Single-tenant systems shouldn't be killed by their own watchdog.
Skip non-llama-server runtimes. oMLX and Ollama use shared daemons; killing them via this code path doesn't free per-model memory.
Skip spec.evictionProtection: true workloads. Per-service opt-out for production-critical inference.
Lowest spec.priority wins. The same enum used for GPU scheduling: batch < low < normal < high < critical.
Tie-break by largest RSS, then by oldest StartedAt for determinism.

Evicted services are stamped into a respawn block until pressure clears, so the controller can't immediately re-spawn the workload the watchdog just killed. The block lifts automatically when the agent observes Normal pressure.

Enabling protection

Eviction is opt-in. The watchdog runs and surfaces conditions by default; --eviction-enabled activates the kill path.

These are flags on the metal-agent itself, not Helm values. The agent reads them from the ProgramArguments array of its launchd plist, ~/Library/LaunchAgents/com.llmkube.metal-agent.plist. Add them after the arguments already there:

<key>ProgramArguments</key>
<array>
    <string>/usr/local/bin/llmkube-metal-agent</string>
    <!-- ...the arguments already in the plist... -->
    <string>--eviction-enabled</string>
    <string>--memory-pressure-critical</string>
    <string>0.10</string>
    <string>--memory-pressure-warning</string>
    <string>0.20</string>
</array>

Reload the agent so launchd picks up the edited plist. Engines keep serving across the reload and the new agent adopts them.

launchctl bootout gui/$(id -u)/com.llmkube.metal-agent
while launchctl print gui/$(id -u)/com.llmkube.metal-agent >/dev/null 2>&1; do sleep 1; done
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.llmkube.metal-agent.plist

make install-metal-agent rewrites the installed plist from the template on every run, so re-add these flags after a reinstall. The defaults are 0.10 (critical) and 0.20 (warning) of available memory; eviction stays off until --eviction-enabled is set.

Mark a specific InferenceService as never-evict:

apiVersion: inference.llmkube.dev/v1alpha1
kind: InferenceService
metadata:
  name: chat-prod
spec:
  modelRef: qwen3-8b
  priority: critical
  evictionProtection: true

Metrics worth alerting on

llmkube_metal_agent_memory_pressure_level: Gauge: 0 = Normal, 1 = Warning, 2 = Critical. Page on Critical = "host memory is under stress."
llmkube_metal_agent_evictions_total: Counter: workloads stopped by the watchdog. Non-zero rate means the agent is actively pruning under pressure.
llmkube_metal_agent_evictions_skipped_total{reason}: Counter: tracks why an eviction was skipped. Labels include disabled, below_guard, empty, floor, all_protected, runtime_ineligible.
llmkube_metal_agent_process_rss_bytes{name}: Gauge: per-process resident set size. Useful for capacity planning and spotting the model that's growing.