Linux Host Agent (experimental)
The Metal agent also starts on Linux, on arm64 and
amd64. It joins the cluster the same way a Mac does (a MetalNode, a Lease
heartbeat, the dial-back check and the in-cluster relay), and serves models
through an engine on the host.
Models served by a Linux host use hardware.accelerator: metal, like a Mac’s: metal is what sends a service to a host agent instead of in-cluster pods. On
Linux the agent logs Metal not available; Linux host support is experimental at startup, then whether it found a Hailo device node
(/dev/hailo*). Any other host without Metal still exits.
Tracking issue: #2481.
What has been tested
| Status | |
|---|---|
linux/arm64 agent, runtime: ollama with stock Ollama on CPU | Tested in kind: an agent container next to stock Ollama, joined by hand, reached Reachable=True and served chat completions through the relay |
| Linux arm64 host with a 16K-page kernel (Raspberry Pi 5) | The agent runs on Raspberry Pi OS’s 6.18.50+rpt-rpi-2712 kernel and served a kind cluster on the same LAN |
| linux/amd64 | Builds. Untested on hardware |
llama.cpp (llama-server) on Linux | Untested. The agent’s llama.cpp defaults (flash attention, mlock) are tuned for Apple Silicon |
llmkube metal join | Refuses to run off macOS. Join the host by hand |
MetalNode hardware | Not reported off macOS: kubectl get metalnode shows no memory or chip for a Linux host |
llmkube deploy for a Linux host | Untested. Write the Model and InferenceService by hand |
Get the agent
There is no Linux release archive yet. Build the agent from a checkout, for
the host’s architecture (arm64 or amd64):
GOOS=linux GOARCH=arm64 CGO_ENABLED=0 go build -o llmkube-metal-agent ./cmd/metal-agent The binary is static and needs nothing else from the host. Copy it to the
host and put it on PATH.
Keep hosts apart
Every agent watches one namespace (--namespace) and considers every metal service in it, so keep a Linux host’s services apart from other hosts’:
- Use a namespace of its own (recommended). Create the host’s identity in
that namespace (
llmkube metal token create edge-1 -n edge) and put its Models and InferenceServices there. No other agent sees them. - Or allowlist every agent in a shared namespace, Macs included, with
--inference-service-allowlist(comma-separated InferenceService names). An allowlist only limits the agent it is set on. An agent without one claims the Linux host’s services too: every agent registers the Ollama runtime, so another host running Ollama can quietly serve the registry’s copy of the same tag, and one without it refuses the service and writes status over the Linux host’s.
Making placement on a MetalNode explicit is proposed in #2481.
Join the host by hand
llmkube metal join runs only on macOS, so on Linux you create the host’s
identity as for a Mac, mint its token yourself and write its kubeconfig.
Where you have cluster access, create the namespace and the host’s identity.
This creates its ServiceAccount (llmkube-metal-<name>), RBAC, Lease and MetalNode, plus a join Secret:
kubectl create namespace edge
llmkube metal token create edge-1 -n edge Ignore the llmkube metal join line it prints. Delete the unused join Secret,
which revokes that join string, and mint the agent’s token directly:
kubectl -n edge delete secret llmkube-metal-edge-1-join
kubectl -n edge create token llmkube-metal-edge-1 --duration=168h > token
chmod 600 token Write a kubeconfig that reads that token file, with an API server address the
host can reach and the cluster’s CA. kubectl config view --raw --minify shows both for your current context; a kind cluster’s 127.0.0.1 address is
not reachable from another host.
apiVersion: v1
kind: Config
clusters:
- name: llmkube
cluster:
server: https://<api-server>:6443
certificate-authority-data: <base64 CA>
users:
- name: edge-1
user:
tokenFile: /path/to/token
contexts:
- name: edge-1
context: {cluster: llmkube, user: edge-1, namespace: edge}
current-context: edge-1 Copy both files to the host into a directory the agent’s user owns, readable only by that user (0600), and start the agent as that user (not root) under the supervisor of your choice, such as a systemd unit or a container. Renewal writes the new token to a temporary file in the token’s directory and renames it over the old one, so the agent’s user needs write access to that directory: in a container, mount the directory, not the token file alone.
llmkube-metal-agent --kubeconfig /path/to/kubeconfig --namespace edge --metal-node edge-1 --renew-service-account-token --runtime ollama --renew-service-account-token keeps the token file current: the agent
renews at half the token’s lifetime, which needs the API server. A host
switched off or off the network for longer than the token’s lifetime (7 days
with --duration=168h) always comes back with an expired token, and one away
for more than half of it may (less if the API server caps token lifetimes).
The agent then logs the agent's token has expired and cannot be renewed with the kubectl create token command to run; mint a new token as above and
write it over the token file.
The model store and state directory default to $XDG_DATA_HOME/llmkube/models and $XDG_DATA_HOME/llmkube/metal-agent (~/.local/share/llmkube/... when XDG_DATA_HOME is unset); --model-store and --state-dir override them. A model store under /tmp or /var/tmp is
refused.
The cluster reaches the agent on its ingress port (--ingress-port, 9443), so
open that port to the cluster in the host’s firewall. The MetalNode lists
the host’s LAN addresses first, then tailnet ones, for the dial-back to try;
with the API server on loopback (a kind cluster on the same host), Docker
Desktop’s 192.168.65.254 comes last, after them. Then wait for the MetalNode to report Reachable=True:
kubectl get metalnode edge-1 -n edge To remove the host, stop the agent and run llmkube metal remove edge-1 -n edge where you have cluster access.
Engines
As on a Mac, spec.runtime picks an InferenceService’s engine and the agent’s --runtime is the fallback. On Linux the agent serves only the two engines
below: it refuses to start when --runtime or a binary flag selects a
macOS-only engine (mlx-server, omlx, vllm-swift, tensorfold, or their --mlx-server-bin, --omlx-bin, --vllm-swift-bin, --tensorfold-bin), and
a service whose spec.runtime names one is refused with RuntimeNotAvailable.
- Ollama (
runtime: ollama): the agent talks to an Ollama-compatible daemon on--ollama-port(11434, Ollama’s default) and does not start it. Run the daemon under its own supervisor, listening on127.0.0.1(OLLAMA_HOST=127.0.0.1): the agent warns when it also answers on the host’s address, where it bypasses the ingress. - llama.cpp (
runtime: llamacpp, and the agent’s default--runtime llama-server): the agent startsllama-server, found in/opt/homebrew/bin,/usr/local/binor/usr/bin, or named with--llama-server. It is untested on Linux. Without llama-server the agent starts only when--runtimeselects another engine, and refuses a service that selects llama.cpp withRuntimeNotAvailable.
Memory
On Linux the agent reads /proc/meminfo, capped by the cgroup v2 limits from
its own cgroup up to the root: total memory is the smallest memory.max on
that path, and available memory the least room left under any of them. A
systemd MemoryMax= or a container --memory limit therefore sizes the
download cap, and the budget for engines the agent starts, which inherit its
cgroup.
Only cgroup v2 limits are read. On a host whose memory controller is on cgroup v1, as on a hybrid v1/v2 host, the agent warns once at startup that its memory figures are the host’s: it then admits engines against host memory, while the v1 limit can still OOM-kill them.
Ollama is a daemon the agent only talks to, running outside its cgroup, so runtime: ollama services are admitted against the host’s MemTotal and MemAvailable, whatever the agent’s own limit, and the memory watchdog never
picks one to evict. llmkube_metal_agent_memory_budget_bytes reports that
host-wide budget.
For a fit decision that does not depend on the machine, as in tests, set spec.hardware.memoryBudget on the Model. A budget larger than the one its
engine is admitted against (the cgroup’s, or the host’s for Ollama) is
clamped to it with a MemoryBudgetClamped Warning event, unless the agent
runs with --allow-model-memory-budget-override.