Sentinel

What it is

Sentinel turns whatever machines I have into one pool for running local models. A daemon runs on each machine and fronts that machine’s Ollama. The daemons find each other on the network, elect a leader, and the leader serves one API that any OpenAI-compatible client can point at. Where a request actually runs is the fleet’s problem, not the client’s.

Why I built it

Local models live wherever the GPU memory is. Pointing a tool at one specific host works until that machine sleeps, gets rebooted, or the model moves somewhere else, and then every config that hardcoded the host is wrong. I wanted the opposite arrangement: machines with unpredictable lifetimes offer capacity, consumers use the pool, and nobody tracks which box is up.

How it works

The diagram above follows one chat request. Daemons discover each other over mDNS and run a modified Bully election where the best hardware wins. Winning switches on an embedded gateway inside that same daemon (registry, scheduler, router, and dispatch), so there’s no separate control plane to install. Every daemon, the leader included, registers with the current gateway and heartbeats.

When a request arrives, the router scores each node on whether the model is already loaded, free VRAM, load headroom, and active jobs, then dispatches over gRPC. Tokens stream back to the client as server-sent events. A sleeping machine can be woken with Wake-on-LAN, and the leader exposes Prometheus metrics so the fleet shows up on the same dashboards as everything else.

Decisions

Failover by reconstruction, not replication. There’s no Raft and no replicated log. A running job survives a leader crash because the node executing it reports it to the new leader on registration, with its fencing epoch, and the new leader resumes polling. The cost is explicit: a job that was queued but not started dies with the leader. The submit response says so, which makes “accepted” mean “queued”, not “guaranteed”.

Fencing epochs on every assignment. The gateway bumps a job’s epoch each time it assigns it. A daemon rejects a submission with an older epoch than the one it’s running, so a stale leader can’t double-dispatch work.

At-most-once where re-running is dangerous. Shell jobs are never re-run after a node failure unless they’re marked idempotent. Inference retries are capped at three. A job absorbed from a failover report carries no payload, so it can finish but can’t be silently re-run empty.

Terminal states need evidence. Finished results stay on the producing node until the gateway acknowledges them, and a poll error is recorded as unreachable, never as completed.

In use

Version 0.3.0 has served my local models since September 2026. One open item, stated plainly: tool calls pass through the API, but I haven’t verified any local model against the tool-call contract yet. That compatibility work is in progress.

API example

An illustrative session against a leader on the local network (annotated, not a captured transcript; the hostname and model are examples):

# Any OpenAI-style client works; this is the raw request
$ curl -N http://sentinel.local:50053/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{"model": "llama3.1:8b", "stream": true,
         "messages": [{"role": "user", "content": "Summarize this log line"}]}'
data: {"choices":[{"delta":{"content":"The"}}]}
...

# Which daemon is leading right now
$ curl http://sentinel.local:50053/api/leader

# Nodes, free VRAM, and job counts across the fleet
$ curl http://sentinel.local:50053/api/cluster

# Wake a sleeping node from its stored MAC address
$ curl -X POST http://sentinel.local:50053/api/nodes/wake/node-c

The repository is private, so there’s no source link here.