A model provider on a shelf

A provider is a URL, a key, a model name.

When your apps call OpenAI or Anthropic, everything hard hides behind one clean endpoint. Weights, serving software, GPU memory, routing — all invisible. The question that made this project interesting: could a box on a shelf behave the same way?

128GB of unified memory, and everything providers hide.

NVIDIA GB10 Grace Blackwell, Ubuntu on ARM64, a headline claim of 1 petaFLOP at FP4 — a qualifier that matters. Real performance depends on the model, the runtime, the quantization, and how much runs at once.

No benchmark numbers here on purpose: none have been captured yet, and we'd rather say that plainly.

vLLM proved it. llama.cpp runs it.

vLLM served the first model and proved the box could host a real API service. The main path moved to llama.cpp because the Qwen model we wanted ships as a quantized GGUF file — trading precision for memory and speed on a 128GB box.

local-reasoning · gpt-oss-20b — then qwen36-llamacpp · Qwen3.6-35B Q4

LiteLLM is what makes it a provider.

Clients never talk to a raw model server. LiteLLM presents one stable OpenAI-compatible /v1 with model aliases and virtual keys. Swap the backend behind an alias and nothing downstream breaks.

base_url + api_key + model — the same three lines the team already writes for OpenAI.

Operator, team, public — never straight to the backend.

SSH over Tailscale to operate it. Tailscale Serve for the team to use it privately. ngrok when a demo needs public access — and public always goes through the gateway, so keys stay enforced and revocable.

Agents plug in — and the meter stops.

OpenCode and Nanoclaw wired straight into the GX10 through the standard /v1/chat/completions path, like any other provider. And the marginal cost of one more prompt, one more agent loop, stops feeling counted. That changes how you experiment more than any benchmark would.