Daniel's got a whole thing this week about what he's calling a pure AI inference server, and the first thing he wants to do is kill the name "AI server" because he thinks it's misleading. His argument is that most people hear that and picture a box with a chat interface, some agent frameworks, ComfyUI, Open WebUI — the whole application layer living right on the machine. What he wants instead is a machine whose only job is to run models. Weights stored locally, inference engines on top, everything served out over an API, accessible through Tailscale or a Cloudflare tunnel. No UI, no MCP servers, no clients, no agents. All of that lives on other machines and talks to this box. The applications are ephemeral and swappable. The inference layer is stable infrastructure you set up once and mostly leave alone. And he wants us to walk through what it actually takes to build that properly, starting with the engine layer, then the multi-model problem, concurrency and batching, weights management, the OS and container decisions, the API surface, exposure and auth, and how VRAM decisions flow from software choices rather than the other way round. He wants this to end up as something close to a considered build spec — the choices, the order to make them in, where the sharp edges are. Not a shopping list.
The name thing is right and it's doing more damage than he even says. I've watched three different people in the past month build what they called an AI server and every single one of them ended up with a machine that's simultaneously trying to be a database, a web server, a chat frontend, and an inference box, and then they're surprised when it falls over. The decoupling argument is the whole thing. Inference is a compute workload. It should look like a database — you connect to it, you query it, you get results back, and you never think about what's happening inside the box.
The database analogy is good. Nobody runs their web app on the Postgres machine.
Right. And nobody would. So the first real decision is the engine layer, and Daniel's list is the right one — vLLM, llama.cpp server mode, Ollama, TGI, SGLang, TensorRT-LLM. But those are not six versions of the same thing. They split pretty cleanly into two camps, and the split is whether the thing was designed from the ground up as a serving layer or whether it's a model runner that happens to expose a port.
And which is which?
vLLM and SGLang are the two that were built for serving from day one. vLLM came out of Berkeley with PagedAttention, which is this memory management technique that treats the KV cache the way an operating system treats virtual memory — it pages it, so you're not allocating a giant contiguous block per sequence. That means you can handle way more concurrent requests on the same hardware because the memory fragmentation that kills throughput on naive implementations just... doesn't happen. They've since added prefix caching, chunked prefill, speculative decoding. This is a serious piece of infrastructure. SGLang came out of Stanford and their big contribution is RadixAttention, which is a different approach to caching that automatically reuses KV cache across requests that share a prefix — which in practice is almost all of them, because system prompts are long and they're the same every time. Those two are in a league of their own for throughput on concurrent workloads.
And the other camp?
Llama.cpp server mode and Ollama. Llama.cpp is an incredible project — it's the reason you can run models on a laptop CPU and get usable performance. The server mode exposes an OpenAI-compatible API, and for single-user or low-concurrency scenarios it's fine. But it was not architected as a serving engine. The batching is simpler, the memory management is not PagedAttention-level sophisticated, and under real concurrent load it shows. Ollama sits on top of llama.cpp — it wraps it, adds model management, adds a nicer CLI, but under the hood it's the same engine with the same limitations. It's a convenience wrapper, and for what Daniel's describing — a machine whose entire purpose is to serve multiple models to multiple consumers — it is the wrong tool.
So Ollama is the thing people reach for because it's easy, and then they're confused about why their inference server can't handle more than two people at once.
And the Ollama team has never claimed otherwise — they built a great tool for running models locally. The community turned it into a server and then got frustrated when it wasn't one. TGI sits somewhere in the middle. HuggingFace built it, it's got proper continuous batching, it's got watermarking for detecting generated text, it supports a wide range of model architectures. It's more of a serving engine than llama.cpp but it's less battle-tested at scale than vLLM. The development pace has been... uneven. And TensorRT-LLM is the NVIDIA option — it's the fastest thing you can run on NVIDIA hardware, full stop, but you pay for that speed in complexity. You're building engines for specific model architectures, the build process is involved, and the documentation assumes you already know what you're doing.
So if Daniel's building a pure inference server, the engine choice is vLLM or SGLang, and everything else is a compromise he accepts for reasons other than serving performance.
For the general-purpose LLM, yes. But here's where the multi-model problem makes it interesting. He's not just running one model. He wants Whisper, possibly an image generation model, a general LLM, maybe an embedding model, maybe a model backing an agent. Those don't all run on vLLM. Whisper doesn't. Image generation models don't. So you're already in a world of multiple engines, and the question is how you colocate them.
Which I think is the crux of what he's asking. How do you host all of these on one machine without them stepping on each other?
The honest answer is that it's still a mess and most people are hand-rolling it. But let me walk through the actual mechanics. You've got one GPU — or maybe two, but let's say one for simplicity. You've got some amount of VRAM, call it twenty-four gigs on a 4090 or forty-eight on an A6000. Each model you load consumes a chunk of that VRAM for the weights, plus additional memory for the KV cache per concurrent request. If you keep everything resident, you need enough VRAM to hold all the models simultaneously. That's the simple case — you load them all, they stay loaded, and you just route requests to the right engine.
And if you don't have enough VRAM for that?
Then you swap. A model gets unloaded from VRAM, another one gets loaded in, and the question is how long that takes and what it costs. Loading a seven-billion-parameter model at four-bit quantisation from an NVMe drive takes... on the order of five to ten seconds. A seventy-billion-parameter model might take thirty to forty seconds. That's the cold-start latency your request is eating. If you're swapping on every request, your inference server is basically unusable for interactive work. So the practical answer is you keep your high-traffic models resident and you accept that the low-traffic ones will have a cold-start penalty, or you buy enough VRAM to hold everything.
And the orchestration layer for this? Is there something that handles swap scheduling sensibly?
Not really, and that's the gap. There are projects that nibble at the edges — there's a thing called Aphrodite that was forked from vLLM and adds some multi-model management features, there are Kubernetes operators that handle GPU workload scheduling. But for a single machine running multiple inference engines, the tooling is thin. Most people end up writing a small controller that watches which models are loaded, tracks request patterns, and makes swap decisions. It's not a solved problem. Daniel's instinct that everyone is hand-rolling this is correct.
So you're essentially building a small scheduler yourself, or you're overprovisioning VRAM to avoid the problem entirely.
And that's the software decision that determines the hardware bill, which is exactly what he said. If you decide you need Whisper resident, an embedding model resident, and a general LLM resident, you add up their VRAM footprints at your chosen quantisation level, add your KV cache budget for the concurrency you expect, and that's your minimum VRAM. The hardware follows the software architecture, not the other way round.
Let's talk about that concurrency and batching, because he raised the tension between interactive requests and batch jobs sharing the same hardware.
This is where the engine choice really matters. vLLM and SGLang both implement continuous batching, which means they don't wait for a batch to fill up before running inference — they dynamically add and remove requests from the batch as they arrive and complete. That's the key to handling interactive traffic without destroying latency. Under the hood, the engine is constantly assembling the largest batch it can without making anyone wait too long. You can tune the maximum batch size and the maximum wait time, and the engine figures out the rest.
And batch jobs — the "here's a pile of work, I don't care about latency" case?
That's a different scheduling problem. If you just throw batch work into the same queue as interactive requests, the batch work will consume all the available throughput and interactive latency will spike. The right answer is some form of priority queueing or separate request queues with weighted scheduling. vLLM doesn't have native priority classes — you'd need to run separate engine instances or put something in front. Or you handle it at the application layer: your batch client submits work at a lower rate, or only during off-peak hours, or you run a dedicated batch engine instance on a portion of the GPU. For a single machine, the simplest approach is time-based — batch work runs when interactive traffic is low, and you throttle it when interactive requests arrive. It's crude but it works.
And the thing in front — is that a gateway? A router?
That's the API surface question, and it connects to what he asked about protocols. Most of these engines expose an OpenAI-compatible API. vLLM does, SGLang does, llama.cpp server mode does, Ollama does. That's the de facto standard — the chat completions endpoint, the models list endpoint. But Whisper doesn't speak that protocol. Image generation models use their own thing — typically something like the Stable Diffusion web UI API or a custom gRPC interface. Embedding models usually speak OpenAI-compatible too, but not always. So even if you standardise on the OpenAI API format for LLMs, you still have protocol diversity across model types, and something needs to normalise that.
So you do need a gateway.
You need something. LiteLLM is the most common choice — it's a proxy that sits in front of multiple backends and presents a unified OpenAI-compatible API. You configure it with the models you have, where they live, and it routes requests. It also handles load balancing, fallbacks, and it gives you observability — request logs, token counts, latency metrics. For Daniel's setup, LiteLLM or something like it is probably the right call. It's not heavy, it runs as a container, and it solves the protocol normalisation problem without adding much complexity.
What about the weights management problem? He's right that nobody writes about this.
It's a genuine gap. The happy path is you download weights from HuggingFace once and they sit there forever. The reality is models get updated — bug fixes, new quantisations, fine-tuned variants — and you want to pull those updates without breaking anything. The tooling for this is basically nonexistent. HuggingFace Hub has a Python library that lets you download models programmatically, and you can wrap that in a cron job or a systemd timer that checks for new versions. But versioning is loose — most repos just use commit hashes, and "latest" is a moving target. Rolling back means keeping the old weights around and pointing the engine at a different directory. Avoiding a download eating your disk means checking available space before pulling. Avoiding swapping a model out from under a running workload means... you don't. You download to a staging directory, verify the download, then do a cutover when the engine is between requests, or you accept a brief outage.
None of that is automatable in a clean way without building it yourself.
Not cleanly, no. There are projects that touch on it — HuggingFace has a thing called TGI that includes some model update handling, and there are MLOps platforms that do model registry and deployment. But for a single inference server, the tooling is "write a script." It's one of those problems that's simple enough that nobody's built a product for it and annoying enough that everyone who does it themselves spends a weekend on it.
The OS and container question, then. He asked bare metal Linux versus containers, which distribution, how drivers constrain the choice.
The driver constraint is the binding one. If you're on NVIDIA hardware, you need the NVIDIA driver and CUDA toolkit, and those have specific kernel and distribution compatibility requirements. NVIDIA officially supports RHEL, Ubuntu LTS, and SUSE. Unofficially, they work on most things, but if you want the path of least resistance, Ubuntu LTS is the default answer and it's the default for good reason — it's what NVIDIA tests against, it's what most of the tooling assumes, and when something breaks, you'll find forum posts from people who had the same problem on the same distribution. ROCm for AMD is more restrictive — effectively Ubuntu and RHEL, with a strong preference for specific point releases.
So Ubuntu LTS, and then the question is whether you containerise the engines or run them on the host.
GPU passthrough to containers used to be painful. It's not anymore. The NVIDIA Container Toolkit handles exposing the GPU to Docker containers cleanly — you pass the gpus flag, the driver and CUDA libraries are mounted into the container, and it works. The advantage of containerising is that different engines have different Python dependency trees and they will conflict if you install them all on the host. vLLM wants one version of PyTorch, something else wants another, and suddenly you're in dependency hell. Containers solve that. Each engine gets its own image, its own environment, and they don't step on each other.
And the orchestration — Docker Compose versus Kubernetes on a single node?
Kubernetes on a single node is overkill for this and Daniel knows it. Docker Compose is the right level of abstraction. You define your services — vLLM container, Whisper container, LiteLLM container — you wire up the ports and the GPU access, and you're done. Systemd units are the even simpler option if you don't want the container overhead — you run each engine as a systemd service with a virtual environment, and you manage them with systemctl. That's lighter weight, but you lose the environment isolation. For a machine that's going to run for years without much intervention, I'd lean toward Docker Compose. The reproducibility is worth the small overhead.
And the exposure question. Tailscale versus Cloudflare tunnel versus LAN-only.
LAN-only is the simplest and the most secure, and if all your consumers are on the same network, that's the right answer. You bind the engines to localhost or the LAN interface and you're done. Tailscale is the next step up — it gives you a WireGuard mesh network with each machine getting a stable IP address and DNS name, and the traffic is end-to-end encrypted. For a single person accessing their inference server from a laptop when they're not at home, Tailscale is basically perfect. Cloudflare Tunnel is the option when you need to expose the server to the public internet without opening ports — it creates an outbound connection to Cloudflare's edge and traffic comes in through that. The tradeoff is that Cloudflare terminates TLS, so they can see the traffic in plaintext. For an inference API where the prompts and responses might be sensitive, that matters.
And authentication?
API keys. Every engine that exposes an OpenAI-compatible API supports API key authentication, usually as a simple pre-shared key. You set an environment variable with a long random string, and clients have to include it in the Authorization header. It's not sophisticated, but for a single-user or small-team inference server it's adequate. If you're exposing it more broadly, you'd put something more robust in front — Authentik, Authelia, or just Tailscale's built-in ACLs if you're using Tailscale. Tailscale ACLs are actually really good for this — you can tag specific machines and define which services they can reach, and it's all enforced at the network layer before traffic even hits the application.
Let's go back to VRAM from the software angle, because he specifically asked for that. How much you actually need.
Let me build it up. A seven-billion-parameter model at four-bit quantisation takes about four gigs of VRAM for the weights. A Whisper large-v3 model takes about three gigs. An embedding model like BGE-M3 takes maybe a gig. That's eight gigs for the models themselves. Then you need KV cache for concurrent requests. A rough rule of thumb is about one to two gigs per concurrent user for a seven-billion-parameter model, depending on context length. If you want to handle four concurrent users with reasonable context windows, that's another four to eight gigs. So you're at twelve to sixteen gigs before any headroom. A twenty-four-gig card handles that comfortably. If you step up to a seventy-billion-parameter model, the weights alone at four-bit are about forty gigs, and now you're looking at a forty-eight-gig card minimum, and that's before KV cache.
So the quantisation decision is really a VRAM budget decision dressed up as a quality decision.
It's both, but the budget drives it. Four-bit quantisation is the sweet spot for most use cases — the quality loss is barely measurable on benchmarks, and the memory savings are enormous. Eight-bit is higher quality but you need roughly double the VRAM. FP16 is the reference quality but the memory requirements are impractical for anything but the smallest models. The decision tree is: what models do you need resident, what quantisation can you tolerate, how much concurrency do you need, and that gives you a VRAM number. Then you buy the card that meets it.
If Daniel were building this today, what's the sequence of decisions?
Start with the models. Decide what needs to be resident — probably Whisper, an embedding model, and a general LLM at minimum. Pick the LLM size based on the quality you need — seven billion is the workhorse, seventy billion if you need more reasoning capability. That gives you the VRAM requirement. Buy the GPU. Install Ubuntu LTS, the NVIDIA driver and CUDA toolkit, and Docker. Containerise vLLM for the LLM, a separate container for Whisper, a separate container for the embedding model. Put LiteLLM in front as the API gateway. Configure each engine with an API key. Expose LiteLLM over Tailscale if you need remote access, LAN-only otherwise. Write a small script for weights updates — cron job that checks HuggingFace, downloads to staging, cuts over during low-traffic hours. That's the core of it.
And the sharp edges?
The multi-engine orchestration is the sharpest one. There's no off-the-shelf solution for colocating vLLM, a Whisper engine, and an embedding model on one GPU and managing their VRAM. You're either overprovisioning VRAM so everything fits, or you're writing a scheduler. The second sharp edge is the weights update problem — the tooling is thin and you'll end up scripting it yourself. The third is that when something breaks at the intersection of the NVIDIA driver, the container toolkit, and a specific engine version, the error messages are opaque and the fix is usually a specific combination of versions that someone figured out and posted on a GitHub issue.
The NVIDIA driver version roulette.
It's real. And the fourth sharp edge is that the API compatibility isn't as uniform as it looks. The OpenAI-compatible endpoints mostly work the same way, but there are edge cases — streaming tokens, function calling, logprobs — where engines diverge. LiteLLM papers over a lot of that, but not all of it.
Hilbert: The NVIDIA driver version isn't the problem. The problem is CUDA toolkit eleven point eight versus twelve point one and the fact that half the engines in that list pinned themselves to one or the other and never updated.
That's... actually more specific.
Hilbert: I ran a render farm for an architectural visualisation firm in two thousand eight. Twelve machines, each with two GPUs, and we had exactly this problem with the rendering engines. One wanted CUDA ten, one wanted eleven, and they couldn't coexist on the same driver. We ended up dedicating machines to specific engine versions. The inference people are heading for the same wall.
The containerisation solves that, though. Each engine gets its own image with its own CUDA version.
Hilbert: Until the container toolkit itself has a minimum driver requirement and you need a driver that supports both CUDA eleven point eight and twelve point one containers simultaneously. Which works now, mostly. It didn't in two thousand nineteen.
The sharp edge is real but it's been sanded down somewhat.
Hilbert: It's been sanded down until the next major CUDA release breaks backward compatibility again. NVIDIA does it every three years like clockwork.
That's actually worth flagging for Daniel's setup. If he pins the driver and the container toolkit versions at the start and doesn't touch them, he'll be fine. The problems start when you try to stay current.
Hilbert: The weights thing he asked about. In two thousand nine we had a similar problem with texture libraries — new versions would drop, artists would pull them, and suddenly a render would look different and nobody knew why. We solved it by never updating anything that was currently in a project. New version went to a staging directory, got validated against a test scene, and only got promoted when the project was done. The inference people need the same thing and nobody's built it.
A model release pipeline. That's essentially what he's describing.
Hilbert: It's not complicated. It's just boring, and boring things don't get GitHub stars.
The weights management gap in a sentence.
One thing I keep wondering about — he said the applications are ephemeral and you swap them constantly, but the inference layer is stable. Is that actually true in practice, or do the applications drive requirements back down into the inference layer?
They do, and that's the tension. An agent framework wants function calling support. A chat interface wants streaming. A batch processing pipeline wants maximum throughput and doesn't care about latency. Those are different performance profiles, and the engine configuration that's optimal for one is suboptimal for another. You can tune for the common case — probably interactive chat with streaming — and accept that batch jobs get whatever's left, or you run separate engine instances with different configurations. But separate instances means partitioning your VRAM, and now you're back to the multi-model colocation problem.
The decoupling is real at the architectural level, but the applications still shape the inference layer's configuration even if they don't live on the same machine.
They shape it, but they don't own it. And that's the distinction that matters. When the applications live on the inference server, you reconfigure the server every time you change an application. When they're decoupled, the inference layer has a stable configuration that serves multiple applications, and you accept that no single application gets a perfectly tuned backend. The tradeoff is configurability versus stability, and for what Daniel's describing, stability wins.
The misconception people have about this whole setup — what's the one thing most people get wrong?
That Ollama is a serving engine. It's not. It's a model runner with a convenience layer, and for single-user local inference it's great. For a multi-model, multi-user inference server, it's the wrong foundation. The right answer is vLLM or SGLang for LLMs, plus dedicated engines for other model types, with something like LiteLLM normalising the API surface in front.
The thing I'll be watching is whether the orchestration layer for multi-model colocation actually materialises as a product or whether it stays in the realm of hand-rolled scripts. Daniel's right that it's the crux of the whole thing, and right now the answer is "build it yourself."
The other thing worth watching is whether the engine developers start treating multi-model serving as a first-class problem. Right now each engine assumes it's the only thing on the GPU. That assumption is going to break as more people build boxes like this.
This has been My Weird Prompts. Thanks to our producer Hilbert Flumingtop.
Find us at my weird prompts dot com or email us at show at my weird prompts dot com.
We'll be back soon.
