How to host an open-weight model

This year I moved from running cloud infrastructure to running models on it. These are the notes I wish I had on day one.

Three pieces

The model is a stateless calculator. Tokens in, tokens out.

The inference engine (vLLM, SGLang, Ollama) loads the model onto the GPU and serves it. It batches requests, manages the cache and streams tokens back.

The harness is the agent around the model. It holds the conversation, calls tools and decides what goes into each request.

Reading a model name

Names loosely follow author/name-version-size-precision. Size is the number of weights in billions: 9B, 27B. At 16-bit (BF16) each weight is 2 bytes, so about 2 GB per billion. A 27B model needs around 54 GB just to load. FP32 is full precision.

Precision is how many bits each weight is stored in, which is quantization.

W4A16 means 4-bit weights with 16-bit activations. NVFP4 is a 4-bit float with small per-block scales, native on NVIDIA Blackwell. The scales are why the files come out a bit over a quarter of the BF16 size.

Quantization costs a little precision per weight. It’s hard to notice except on hard reasoning, and big models absorb it better. A bigger model at 4 bits often beats a smaller one at 16-bit.

MoE (mixture of experts) models run only some of their weights per token, so they are faster than a dense model of the same total size. All the weights still have to fit in memory.

The file format usually picks the runtime. Safetensors goes to vLLM or SGLang. GGUF goes to llama.cpp, Ollama or LM Studio. MLX models are folders of safetensors for Apple’s MLX.

The weights get split evenly across the GPUs. With tensor parallelism the GPU count has to divide the model’s attention-head and KV-head counts. Those are usually powers of two and servers come in 2, 4 or 8 cards, so it’s usually 1, 2, 4 or 8. Pipeline parallelism allows other counts. Fewer GPUs is better if the model fits, though extra GPUs also grow the KV pool.

Where GPU memory goes

Three things: weights, compute space and the KV cache pool.

Weights are loaded once and shared by every request. Compute space is small and scales with max-num-batched-tokens; vLLM profiles it at startup. The KV pool is everything left over. Its size is fixed at startup and shared by all running requests. gpu-memory-utilization is the fraction of each GPU’s memory vLLM may use, and the default is around 0.9 (0.92 in the latest docs).

The pool decides how much work the server can do.

1. Every token a request holds takes a slice of the pool: prompt, tools, history, data and reply.
2. A request’s share grows token by token and stops at max-model-len. That value has to fit in the pool and within the length the model was trained on.
3. Pool divided by max-model-len is roughly how many full-size requests run at once, up to max-num-seqs. The rest wait. vLLM hands out the pool in blocks, so it’s approximate.
4. Storing the cache at 8 bits (kv-cache-dtype fp8) roughly doubles the pool. The quality cost is small on most models and larger on some, so test yours.
5. Long context isn’t equally good all the way through. Quality drops as it fills, and a long prompt means a slow first token.

The knobs

Tuning trades latency (how fast one user gets an answer) against throughput (how much the server handles in total). These are vLLM flag names.

Memory: gpu-memory-utilization, max-model-len, kv-cache-dtype. Scheduling: max-num-seqs and max-num-batched-tokens. Parallelism: tensor-parallel-size.

For more users, raise max-num-seqs, use --kv-cache-dtype fp8 and raise gpu-memory-utilization a bit. Tensor-parallel size is mostly decided by whether the model fits, not by latency.

Chunked prefill and prefix caching are on by default in current vLLM (V1). The flags matter on older versions or to turn them off. Prefix caching is the cheap win when many requests share the same start, like a system prompt or tool list. The engine computes it once and reuses it.

Measuring it

Three numbers: TTFT (time to first token), TPOT (time per output token) and TPS (tokens per second across all users).

Three runs are enough to start. A baseline with one user and a few dozen requests. A saturation run with the same prompt at rising concurrency, 2 up to 128, until throughput stops growing. Then mixed shapes: long prompts with short answers, and the reverse.

guidellm or the engine’s own benchmark script will do it. Whatever you use, check its numbers against a second tool once.

What the model can’t do for you

The harness resends the whole conversation every turn, so a longer session takes a bigger slice of the pool. Keeping it small is the harness’s job: summarise old turns, drop old tool output, or start fresh.