In this guide, I will show you how I set up a local, OpenAI-compatible llama-server in Docker under WSL2 with CUDA acceleration, taking you from the basic arrangement to the first working request.
The finished setup runs llama-server in a container, uses the GPU through WSL2, and is configured with exactly one slot, -np 1, and at least 128k context. I need that size from my own experience because I run a coding agent such as OpenCode against it: smaller contexts trigger compaction too often and make the work tedious. A larger context would be nice, but it no longer fits into the VRAM of my setup. It is meant for one local workstation, not for a multi-user server where several people send requests at the same time.
Any client that speaks the OpenAI-compatible API can call the resulting endpoint, while the models run locally on your own machine instead of in […]