Building local LLM stacks with Docker: what enterprise teams need to know
A developer's detailed walkthrough of building a containerized GenAI chatbot highlights the practical challenges enterprise teams face when deploying local LLM infrastructure.
The setup uses Docker Compose to orchestrate four services: Ollama for LLM inference, FastAPI for the backend, ChromaDB for vector storage, and Streamlit for the UI. This mirrors real-world AI platforms where separation of concerns matters.
What's worth noting
The architecture itself is standard microservices. The interesting part is the documented errors and fixes:
Network configuration: Containers must reference each other by service name (http://ollama:11434), not localhost. This trips up developers used to local development.
Model persistence: Ollama models don't automatically exist in containers. They need explicit pulling (docker exec -it container ollama pull mistral) and volume mounts for persistence (-v ollama:/root/.ollama).
Port conflicts: Host-level Ollama installations (common via Snap on Ubuntu) block container ports. The fix requires stopping host services or remapping ports.
GPU passthrough: While not covered in depth here, production deployments need --gpus=all flags and NVIDIA Container Toolkit for acceptable inference performance.
Why this matters
Local LLM deployment is moving from hobby projects to enterprise consideration. Privacy requirements, API cost control, and vendor independence are driving interest.
Docker provides reproducible environments, but the gap between development tutorials and production-ready systems is real. Hardware requirements start at 4 CPU cores and 8GB RAM, but realistic workloads need more. Storage is per-model: base images under 1GB, models add gigabytes.
The trade-offs are clear. Docker adds operational overhead (storage, memory limits, networking complexity) but delivers isolation and portability. For teams evaluating local LLM infrastructure, these documented friction points are more valuable than the happy path.
Ollama's been shipping production-ready Docker images since October 2023. ChromaDB supports persistent volumes. The tooling exists. The question is whether your use case justifies the operational complexity versus cloud APIs.
History suggests: start simple, containerize when complexity justifies it, not before.