Architecture decision
Specify model revision, precision, context and concurrency. Test response latency with real prompts before sizing production capacity.
GLOBALATLAS AI · LLM hosting
LLM hosting
A model service needs weights in memory, runtime buffers, a KV cache and an application endpoint.
From a user request to model output
- 01
Application API: access and request limits
- 02
Model server: weights, context and KV cache
- 03
Persistent storage: model versions and logs
Configuration requirements
Separate the model server from the public API if their scaling and security requirements differ.