Architecture decision
A GPU server and a managed endpoint are different products. The request must name the operating model and the unit used to measure consumption.
GLOBALATLAS AI · AI inference
AI inference
Inference executes a trained model. Input size, batching and context length determine working memory as well as model weights.
Latency or batch throughput?
- 01
Interactive requests: measure first response latency
- 02
Batch jobs: measure completed inputs per time interval
- 03
Track peak memory for the chosen input length
Configuration requirements
Decide between a GPU VM you operate and a managed endpoint. The public Finder currently checks GPU VM offers, not a managed inference SLA.