Dedicated GPU node
One customer per GPU server in the initial architecture. Shared API keys do not substitute for infrastructure isolation.
INFRASTRUCTURE / DEDICATED GPU
Compute chosen for your workload.
Each customer deployment starts with its own GPU server. We size capacity against the model, expected demand, and data location.
Inference runtime
vLLM loads the selected model, manages batching and KV cache, and exposes an OpenAI-compatible endpoint behind the gateway.
Persistent storage
Model weights, caches, gateway data, and operational records use persistent storage with planned backup paths.
Private control plane
The management application is separate from customer GPU nodes. Operational connectivity is designed for private networking.
Deployment boundary
Inference and gateway ports stay internal. Only the public TLS endpoint is exposed to customer applications.
Capacity planning
GPU choice considers weights, KV cache, runtime overhead, concurrency, and target latency.
A single GPU node is not high availability. Redundancy and failover are scoped separately when the required service level calls for them.