Requirements & Selection
Start with your target model, context length and number of concurrent users. Check whether a single device is sufficient, then consider model replicas or model parallelism across devices.
Configurations are based on validation of specific models and tasks. View validated configurations →
01 / Use-Case Solutions
A model’s memory requirements determine whether it needs multiple devices. Concurrency targets determine whether you need more model replicas. Each calls for a different way to scale.
Each Cube runs a complete replica of the same model. A load balancer distributes requests to increase concurrency and aggregate throughput.
Discuss concurrency solutions →Model parallelism distributes one model across multiple Cubes. High-speed networking connects the nodes so they can host a model that is too large for a single device.
See multi-device configurations ↓First confirm that the model runs, then choose how to scale based on concurrency, aggregate throughput and latency targets.
One full model instance on a single device
Start with an estimate of 8 concurrent requests per device, then confirm through load testing with the same model, context and request size.
Add Cubes running identical model replicas and configure load balancing. For a target of around 16 concurrent requests, start by evaluating 2 devices, then adjust based on load testing.
Assess concurrency options →Model parallelism addresses capacity
The first 4 devices provide enough capacity to run the model. To increase aggregate throughput, add a second or third group of 4 devices, with each group running its own replica. Distribute requests between groups using load balancing.
Each group of 4 devices jointly hosts one model instance, rather than four independent services. To serve more concurrent requests, replicate the entire group with the required high-speed networking and load balancing.
Discuss cluster expansion →Final device counts depend on model compatibility and load tests covering context length, concurrent users, time to first token and aggregate throughput. Model parallelism across devices also requires suitable high-speed interconnects such as 200GbE / RoCEv2.
02 / Configuration Lists
GB10 · 128 GB unified system memory per device
Choose 1 / 2 / 4 TB of storage to suit your models.
Single-node model service
Recommended Accessories
Monitor1 unit
Keyboard & Mouse1 set
Dock1 unitConnect peripherals to work directly on the device.
Two-node model parallelism
Recommended Accessories
Monitor1 unitKeyboard & Mouse1 setDock1 unit
DAC cable1 cableOne QSFP56-DAC cable directly connects the two devices.
Multi-node distributed inference
Recommended Accessories
Monitor1 unitKeyboard & Mouse1 setDock1 unitDAC cableN cablesN devices with N QSFP56-DAC cables;
the switch must provide at least N compatible 200G ports.
Accessories may be reused or selected separately. Two-device and multi-device lists assume one shared set of peripherals. DAC cables and a 200G switch support multi-device model parallelism; final accessories are subject to the agreed solution.
03 / Get a Solution
Use your application tasks and performance targets
to assess model, device and network configurations.
Target applications, model versions, input types and context length.
Concurrent users, request size, time to first token and aggregate throughput targets.
Data storage locations, restrictions on internal and external network access, and existing network equipment.
Budget range, deployment timeline and acceptance criteria.
Confirm models, device counts, recommended accessories and required services in one plan.
