Infrastructure & AI Platform
Performance is a property of the system.
GPUs deliver only what the rest of the cluster allows. ARC designs compute, networking, storage, orchestration, and operations together, manages them through the AI Platform, and validates the result against your jobs.
- Scale-up
- NVLink within a server or rack
- Scale-out
- InfiniBand or Ethernet with RoCE and RDMA
- Schedulers
- Kubernetes and Slurm
- Acceptance
- Representative jobs, criteria agreed in advance
Architecture
One cluster, every layer.
NVLink connects GPUs within a system. The scale-out network connects systems. Storage and scheduling complete the environment, and telemetry covers all of it.
The stack
Every layer, designed together.
Select a layer to see what ARC delivers and why it matters. Facilities sit beneath every layer: power, cooling, rack density, cabling, and serviceability.
Layer 1 · Compute
The right system for the model and the job.
- What ARC delivers
- NVIDIA HGX B300 and GB300 NVL72 systems, H100 and H200 for existing Hopper environments, and planning for Vera Rubin NVL72.
- Why it matters
- GPU memory, host CPU and RAM, and system size set what a job can do. We size them to your model, precision, and batch size.
Layer 2 · Networking
Scale up within a system. Scale out between systems.
- What ARC delivers
- Scale-up: NVLink connects GPUs within a server or rack. Scale-out: InfiniBand or Ethernet with RoCE, using RDMA between servers. NIC and DPU integration and designed failure domains.
- Why it matters
- In distributed training, the slowest link sets the pace. Topology follows your parallelism strategy, not a default.
Layer 3 · Storage
Keep GPUs fed and long jobs recoverable.
- What ARC delivers
- Local NVMe, shared file systems, and object storage, with separate paths for datasets, model artifacts, and checkpoints.
- Why it matters
- Slow data loading leaves GPUs idle. Slow checkpointing makes every failure more expensive.
Layer 4 · Orchestration
Share capacity without losing control.
- What ARC delivers
- Kubernetes or Slurm, run by your team or by ARC, managed through the AI Platform.
- Why it matters
- When several teams share a cluster, fair allocation and visibility matter as much as raw capacity.
Layer 5 · Operations
Keep the cluster healthy in production.
- What ARC delivers
- Observability across nodes, network, storage, and scheduler; controlled change; incident response; recovery and lifecycle management.
- Why it matters
- Hardware fails at scale. What matters is how quickly a fault is isolated and the system returns to service.
Layer 6 · Inference runtime
Set the latency and cost of every request.
- What ARC delivers
- Request routing, batching, caching, and concurrency tuned to your traffic and response targets.
- Why it matters
- In production, runtime decisions affect cost per token as much as the hardware does.
AI Platform
The orchestration and management layer.
The AI Platform supports every ARC product. It decides who gets which resources, when work runs, and where it lands. ARC configures and operates it within the boundaries you set.
Unified orchestration
One management layer across configured GPU, CPU, memory, and storage pools, on Kubernetes or Slurm.
Multi-team allocation
Project boundaries, permissions, quotas, and priorities that let teams share capacity fairly.
Task-level controls
GPU, CPU, memory, storage, and runtime requirements set at submission, with lifecycle control for each job.
Resource visibility
Allocation and job activity tied to usage reporting and capacity planning.
Terms that are often confused
| Term | Question it answers | What it does |
|---|---|---|
| Resource allocation | Which team gets how much | Divides capacity among projects and teams through quotas, budgets, and priorities. |
| Scheduling | When work runs | Orders queued jobs according to priority, policy, and available capacity. |
| Job placement | Where work lands | Puts each job on specific nodes, accounting for GPU type, NVLink domain, and network topology. |
| Request routing | Where traffic goes | Sends inference requests across serving replicas based on load and latency targets. |
Memory and workload fit
Host RAM and GPU memory do different jobs.
Telemetry shows which one is the constraint. The right fix depends on the model and runtime: batch size, precision, checkpointing strategy, model sharding, or the data pipeline.
Host RAM
Data loading and CPU-side work
Holds input pipelines, preprocessing, and runtime processes on the server’s CPUs.
GPU memory
Model state on the accelerator
Holds weights, activations, optimizer state during training, and the KV cache during inference.
Where bottlenecks hide
Optimize the constraint that matters.
Training
Collective communication, storage throughput, or checkpoint recovery.
Inference
GPU memory, request scheduling, batching, or concurrency.
We use observability data and workload behavior to find the real constraint before deciding what to change.
Validation
Tested against real work before handoff.
Acceptance criteria are agreed before bring-up begins. This is what they cover.
- Check
Hardware health
Burn-in and component checks across every node.
- Check
Network communication
Collective and point-to-point tests across the scale-out network.
- Check
Storage I/O
Dataset reads, checkpoint writes, and restores.
- Check
Software consistency
Drivers, firmware, and images match the agreed baseline.
- Check
Representative jobs
Your own workloads, run before acceptance.
Work through the architecture with us.
Bring your current bottleneck or the requirements for a new environment.
or email contact@arccloudai.com