Running Distributed GPU Workloads on KubeRay: What Breaks at Scale and What It Actually Costs

The KubeRay quickstart is honest about what it shows and quiet about what it hides. Apply a RayCluster, watch the operator spin up a head and some workers, run a toy job, see it fan out. It works. The gap between that and a cluster carrying real GPU workloads all day is where the operational lessons live.

This is a field guide to that gap: the parts of a Ray-on-Kubernetes cluster that hold under load, the parts that do not, and the cost questions that decide whether the whole thing was worth standing up.

The model, briefly

KubeRay's operator reconciles three custom resources. RayCluster is a long-lived cluster you submit jobs to. RayJob creates a cluster, runs one job, and tears it down. RayService fronts a Serve deployment. For batch GPU work the choice is usually RayCluster for an interactive, shared pool or RayJob for one-shot pipelines that should not outlive their output.

Inside either, Ray's own scheduler — not the Kubernetes scheduler — places tasks and actors onto workers by logical resources. Kubernetes schedules pods; Ray schedules work onto pods. Most scaling confusion traces back to forgetting which scheduler owns which decision.

apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: gpu-pool
spec:
  rayVersion: "2.9.0"
  enableInTreeAutoscaling: true
  headGroupSpec:
    rayStartParams:
      num-gpus: "0"
    template:
      spec:
        containers:
          - name: ray-head
            image: registry.internal/ray-gpu:latest
            resources:
              requests: { cpu: "4", memory: 16Gi }
  workerGroupSpecs:
    - groupName: gpu
      replicas: 0
      minReplicas: 0
      maxReplicas: 32
      rayStartParams:
        num-gpus: "1"
      template:
        spec:
          containers:
            - name: ray-worker
              image: registry.internal/ray-gpu:latest
              resources:
                limits: { nvidia.com/gpu: 1 }
                requests: { cpu: "8", memory: 32Gi }

enableInTreeAutoscaling lets Ray's autoscaler add and remove worker pods based on pending task demand. Ray does not talk to the node layer directly: it creates worker pods, and when those pods cannot be scheduled for want of capacity the Kubernetes Cluster Autoscaler reacts to the Pending pods and provisions nodes. Two autoscalers in a trench coat, coupled only through the scheduler. When scaling feels sluggish, the latency is almost always in the lower layer waiting on a node, not in Ray.

What breaks at scale

A ten-GPU cluster forgives a lot. A hundred-GPU cluster collects on every shortcut.

The head node becomes the bottleneck

Task submission and actor lifecycle run through the head node, and its Global Control Store holds the cluster's metadata and actor registry. Object data itself moves worker-to-worker under Ray's distributed ownership, not through the head, but the coordination still funnels there. At small scale it is invisible. As task and actor counts climb, the head's CPU and memory become the ceiling on how fast the cluster can dispatch work. Submitting millions of tiny tasks is the fast path to a saturated head and a cluster that looks busy while the GPUs idle.

The fixes are structural: batch fine-grained work into coarser tasks, give the head real CPU and memory rather than the default, and keep GPU workers off the head group so coordination never competes with compute.

TODO(founder): the scale where the head first became the bottleneck — task rate, head spec before and after, and what the symptom looked like from the outside.

The object store spills, then thrashes

Ray keeps task outputs in a shared-memory object store on each node, backed by disk when memory fills. Spilling to fast local disk is fine. Spilling to slow disk, or spilling constantly because the working set never fits, turns a compute job into an IO job. GPU utilization graphs go sawtooth and nobody can say why.

Watch the spill metrics, not just GPU utilization. A cluster bound by object spilling has perfectly healthy-looking GPUs that are starved between bursts.

TODO(founder): a spilling incident — what the working set was, what disk it spilled to, and the change that fixed it.

GPU fragmentation and fractional allocation

Ray allocates GPUs as logical numbers. Request num_gpus=1 for work that uses a fraction and the cluster strands capacity: cards reserved, barely used, unavailable to anything else. Request fractions without accounting for VRAM and tasks collide in memory. There is no scheduler that reads your actual kernel; the number you write is the number Ray honors.

import ray

@ray.remote(num_gpus=0.5, num_cpus=4)
def infer(batch, weights_ref):
    model = load(ray.get(weights_ref))   # weights broadcast once
    return model.predict(batch)

The weights_ref pattern matters at scale for the same reason it matters anywhere: broadcast the large immutable object once through the object store, do not let every task reload it. The difference between doing this and not doing it is the difference between network-bound and compute-bound.

TODO(founder): the fragmentation you actually hit — how much GPU capacity was stranded before tuning the fractions, and how you measured it.

The unglamorous ones

Image pulls of multi-gigabyte CUDA images stall pod startup and make autoscaling feel broken when it is just waiting on a registry. NCCL and multi-node collective communication need the right network and host settings or they hang with no useful error. And a single Kubernetes node draining for maintenance takes a slice of the GPU pool with it, so jobs must tolerate worker loss as a normal event, not an exception.

TODO(founder): which of these bit hardest in practice, and the war story behind it.

What it actually costs

GPU clusters have three cost layers, and conflating them is how budgets get surprised.

There is the GPU time itself — owned cards depreciating or rented cards billing by the hour. There is the overhead that does not show up in a per-GPU price: the head node, the control plane, storage for the object store and images, and the network. And there is the cost of idle — capacity provisioned and paid for while no job runs, which minReplicas: 0 and scale-to-zero exist to kill.

The lever that moves the bill most is utilization. A cluster at high steady utilization makes owned hardware look cheap; a cluster that spikes and idles makes rented capacity look cheap. Autoscaling to zero between jobs is the single highest-leverage cost control, because the most expensive GPU is the one drawing power for nothing.

TODO(founder): the real numbers — your GPU-hour cost owned versus rented, the overhead as a fraction of GPU spend, the idle you eliminated with scale-to-zero, and the monthly figure before and after. This is the section readers came for.

What KubeRay earns

For all the failure modes, KubeRay earns its place. It turns a heterogeneous pile of GPU nodes into one elastic, scriptable pool; it scales to zero so idle costs nothing; and it survives node churn because Ray reschedules around lost workers by default. The work is in respecting the two schedulers, keeping the head and object store healthy, and writing tasks coarse enough to dispatch and fault-tolerant enough to lose. Do that, and the cluster stops being the thing you fight and goes back to being the thing you run jobs on.