Running distributed backtesting on KubeRay with consumer GPUs

A backtest is embarrassingly parallel right up until it is not. Each parameter set runs on its own slice of history with no shared state, so the work fans out cleanly. The trouble starts at the edges: loading the same market data into every worker, keeping GPU memory inside the lines of a card that was built for games, and noticing when a worker dies quietly and leaves a hole in your results grid.

This is the story of moving one such sweep off a single workstation and onto a small Kubernetes cluster of consumer GPUs, driven by Ray through the KubeRay operator. The shape of the problem is common. The interesting part is what it costs and what breaks.

Why consumer GPUs at all

Datacenter GPUs are the obvious answer and often the wrong one for this workload. Backtesting here is throughput-bound, not latency-bound, and it does not need the memory bandwidth, NVLink, or ECC that justify a datacenter card's price. The model that scores each candidate is small enough to fit several copies on one consumer card.

The pitch was simple: more cards for the same money buys more parallel backtests per wall-clock hour, and wall-clock hour is the metric that decides whether you iterate on a strategy twice a day or twice a week.

TODO(founder): the concrete trigger — how long the full sweep took on one machine, and how many parameter sets were in the grid.

The cluster shape

The cluster is a handful of bare-metal nodes, each with one or more consumer GPUs, joined to a single Kubernetes control plane. KubeRay's operator turns a RayCluster custom resource into a head pod plus a pool of worker pods, and Ray's scheduler hands GPU work to whichever worker has a free card.

apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: backtest
spec:
  rayVersion: "2.9.0"
  enableInTreeAutoscaling: true
  headGroupSpec:
    rayStartParams:
      num-gpus: "0"
    template:
      spec:
        containers:
          - name: ray-head
            image: registry.internal/backtest-ray:latest
            resources:
              requests:
                cpu: "2"
                memory: 8Gi
  workerGroupSpecs:
    - groupName: gpu-workers
      replicas: 4
      minReplicas: 0
      maxReplicas: 8
      rayStartParams:
        num-gpus: "1"
      template:
        spec:
          containers:
            - name: ray-worker
              image: registry.internal/backtest-ray:latest
              resources:
                limits:
                  nvidia.com/gpu: 1
                requests:
                  cpu: "4"
                  memory: 16Gi

Two decisions in that manifest matter more than they look. The head group requests zero GPUs, so the scheduler never wastes a card coordinating work. And the worker group sets minReplicas: 0, so an idle cluster scales the GPU pool to nothing and stops drawing power between sweeps.

The NVIDIA device plugin has to be running on each node for nvidia.com/gpu to be schedulable. That is table stakes, but it is also the first thing to check when pods sit Pending forever.

Sharding the sweep

The driver turns the parameter grid into a list of Ray tasks, each one a self-contained backtest. Ray packs them onto the GPU workers and streams results back as they finish.

import ray

ray.init(address="auto")

@ray.remote(num_gpus=0.25)
def score(params: dict, data_ref) -> dict:
    history = ray.get(data_ref)        # shared, loaded once
    equity = simulate(history, params)
    return {"params": params, "sharpe": sharpe(equity)}

data_ref = ray.put(load_history())     # one copy in the object store
grid = expand(parameter_space())
results = ray.get([score.remote(p, data_ref) for p in grid])

num_gpus=0.25 is the lever that makes consumer cards pay off. Ray treats GPU allocation as a fractional bookkeeping number, so four backtests share one physical card. The model is small, the card is not saturated by one copy, and packing four onto it pushes utilization toward full without running out of VRAM. Tune the fraction to the model that actually fits.

ray.put is the other half. Loading the price history once into Ray's object store and passing a reference means every worker reads the same immutable copy instead of each task re-reading gigabytes from disk. Skip this and the job is bound by data loading, not compute, and the GPUs sit half-idle.

One caveat on ray.init(address="auto"): auto discovers a Ray cluster only from inside it. In the KubeRay setup that means the driver runs as a RayJob the operator submits, or from a shell in the head pod. To drive the same job from a workstation outside the cluster, point Ray at the head's client server instead — ray.init("ray://<head-service>:10001") — because auto will not find a remote cluster.

What broke

The cluster ran. Then it ran into the things consumer hardware does not promise.

Thermals and clock throttling. Consumer cards under sustained 100% load throttle to stay inside their thermal envelope. Backtesting is exactly that kind of load. The cards did not fail; they quietly got slower, and the slowdown did not show up anywhere except in wall-clock time creeping past the estimate.

TODO(founder): the real throttling story — which cards, what the clocks dropped to, and whether better airflow or a power-limit cap fixed it.

Fractional GPU memory, not just compute. num_gpus=0.25 reserves a quarter of a card's scheduling slot. It does not partition VRAM. Pack four tasks whose combined memory exceeds the card and one of them dies with an out-of-memory error mid-backtest. The fix is to size the fraction against real memory use, not optimism.

TODO(founder): the OOM incident — how it surfaced, how long it took to trace to VRAM rather than a code bug.

Silent worker loss. A pod evicted under node pressure, or a card that fell off the bus and needed a node reboot, takes its in-flight tasks with it. Ray reschedules the task, which is the behavior you want, but only if the driver is structured to let it. A grid collected by index into a pre-sized array would have shipped with holes.

TODO(founder): the worst failure during a real sweep, and how you noticed the results were incomplete.

What it actually cost

This is the question the whole exercise exists to answer, and it splits into capital and running cost.

The capital side is the cluster itself: consumer cards, the nodes to hold them, and the power and cooling to run them. The running side is electricity and the hours of your attention the setup demands. Against that stands the alternative: renting datacenter GPUs by the hour and paying nothing when idle.

The honest comparison depends on duty cycle. Hardware you own is cheapest when it is busy and a liability when it sits idle. Rented capacity is the reverse. The break-even is a utilization number, not an opinion.

TODO(founder): the real figures — what the consumer-GPU cluster cost to build, the per-sweep electricity, the hourly rate of the datacenter GPU you compared against, and the utilization point where owning beat renting. This table is the post.

When this makes sense

Consumer GPUs win for this workload when three things hold: the job is throughput-bound and tolerant of a slow card, the model fits comfortably in consumer VRAM with room to pack several copies, and the cluster stays busy enough to earn its capital cost. Miss any one and renting datacenter cards by the hour is the cheaper, calmer choice.

KubeRay did the unglamorous work well: it turned a pile of mismatched machines into a single elastic GPU pool, scaled it to zero between runs, and rescheduled work around the failures that consumer hardware guarantees you will have. The strategy questions are yours. The infrastructure, at least, can stop being the bottleneck.