The NUMA that wasn't there

Aditya Kumar · 12 September 2026

TL;DR — Vx already models a GPU's memory as a space you can be refused access to. A CPU's NUMA domains are the same relation, so modelling them needed no new machinery. The compiler predicted that a remote access costs 2.26× a local one; the hardware said 2.35×. Getting to that number meant first throwing away a day of measurements taken on a machine that reports two NUMA nodes it does not have.

Every memory space in Vx's fleet so far has been silicon you have to rent. NUMA is the one that is already inside the machine on your desk — and it turns out to be the sharpest test of the idea, because the two memories are physically identical.

Why NUMA is a better demo than a GPU

Vx puts the memory space in the type. A tensor in GPU_HBM is a different type from one in CPU_DRAM, and a host read of the first is refused at compile time rather than faulting at run time.

Someone can reasonably dismiss that as tracking hardware kinds. Of course HBM and DRAM are different — they are different chips. The type system is just writing down which one you have.

NUMA removes that objection entirely. On a two-socket server, node 0's memory and node 1's memory are the same DDR4, at the same speed, from the same order. Nothing about the bytes differs. The only thing separating them is which core is asking. If a type system can express that, it is expressing locality itself rather than a taxonomy of parts.

The model needed nothing new

A NUMA domain is a memory that some execution units reach cheaply and others reach across a link. That is structurally identical to two GPUs on a node, which Vx already describes as HBM and PEER_HBM with a priced edge between them. So the machine file for a two-socket Xeon is the same shape:

Memory HBM {
  capacity: 96 GiB, bandwidth: 140 GB/s, managed: explicit, scope: device, node: 0
}
Memory PEER_HBM {
  capacity: 96 GiB, bandwidth: 140 GB/s, managed: explicit, scope: device, node: 1
}

Topology Device {
  arch: x86_64,
  memory: Memory::HBM,
  visible: [Memory::HBM, Memory::PEER_HBM, Memory::L2],
  transfer Memory::HBM -> Memory::PEER_HBM : 62 GB/s,
  transfer Memory::PEER_HBM -> Memory::HBM : 62 GB/s
}

The names say which role a space plays, not what it is made of — there is no HBM anywhere near this part. What the pair says is "a tile lives in one domain, and reaching it from the other crosses a priced link", and that is true of two sockets exactly as it is of two GPUs.

The immediate payoff is a program the compiler now refuses. Before, the whole box was one 60 GiB space, so a 40 GiB tile was admitted — and it could not possibly be local, because no node holds 40 GiB:

Error[E6009]: transferred tensor needs 42949672960 bytes
              but memory space 'HBM' has capacity 32212254720 bytes

That refusal costs nothing to obtain. It runs on a laptop, before a machine is booked.

The machine that lied

The interesting half is the prediction. Two declared rates — 140 GB/s to a node's own memory, 62 GB/s across the interconnect — imply a remote access costs about 2.3× a local one, and that is a number you can go and falsify.

We started on a c4.8xlarge, because it is the exact part the machine file describes: a dual-socket Xeon E5-2666 v3. Everything checked out. lscpu matched. numactl -H reported two nodes with the right CPU lists. The ACPI distance table declared 10 for local and 21 for remote. /proc/PID/numa_maps confirmed that binding an allocation to a node put every single page there — and so did move_pages(), which is the kernel's own answer to the question "which node is this page on", reporting 64 of 64 probed pages on the node we asked for, in both directions.

Then the measurements came back:

read, all four (cpu node, memory node) pairs :  98.0  96.7  98.1  98.8  GB/s
pointer-chase latency                        : 119.0 117.3 117.8 117.8  ns

Under 1% spread. No NUMA effect at all — and worse than that, the numbers are not merely flat, they are impossible:

• One socket of that part peaks at 68.3 GB/s (four DDR4-2133 channels). We measured 98 bound to a single node. One socket cannot deliver that.
• QPI peaks at 38.4 GB/s. We measured a "remote" read at 96.7. Data crossing that link cannot exceed 38.4.
• Binding to one node measured the same as interleaving across both.

The guest's NUMA topology is cosmetic. Xen synthesizes the firmware tables, and the pages are spread across both sockets whatever the guest is told. The kernel is not lying: it is answering truthfully about guest nodes, and those are the thing that is not backed by locality. But the practical effect is worse than a lie, because the answer is indistinguishable from a correct one — move_pages() reports the same 64 of 64 on a c5.metal where the placement is real and a remote read costs 2.35×, and on this machine where it costs nothing at all. Every verification step passed while the thing being verified was fiction.

We checked it was not one unlucky instance. A second c4.8xlarge, provisioned separately on a different host, gives the same picture: reads of 98.9, 98.0, 98.0 and 99.1 GB/s across the four pairs, pointer-chase latencies within 1.5% of each other, and a single-node bind measuring 99.4 against 96.7 interleaved. It is policy, not placement.

The lesson is not about AWS. It is that a check and the thing it checks are not the same thing, and the only evidence that survived here was a rate the hardware could not produce. So the tooling now refuses rather than reporting: bind to one node, interleave across both, and if the two agree within 10% the bind confined nothing and every number above it is measuring one undivided pool.

The number

On a c5.metal — bare metal, no hypervisor between the guest and the sockets — the same probe gives a clean signal:

The 48-thread pair is a ratio of 2.348×. The single-threaded pair is 1.50×, and lower for a reason worth knowing: one core cannot saturate a socket's memory controllers, so that row is bounded by how many misses it can keep in flight rather than by the memory. It matters later.

Predicted 2.258×. Measured 2.348×. A 4% error on a figure derived from a datasheet and never previously checked.

The measurement also settled something the model had to guess. Cascade Lake carries UPI at 20.8 GB/s per link per direction, and a two-socket board may wire two or three links between the pair; the file assumed three, which is the optimistic end. The measured 51.55 GB/s is more than two links could physically carry — their ceiling is 41.6 — so three it is. The guess was right, which is not a reason to have made it.

And the sanity check that the c4 failed: 121 GB/s is 86% of one socket's 140 GB/s peak, and comfortably below it. Staying below peak is the load-bearing property. A bind that measures above one socket's ceiling is not confining anything.

What placement is actually worth

Pricing a hop is one thing; making the allocation land there is another. A Memory space declaring node: N now reaches the runtime as a number it can act on, and the backend binds the pages with mbind. The kernel agrees: move_pages reports 64 of 64 probed pages on the requested node, in both directions.

So what does that buy? Here is the honest table, and the middle row is the one that matters:

1.34× is the number, and 2.50× is not. Interleaving costs nothing, needs no source change, and is what anyone already does on a two-socket box. Quoting the speedup against naive first-touch would be comparing against the worst case — the same move as a benchmark reporting a fourfold speedup for a quarter of the work.

What it delivers today, which is less

That 1.34× needs the threads placed as well as the pages, and Vx's host backend runs an outlined kernel on one thread — vx_host_call_kernel is a single ffi_call, with no pool to spread across. So the aggregate figure is not reachable through that path at all.

What is reachable is one thread's local-versus-remote. A dispatch now pins itself to the node holding its arguments, and single-threaded that is worth about 1.5× on memory-bound work — 12.7–13.8 GB/s local against 8.7–9.0 remote. A threaded host backend is what would turn that into the larger number, and it is a different piece of work.

It is worth being plain about this rather than quoting the best figure in the post. The compile-time half — refusing a tile no node can hold, pricing the hop, and doing both on a laptop — is complete and validated. The runtime half is real, verified against the kernel, and currently capped by something unrelated to NUMA.

The thing that kept recurring

Three separate bugs this week had one shape: a declaration nobody downstream was obliged to honour.

A machine reported a NUMA topology the hardware did not have. A memory space declared managed: cached, meaning the host may read it, and the runtime then allocated it with cudaMalloc where the host cannot — a segfault three layers from the declaration that caused it. And an allocator tagged memory with an id that promised one thing while a different code path had produced it, which aborted the first program that transferred between two domains.

In every case the checks passed. A declaration that nothing downstream must respect looks exactly like one that is honoured, right up until you measure something that could not be true. That is the argument for putting placement in the type system, and it is also the argument for the guards: the NUMA probe now refuses to report a ratio from a machine whose nodes are cosmetic, and the runtime warns when it cannot honour what a model claimed.

The machine files, the probe, and the placement benchmark are all in the repository — fleet/xeon-8275cl.vx, utils/campaign/run_numa_probe.sh and utils/campaign/placement_bench.c. The tracking issue is Vx#515. Vx is Apache 2.0 with the LLVM exception — install it, or read Carrying facts across boundaries for the other five places a fact is carried across a boundary rather than re-derived.