Handing the PTX back to the compiler

TL;DR — glm-vx runs GLM models with GPU kernels written in Vx. Its kernels had 99 pieces of hand-written PTX in 25 source files. We moved most of them to MLIR ops. A test pins each PTX instruction that the old text had, with its exact count, and the registers and spills that ptxas reports. 14 of the 20 pinned kernels kept every pinned instruction. In the other 6, the compiler wrote fewer instructions, because it could now see the code around the old asm. No kernel needs more registers. One needs one register fewer, and one spills 8 bytes where it spilled 40. In four places the compiler made worse code. One needed a fix, and three keep their asm. One of these had the same instructions in another order, and only a timing on an H200 showed it. All of glm-vx's hardware tests for these kernels passed on an H200.

Why the PTX was there

A Vx function can hold an mlir! block: MLIR ops that the compiler puts into the function as they are. glm-vx used these blocks for the GPU instructions that Vx code did not have a way to say: tensor-core mma.sync, warp shuffles and votes, cp.async copies, TMA loads and mbarriers on the H100, and loads with cache hints. Many of these blocks held one op, llvm.inline_asm, with PTX text inside it:

%r = llvm.inline_asm has_side_effects "{ .reg .pred q; .reg .b32 m; setp.ne.s32 q, $1, 0;
    vote.sync.ballot.b32 m, q, -1; popc.b32 $0, m; }", "=r,r" %p : (i32) -> i32

To LLVM, this text is a black box. LLVM cannot remove an instruction in it, merge it with the code around it, or move it. That is sometimes what you want. Usually it only stops the compiler from doing its job. The NVVM dialect of MLIR has an op for almost all of these instructions, so we moved them.

A test that pins the old PTX

A move like this can lose an instruction without a test failing. A 16-byte load can become two 8-byte loads, or a .nc cache hint can go away, and the kernel still gives the right answer. So before we changed a line, we recorded what the kernels had.

For each of the 20 kernels that had inline PTX, gpu/spawn/ptx_pins.json keeps each special instruction form of the old text, with all its qualifiers, for example shfl.sync.down.b32 or ld.global.nc.v4.b32. It keeps the count of that form in the whole kernel, and the registers, stack and spill bytes from ptxas 13.1. The test builds each kernel and checks the exact count of each form. The registers, stack and spills must be no more than before. Plain adds, moves and compares are not pinned, because LLVM can choose them differently and be right.

We broke the build on purpose to see the test fail: an f16 mma in place of bf16, one shuffle fewer, a 16-byte load split in two, one register more, and a spill. Each failed the test for its kernel.

Most kernels: the same PTX

The old PTX became these ops:

Old PTXMLIR op
mma.sync (bf16, f16)nvvm.mma.sync
shfl.sync, vote.sync.ballot, match.any.syncnvvm.shfl.sync, nvvm.vote.sync, nvvm.match.sync
popcllvm.intr.ctpop
cp.async and its commit and waitnvvm.cp.async.shared.global and its ops
ld.global.nc.v4an invariant llvm.load of a vector
cvt.rn.bf16x2.f32, cvt.rn.bf16.f32llvm.fptrunc
cvt.rn.satfinite.e4m3x2.f32nvvm.convert.f32x2.to.f8x2
TMA load, mbarrier arrive and wait, fences, prefetchthe nvvm ops of the same names

In 14 of the 20 kernels, every pinned form kept its exact count, and ptxas gave the same registers. One of them, index_q_bf16_at, now needs 13 registers, not 14. The 499 inline-asm regions in the PTX of these kernels went down to 81. 45 of the 81 are not ours. LLVM's NVPTX backend writes an exit; as inline asm after a failed bounds check.

Six kernels: better PTX

In six kernels the pins failed, because the new PTX had fewer instructions. The compiler now saw both the old asm and the code around it, so it could remove work that the asm hid. We read each change, made sure the new PTX does the same work, and recorded the new counts. A removed form stays in the pins at count 0, so it cannot come back without a test failure.

A ballot that only needs to know "any". The top-k kernels count the lanes of a warp with a flag set: a ballot, then a popc. Some callers only test whether the count is above zero. With the asm, the flag went from a predicate to an integer and back, and the popc stayed:

selp.b32          %r69, 1, 0, %p3;
add.s32           %r262, %r68, %r69;
{ .reg .pred q; .reg .b32 m; setp.ne.s32 q, %r262, 0;
  vote.sync.ballot.b32 m, q, -1; popc.b32 %r261, m; }
setp.lt.s32       %p133, %r261, 1;

With the ops, LLVM sees that "count > 0" is "ballot is not 0". It ballots the predicate directly and drops the popc:

or.pred               %p153, %p2, %p3;
vote.sync.ballot.b32  %r325, %p153, -1;
setp.eq.b32           %p154, %r325, 0;

The three top-k kernels have 18 fewer popc.b32. In topk_mb_list_at, which ptxas fits in 32 registers, the spill stores went from 40 bytes to 8.

The mbarrier wait loop. In the H100 GEMM, the asm turned the result of mbarrier.try_wait into an integer with a selp, and the loop compared the integer with zero. Now the loop branches on the predicate:

// before
{ .reg .pred p; mbarrier.try_wait.parity.shared::cta.b64 p, [%rd249], %r122; selp.u32 %r121, 1, 0, p; }
setp.eq.b32  %p9, %r121, 0;
@%p9 bra     $L__BB0_8;

// after
mbarrier.try_wait.parity.shared.b64 %p9, [%rd65], %r14;
not.pred     %p10, %p9;
@%p10 bra    $L__BB0_8;

LLVM writes .shared where the asm had .shared::cta. PTX defines the two as the same address space.

A bf16 store. The epilogue of the GEMM converts each f32 result to bf16 and stores 16 bits. The asm converted into a 16-bit register and then widened it to 32 bits with cvt.u32.u16, only for the store to use the low 16 bits again. Now the store takes the 16-bit register, and fp8_gemm_m64 has 64 fewer instructions:

// before
{ .reg .b16 h; cvt.rn.bf16.f32 h, %r393; cvt.u32.u16 %r199, h; }
st.global.b16  [%rd65], %r199;

// after
cvt.rn.bf16.f32  %rs1, %r327;
st.global.b16    [%rd74], %rs1;

Four places where seeing more made it worse

An op that LLVM can see is also an op that LLVM can move. Four times that cost us. The pins caught three of them, and a GPU timing caught the fourth.

With the ops, index_scores_mma_at took 37.0 µs against 35.6 µs with the asm (1.04×), at 200,000 tokens of context, for GLM-5.3's indexer (32 heads of 128 dims, top 2048 of one query, f32 keys), on one H200, batch 1, against glm-vx 7970a7b with its inline PTX. It was slower at all 6 context lengths, 1.046× as long in the geometric mean (1.026× to 1.079×). index_scores_mma16_at, the same kernel over bf16 keys, has the same ops and was not slower: 0.989× in the geometric mean, faster at 4 of the 6 lengths. So it keeps its ops.

Tokens of contextmma_at, ops (µs)asm (µs)ops / asmmma16_at, ops (µs)asm (µs)ops / asm
3,0017.3 [7.2–7.4]6.9 [6.9–7.1]1.0586.9 [6.8–6.9]7.0 [7.0–7.0]0.986
8,1928.2 [8.1–8.2]7.6 [7.6–7.8]1.0797.2 [7.1–7.3]7.2 [7.2–7.2]1.000
32,76812.0 [11.9–12.1]11.7 [11.5–11.8]1.0269.4 [9.4–9.7]9.6 [9.6–9.7]0.979
65,53619.0 [18.8–19.0]18.4 [18.3–18.4]1.03313.0 [13.0–13.1]13.2 [13.1–13.2]0.985
131,07228.4 [28.4–28.6]27.3 [27.2–28.0]1.04021.3 [21.2–21.5]21.6 [21.5–21.7]0.986
200,00037.0 [36.7–37.2]35.6 [35.6–35.9]1.03927.9 [27.9–27.9]27.9 [27.9–28.3]1.000

Each time is the median of 3 rounds, with the fastest and the slowest round in brackets. The builds ran in alternating order, on an idle GPU. A round is the median of 20 launches, each alone after a copy that flushes L2. Both builds come from the vxc of Vx 10c6f478f, with driver 580.173.02 and CUDA 13.0. The ops build is glm-vx 8eae0c0. Every run checked every output. The times of the top-k kernels in the same runs are in the folder too. The script, the logs, every run and the full setup are in glm-vx's benchmark folder.

What stays PTX

Besides the lane-id reads, the barrier inits and the warp body above, two kinds of PTX stay:

One of those MoE kernels has a plain Vx version now. glm-vx's decode MoE down projection as a Vx lane matmul @ took 11.90 µs against 13.61 µs for the inline-PTX kernel (0.874×), for one decode step of GLM-5.3 in block FP8 at one rank of tensor parallelism 8 (6144 rows, 9 experts of width 256), batch 1, on one H200. Both read the same 14.2 MB and use f16 activations and f32 sums. Each time is the median of 10 rounds, from 11.66 to 12.22 µs against 13.47 to 14.42 µs. The setup and every run are in its benchmark folder. Its routing step keeps one small PTX helper, a shuffle, a ballot and a bfind, that Vx cannot write yet.

The converted kernels are on glm-vx main. On an H200, all of glm-vx's hardware tests for them passed: top-k, index scores, index chunk and both fp8 GEMMs. The pins check the PTX and the registers. The tests check the results. Only a timing checks the speed.