Vx fully embraces {ML, IR, MLIR}

TL;DR — {ML, IR, MLIR} is shell brace expansion for the three things this post is about. ML: tensors are built-in values, and a @ b becomes a linalg.matmul. IR: Vx has its own intermediate representation, and a library can ship as it. MLIR: Vx has its own MLIR dialect. A GPU loop goes through it to a kernel with shared memory and warp shuffles. Inside Vx source, mlir! blocks write the core library and the GPU instructions. Each block states what it writes, and the compiler checks that against the MLIR. The last section lists what “fully” does not cover yet.

ML is machine learning: the tensor programs that Vx compiles. An IR, or intermediate representation, is the form a compiler holds a program in between the source and the machine code. MLIR is the LLVM project's framework for IRs. An MLIR IR is made of dialects: sets of operations at one level, such as linear algebra, GPU kernels or LLVM instructions. A compiler lowers a program from higher dialects into lower ones, until only machine code is left.

The first version of this post had one mlir! example, and its programs no longer compile. This version is new. Every program here was compiled with vxc built from commit a24dbc65.

ML: tensors the compiler understands

The element type and the shape of a tensor are part of its type. Matrix multiplication is the @ operator:

fn main() -> i32 {
  let mut a = Tensor<f32, [2, 3]>::new();
  let mut b = Tensor<f32, [3, 2]>::new();
  for i in 0..2 {
    for k in 0..3 {
      a[i][k] = (i * 3 + k + 1) as f32;
      b[k][i] = 1.0;
    }
  }
  let c = a @ b;
  print(c[1][0]);
  return 0;
}

It prints 15, the sum of row 1 of a. The @ does not become three loops. It becomes one linalg.matmul, which still says “a 2×3 times a 3×2” (vxc matmul.vx --action emit-mlir):

linalg.matmul ins(%alloc, %alloc_0 : memref<2x3xf32>, memref<3x2xf32>) outs(%alloc_2 : memref<2x2xf32>)

So a later stage can choose how to run it. When the product is placed on an NVIDIA GPU, the runtime gives it to cuBLAS.

std::tensor has the reductions that a model needs, such as sum, max, argmax and softmax_inplace. grad, jvp and vjp differentiate a function. The compiler turns grad(f, x) into a call that the Enzyme plugin fills in, on the LLVM IR.

IR: Vx's own, and a library can ship as it

Between the source and MLIR, Vx holds the program in its own IR, the flat HIR. It is a list of simple instructions, made after the type check and the borrow check. The default code generator reads it.

The flat HIR is also a file format. --action emit-interface checks a module, then writes its signatures and its flat-HIR bodies to a .vxlib file. A later compile can import that file without the source:

fn relu(x : f32) -> f32 {
  if x > 0.0 {
    return x;
  }
  return 0.0;
}
$ vxc activations.vx --action emit-interface -o activations.vxlib
Wrote module interface (790 bytes) to activations.vxlib
  fn_sigs 1/1  methods 0/0  bodies 1/1  structs 0/0  enums 0/0  (nothing skipped)
$ rm activations.vx
import activations;

fn main() -> i32 {
  print(relu(-2.5) + relu(4.0));
  return 0;
}

This prints 4, with activations.vx deleted.

MLIR: a dialect of its own

No upstream dialect says “this buffer is in GPU memory” or “run this region on that device”. The vx dialect says it. It has 16 operations, defined in TableGen in include/VxDialect.td. Two of them, trimmed:

def Vx_SpawnOp : Vx_Op<"spawn"> {
    let summary = "Spawns execution on a specific topology.";
    let arguments = (ins I32Attr:$topology);
    let regions = (region AnyRegion:$body);
}

def Vx_LaneReduceOp : Vx_Op<"lane_reduce", [MemoryEffects<[MemRead, MemWrite]>]> {
    let summary = "Combine the first elements of a tensor over the lanes.";
    let arguments = (ins AnyMemRef:$input, I32:$count, StrAttr:$kind,
                     UnitAttr:$ordered, UnitAttr:$subgroup, UnitAttr:$serial);
    let results = (outs F32:$result);
}

Here is a GPU program that uses both. Each block of 256 lanes divides one row by the sum of the row. With across blocks and across lanes, the passes of a loop run at the same time. They run on the blocks and the threads of the GPU. Memory::SMEM is the shared memory of one block. lanes::sum adds the row over the lanes:

// Each block of 256 lanes divides one row of `x` by the row's sum.
fn normalize(out : &mut Tensor<f32, [?], Memory::HBM>, x : &Tensor<f32, [?], Memory::HBM>, rows : i32) -> void {
  spawn on(Topology::Device) {
    for b in 0..rows across blocks {
      let mut part = unsafe { Tensor<f32, [256], Memory::SMEM>::uninit() };
      for t in 0..256 across lanes {
        part[t] = x[b * 256 + t];
      }
      let s = lanes::sum(&part, 256);
      for t in 0..256 across lanes {
        out[b * 256 + t] = x[b * 256 + t] / s;
      }
    }
  }
  return;
}

fn main() -> i32 {
  let mut host = Tensor<f32, [512]>::fill(1.0);
  for i in 0..256 {
    host[256 + i] = i as f32;
  }
  let x = transfer(host, Memory::HBM);
  let mut out = transfer(Tensor<f32, [512]>::fill(0.0), Memory::HBM);
  normalize(&mut out, &x, 2);
  let r = transfer(out, Memory::CPU_DRAM);
  println!("{} {} {}", r[0], r[256 + 1], r[511]);
  return 0;
}
$ vxc rows.vx --host default --machine fleet/a100-80.vx --run
0.00390625 0.000030637257 0.0078125

Row 0 is all ones, so each value is 1/256. Row 1 is 0 to 255, which add up to 32640. The machine file fleet/a100-80.vx describes an A100: its memories, its chip, and its levels of blocks and lanes.

Stage 1: the vx dialect

In the output of --action emit-mlir, the region is a vx.spawn. The loops are still loops. The address space 3 on the memref is shared memory. The chip comes from the machine file. Trimmed:

vx.spawn topology(1036766083) {
  %alloca_1 = memref.alloca() {alignment = 16 : i64} : memref<256xf32, 3>
  vx.launch_dim %arg2 axis(0)
  vx.launch_dim %c256_i32 axis(3)
  %block_id_x = gpu.block_id  x
  ...
  vx.barrier
  %23 = "vx.lane_reduce"(%alloca_1, %c256_i32_10) <{kind = "sum", ordered}> : (memref<256xf32, 3>, i32) -> f32
  ...
  %42 = arith.divf %41, %23 : f32
  memref.store %42, %arg0[%35] : memref<?xf32>
  ...
  vx.drop %alloca_1 : memref<256xf32, 3>
  vx.yield
} {arch = "nvptx64", chip = "sm_80", topology_name = "Device", vx.across}

vx.launch_dim gives the launch shape: rows blocks on axis 0, and 256 threads in a block (axis 3). The compiler put a vx.barrier after the first lane loop, so that every lane writes part before the sum reads it.

Stage 2: Vx's passes, into upstream dialects

The pass convert-vx-to-standard moves the region out into a kernel. The host launches it with the shape from stage 1:

vx.launch @vx_npu_kernel_0(%arg0, %arg1, %arg2, %arg2) {topology = 1036766083 : i32, vx.across, vx.launch_block = "256", vx.launch_grid = "$0", ...}

The kernel has two copies. The GPU copy is in MLIR's gpu dialect. There, vx.lane_reduce becomes 5 shuffles in each warp of 32 lanes. The first lane of each warp writes its sum to shared memory, and after a barrier the warps add those sums with 5 more shuffles:

gpu.module @vx_kernels attributes {vx.chip = "sm_80"} {
  memref.global "private" @vx_npu_kernel_0_smem_1 : memref<256xf32, 3> {alignment = 16 : i64}
  memref.global "private" @vx_npu_kernel_0_smem_0 : memref<32xf32, 3> {alignment = 16 : i64}
  gpu.func @vx_npu_kernel_0(%arg0: memref<?xf32>, %arg1: memref<?xf32>, %arg2: i32) kernel {
    ...
    gpu.barrier
    ...
    %shuffleResult, %valid = gpu.shuffle  down %28, %c16_i32, %c32_i32 : f32
    %29 = arith.addf %28, %shuffleResult : f32
    %shuffleResult_7, %valid_8 = gpu.shuffle  down %29, %c8_i32, %c32_i32_6 : f32
    %30 = arith.addf %29, %shuffleResult_7 : f32
    ...

The CPU copy is a vx.kernel. There, the same operation becomes loops that add in the same order, so the two copies give the same bits. This machine has no GPU, so the CPU copy printed the result above.

From here, upstream MLIR passes do the work. convert-gpu-to-nvvm and gpu-module-to-binary make PTX from the GPU module. The host code goes to the llvm dialect and then to LLVM IR.

Stage 3: PTX

PTX is NVIDIA's assembly language for GPUs. vxc --emit-ptx writes the kernels to a file, and a .json file beside it gives the parameters and the launch shape of each kernel. The kernel has the name of the function. The middle of it:

.version 7.6
.target sm_80
...
.visible .entry normalize(
...
	.shared .align 16 .b8 vx_npu_kernel_0_smem_1[1024];
	.shared .align 16 .b8 vx_npu_kernel_0_smem_0[128];
...
	bar.sync 	0;
	...
	shfl.sync.down.b32 	%r11, %r35, 16, 31, -1;
	add.rn.f32 	%r12, %r35, %r11;
	shfl.sync.down.b32 	%r13, %r12, 8, 31, -1;
	add.rn.f32 	%r14, %r12, %r13;
	...
	st.shared.b32 	[%rd5], %r5;
	bar.sync 	0;
	...
	shfl.sync.idx.b32 	%r6, %r29, 0, 31, -1;
	bar.sync 	0;
	...
	div.rn.f32 	%r31, %r30, %r6;
	st.global.b32 	[%rd23], %r31;

The last shfl.sync.idx gives the sum of lane 0 to every lane of the warp.

MLIR inside Vx source

The traffic also goes the other way. A Vx function can contain MLIR, in an mlir! block. Here is pow as one MLIR operation:

fn pow(x : f64, n : i32) -> f64 {
  return mlir!( inputs : (%x = x : f64, %n = n : i32), returns : f64, dialects : ["math"]) {
    %r = math.fpowi %x, %n : f64, i32
    macro.yield %r : f64
  };
}

fn main() -> i32 {
  print(pow(2.0, 10));
  print(" ");
  print(pow(1.5, -2));
  print("\n");
  return 0;
}

It prints 1024 0.4444444444444444. inputs binds each Vx value to an MLIR name, returns gives the result type, and dialects lists the dialects that the block uses. macro.yield gives the result back to Vx. The block becomes a small function, and math.fpowi becomes an LLVM intrinsic (vxc pow.vx --action emit-llvm, trimmed):

llvm.func hidden @vx_flat_mlir_0(%arg0: f64, %arg1: i32) -> f64 {
  %0 = llvm.intr.powi(%arg0, %arg1) : (f64, i32) -> f64
  llvm.return %0 : f64
}

The core library declares no extern function. About 130 mlir! blocks write its integer methods, its math functions, its pointers and its atomics. So core also works in a GPU kernel, which has no C library.

One block for the CPU and the GPU

The atomics of core::sync::atomic are an example. A macro_rules macro stamps one function for each integer type and each operation. $op is the operation, such as add or max, and $mlir is the MLIR type. The case for Relaxed, trimmed:

fn $name<M : Memory, D : Topology>(t : &Tensor<$t, [?], Memory::M>, i : i32, val : $t, order : AtomicOrdering) on Topology::D -> $t
ensures atomic(t)
{
  assert(i >= 0 && i < $len(t), "atomic read-modify-write: the index is outside the tensor");
  match order {
    AtomicOrdering::Relaxed => {
      return mlir!( inputs : (%t = t : memref<?x$mlir>, %i = i : i32, %x = val : $mlir), returns : $t,
                    dialects : ["memref", "arith", "llvm"], ensures : [atomic(t)]) {
        %a = memref.extract_aligned_pointer_as_index %t : memref<?x$mlir> -> index
        ...
        %p = llvm.getelementptr %base[%k] : (!llvm.ptr, i64) -> !llvm.ptr, $mlir
        %r = llvm.atomicrmw $op %p, %x monotonic : !llvm.ptr, $mlir
        macro.yield %r : $mlir
      };
    }
    ...
  }
}

A program calls it as an ordinary function. This one counts the values of x. Each block counts in its shared memory first, and then adds its counts to the totals in GPU memory:

import core::sync::atomic;

// Counts the values of `x`, which are 0 to 7. Each block counts in shared memory first,
// then adds its counts to the totals in GPU memory.
fn histogram(counts : &Tensor<u32, [8], Memory::HBM>, x : &Tensor<i32, [?], Memory::HBM>, blocks : i32) -> void {
  spawn on(Topology::Device) {
    for b in 0..blocks across blocks {
      let local = Tensor<u32, [8], Memory::SMEM>::fill(0u32);
      for l in 0..256 across lanes {
        let _old = atomic_fetch_add_u32(&local, x[b * 256 + l], 1u32, AtomicOrdering::Relaxed);
      }
      for l in 0..8 across lanes {
        let _old = atomic_fetch_add_u32(counts, l, local[l], AtomicOrdering::Relaxed);
      }
    }
  }
  return;
}

fn main() -> i32 {
  let mut values = Tensor<i32, [1024]>::fill(0);
  for i in 0..1024 {
    values[i] = (i * 5) % 8;
  }
  let x = transfer(values, Memory::HBM);
  let counts = transfer(Tensor<u32, [8]>::fill(0u32), Memory::HBM);
  histogram(&counts, &x, 4);
  let c = transfer(counts, Memory::CPU_DRAM);
  println!("{} {} {} {} {} {} {} {}", c[0], c[1], c[2], c[3], c[4], c[5], c[6], c[7]);
  return 0;
}

It prints 128 128 128 128 128 128 128 128. The memory space M is a type parameter, so the compiler makes one copy of the function for shared memory and one for GPU memory. In the kernel, the add to local is a shared-memory atomic, and the add to counts is a global one. The ordering Relaxed is a constant, so the other cases of the match are gone:

	atom.shared.add.u32 	%r12, [%rd19], 1;
	...
	bar.sync 	0;
	...
	atom.global.add.u32 	%r14, [%rd25], %r13;

On the CPU, the same block becomes an LLVM atomicrmw. That is how this machine printed the counts.

A block says what it writes

The passes of an across loop run at the same time. So the compiler proves that no two passes write the same place, and that no pass reads a place that another pass writes. A loop that it cannot prove is an error (E6032).

The proof does not read MLIR. So an mlir! block with no contract is refused in an across loop:

Error[E6032] at 151:24: an `mlir!` block in an `across` loop may write memory that the checker cannot see

The ensures entry of a block is its contract. ensures : [atomic(t)] above says that each access to t is atomic, so two passes can add to one element. ensures : [writes_only()] says that the block writes no memory. The function that holds the block states the same contract after its signature, and a call in the loop uses it.

Here is an instruction that only a GPU has. nvvm.vote.sync ballot gives each lane of a warp one bit for each lane whose condition is true. math.ctpop counts the bits:

// How many lanes of the warp have `p` true. It writes no memory.
fn count_true(p : bool) on Topology::Device -> i32
ensures writes_only()
{
  return mlir!( inputs : (%p = p : i1), returns : i32, dialects : ["nvvm", "llvm", "math"], ensures : [writes_only()]) {
    %all = llvm.mlir.constant(-1 : i32) : i32
    %bits = nvvm.vote.sync ballot %all, %p -> i32
    %r = math.ctpop %bits : i32
    macro.yield %r : i32
  };
}

// Each lane gets the number of positive values in its warp.
pub fn count_positive(out : &mut Tensor<i32, [?], Memory::HBM>, x : &Tensor<f32, [?], Memory::HBM>, warps : i32) -> void {
  spawn on(Topology::Device) {
    for w in 0..warps across blocks {
      for l in 0..32 across lanes {
        out[w * 32 + l] = count_true(x[w * 32 + l] > 0.0);
      }
    }
  }
  return;
}

In the PTX of count_positive, the block is two instructions:

	vote.sync.ballot.b32 	%r9, %p3, -1;
	popc.b32 	%r10, %r9;

Take out the two ensures, and the call is refused:

Error[E6032] at 17:27: the call to `count_true` may write through an argument, and the checker does not read its body. Write its work in the loop

The compiler checks the contract

A contract that nothing checks is a comment. So the compiler reads the MLIR of the block, and compares each operation on an input with the contract. This block says atomic(t), but it does a plain load, an add and a store:

// Says that each access to `t` is atomic, but it loads, adds and stores.
fn add_one(t : &Tensor<i32, [?], Memory::HBM>, i : i32) on Topology::Device -> i32
ensures atomic(t)
{
  return mlir!( inputs : (%t = t : memref<?xi32>, %i = i : i32), returns : i32, dialects : ["memref", "arith"], ensures : [atomic(t)]) {
    %k = arith.index_cast %i : i32 to index
    %old = memref.load %t[%k] : memref<?xi32>
    %one = arith.constant 1 : i32
    %new = arith.addi %old, %one : i32
    memref.store %new, %t[%k] : memref<?xi32>
    macro.yield %old : i32
  };
}
Error[E8007] at 5:17: the block says `atomic(t)`, but `%old = memref.load %t[%k] : memref<?xi32>` reads or writes `%t` without an atomic operation

Some things stay trusted. An input of type !llvm.ptr is an address, and the compiler cannot see which tensor it points into. The compiler also trusts that the PTX of an operation does what the operation says.

The MLIR itself is checked when the program compiles, not when it runs. A block that is not valid MLIR is an error at the block, with the message of the MLIR parser:

The `mlir!` block at line 2, column 17 is not valid MLIR: custom op 'arith.no_such_op' is unknown
The `mlir!` block at line 2, column 17 takes its input `%t` as `memref<4xf32>`, but the value is a `memref<8xf32>`: 'memref.cast' op operand type 'memref<8xf32>' and result type 'memref<4xf32>' are cast incompatible

A block can also name the arguments that it changes, with clobbers. The compile-time interpreter cannot run MLIR, but with clobbers it knows which values the block changed.

What “fully” does not cover yet

So “fully” describes the direction more than the finish line. But the direction is concrete. Placement is in the types, and it stays in Vx's own IR and in the vx dialect until it becomes kernels and copies. And the MLIR that a programmer writes by hand gets a contract that the compiler checks.

Rewritten 24 October 2026. The first version's programs no longer compiled, because ::uninit() now needs unsafe. This version adds the GPU loops, the lane sums and the mlir! contracts. A .vxlib is now checked when it is written.

For why keeping structure until code generation matters, see Premature de-optimization is the root of all evil. For GPU tiles in shared memory, see Tiles, without a tile type. The dialect is in include/VxDialect.td, its lowering in src/dialect/VxLowering.cpp, and the lane sums in docs/lane_reductions.md.