<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:itunes="http://www.itunes.com/dtds/podcast-1.0.dtd" xmlns:googleplay="http://www.google.com/schemas/play-podcasts/1.0"><channel><title><![CDATA[Shawn Westfall]]></title><description><![CDATA[Security practitioner with 14 years in incident response, red teaming, exploit development, and malware reversing. Now applying AI to adversarial problems—and sharing the experiments, field notes, and lessons that hold up.]]></description><link>https://blog.swesty.net</link><image><url>https://substackcdn.com/image/fetch/$s_!eYc_!,w_256,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd47a4abd-91a7-4f70-ba6a-27197d589b8d_665x665.png</url><title>Shawn Westfall</title><link>https://blog.swesty.net</link></image><generator>Substack</generator><lastBuildDate>Sun, 22 Nov 2026 19:11:14 GMT</lastBuildDate><atom:link href="https://blog.swesty.net/feed" rel="self" type="application/rss+xml"/><copyright><![CDATA[Shawn Westfall]]></copyright><language><![CDATA[en]]></language><webMaster><![CDATA[pneumacap@substack.com]]></webMaster><itunes:owner><itunes:email><![CDATA[pneumacap@substack.com]]></itunes:email><itunes:name><![CDATA[Shawn Westfall]]></itunes:name></itunes:owner><itunes:author><![CDATA[Shawn Westfall]]></itunes:author><googleplay:owner><![CDATA[pneumacap@substack.com]]></googleplay:owner><googleplay:email><![CDATA[pneumacap@substack.com]]></googleplay:email><googleplay:author><![CDATA[Shawn Westfall]]></googleplay:author><itunes:block><![CDATA[Yes]]></itunes:block><item><title><![CDATA[Beyond Kernel 7: Closing the Gap to cuBLAS on an H100 FP32 GEMM]]></title><description><![CDATA[I am currently taking Vizuara&#8217;s Kernel Engineering Workshop, taught by Dr. Raj Dandekar and Shubham Panchal.]]></description><link>https://blog.swesty.net/p/beyond-kernel-7-closing-the-cublas-gap</link><guid isPermaLink="false">https://blog.swesty.net/p/beyond-kernel-7-closing-the-cublas-gap</guid><dc:creator><![CDATA[Shawn Westfall]]></dc:creator><pubDate>Tue, 10 Nov 2026 00:00:00 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/7513c107-21b5-4e99-8c58-456f3b1720e5_1864x270.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>I am currently taking <a href="https://kernelworkshop.vizuara.ai/">Vizuara&#8217;s Kernel Engineering Workshop</a>, taught by Dr. Raj Dandekar and Shubham Panchal. The course is about understanding how GPUs execute programs, then using that understanding to write faster kernels: the functions that run on the GPU. It starts with parallel execution, CUDA, and the memory hierarchy, then works through matrix multiplication, tensor cores, profiling, attention, and modern GPU programming tools. The lectures are paired with coding sessions so the explanations can be tested in working implementations.</p><p>This article grew out of the matrix multiplication work. Before moving on to tensor cores, we worked through seven versions of a kernel running on CUDA cores. Each version addressed a different source of overhead, from the way threads accessed global memory to the amount of work they kept in registers. By the seventh version the kernel was considerably more sophisticated, but there was still a gap to NVIDIA&#8217;s cuBLAS library. The instructor challenged us to see whether we could close it.</p><p>The course was already taking a substantial amount of my free time, so I explored the challenge without turning it into another lengthy assignment: I gave an LLM the existing kernels and a way to run experiments through my Modal account. The model did much of the implementation and profiling analysis, using Nsight Compute to inspect the results. My goals were to learn about optimizations beyond those covered in class and to share the findings with the course authors and other students.</p><p>The result was a small but repeatable win for one specific case, and a remaining gap everywhere else:</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!7JyC!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3baf9ae2-519a-4cba-8b12-b5c0d50de8ab_1864x270.png 424w, https://substackcdn.com/image/fetch/$s_!7JyC!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3baf9ae2-519a-4cba-8b12-b5c0d50de8ab_1864x270.png 848w, https://substackcdn.com/image/fetch/$s_!7JyC!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3baf9ae2-519a-4cba-8b12-b5c0d50de8ab_1864x270.png 1272w, https://substackcdn.com/image/fetch/$s_!7JyC!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3baf9ae2-519a-4cba-8b12-b5c0d50de8ab_1864x270.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!7JyC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3baf9ae2-519a-4cba-8b12-b5c0d50de8ab_1864x270.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/3baf9ae2-519a-4cba-8b12-b5c0d50de8ab_1864x270.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Beyond Kernel 7: Closing the Gap to cuBLAS on an H100 FP32 GEMM &#8212; table 1&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:true,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Beyond Kernel 7: Closing the Gap to cuBLAS on an H100 FP32 GEMM &#8212; table 1" title="Beyond Kernel 7: Closing the Gap to cuBLAS on an H100 FP32 GEMM &#8212; table 1" srcset="https://substackcdn.com/image/fetch/$s_!7JyC!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3baf9ae2-519a-4cba-8b12-b5c0d50de8ab_1864x270.png 424w, https://substackcdn.com/image/fetch/$s_!7JyC!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3baf9ae2-519a-4cba-8b12-b5c0d50de8ab_1864x270.png 848w, https://substackcdn.com/image/fetch/$s_!7JyC!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3baf9ae2-519a-4cba-8b12-b5c0d50de8ab_1864x270.png 1272w, https://substackcdn.com/image/fetch/$s_!7JyC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F3baf9ae2-519a-4cba-8b12-b5c0d50de8ab_1864x270.png 1456w" sizes="100vw" fetchpriority="high"></picture><div></div></div></div></figure></div><p>All of these were measured on one NVIDIA H100 80GB with FP32 inputs and arithmetic and TF32 disabled for cuBLAS. This was an investigation of a particular FP32 workload, not a replacement for cuBLAS across arbitrary matrix sizes and precision modes. The rest of the article explains the operation, the seven kernels that established the starting point, and then each later change: the ones that helped, and the ones that looked promising in a profile but did not improve runtime.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!gOqV!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb48b2159-3ff7-412d-9859-a6d6c6269489_1864x1102.png 424w, https://substackcdn.com/image/fetch/$s_!gOqV!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb48b2159-3ff7-412d-9859-a6d6c6269489_1864x1102.png 848w, https://substackcdn.com/image/fetch/$s_!gOqV!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb48b2159-3ff7-412d-9859-a6d6c6269489_1864x1102.png 1272w, https://substackcdn.com/image/fetch/$s_!gOqV!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb48b2159-3ff7-412d-9859-a6d6c6269489_1864x1102.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!gOqV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb48b2159-3ff7-412d-9859-a6d6c6269489_1864x1102.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b48b2159-3ff7-412d-9859-a6d6c6269489_1864x1102.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Beyond Kernel 7: Closing the Gap to cuBLAS on an H100 FP32 GEMM &#8212; chart 1&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:false,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Beyond Kernel 7: Closing the Gap to cuBLAS on an H100 FP32 GEMM &#8212; chart 1" title="Beyond Kernel 7: Closing the Gap to cuBLAS on an H100 FP32 GEMM &#8212; chart 1" srcset="https://substackcdn.com/image/fetch/$s_!gOqV!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb48b2159-3ff7-412d-9859-a6d6c6269489_1864x1102.png 424w, https://substackcdn.com/image/fetch/$s_!gOqV!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb48b2159-3ff7-412d-9859-a6d6c6269489_1864x1102.png 848w, https://substackcdn.com/image/fetch/$s_!gOqV!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb48b2159-3ff7-412d-9859-a6d6c6269489_1864x1102.png 1272w, https://substackcdn.com/image/fetch/$s_!gOqV!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb48b2159-3ff7-412d-9859-a6d6c6269489_1864x1102.png 1456w" sizes="100vw"></picture><div></div></div></div><figcaption class="image-caption"><strong>Figure 1.</strong> Speed relative to cuBLAS at each step, for three square sizes. Each point is the best configuration of the kernel introduced at that step, divided by cuBLAS measured in the same run (cuBLAS's own time moved 1&#8211;2% between runs). The first point is the course authors' kernel 7 measurement, which may have used different hardware or settings, so its link to kernel 8 is dashed. The last point is the 15-round confirmation. Block swizzling, tested between kernels 9 and 10, is omitted because it was only profiled; Figure 10 covers it.</figcaption></figure></div><h2>The operation: GEMM and the precision of the comparison</h2><p>The operation is matrix multiplication, usually discussed under the name GEMM: general matrix multiplication. For A with dimensions M &#215; K and B with dimensions K &#215; N, the product C is M &#215; N, and each output element is the dot product of a row of A and a column of B:</p><p>C[m, n] = &#931;<sub>k</sub> A[m, k] &#215; B[k, n]</p><p>The general BLAS routine computes C = &#945;AB + &#946;C; these experiments computed only AB, for square matrices whose sides are multiples of the tile size. NVIDIA&#8217;s <a href="https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html">matrix multiplication background guide</a> covers the operation and its performance characteristics. That keeps the scope clear: a library routine also supports scaling, accumulation into an existing output, transposition, other layouts, and many shapes that these kernels did not attempt.</p><p>A straightforward implementation performs K multiply-adds for each of the M &#215; N outputs, about 2MNK floating-point operations. The work is done by FFMA instructions: fused floating-point multiply-adds, where the multiplication and addition share a single final rounding. In throughput accounting, an FFMA still counts as two operations.</p><p>The inputs and arithmetic in the comparison were FP32: four-byte values with one sign bit, eight exponent bits, and 23 stored fraction bits. That is a stronger statement than saying the arrays had a 32-bit type, because a library can accept FP32 arrays and use a different internal arithmetic mode. TF32 (TensorFloat-32) is the important example on NVIDIA GPUs. It keeps FP32&#8217;s exponent range but uses fewer significand bits for the multiplication inputs on the tensor cores, which buys much higher throughput at the cost of different numerics (<a href="https://developer.nvidia.com/blog/accelerating-ai-training-with-tf32-tensor-cores/">NVIDIA&#8217;s TF32 introduction</a>). TF32 was disabled for the cuBLAS reference here, so both sides ran FFMA on the CUDA cores. A claim of beating cuBLAS without that qualification would leave out an essential part of the comparison.</p><h2>The hardware model behind the changes</h2><p>A CUDA kernel launches a grid of thread blocks. Threads within each block are grouped into warps of 32, and the GPU&#8217;s streaming multiprocessors (SMs) schedule those warps&#8217; instructions. Global memory holds the matrices and is visible across the device. Shared memory belongs to a block and lets its threads reuse a loaded tile without asking global memory for the same data again. Registers hold per-thread values, including the partial sums, right next to the arithmetic. Moving data into a faster level does not remove the need to deliver it to the next one.</p><p>That gives three questions to ask of any version. Are neighboring lanes of a warp requesting data efficiently? Is each loaded value reused enough times to justify moving it? And is there enough independent work available while a load, or an instruction dependency, is waiting? The first seven kernels largely answered the first two. The later kernels were about the third: how reliably the layout could keep the arithmetic units supplied.</p><h2>The foundation: the first seven kernels</h2><p>The numbering follows <a href="https://github.com/swesty/Kernel-Engg/tree/9d1e6110dfade157c55f686907a9611f2ee83ff0">my fork of the course repository</a>. All seven were Python CUDA kernels written with Numba&#8217;s <code>@cuda.jit</code> and CuPy arrays: Python was the source language, and Numba compiled the device code for the GPU. Warp tiling was already in the course&#8217;s kernel 6, and kernel 7 tuned that design. Kernels 8 to 14 are my extension of the course work.</p><p>The important distinction in this part is between changing <em>what</em> each thread computes and changing <em>where</em> its operands live.</p><h3>Kernel 1: a correct baseline</h3><p><a href="https://github.com/swesty/Kernel-Engg/blob/9d1e6110dfade157c55f686907a9611f2ee83ff0/lecture8/Lecture8_MatMul_Kernels.ipynb">Kernel 1</a> assigned one output element to each thread. That is an easy starting point for checking correctness, but its mapping sent neighboring lanes of a warp down different output <em>rows</em>. The arrays are row-major, so adjacent columns sit next to each other in memory and adjacent rows do not. Holding k fixed and looking across one warp: every lane requested the same B value, while the A reads and C writes were each separated by a full row stride. Kernel 1 read its operands directly from global memory, with no shared-memory staging.</p><p>That mattered before any attempt to pipeline the computation. A GPU moves memory in transactions, not by charging only for the bytes the program uses, and scattered lane requests can need many more transactions than a contiguous group.</p><h3>Kernel 2: make neighboring lanes match the layout</h3><p><a href="https://github.com/swesty/Kernel-Engg/blob/9d1e6110dfade157c55f686907a9611f2ee83ff0/lecture8/Lecture8_MatMul_Kernels.ipynb">Kernel 2</a> swapped the mapping so neighboring lanes worked across output columns. For a fixed row and k, every lane needed the same A[m, k], each needed a neighboring B[k, n], and the output writes advanced across columns. The arithmetic was unchanged; the improvement came entirely from how the warp requested its data.</p><p>This is coalescing: how a warp&#8217;s memory requests can be combined. It does not guarantee one physical transaction per access, since alignment, access width, and the exact addresses still matter. The useful change was that neighboring lanes no longer made row-strided A requests and C writes.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!m8QG!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac602a99-3296-4524-a373-15c5bcea6290_2880x1680.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!m8QG!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac602a99-3296-4524-a373-15c5bcea6290_2880x1680.png 424w, https://substackcdn.com/image/fetch/$s_!m8QG!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac602a99-3296-4524-a373-15c5bcea6290_2880x1680.png 848w, https://substackcdn.com/image/fetch/$s_!m8QG!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac602a99-3296-4524-a373-15c5bcea6290_2880x1680.png 1272w, https://substackcdn.com/image/fetch/$s_!m8QG!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac602a99-3296-4524-a373-15c5bcea6290_2880x1680.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!m8QG!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac602a99-3296-4524-a373-15c5bcea6290_2880x1680.png" width="2880" height="1680" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ac602a99-3296-4524-a373-15c5bcea6290_2880x1680.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1680,&quot;width&quot;:2880,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Two 4-by-4 output grids show lane IDs 0 through 3 down a column for kernel 1 and across a row for kernel 2. Lists describe the resulting A, B, and C address strides.&quot;,&quot;title&quot;:&quot;Kernel 1 maps neighboring lanes to different output rows: at one loop index their A loads and C stores are row-strided, while every lane requests the same B value. Kernel 2 maps lanes to output columns: A becomes a same-address request, and B loads and C stores are contiguous. Four of the 32 lanes are drawn. These are global-memory access patterns, not shared-memory bank mappings.&quot;,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Two 4-by-4 output grids show lane IDs 0 through 3 down a column for kernel 1 and across a row for kernel 2. Lists describe the resulting A, B, and C address strides." title="Kernel 1 maps neighboring lanes to different output rows: at one loop index their A loads and C stores are row-strided, while every lane requests the same B value. Kernel 2 maps lanes to output columns: A becomes a same-address request, and B loads and C stores are contiguous. Four of the 32 lanes are drawn. These are global-memory access patterns, not shared-memory bank mappings." srcset="https://substackcdn.com/image/fetch/$s_!m8QG!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac602a99-3296-4524-a373-15c5bcea6290_2880x1680.png 424w, https://substackcdn.com/image/fetch/$s_!m8QG!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac602a99-3296-4524-a373-15c5bcea6290_2880x1680.png 848w, https://substackcdn.com/image/fetch/$s_!m8QG!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac602a99-3296-4524-a373-15c5bcea6290_2880x1680.png 1272w, https://substackcdn.com/image/fetch/$s_!m8QG!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fac602a99-3296-4524-a373-15c5bcea6290_2880x1680.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"><strong>Figure 2.</strong> Kernel 1 maps neighboring lanes to different output rows: at one loop index their A loads and C stores are row-strided, while every lane requests the same B value. Kernel 2 maps lanes to output columns: A becomes a same-address request, and B loads and C stores are contiguous. Four of the 32 lanes are drawn. These are global-memory access patterns, not shared-memory bank mappings.</figcaption></figure></div><p>For one aligned, full-warp FP32 store, the difference is concrete. Kernel 1 wrote 32 four-byte values to 32 different rows. In the notebook&#8217;s example the C row stride is 800 &#215; 4 = 3,200 bytes, so those writes touched 32 separate 32-byte sectors, and the 128 bytes written filled one eighth of them. Kernel 2 wrote the same 128 bytes across adjacent columns, filling four aligned sectors completely.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!nTz4!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffabf6772-b7c4-4365-b06e-934e21b81bd5_2880x1920.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!nTz4!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffabf6772-b7c4-4365-b06e-934e21b81bd5_2880x1920.png 424w, https://substackcdn.com/image/fetch/$s_!nTz4!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffabf6772-b7c4-4365-b06e-934e21b81bd5_2880x1920.png 848w, https://substackcdn.com/image/fetch/$s_!nTz4!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffabf6772-b7c4-4365-b06e-934e21b81bd5_2880x1920.png 1272w, https://substackcdn.com/image/fetch/$s_!nTz4!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffabf6772-b7c4-4365-b06e-934e21b81bd5_2880x1920.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!nTz4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffabf6772-b7c4-4365-b06e-934e21b81bd5_2880x1920.png" width="2880" height="1920" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/fabf6772-b7c4-4365-b06e-934e21b81bd5_2880x1920.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1920,&quot;width&quot;:2880,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Thirty-two separate eight-word sectors each have one filled requested word for kernel 1. Four sectors have all eight words used for kernel 2. The requested-byte fractions are 12.5 percent and 100 percent.&quot;,&quot;title&quot;:&quot;One aligned, full-warp FP32 store instruction requests 128 bytes. The row-strided kernel 1 mapping touches 32 different 32-byte sectors and uses one word in each; dotted outlines mark the seven words per sector the instruction does not request, and the address gaps between rows are omitted. Kernel 2 packs the same stores into four fully used sectors. The fractions describe one instruction's requested bytes, not total DRAM traffic, cache behavior, or an expected speedup.&quot;,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Thirty-two separate eight-word sectors each have one filled requested word for kernel 1. Four sectors have all eight words used for kernel 2. The requested-byte fractions are 12.5 percent and 100 percent." title="One aligned, full-warp FP32 store instruction requests 128 bytes. The row-strided kernel 1 mapping touches 32 different 32-byte sectors and uses one word in each; dotted outlines mark the seven words per sector the instruction does not request, and the address gaps between rows are omitted. Kernel 2 packs the same stores into four fully used sectors. The fractions describe one instruction's requested bytes, not total DRAM traffic, cache behavior, or an expected speedup." srcset="https://substackcdn.com/image/fetch/$s_!nTz4!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffabf6772-b7c4-4365-b06e-934e21b81bd5_2880x1920.png 424w, https://substackcdn.com/image/fetch/$s_!nTz4!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffabf6772-b7c4-4365-b06e-934e21b81bd5_2880x1920.png 848w, https://substackcdn.com/image/fetch/$s_!nTz4!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffabf6772-b7c4-4365-b06e-934e21b81bd5_2880x1920.png 1272w, https://substackcdn.com/image/fetch/$s_!nTz4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffabf6772-b7c4-4365-b06e-934e21b81bd5_2880x1920.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"><strong>Figure 3.</strong> One aligned, full-warp FP32 store instruction requests 128 bytes. The row-strided kernel 1 mapping touches 32 different 32-byte sectors and uses one word in each; dotted outlines mark the seven words per sector the instruction does not request, and the address gaps between rows are omitted. Kernel 2 packs the same stores into four fully used sectors. The fractions describe one instruction's requested bytes, not total DRAM traffic, cache behavior, or an expected speedup.</figcaption></figure></div><h3>Kernel 3: reuse tiles in shared memory</h3><p><a href="https://github.com/swesty/Kernel-Engg/blob/9d1e6110dfade157c55f686907a9611f2ee83ff0/lecture8/Lecture8_MatMul_Kernels.ipynb">Kernel 3</a> introduced shared-memory tiles. Threads in a block cooperatively loaded portions of A and B, synchronized, and reused those values to compute a tile of C, so a value that contributed to several outputs no longer needed a separate global request for each use.</p><p>Consider a block computing a B<sub>M</sub> &#215; B<sub>N</sub> tile of C from a chunk of B<sub>K</sub> values along K. It needs an A panel of B<sub>M</sub> &#215; B<sub>K</sub> values and a B panel of B<sub>K</sub> &#215; B<sub>N</sub> values, and those panels support B<sub>M</sub> &#215; B<sub>N</sub> &#215; B<sub>K</sub> multiply-adds. That reuse is what arithmetic intensity measures: operations performed per byte moved at a given memory level. For this simplified FP32 tile, ignoring output traffic, caches, and redundant loads:</p><p>Arithmetic intensity &#8776; 2 B<sub>M</sub> B<sub>N</sub> B<sub>K</sub> / [4 (B<sub>M</sub> B<sub>K</sub> + B<sub>K</sub> B<sub>N</sub>)] FLOP per byte</p><p>For a square B &#215; B tile this simplifies to B/4 FLOP per byte of panel traffic. It&#8217;s a model, not a measurement, but it explains why reuse matters: the arithmetic grows faster than the panel traffic.</p><p>The cost was synchronization. Threads had to finish loading a shared tile before anyone consumed it, and could not overwrite it until the whole block was done reading. The tile created reuse, and it also created dependencies that later versions had to manage.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!vrB5!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F122af1f7-be92-43b8-b4e9-ea245fb88c59_2880x1790.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!vrB5!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F122af1f7-be92-43b8-b4e9-ea245fb88c59_2880x1790.png 424w, https://substackcdn.com/image/fetch/$s_!vrB5!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F122af1f7-be92-43b8-b4e9-ea245fb88c59_2880x1790.png 848w, https://substackcdn.com/image/fetch/$s_!vrB5!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F122af1f7-be92-43b8-b4e9-ea245fb88c59_2880x1790.png 1272w, https://substackcdn.com/image/fetch/$s_!vrB5!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F122af1f7-be92-43b8-b4e9-ea245fb88c59_2880x1790.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!vrB5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F122af1f7-be92-43b8-b4e9-ea245fb88c59_2880x1790.png" width="2880" height="1790" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/122af1f7-be92-43b8-b4e9-ea245fb88c59_2880x1790.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1790,&quot;width&quot;:2880,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Shared A and B tiles highlight a column and row whose outer product updates the output tile. A four-step cycle shows loading, a barrier, computation, and a second barrier.&quot;,&quot;title&quot;:&quot;Kernel 3 cooperatively loads A and B tiles from global into shared memory. At each inner K step, a column of A and a row of B contribute an outer product to the output tile. The first cuda.syncthreads() makes the loaded inputs visible before use; the second prevents an early overwrite while other threads still read them. Both are device-side, block-wide barriers, not the host-side cuda.synchronize(). The drawing uses 4 &#215; 4 tiles; the repository kernel uses 32 &#215; 32.&quot;,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Shared A and B tiles highlight a column and row whose outer product updates the output tile. A four-step cycle shows loading, a barrier, computation, and a second barrier." title="Kernel 3 cooperatively loads A and B tiles from global into shared memory. At each inner K step, a column of A and a row of B contribute an outer product to the output tile. The first cuda.syncthreads() makes the loaded inputs visible before use; the second prevents an early overwrite while other threads still read them. Both are device-side, block-wide barriers, not the host-side cuda.synchronize(). The drawing uses 4 &#215; 4 tiles; the repository kernel uses 32 &#215; 32." srcset="https://substackcdn.com/image/fetch/$s_!vrB5!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F122af1f7-be92-43b8-b4e9-ea245fb88c59_2880x1790.png 424w, https://substackcdn.com/image/fetch/$s_!vrB5!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F122af1f7-be92-43b8-b4e9-ea245fb88c59_2880x1790.png 848w, https://substackcdn.com/image/fetch/$s_!vrB5!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F122af1f7-be92-43b8-b4e9-ea245fb88c59_2880x1790.png 1272w, https://substackcdn.com/image/fetch/$s_!vrB5!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F122af1f7-be92-43b8-b4e9-ea245fb88c59_2880x1790.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"><strong>Figure 4.</strong> Kernel 3 cooperatively loads A and B tiles from global into shared memory. At each inner K step, a column of A and a row of B contribute an outer product to the output tile. The first cuda.syncthreads() makes the loaded inputs visible before use; the second prevents an early overwrite while other threads still read them. Both are device-side, block-wide barriers, not the host-side cuda.synchronize(). The drawing uses 4 &#215; 4 tiles; the repository kernel uses 32 &#215; 32.</figcaption></figure></div><h3>Kernel 4: several outputs per thread</h3><p><a href="https://github.com/swesty/Kernel-Engg/blob/9d1e6110dfade157c55f686907a9611f2ee83ff0/lecture8/Lecture8_MatMul_Kernels.ipynb">Kernel 4</a> moved more of that reuse into individual threads. Each thread kept a vertical strip of eight accumulators in registers, so one B value loaded per K-step fed eight multiply-adds with different A values. This is one-dimensional register blocking, and it made the resource trade visible: eight accumulators need more registers than one, and a change that reduces loads can still lose if its register demand leaves too little active work on the SM or causes spilling.</p><h3>Kernel 5: reuse both operands in a register patch</h3><p><a href="https://github.com/swesty/Kernel-Engg/blob/9d1e6110dfade157c55f686907a9611f2ee83ff0/lecture10/matmul-05.py">Kernel 5</a> expanded the strip into a two-dimensional patch. The repository configuration used a 4 &#215; 4 patch per thread inside a 64 &#215; 64 block tile, with B<sub>K</sub> = 16 and 256 threads. For a T<sub>M</sub> &#215; T<sub>N</sub> patch, a thread loads T<sub>M</sub> A values and T<sub>N</sub> B values and uses their outer product to update T<sub>M</sub> &#215; T<sub>N</sub> accumulators, so every A value is reused across T<sub>N</sub> columns and every B value across T<sub>M</sub> rows. The key change was not that the thread did more work, but that it did more multiply-adds per operand loaded into registers.</p><p>Block tiling and register blocking both increase reuse, at different levels. Block tiling cut repeated global requests by sharing panels among threads; register blocking cut repeated operand delivery to the arithmetic within a thread.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!b1UQ!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde8c2251-a41e-4008-8093-22095c6b47c8_2880x1820.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!b1UQ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde8c2251-a41e-4008-8093-22095c6b47c8_2880x1820.png 424w, https://substackcdn.com/image/fetch/$s_!b1UQ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde8c2251-a41e-4008-8093-22095c6b47c8_2880x1820.png 848w, https://substackcdn.com/image/fetch/$s_!b1UQ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde8c2251-a41e-4008-8093-22095c6b47c8_2880x1820.png 1272w, https://substackcdn.com/image/fetch/$s_!b1UQ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde8c2251-a41e-4008-8093-22095c6b47c8_2880x1820.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!b1UQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde8c2251-a41e-4008-8093-22095c6b47c8_2880x1820.png" width="2880" height="1820" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/de8c2251-a41e-4008-8093-22095c6b47c8_2880x1820.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1820,&quot;width&quot;:2880,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;An eight-cell vertical output strip uses eight A operands and one repeated B operand. Beside it, a four-by-four output patch combines a four-value A column with a four-value B row.&quot;,&quot;title&quot;:&quot;Kernel 4 keeps an eight-element accumulator strip, reusing one B value across eight A values. Kernel 5's 4 &#215; 4 patch uses four A and four B operands in an outer product. Per K-step, the source-level count goes from eight FFMAs using nine operand values to sixteen FFMAs using eight. These are logical reuse counts, not measured traffic or guaranteed register allocation.&quot;,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="An eight-cell vertical output strip uses eight A operands and one repeated B operand. Beside it, a four-by-four output patch combines a four-value A column with a four-value B row." title="Kernel 4 keeps an eight-element accumulator strip, reusing one B value across eight A values. Kernel 5's 4 &#215; 4 patch uses four A and four B operands in an outer product. Per K-step, the source-level count goes from eight FFMAs using nine operand values to sixteen FFMAs using eight. These are logical reuse counts, not measured traffic or guaranteed register allocation." srcset="https://substackcdn.com/image/fetch/$s_!b1UQ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde8c2251-a41e-4008-8093-22095c6b47c8_2880x1820.png 424w, https://substackcdn.com/image/fetch/$s_!b1UQ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde8c2251-a41e-4008-8093-22095c6b47c8_2880x1820.png 848w, https://substackcdn.com/image/fetch/$s_!b1UQ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde8c2251-a41e-4008-8093-22095c6b47c8_2880x1820.png 1272w, https://substackcdn.com/image/fetch/$s_!b1UQ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fde8c2251-a41e-4008-8093-22095c6b47c8_2880x1820.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"><strong>Figure 5.</strong> Kernel 4 keeps an eight-element accumulator strip, reusing one B value across eight A values. Kernel 5's 4 &#215; 4 patch uses four A and four B operands in an outer product. Per K-step, the source-level count goes from eight FFMAs using nine operand values to sixteen FFMAs using eight. These are logical reuse counts, not measured traffic or guaranteed register allocation.</figcaption></figure></div><h3>Kernel 6: organize the work at warp level</h3><p><a href="https://github.com/swesty/Kernel-Engg/blob/9d1e6110dfade157c55f686907a9611f2ee83ff0/lecture10/matmul-06.py">Kernel 6</a> added a level between the block tile and the thread patch: the block&#8217;s output tile was divided into warp-sized regions, and the 32 lanes of each warp covered their region together. It also stored A transposed in shared memory and expressed groups of four adjacent accesses, aligning the layout with the way threads consumed operands. (A source pattern that touches four adjacent values doesn&#8217;t prove the compiler emitted a vector load; the generated code has to be checked.)</p><p>The transpose matters because shared memory is split into 32 banks of four-byte words. Different addresses in the same bank force a warp&#8217;s access into extra service steps, so rearranging a shared tile can help even though the mathematics is unchanged. What counts is the set of addresses the lanes request for one instruction, not whether a drawing of the matrix looks contiguous. NVIDIA&#8217;s <a href="https://docs.nvidia.com/cuda/cuda-c-best-practices-guide/index.html">CUDA Best Practices Guide</a> covers coalescing, shared-memory reuse, and bank conflicts.</p><h3>Kernel 7: tune the geometry</h3><p><a href="https://github.com/swesty/Kernel-Engg/blob/9d1e6110dfade157c55f686907a9611f2ee83ff0/lecture10/matmul-07.py">Kernel 7</a> ran a Python autotuning search over 11 candidate configurations of block tile, warp tile, K-chunk depth, and thread patch. Numba compiled each candidate, and the harness timed them and kept the fastest. No single tile size wins by maximizing reuse alone: a larger tile reduces repeated loads but uses more registers or shared memory, a deeper K chunk reduces loop overhead but needs more buffering, and a warp arrangement that helps one operand&#8217;s access pattern can complicate the other&#8217;s. The search made those choices empirical.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!XUvL!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1495966-5c40-47a3-a3e2-563de5da2a35_2880x1720.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!XUvL!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1495966-5c40-47a3-a3e2-563de5da2a35_2880x1720.png 424w, https://substackcdn.com/image/fetch/$s_!XUvL!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1495966-5c40-47a3-a3e2-563de5da2a35_2880x1720.png 848w, https://substackcdn.com/image/fetch/$s_!XUvL!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1495966-5c40-47a3-a3e2-563de5da2a35_2880x1720.png 1272w, https://substackcdn.com/image/fetch/$s_!XUvL!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1495966-5c40-47a3-a3e2-563de5da2a35_2880x1720.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!XUvL!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1495966-5c40-47a3-a3e2-563de5da2a35_2880x1720.png" width="2880" height="1720" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/b1495966-5c40-47a3-a3e2-563de5da2a35_2880x1720.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1720,&quot;width&quot;:2880,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A 64-by-64 block tile contains eight warp rectangles. One warp is enlarged to show 32 lane-numbered thread patches arranged in four rows and eight columns, each covering four-by-four outputs.&quot;,&quot;title&quot;:&quot;The displayed kernel 6 configuration divides a 64 &#215; 64 output tile into eight 16 &#215; 32 warp tiles, each holding 32 thread-owned 4 &#215; 4 patches. Kernel 7 keeps this ownership hierarchy and searches its dimensions, including the K-chunk depth. A valid source-level geometry doesn't establish the best compiled resource use or runtime.&quot;,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A 64-by-64 block tile contains eight warp rectangles. One warp is enlarged to show 32 lane-numbered thread patches arranged in four rows and eight columns, each covering four-by-four outputs." title="The displayed kernel 6 configuration divides a 64 &#215; 64 output tile into eight 16 &#215; 32 warp tiles, each holding 32 thread-owned 4 &#215; 4 patches. Kernel 7 keeps this ownership hierarchy and searches its dimensions, including the K-chunk depth. A valid source-level geometry doesn't establish the best compiled resource use or runtime." srcset="https://substackcdn.com/image/fetch/$s_!XUvL!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1495966-5c40-47a3-a3e2-563de5da2a35_2880x1720.png 424w, https://substackcdn.com/image/fetch/$s_!XUvL!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1495966-5c40-47a3-a3e2-563de5da2a35_2880x1720.png 848w, https://substackcdn.com/image/fetch/$s_!XUvL!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1495966-5c40-47a3-a3e2-563de5da2a35_2880x1720.png 1272w, https://substackcdn.com/image/fetch/$s_!XUvL!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb1495966-5c40-47a3-a3e2-563de5da2a35_2880x1720.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"><strong>Figure 6.</strong> The displayed kernel 6 configuration divides a 64 &#215; 64 output tile into eight 16 &#215; 32 warp tiles, each holding 32 thread-owned 4 &#215; 4 patches. Kernel 7 keeps this ownership hierarchy and searches its dimensions, including the K-chunk depth. A valid source-level geometry doesn't establish the best compiled resource use or runtime.</figcaption></figure></div><p>By this point the kernel had substantial data reuse. The remaining question was how consistently it could feed operands and instructions to the arithmetic units. If you want to watch kernels 1 to 7 step by step, the repository includes an interactive visualizer of each one&#8217;s thread ownership, global-memory sectors, shared-memory tiles, and registers:</p><div class="captioned-image-container"><figure><a class="image-link image2" target="_blank" href="https://inference.swesty.net/viz/gemm-kernels-visualizer.html" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!4jOv!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa19054c8-9f41-4521-b4a8-ee0479f6998a_1864x1726.png 424w, https://substackcdn.com/image/fetch/$s_!4jOv!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa19054c8-9f41-4521-b4a8-ee0479f6998a_1864x1726.png 848w, https://substackcdn.com/image/fetch/$s_!4jOv!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa19054c8-9f41-4521-b4a8-ee0479f6998a_1864x1726.png 1272w, https://substackcdn.com/image/fetch/$s_!4jOv!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa19054c8-9f41-4521-b4a8-ee0479f6998a_1864x1726.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!4jOv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa19054c8-9f41-4521-b4a8-ee0479f6998a_1864x1726.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/a19054c8-9f41-4521-b4a8-ee0479f6998a_1864x1726.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Beyond Kernel 7: Closing the Gap to cuBLAS on an H100 FP32 GEMM &#8212; GEMM kernels visualizer (static screenshot)&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:&quot;https://inference.swesty.net/viz/gemm-kernels-visualizer.html&quot;,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Beyond Kernel 7: Closing the Gap to cuBLAS on an H100 FP32 GEMM &#8212; GEMM kernels visualizer (static screenshot)" title="Beyond Kernel 7: Closing the Gap to cuBLAS on an H100 FP32 GEMM &#8212; GEMM kernels visualizer (static screenshot)" srcset="https://substackcdn.com/image/fetch/$s_!4jOv!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa19054c8-9f41-4521-b4a8-ee0479f6998a_1864x1726.png 424w, https://substackcdn.com/image/fetch/$s_!4jOv!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa19054c8-9f41-4521-b4a8-ee0479f6998a_1864x1726.png 848w, https://substackcdn.com/image/fetch/$s_!4jOv!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa19054c8-9f41-4521-b4a8-ee0479f6998a_1864x1726.png 1272w, https://substackcdn.com/image/fetch/$s_!4jOv!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa19054c8-9f41-4521-b4a8-ee0479f6998a_1864x1726.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></a><figcaption class="image-caption"><strong>Figure 7.</strong> Interactive thread-view and memory-view visualizer for kernels 1&#8211;7 (plus the course's tensor-core examples), from the Kernel-Engg repository. Pick a kernel tab, then step through the loop or drag the warp and lane sliders. Its timings column is from the course's T4 session, not the H100 runs in this article. <a href="https://inference.swesty.net/viz/gemm-kernels-visualizer.html">Open the interactive visualizer</a>.</figcaption></figure></div><h2>What the measurements can and can&#8217;t tell us</h2><p>The starting point came from the course authors&#8217; H100 measurements, not from my runs. The <a href="https://github.com/swesty/Kernel-Engg/blob/9d1e6110dfade157c55f686907a9611f2ee83ff0/kernel08-cp-async/LAB_NOTEBOOK.md#L55-L60">starting-point record in my lab notes</a> preserves it: 86.6 &#181;s for the course&#8217;s kernel 7 and 56.1 &#181;s for cuBLAS at 1024 &#215; 1024 FP32, about 65% of cuBLAS speed. I used those numbers as context for the challenge, not as a baseline. The record doesn&#8217;t capture the hardware model, software versions, clock policy, or repetition details, and the course&#8217;s measurement may have used different hardware or settings from mine. That likely accounts for its cuBLAS time (56.1 &#181;s) being faster than the 64.6 &#181;s cuBLAS measured in my confirmation run. So I never subtract my result from 86.6 &#181;s and call the difference a speedup; every comparison below is against cuBLAS measured in the same run.</p><p>There is also an implementation boundary. Kernels 8 to 14 were written in CUDA C++ with inline PTX, compiled at runtime with NVRTC through CuPy&#8217;s <code>RawKernel</code>, with separate nvcc builds for inspecting SASS (the GPU&#8217;s machine instructions). That is not a sequence of edits to the Numba kernel under the same compiler, so any comparison across the boundary includes changes in code generation as well as in the optimization itself.</p><p>The investigation used timings and profiles for different jobs. Timings decided whether a candidate was useful. Nsight Compute, including its per-instruction source view, suggested where to look next. A proposed explanation stayed a hypothesis until the code and the timings supported it. A counter comparison at one fixed configuration and the fastest result from a configuration sweep also answer different questions, and I&#8217;ve tried to say which one each number is.</p><h2>Kernel 8: asynchronous copies</h2><p>A GEMM kernel repeatedly loads chunks along the shared K dimension. In a simple implementation the block loads a chunk, waits until it&#8217;s usable, does the arithmetic, and then moves on, which is the load, synchronize, compute cycle in Figure 4. During those waits the arithmetic units may have nothing to issue from that warp.</p><p><a href="https://github.com/swesty/Kernel-Engg/blob/9d1e6110dfade157c55f686907a9611f2ee83ff0/kernel08-cp-async/matmul_cp_async.cu#L378-L397">Kernel 8</a> introduced per-thread <code>cp.async</code> copies from global memory into a multi-stage shared-memory ring buffer: a fixed set of reusable buffers, called stages, visited in a repeating order (with three stages: 0, 1, 2, then back to 0). Each stage holds one K chunk. The pipeline loads a future chunk into one stage while computing on a ready chunk in another, and reuses a stage only after its previous contents are no longer needed.</p><p>The <a href="https://github.com/swesty/Kernel-Engg/blob/9d1e6110dfade157c55f686907a9611f2ee83ff0/kernel08-cp-async/matmul_cp_async.cu#L94-L109">cp.async primitives</a> issue an asynchronous global-to-shared copy, and issuing a copy is not the same as having its result. <code>cp.async.wait_group</code> waits on the issuing thread&#8217;s own copy groups, and <code>__syncthreads()</code> then makes every thread&#8217;s copies visible across the block: copy completion and coordination between threads are separate requirements. The benefit is a chance to put useful arithmetic between issuing a request and waiting for it, instead of stopping after every load. NVIDIA&#8217;s <a href="https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/async-copies.html">asynchronous copy documentation</a> describes the model.</p><p>More stages aren&#8217;t free: they occupy shared memory, and if buffering reduces the number of blocks that fit on an SM the trade can go the wrong way. So stage count went into the configuration search.</p><p>At the fixed configuration used for the profile (128 &#215; 64 tiles, B<sub>K</sub> = 8, n = 1,024), the share of warp cycles spent waiting on global memory (Nsight&#8217;s long-scoreboard stall) fell from 46.8% with one stage to 2.5% with three, below cuBLAS&#8217;s 6.6%. The best 1,024 configuration in that sweep reached about 89% of cuBLAS. The single-stage control was not identical to the course&#8217;s kernel 7, so this doesn&#8217;t isolate the full difference between the course implementation and this one. The remaining gap was in how often the scheduler issued instructions, not in memory.</p><p>This was ordinary per-thread <code>cp.async</code>. TMA, the Hopper Tensor Memory Accelerator, appears later only as a proposed experiment; it is not part of any measured result.</p><h2>Kernel 9: the next operand bottleneck</h2><p>Kernels 8 and 9 apply the same idea at two levels of the hierarchy: prepare future operands while computing with ones that are already ready. Kernel 8 pipelined global-to-shared copies. <a href="https://github.com/swesty/Kernel-Engg/blob/9d1e6110dfade157c55f686907a9611f2ee83ff0/kernel08-cp-async/matmul_cp_async.cu#L398-L433">Kernel 9</a> kept that pipeline and added shared-to-register prefetching into two alternating sets of registers. One set holds the operands for the current K-step while ordinary shared-memory loads fill the other for the next step (no second <code>cp.async</code> path is involved). That doesn&#8217;t make a dependent multiply-add independent of its inputs; it moves the load earlier, so the inputs are more likely to be ready when the multiply-add needs them.</p><p>A separate switch stored A transposed in shared memory, so a thread&#8217;s eight A values for a K-step could be fetched with two 128-bit (<code>float4</code>) loads instead of eight scalar loads. As with kernel 6, the generated instructions are what count: a source expression that mentions four values isn&#8217;t proof of one vector load.</p><p>Two Nsight metrics were useful here because they describe different halves of the hypothesis. The short-scoreboard stall tracks a class of dependency waits that includes shared-memory results; it is not &#8220;all time spent loading shared memory.&#8221; Issue-active is how often the scheduler issued an instruction; it is not a percentage of peak FLOP/s. The <a href="https://docs.nvidia.com/nsight-compute/ProfilingGuide/">Nsight Compute Profiling Guide</a> is the reference for both. With both changes, short-scoreboard stalls fell from 0.38 to 0.13 cycles per issued instruction and issue-active rose from 52% to 62%. The best 1,024 configuration reached about 97% of cuBLAS.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!mWFq!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce755c5c-a804-46aa-ad1a-1f947a65873d_2880x1840.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!mWFq!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce755c5c-a804-46aa-ad1a-1f947a65873d_2880x1840.png 424w, https://substackcdn.com/image/fetch/$s_!mWFq!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce755c5c-a804-46aa-ad1a-1f947a65873d_2880x1840.png 848w, https://substackcdn.com/image/fetch/$s_!mWFq!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce755c5c-a804-46aa-ad1a-1f947a65873d_2880x1840.png 1272w, https://substackcdn.com/image/fetch/$s_!mWFq!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce755c5c-a804-46aa-ad1a-1f947a65873d_2880x1840.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!mWFq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce755c5c-a804-46aa-ad1a-1f947a65873d_2880x1840.png" width="2880" height="1840" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/ce755c5c-a804-46aa-ad1a-1f947a65873d_2880x1840.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1840,&quot;width&quot;:2880,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A serial row alternates load and compute. Below it, copy and compute rows show the next global chunk loaded while the current shared chunk is used, and a second pair of rows shows register prefetch overlapping FFMA work.&quot;,&quot;title&quot;:&quot;Two levels of overlap. The global-to-shared pipeline uses per-thread cp.async copies to prepare future K chunks while the kernel computes on a ready shared-memory stage. Register prefetching does the same one level closer, preparing the next operand fragments while the current ones are consumed. These are schematic orderings, not measured timelines; completion and synchronization are still required before use.&quot;,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A serial row alternates load and compute. Below it, copy and compute rows show the next global chunk loaded while the current shared chunk is used, and a second pair of rows shows register prefetch overlapping FFMA work." title="Two levels of overlap. The global-to-shared pipeline uses per-thread cp.async copies to prepare future K chunks while the kernel computes on a ready shared-memory stage. Register prefetching does the same one level closer, preparing the next operand fragments while the current ones are consumed. These are schematic orderings, not measured timelines; completion and synchronization are still required before use." srcset="https://substackcdn.com/image/fetch/$s_!mWFq!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce755c5c-a804-46aa-ad1a-1f947a65873d_2880x1840.png 424w, https://substackcdn.com/image/fetch/$s_!mWFq!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce755c5c-a804-46aa-ad1a-1f947a65873d_2880x1840.png 848w, https://substackcdn.com/image/fetch/$s_!mWFq!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce755c5c-a804-46aa-ad1a-1f947a65873d_2880x1840.png 1272w, https://substackcdn.com/image/fetch/$s_!mWFq!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fce755c5c-a804-46aa-ad1a-1f947a65873d_2880x1840.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"><strong>Figure 8.</strong> Two levels of overlap. The global-to-shared pipeline uses per-thread cp.async copies to prepare future K chunks while the kernel computes on a ready shared-memory stage. Register prefetching does the same one level closer, preparing the next operand fragments while the current ones are consumed. These are schematic orderings, not measured timelines; completion and synchronization are still required before use.</figcaption></figure></div><p>The two switches were also measured separately, and alone each gave mixed results:</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!5lN4!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd9e22375-5929-4b36-a9ca-68778bc83276_1864x362.png 424w, https://substackcdn.com/image/fetch/$s_!5lN4!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd9e22375-5929-4b36-a9ca-68778bc83276_1864x362.png 848w, https://substackcdn.com/image/fetch/$s_!5lN4!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd9e22375-5929-4b36-a9ca-68778bc83276_1864x362.png 1272w, https://substackcdn.com/image/fetch/$s_!5lN4!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd9e22375-5929-4b36-a9ca-68778bc83276_1864x362.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!5lN4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd9e22375-5929-4b36-a9ca-68778bc83276_1864x362.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/d9e22375-5929-4b36-a9ca-68778bc83276_1864x362.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Beyond Kernel 7: Closing the Gap to cuBLAS on an H100 FP32 GEMM &#8212; chart 2&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Beyond Kernel 7: Closing the Gap to cuBLAS on an H100 FP32 GEMM &#8212; chart 2" title="Beyond Kernel 7: Closing the Gap to cuBLAS on an H100 FP32 GEMM &#8212; chart 2" srcset="https://substackcdn.com/image/fetch/$s_!5lN4!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd9e22375-5929-4b36-a9ca-68778bc83276_1864x362.png 424w, https://substackcdn.com/image/fetch/$s_!5lN4!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd9e22375-5929-4b36-a9ca-68778bc83276_1864x362.png 848w, https://substackcdn.com/image/fetch/$s_!5lN4!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd9e22375-5929-4b36-a9ca-68778bc83276_1864x362.png 1272w, https://substackcdn.com/image/fetch/$s_!5lN4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd9e22375-5929-4b36-a9ca-68778bc83276_1864x362.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption"><strong>Figure 9.</strong> Kernel 9's two changes, alone and together, at n = 1,024 and two fixed configurations. In the timing sweep, transposing A alone helped slightly and prefetching alone was slightly slower; in the profile configuration, transposing alone was much slower. Only the combination was clearly faster in both. The highlighted cell is the combination.</figcaption></figure></div><p>The consistent result was the combination. Without prefetch, the transposed layout&#8217;s wider loads still stalled the warp that issued them; without the transpose, prefetch needed eight scalar loads per fragment. Together, the loads could move earlier <em>and</em> there were fewer of them. That win was limited to n = 1,024: at 2,048 and 4,096, kernel 9 gave no gain (79&#8211;83% of cuBLAS).</p><h2>A plausible optimization that didn&#8217;t improve runtime</h2><p>The larger matrices suggested another opportunity. At n = 4,096 each matrix is 64 MB, more than the H100&#8217;s 50 MB L2 cache, and blocks launched in plain row order. Changing the order in which blocks process output tiles (block swizzling) can make neighboring blocks reuse more of the same input panels through L2, and if fewer bytes reach DRAM it&#8217;s reasonable to expect the kernel to get faster. I grouped launches eight block-rows at a time, similar to Triton&#8217;s <code>GROUP_M</code>.</p><p>The swizzle did what it was designed to do. DRAM reads fell by 22% and 53% on the two configurations tested, and the L2 hit rate rose. Runtime didn&#8217;t follow:</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Lyvw!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Facfe4310-a77c-4bca-8829-51c543241371_1864x1046.png 424w, https://substackcdn.com/image/fetch/$s_!Lyvw!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Facfe4310-a77c-4bca-8829-51c543241371_1864x1046.png 848w, https://substackcdn.com/image/fetch/$s_!Lyvw!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Facfe4310-a77c-4bca-8829-51c543241371_1864x1046.png 1272w, https://substackcdn.com/image/fetch/$s_!Lyvw!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Facfe4310-a77c-4bca-8829-51c543241371_1864x1046.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!Lyvw!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Facfe4310-a77c-4bca-8829-51c543241371_1864x1046.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/acfe4310-a77c-4bca-8829-51c543241371_1864x1046.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Beyond Kernel 7: Closing the Gap to cuBLAS on an H100 FP32 GEMM &#8212; chart 3&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Beyond Kernel 7: Closing the Gap to cuBLAS on an H100 FP32 GEMM &#8212; chart 3" title="Beyond Kernel 7: Closing the Gap to cuBLAS on an H100 FP32 GEMM &#8212; chart 3" srcset="https://substackcdn.com/image/fetch/$s_!Lyvw!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Facfe4310-a77c-4bca-8829-51c543241371_1864x1046.png 424w, https://substackcdn.com/image/fetch/$s_!Lyvw!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Facfe4310-a77c-4bca-8829-51c543241371_1864x1046.png 848w, https://substackcdn.com/image/fetch/$s_!Lyvw!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Facfe4310-a77c-4bca-8829-51c543241371_1864x1046.png 1272w, https://substackcdn.com/image/fetch/$s_!Lyvw!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Facfe4310-a77c-4bca-8829-51c543241371_1864x1046.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption"><strong>Figure 10.</strong> Block swizzling at n = 4,096 (Nsight Compute replay times). Each arrow runs from the row-order launch (hollow) to the swizzled launch (filled). Swizzling cut DRAM reads on both configurations, but one got 14% slower and the other 4% faster, and neither approached cuBLAS, which reads less still and runs faster.</figcaption></figure></div><p>The profile explains why. Time spent waiting on global memory was at most 6% in every case, swizzled or not. At n = 4,096 the best later kernel needs about 0.13 TB/s of HBM bandwidth, roughly 4% of what the H100 provides. DRAM traffic was not what limited these kernels, so cutting it changed nothing that mattered. This is the clearest example of a successful local change that wasn&#8217;t a performance improvement, and it would have been easy to keep the change on the strength of the memory counters alone. (The switch stayed on in later kernels because it didn&#8217;t hurt.)</p><h2>Larger thread patches and less address bookkeeping</h2><h3>Kernel 10: bigger patches</h3><p><a href="https://github.com/swesty/Kernel-Engg/blob/9d1e6110dfade157c55f686907a9611f2ee83ff0/kernel08-cp-async/matmul_cp_async.cu#L148-L203">Kernel 10</a> increased the per-thread patch to 16 &#215; 8 or 8 &#215; 16, the same 128 outputs per thread that cuBLAS&#8217;s own kernel uses at n = 4,096. A T<sub>M</sub> &#215; T<sub>N</sub> outer product uses T<sub>M</sub> + T<sub>N</sub> operand values to update T<sub>M</sub> &#215; T<sub>N</sub> accumulators, so larger patches make each operand serve more multiply-adds. But accumulator count grows with the product of the dimensions, and these configurations ran at about 255 registers per thread.</p><p>The layout had to change with the patch. With eight thread columns, contiguous B columns would put two lanes in the same shared-memory bank, a two-way conflict, so kernel 10 used a strided mapping for B columns that is conflict-free. Both the amount of work per thread and the way threads fetched operands changed, so &#8220;a larger tile&#8221; isn&#8217;t a complete explanation of the result.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!NIB7!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c9aa178-7e8e-4ef8-817e-62a5cd9f4b67_2880x1660.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!NIB7!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c9aa178-7e8e-4ef8-817e-62a5cd9f4b67_2880x1660.png 424w, https://substackcdn.com/image/fetch/$s_!NIB7!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c9aa178-7e8e-4ef8-817e-62a5cd9f4b67_2880x1660.png 848w, https://substackcdn.com/image/fetch/$s_!NIB7!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c9aa178-7e8e-4ef8-817e-62a5cd9f4b67_2880x1660.png 1272w, https://substackcdn.com/image/fetch/$s_!NIB7!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c9aa178-7e8e-4ef8-817e-62a5cd9f4b67_2880x1660.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!NIB7!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c9aa178-7e8e-4ef8-817e-62a5cd9f4b67_2880x1660.png" width="2880" height="1660" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/7c9aa178-7e8e-4ef8-817e-62a5cd9f4b67_2880x1660.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1660,&quot;width&quot;:2880,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;A small eight-by-eight patch is compared with a sixteen-by-eight patch, with counts of operand values, FFMA updates, and accumulators.&quot;,&quot;title&quot;:&quot;At one K-step, a TM &#215; TN patch performs TM &#215; TN FFMA updates using TM + TN operand values. Enlarging an 8 &#215; 8 patch to 16 &#215; 8 raises arithmetic per operand but doubles accumulator storage from 64 values to 128. These are reuse counts, not an isolated timing comparison.&quot;,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="A small eight-by-eight patch is compared with a sixteen-by-eight patch, with counts of operand values, FFMA updates, and accumulators." title="At one K-step, a TM &#215; TN patch performs TM &#215; TN FFMA updates using TM + TN operand values. Enlarging an 8 &#215; 8 patch to 16 &#215; 8 raises arithmetic per operand but doubles accumulator storage from 64 values to 128. These are reuse counts, not an isolated timing comparison." srcset="https://substackcdn.com/image/fetch/$s_!NIB7!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c9aa178-7e8e-4ef8-817e-62a5cd9f4b67_2880x1660.png 424w, https://substackcdn.com/image/fetch/$s_!NIB7!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c9aa178-7e8e-4ef8-817e-62a5cd9f4b67_2880x1660.png 848w, https://substackcdn.com/image/fetch/$s_!NIB7!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c9aa178-7e8e-4ef8-817e-62a5cd9f4b67_2880x1660.png 1272w, https://substackcdn.com/image/fetch/$s_!NIB7!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7c9aa178-7e8e-4ef8-817e-62a5cd9f4b67_2880x1660.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"><strong>Figure 11.</strong> At one K-step, a TM &#215; TN patch performs TM &#215; TN FFMA updates using TM + TN operand values. Enlarging an 8 &#215; 8 patch to 16 &#215; 8 raises arithmetic per operand but doubles accumulator storage from 64 values to 128. These are reuse counts, not an isolated timing comparison.</figcaption></figure></div><p>At n = 4,096, shared-memory loads fell from 336 million to 101 million, matching cuBLAS, and performance reached about 90% of cuBLAS (88% at 2,048). The n = 1,024 result dipped at this step, from 97% to 92%, but that dip was on the same 128 &#215; 64 configuration as before, not on the new large patches. It coincided with the B-column remap, a later test (kernel 13) ruled the remap out, and I never isolated the cause, so I treat it as likely run-to-run variation rather than a property of larger patches.</p><h3>Kernel 11: hoist the address math</h3><p>At this point, Nsight&#8217;s per-instruction view showed the hot loop spending a lot of issue slots on integer instructions (<code>LEA</code>, <code>IMAD</code>, <code>LDC</code>) that cuBLAS&#8217;s hot loop didn&#8217;t execute at all. At 255 registers, the compiler was re-reading the matrix dimensions from constant memory and recomputing every <code>cp.async</code> source address for every chunk. <a href="https://github.com/swesty/Kernel-Engg/blob/9d1e6110dfade157c55f686907a9611f2ee83ff0/kernel08-cp-async/matmul_cp_async.cu#L217-L293">Kernel 11</a> precomputed each thread&#8217;s copy pointers and shared-memory offsets once (hoisting them out of the loop), advanced them by fixed strides, and replaced the ring buffer&#8217;s <code>t % STAGES</code> with add-and-wrap. None of this reduces the multiply-adds the product needs; it reduces the work around them.</p><p>Overhead instructions per FFMA fell from 0.170 to 0.075 (cuBLAS: 0.131), registers dropped from 255 to 231, and the larger sizes reached 94&#8211;95% of cuBLAS. Not every prediction held: wait stalls fell to 0.08 rather than the predicted 0.02, and the per-instruction view showed why. The running pointer update had compiled into a chain of dependent 64-bit multiply-adds issued in one burst at each chunk boundary.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!JYca!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3bfc8cc-2dba-4aef-b405-4f7e334d0573_2880x1750.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!JYca!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3bfc8cc-2dba-4aef-b405-4f7e334d0573_2880x1750.png 424w, https://substackcdn.com/image/fetch/$s_!JYca!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3bfc8cc-2dba-4aef-b405-4f7e334d0573_2880x1750.png 848w, https://substackcdn.com/image/fetch/$s_!JYca!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3bfc8cc-2dba-4aef-b405-4f7e334d0573_2880x1750.png 1272w, https://substackcdn.com/image/fetch/$s_!JYca!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3bfc8cc-2dba-4aef-b405-4f7e334d0573_2880x1750.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!JYca!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3bfc8cc-2dba-4aef-b405-4f7e334d0573_2880x1750.png" width="2880" height="1750" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e3bfc8cc-2dba-4aef-b405-4f7e334d0573_2880x1750.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1750,&quot;width&quot;:2880,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Before-and-after flow diagrams compare repeated global and shared address derivation with setup once, followed by copy, compute, fixed-stride pointer updates, and ring-stage wrapping.&quot;,&quot;title&quot;:&quot;Kernel 11 precomputes copy pointers and shared-memory offsets, then advances them by fixed strides, and the ring-buffer index uses increment-and-wrap instead of modulo. This summarizes the source transformation, not the exact machine-instruction sequence.&quot;,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Before-and-after flow diagrams compare repeated global and shared address derivation with setup once, followed by copy, compute, fixed-stride pointer updates, and ring-stage wrapping." title="Kernel 11 precomputes copy pointers and shared-memory offsets, then advances them by fixed strides, and the ring-buffer index uses increment-and-wrap instead of modulo. This summarizes the source transformation, not the exact machine-instruction sequence." srcset="https://substackcdn.com/image/fetch/$s_!JYca!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3bfc8cc-2dba-4aef-b405-4f7e334d0573_2880x1750.png 424w, https://substackcdn.com/image/fetch/$s_!JYca!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3bfc8cc-2dba-4aef-b405-4f7e334d0573_2880x1750.png 848w, https://substackcdn.com/image/fetch/$s_!JYca!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3bfc8cc-2dba-4aef-b405-4f7e334d0573_2880x1750.png 1272w, https://substackcdn.com/image/fetch/$s_!JYca!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe3bfc8cc-2dba-4aef-b405-4f7e334d0573_2880x1750.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"><strong>Figure 12.</strong> Kernel 11 precomputes copy pointers and shared-memory offsets, then advances them by fixed strides, and the ring-buffer index uses increment-and-wrap instead of modulo. This summarizes the source transformation, not the exact machine-instruction sequence.</figcaption></figure></div><p>Loop unrolling supported several of these changes. The CUDA C++ file unrolls every loop with a compile-time trip count, which is what lets the accumulators and double buffers live in registers and turns kernel 11&#8217;s offsets into instruction immediates (<a href="https://docs.nvidia.com/cuda/cuda-programming-guide/05-appendices/cpp-language-extensions.html">CUDA C++ loop-unrolling controls</a>). The course&#8217;s Numba kernels already unrolled their small inner patch loops but not the K-step loop. I never measured unrolling on its own, and comparisons of instruction counts across different configurations and GPUs can&#8217;t assign a gain to it alone.</p><h3>Kernels 12 and 13: refine the schedule without adding overhead</h3><p>Kernel 11&#8217;s remaining stalls clustered at chunk boundaries, so <a href="https://github.com/swesty/Kernel-Engg/blob/9d1e6110dfade157c55f686907a9611f2ee83ff0/kernel08-cp-async/matmul08.py#L68-L69">kernel 12</a> spread each chunk&#8217;s copies across the K-steps, addressing each from the chunk&#8217;s base so there was no dependency chain, and separately tried a different multiply-add order. Two versions can move the same bytes and perform the same arithmetic but expose very different dependency chains to the scheduler. Issuing a load earlier helps if independent instructions are available before its result is needed; it doesn&#8217;t help if the change adds enough bookkeeping to consume the time it saved.</p><p>Kernel 12 showed both sides. In its own sweep it reached parity with cuBLAS at n = 1,024 (64.9 &#181;s against 64.8, a tie within run-to-run noise, with the alternate FFMA order) and was the best kernel yet at 2,048 (355.6 &#181;s against 340.9, 96%). At 4,096 it gained nothing: 2,779.4 &#181;s against kernel 11&#8217;s 2,772.9. The stalls it targeted did fall, but it executed about 57 million more instructions, and the breakdown wasn&#8217;t the address arithmetic I&#8217;d expected:</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!BqAH!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff3585da5-f2d3-4589-86f7-3defe4cef4c8_1864x626.png 424w, https://substackcdn.com/image/fetch/$s_!BqAH!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff3585da5-f2d3-4589-86f7-3defe4cef4c8_1864x626.png 848w, https://substackcdn.com/image/fetch/$s_!BqAH!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff3585da5-f2d3-4589-86f7-3defe4cef4c8_1864x626.png 1272w, https://substackcdn.com/image/fetch/$s_!BqAH!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff3585da5-f2d3-4589-86f7-3defe4cef4c8_1864x626.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!BqAH!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff3585da5-f2d3-4589-86f7-3defe4cef4c8_1864x626.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/f3585da5-f2d3-4589-86f7-3defe4cef4c8_1864x626.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Beyond Kernel 7: Closing the Gap to cuBLAS on an H100 FP32 GEMM &#8212; chart 4&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Beyond Kernel 7: Closing the Gap to cuBLAS on an H100 FP32 GEMM &#8212; chart 4" title="Beyond Kernel 7: Closing the Gap to cuBLAS on an H100 FP32 GEMM &#8212; chart 4" srcset="https://substackcdn.com/image/fetch/$s_!BqAH!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff3585da5-f2d3-4589-86f7-3defe4cef4c8_1864x626.png 424w, https://substackcdn.com/image/fetch/$s_!BqAH!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff3585da5-f2d3-4589-86f7-3defe4cef4c8_1864x626.png 848w, https://substackcdn.com/image/fetch/$s_!BqAH!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff3585da5-f2d3-4589-86f7-3defe4cef4c8_1864x626.png 1272w, https://substackcdn.com/image/fetch/$s_!BqAH!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ff3585da5-f2d3-4589-86f7-3defe4cef4c8_1864x626.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption"><strong>Figure 13.</strong> Where kernel 12's extra instructions came from at n = 4,096 (256 &#215; 128 tile), from Nsight's per-instruction counts for kernels 11 and 12. Most of the increase is a never-true-predicated shared load that ptxas emitted as filler, plus a kernel parameter reloaded at each copy site. Total executed instructions rose from 2,322 M to 2,379 M.</figcaption></figure></div><p><code>LDS</code> is a shared-memory load; the <code>@!PT</code> predicate is never true, so the instruction does nothing, but it still occupies an issue slot. <code>MOV</code> moves values between registers, and <code>ULDC</code> loads a constant through the uniform datapath. The likely source of both is the per-K-step branch that decided whether to issue a spread copy, though I didn&#8217;t prove that. Either way, a better stall counter wasn&#8217;t evidence of a faster kernel; the runtime decided.</p><p><a href="https://github.com/swesty/Kernel-Engg/blob/9d1e6110dfade157c55f686907a9611f2ee83ff0/kernel08-cp-async/matmul08.py#L70-L72">Kernel 13</a> tested three things. A branch-free version of copy spreading, which always issues every copy and uses a zero-byte copy past the end, didn&#8217;t buy time. Reverting to contiguous B columns, to test whether kernel 10&#8217;s remap had hurt, was worse everywhere, by up to 10% at n = 4,096: the useful layout was the one that suited the warp&#8217;s actual accesses, not the one that looked simplest. And size specialization, compiling the matrix size in as a constant so the compiler can fold address math into immediates, gave about one point at large sizes. A specialized kernel is a legitimate candidate for a fixed workload, but it doesn&#8217;t have the flexibility of a library routine that accepts sizes at runtime, so the results report specialized and generic candidates separately.</p><h2>Kernel 14 and Stream-K</h2><p>At n = 1,024, every kernel so far, cuBLAS included, ran <strong>one warp per scheduler</strong>. The best configuration (kernel 11, 128 &#215; 64 tiles) launched 128 blocks of four warps onto 132 SMs: at most one block per SM, so each of an SM&#8217;s four warp schedulers had exactly one warp to choose from. When that warp stalled on a dependency, the scheduler had nothing else to issue, which is why short-scoreboard and dispatch stalls showed up directly as lost issue slots. Making each individual operation faster doesn&#8217;t fix that; the scheduler needs a second warp.</p><p><a href="https://github.com/swesty/Kernel-Engg/blob/9d1e6110dfade157c55f686907a9611f2ee83ff0/kernel08-cp-async/matmul_cp_async.cu#L563-L599">Kernel 14</a> returned to the hoisted kernel 11 design and added a persistent <a href="https://arxiv.org/abs/2301.03598">Stream-K</a> kernel. It&#8217;s a branch from kernel 11, not kernel 13 with every experiment retained; the <a href="https://github.com/swesty/Kernel-Engg/blob/9d1e6110dfade157c55f686907a9611f2ee83ff0/kernel08-cp-async/matmul08.py#L58-L81">variant definitions</a> show the winning variant uses neither copy spreading nor size specialization. A persistent kernel launches only as many blocks as fit at once (264 here, two per SM) and loops over the work. Stream-K changes the unit of that work: instead of each block owning one output tile&#8217;s entire K reduction, the stream of (tile, K-chunk) steps is divided evenly across blocks, so a block boundary can fall partway through a tile. This implementation does full waves of whole tiles first and splits only what&#8217;s left. At n = 1,024 with 128 &#215; 128 tiles there are just 64 tiles, fewer than the 264 resident blocks, so all of the work is split, and each tile&#8217;s K range is shared by roughly four blocks. Their partial sums are combined with vector <code>atomicAdd</code> into an output that a separate kernel zeroes first.</p><div class="captioned-image-container"><figure><a class="image-link image2 is-viewable-img" target="_blank" href="https://substackcdn.com/image/fetch/$s_!ACwc!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F24b4f4b3-6fe2-49c8-b27d-96aa405ddd1f_2880x1590.png" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!ACwc!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F24b4f4b3-6fe2-49c8-b27d-96aa405ddd1f_2880x1590.png 424w, https://substackcdn.com/image/fetch/$s_!ACwc!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F24b4f4b3-6fe2-49c8-b27d-96aa405ddd1f_2880x1590.png 848w, https://substackcdn.com/image/fetch/$s_!ACwc!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F24b4f4b3-6fe2-49c8-b27d-96aa405ddd1f_2880x1590.png 1272w, https://substackcdn.com/image/fetch/$s_!ACwc!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F24b4f4b3-6fe2-49c8-b27d-96aa405ddd1f_2880x1590.png 1456w" sizes="auto, 100vw"><img src="https://substackcdn.com/image/fetch/$s_!ACwc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F24b4f4b3-6fe2-49c8-b27d-96aa405ddd1f_2880x1590.png" width="2880" height="1590" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/24b4f4b3-6fe2-49c8-b27d-96aa405ddd1f_2880x1590.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:1590,&quot;width&quot;:2880,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Four output tiles, A through D, each contain four K chunks. Three block assignments cross tile boundaries, splitting the K ranges for tiles B and C, and two partial sums for tile B are combined with atomic additions.&quot;,&quot;title&quot;:&quot;Stream-K partitions a stream of (output tile, K-chunk) work instead of assigning whole tiles, so several blocks can contribute to one tile, and their partial sums are added atomically into a pre-zeroed output. The zeroing and atomic accumulation are extra work that must be inside the timing. The partition is illustrative, not the launch configuration.&quot;,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Four output tiles, A through D, each contain four K chunks. Three block assignments cross tile boundaries, splitting the K ranges for tiles B and C, and two partial sums for tile B are combined with atomic additions." title="Stream-K partitions a stream of (output tile, K-chunk) work instead of assigning whole tiles, so several blocks can contribute to one tile, and their partial sums are added atomically into a pre-zeroed output. The zeroing and atomic accumulation are extra work that must be inside the timing. The partition is illustrative, not the launch configuration." srcset="https://substackcdn.com/image/fetch/$s_!ACwc!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F24b4f4b3-6fe2-49c8-b27d-96aa405ddd1f_2880x1590.png 424w, https://substackcdn.com/image/fetch/$s_!ACwc!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F24b4f4b3-6fe2-49c8-b27d-96aa405ddd1f_2880x1590.png 848w, https://substackcdn.com/image/fetch/$s_!ACwc!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F24b4f4b3-6fe2-49c8-b27d-96aa405ddd1f_2880x1590.png 1272w, https://substackcdn.com/image/fetch/$s_!ACwc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F24b4f4b3-6fe2-49c8-b27d-96aa405ddd1f_2880x1590.png 1456w" sizes="auto, 100vw" loading="lazy"></picture><div class="image-link-expand"><div class="pencraft pc-display-flex pc-gap-8 pc-reset"><button class="pencraft pc-reset pencraft icon-container restack-image buttonBase-tqxFUX" tabindex="0" type="button"><svg aria-hidden="true" width="20" height="20" viewBox="0 0 20 20" fill="none" stroke-width="1.5" stroke="var(--color-fg-primary)" stroke-linecap="round" stroke-linejoin="round" xmlns="http://www.w3.org/2000/svg" class="icon-noB79L"><g><path d="M2.53001 7.81595C3.49179 4.73911 6.43281 2.5 9.91173 2.5C13.1684 2.5 15.9537 4.46214 17.0852 7.23684L17.6179 8.67647M17.6179 8.67647L18.5002 4.26471M17.6179 8.67647L13.6473 6.91176M17.4995 12.1841C16.5378 15.2609 13.5967 17.5 10.1178 17.5C6.86118 17.5 4.07589 15.5379 2.94432 12.7632L2.41165 11.3235M2.41165 11.3235L1.5293 15.7353M2.41165 11.3235L6.38224 13.0882"></path></g></svg></button><button class="pencraft pc-reset pencraft icon-container view-image buttonBase-tqxFUX" tabindex="0" type="button"><svg xmlns="http://www.w3.org/2000/svg" width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" class="lucide lucide-maximize2 lucide-maximize-2 icon-noB79L"><polyline points="15 3 21 3 21 9"></polyline><polyline points="9 21 3 21 3 15"></polyline><line x1="21" x2="14" y1="3" y2="10"></line><line x1="3" x2="10" y1="21" y2="14"></line></svg></button></div></div></div></a><figcaption class="image-caption"><strong>Figure 14.</strong> Stream-K partitions a stream of (output tile, K-chunk) work instead of assigning whole tiles, so several blocks can contribute to one tile, and their partial sums are added atomically into a pre-zeroed output. The zeroing and atomic accumulation are extra work that must be inside the timing. The partition is illustrative, not the launch configuration.</figcaption></figure></div><p>This didn&#8217;t make the matrix multiplication smaller. It redistributed the work so each scheduler had two warps, and it added work of its own: zeroing the output, the atomic accumulation, and a different summation order. The profile shows what that bought:</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!DPFp!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ec892a9-46e5-483f-8231-ca3bf97ef307_1864x870.png 424w, https://substackcdn.com/image/fetch/$s_!DPFp!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ec892a9-46e5-483f-8231-ca3bf97ef307_1864x870.png 848w, https://substackcdn.com/image/fetch/$s_!DPFp!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ec892a9-46e5-483f-8231-ca3bf97ef307_1864x870.png 1272w, https://substackcdn.com/image/fetch/$s_!DPFp!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ec892a9-46e5-483f-8231-ca3bf97ef307_1864x870.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!DPFp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ec892a9-46e5-483f-8231-ca3bf97ef307_1864x870.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2ec892a9-46e5-483f-8231-ca3bf97ef307_1864x870.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Beyond Kernel 7: Closing the Gap to cuBLAS on an H100 FP32 GEMM &#8212; chart 5&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Beyond Kernel 7: Closing the Gap to cuBLAS on an H100 FP32 GEMM &#8212; chart 5" title="Beyond Kernel 7: Closing the Gap to cuBLAS on an H100 FP32 GEMM &#8212; chart 5" srcset="https://substackcdn.com/image/fetch/$s_!DPFp!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ec892a9-46e5-483f-8231-ca3bf97ef307_1864x870.png 424w, https://substackcdn.com/image/fetch/$s_!DPFp!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ec892a9-46e5-483f-8231-ca3bf97ef307_1864x870.png 848w, https://substackcdn.com/image/fetch/$s_!DPFp!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ec892a9-46e5-483f-8231-ca3bf97ef307_1864x870.png 1272w, https://substackcdn.com/image/fetch/$s_!DPFp!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2ec892a9-46e5-483f-8231-ca3bf97ef307_1864x870.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption"><strong>Figure 15.</strong> Kernel 11, kernel 14, and cuBLAS at n = 1,024 (Nsight Compute). Stream-K doubled the warps available to each scheduler. Each warp actually stalled more per instruction (2.77 cycles against 1.60), but with two warps, one can issue while the other waits, so the SM issued on 72% of cycles, more than cuBLAS's 70%. Instruction counts were about equal (38 M against 41 M), so time followed the issue rate.</figcaption></figure></div><p>The counterintuitive part is the third panel. The win didn&#8217;t come from any warp stalling less; per-warp stalls went up. It came from having a second warp to cover them. The confirmation timing included both the zeroing kernel and the Stream-K kernel, because a result that counted only the fast part of the computation would be much less useful.</p><h2>The confirmed result</h2><p>The final confirmation ran on an NVIDIA H100 80GB HBM3 (132 SMs) on Modal, with CUDA 13, PyTorch 2.14.1+cu130, and CuPy 14.2. The cuBLAS reference was <code>torch.matmul</code>, with FP32 inputs and arithmetic and TF32 disabled. Each candidate was checked for correctness first. cuBLAS and the candidates were then timed in 15 alternating rounds, each reporting the median of 40 CUDA-event measurements, with a 256 MiB buffer written before each launch to flush L2. Comparing each kernel against cuBLAS within the same round cancels slow drift in clocks and temperature. Those conditions matter when comparing these numbers with benchmarks that reuse warm data.</p><p>At 1024 &#215; 1024, the generic kernel 14 configuration took 61.328 &#181;s against 64.624 &#181;s for cuBLAS. The median paired speed ratio was 1.05295 (105.0&#8211;105.8% across rounds), and the custom kernel was ahead in all 15 rounds. That&#8217;s about 5.3% higher throughput, or 5.1% lower latency. At 2048 &#215; 2048, the best size-specialized kernel 13f took 357.9 &#181;s against 341.3 &#181;s, 95.4% of cuBLAS; at 4096 &#215; 4096 it took 2,761.2 &#181;s against 2,661.1 &#181;s, 96.4%. The best generic kernel 11 configurations reached 93.9% and 95.2% at those sizes. The best numbers across sizes didn&#8217;t all come from one universally winning kernel.</p><p>In absolute terms, kernel 14 at n = 1,024 ran at 35.0 TFLOP/s against cuBLAS&#8217;s 33.2, which is 57% and 54% of the 61.3 TFLOP/s FP32 FFMA ceiling I measured on an H100 in an earlier course assignment. At n = 4,096, k13f reached 49.8 TFLOP/s against cuBLAS&#8217;s 51.6 (81% and 84%). &#8220;Beating cuBLAS&#8221; here means getting closer to the same compute roof than cuBLAS did at one size, not exceeding the hardware.</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!JVkD!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F92ec4d5f-55fc-4f3a-a141-7395d9cf372c_1864x1102.png 424w, https://substackcdn.com/image/fetch/$s_!JVkD!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F92ec4d5f-55fc-4f3a-a141-7395d9cf372c_1864x1102.png 848w, https://substackcdn.com/image/fetch/$s_!JVkD!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F92ec4d5f-55fc-4f3a-a141-7395d9cf372c_1864x1102.png 1272w, https://substackcdn.com/image/fetch/$s_!JVkD!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F92ec4d5f-55fc-4f3a-a141-7395d9cf372c_1864x1102.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!JVkD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F92ec4d5f-55fc-4f3a-a141-7395d9cf372c_1864x1102.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/92ec4d5f-55fc-4f3a-a141-7395d9cf372c_1864x1102.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;Beyond Kernel 7: Closing the Gap to cuBLAS on an H100 FP32 GEMM &#8212; chart 6&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="Beyond Kernel 7: Closing the Gap to cuBLAS on an H100 FP32 GEMM &#8212; chart 6" title="Beyond Kernel 7: Closing the Gap to cuBLAS on an H100 FP32 GEMM &#8212; chart 6" srcset="https://substackcdn.com/image/fetch/$s_!JVkD!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F92ec4d5f-55fc-4f3a-a141-7395d9cf372c_1864x1102.png 424w, https://substackcdn.com/image/fetch/$s_!JVkD!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F92ec4d5f-55fc-4f3a-a141-7395d9cf372c_1864x1102.png 848w, https://substackcdn.com/image/fetch/$s_!JVkD!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F92ec4d5f-55fc-4f3a-a141-7395d9cf372c_1864x1102.png 1272w, https://substackcdn.com/image/fetch/$s_!JVkD!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F92ec4d5f-55fc-4f3a-a141-7395d9cf372c_1864x1102.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div><figcaption class="image-caption"><strong>Figure 16.</strong> Roofline position of the confirmed kernels. The roofs are the FP32 FFMA peak (61.3 TFLOP/s) and HBM copy bandwidth (3.03 TB/s) measured on an H100 in an earlier course assignment, which meet at 20.2 FLOP/byte. Arithmetic intensity comes from Nsight's DRAM counters with C's write-back included; throughput from the confirmation medians. Every point sits roughly 8&#8211;17&#215; to the right of the ridge, so compute issue, not DRAM bandwidth, is the limit. k13f is omitted because its DRAM traffic wasn't profiled.</figcaption></figure></div><p>That picture summarizes the whole investigation. None of the kernels was limited by DRAM bandwidth, so the swizzle&#8217;s traffic reduction had nowhere to go. Every gain from kernel 8 to kernel 14 moved a point straight up toward the roof by raising issue efficiency: hiding latency (kernels 8 and 9), executing fewer overhead instructions (10 and 11), and giving each scheduler more warps (14).</p><h2>What the result establishes</h2><p>The 1,024 result is a repeatable win within the tested setup, with a limited scope. The experiments covered square, tile-multiple shapes, not arbitrary dimensions with edge handling. Stream-K&#8217;s atomic accumulation changes summation order, so results are not bit-reproducible. Correctness was checked by requiring finite outputs and a maximum absolute error, normalized by the reference&#8217;s maximum absolute value, below 10&#8315;&#8308;.</p><p>The remaining gap at larger sizes gives a concrete next question. Stream-K helped the small case, but the measured Stream-K variants lost to the non-Stream-K kernels at 2,048 and 4,096 (89&#8211;94%). Higher register use from inlining the tile body twice is a plausible contributor, not an isolated cause. TMA is another proposed direction, starting with B&#8217;s copies while preserving A&#8217;s working layout. Neither should be described as a measured success yet.</p><h2>Summary</h2><p>The useful progression in this work was the sequence of questions. After improving reuse, I checked whether global-memory latency was exposed. After overlapping those copies, I looked at shared-memory operand delivery. After increasing arithmetic per load, I looked at the address instructions around that arithmetic. And for the small case, the final improvement came from giving each scheduler more than one warp.</p><p>The first seven kernels had already fixed inefficient global accesses and established reuse through shared-memory tiles, register patches, and warp-level organization. The later experiments didn&#8217;t discard that foundation; they investigated what still prevented it from supplying the arithmetic consistently. Async copies reduced exposed global-memory waiting. Register double buffering and the transposed layout targeted the next operand dependency, and only worked together. Larger patches cut shared-load work, and hoisting cut the instructions around the multiply-adds. The swizzle and copy-spreading experiments showed why a better-looking counter didn&#8217;t always mean a better runtime. Stream-K provided enough parallel work to produce a repeatable win at 1,024.</p><p>That sequence is also where the LLM-assisted process needs scrutiny. A plausible explanation can point to a useful experiment, but the explanation alone doesn&#8217;t establish what made the kernel faster. The source, the generated instructions, the correctness checks, and the timings are what let me decide which parts of the story hold up.</p><h2>Evidence for review</h2><p>Course information is on the <a href="https://kernelworkshop.vizuara.ai/">workshop website</a> and the <a href="https://maven.com/rajat-dandekar/kernel-engineering">Maven course page</a>; <a href="https://vizuara.ai/">Vizuara AI Labs</a> is the organization&#8217;s main site. The <a href="https://github.com/VizuaraAI/Kernel-Engg">official course repository</a> has the course source. The course credits <a href="https://siboehm.com/articles/22/CUDA-MMM">Simon Boehm&#8217;s worklog</a> as an inspiration; the explanations here follow the course repository, with the worklog as an external comparison.</p><p><a href="https://github.com/swesty/Kernel-Engg/tree/9d1e6110dfade157c55f686907a9611f2ee83ff0">My fork</a> holds the lab implementations and the later experiments. Every link in this article is pinned to that commit, so the line numbers won&#8217;t drift:</p><ul><li><p>Kernels 1&#8211;4: <a href="https://github.com/swesty/Kernel-Engg/blob/9d1e6110dfade157c55f686907a9611f2ee83ff0/lecture8/Lecture8_MatMul_Kernels.ipynb">Lecture8_MatMul_Kernels.ipynb</a></p></li><li><p>Kernels 5&#8211;7: <a href="https://github.com/swesty/Kernel-Engg/blob/9d1e6110dfade157c55f686907a9611f2ee83ff0/lecture10/matmul-05.py">matmul-05.py</a>, <a href="https://github.com/swesty/Kernel-Engg/blob/9d1e6110dfade157c55f686907a9611f2ee83ff0/lecture10/matmul-06.py">matmul-06.py</a>, <a href="https://github.com/swesty/Kernel-Engg/blob/9d1e6110dfade157c55f686907a9611f2ee83ff0/lecture10/matmul-07.py">matmul-07.py</a></p></li><li><p>Kernels 8&#8211;14 share <a href="https://github.com/swesty/Kernel-Engg/blob/9d1e6110dfade157c55f686907a9611f2ee83ff0/kernel08-cp-async/matmul_cp_async.cu">matmul_cp_async.cu</a>, with build switches and the variant mapping in <a href="https://github.com/swesty/Kernel-Engg/blob/9d1e6110dfade157c55f686907a9611f2ee83ff0/kernel08-cp-async/matmul08.py">matmul08.py</a></p></li><li><p>The investigation narrative: <a href="https://github.com/swesty/Kernel-Engg/blob/9d1e6110dfade157c55f686907a9611f2ee83ff0/kernel08-cp-async/LAB_NOTEBOOK.md">LAB_NOTEBOOK.md</a> and <a href="https://github.com/swesty/Kernel-Engg/blob/9d1e6110dfade157c55f686907a9611f2ee83ff0/kernel08-cp-async/BIG_WINS.md">BIG_WINS.md</a></p></li><li><p>The final confirmation: <a href="https://github.com/swesty/Kernel-Engg/blob/9d1e6110dfade157c55f686907a9611f2ee83ff0/kernel08-cp-async/results/runs/confirm.json">machine-readable JSON</a> and a <a href="https://github.com/swesty/Kernel-Engg/blob/9d1e6110dfade157c55f686907a9611f2ee83ff0/kernel08-cp-async/results/runs/confirm.txt">readable summary</a></p></li><li><p>The interactive visualizer: <a href="https://github.com/swesty/Kernel-Engg/tree/9d1e6110dfade157c55f686907a9611f2ee83ff0/visualizations/kernels">visualizations/kernels</a></p></li></ul><p>The confirmation artifacts date to November 4, 2026, and include source snapshots whose SHA-256 hashes match the linked source. Early profiler evidence is less complete because some profiles were overwritten; the <a href="https://github.com/swesty/Kernel-Engg/blob/9d1e6110dfade157c55f686907a9611f2ee83ff0/kernel08-cp-async/results/MANIFEST.md">results manifest</a> distinguishes surviving and rebuilt records. The performance claims above use confirmation timings, not Nsight replay timings; the charts of profiler counters say so where they&#8217;re used.</p><p>The early lab files are learning artifacts, not production-ready implementations. In particular, the saved kernel 5 and 6 files accumulate into local arrays without an explicit zero-initialization loop, which needs review before using them as standalone examples, and the final correctness checks apply to the later candidates, not retroactively to every early lab file. The kernel 1 and 2 explanations follow the executable index expressions; the notebook&#8217;s opening coalescing comments don&#8217;t consistently match them.</p>]]></content:encoded></item><item><title><![CDATA[vLLM on the NVIDIA DGX Spark (GB10 / sm_121): A Complete Technical Guide]]></title><description><![CDATA[A companion, published after the field notes. I published the field-notes post, vLLM on the DGX Spark, the Hard Way, first: that&#8217;s the operational story, what broke, what I run in production, and the measured numbers.]]></description><link>https://blog.swesty.net/p/vllm-dgx-spark-technical-report</link><guid isPermaLink="false">https://blog.swesty.net/p/vllm-dgx-spark-technical-report</guid><dc:creator><![CDATA[Shawn Westfall]]></dc:creator><pubDate>Mon, 20 Jul 2026 00:00:00 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/9c2c1566-f0e3-4d97-b4ba-b224d8cd9730_1188x380.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<blockquote><p><strong>A companion, published after the field notes.</strong> I published the field-notes post, <em><a href="/blog/vllm-on-dgx-spark-gb10-field-notes">vLLM on the DGX Spark, the Hard Way</a></em>, first: that&#8217;s the operational story, what broke, what I run in production, and the measured numbers. This post is the deep background behind it, the kernel, CUTLASS/FlashInfer, quantization, and backend-oracle detail I had to work through to understand <em>why</em> those field observations happen at all. Researching it was most of the actual work, and it seemed worth putting in one place for anyone hitting the same walls, so I&#8217;m releasing it as a companion. Read the field notes for what to do today; read this for why it works underneath. One caveat: this is a dated implementation audit (2026-06-02 against vLLM <code>main</code>), so treat the version-specific operational bits as a snapshot and defer to the field notes&#8217; timeline for the current state.</p><p><strong>Why this report exists.</strong> vLLM published its DGX Spark deep-dive on 2026-06-01. I deployed its guidance to my production fleet the next day, then ran it under real traffic for seven weeks. This report is the result. The recipe is correct, but high-level: it doesn&#8217;t resolve the implementation details that decide whether a given image actually serves &#8212; SM121 arch targeting, the exact order the backend oracle walks its candidates, FlashInfer&#8217;s cubin-versus-JIT behavior, and when the CUDA-graph escape hatches matter. I worked those out against the source, and that&#8217;s the subject here, next to an honest <a href="#the-scorecard-what-worked-what-didnt-what-i-found">scorecard</a> of what worked, what didn&#8217;t survive production, and what I had to find myself. Where the official blog and the repository diverge, the repository is authoritative for the audited commit. The body is a dated snapshot of that audit; the <a href="#whats-changed-since-the-audit">update timeline</a> records what has changed since.</p><p><strong>Scope.</strong> This is an all-in-one technical reference for running and understanding vLLM on the NVIDIA DGX Spark. It covers the hardware, <em>why</em> the platform has been hard for vLLM, the kernel libraries involved (CUTLASS, FlashInfer, Marlin, QuTLASS), how quantization (NVFP4 / MXFP4 / FP8) is wired up, how vLLM&#8217;s fused-MoE machinery selects backends, what <code>--enforce-eager</code> is and why it has been a crutch, the development history, and concrete guidance on prebuilt containers vs. source builds.</p><p><strong>Provenance.</strong> Code citations are against the vLLM <code>main</code> checkout at commit <code>689e...</code> (2026-06-02). Release-timing and the operational recommendations are reconciled with the official blog <em>&#8220;vLLM on NVIDIA DGX Spark&#8221;</em> (<a href="https://vllm.ai/blog/2026-06-01-vllm-dgx-spark">https://vllm.ai/blog/2026-06-01-vllm-dgx-spark</a>). Where the repo and field experience diverge, the repo wins and the divergence is called out.</p><p><strong>&#9201; Freshness &amp; maintenance: implementation audit dated 2026-06-02 (vLLM <code>main</code> @ <code>689e&#8230;</code>).</strong> <code>v0.22.0</code> is treated throughout as the <strong>first audited stable baseline</strong> containing the Spark fixes discussed, <strong>not</strong> as the latest release as of this post&#8217;s date (stable releases have continued since; see the update timeline). Re-audit backend selection and dependency pins against the exact image or tag you deploy. sm_121 support is a fast-moving target (vLLM stable cadence &#8776; every 2&#8211;3 weeks), so claims carry inline date tags: <strong><code>[verified YYYY-MM-DD]</code></strong> = checked against the repo/blog on that date; <strong><code>[since vX.Y.Z]</code></strong> = release a fix first shipped in. <strong>Time-volatile sections to re-check every vLLM release:</strong> &#167;4.4 (the <code>b12x</code>-is-opt-in exclusion, the single highest-churn fact here; it flips when flashinfer-ai/flashinfer#3134 lands upstream), &#167;7.2&#8211;7.3 (backend auto-selection order), &#167;9 (release&#8594;fix mapping), &#167;10 (recommended image + dependency pins). The architectural material (&#167;1, &#167;3, &#167;5&#8211;&#167;6, &#167;8) is structural and ages slowly. When you re-verify, bump the date tag even if nothing changed: a current date is the signal the claim still holds.</p></blockquote><div><hr></div><h2>What&#8217;s changed since the audit</h2><p>The body below is the 2026-06-02 audit snapshot. Between that audit and this post&#8217;s publication (body assembled 2026-07-20; this timeline last extended 2026-08-05), the following moved. Each entry is either verified on my own production fleet (details and receipts in <a href="/blog/vllm-on-dgx-spark-gb10-field-notes/">the GB10 field notes</a>) or checked against the public release record on the date shown.</p><ul><li><p><strong>2026-06-05 &#8594; 2026-07-05: MTP under real concurrency</strong> <em>[fleet-verified]</em>: in-checkpoint MTP validated at 86.8% acceptance, then hit <code>CUDA error: an illegal memory access</code> every few hours in production, only under concurrent load (upstream #41190/#42084).</p></li><li><p><strong>2026-07-11/12: native FP4 confirmed working in production</strong> <em>[fleet-verified]</em>: a genuine W4A4 checkpoint (4-bit activations, <code>lm_head</code> unquantized) put dense <strong>and</strong> MoE layers on the <code>FLASHINFER_CUTLASS</code> native-FP4 path, faster than forced Marlin on the boxes, and it eliminated the MTP crash (8-way soak, 400/400, zero IMA). &#167;6&#8217;s checkpoint-format emphasis aged well; any lingering &#8220;Marlin is mandatory on GB10&#8221; reading of &#167;5 did not.</p></li><li><p><strong>2026-07-14: vLLM <code>v0.25.1</code> released</strong> <em>[PyPI, checked 2026-07-20]</em>: multiple stable releases now postdate the audited <code>v0.22.0</code> baseline.</p></li><li><p><strong>2026-07-19: official cu129 stable still ships no sm_121 cubins</strong> <em>[fleet-verified]</em>: smoke-testing the v0.25.1 stable candidates reproduced the report&#8217;s central lesson: wheel/cubin coverage, not version number, decides whether a path works. My fleet runs digest-pinned nightlies for exactly this reason.</p></li><li><p><strong>2026-07-27: v0.26.0 released; cu129 aarch64 stable now HAS sm_121 NVFP4 cubins</strong> <em>[fleet scratch-tested 2026-08-05]</em>: <code>auto</code> selects <code>FLASHINFER_CUTLASS</code> for both dense and MoE with no <code>no kernel image is available</code>, which breaks the 2026-07-19 lesson above (the 0.22.0 &#8594; 0.25.1 cu129 cubin drought is over). It still couldn&#8217;t serve my unsloth W4A4 checkpoint, though: EngineCore init dies on a CUTLASS FP8 W8A8 GEMM (<code>cutlass_gemm_caller.cuh:62, Error Internal</code>), a path the same checkpoint clears on my cu13-line nightly. The lesson refines rather than breaks: the failure moved from a missing NVFP4 kernel image to an FP8 CUTLASS error, still wheel/CUDA-line-specific, not version-specific.</p></li><li><p><strong>2026-07-29: #41190 (MTP illegal-memory-access) fixed on <code>main</code></strong> <em>[PR #48245/#49620]</em>: root cause turned out to be a kernel-agnostic scheduler bug (async-decode output-placeholder underflow under KV-pressure preemption), not the drafter kernel. Not yet in a tagged stable; upstream marks it a probable &#8220;may fix.&#8221; Stable-wheel operators stay exposed until a nightly carries it.</p></li><li><p><strong>2026-08-05: #42084 (prefix-caching + spec-decode OOB) appears resolved on a v0.26.1-dev nightly</strong> <em>[fleet scratch-tested]</em>: prefix caching and MTP now run together via an experimental Mamba/SSM cache-<code>align</code> mode (it aligns the attention block size to the linear-attention page size). Validated with a shared-prefix + MTP soak (40% cache hit, 86.9% KV, zero OOB or asserts). This lifts the prefix-caching-vs-spec-decode either/or on recent nightlies, with the caveat that vLLM marks the align path experimental. See the field notes&#8217; 2026-08-05 update for the run detail.</p></li><li><p><strong>2026-08-05: #31424 (NGC container lag) closed <code>not_planned</code></strong>: the vendor-container lag is now permanent-by-neglect.</p></li><li><p><strong>Re-verified 2026-08-05:</strong> &#167;4.4&#8217;s <code>b12x</code> auto-selection exclusion still holds, checked in <code>_POSSIBLE_NVFP4_KERNELS</code> (dense) and the MoE oracle at v0.26.0 and <code>main</code>; b12x is excluded from <code>auto</code> on both paths and the flashinfer-ai/flashinfer#3134 guard has not landed. <strong>Still not re-audited, re-check before relying on:</strong> &#167;7.2&#8211;7.3 (backend auto-selection order) and &#167;10 (image and dependency pins).</p></li></ul><div><hr></div><h2>The scorecard: what worked, what didn&#8217;t, what I found</h2><p>I adopted the blog&#8217;s recipe in one commit (2026-06-02, one day after publication) and have run it under 24/7 agent traffic since. The verdict, with receipts:</p><h3>What worked</h3><ul><li><p><strong>The <code>cu130-nightly</code> image track</strong>: adopted as a gitsha-pinned nightly (0.22.1rc1.dev26, CUDA 13.0.2), per the blog&#8217;s own &#8220;pin a digest&#8221; clause. It unlocked native FP4 immediately: <code>--moe-backend auto</code> &#8594; <code>FLASHINFER_CUTLASS</code>, <strong>~55.8&#8211;58.5 tok/s vs ~50 on forced Marlin</strong> (2026-06-02, batch-1 decode). Still the fleet&#8217;s serving basis today.</p></li><li><p><strong>Trusting <code>auto</code> backend selection, on that image track.</strong> The blog&#8217;s claim that &#8220;recent FlashInfer CUTLASS paths are much stronger than older Spark guidance suggests&#8221; measured out on the boxes the same day.</p></li><li><p><strong>CUDA graphs enabled.</strong> I retired <code>--enforce-eager</code> fleet-wide in the same commit (it had been a v0.20.0 Mamba graph-capture workaround, &#167;8) and never looked back.</p></li><li><p><strong>Built-in <code>--reasoning-parser nemotron_v3</code></strong> over NVIDIA&#8217;s HF-card parser plugin: simpler, coherent, no plugin file.</p></li><li><p><strong>The JIT pre-warm ping</strong> (<code>max_tokens: 3</code> at boot): I run the same trick the blog describes.</p></li></ul><h3>What didn&#8217;t, or didn&#8217;t survive production</h3><ul><li><p><strong><code>auto</code> on the images that predated the blog.</strong> On cu129/v0.20.0 images, <code>auto</code> crashed; my cards from that era read &#8220;cu129 stock: auto crashes; marlin correct.&#8221; The blog&#8217;s advice was true of the image track it tested and wrong for the installed base that preceded it. Reconciling that (blog vs. repo vs. my logs) is how I learned it was a <strong>version gap, not a dispute</strong>, and it&#8217;s why every claim in this report carries a commit and date.</p></li><li><p><strong>The demo-shaped numeric defaults.</strong> <code>--gpu-memory-utilization 0.85</code> matches my own single-model figure, but co-residency forced 0.3&#8211;0.4, and taught me the unified-memory rule the hard way: <strong>the GPU reservation is unswappable, so the reservation itself is the memory pressure</strong> (0.6 &#8776; 71 GiB got my 35B OOM-killed on 2026-07-03). <code>--max-num-seqs 4</code> and <code>--max-model-len 131072</code> became 4/8/16 and 32K/262K per box role. None of these are errors in the blog; they&#8217;re demo-shaped numbers that don&#8217;t transfer to a multi-model production box.</p></li><li><p><strong>MTP as &#8220;evaluate.&#8221;</strong> I productionized <code>--speculative-config</code> MTP and paid the tuition: 86.8% acceptance in validation, then <code>CUDA illegal memory access</code> every few hours <strong>only under concurrent load</strong> (upstream #41190/#42084). The configuration that avoided it, a genuine W4A4 checkpoint putting experts <em>and</em> drafter on the CUTLASS path, is the report&#8217;s biggest self-discovered lesson (below). The blog&#8217;s &#8220;validate before relying on it&#8221; clause was fully vindicated; batch-1 validation was not enough.</p></li><li><p><strong>The fp8 KV-cache caution: inverted by measurement.</strong> The blog frames <code>--kv-cache-dtype fp8</code> as a memory-pressure concession (&#8220;may affect predictability&#8230; noticeable performance cost&#8230; avoid unless memory pressure requires it and quality checks pass&#8221;). I ran the quality check it prescribes (2026-07-20, A/B on the production 35B, temp-0 task-scored extraction + greedy probes): <strong>byte-identical extraction output across arms 10/10 docs, fp8 </strong><em><strong>more</strong></em><strong> deterministic intra-arm (10/10 vs 9/10), and ~27% faster decode</strong> (96.6 vs 76.3 tok/s, halved KV traffic on a bandwidth-bound box, with MTP re-reading KV every verification step), plus double the 262K-context capacity in the same pool (~3.2 vs ~1.6 sequences). Not an error in the blog (its escape clause is exactly what I executed), but on decode-bound long-context serving the cost-benefit framing runs the other way: fp8 KV is a <em>default with a per-checkpoint validation step</em>, not a concession.</p></li></ul><h3>What I found that the blog never mentions</h3><ul><li><p><strong>The checkpoint&#8217;s quant format decides the FP4 backend, not your flags</strong> (W4A4 &#8594; native CUTLASS; W4A16 &#8594; Marlin fallback). The load-bearing rule of this entire report (&#167;6), and the actual fix for the MTP crashes.</p></li><li><p><strong><code>b12x</code> is opt-in, not auto</strong> at the audited commit (&#167;4.4): <code>auto</code> cannot select the Spark-native MoE path the blog&#8217;s era shipped.</p></li><li><p><strong>Backend flags the recipe needs but the blog never names</strong>: <code>--attention-backend</code> (TRITON_ATTN / flashinfer per model), <code>--mamba-ssm-cache-dtype float32</code> for hybrid-Mamba, <code>--max-num-batched-tokens</code> prefill chunking, and <code>--async-scheduling</code>, including the discovery that spec-decode configurations can silently disable async scheduling and tax prefill (&#167;the n-gram lesson in my <a href="/blog/vllm-on-dgx-spark-gb10-field-notes/">field notes</a>).</p></li><li><p><strong>Coordinator interactions</strong>: behind a llamactl-style proxy, <code>--served-model-name</code> 404s every request; the blog&#8217;s standalone recipe breaks unmodified behind a gateway.</p></li><li><p><strong>Unified-memory operational arithmetic</strong>: KV-pool math from the boot log (per-token cost, boot floor &#8776; 0.24 at 262K ctx, &#8805;15 GiB host headroom) that turns <code>--gpu-memory-utilization</code> from folklore into subtraction.</p></li></ul><div><hr></div><h2>1. The DGX Spark architecture</h2><h3>1.1 What the box is</h3><p>The DGX Spark is built on the <strong>NVIDIA GB10 Grace Blackwell superchip</strong>, a Grace <strong>ARM64/aarch64</strong> CPU and a Blackwell GPU fused onto one package with a <strong>128 GB unified LPDDR5X memory pool</strong> shared between CPU and GPU. This single fact drives most of the operational quirks: there is no separate &#8220;VRAM&#8221;; weights, KV cache, the OS page cache, and the container runtime all draw from the same 128 GB. Memory bandwidth (~273 GB/s class) is far below a datacenter card, which is why decode throughput is the binding constraint on Spark, not compute.</p><p>vLLM recognizes the unified-memory nature of the platform explicitly:</p><ul><li><p><code>vllm/utils/mem_utils.py:110</code>: <em>&#8220;On UMA (Unified Memory Architecture) platforms where CPU and GPU share physical memory (e.g. GH200, DGX Spark, Jetson Orin)&#8230;&#8221;</em></p></li><li><p><code>vllm/platforms/interface.py:799</code>: <code>is_integrated_gpu()</code> documents DGX Spark as a UMA device.</p></li></ul><h3>1.2 The compute capability that matters: <code>sm_121</code></h3><p>The GB10 GPU reports <strong>compute capability 12.1</strong>, i.e. <code>sm_121</code> (architecture-specific target <code>sm_121a</code>). This is the single most important number in this entire document. It places Spark in the <strong>&#8220;SM12x family&#8221;</strong> of consumer/workstation Blackwell parts:</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Bvl3!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc22013e9-c20e-4502-ac50-e06af4dd552f_1188x380.png 424w, https://substackcdn.com/image/fetch/$s_!Bvl3!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc22013e9-c20e-4502-ac50-e06af4dd552f_1188x380.png 848w, https://substackcdn.com/image/fetch/$s_!Bvl3!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc22013e9-c20e-4502-ac50-e06af4dd552f_1188x380.png 1272w, https://substackcdn.com/image/fetch/$s_!Bvl3!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc22013e9-c20e-4502-ac50-e06af4dd552f_1188x380.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!Bvl3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc22013e9-c20e-4502-ac50-e06af4dd552f_1188x380.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/c22013e9-c20e-4502-ac50-e06af4dd552f_1188x380.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;vLLM on the NVIDIA DGX Spark (GB10 / sm_121): A Complete Technical Guide &#8212; table 1&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="vLLM on the NVIDIA DGX Spark (GB10 / sm_121): A Complete Technical Guide &#8212; table 1" title="vLLM on the NVIDIA DGX Spark (GB10 / sm_121): A Complete Technical Guide &#8212; table 1" srcset="https://substackcdn.com/image/fetch/$s_!Bvl3!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc22013e9-c20e-4502-ac50-e06af4dd552f_1188x380.png 424w, https://substackcdn.com/image/fetch/$s_!Bvl3!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc22013e9-c20e-4502-ac50-e06af4dd552f_1188x380.png 848w, https://substackcdn.com/image/fetch/$s_!Bvl3!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc22013e9-c20e-4502-ac50-e06af4dd552f_1188x380.png 1272w, https://substackcdn.com/image/fetch/$s_!Bvl3!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc22013e9-c20e-4502-ac50-e06af4dd552f_1188x380.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div></figure></div><p>vLLM&#8217;s own source documents the mapping in two kernel generators:</p><blockquote><p><code>csrc/quantization/marlin/generate_kernels.py:18</code> and <code>csrc/moe/marlin_moe_wna16/generate_kernels.py:18</code>: <code># SM89 and the SM12x family (SM120 RTX 5090, SM121 DGX Spark GB10)</code></p></blockquote><h3>1.3 The trap: <code>sm_121</code> is <em>not</em> <code>sm_100</code></h3><p>Datacenter Blackwell (<code>sm_100</code>) and workstation Blackwell (<code>sm_120</code>/<code>sm_121</code>) are <strong>different architectures with different tensor-core instruction sets</strong>, even though both are marketed as &#8220;Blackwell.&#8221; A kernel compiled and tuned for <code>sm_100</code> will not run on <code>sm_121</code>. In CUTLASS terms they require entirely separate kernel instantiations: different <code>BlockwiseScaleConfig</code>, different warp-specialized kernel schedules, and different <code>cutlass::arch::Sm1xx</code> tags (see &#167;3). The practical consequence: <strong>almost all generic &#8220;Blackwell NVFP4&#8221; guidance on the internet is written for <code>sm_100</code> and does not transfer to Spark.</strong> This is the root cause of nearly every Spark gotcha.</p><h3>1.4 What the tensor cores natively support</h3><p>The Marlin generator records the key hardware fact about the SM12x family:</p><blockquote><p><code>csrc/quantization/marlin/generate_kernels.py:18-23</code>:</p><pre><code># SM89 and the SM12x family (SM120 RTX 5090, SM121 DGX Spark GB10)
# fully support mma.sync.aligned.m16n8k32.row.col.f32.e4m3.e4m3.f32.
# SM90 and SM100 can use this PTX, but it's simulated
# with FP16 MMA, so it cannot achieve any acceleration.
if arch == 89 or arch // 10 == 12:
    SUPPORT_FP8 = True</code></pre></blockquote><p>So <code>sm_121</code> has <strong>native FP8 (<code>e4m3</code>) tensor-core MMA</strong>, and (via <code>OpClassBlockScaledTensorOp</code>) native <strong>block-scaled FP4</strong> tensor cores. The hardware is capable; the historical problems were entirely about <em>software coverage</em> for this specific architecture.</p><div><hr></div><h2>2. Why DGX Spark has been hard for vLLM</h2><p>Four compounding reasons:</p><p><strong>1. Architecture fragmentation + arch-specific cubins.</strong> Modern quantized GEMM kernels (CUTLASS, FlashInfer&#8217;s CuteDSL kernels) are compiled into <strong>architecture-specific cubins</strong>. A wheel that ships <code>sm_100a</code> and <code>sm_120a</code> cubins but <em>not</em> <code>sm_121a</code> will fail at runtime on Spark, not with a graceful fallback, but with a hard error or, worse, silent garbage. The canonical symptom:</p><blockquote><p><code>csrc/.../fp4/nvfp4_scaled_mm_entry.cu:66-68</code> (runtime SM dispatch):</p><pre><code>STD_TORCH_CHECK_NOT_IMPLEMENTED(
    false, "No compiled nvfp4 mm kernel for SM ", sm,
    ". Recompile with CUDA &gt;= 12.8 and CC &gt;= 100.");</code></pre></blockquote><p><strong>2. <code>sm_121</code> is the newest, least-covered target.</strong> <code>sm_120</code> (RTX 5090) support landed in vLLM in mid-2025; <code>sm_121</code> is a strict latecomer that had to be threaded through every kernel&#8217;s arch list one at a time (see &#167;9). For a long window, <code>sm_121</code> was <em>partially</em> covered: some kernels had it, others silently excluded it, producing inconsistent behavior.</p><p><strong>3. aarch64.</strong> Spark is ARM64. Historically the CUDA Python ecosystem was x86-first; aarch64 CUDA wheels were patchy. This is now largely solved (PyTorch &#8805;2.11.0 ships aarch64 CUDA wheels on PyPI, and vLLM publishes aarch64 wheels), but it added friction during the early Spark period and still bites anyone on older torch.</p><p><strong>4. CUDA toolkit floor.</strong> <code>sm_121</code> kernels require <strong>CUDA &#8805; 12.8</strong> to even compile; CUDA 13.0 is preferred (it introduces <em>family</em> targets, see &#167;3.4). Building against an older toolkit silently drops Spark from the arch list:</p><blockquote><p><code>CMakeLists.txt:99-110</code>:</p><pre><code>if(CMAKE_CUDA_COMPILER_VERSION VERSION_GREATER_EQUAL 13.0)
  set(CUDA_SUPPORTED_ARCHS "7.5;8.0;8.6;8.7;8.9;9.0;10.0;11.0;12.0")   # family targets
elseif(CMAKE_CUDA_COMPILER_VERSION VERSION_GREATER_EQUAL 12.8)
  set(CUDA_SUPPORTED_ARCHS "7.5;8.0;8.6;8.7;8.9;9.0;10.0;10.1;10.3;12.0;12.1")
else()
  set(CUDA_SUPPORTED_ARCHS "7.0;7.5;8.0;8.6;8.7;8.9;9.0")               # no sm_121 at all
endif()</code></pre></blockquote><p>The rest of this guide explains the moving parts that these four problems touch.</p><div><hr></div><h2>3. CUTLASS: the templated GEMM engine</h2><h3>3.1 What CUTLASS is and why it matters</h3><p><strong>CUTLASS</strong> (CUDA Templates for Linear Algebra Subroutines) is NVIDIA&#8217;s open-source C++ template library for building high-performance GEMMs directly on the tensor cores. vLLM uses CUTLASS 3.x (&#8220;c3x&#8221;) to implement its <em>first-choice</em> quantized matmuls: FP8 W8A8, NVFP4 W4A4, and the blockwise-scaled variants. When CUTLASS works for your arch + dtype, it is typically the fastest path, which is why the backend oracles try it before falling back to Marlin.</p><p>Key files:</p><ul><li><p><code>csrc/quantization/w8a8/cutlass/scaled_mm_c3x_sm120.cu</code>: FP8/INT8 scaled-MM entry for SM120/121</p></li><li><p><code>csrc/quantization/w8a8/cutlass/c3x/scaled_mm_sm120_fp8_dispatch.cuh</code>: per-tensor FP8, with M-shape-based tile selection</p></li><li><p><code>csrc/quantization/w8a8/cutlass/c3x/scaled_mm_blockwise_sm120_fp8_dispatch.cuh</code>: blockwise FP8</p></li><li><p><code>csrc/quantization/fp4/nvfp4_scaled_mm_sm120_kernels.cu</code>: NVFP4 GEMM</p></li><li><p><code>csrc/quantization/fp4/nvfp4_blockwise_moe_kernel.cu</code>: NVFP4 blockwise MoE</p></li><li><p><code>csrc/quantization/fp4/nvfp4_scaled_mm_entry.cu</code>: runtime SM dispatch (the source of the &#8220;No compiled nvfp4 mm kernel for SM &#8230;&#8221; error)</p></li></ul><h3>3.2 Why CUTLASS needs <em>separate</em> kernels for <code>sm_100</code> vs <code>sm_120</code>/<code>sm_121</code></h3><p>CUTLASS kernels are parameterized by an <strong>architecture tag</strong> that selects the MMA atoms and the warp-specialized scheduling strategy. Datacenter and workstation Blackwell get different tags and different schedules:</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!R4XI!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F08c986fb-00ef-4d96-abde-d63304f233ad_2064x302.png 424w, https://substackcdn.com/image/fetch/$s_!R4XI!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F08c986fb-00ef-4d96-abde-d63304f233ad_2064x302.png 848w, https://substackcdn.com/image/fetch/$s_!R4XI!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F08c986fb-00ef-4d96-abde-d63304f233ad_2064x302.png 1272w, https://substackcdn.com/image/fetch/$s_!R4XI!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F08c986fb-00ef-4d96-abde-d63304f233ad_2064x302.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!R4XI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F08c986fb-00ef-4d96-abde-d63304f233ad_2064x302.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/08c986fb-00ef-4d96-abde-d63304f233ad_2064x302.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;vLLM on the NVIDIA DGX Spark (GB10 / sm_121): A Complete Technical Guide &#8212; table 2&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="vLLM on the NVIDIA DGX Spark (GB10 / sm_121): A Complete Technical Guide &#8212; table 2" title="vLLM on the NVIDIA DGX Spark (GB10 / sm_121): A Complete Technical Guide &#8212; table 2" srcset="https://substackcdn.com/image/fetch/$s_!R4XI!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F08c986fb-00ef-4d96-abde-d63304f233ad_2064x302.png 424w, https://substackcdn.com/image/fetch/$s_!R4XI!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F08c986fb-00ef-4d96-abde-d63304f233ad_2064x302.png 848w, https://substackcdn.com/image/fetch/$s_!R4XI!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F08c986fb-00ef-4d96-abde-d63304f233ad_2064x302.png 1272w, https://substackcdn.com/image/fetch/$s_!R4XI!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F08c986fb-00ef-4d96-abde-d63304f233ad_2064x302.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div></figure></div><p>These are <em>distinct compiled kernels</em>; there is no shared binary. That is exactly why &#8220;it works on my B200&#8221; tells you nothing about Spark.</p><h3>3.3 The <code>enable_sm120_family</code> guard: one binary for SM120 <em>and</em> SM121</h3><p>The mechanism that lets a single compiled kernel serve both RTX-class (<code>sm_120</code>) and Spark (<code>sm_121</code>) is a thin device-side wrapper:</p><blockquote><p><code>csrc/cutlass_extensions/common.hpp</code> (&#8776; lines 128-142):</p><pre><code>// SM12x family includes SM120 (RTX 5090) and SM121 (DGX Spark GB10)
template &lt;typename Kernel&gt;
struct enable_sm120_family : Kernel {
  template &lt;typename... Args&gt;
  CUTLASS_DEVICE void operator()(Args&amp;&amp;... args) {
#if defined __CUDA_ARCH__
  #if (__CUDA_ARCH__ &gt;= 1200 &amp;&amp; __CUDA_ARCH__ &lt; 1300)
    Kernel::operator()(std::forward&lt;Args&gt;(args)...);
  #else
    printf("This kernel only supports sm120f.\n");
    asm("trap;");
  #endif
#endif
  }
};</code></pre></blockquote><p>The range check <code>__CUDA_ARCH__ &gt;= 1200 &amp;&amp; &lt; 1300</code> admits 1200 (<code>sm_120</code>) <strong>and</strong> 1210 (<code>sm_121</code>), the whole SM12x family. Several Spark bugs over the past months were precisely cases where a kernel used a narrower <code>sm_120</code>-only guard and thus <em>trapped or fell through on <code>sm_121</code></em>; the fixes consisted of routing those kernels through <code>enable_sm120_family</code> (e.g. the per-tensor FP8 dispatch fix, PR #41215).</p><h3>3.4 The <code>a</code> vs <code>f</code> suffix, and <code>cuda_archs_loose_intersection</code></h3><p><code>nvcc</code> arch targets carry a suffix:</p><ul><li><p><strong><code>12.1a</code> (architecture-specific):</strong> one cubin for exactly <code>sm_121</code>. To cover the family you must list every member: <code>12.0a;12.1a</code>.</p></li><li><p><strong><code>12.0f</code> (family):</strong> <em>one</em> cubin that runs across the whole SM12x family: <code>sm_120</code>, <code>sm_121</code>, and future SM12x. <strong>Requires CUDA &#8805; 13.0.</strong> (See NVIDIA&#8217;s family-specific-architecture docs.)</p></li></ul><p>vLLM&#8217;s CMake picks the right form by toolkit version. For example, scaled-MM:</p><blockquote><p><code>CMakeLists.txt:755-758</code>:</p><pre><code>if(${CMAKE_CUDA_COMPILER_VERSION} VERSION_GREATER_EQUAL 13.0)
  cuda_archs_loose_intersection(SCALED_MM_ARCHS "12.0f" "${CUDA_ARCHS}")
else()
  cuda_archs_loose_intersection(SCALED_MM_ARCHS "12.0a;12.1a" "${CUDA_ARCHS}")
endif()</code></pre></blockquote><p>The same pattern appears for <code>FP4_ARCHS</code>, <code>MLA_ARCHS</code>, <code>MARLIN_*_ARCHS</code>, etc. The helper <code>cuda_archs_loose_intersection</code> (<code>cmake/utils.cmake:368-416</code>) is what makes the <em>family</em> form match a user&#8217;s specific arch: given source <code>12.0f</code> and a target list containing <code>12.1a</code>, it strips suffixes, compares the <strong>major version</strong> (both <code>12</code>), and matches, so a single <code>12.0f</code> build covers Spark. This family-level matching (added in PR #38126, &#8220;Fix DGX Spark logic&#8221;) is why a CUDA-13 build with just <code>12.0f</code> correctly serves <code>sm_121</code>.</p><p><strong>Operator takeaway:</strong> if you build from source on CUDA 12.8/12.9 you must list <code>12.1a</code> (or <code>12.0a;12.1a</code>); on CUDA 13.0+ <code>12.0f</code> suffices. The in-repo Docker Grace-Blackwell example uses <code>'9.0 10.0+PTX'</code>. That is for GH200/GB200 and is <strong>wrong for GB10</strong>; you must override it.</p><div><hr></div><h2>4. FlashInfer: the kernel-serving layer</h2><h3>4.1 What FlashInfer is in vLLM</h3><p><strong>FlashInfer</strong> is a kernel library that vLLM leans on heavily for inference-optimized kernels. vLLM&#8217;s wrapper is <code>vllm/utils/flashinfer.py</code> (~1000 lines). Surface area spans:</p><ul><li><p><strong>Attention</strong>: paged/ragged batch decode &amp; prefill, TRT-LLM attention backends, MLA (DeepSeek), FP4 KV cache.</p></li><li><p><strong>Fused MoE</strong>: TRT-LLM BF16/FP8/FP4 block-scale MoE, CUTLASS fused MoE, CuteDSL NVFP4 MoE, and the SM12x <strong><code>b12x</code></strong> fused MoE.</p></li><li><p><strong>Quantized GEMM</strong>: FP4/MXFP4 quantize, NVFP4 scaled MM, block-scale FP8.</p></li><li><p><strong>RoPE</strong> and <strong>collective all-reduce</strong> (MNNVL/NVLink).</p></li></ul><h3>4.2 How FlashInfer ships kernels: prebuilt cubins vs. JIT</h3><p>This is the crux of FlashInfer&#8217;s relationship with Spark. FlashInfer kernels arrive one of two ways (<code>vllm/utils/flashinfer.py:27-63</code>):</p><ul><li><p><strong>Prebuilt cubins</strong>: the <code>flashinfer-cubin</code> package (pinned <code>flashinfer-cubin==0.6.11.post2</code> in <code>requirements/cuda.txt:13</code>) downloads precompiled cubins from an NVIDIA artifact repository. Fast, but <strong>only covers the arches NVIDIA chose to build.</strong></p></li><li><p><strong>JIT</strong>: if cubins are missing, FlashInfer compiles kernels at first request, which <strong>requires <code>nvcc</code> to be present</strong> in the container: </p><pre><code>if not has_flashinfer_cubin() and shutil.which("nvcc") is None:
    # FlashInfer unavailable: no cubins and no nvcc to JIT
    return False</code></pre></li></ul><p><strong>Why Spark broke:</strong> for a long while the prebuilt FlashInfer wheels shipped <strong>zero <code>sm_120</code>/<code>sm_121</code> cubins</strong>, and stock inference images don&#8217;t carry <code>nvcc</code> (it&#8217;s in the <em>devel</em> image, not <em>runtime</em>). So FlashInfer&#8217;s FP4 path either errored (<code>Failed to run cutlass FP4 gemm on sm120/sm121 &#8230; Error Internal</code>) or (the dangerous case) produced incoherent tokens with no error. This is the original sin behind the field rule &#8220;force Marlin on Spark.&#8221;</p><h3>4.3 <code>b12x</code>: the modern SM12x MoE / FP4 path</h3><p>The newer, Spark-native path is <strong><code>b12x</code></strong> (FlashInfer&#8217;s SM12x block-scaled GEMM, built on <strong>CuteDSL</strong>, the CUTLASS C++ tensor-layout DSL). It exists in two forms:</p><ul><li><p><strong>MoE expert:</strong> <code>vllm/model_executor/layers/fused_moe/experts/flashinfer_b12x_moe.py</code>:</p><blockquote><p><em>&#8220;FlashInfer CuteDSL fused MoE expert for SM12x (SM120/SM121, RTX Pro 6000 / DGX Spark). Uses <code>b12x_fused_moe</code> &#8230; which fuses token dispatch, two GEMMs, SwiGLU activation, and topk-weight reduction into a single kernel call. Input quantization (BF16&#8594;FP4) is performed inside the kernel &#8230; Only NVFP4 &#8230; is supported.&#8221;</em></p></blockquote><p>Notably, it takes <strong>BF16 hidden states directly</strong> and quantizes inside the kernel; scale factors are pre-converted to MMA layout once in <code>process_weights_after_loading</code>.</p></li><li><p><strong>Dense linear:</strong> <code>vllm/model_executor/kernels/linear/nvfp4/flashinfer.py:221</code>:</p><blockquote><p><code>class FlashInferB12xNvFp4LinearKernel(NvFp4LinearKernel): """NVFP4 GEMM via FlashInfer's b12x CuTe DSL warp-level MMA kernel (SM120+)."""</code></p></blockquote></li></ul><p>Both require CC &#8805; 12.0. <code>b12x</code> landed in vLLM in <strong>PR #40082</strong> (first released in <strong>v0.22.0</strong>).</p><h3>4.4 The critical nuance: <code>b12x</code> is <strong>opt-in</strong>, not auto <code>[verified 2026-06-02: highest-churn fact; re-check each release]</code></h3><p>This is the single most under-appreciated fact for Spark operators, and it refines the common &#8220;just leave backends on <code>auto</code>&#8221; advice. The official blog recommends leaving backend selection on <code>auto</code>, but <strong>at the audited vLLM <code>main</code> commit (<code>689e&#8230;</code>, 2026-06-02), <code>b12x</code> is deliberately excluded from auto-selection</strong> until an upstream CUTLASS <code>sm_121</code> MMA-guard issue (flashinfer-ai/flashinfer#3134) is resolved:</p><blockquote><p><code>vllm/model_executor/layers/fused_moe/oracle/nvfp4.py:170-173</code>:</p><pre><code># NOTE: the kernels are selected in the following order.
# FLASHINFER_B12X is intentionally excluded from auto-selection until
# the upstream CUTLASS SM121 MMA op guard is resolved; use
# moe_backend="flashinfer_b12x" to opt in explicitly.</code></pre><p>and the matching note for dense linears in <code>vllm/model_executor/kernels/linear/__init__.py:383-395</code>:</p><pre><code># FlashInferB12xNvFp4LinearKernel excluded from auto-selection until
# upstream CUTLASS SM121 MMA op guard is resolved; use
# VLLM_NVFP4_GEMM_BACKEND=flashinfer-b12x to opt in explicitly.</code></pre></blockquote><p>So on that code path, <code>--moe-backend auto</code> will <strong>not</strong> pick the Spark-native <code>b12x</code> kernel; it walks the list below (&#167;7) and lands on whatever else supports <code>sm_121</code>. To actually use <code>b12x</code> you pass <code>--moe-backend flashinfer_b12x</code> (and/or <code>--linear-backend flashinfer-b12x</code>). This is why the audited-commit Spark recipe is <strong>either</strong> &#8220;force Marlin,&#8221; <strong>&#8220;explicitly opt into <code>b12x</code> and validate,&#8221;</strong> or, as my production fleet later confirmed, get onto the <code>FLASHINFER_CUTLASS</code> native path with a genuine W4A4 checkpoint. Re-check this against the exact release you deploy: it is expected to change once the upstream guard lands.</p><blockquote><p><strong>Update <code>[verified 2026-08-10]</code>:</strong> the exclusion still holds at v0.26.0 and <code>main</code>, but two details in the quoted source comments have since moved &#8212; treat the comments above as a 2026-06-02 snapshot:</p><ul><li><p><strong>The env-var opt-in is gone.</strong> <code>VLLM_NVFP4_GEMM_BACKEND</code> was deprecated in v0.23 (<code>deprecated_env(..., "v0.23", "Use --linear-backend.")</code>) and <strong>removed in v0.25</strong>. On any current image, <code>VLLM_NVFP4_GEMM_BACKEND=flashinfer-b12x</code> is a <strong>silent no-op</strong> &#8212; stale recipes still hand it out. The only surviving opt-in is the CLI value: <code>--linear-backend flashinfer_b12x</code> / <code>--moe-backend flashinfer_b12x</code>.</p></li><li><p><strong>The exclusion is global, not <code>sm_121</code>-scoped.</strong> <code>FLASHINFER_B12X</code> is absent from <code>AVAILABLE_BACKENDS</code> on <em>every</em> GPU &#8212; an <code>sm_120</code> RTX PRO 6000 also falls to Marlin under <code>auto</code>. <code>sm_121</code> is the <em>reason</em> cited in the comment, not the <em>scope</em>. The still-open <a href="https://github.com/vllm-project/vllm/pull/47577">PR #47577</a> would re-add auto-selection for <code>sm_120</code> only and deliberately keep <code>sm_121</code> on the explicit opt-in &#8212; <code>auto</code> keeps skipping <code>b12x</code> and falling through to whatever the image + checkpoint actually support (<code>FLASHINFER_CUTLASS</code> on a genuine W4A4 checkpoint with <code>sm_121</code> cubins, Marlin on a W4A16 checkpoint &#8212; see &#167;4.4), <em>not</em> an unconditional Marlin default. So b12x remains <strong>opt-in-only, never auto-selected on a Spark, in every released version</strong> &#8212; and forcing it currently carries a cluster of open bugs (activation memory roughly doubling &#8594; ~55% less KV cache, #49079; an Xid 31 MMU fault under concurrent chunked prefill, #50189).</p></li></ul></blockquote><h3>4.5 <code>flashinfer_cutlass_moe</code> vs <code>flashinfer_b12x_moe</code></h3><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!dvPC!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F297fb457-e577-48fe-9c8d-7d1f47c14d73_1618x458.png 424w, https://substackcdn.com/image/fetch/$s_!dvPC!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F297fb457-e577-48fe-9c8d-7d1f47c14d73_1618x458.png 848w, https://substackcdn.com/image/fetch/$s_!dvPC!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F297fb457-e577-48fe-9c8d-7d1f47c14d73_1618x458.png 1272w, https://substackcdn.com/image/fetch/$s_!dvPC!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F297fb457-e577-48fe-9c8d-7d1f47c14d73_1618x458.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!dvPC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F297fb457-e577-48fe-9c8d-7d1f47c14d73_1618x458.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/297fb457-e577-48fe-9c8d-7d1f47c14d73_1618x458.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;vLLM on the NVIDIA DGX Spark (GB10 / sm_121): A Complete Technical Guide &#8212; table 3&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="vLLM on the NVIDIA DGX Spark (GB10 / sm_121): A Complete Technical Guide &#8212; table 3" title="vLLM on the NVIDIA DGX Spark (GB10 / sm_121): A Complete Technical Guide &#8212; table 3" srcset="https://substackcdn.com/image/fetch/$s_!dvPC!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F297fb457-e577-48fe-9c8d-7d1f47c14d73_1618x458.png 424w, https://substackcdn.com/image/fetch/$s_!dvPC!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F297fb457-e577-48fe-9c8d-7d1f47c14d73_1618x458.png 848w, https://substackcdn.com/image/fetch/$s_!dvPC!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F297fb457-e577-48fe-9c8d-7d1f47c14d73_1618x458.png 1272w, https://substackcdn.com/image/fetch/$s_!dvPC!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F297fb457-e577-48fe-9c8d-7d1f47c14d73_1618x458.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div></figure></div><p>PR #39825 (&#8220;Disable FlashInfer CUTLASS MoE on SM121&#8221;) was the stopgap that kept the <em>older</em> path from misbehaving on Spark before <code>b12x</code> existed.</p><div><hr></div><h2>5. Marlin: the universal fallback</h2><h3>5.1 What Marlin is</h3><p><strong>Marlin</strong> (originating from IST-DASLab, integrated via Neural Magic; <code>csrc/quantization/marlin/marlin.cu</code>) is a <strong>mixed-precision, weight-only quantized GEMM</strong>: 16-bit activations (FP16/BF16) multiplied by low-bit weights. The &#8220;<strong>wna16</strong>&#8221; naming = <em>weight n-bit, activation 16-bit</em>. It supports a broad matrix of weight formats (<code>csrc/quantization/marlin/generate_kernels.py:29-100</code>): AWQ-INT4, GPTQ-INT4/INT8, FP8 (<code>kFE4M3fn</code>), <strong>NVFP4</strong> (<code>kFE2M1f</code> weights, <code>kFE4M3fn</code> scales), MXFP4, and MXFP8.</p><h3>5.2 Why Marlin is the robust fallback on Spark</h3><p>Two structural reasons:</p><p><strong>1. It is template-generated, not arch-pinned.</strong> Marlin emits kernels from Jinja2 templates parameterized by thread/tile/group config, so the <em>same</em> source covers <code>sm_80</code>, <code>sm_89</code>, <code>sm_120</code>, <code>sm_121</code> without bespoke per-arch kernel files the way CUTLASS does. Its CMake arch lists explicitly include the SM12x family:</p><blockquote><p><code>CMakeLists.txt</code>: <code>MARLIN_FP8_ARCHS</code> = <code>"8.9;12.0a;12.1a"</code> (CUDA&lt;13) / <code>"8.9;12.0f"</code> (CUDA&#8805;13); <code>MARLIN_MOE_FP8_ARCHS</code> = <code>"8.9;12.0;12.1"</code>.</p></blockquote><p><strong>2. It covers </strong><em><strong>both</strong></em><strong> dense linears and MoE.</strong> Dense: <code>csrc/quantization/marlin/</code>. MoE: <code>csrc/moe/marlin_moe_wna16/</code>. So forcing Marlin fixes the <em>whole</em> model in one move, which is exactly why it became the Spark workaround when CUTLASS/FlashInfer FP4 paths were broken for <em>both</em> the dense and MoE halves of a model.</p><h3>5.3 Marlin for NVFP4</h3><p>Marlin has a real FP4 path. The NVFP4 linear kernel is <code>MarlinNvFp4LinearKernel</code> (&#8220;NVFP4 weight-only GEMM via Marlin (W4A16)&#8221;, <code>vllm/model_executor/kernels/linear/nvfp4/marlin.py:18</code>). It performs a non-trivial scale repacking, converting the FP8-E4M3 block scales into Marlin&#8217;s internal &#8220;S0E5M3&#8221; dequant layout (<code>vllm/.../utils/marlin_utils_fp4.py:53-114</code>). You route to it explicitly with <code>VLLM_NVFP4_GEMM_BACKEND=marlin</code> (env, <strong>deprecated</strong> per the blog) or, preferably, the CLI <code>--linear-backend marlin</code> / <code>--moe-backend marlin</code>.</p><h3>5.4 Where Marlin sits in selection, and #40923</h3><p>In the NVFP4 MoE oracle (&#167;7) Marlin sits near the <strong>end</strong> of the auto list, a deliberate &#8220;always works, decent speed&#8221; backstop after the CUTLASS/FlashInfer attempts. PR #40923 (&#8220;Marlin MoE: include SM 12.x in default arch list&#8221;) added the SM12x family to the <em>default</em> Marlin-MoE build arch list so stock builds carry Spark Marlin-MoE kernels out of the box (currently only in <code>v0.22.1rc0</code>).</p><p><strong>Cost of Marlin:</strong> as a weight-only W*A16 kernel it dequantizes weights to 16-bit and runs the matmul in 16-bit, so it does not exploit Spark&#8217;s native FP4/FP8 tensor cores as fully as a true block-scaled CUTLASS/<code>b12x</code> kernel can. On a bandwidth-bound box like Spark this is often acceptable, but it&#8217;s the reason Marlin is the <em>fallback</em>, not the <em>goal</em>.</p><div><hr></div><h2>6. NVFP4 / MXFP4 / FP8 quantization on Spark</h2><h3>6.1 NVFP4 format</h3><p>NVFP4 is a <strong>4-bit float with two-level block scaling</strong>:</p><ul><li><p><strong>Elements:</strong> <code>e2m1</code> (2-bit exponent, 1-bit mantissa; max 6.0), packed 2-per-byte.</p></li><li><p><strong>Per-block scale:</strong> <code>FP8 e4m3</code>, one per <strong>16</strong> contiguous elements along K.</p></li><li><p><strong>Global scale:</strong> an <code>fp32</code> scalar (one for weights, one for activations); the kernel combines them as <code>alpha = 1 / (input_global_scale * weight_global_scale)</code>.</p></li></ul><p>So the effective value is <code>element &#215; fp8_block_scale &#215; fp32_global_scale</code>. (See <code>vllm/.../quantization/modelopt.py</code> around the NVFP4 method, and <code>csrc/.../fp4/nvfp4_utils.cuh</code>.) This block structure is why NVFP4 needs <em>block-scaled</em> tensor-core MMA (<code>OpClassBlockScaledTensorOp</code>) to run fast, which only <code>sm_100</code>/<code>sm_120</code>/<code>sm_121</code> provide, in arch-specific form.</p><h3>6.2 The dense-vs-MoE split: the gotcha that wasted everyone&#8217;s time</h3><p>NVFP4 weights appear in <strong>two independent code paths</strong>:</p><ul><li><p><strong>Dense linear layers</strong> &#8594; <code>ModelOptNvFp4LinearMethod</code>, served by the <em>linear</em> kernel oracle (<code>cutlass_scaled_fp4_mm</code>, FlashInfer, or Marlin).</p></li><li><p><strong>MoE expert weights</strong> &#8594; the <em>fused-MoE</em> oracle (&#167;7).</p></li></ul><p>These are selected <strong>independently</strong>. The historical Spark trap: the model-card env vars <code>VLLM_USE_FLASHINFER_MOE_FP4=1</code> / <code>VLLM_FLASHINFER_MOE_BACKEND=&#8230;</code> only reroute the <strong>MoE</strong> half. The <strong>dense</strong> NVFP4 linears keep going through the broken CUTLASS FP4 path and crash or emit garbage. <strong>The only single-switch fix that covers both halves is Marlin</strong> (<code>--linear-backend marlin --moe-backend marlin</code>), or, on current images, explicitly opting <em>both</em> into <code>b12x</code>.</p><h3>6.3 The kernels that consume NVFP4, and the runtime SM dispatch</h3><p>The CUTLASS FP4 entry point dispatches on the <em>runtime</em> SM and fails loudly if the matching cubin wasn&#8217;t compiled in:</p><blockquote><p><code>csrc/.../fp4/nvfp4_scaled_mm_entry.cu</code> (&#8776; 48-68):</p><pre><code>const int32_t sm = get_sm_version_num();
#if defined(ENABLE_NVFP4_SM100) ...
  if (sm &gt;= 100 &amp;&amp; sm &lt; 120) { cutlass_scaled_fp4_mm_sm100a(...); return; }
#endif
#if defined(ENABLE_NVFP4_SM120) ...
  if (sm &gt;= 120 &amp;&amp; sm &lt; 130) { cutlass_scaled_fp4_mm_sm120a(...); return; }   // SM120/121
#endif
  STD_TORCH_CHECK_NOT_IMPLEMENTED(false, "No compiled nvfp4 mm kernel for SM ", sm, ...);</code></pre></blockquote><p>This device guard + runtime dispatch was added/hardened in PR #29711. The lesson: a wheel built without <code>ENABLE_NVFP4_SM120</code> / the <code>12.1a</code> arch simply has no Spark FP4 kernel, and you hit the <code>NOT_IMPLEMENTED</code> at first FP4 matmul.</p><h3>6.4 NVFP4 vs MXFP4 (and who uses which)</h3><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!rQVw!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58c92479-18b8-4f2c-9ee0-c4e02bcae9bd_1126x380.png 424w, https://substackcdn.com/image/fetch/$s_!rQVw!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58c92479-18b8-4f2c-9ee0-c4e02bcae9bd_1126x380.png 848w, https://substackcdn.com/image/fetch/$s_!rQVw!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58c92479-18b8-4f2c-9ee0-c4e02bcae9bd_1126x380.png 1272w, https://substackcdn.com/image/fetch/$s_!rQVw!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58c92479-18b8-4f2c-9ee0-c4e02bcae9bd_1126x380.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!rQVw!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58c92479-18b8-4f2c-9ee0-c4e02bcae9bd_1126x380.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/58c92479-18b8-4f2c-9ee0-c4e02bcae9bd_1126x380.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;vLLM on the NVIDIA DGX Spark (GB10 / sm_121): A Complete Technical Guide &#8212; table 4&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="vLLM on the NVIDIA DGX Spark (GB10 / sm_121): A Complete Technical Guide &#8212; table 4" title="vLLM on the NVIDIA DGX Spark (GB10 / sm_121): A Complete Technical Guide &#8212; table 4" srcset="https://substackcdn.com/image/fetch/$s_!rQVw!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58c92479-18b8-4f2c-9ee0-c4e02bcae9bd_1126x380.png 424w, https://substackcdn.com/image/fetch/$s_!rQVw!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58c92479-18b8-4f2c-9ee0-c4e02bcae9bd_1126x380.png 848w, https://substackcdn.com/image/fetch/$s_!rQVw!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58c92479-18b8-4f2c-9ee0-c4e02bcae9bd_1126x380.png 1272w, https://substackcdn.com/image/fetch/$s_!rQVw!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F58c92479-18b8-4f2c-9ee0-c4e02bcae9bd_1126x380.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div></figure></div><p>MXFP4 (<code>vllm/.../quantization/mxfp4.py</code>) is what the GPT-OSS family ships in. The block-scaled FP4/MXFP4 CUTLASS kernels for <code>sm_120</code>/<code>sm_121</code> are partly provided through <strong>QuTLASS</strong> (<code>cmake/external_projects/qutlass.cmake</code>, fetched from IST-DASLab; requires CUDA 12.8+ and CC <code>10.0f</code>/<code>12.0f</code>).</p><h3>6.5 FP8: the path of least resistance on Spark</h3><p>Because <code>sm_121</code> has <strong>native <code>e4m3</code> FP8 MMA</strong> (&#167;1.4) and FP8 is covered by both CUTLASS (<code>scaled_mm_sm120_fp8_dispatch</code>) and Marlin, <strong>FP8 is usually far less fragile than NVFP4 on Spark</strong>; on the audited paths it needed no Marlin dance. Treat that as image- and model-dependent rather than a law (Spark FP8 failures have surfaced in specific nightlies and models); smoke-test the exact model and container before relying on <code>auto</code>. On a 128 GB unified box the memory saving of FP4 over FP8 is often irrelevant for the models people actually run, so a practical rule is: <strong>prefer FP8 (or BF16) unless a model is only published in NVFP4/MXFP4.</strong></p><div><hr></div><h2>7. How vLLM does MoE, and how it lands on Spark</h2><h3>7.1 The modular fused-MoE architecture</h3><p>vLLM decomposes a fused-MoE layer into composable stages (<code>vllm/model_executor/layers/fused_moe/modular_kernel.py:45-79</code>):</p><pre><code>[Router] &#8594; [Quantize-Dispatch] &#8594; [Permute-Experts-Unpermute] &#8594; [Combine]</code></pre><p>implemented via three classes: <code>FusedMoEPrepareAndFinalizeModular</code> (prep + finalize), <code>FusedMoEExpertsModular</code> (the expert GEMMs + activation + optional quant), and <code>FusedMoEModularKernel</code> (the glue). Each concrete backend is an &#8220;experts&#8221; implementation under <code>vllm/model_executor/layers/fused_moe/experts/</code> (e.g. <code>flashinfer_b12x_moe.py</code>, <code>marlin_moe.py</code>, <code>triton_moe.py</code>, <code>flashinfer_cutlass_moe.py</code>).</p><h3>7.2 The oracle: how a backend is chosen <code>[verified 2026-06-02]</code></h3><p>For NVFP4 MoE, <code>select_nvfp4_moe_backend()</code> (<code>oracle/nvfp4.py:160-333</code>) walks a priority list and asks each backend whether it supports the current config/device:</p><pre><code>AVAILABLE_BACKENDS = [
    NvFp4MoeBackend.FLASHINFER_TRTLLM,
    NvFp4MoeBackend.FLASHINFER_CUTEDSL,
    NvFp4MoeBackend.FLASHINFER_CUTEDSL_BATCHED,
    NvFp4MoeBackend.FLASHINFER_CUTLASS,
    NvFp4MoeBackend.VLLM_CUTLASS,
    NvFp4MoeBackend.MARLIN,
    NvFp4MoeBackend.EMULATION,
]
# FLASHINFER_B12X intentionally excluded from auto-selection (opt in explicitly)</code></pre><p>The <code>--moe-backend</code> CLI value maps onto this enum in <code>map_nvfp4_backend()</code> (<code>oracle/nvfp4.py:141-157</code>), and the available options + their docstrings (including <em>&#8220;flashinfer_b12x: &#8230; for SM12x (RTX Pro 6000 / DGX Spark)&#8221;</em>) live in <code>vllm/config/kernel.py:171-190</code>.</p><h3>7.3 What this means on Spark</h3><ul><li><p><code>--moe-backend auto</code> on <code>sm_121</code> will <strong>skip <code>b12x</code></strong> (opt-in only, at the audited commit, &#167;4.4) and try TRT-LLM / CuteDSL / CUTLASS paths, then <code>VLLM_CUTLASS</code>, then <strong>Marlin</strong>, then emulation. Whether the earlier entries actually run on <code>sm_121</code> depends on the image&#8217;s cubins; this is why <code>auto</code> behavior on Spark has historically been a coin-flip and why people pinned Marlin.</p></li><li><p>For a <strong>predictable</strong> Spark deployment you have two solid choices:</p><ol><li><p><strong>Force Marlin</strong> for both linear and MoE: guaranteed-correct, slightly slower.</p></li><li><p><strong>Explicitly opt into <code>b12x</code></strong> (<code>--moe-backend flashinfer_b12x</code>, <code>--linear-backend flashinfer-b12x</code>) on a <code>v0.22.0+</code> image (the fast native path) <em>and validate output coherence</em> before trusting it (the <code>b12x</code> exclusion-from-auto exists precisely because the upstream guard isn&#8217;t fully settled).</p></li></ol></li></ul><div><hr></div><h2>8. CUDA graphs and <code>--enforce-eager</code></h2><h3>8.1 What CUDA graphs do, and what <code>enforce_eager</code> turns off</h3><p>vLLM captures the decode forward pass into a <strong>CUDA graph</strong> so the GPU replays a recorded sequence of kernel launches with near-zero per-launch CPU overhead. <code>enforce_eager</code> disables this entirely:</p><blockquote><p><code>vllm/config/model.py</code>: <code>enforce_eager: bool = False</code>: <em>&#8220;If True, we will disable CUDA graph and always execute the model in eager mode. If False, we will use CUDA graph and eager execution in hybrid for maximal performance and flexibility.&#8221;</em></p></blockquote><p>When set, vLLM forces <code>cudagraph_mode = NONE</code> and clears the capture-size list.</p><h3>8.2 Piecewise vs. full capture (and the V1 default)</h3><p><code>vllm/config/compilation.py:599-634</code> defines the modes:</p><ul><li><p><code>NONE</code>: no capture.</p></li><li><p><code>PIECEWISE</code>: capture everything <em>except</em> graph-incompatible ops (e.g. some attention), which run outside the graph for flexibility.</p></li><li><p><code>FULL</code> / <code>FULL_DECODE_ONLY</code>.</p></li><li><p><strong><code>FULL_AND_PIECEWISE</code></strong>: full graphs for decode batches, piecewise for prefill/mixed. <strong>This is the V1 default</strong> and the most performant for most models.</p></li></ul><p>Graphs are captured at a discrete set of batch sizes (<code>cudagraph_capture_sizes</code>), auto-generated roughly as <code>[1,2,4] + range(8,256,8) + range(256, max, 16)</code>, with <code>max_cudagraph_capture_size</code> defaulting to <code>min(max_num_seqs*2, 512)</code>. This is the mechanism behind the field trick of capping <code>--max-num-seqs</code> to bound the capture set.</p><h3>8.3 Why Mamba / hybrid (<code>nemotron_h</code>) models need eager on Spark</h3><p>First, the scope: the official blog&#8217;s benchmark path kept CUDA graphs <strong>enabled</strong>; this section covers a separate field failure mode, not a blanket Spark rule. Hybrid SSM/Mamba models maintain <strong>recurrent state</strong> and use <strong>Triton</strong> kernels with persistent in-place state updates. Graph capture freezes kernel-launch semantics and pointer bindings, which collides with that dynamic state; hence Mamba models implement special <code>copy_inputs_before_cuda_graphs</code> / <code>get_seqlen_agnostic_capture_inputs</code> hooks (<code>vllm/model_executor/models/mamba.py:270-274</code>, <code>mamba2.py</code>, <code>jamba.py</code>, <code>nemotron_h.py</code>). On <code>sm_121</code> specifically, graph capture of the Mamba-2 Triton path has surfaced an <em>&#8220;illegal instruction&#8221;</em> during capture in my deployments, so the operational fix has been <code>--enforce-eager</code>, applied only when a specific model/image combination shows graph-capture instability, and documented with the tested model, image, CUDA version, and error.</p><h3>8.4 Why eager is a crutch, not a destination</h3><p><code>enforce_eager</code> pays for itself in launch overhead on every decode step; field measurements on the Nemotron hybrid models put the cost around <strong>~37%</strong> throughput on Spark, and the broader range is roughly 5&#8211;30% depending on batch size and model depth. It&#8217;s an <strong>escape hatch</strong> for a kernel/capture incompatibility, not a tuning choice. The right long-term fix is upstream: make the Mamba-2 Triton kernels capture-safe on <code>sm_121</code> (or provide a CUDA-graph-compatible SSM path), at which point Spark hybrid models can drop <code>--enforce-eager</code> and recover that throughput. Until then it remains model-specific: apply it to <code>nemotron_h</code>/Mamba hybrids, <strong>not</strong> to dense or standard-MoE models that capture fine.</p><div><hr></div><h2>9. The history: how Spark support was built</h2><p>Support arrived in two waves: RTX-class <code>sm_120</code> first, then Spark <code>sm_121</code> threaded in behind it. Release attributions below are verified with <code>git tag --contains</code> <code>[verified 2026-06-02]</code>.</p><p><strong>Wave 1, <code>sm_120</code> (RTX Blackwell) groundwork, mid-2025:</strong></p><ul><li><p>#17280 CUTLASS W8A8 FP8 for <code>sm_120</code>; #21309 NVFP4 W4A4 for <code>sm_120</code>; #22131 block FP8; #29242 NVFP4 MoE CUTLASS; #29711 device guard + runtime SM dispatch for <code>cutlass_scaled_fp4_mm</code>.</p></li></ul><p><strong>Wave 2, <code>sm_121</code> (DGX Spark) era, 2026:</strong></p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!SVw0!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffda3f8bb-7759-4a41-a48e-eb0c528d83ee_1938x690.png 424w, https://substackcdn.com/image/fetch/$s_!SVw0!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffda3f8bb-7759-4a41-a48e-eb0c528d83ee_1938x690.png 848w, https://substackcdn.com/image/fetch/$s_!SVw0!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffda3f8bb-7759-4a41-a48e-eb0c528d83ee_1938x690.png 1272w, https://substackcdn.com/image/fetch/$s_!SVw0!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffda3f8bb-7759-4a41-a48e-eb0c528d83ee_1938x690.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!SVw0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffda3f8bb-7759-4a41-a48e-eb0c528d83ee_1938x690.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/fda3f8bb-7759-4a41-a48e-eb0c528d83ee_1938x690.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;vLLM on the NVIDIA DGX Spark (GB10 / sm_121): A Complete Technical Guide &#8212; table 5&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="vLLM on the NVIDIA DGX Spark (GB10 / sm_121): A Complete Technical Guide &#8212; table 5" title="vLLM on the NVIDIA DGX Spark (GB10 / sm_121): A Complete Technical Guide &#8212; table 5" srcset="https://substackcdn.com/image/fetch/$s_!SVw0!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffda3f8bb-7759-4a41-a48e-eb0c528d83ee_1938x690.png 424w, https://substackcdn.com/image/fetch/$s_!SVw0!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffda3f8bb-7759-4a41-a48e-eb0c528d83ee_1938x690.png 848w, https://substackcdn.com/image/fetch/$s_!SVw0!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffda3f8bb-7759-4a41-a48e-eb0c528d83ee_1938x690.png 1272w, https://substackcdn.com/image/fetch/$s_!SVw0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffda3f8bb-7759-4a41-a48e-eb0c528d83ee_1938x690.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div></figure></div><p><strong>The key takeaway:</strong> the cluster of fixes that make Spark coherent end-to-end, the SM121 FP8 exclusion fix (#35568) and the <code>b12x</code> integration (#40082), <strong>first ship together in <code>v0.22.0</code> (2026-05-29).</strong> Anything older (including the popular <code>v0.20.0</code>) lacks <code>b12x</code> and the exclusion fix; it can be made to work only by <strong>forcing Marlin</strong>. Release cadence is ~2&#8211;3 weeks.</p><div><hr></div><h2>10. Prebuilt containers vs. rolling your own</h2><h3>10.1 Decision summary <code>[verified 2026-06-02: image tags &amp; version floors drift; re-check each release]</code></h3><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Ib2f!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F940f6d61-99db-4de7-9d42-331f2dce69bb_7050x380.png 424w, https://substackcdn.com/image/fetch/$s_!Ib2f!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F940f6d61-99db-4de7-9d42-331f2dce69bb_7050x380.png 848w, https://substackcdn.com/image/fetch/$s_!Ib2f!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F940f6d61-99db-4de7-9d42-331f2dce69bb_7050x380.png 1272w, https://substackcdn.com/image/fetch/$s_!Ib2f!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F940f6d61-99db-4de7-9d42-331f2dce69bb_7050x380.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!Ib2f!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F940f6d61-99db-4de7-9d42-331f2dce69bb_7050x380.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/940f6d61-99db-4de7-9d42-331f2dce69bb_7050x380.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;vLLM on the NVIDIA DGX Spark (GB10 / sm_121): A Complete Technical Guide &#8212; table 6&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="vLLM on the NVIDIA DGX Spark (GB10 / sm_121): A Complete Technical Guide &#8212; table 6" title="vLLM on the NVIDIA DGX Spark (GB10 / sm_121): A Complete Technical Guide &#8212; table 6" srcset="https://substackcdn.com/image/fetch/$s_!Ib2f!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F940f6d61-99db-4de7-9d42-331f2dce69bb_7050x380.png 424w, https://substackcdn.com/image/fetch/$s_!Ib2f!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F940f6d61-99db-4de7-9d42-331f2dce69bb_7050x380.png 848w, https://substackcdn.com/image/fetch/$s_!Ib2f!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F940f6d61-99db-4de7-9d42-331f2dce69bb_7050x380.png 1272w, https://substackcdn.com/image/fetch/$s_!Ib2f!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F940f6d61-99db-4de7-9d42-331f2dce69bb_7050x380.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div></figure></div><p><strong>You almost never need a from-scratch native compile on the Spark itself</strong> unless you&#8217;re editing kernels; the published aarch64 images/wheels exist precisely so you don&#8217;t. From-source is the path for kernel work or when a needed fix is on <code>main</code> but not yet released.</p><h3>10.2 If you build your own: the arch traps</h3><ul><li><p><strong>List the right arch.</strong> CUDA 12.8/12.9 &#8594; <code>12.1a</code> (or <code>12.0a;12.1a</code>). CUDA 13.0+ &#8594; <code>12.0f</code> (family target covers Spark). Below 12.8, <code>sm_121</code> cannot be built at all (<code>CMakeLists.txt:99-110</code>).</p></li><li><p><strong>Do not copy the in-repo Grace-Blackwell Docker example&#8217;s</strong> <code>torch_cuda_arch_list='9.0 10.0+PTX'</code>. That targets GH200/GB200, <strong>not</strong> GB10. Override it with <code>12.1a</code>/<code>12.0f</code>.</p></li><li><p><strong>Match the dependency pins to the revision you build.</strong> Current <code>main</code> (<code>requirements/cuda.txt</code>) pins <code>flashinfer-python==0.6.11.post2</code>, <code>flashinfer-cubin==0.6.11.post2</code>, and <code>nvidia-cutlass-dsl[cu13]==4.5.2</code> <code>[verified 2026-06-02: these pins move; read the pin in the exact tag you build]</code>. (Older field notes citing cutlass-dsl 4.4.2 are stale.) <strong>Update <code>[verified 2026-08-10]</code>:</strong> this pin has since diverged <em>by build path</em>. The upstream <code>main</code> pin is not the whole story &#8212; Luke Alonso&#8217;s <code>b12x</code> kernel library (the one eugr&#8217;s experimental fork image builds against) now requires <code>nvidia-cutlass-dsl==4.6.0</code>, while <code>4.5.0</code> is known to emit PTX that <code>ptxas</code> rejects on <code>sm_121</code>/<code>sm_120</code> (<a href="https://github.com/NVIDIA/cutlass/issues/3227">NVIDIA/cutlass#3227</a>, fixed in 4.5.1/4.5.2). So &#8220;4.4.2 is stale&#8221; is now itself stale: read the cutlass-dsl pin in the exact tag <strong>and</strong> build path you deploy.</p></li><li><p><strong>Include <code>nvcc</code> if you rely on FlashInfer JIT</strong>; otherwise FlashInfer disables itself when prebuilt cubins are missing for your arch (&#167;4.2).</p></li></ul><h3>10.3 The non-negotiable: validate on the image you deploy</h3><p>Because Spark correctness hinges on whether the <em>specific image</em> carries <code>sm_121</code> cubins and which backend <code>auto</code> lands on, <strong>always smoke-test before trusting a config</strong>, watching for <em>both</em> the hard error and silent garbage:</p><ol><li><p>Serve one NVFP4 <strong>MoE</strong> model and one NVFP4-heavy <strong>dense</strong> model on the candidate image.</p></li><li><p>Watch logs for <code>Failed to run cutlass FP4 gemm on sm120/sm121 &#8230; Error Internal</code> or <code>No compiled nvfp4 mm kernel for SM 121</code>.</p></li><li><p>Inspect actual generated text for coherence (the silent-garbage failure mode produces no error).</p></li><li><p>Try, in order: (a) <code>auto</code>, (b) explicit <code>--moe-backend flashinfer_b12x --linear-backend flashinfer-b12x</code>, (c) forced Marlin. Adopt the fastest that is <em>coherent and stable</em>.</p></li><li><p>Fire a warm-up request (<code>max_tokens=3</code>, same model + chat template) at startup: the first request JIT-compiles Inductor/FlashInfer kernels (~25 s) and you don&#8217;t want a user to eat it.</p></li></ol><h3>10.4 Backend selection cheat-sheet for Spark</h3><ul><li><p><strong>FP8 / BF16 model:</strong> <code>auto</code> is usually fine; FP8 uses native <code>sm_121</code> tensor cores, so no Marlin dance in the common case. Treat it as image- and model-dependent, though: the v0.26.0 cu129 stable actually died on a CUTLASS FP8 W8A8 GEMM for my checkpoint (see the update timeline), so smoke-test the exact model and container before trusting <code>auto</code>.</p></li><li><p><strong>NVFP4 model, want guaranteed-correct:</strong> <code>--moe-backend marlin --linear-backend marlin</code>.</p></li><li><p><strong>NVFP4 model, want fastest native path:</strong> <code>--moe-backend flashinfer_b12x --linear-backend flashinfer-b12x</code> on <code>v0.22.0+</code>, then validate coherence.</p></li><li><p><strong>Hybrid Mamba (<code>nemotron_h</code>):</strong> add <code>--enforce-eager</code> <strong>if</strong> your model/image combination shows the &#167;8.3 graph-capture failure; it is a documented per-model workaround, not a Spark default.</p></li><li><p>Prefer <strong>CLI flags</strong> over the <code>VLLM_*</code> env vars: <code>VLLM_NVFP4_GEMM_BACKEND</code> was deprecated in v0.23 and <strong>removed in v0.25</strong> (a no-op on current images), and <code>VLLM_USE_FLASHINFER_MOE_FP4</code> is deprecated. Use <code>--moe-backend</code> / <code>--linear-backend</code>.</p></li></ul><div><hr></div><h2>11. Appendix: glossary and file index</h2><h3>Glossary</h3><ul><li><p><strong>GB10 / Grace Blackwell</strong>: the DGX Spark superchip (Grace ARM CPU + Blackwell GPU, unified memory).</p></li><li><p><strong><code>sm_121</code> / CC 12.1</strong>: DGX Spark&#8217;s compute capability; workstation-Blackwell SM12x family, distinct from datacenter <code>sm_100</code>.</p></li><li><p><strong><code>a</code> vs <code>f</code> arch suffix</strong>: <code>12.1a</code> = arch-specific cubin (one SM); <code>12.0f</code> = family cubin (whole SM12x, needs CUDA 13).</p></li><li><p><strong>CUTLASS</strong>: NVIDIA C++ template GEMM library; vLLM&#8217;s first-choice quantized matmul; needs arch-specific kernels.</p></li><li><p><strong>FlashInfer</strong>: inference kernel library (attention, MoE, FP4/FP8 GEMM); ships prebuilt cubins or JITs at runtime.</p></li><li><p><strong>CuteDSL</strong>: CUTLASS&#8217;s tensor-layout DSL; basis of the <code>b12x</code> SM12x kernels.</p></li><li><p><strong><code>b12x</code></strong>: FlashInfer&#8217;s SM12x block-scaled fused-MoE / FP4-GEMM path; fast on Spark but <strong>opt-in</strong> (not auto, at the audited commit, &#167;4.4).</p></li><li><p><strong>Marlin / wna16</strong>: template-generated weight-only (W4/W8 &#183;A16) GEMM; broad arch coverage; Spark&#8217;s robust fallback for both dense and MoE.</p></li><li><p><strong>QuTLASS</strong>: IST-DASLab CUTLASS extension providing FP4/MXFP4 kernels for <code>sm_100</code>/<code>sm_120</code>.</p></li><li><p><strong>NVFP4</strong>: 4-bit float (<code>e2m1</code>), 16-element blocks, FP8-<code>e4m3</code> block scales + FP32 global scale.</p></li><li><p><strong>MXFP4</strong>: 4-bit float, 32-element blocks, <code>e8m0</code> block scales; GPT-OSS format.</p></li><li><p><strong>enforce-eager</strong>: disables CUDA-graph capture; needed for Mamba/hybrid on Spark; ~5&#8211;37% throughput cost.</p></li></ul><h3>File index (paths under the vLLM repo root)</h3><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!_6Kf!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2dc947d1-928c-4299-acce-f3956e9dec5d_2440x1856.png 424w, https://substackcdn.com/image/fetch/$s_!_6Kf!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2dc947d1-928c-4299-acce-f3956e9dec5d_2440x1856.png 848w, https://substackcdn.com/image/fetch/$s_!_6Kf!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2dc947d1-928c-4299-acce-f3956e9dec5d_2440x1856.png 1272w, https://substackcdn.com/image/fetch/$s_!_6Kf!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2dc947d1-928c-4299-acce-f3956e9dec5d_2440x1856.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!_6Kf!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2dc947d1-928c-4299-acce-f3956e9dec5d_2440x1856.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/2dc947d1-928c-4299-acce-f3956e9dec5d_2440x1856.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;vLLM on the NVIDIA DGX Spark (GB10 / sm_121): A Complete Technical Guide &#8212; table 7&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="vLLM on the NVIDIA DGX Spark (GB10 / sm_121): A Complete Technical Guide &#8212; table 7" title="vLLM on the NVIDIA DGX Spark (GB10 / sm_121): A Complete Technical Guide &#8212; table 7" srcset="https://substackcdn.com/image/fetch/$s_!_6Kf!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2dc947d1-928c-4299-acce-f3956e9dec5d_2440x1856.png 424w, https://substackcdn.com/image/fetch/$s_!_6Kf!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2dc947d1-928c-4299-acce-f3956e9dec5d_2440x1856.png 848w, https://substackcdn.com/image/fetch/$s_!_6Kf!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2dc947d1-928c-4299-acce-f3956e9dec5d_2440x1856.png 1272w, https://substackcdn.com/image/fetch/$s_!_6Kf!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2dc947d1-928c-4299-acce-f3956e9dec5d_2440x1856.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div></figure></div><div><hr></div><p><em>Compiled from the vLLM <code>main</code> source tree (2026-06-02) and the official vLLM DGX Spark blog (2026-06-01). Backend selection details, especially the <code>b12x</code> opt-in, reflect current <code>main</code>; re-check the oracle comments in <code>oracle/nvfp4.py</code> against the exact image you deploy, since the upstream CUTLASS sm_121 MMA-guard fix may flip <code>b12x</code> into auto-selection in a future release.</em></p>]]></content:encoded></item><item><title><![CDATA[vLLM on the DGX Spark, the Hard Way: Seven Months of GB10 Field Notes]]></title><description><![CDATA[Every performance number in this post was measured on my own two DGX Sparks and is labeled with the most precise identifiers I have for that run: image digest or pinned gitsha, checkpoint, and date.]]></description><link>https://blog.swesty.net/p/vllm-on-dgx-spark-gb10-field-notes</link><guid isPermaLink="false">https://blog.swesty.net/p/vllm-on-dgx-spark-gb10-field-notes</guid><dc:creator><![CDATA[Shawn Westfall]]></dc:creator><pubDate>Sun, 19 Jul 2026 00:00:00 GMT</pubDate><enclosure url="https://substack-post-media.s3.amazonaws.com/public/images/3248a3b4-a7b1-4e25-a32b-627769f4a56e_1360x304.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<blockquote><p>Every performance number in this post was measured on my own two DGX Sparks and is labeled with the most precise identifiers I have for that run: image digest or pinned gitsha, checkpoint, and date. Where I cite someone else&#8217;s number, I say so. Where I was wrong earlier, the timeline says that too.</p></blockquote><h2>Why this post exists</h2><p>The DGX Spark&#8217;s serving ecosystem moved faster than anyone&#8217;s documentation could; NVIDIA&#8217;s launch guidance and the vLLM team&#8217;s Spark deep-dive were each accurate for the moment they were written, and the platform kept moving underneath them. What that left for operators to discover in production: which image builds carry <code>sm_121</code> kernels and which don&#8217;t, why some NVFP4 checkpoints crash or emit garbage on paths that work fine on datacenter Blackwell, what <code>--gpu-memory-utilization</code> really means on a unified 128 GB pool, and what happens to speculative decoding under <em>concurrent</em> load rather than in a batch-1 demo. I wrote my findings down after each incident. This post is that document, cleaned up, with the receipts, offered as a snapshot with dates attached, since it will age the same way.</p><p>Two boxes inform everything here: <strong>Spark-1</strong> and <strong>Spark-2</strong>, both GB10 (aarch64, <code>sm_121a</code>, 128 GB unified, ~273 GB/s), serving production traffic for a homelab agent fleet (reasoning, tool-calling, and structured extraction), 24/7, behind a model gateway. This is not a benchmark rig; every lesson below came out of something breaking. For the kernel-level <em>why</em> behind these notes (the CUTLASS/FlashInfer/Marlin mechanics, the backend oracle, and the arch details), see the companion <a href="/blog/vllm-dgx-spark-technical-report">technical report</a>.</p><h2>The five-minute model of what vLLM is doing on this box</h2><p>Everything in this post follows from four pieces of vLLM machinery. If you already operate vLLM, skip ahead; if not, these four paragraphs are the vocabulary the rest of the post uses.</p><p><strong>1. Serving has two phases with opposite bottlenecks.</strong> <em>Prefill</em> processes your whole prompt in parallel; it&#8217;s compute-bound, measured in thousands of tokens/second, and determines time-to-first-token. <em>Decode</em> generates one token at a time, and each token requires re-reading the model&#8217;s active weights from memory; it&#8217;s memory-bandwidth-bound, which on a 273 GB/s unified-memory box is the budget everything competes for. This split explains most of the numbers here: why a 35B MoE with only 3B <em>active</em> parameters decodes fast on a Spark (decode reads the active set, not the total), why a weight-only quantization like Marlin costs little at single-user decode but shows up at prefill and concurrency (those are compute-heavy), and why my tables always report the phases separately. Two caveats the rest of the post leans on. First, the <em>short-context limit</em>: decode actually re-reads the active weights <em>and</em> the KV cache of every token attended so far, so at long context the KV term grows and decode throughput falls with sequence length, which is why KV-pool sizing (&#167;9) is a performance concern, not just a capacity one. Second, what <em>Marlin</em> means throughout: it&#8217;s vLLM&#8217;s weight-only 4-bit kernel that keeps weights 4-bit in memory and dequantizes them to BF16/FP16 <em>inside</em> the GEMM, so it wins on bandwidth-bound decode but runs the matmul in high precision, never touching the native FP4 tensor cores. That&#8217;s why its cost surfaces in compute-bound prefill and under concurrency, not at batch-1 decode.</p><p><strong>2. Kernels are chosen by a chain with three links, and any link can break.</strong> When vLLM loads a model it reads the checkpoint&#8217;s quantization config (<code>config.json</code>), runs it through a backend &#8220;oracle&#8221; (<code>select_nvfp4_moe_backend()</code> in <code>vllm/model_executor/layers/fused_moe/oracle/nvfp4.py</code>) that maps <em>quant format + GPU architecture</em> to a ranked list of candidate kernel backends. That ranked list is the oracle&#8217;s <code>AVAILABLE_BACKENDS</code> constant: the <code>['FLASHINFER_TRTLLM', &#8230; 'FLASHINFER_CUTLASS', &#8230; 'MARLIN', 'EMULATION']</code> line you&#8217;ll see in my boot logs. The winning backend then still has to find a <em>compiled kernel binary (cubin) for your GPU inside the wheel you actually installed</em>. Three links: checkpoint declares, oracle selects, wheel delivers. Each fails differently: a W4A16 checkpoint means the oracle <em>correctly</em> selects Marlin (nothing to fix); a wheel without <code>sm_121</code> cubins means the selected backend dies at runtime (<code>no kernel image is available</code>); and a version-gated integration mismatch can silently drop you onto slow fallbacks. Most of this post&#8217;s timeline is one of those three links breaking.</p><p><strong>3. The KV cache is a pre-allocated pool, not a meter.</strong> vLLM (PagedAttention) grabs one big block of memory at startup, sized by <code>--gpu-memory-utilization</code>, and pages sequences into it. It&#8217;s <em>capacity</em>; the boot log tells you the pool size in GiB and in tokens, and dividing them gives the per-token cost that makes context-length planning arithmetic instead of folklore (worked example in the unified-memory section). On a unified-memory box the grab itself is the danger: it can&#8217;t be swapped, so an oversized pool <em>is</em> the OOM.</p><p><strong>4. Speculative decoding buys speed with a coupled scheduler.</strong> A cheap drafter proposes k tokens; the full model verifies them in a single forward pass; accepted tokens cost almost nothing extra. Acceptance depends entirely on how <em>predictable</em> your output is to that drafter (code and structured JSON accept well, free prose doesn&#8217;t) and the metric vLLM reports (&#8220;acceptance length&#8221;) includes a guaranteed bonus token, so read it carefully. The part nobody advertises: the drafter is wired into the scheduler, so turning speculation on can change scheduling behavior itself: disable async scheduling, cap the token budget per step. A &#8220;free&#8221; flag can therefore tax prefill and TTFT even while decode acceptance looks healthy. That coupling is exactly how my n-gram experiment lost (full numbers below).</p><h2>The timeline: from launch-day optimism to a config that survives</h2><p>This is the part most guides skip: <em>when</em> each fact became true. sm_121 support moved so fast that advice written eight weeks apart contradicts itself, including my own. Dates below come from my git history, incident notes, and the vLLM release record.</p><p><strong>2025-12-20: the origin.</strong> NVIDIA&#8217;s launch-window vLLM container documentation (NGC <code>vllm:25.11-py3</code> and its release notes) claimed clean &#8220;DGX Spark functional support&#8221; and NVFP4-on-Blackwell. I wrote my first Spark serving doc from those sources, taking the claims at face value. What wasn&#8217;t yet visible in that material: <code>sm_121</code> is consumer Blackwell, not the <code>sm_100</code> the FP4 kernel work had targeted, and none of the then-shipped FlashInfer wheels contained sm_121 cubins, details the launch-window docs preceded rather than concealed. Everything below is the gap between documentation written at launch and a working box, closing in real time.</p><p><strong>2026-02 &#8594; 2026-05: upstream catches up, in pieces.</strong> The vLLM release record tells the real support story: <code>enable_sm120_family</code> (#33517, Feb); &#8220;Fix DGX Spark logic&#8221; (#38126, &#8594; v0.19.0); disable the broken FlashInfer CUTLASS MoE on sm_121 as a stopgap (#39825, &#8594; v0.20.0); fix the wrong sm_121 <em>exclusion</em> from Marlin/CUTLASS FP8 (#35568) and land the b12x sm_121 kernels (#40082), both only in <strong>v0.22.0 (2026-05-29)</strong>. In my tested pre-v0.22 images and checkpoints, native NVFP4 paths crashed (<code>cutlass FP4 gemm ... sm120/sm121 ... Error Internal</code>), silently emitted garbage, or required Marlin; behavior varies with the image build, packaged wheels, checkpoint, and backend, so treat this as my observed floor, not a universal law.</p><p><strong>2026-05-31: the empirical break.</strong> After weeks of intermittent failures I isolated the pattern on my own boxes: prebuilt FlashInfer wheels shipped <strong>zero</strong> sm_120/121 cubins; the model-card env vars rerouted only MoE layers while dense NVFP4 linears still hit the broken CUTLASS path; and the reliable fix was forcing <strong>Marlin for both backends</strong> (plus <code>--enforce-eager</code> for hybrid-Mamba models, a graph-capture bug fixed later in v0.22.0). Forced-Marlin became my fleet default, correct at the time, and understandably carried forward by community guides (including mine) as if it were a permanent hardware rule.</p><p><strong>2026-06-01/02: reconciliation, then past Marlin.</strong> The vLLM team published its Spark deep-dive claiming <code>auto</code> backends were fine, which contradicted my validated config. Reconciling blog-vs-repo-vs-my-logs showed the disagreement was a <strong>version gap</strong>, not a dispute: my Marlin era ran on v0.20.0, which simply lacked the v0.22.0 kernels. On 2026-06-02 I moved to a digest-pinned cu130 nightly and <code>--moe-backend auto</code> selected FLASHINFER_CUTLASS native FP4, coherent, no Marlin, no enforce-eager, and <em>faster</em> (~50.2 &#8594; ~55.8/58.5 tok/s on my Nemotrons). Meanwhile stock <code>v0.22.0-cu129</code> (same vLLM version, different wheel) still crashed on <code>auto</code> because the cu129 wheel didn&#8217;t bake the sm_121 MoE cubin. That&#8217;s when &#8220;which vLLM version?&#8221; stopped being the right question and &#8220;does <em>this wheel</em> carry the cubins?&#8221; replaced it.</p><p><strong>2026-06-05 &#8594; 2026-07-05: the MTP production saga.</strong> I deployed the 35B Qwen with in-checkpoint MTP: 86.8% acceptance in validation, then <code>CUDA error: an illegal memory access</code> crashes every few hours in production, only under concurrent load (upstream #41190/#42084; the failure signature is a broken draft path poisoning the accepted-token buffer). An image bump to a 0.23-era nightly fixed the <em>load-time</em> bug (#35031 via #46316) but not the runtime crash, so spec-decode went off. Along the way, a second, unrelated checkpoint lesson: the W4A16 build&#8217;s 4-bit <code>lm_head</code> produced <code>!!!!</code> garbage on greedy structured-extraction output; quantization choices inside the checkpoint, not the engine, decided both failures.</p><p><strong>2026-07-11/12: the W4A4 resolution.</strong> Swapping to a genuine W4A4 checkpoint (4-bit activations, <code>lm_head</code> unquantized) moved experts <em>and</em> drafter onto the FLASHINFER_CUTLASS path, and the crash was gone: 8-way soak, 400/400, zero IMA, then re-enabled in production and bumped to <code>num_speculative_tokens: 3</code> (mean acceptance length ~3.1 output tokens per verification step, bonus token included; ~95% draft acceptance on real traffic). The upstream bugs remain open; I didn&#8217;t patch vLLM, I <em>routed around the crashing kernel path via configuration</em>, checkpoint format and drafter backend together, so neither alone gets the credit. The transferable lesson stands regardless: on sm_121, the checkpoint&#8217;s quant format (W4A4 vs W4A16) decides which kernels you run, and therefore which bugs you hit.</p><p><strong>2026-07-19: today.</strong> Both boxes on digest-pinned nightlies (0.22/0.23-era), native FP4, MTP where the model ships heads, everything below measured on this config. I smoke-tested the v0.25.1 stable candidates the same day this post was assembled, and the timeline&#8217;s lesson reproduced on cue: the official cu129 stable still ships no sm_121 cubins (dead at engine init), and the promising new cu13 tag turned out to be a mis-stamped dev artifact decoding 2.6&#215; slow (details in the image table below). Seven months in, the rule hasn&#8217;t changed: <strong>there is still no viable aarch64 stable for GB10 native FP4; validate the wheel you actually pulled, then pin it.</strong></p><p><strong>Update: 2026-08-05 (post-publication).</strong> Three developments the timeline&#8217;s own logic predicted: (1) <strong>v0.26.0</strong> (2026-07-27): I smoke-tested its cu129 aarch64 stable on a scratch GB10 instance, and the <code>sm_121</code> NVFP4 cubins are finally present. <code>auto</code> selects <code>FLASHINFER_CUTLASS</code> for both dense and MoE NVFP4 with no <code>no kernel image is available</code>, breaking the three-release cu129 drought. It still could not serve my unsloth checkpoint, though: EngineCore init dies during the profiling run on a CUTLASS FP8 W8A8 GEMM (<code>cutlass_gemm_caller.cuh:62, Error Internal</code>), a path the same checkpoint clears on my v0.23-era nightly. So the operator bottom line is unchanged (no cu129 stable serves this GB10 NVFP4+FP8 workload yet), but the failure moved from a missing NVFP4 kernel image to an FP8 CUTLASS error, worth re-testing per checkpoint. (Scratch-tested 2026-08-05; not yet root-caused or filed.) (2) <strong>#41190 (the MTP illegal-memory-access) was fixed on main 2026-07-29</strong> (#48245/#49620), though not yet in a tagged stable release and marked a probable (&#8220;may fix&#8221;) rather than confirmed resolution, so stable-wheel operators remain exposed until a nightly carries it. (3) <strong>Prefix caching + MTP now coexist</strong> on a v0.26.1-dev nightly: the either/or that #42084 forced looks resolved via an experimental Mamba/SSM cache-align mode (tested on my GDN-hybrid 35B: zero out-of-bounds under a shared-prefix + MTP soak). All are elaborated in the sections below; noted here to keep the timeline honest about its own aging.</p><h2>The one rule that explains all the FP4 confusion</h2><p>A widely-shared rule (reasonably derived from what its authors tested) says &#8220;GB10 has no native FP4 compute, so FP4 MoE must run on Marlin.&#8221; My production logs show the rule has an important exception:</p><pre><code>Using FlashInferCutlassNvFp4LinearKernel for NVFP4 GEMM
Using 'FLASHINFER_CUTLASS' NvFp4 MoE backend out of potential backends:
  ['FLASHINFER_TRTLLM', 'FLASHINFER_CUTEDSL', 'FLASHINFER_CUTEDSL_BATCHED',
   'FLASHINFER_CUTLASS', 'VLLM_CUTLASS', 'MARLIN', 'EMULATION']</code></pre><p>That&#8217;s a Qwen3.6-35B-A3B NVFP4 MoE on <code>--moe-backend auto</code>, coherent output, running for days under real load, on a GB10. No Marlin anywhere.</p><p><strong>Native FP4 on sm_121 needs two conditions, and both must hold:</strong></p><ol><li><p><strong>The image&#8217;s FlashInfer wheel actually bakes <code>sm_121</code> cubins.</strong> Source support is not wheel support. Stock <code>v0.22.0-cu129</code> has the sm_121 kernels <em>in source</em> (#40082) but the cu129 wheel ships without the MoE cubin; <code>auto</code> dies with <code>no kernel image is available</code>. The cu130 nightlies bake it. This is an image property; smoke-test the exact digest you deploy.</p></li><li><p><strong>The checkpoint is genuine W4A4</strong> (<code>nvfp4-pack-quantized</code>: 4-bit weights <em>and</em> activations). A <strong>W4A16 weight-only</strong> &#8220;NVFP4&#8221; checkpoint has no FP4 activations, so there is no FP4 matmul to dispatch; <code>auto</code> lands on Marlin <em>correctly</em>, and no flag will change it.</p></li></ol><p>Condition 2 is the one nobody talks about, and it explains the contradictory community reports. I ran three quantizations of the <em>same model</em> (Qwen3.6-35B-A3B) on the same image:</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Baj7!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1b38d15f-5a51-4bbf-8cdf-b2c111d3eb42_1360x304.png 424w, https://substackcdn.com/image/fetch/$s_!Baj7!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1b38d15f-5a51-4bbf-8cdf-b2c111d3eb42_1360x304.png 848w, https://substackcdn.com/image/fetch/$s_!Baj7!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1b38d15f-5a51-4bbf-8cdf-b2c111d3eb42_1360x304.png 1272w, https://substackcdn.com/image/fetch/$s_!Baj7!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1b38d15f-5a51-4bbf-8cdf-b2c111d3eb42_1360x304.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!Baj7!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1b38d15f-5a51-4bbf-8cdf-b2c111d3eb42_1360x304.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/1b38d15f-5a51-4bbf-8cdf-b2c111d3eb42_1360x304.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;vLLM on the DGX Spark, the Hard Way: Seven Months of GB10 Field Notes &#8212; table 1&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="vLLM on the DGX Spark, the Hard Way: Seven Months of GB10 Field Notes &#8212; table 1" title="vLLM on the DGX Spark, the Hard Way: Seven Months of GB10 Field Notes &#8212; table 1" srcset="https://substackcdn.com/image/fetch/$s_!Baj7!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1b38d15f-5a51-4bbf-8cdf-b2c111d3eb42_1360x304.png 424w, https://substackcdn.com/image/fetch/$s_!Baj7!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1b38d15f-5a51-4bbf-8cdf-b2c111d3eb42_1360x304.png 848w, https://substackcdn.com/image/fetch/$s_!Baj7!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1b38d15f-5a51-4bbf-8cdf-b2c111d3eb42_1360x304.png 1272w, https://substackcdn.com/image/fetch/$s_!Baj7!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1b38d15f-5a51-4bbf-8cdf-b2c111d3eb42_1360x304.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div></figure></div><p>So when a guide says Marlin is required, it&#8217;s worth checking which checkpoint was tested; most popular recipes use the W4A16 builds, which made the conclusion easy to reach. If it was an <code>nvidia/...</code> modelopt W4A16 build (most of the popular recipes are), their observation was correct <em>for that checkpoint</em>; it just doesn&#8217;t generalize into a hardware rule.</p><p><strong>Decision tree:</strong></p><pre><code>Is the checkpoint W4A4 (nvfp4-pack-quantized, activations num_bits: 4)?
&#9500;&#9472; NO (W4A16 weight-only) &#8594; auto &#8594; Marlin. This is correct; don't fight it.
&#9474;    At batch-1 you barely pay for it: decode is bandwidth-bound and 4-bit
&#9474;    weight storage already captures the win. Marlin costs you prefill + concurrency.
&#9492;&#9472; YES &#8594; does your image's flashinfer wheel bake sm_121 cubins?
     &#9500;&#9472; NO (e.g. stock v0.22.0-cu129) &#8594; force --moe-backend marlin, or change image
     &#9492;&#9472; YES (cu130 nightlies, pinned by gitsha) &#8594; --moe-backend auto
          &#8594; FLASHINFER_CUTLASS native FP4, dense AND MoE. Faster than Marlin,
            full cudagraphs, and (0.23-era) --attention-backend flashinfer
            works even on hybrid attention/Mamba/GDN models.</code></pre><p>One more wrinkle from the release archaeology: the <em>fastest</em> sm_121 path, <code>flashinfer_b12x</code> (#40082), is intentionally excluded from <code>auto</code>-selection pending an upstream CUTLASS MMA-guard fix, so <code>auto</code> gives you a working-and-fast path, not necessarily the theoretical-fastest one. Precision note, since people assume this is MoE-only: it isn&#8217;t. The dense NVFP4 linear oracle (<code>_POSSIBLE_NVFP4_KERNELS</code> in <code>vllm/model_executor/kernels/linear/__init__.py</code>) comments b12x out of its auto candidate list with the same rationale as the MoE oracle, so <code>auto</code> never even tests it on either path. My boot log&#8217;s <code>FlashInferCutlassNvFp4LinearKernel</code> for the dense GEMM is <code>auto</code> walking past the policy-excluded b12x (and a higher-priority CuteDSL kernel my wheel can&#8217;t use) down to CUTLASS. Both dense and MoE land on CUTLASS for the same reason: b12x is opt-in only (<code>--linear-backend flashinfer_b12x</code> / <code>moe-backend flashinfer_b12x</code>), never auto-selected. I haven&#8217;t needed to <em>force</em> <code>b12x</code>; CUTLASS is already ahead of Marlin.</p><h2>Which images actually work (July 2026)</h2><p>The single most common failure mode on Spark is an image whose wheels predate <code>sm_121</code>. Verified state as of <strong>2026-07-19</strong> (Docker Hub API + my deployments):</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!05GS!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe010fec2-13e9-4e91-8b75-64004a5b4a7e_6194x620.png 424w, https://substackcdn.com/image/fetch/$s_!05GS!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe010fec2-13e9-4e91-8b75-64004a5b4a7e_6194x620.png 848w, https://substackcdn.com/image/fetch/$s_!05GS!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe010fec2-13e9-4e91-8b75-64004a5b4a7e_6194x620.png 1272w, https://substackcdn.com/image/fetch/$s_!05GS!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe010fec2-13e9-4e91-8b75-64004a5b4a7e_6194x620.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!05GS!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe010fec2-13e9-4e91-8b75-64004a5b4a7e_6194x620.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/e010fec2-13e9-4e91-8b75-64004a5b4a7e_6194x620.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;vLLM on the DGX Spark, the Hard Way: Seven Months of GB10 Field Notes &#8212; table 2&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="vLLM on the DGX Spark, the Hard Way: Seven Months of GB10 Field Notes &#8212; table 2" title="vLLM on the DGX Spark, the Hard Way: Seven Months of GB10 Field Notes &#8212; table 2" srcset="https://substackcdn.com/image/fetch/$s_!05GS!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe010fec2-13e9-4e91-8b75-64004a5b4a7e_6194x620.png 424w, https://substackcdn.com/image/fetch/$s_!05GS!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe010fec2-13e9-4e91-8b75-64004a5b4a7e_6194x620.png 848w, https://substackcdn.com/image/fetch/$s_!05GS!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe010fec2-13e9-4e91-8b75-64004a5b4a7e_6194x620.png 1272w, https://substackcdn.com/image/fetch/$s_!05GS!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe010fec2-13e9-4e91-8b75-64004a5b4a7e_6194x620.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div></figure></div><p>Rules that survived contact:</p><ul><li><p><strong>Check <code>/version</code> before anything else: a tag&#8217;s name is a claim, not a fact.</strong> The cu13 &#8220;0.25.1&#8221; tag above passed every kernel check and would have shipped a 2.6&#215; decode regression; the one-line <code>curl :PORT/version</code> exposed it as a dev artifact instantly. My best hypothesis for the slowdown: with a bogus version stamp, version-gated integrations (FlashInfer wheel matching, autotune caches, compat shims) can&#8217;t resolve, and the build silently takes fallback kernel paths; the uniform ~2&#215; hit across <em>both</em> prefill and decode fits systemic fallback, not one bad kernel. I didn&#8217;t profile further; a mis-stamped build is disqualified either way.</p></li><li><p><strong>Pin <code>nightly-&lt;gitsha&gt;</code> or a digest, never a moving tag.</strong> I froze my production image as a locally-tagged derivative so upstream tag drift can&#8217;t change what&#8217;s running.</p></li><li><p><strong>Smoke-test the exact image before deploying</strong>: boot the model on a scratch port, hit the raw <code>/v1/completions</code> endpoint (bypasses chat template and reasoning parser), and watch for both the loud failure (<code>no kernel image is available</code>, <code>cutlass FP4 gemm &#8230; Error Internal</code>) and the quiet one (silent garbage tokens).</p></li><li><p><strong>A reasoning model returning empty output is usually not broken.</strong> If <code>finish_reason: length</code> and both <code>content</code> and <code>reasoning_content</code> are empty, the model hit <code>max_tokens</code> inside an unclosed <code>&lt;think&gt;</code> block. Raise the budget or pass <code>chat_template_kwargs: {"enable_thinking": false}</code> before you blame the kernels.</p></li></ul><h2>The MTP saga: speculative decoding that survives real traffic</h2><p>Qwen3.6 ships in-checkpoint MTP heads, and single-stream decode on a bandwidth-bound box is exactly where speculative decoding pays. Getting it to <em>stay up</em> taught me the most expensive lesson in this post:</p><p><strong>Batch-1 validation lies.</strong> My first MTP deployment (W4A16 checkpoint &#8594; Marlin experts, drafter forced to a Triton MoE backend) validated clean at batch-1 with 86.8% acceptance, then died in production with <code>CUDA error: an illegal memory access</code> <strong>only under concurrent load</strong>, taking both the coordinator tool-calling endpoint and the memory-extraction alias down with it. Upstream issues existed (#41190/#42084); an image bump didn&#8217;t fix it; I ran months with spec-decode off.</p><p><strong>The crash was avoided by configuration, not a patch.</strong> Moving to the W4A4 build routed both the experts <em>and</em> the drafter through FLASHINFER_CUTLASS, a different kernel path than the crashing Marlin+Triton combination. The W4A4/CUTLASS configuration eliminated the crash in my concurrency soak, but checkpoint format and drafter backend changed together, so the result cannot be attributed to either change alone. Before re-enabling in production I soaked it at 8-way concurrency: 400/400 requests, <strong>zero</strong> illegal-memory-access, 73% acceptance under load.</p><p><strong>What MTP actually delivers on GB10</strong> (my production numbers, <code>num_speculative_tokens: 3</code>):</p><ul><li><p><strong>Mean acceptance length &#8776;3.1 output tokens per verification step</strong> under 8-way burst: note vLLM&#8217;s metric includes the guaranteed bonus token, so with <code>n=3</code> that&#8217;s ~2.1 accepted <em>draft</em> tokens per step (per-position draft acceptance 84.2% / 67.8% / 57.5%: the falling marginal value is why you keep n at 1&#8211;3, unlike block-verify methods that run 7&#8211;15).</p></li><li><p>~<strong>94.8% cumulative acceptance over ~20 h of real coordinator + extraction traffic</strong>, far above any synthetic number, because real agent workloads are predictable, structured output. Measure acceptance on <em>your</em> traffic via <code>/metrics | grep -i spec_decode</code>, not on a demo prompt.</p></li><li><p>Thinking-mode decode is <em>faster</em> than answer-mode with MTP on: structured reasoning accepts better than free prose.</p></li></ul><h3>Measured: native FP4 + MTP vs the published Marlin numbers</h3><p>I ran the same single-stream methodology the community playbook published (prefix-cache-defeating nonces, forced full-length generation, server-side token counts, 3-run medians, ~6K-token prompt) against my <strong>production</strong> 35B, native FLASHINFER_CUTLASS, W4A4, MTP-3, vLLM 0.23.1rc1-era, measured 2026-07-19:</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!iAQJ!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F193b2c38-5b78-4d99-b143-6ecda7f0d176_2118x304.png 424w, https://substackcdn.com/image/fetch/$s_!iAQJ!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F193b2c38-5b78-4d99-b143-6ecda7f0d176_2118x304.png 848w, https://substackcdn.com/image/fetch/$s_!iAQJ!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F193b2c38-5b78-4d99-b143-6ecda7f0d176_2118x304.png 1272w, https://substackcdn.com/image/fetch/$s_!iAQJ!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F193b2c38-5b78-4d99-b143-6ecda7f0d176_2118x304.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!iAQJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F193b2c38-5b78-4d99-b143-6ecda7f0d176_2118x304.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/193b2c38-5b78-4d99-b143-6ecda7f0d176_2118x304.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;vLLM on the DGX Spark, the Hard Way: Seven Months of GB10 Field Notes &#8212; table 3&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="vLLM on the DGX Spark, the Hard Way: Seven Months of GB10 Field Notes &#8212; table 3" title="vLLM on the DGX Spark, the Hard Way: Seven Months of GB10 Field Notes &#8212; table 3" srcset="https://substackcdn.com/image/fetch/$s_!iAQJ!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F193b2c38-5b78-4d99-b143-6ecda7f0d176_2118x304.png 424w, https://substackcdn.com/image/fetch/$s_!iAQJ!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F193b2c38-5b78-4d99-b143-6ecda7f0d176_2118x304.png 848w, https://substackcdn.com/image/fetch/$s_!iAQJ!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F193b2c38-5b78-4d99-b143-6ecda7f0d176_2118x304.png 1272w, https://substackcdn.com/image/fetch/$s_!iAQJ!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F193b2c38-5b78-4d99-b143-6ecda7f0d176_2118x304.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div></figure></div><p>The honest read: <strong>these measurements show no batch-1 native-FP4 decode advantage, and they don&#8217;t isolate the backend, either.</strong> Checkpoint format, machine load, measurement date, and tooling all differ across the rows, so this is convergent evidence, not a controlled experiment. It is <em>consistent</em> with batch-1 parity: decode is memory-bandwidth-bound and 4-bit weight storage is identical either way; my own W4A16-era measurement (102.7 t/s) matches the published Marlin number almost exactly, and my W4A4 medians land lower mostly because the box was serving background production traffic during the run (top of spread 97.6; window acceptance 76.4% vs ~80% idle). Calling it backend <em>equivalence</em> would need a same-checkpoint, same-load A/B I haven&#8217;t run. Native FP4&#8217;s real wins are <strong>prefill/concurrency (compute-bound paths), full cudagraphs on the hybrid arch, and (decisively for me) the kernel path on which MTP stopped crashing.</strong> The practical takeaway: don&#8217;t switch backends expecting a batch-1 decode win, and when evaluating Marlin, look at prefill and concurrent load: that&#8217;s where the difference actually lives.</p><p>Patterns from this configuration (one model, one serving config, corroborated where noted, not universal laws): prefill held ~6.3K t/s in both modes on this Qwen3.6-35B setup (TTFT &#8776; prompt_tokens &#247; 6300; the vlaicu measurement of the same model reports the same ~6.2K, and note my Nemotron prefills at 8.5K; &#8216;flat across modes&#8217; generalizes, the absolute number doesn&#8217;t); <strong>thinking decoded ~25% faster than answering</strong> under MTP for this workload and acceptance profile (102.5 vs 82.2); per-position draft acceptance decayed 86.6% &#8594; 77.2% &#8594; 65.5% across positions 0/1/2 in the same window.</p><p>And the no-speculator baseline, same script, same day, my Nemotron-3-Nano-30B-A3B-NVFP4 (native FP4, no spec-decode, vLLM 0.22.1rc1-era):</p><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!U2mn!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6d2ca166-b325-4828-aff8-b42e8e7fb797_1126x304.png 424w, https://substackcdn.com/image/fetch/$s_!U2mn!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6d2ca166-b325-4828-aff8-b42e8e7fb797_1126x304.png 848w, https://substackcdn.com/image/fetch/$s_!U2mn!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6d2ca166-b325-4828-aff8-b42e8e7fb797_1126x304.png 1272w, https://substackcdn.com/image/fetch/$s_!U2mn!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6d2ca166-b325-4828-aff8-b42e8e7fb797_1126x304.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!U2mn!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6d2ca166-b325-4828-aff8-b42e8e7fb797_1126x304.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/6d2ca166-b325-4828-aff8-b42e8e7fb797_1126x304.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;vLLM on the DGX Spark, the Hard Way: Seven Months of GB10 Field Notes &#8212; table 4&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="vLLM on the DGX Spark, the Hard Way: Seven Months of GB10 Field Notes &#8212; table 4" title="vLLM on the DGX Spark, the Hard Way: Seven Months of GB10 Field Notes &#8212; table 4" srcset="https://substackcdn.com/image/fetch/$s_!U2mn!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6d2ca166-b325-4828-aff8-b42e8e7fb797_1126x304.png 424w, https://substackcdn.com/image/fetch/$s_!U2mn!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6d2ca166-b325-4828-aff8-b42e8e7fb797_1126x304.png 848w, https://substackcdn.com/image/fetch/$s_!U2mn!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6d2ca166-b325-4828-aff8-b42e8e7fb797_1126x304.png 1272w, https://substackcdn.com/image/fetch/$s_!U2mn!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6d2ca166-b325-4828-aff8-b42e8e7fb797_1126x304.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div></figure></div><p>Two things worth noticing: thinking &#8776; answering exactly as bandwidth-bound theory predicts when there&#8217;s no draft head (the Qwen thinking-speedup is <em>entirely</em> MTP), and the Mamba-hybrid prefills ~35% faster than the attention-hybrid Qwen (8.5K vs 6.3K t/s). Also 64.2 t/s is ~10% above my own 2026-06-02 measurement of the same deployment (58.5), measured with a different tool; methodology moves numbers, which is exactly why every figure here states its method and date.</p><p>Two spec-decode paths I evaluated but do <strong>not</strong> run:</p><ul><li><p><strong>n-gram lookup</strong>: the best story in this post, because I got it wrong in public first. My initial A/B showed n-gram <em>losing</em> 14% on my extraction workload despite 62% acceptance, and I wrote it up as &#8220;the free flag isn&#8217;t free.&#8221; A reviewer pushed back: my explanation blended three effects, and the prefill penalty looked configuration-specific. The controlled retest (every arm with an explicit <code>--max-num-batched-tokens</code>, prefix caching pinned off, k swept 1&#8594;4, token-equality checks) <strong>flipped the verdict entirely</strong>: extract-workload decode went 61.5 &#8594; <strong>117.3 tok/s (+91%) at k=3</strong>, with <em>no</em> prefill penalty in any arm. The original loss was my configuration all along (an unset scheduler budget interacting with speculation settings), exactly what the boot warning had been trying to tell me. The k-sweep shows the real economics: acceptance decays with depth (80.8% at k=1 &#8594; 64% at k=4) while throughput saturates at k=3. Two caveats survived the retest: speculative and plain decode are <em>not</em> token-identical at temperature 0 on this build (every spec arm diverged from the control&#8217;s greedy output at the same early position: deterministic batch-shape numerics, so validate output quality, don&#8217;t assume equivalence), and all of this is single-stream: speculation taxes throughput under concurrent load, so it ships with <code>--speculative-disable-by-batch-size</code> or not at all. The meta-lesson outranks the numbers: <strong>a configuration confound can invert your benchmark&#8217;s conclusion, and the warning that would have told you is in the boot log.</strong></p><p>And then the story turned again. Before promoting, I ran the production gates (an 8-way soak plus an output-quality gate with a control arm), and the quality gate killed it: the no-spec control produced <strong>20/20</strong> perfect JSON extractions, while ngram produced <strong>0/20</strong>, with version strings losing their final component (<code>"0.22"</code> for <code>"0.22.0"</code>) and JSON keys mutating mid-document (<code>"sentence"</code> &#8594; <code>"sent"</code>). The GPU proposer failed identically, localizing the defect to the shared verification path. The temperature-0 divergence I&#8217;d politely called &#8220;batch-shape numerics&#8221; was token corruption all along. (Two more gate catches on the way: prefix caching &#215; ngram hard-asserts in the Mamba-2 layers on this arch, and the burst-protection flag most guides recommend doesn&#8217;t exist on this build in either spelling.) Final ledger for n-gram on this box: +91% throughput, mechanically real, generating subtly wrong tokens, worthless <strong>on this hybrid Mamba/GDN architecture</strong> (the corruption lives in the shared verification path, upstream #39273; CPU and GPU proposers fail identically, which localizes it there, and on a pure-attention target n-gram may well be fine). Three verdicts in three days, each overturned by a better-controlled experiment than the last. If there&#8217;s one sentence to take from this post, it&#8217;s the last gate&#8217;s: <strong>speculative-decoding throughput without output validation is not a result.</strong></p></li><li><p><strong>DFlash</strong>: block-diffusion drafters, the biggest published single-stream wins (2.2&#8211;2.7&#215; on GB10, per AEON-7&#8217;s measurements). The caveat that keeps it off my fleet: on <strong>quantized (NVFP4/FP8) targets under stock vLLM, acceptance collapses</strong> (~4 of 15 tokens &#8594; a net slowdown); the working paths need a patched sm121 build or a BF16 target, and DFlash requires BF16 KV, incompatible with my fp8-KV recipes. MTP at ~95% real-traffic acceptance with zero extra memory sets a high bar.</p></li></ul><h2>Unified memory: the OOM you get with &#8220;free&#8221; RAM showing</h2><p><code>--gpu-memory-utilization</code> on Spark is a fraction of the whole 128 GB pool, shared with the OS, page cache, and every other container. Two facts make it dangerous:</p><ol><li><p><strong>The reservation is unswappable.</strong> On unified memory, vLLM&#8217;s KV-pool grab can&#8217;t be paged out. Set 0.6 on a box that&#8217;s also running a gateway and observability stack and you&#8217;ve made the kernel OOM-killer&#8217;s decision for it: that exact misconfiguration cost me a 2h40m production outage (instance reserved ~71 GiB, host hit ~2 GiB free with swap exhausted, <code>oom_kill</code>).</p></li><li><p><strong>Lowering it shrinks KV </strong><em><strong>capacity</strong></em><strong>, not speed.</strong> The pool is capacity, not a performance knob. My 35B runs at <strong>0.3</strong> (&#8776;34 GiB) while keeping the model&#8217;s full 262K context, because the math works:</p></li></ol><pre><code>KV cost  &#8776; 12.4 KB/token          &#8592; boot log: 7.15 GiB &#247; 617,494 pool tokens
                                     (Qwen3.6-35B-A3B, KV cache in fp8)
1 &#215; 262K sequence &#8776; 3.1 GiB
0.3 util &#8594; 7.15 GiB KV pool &#8594; 2.36x concurrency at full 262K context
                                     (vLLM prints this: Maximum concurrency 2.36x)
real coordinator/extraction traffic touches &lt; 2% of that</code></pre><p>A clarification that trips people up: <strong>two quantizations coexist in this server and they are different knobs.</strong> The <em>weights</em> are NVFP4 (that&#8217;s the checkpoint format the whole kernel-selection story is about); the <em>KV cache</em> is fp8 because I pass <code>--kv-cache-dtype fp8</code>. The 12.4 KB/token figure is a property of the KV dtype and the model&#8217;s attention layout; the weight quantization plays no part in it. (A widely-quoted figure for this same model family is ~16.6 KB/token, measured on a different checkpoint, vLLM build, and config. Hybrid attention/SSM models are exactly where per-token KV accounting varies across versions, another reason the formula beats the folklore: your own boot log prints both numbers; divide them.)</p><p>Read <code>Available KV cache memory</code> and <code>GPU KV cache size ... tokens</code> from your own boot log, and size <code>gpu-memory-utilization</code> from <code>max-num-seqs &#215; max-model-len</code> plus slack, not from a recipe&#8217;s 0.85 default, which assumes the box serves nothing else. If a load fails with memory apparently free, NVIDIA&#8217;s documented <em>debugging</em> workaround for UMA memory-reporting is to flush the page cache (<code>sync; echo 3 &gt; /proc/sys/vm/drop_caches</code>) and then restart the affected application; the cache holds unified memory CUDA can&#8217;t reclaim. Treat it as a diagnostic step, not routine ops.</p><h2>Smaller sharp edges, quickly</h2><ul><li><p><strong>Tool calling fails closed, loudly, and only at request time.</strong> Without <code>--enable-auto-tool-choice --tool-call-parser &lt;family&gt;</code>, the model loads fine and <code>/v1/models</code> is green; then every agent request 400s (<code>"auto" tool choice requires&#8230;</code>). The parser tracks the <em>model family</em>, not your client: <code>qwen3_coder</code>/<code>qwen3_xml</code> for Qwen3.6, <code>nemotron_v3</code>/nano parsers for Nemotron. The wrong parser fails <em>open</em>: the call text sits in <code>content</code>, <code>tool_calls</code> stays null, and your agent silently stalls after the think block.</p></li><li><p><strong>Hybrid-Mamba graph capture</strong> (<code>nemotron_h</code>): needed <code>--enforce-eager</code> on v0.20.0 (~37% throughput tax), fixed by v0.22.0; drop the flag and take the free speed.</p></li><li><p><strong><code>max_tokens</code> starvation on reasoning models</strong> breaks tool-call parsing in ways that look like model bugs. Budget ~10K+ output tokens for reasoning models in agent pipelines. It presents <em>identically</em> to the wrong-parser stall above (the agent hangs after the think block with no error), but the two are distinguishable: <strong>wrong parser</strong> gives <code>tool_calls</code> null with the call text sitting in <code>content</code>; <strong>budget starvation</strong> gives <code>finish_reason: length</code> with an unclosed <code>&lt;think&gt;</code> block. Check which before blaming the model.</p></li><li><p><strong>ConnectX-7 clustering: the 200 Gb/s per QSFP cable arrives as two ~100 Gb/s interfaces: configure both or you get half.</strong> Each port exposes two RoCE &#8220;twins&#8221; (each behind an x4 PCIe Gen5 link); my single-interface <code>ib_write_bw</code> measured ~111 Gb/s, consistent with one twin, so full NCCL bandwidth needs <code>NCCL_IB_HCA=&lt;twin1&gt;,&lt;twin2&gt;</code>. DGX Spark also doesn&#8217;t support GPUDirect RDMA; CUDA-memory traffic takes NVIDIA&#8217;s documented host-pinned fallback. And multi-node vLLM needs more than NCCL: a Ray cluster (or multi-node multiprocessing) with matching runtime environments and model paths, plus an explicitly selected distributed executor. TP=2 across two one-GPU Sparks works; PP=2 can reduce cross-node tensor-parallel collectives where the model supports it.</p></li><li><p><strong>Cold start is real</strong>: ~2.5 min weight load + ~25 s first-request JIT on my 30B-class models. Keep instances warm (no idle-timeout, no on-demand start in the request path) and fire a <code>max_tokens: 3</code> warm-up ping at boot.</p></li></ul><h2>What I&#8217;d tell you to deploy today</h2><p>For a 30&#8211;35B-class MoE NVFP4 reasoning model on one GB10, July 2026:</p><ol><li><p><strong>Checkpoint:</strong> a genuine W4A4 build (check <code>config.json</code> for <code>nvfp4-pack-quantized</code> with activation <code>num_bits: 4</code>; confirm <code>lm_head</code> is in the quant ignore-list if you do structured extraction; 4-bit lm_head gave me <code>!!!!</code> garbage on greedy JSON output).</p></li><li><p><strong>Image:</strong> a cu130-line nightly pinned by gitsha (mine: 0.23.1rc1-era), or v0.25.1 stable <em>after</em> an sm_121 smoke test.</p></li><li><p><strong>Flags:</strong> <code>--moe-backend auto</code> (verify the boot log says FLASHINFER_CUTLASS), <code>--kv-cache-dtype fp8</code>, MTP at <code>num_speculative_tokens: 3</code> <strong>after</strong> an 8-way soak, tool-call + reasoning parsers matched to the model family, and <code>--gpu-memory-utilization</code> computed from the KV math above, not defaulted.</p></li><li><p><strong>Process:</strong> pin digests, smoke-test raw completions on every image change, watch <code>/metrics</code> spec-decode acceptance and KV utilization, and never validate spec-decode at batch-1.</p></li></ol><h2>Still open: what I&#8217;m waiting on, and why it matters</h2><p>A snapshot of the upstream gaps that shape my config today. Each of these is a reason some recommendation above reads &#8220;pinned nightly&#8221; or &#8220;re-test later&#8221; instead of &#8220;just do X.&#8221;</p><ul><li><p><strong>vLLM #41190: illegal memory access in MTP speculative decoding (fixed on main 2026-07-29 via #48245/#49620; not yet in a tagged stable release, and upstream marks it a probable &#8220;may fix&#8221; rather than a confirmed resolution).</strong> The crash that took my 35B down for a month. Failure signature: the drafter path poisons the accepted-token buffer (all <code>-1</code>) and the engine dies with <code>CUDA error: an illegal memory access</code>, <em>only under concurrent load</em>, which is why batch-1 validation sails past it. The upstream fix locates the root cause deeper than my symptom: async-decode output-placeholder accounting underflows when KV-pressure preemption rolls back an in-flight request. That&#8217;s a scheduler bug, kernel-agnostic, which is precisely why concurrency (KV pressure, then preemption) triggers it and batch-1 never can. Why it&#8217;s pertinent: I didn&#8217;t patch it, I <strong>routed around it</strong>. The W4A4/CUTLASS configuration avoids the crashing combination, but since the true cause is scheduler-level, I can&#8217;t yet say whether I <em>avoided</em> or merely <em>masked</em> it. I ran that soak: W4A4/CUTLASS held at 100% KV utilization under concurrent MTP load with zero illegal-memory-access, but with prefix caching off, vLLM&#8217;s admission control never actually preempted an in-flight request, so saturation alone did not reproduce the crash. That is strong evidence the config is robust in practice, and a hint that the #41190 rollback needs conditions this config avoids, but it is not a formal proof of avoided over masked. Anyone on a W4A16 NVFP4 checkpoint (the Marlin path) with MTP under real concurrency stayed exposed until a nightly carries #48245, one more reason to run digest-nightlies, not stables. My rule stands regardless: no spec-decode config goes to production without an 8-way soak.</p></li><li><p><strong>vLLM #42084: out-of-bounds gather with prefix caching + spec decode on <code>sm_121a</code> (open, but see the 2026-08-05 update below).</strong> Constrains which feature <em>combinations</em> are safe on this architecture. The mechanism is hybrid-architecture-specific: the linear-attention state block table (Mamba-2 layers in Nemotron, Gated DeltaNet layers in Qwen3.6, both managed by vLLM&#8217;s SSM cache, which it confusingly calls &#8220;Mamba&#8221; even for GDN) and the speculative extension disagree on how many blocks a sequence owns, so the <code>torch.gather</code> indexes past the end; a pure-attention model has no second state block table to desync, so it is likely unaffected. Pertinent because prefix caching and speculation are both individually attractive for agent workloads: this is the reason I validate them together, not separately, before enabling both. (I did lose the production 35B to an illegal-memory-access around 2026-08-04 under concurrent extraction load, but with no logs and a coincident llamactl-restart incident on the same box I can&#8217;t attribute it &#8212; not to #42084, and not confidently to any particular feature combination &#8212; so I don&#8217;t count it as evidence either way.) <strong>Update 2026-08-05:</strong> on a v0.26.1-dev nightly I ran prefix caching + MTP together on my GB10 Qwen3.6-35B (the Gated DeltaNet hybrid, <em>not</em> classic Mamba) under a shared-prefix concurrent soak (40% cache hit, 86.9% KV): zero out-of-bounds, zero state-cache asserts, coherent output throughout. #42084 appears resolved via a new experimental cache-&#8221;align&#8221; mode that sets the attention block size to match the linear-attention page size. So the prefix-caching-vs-spec-decode either/or looks liftable on recent nightlies, with the caveat that vLLM still marks the align path experimental.</p></li><li><p><strong>The <code>b12x</code> backend is deliberately excluded from <code>auto</code> selection</strong> pending an upstream CUTLASS sm_121 MMA-guard fix (the exclusion is a documented choice in vLLM&#8217;s NVFP4 oracle). <code>b12x</code> (#40082) is the purpose-built sm_121 path and plausibly the fastest one, meaning some performance is intentionally left on the table today. When that guard fix lands, <code>auto</code> may silently start selecting <code>b12x</code>, which is both good news and a behavior change to catch in the boot log. I re-bench on every image bump partly for this.</p></li><li><p><strong>No aarch64 stable wheel has shipped <code>sm_121</code> cubins in three consecutive releases</strong> (v0.22.0 &#8594; v0.25.1, verified by me on the day each mattered), and the first CUDA-13 aarch64 &#8220;stable&#8221; tag turned out to be a mis-stamped dev artifact. This is the single reason my production images are digest-pinned <em>nightlies</em>, not a preference for living dangerously, but the only track whose wheels currently carry the kernels. The day a properly version-stamped cu13 aarch64 stable appears (check <code>/version</code> first), the calculus flips.</p></li><li><p><strong>NGC container lag (vllm #31424).</strong> NVIDIA&#8217;s official vLLM containers trail upstream by months: the newest NGC tag ships 0.22.1 while upstream stable is 0.25.1. Pertinent because &#8220;use the vendor container&#8221; is the default advice in most enterprise settings, and on this platform it currently can&#8217;t serve the newest model architectures or fixes. (NVIDIA closed #31424 as <code>not_planned</code>, so the lag is effectively permanent-by-neglect, which is <em>why</em> the vendor container can&#8217;t be the default on this platform.)</p></li><li><p><strong>Speculative decoding&#8217;s scheduler coupling</strong>: on this build, CPU n-gram speculation disables async scheduling, and the per-step token budget derives from your batch/speculation settings (vLLM says both in boot warnings). Mostly <em>mitigable from your side</em>: my controlled retest showed that simply setting <code>--max-num-batched-tokens</code> explicitly eliminated the penalty that sank my first experiment, and <code>ngram_gpu</code> (present and working on my 0.22-era build) didn&#8217;t disable async scheduling at all. What remains open upstream, now upgraded from observation to confirmed defect: n-gram speculation on this build <strong>corrupts structured output</strong> (control 20/20 clean vs ngram 0/20; dropped token components inside version strings, mutated JSON keys; identical signature from CPU and GPU proposers &#8594; shared verify-path bug), and prefix caching &#215; ngram hard-asserts in <code>mamba_mixer2</code> on the hybrid arch. Before filing, I duplicate-checked the tracker, and both bugs were already known, which is its own lesson: the assert is precisely open PR vllm#46424 (my data extends it from MTP to ngram), and the corruption belongs to the vllm#39273 hybrid-corruption cluster. I also tested vllm#40875&#8217;s config-only workaround (<code>prompt_lookup_min=8</code>): it did not mitigate on this Mamba2 hybrid: corruption changed shape but persisted, 0/20. My contribution upstream is corroborating data on existing issues, not new tickets, and until that cluster resolves, no n-gram config passes my quality gate regardless of its throughput.</p></li><li><p><strong>One I owe upstream:</strong> in my n-gram A/B, temperature-0 output <em>lengths diverged run-to-run</em> with speculation on (502&#8211;1116 tokens) while the control was deterministic (529 every run). Speculative decoding is supposed to preserve the target distribution; numerics on this path apparently aren&#8217;t bit-stable on this build. I haven&#8217;t filed a reproducer yet; until then treat it as my observation, not a confirmed upstream bug.</p></li></ul><h2>Appendix: every knob this post mentions</h2><div class="captioned-image-container"><figure><div class="image-link image2" data-component-name="Image2ToDOM"><div class="image2-inset"><picture><source type="image/webp" srcset="https://substackcdn.com/image/fetch/$s_!Hl7o!,w_424,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd57ee23-9b7d-47e8-a155-624fd427541b_3388x1156.png 424w, https://substackcdn.com/image/fetch/$s_!Hl7o!,w_848,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd57ee23-9b7d-47e8-a155-624fd427541b_3388x1156.png 848w, https://substackcdn.com/image/fetch/$s_!Hl7o!,w_1272,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd57ee23-9b7d-47e8-a155-624fd427541b_3388x1156.png 1272w, https://substackcdn.com/image/fetch/$s_!Hl7o!,w_1456,c_limit,f_webp,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd57ee23-9b7d-47e8-a155-624fd427541b_3388x1156.png 1456w" sizes="100vw"><img src="https://substackcdn.com/image/fetch/$s_!Hl7o!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd57ee23-9b7d-47e8-a155-624fd427541b_3388x1156.png" data-attrs="{&quot;src&quot;:&quot;https://substack-post-media.s3.amazonaws.com/public/images/fd57ee23-9b7d-47e8-a155-624fd427541b_3388x1156.png&quot;,&quot;srcNoWatermark&quot;:null,&quot;fullscreen&quot;:null,&quot;imageSize&quot;:null,&quot;height&quot;:null,&quot;width&quot;:null,&quot;resizeWidth&quot;:null,&quot;bytes&quot;:null,&quot;alt&quot;:&quot;vLLM on the DGX Spark, the Hard Way: Seven Months of GB10 Field Notes &#8212; table 5&quot;,&quot;title&quot;:null,&quot;type&quot;:null,&quot;href&quot;:null,&quot;belowTheFold&quot;:true,&quot;topImage&quot;:false,&quot;internalRedirect&quot;:null,&quot;isProcessing&quot;:false,&quot;align&quot;:null,&quot;offset&quot;:false}" class="sizing-normal" alt="vLLM on the DGX Spark, the Hard Way: Seven Months of GB10 Field Notes &#8212; table 5" title="vLLM on the DGX Spark, the Hard Way: Seven Months of GB10 Field Notes &#8212; table 5" srcset="https://substackcdn.com/image/fetch/$s_!Hl7o!,w_424,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd57ee23-9b7d-47e8-a155-624fd427541b_3388x1156.png 424w, https://substackcdn.com/image/fetch/$s_!Hl7o!,w_848,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd57ee23-9b7d-47e8-a155-624fd427541b_3388x1156.png 848w, https://substackcdn.com/image/fetch/$s_!Hl7o!,w_1272,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd57ee23-9b7d-47e8-a155-624fd427541b_3388x1156.png 1272w, https://substackcdn.com/image/fetch/$s_!Hl7o!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffd57ee23-9b7d-47e8-a155-624fd427541b_3388x1156.png 1456w" sizes="100vw" loading="lazy"></picture><div></div></div></div></figure></div><h2>Sources &amp; receipts</h2><ul><li><p>vLLM team, &#8220;vLLM on NVIDIA DGX Spark&#8221;: <a href="https://vllm.ai/blog/2026-06-01-vllm-dgx-spark">https://vllm.ai/blog/2026-06-01-vllm-dgx-spark</a></p></li><li><p>NVIDIA DGX Spark playbooks: <a href="https://build.nvidia.com/spark/vllm/instructions">https://build.nvidia.com/spark/vllm/instructions</a> &#183; <a href="https://github.com/NVIDIA/dgx-spark-playbooks">https://github.com/NVIDIA/dgx-spark-playbooks</a></p></li><li><p>vLLM PRs referenced: #38126, #39825 (v0.20.0); #35568, #40082 (v0.22.0); #40923 (v0.22.1rc0). Upstream spec-decode IMA issues: #41190, #42084.</p></li><li><p>Community playbook with the broadest recipe coverage, the most useful single Spark reference I&#8217;ve found, whose Marlin guidance this post extends with checkpoint-format context: <a href="https://vlaicu.io/posts/dgx-vllm/">https://vlaicu.io/posts/dgx-vllm/</a></p></li><li><p>Community reports illustrating the software-immaturity-vs-hardware-limitation distinction this post draws: a slow-tokens/s thread where the cause is corrected from hardware to an immature kernel stack (<a href="https://forums.developer.nvidia.com/t/slow-performance-for-redhatai-qwen3-coder-next-nvfp4/363129">https://forums.developer.nvidia.com/t/slow-performance-for-redhatai-qwen3-coder-next-nvfp4/363129</a>), and the &#8220;SM121 software support is severely lacking&#8221; thread (<a href="https://forums.developer.nvidia.com/t/dgx-spark-sm121-software-support-is-severely-lacking-official-roadmap-needed/357663">https://forums.developer.nvidia.com/t/dgx-spark-sm121-software-support-is-severely-lacking-official-roadmap-needed/357663</a>)</p></li><li><p>AEON-7 DFlash GB10 measurements: <a href="https://github.com/AEON-7/vllm-dflash">https://github.com/AEON-7/vllm-dflash</a> &#183; Z-Lab DFlash: <a href="https://github.com/z-lab/dflash">https://github.com/z-lab/dflash</a></p></li><li><p>Checkpoints: <code>unsloth/Qwen3.6-35B-A3B-NVFP4</code> (deployed), <code>nvidia/Qwen3.6-35B-A3B-NVFP4</code>, <code>nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4</code>, <code>nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-NVFP4</code></p></li><li><p>Bench methodology: single-stream prefill/decode script measuring TTFT, prefill tok/s, decode tok/s from server token counts, with prefix-cache-defeating nonces and forced full-length generation (<code>ignore_eos</code>), adapted from the vlaicu.io playbook &#167;10.</p></li></ul>]]></content:encoded></item></channel></rss>