What FuTransformer is

FuTransformer is a CUDA implementation of a Llama-style Transformer block designed specifically for NVIDIA Hopper GPUs, with distinct precision regimes for the two main phases of inference:

  • Prefill: FP8 matrix multiplications and activations for processing the prompt.
  • Decode: INT4 projection weights and an INT4 KV-cache path for single-token generation.

The design is built around the observation that:

  • Prefill “reuses weights across tokens and is dominated by matrix multiplication and activation data movement” FuTransformer.
  • Single-token decode has “limited reuse and is dominated by weight and KV-cache reads” FuTransformer.

FuTransformer applies the CODA GEMM-plus-epilogue approach to inference, then pushes it further with precision-aware layout choices, epilogue fusion, and cache quantization strategies tailored to Hopper’s Tensor Cores FuTransformer.

The result: a Hopper-specific block implementation that accelerates both prefill and decode relative to compiled PyTorch and a CODA baseline at BF16/FP8, and a dedicated INT4 decode path that delivers additional speedups with controlled perplexity loss FuTransformer.

---

The two phases: prefill vs decode

A Llama-style Transformer block performs the same logical computation in both prefill and decode, but the systems constraints differ sharply.

Prefill characteristics

During prefill you process a full prompt sequence, often with non-trivial batch size:

  • The *same* projection and feed-forward weights are reused across many tokens.
  • Work is dominated by:
  • Large matrix multiplications (QKV, attention output, MLP projections).
  • Activation data movement through the layers FuTransformer.

That shapes what matters:

  • Arithmetic intensity is relatively high.
  • Making GEMM fast and minimizing activation bandwidth gives you most of the win.
  • Weight precision can be reduced so long as epilogue and normalization logic keep errors bounded.

Decode characteristics

During single-token decode:

  • Each step processes one token per sequence, reusing weights but not across many time steps.
  • The dominant cost is:
  • Re-reading projection weights.
  • Re-reading and updating the KV-cache FuTransformer.

So the bottleneck flips:

  • It’s less about raw FLOPs, more about:
  • Memory bandwidth to weights and KV-cache.
  • Cache layout and quantization that reduce bytes moved per step.

FuTransformer’s architecture mirrors these constraints:

  • FP8-heavy, GEMM-centric prefill path.
  • Extremely bandwidth-lean INT4 decode with Tensor Core–friendly layouts.

---

How the FP8 prefill path works

FuTransformer’s prefill path layers several optimizations on top of the CODA “GEMM-plus-epilogue” pattern.

GEMM-plus-epilogue on Hopper

CODA introduced a pattern where:

  • A single GEMM kernel directly computes the matrix multiplication.
  • The “epilogue” (e.g., bias add, activation, normalization/scale) is fused into the same kernel to avoid extra reads/writes.

FuTransformer “applies CODA’s GEMM-plus-epilogue approach to inference” for Llama-style blocks on Hopper FuTransformer. For prefill, it then adds:

  • Direct KV-cache placement.
  • RMSNorm gain folding.
  • FP8 output writes with delayed scaling.
  • Support for prefill without padding FuTransformer.

This means the GEMM kernel isn’t just computing QKV/MLP; it’s also:

  • Normalizing activations.
  • Quantizing and writing them to FP8.
  • Emitting keys/values directly into cache layouts that the decode kernel can consume.

Folding RMSNorm gain into weights

Llama-style blocks use RMSNorm with a fixed, learned gain vector. FuTransformer “folds the fixed RMSNorm gain into weights” FuTransformer. Practically:

  • The normalization gain is pre-multiplied into the weight matrices.
  • At runtime, the block can:
  • Apply RMS normalization without a separate scaling pass.
  • Let the GEMM epilogue handle the reduced-form normalization.

The payoff: fewer memory reads and a simpler kernel epilogue.

Direct KV-cache placement

The attention block generates K and V for each token. Instead of:

  1. Running projection.
  2. Writing intermediate tensors.
  3. Launching a separate kernel to copy/rearrange them into KV-cache.

FuTransformer “adds direct KV-cache placement” [FuTransformer](http://arxiv.org/abs/2610.23796v1]:

  • The attention projection kernel writes K and V directly into the KV-cache layout used for decode.
  • This avoids an extra transformation and aligns the memory layout with the later INT4 decode pathway.

FP8 outputs with delayed scaling

To exploit Hopper’s FP8 Tensor Cores without losing too much accuracy, FuTransformer:

  • “[Writes] FP8 output directly with delayed scaling” FuTransformer.

The mechanism implied by that description:

  • The kernel computes in a higher-precision path where needed (e.g., accumulation).
  • It writes activations out already quantized to FP8, but scaling factors are chosen or updated later (“delayed”) to maintain numeric stability.

This removes separate FP8 quantization passes and keeps activation bandwidth low during prefill.

No-padding prefill

FuTransformer’s prefill path “support[s] prefill without padding” FuTransformer, allowing:

  • Processing input sequences without padding them up to a multiple of some tile length.
  • Avoiding wasted computation on padding tokens, which matters for latency-sensitive workloads and variable-length prompts.

On an H100, these pieces combine into:

  • Up to 1.48× faster prefill than compiled PyTorch FP8 [FuTransformer](http://arxiv.org/abs/2610.23796v1].
  • A throughput of 1.29 PFLOP/s for prefill [FuTransformer](http://arxiv.org/abs/2610.23796v1].

For the gate/up projection in a SwiGLU MLP at the 8B model shape:

  • FuTransformer’s BF16 path is 1.13× faster than CODA [FuTransformer](http://arxiv.org/abs/2610.23796v1].
  • The FP8 path is 2.02–2.18× faster than the evaluated BF16 CODA path [FuTransformer](http://arxiv.org/abs/2610.23796v1].

Those are kernel-level numbers, but they give a lower bound on the stack-level speedups you can expect when prefill dominates.

---

How the INT4 decode path works

For single-token decode, FuTransformer changes the quantization regime and memory strategy.

INT4 projection weights and KV-cache copy

The decode kernels:

  • Use INT4 projection weights.
  • Use an INT4 KV-cache copy [FuTransformer](http://arxiv.org/abs/2610.23796v1].

That is, both:

  • The matrices used for projecting Q/K/V and MLP activations, and
  • The key/value tensors stored in cache

are kept in 4-bit integer form on device and dequantized on the fly.

Why this matters for decode:

  • Each time step must re-read the full set of weights and keys/values.
  • Dropping to 4 bits cuts the bytes moved per token by roughly 4× vs BF16 (ignoring overhead for scales and layout).
  • The main risk is quantization error; FuTransformer mitigates this with layout choices and a transform described below.

Tensor Core fragment layouts and direct dequantization

Weights and keys “follow Tensor Core fragment layouts for direct dequantization into MMA operands without full intermediate tiles” [FuTransformer](http://arxiv.org/abs/2610.23796v1]. Concretely:

  • The INT4 data is laid out in memory in the same pattern that Tensor Cores expect for their matrix fragments.
  • When a kernel loads data:
  • It can directly unpack and dequantize into the Tensor Core’s MMA operand registers.
  • It avoids materializing full dequantized tiles in global or shared memory.

This trading pattern:

  • Lowers on-chip memory traffic and register pressure.
  • Preserves the GEMM-friendly memory layout that Hopper’s Tensor Cores want, even with INT4.

Block-diagonal Hadamard transform on KV-cache

Quantizing KV-cache to INT4 can be especially lossy in outlier channels—dimensions whose magnitude spikes, driving the global scale.

FuTransformer adds “a block-diagonal Hadamard transform [that] reduces cache quantization error from outlier channels” [FuTransformer](http://arxiv.org/abs/2610.23796v1]. In effect:

  • The cache channels are transformed via a block-diagonal Hadamard matrix before quantization.
  • This redistributes energy across dimensions, so:
  • Outliers are “smeared” across channels.
  • The per-block quantization range is better utilized.

Then, at use time, the inverse structural effect is accounted for when reading data back.

The payoff is reduced quantization error for the same 4-bit precision, which shows up both in perplexity and in how aggressively you can compress cache activations.

Speedups vs PyTorch and vs FP8

Across 8B and 70B Llama-like shapes and batch sizes 1–32, the INT4 decode implementation is:

  • 1.3–2.4× faster than the faster of PyTorch’s BF16 and INT4 blocks at each configuration point [FuTransformer](http://arxiv.org/abs/2610.23796v1].
  • More specifically:
  • 2.1–3.3× faster than compiled, CUDA-graphed BF16 blocks [FuTransformer](http://arxiv.org/abs/2610.23796v1].
  • 1.3–6.3× faster than the same block using PyTorch’s INT4 projection kernel, which “targets a single query row and loses to BF16 at batch 32” [FuTransformer](http://arxiv.org/abs/2610.23796v1].
  • 31–51% faster than FuTransformer’s own FP8 path on decode [FuTransformer](http://arxiv.org/abs/2610.23796v1].

End-to-end, for Llama‑3.1‑8B decode:

  • FuTransformer “takes 2.8 ms per step at batch one, versus 8.1 ms for the compiled transformers implementation” [FuTransformer](http://arxiv.org/abs/2610.23796v1].

This is the number to map directly to your per-token latency budget: on H100, a ~3× reduction in decode step time at batch 1 for this model size.

---

Accuracy and perplexity trade-offs

Running INT4 end-to-end is only useful if it stays close to the original model behavior.

On WikiText-2, decoding every token through the INT4 weight path gives:

  • Baseline perplexity (compiled path): 6.24 [FuTransformer](http://arxiv.org/abs/2610.23796v1].
  • With straightforward round-to-nearest INT4 weights: 7.23 [FuTransformer](http://arxiv.org/abs/2610.23796v1].
  • With GPTQ in the same INT4 format: 6.99 [FuTransformer](http://arxiv.org/abs/2610.23796v1].
  • Adding the rotated INT4 cache (with Hadamard-style transform) “adds 0.20” perplexity [FuTransformer](http://arxiv.org/abs/2610.23796v1].

So relative to 6.24:

  • Pure INT4 weights and cache with round-to-nearest give +0.99 perplexity.
  • GPTQ-compressed weights plus rotated cache give +0.95 perplexity.

The authors also note that “the kernels match a quantized reference computation within 0.001 perplexity” [FuTransformer](http://arxiv.org/abs/2610.23796v1]. That is:

  • Once you specify a quantized reference (e.g., GPTQ INT4), FuTransformer’s kernels reproduce its behavior extremely closely; most of the perplexity delta is due to quantization, not kernel math differences.

Operationally, this means:

  • You can treat perplexity degradation as a function of your quantization scheme, not of the kernel.
  • Kernel choice is dominated by speed and bandwidth considerations as long as you match the reference quantizer.

---

How FuTransformer fits in a Llama inference stack

FuTransformer is a block-level CUDA implementation. To use it in practice in an agentic or batch-serving system, you’d:

  • Plug it in as a drop-in replacement for the default Transformer block kernels in a Llama-style model graph.
  • Route requests as:
  • FP8 prefill for the initial prompt.
  • INT4 decode for per-token steps.

Because FuTransformer:

  • Supports prefill without padding [FuTransformer](http://arxiv.org/abs/2610.23796v1], it is well-suited to workloads with heterogeneous sequence lengths.
  • Exploits INT4 cache and weight layouts targeted at Hopper Tensor Cores, it’s tied closely to this hardware generation [FuTransformer](http://arxiv.org/abs/2610.23796v1].

When integrating:

  • You must ensure model weights are available in both FP8/BF16 (for prefill) and INT4 (for decode) formats, or at least in INT4 plus whatever is needed for prefill.
  • You need to allocate KV-cache memory in the INT4 layout expected by FuTransformer’s decode kernels from the start, as prefill directly writes into that layout.

The stack then looks roughly like:

  1. Host-level runtime orchestrates prefill vs decode phases per request.
  2. FuTransformer kernels handle Llama-style block computation on H100:
  • FP8 GEMM+epilogue prefill with direct KV placement.
  • INT4 decode, reusing the same KV-cache.
  1. Quantization tooling (e.g., GPTQ in INT4 format) prepares weight and cache quantization parameters to control perplexity [FuTransformer](http://arxiv.org/abs/2610.23796v1].

---

Why it matters now

From the reported results:

  • Kernel-level FP8 prefill: up to 1.48× PyTorch FP8 at 1.29 PFLOP/s [FuTransformer](http://arxiv.org/abs/2610.23796v1].
  • SwiGLU gate/up at 8B:
  • BF16 FuTransformer: 1.13× CODA [FuTransformer](http://arxiv.org/abs/2610.23796v1].
  • FP8 FuTransformer: 2.02–2.18× CODA BF16 [FuTransformer](http://arxiv.org/abs/2610.23796v1].
  • INT4 decode:
  • 2.1–3.3× faster than compiled BF16 PyTorch blocks.
  • 1.3–6.3× faster than PyTorch’s INT4 block.
  • 31–51% faster than FuTransformer FP8 decode [FuTransformer](http://arxiv.org/abs/2610.23796v1].
  • End-to-end decode step for Llama‑3.1‑8B at batch=1: 2.8 ms vs 8.1 ms [FuTransformer](http://arxiv.org/abs/2610.23796v1].

If you are building:

  • Agentic systems with long-running sessions, or
  • High-throughput text generation services on H100,

these numbers translate directly into:

  • More tokens per second per GPU.
  • Lower marginal cost per request at fixed hardware.
  • Better ability to mix large-batch prefill with low-latency single-token decode by matching each phase with its optimal precision/layout.

Because the kernels match a quantized reference within 0.001 perplexity, you can use FuTransformer as the kernel backend for whatever quantization pipeline you already trust [FuTransformer](http://arxiv.org/abs/2610.23796v1]. The critical engineering decisions move to:

  • Which quantization (e.g., GPTQ) you use.
  • How much perplexity degradation you can afford for your product UX.

---

What is not documented

The sources do not document:

  • Implementation details of the CODA GEMM-plus-epilogue approach beyond its name and the fact FuTransformer builds on it.
  • Exact CUDA kernel code, launch configurations, tile/block sizes, or memory hierarchy usage.
  • Specific Tensor Core fragment shapes, FP8 formats, or scale schemes used in practice.
  • How FuTransformer integrates with specific frameworks (e.g., PyTorch API surface, graph capture, or compilation flows).
  • Memory footprint comparisons between FP8/INT4 FuTransformer and baseline implementations.
  • Behavior on non-Hopper GPUs or non-Llama architectures, or any training-time applications.