{"id":564593,"date":"2026-03-05T14:04:39","date_gmt":"2026-03-05T14:04:39","guid":{"rendered":"https:\/\/Blockchain.News\/news\/flashattention-4-nvidia-blackwell-b200-optimization"},"modified":"2026-03-05T14:04:39","modified_gmt":"2026-03-05T14:04:39","slug":"flashattention-4-hits-71-gpu-utilization-on-nvidia-blackwell-b200","status":"publish","type":"post","link":"https:\/\/e-bitco.in\/index.php\/2026\/03\/05\/flashattention-4-hits-71-gpu-utilization-on-nvidia-blackwell-b200\/","title":{"rendered":"FlashAttention-4 Hits 71% GPU Utilization on NVIDIA Blackwell B200"},"content":{"rendered":"
\n

Terrill Dicki<\/a> Mar 05, 2026 14:04<\/span> <\/p>\n

Together AI’s FlashAttention-4 achieves 1,605 TFLOPs\/s on B200 GPUs, up to 2.7x faster than Triton. New pipelining overcomes asymmetric hardware scaling bottlenecks.<\/p>\n

\"FlashAttention-4 <\/a> <\/figure>\n

Together AI has released FlashAttention-4, achieving up to 1,605 TFLOPs\/s on NVIDIA’s Blackwell B200 GPUs\u2014representing 71% hardware utilization and marking a 2.7x speedup over Triton implementations. The release addresses a fundamental challenge in modern AI hardware: tensor core throughput is scaling far faster than other critical resources.<\/p>\n

For context, NVIDIA’s market cap sits at $4.49 trillion as of March 4, 2026, with shares trading at $179.86. The company released its own Flash Attention optimization guide for Blackwell GPUs just yesterday, signaling the growing importance of attention optimization in production AI workloads.<\/p>\n

The Asymmetric Scaling Problem<\/h2>\n

Here’s what makes this interesting. From Hopper H100 to Blackwell B200, BF16 tensor core throughput jumped from 1 to 2.25 PFLOPs. But special function units for exponential operations and shared memory bandwidth? Unchanged. That creates a bottleneck nobody was expecting.<\/p>\n

The Together AI team discovered that the forward pass isn’t compute-bound at all on B200\u2014it’s bottlenecked by exponential calculations in softmax. The backward pass? Shared memory traffic dominates. Traditional attention optimization focused on the wrong constraints.<\/p>\n

How FA4 Solves It<\/h2>\n

The forward pass uses a ping-pong schedule processing two query tiles per CTA, with dedicated warpgroups handling softmax while others issue matrix operations. The clever bit: software emulation of the exponential function using FMA units alongside hardware MUFU.EX2, effectively doubling exponential throughput.<\/p>\n

Conditional online softmax rescaling skips small corrections entirely. If the max jump stays below a threshold, the kernel avoids unnecessary vector operations. Final normalization still produces correct results\u2014but the critical path shrinks considerably.<\/p>\n

The backward pass exploits Blackwell’s new 2-CTA MMA mode, partitioning output accumulators across CTA pairs. Each CTA stages half of operand B while keeping only its accumulator slice, roughly halving shared memory traffic. Global atomic reductions for dQ gradients also drop by half.<\/p>\n

Performance Numbers<\/h2>\n

Against cuDNN 9.13, FlashAttention-4 delivers 1.1-1.3x improvement on forward passes and consistent gains on backward passes at large sequence lengths. The Triton comparison shows the starkest difference\u2014up to 2.7x faster forward performance.<\/p>\n

Deterministic mode, which serializes global reductions for reproducible training, still achieves 85-90% of non-deterministic throughput. That’s significant for teams requiring exact reproducibility across training runs.<\/p>\n

The Broader Picture<\/h2>\n

FlashAttention has evolved rapidly since its May 2022 debut. Version 1 achieved 25-40% utilization on A100s. FA2 pushed that to 50-73% in July 2023. FA3 targeted Hopper GPUs specifically, hitting 75% utilization with FP16 and nearly 1.2 PFLOPS with FP8.<\/p>\n

FA4 represents a philosophical shift\u2014algorithm and kernel co-design that accounts for asymmetric hardware evolution. The techniques have already been partially incorporated into cuDNN 9.13 and 9.14 through collaboration with NVIDIA’s teams.<\/p>\n

The implementation uses CuTe-DSL, CUTLASS’s Python kernel DSL, cutting compile times by 20-30x versus C++ templates. For teams running large-scale training on Blackwell hardware, the efficiency gains compound across millions of attention operations daily.<\/p>\n

Image source: Shutterstock<\/i><\/span> Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

Terrill Dicki Mar 05, 2026 14:04 Together AI’s FlashAttention-4 achieves 1,605 TFLOPs\/s on B200 GPUs, up to 2.7x faster than Triton. New pipelining overcomes asymmetric hardware scaling bottlenecks. Together AI has released FlashAttention-4, achieving up to 1,605 TFLOPs\/s on NVIDIA’s Blackwell B200 GPUs\u2014representing 71% hardware utilization and marking a 2.7x speedup over Triton implementations. The […]<\/p>\n","protected":false},"author":1,"featured_media":564594,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[12],"tags":[20916,21929,23987,23503,25,2148],"class_list":{"0":"post-564593","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-blockchain","8":"tag-ai-infrastructure","9":"tag-blackwell","10":"tag-flashattention-4","11":"tag-gpu-optimization","12":"tag-news","13":"tag-nvidia"},"_links":{"self":[{"href":"https:\/\/e-bitco.in\/index.php\/wp-json\/wp\/v2\/posts\/564593","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/e-bitco.in\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/e-bitco.in\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/e-bitco.in\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/e-bitco.in\/index.php\/wp-json\/wp\/v2\/comments?post=564593"}],"version-history":[{"count":0,"href":"https:\/\/e-bitco.in\/index.php\/wp-json\/wp\/v2\/posts\/564593\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/e-bitco.in\/index.php\/wp-json\/wp\/v2\/media\/564594"}],"wp:attachment":[{"href":"https:\/\/e-bitco.in\/index.php\/wp-json\/wp\/v2\/media?parent=564593"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/e-bitco.in\/index.php\/wp-json\/wp\/v2\/categories?post=564593"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/e-bitco.in\/index.php\/wp-json\/wp\/v2\/tags?post=564593"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}