News
NVIDIA says three DGX Spark optimizations lifted Ling-3.0-flash throughput and cut latency in GSM8K testing, with a caveat on stacking them.
NVIDIA said three software optimizations can materially improve inference on DGX Spark when running Inclusion AI’s Ling-3.0-flash model, with the biggest gains coming from speculative decoding once the components are configured to work together. In a post published Oct. 27, 2026, the company reported faster token generation and lower end-to-end latency, while also warning that one integration misstep can make the combined setup slower than the unoptimized baseline.
What NVIDIA measured
The evaluation used Ling-3.0-flash-fp4, a mixture-of-experts model with 124 billion total parameters and 5.1 billion active parameters per token, according to NVIDIA’s post. The baseline followed the company’s cookbook configuration, including FlashInfer MXFP4 MoE kernels, NVIDIA CUTLASS dense GEMM and fp32 LM-head output. NVIDIA said profiling showed the decoding path was memory-bound, with weight reads dominating execution time, and that the MoE computation and output-layer projection were the main targets for optimization.
On a synthetic workload with 32,000 input tokens, 2,000 output tokens and batch size 1, the first optimization, swapping in the Humming MXFP4 MoE backend, reduced time per output token from 27.71 ms to 25.35 ms and increased decode throughput from 36.09 tok/s to 39.45 tok/s, or 9.3% relative to baseline, according to NVIDIA. Adding online FP8 LM-head quantization cut TPOT further to 22.76 ms and raised throughput to 43.94 tok/s, a cumulative gain of 21.7% versus baseline.
The third technique, multi-token prediction speculative decoding with NEXTN, had the largest effect in the synthetic test. NVIDIA said the full three-optimization stack brought TPOT down to 9.72 ms and lifted decode throughput to 102.88 tok/s, a 185.1% improvement over baseline on that synthetic setup. The company said the MTP accept length reached 3.90 tokens per round in that test, but cautioned that the result does not establish comparable gains on other workloads.
End-to-end GSM8K results were more modest
NVIDIA also reported a stricter end-to-end comparison across all 1,319 GSM8K test questions using five-shot prompting, temperature 0.6, top_p 0.95, top_k 29 and max_tokens 32,768, with chain-of-thought enabled. In that run, the baseline posted 26.45 ms TPOT and 37.81 tok/s, while the fully optimized configuration reached 18.56 ms TPOT and 53.89 tok/s, a 42.5% throughput increase. Mean latency fell from 15.21 seconds to 11.49 seconds, a 24.5% reduction, while GSM8K accuracy moved from 96.89% to 96.82%, a difference of 0.08 percentage points, according to the post.
The company said the gap between synthetic and real-workload gains reflected several factors, including more repetitive synthetic outputs, simpler greedy sampling in the synthetic test and shorter real outputs, which dilute the fixed costs of each decode step. NVIDIA noted that the median GSM8K output length was about 310 tokens, versus a fixed 2,000-token synthetic output, and said the synthetic benchmark therefore overstates speculative-decoding benefits for other workloads.
Integration caveat matters
NVIDIA said the three techniques do not overlap conceptually: Humming and FP8 LM-head quantization reduce the cost of each decode step, while MTP reduces the number of steps. But the company also flagged a deployment issue. When online FP8 LM-head quantization and MTP are enabled together, the draft model must share the full LM-head module with the target model. Without that interface, NVIDIA said the framework can silently fall back to a broken configuration in which accept length drops to 1.00 and throughput falls below the baseline.
The post’s practical warning is that startup validation matters as much as benchmark numbers. NVIDIA said developers should verify that each optimization is active at launch rather than inferring configuration state from later performance alone. The article did not provide a next scheduled catalyst beyond the published guidance.
