{"id":554801,"date":"2026-02-12T06:48:44","date_gmt":"2026-02-12T06:48:44","guid":{"rendered":"https:\/\/Blockchain.News\/news\/together-ai-cpd-cache-aware-inference-40-percent-faster"},"modified":"2026-02-12T06:48:44","modified_gmt":"2026-02-12T06:48:44","slug":"together-ai-achieves-40-faster-llm-inference-with-cache-aware-architecture","status":"publish","type":"post","link":"https:\/\/e-bitco.in\/index.php\/2026\/02\/12\/together-ai-achieves-40-faster-llm-inference-with-cache-aware-architecture\/","title":{"rendered":"Together AI Achieves 40% Faster LLM Inference With Cache-Aware Architecture"},"content":{"rendered":"
\n

Joerg Hiller<\/a> Feb 12, 2026 06:48<\/span> <\/p>\n

Together AI’s new CPD system separates warm and cold inference workloads, delivering 35-40% higher throughput for long-context AI applications on NVIDIA B200 GPUs.<\/p>\n

\"Together <\/a> <\/figure>\n

Together AI has unveiled a cache-aware disaggregated inference architecture that boosts throughput by up to 40% for large language models handling long prompts\u2014a development that could reshape economics for AI-native applications from coding copilots to retrieval-augmented systems.<\/p>\n

The new system, called cache-aware prefill-decode disaggregation (CPD), tackles a growing bottleneck in AI infrastructure: serving prompts exceeding 100,000 tokens without crippling latency. As context windows expand across the industry, the compute required to process new versus repeated context has become a critical performance divider.<\/p>\n

How CPD Works<\/h2>\n

The architecture splits inference into three distinct node types rather than the traditional two-tier approach. Pre-prefill nodes handle “cold” requests\u2014prompts with little reusable context that require full computation. Standard prefill nodes prioritize “warm” requests that can pull cached key-value states instead of recomputing them. Decode nodes remain isolated for latency-sensitive token generation.<\/p>\n

A three-level KV-cache hierarchy underpins the system: GPU memory at the fastest tier, host DRAM in the middle, and a cluster-wide distributed cache connected via RDMA at the base. When a cold request gets processed, its KV state writes to the distributed cache. Subsequent similar requests fetch this state in bulk, converting seconds of compute into hundreds of milliseconds of transfer.<\/p>\n

The router makes real-time decisions by estimating how much of each incoming prompt can be served from cache, steering low-reuse requests to pre-prefill nodes while high-reuse traffic takes the fast path.<\/p>\n

Benchmark Results on B200 GPUs<\/h2>\n

Testing on NVIDIA B200 GPUs with tensor parallelism across four GPUs per node revealed stark differences from conventional approaches. The baseline system\u2014two prefill nodes sharing capacity\u2014saturated around 0.75-0.8 queries per second per GPU. CPD sustained approximately 1.1-1.15 QPS per GPU before hitting the same wall.<\/p>\n

Latency improvements proved equally significant. Under increasing load, the baseline’s median time-to-first-token climbed past one second and into multi-second territory. CPD maintained sub-second to low-second median TTFT even at QPS levels where the baseline had already saturated.<\/p>\n

The 600-second steady-state tests used a synthetic coding agent workload designed to mirror real AI-assisted development scenarios\u2014large shared codebase context with multi-turn interactions.<\/p>\n

Why This Matters for AI Infrastructure<\/h2>\n

The performance gap highlights a shift in what determines inference efficiency. Raw model execution speed matters less than system-level scheduling when context windows grow large. A 100K-token prompt that initially requires seconds of compute can drop to a few hundred milliseconds once its context warms in the cache hierarchy.<\/p>\n

For organizations running AI agents, multi-turn chatbots, or retrieval-augmented generation at scale, the throughput gains translate directly to infrastructure cost savings\u2014or the ability to serve more users on existing hardware. The 35-40% improvement in sustainable QPS represents meaningful capacity that compounds across large deployments.<\/p>\n

Together AI’s approach suggests that as foundation models continue expanding context capabilities, the infrastructure layer will need increasingly sophisticated workload separation to avoid letting expensive cold prompts dominate shared resources.<\/p>\n

Image source: Shutterstock<\/i><\/span> Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

Joerg Hiller Feb 12, 2026 06:48 Together AI’s new CPD system separates warm and cold inference workloads, delivering 35-40% higher throughput for long-context AI applications on NVIDIA B200 GPUs. Together AI has unveiled a cache-aware disaggregated inference architecture that boosts throughput by up to 40% for large language models handling long prompts\u2014a development that could […]<\/p>\n","protected":false},"author":1,"featured_media":554802,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[12],"tags":[20916,22044,2572,25,24145,19527],"class_list":{"0":"post-554801","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-blockchain","8":"tag-ai-infrastructure","9":"tag-llm-inference","10":"tag-machine-learning","11":"tag-news","12":"tag-nvidia-b200","13":"tag-together-ai"},"_links":{"self":[{"href":"https:\/\/e-bitco.in\/index.php\/wp-json\/wp\/v2\/posts\/554801","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/e-bitco.in\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/e-bitco.in\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/e-bitco.in\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/e-bitco.in\/index.php\/wp-json\/wp\/v2\/comments?post=554801"}],"version-history":[{"count":0,"href":"https:\/\/e-bitco.in\/index.php\/wp-json\/wp\/v2\/posts\/554801\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/e-bitco.in\/index.php\/wp-json\/wp\/v2\/media\/554802"}],"wp:attachment":[{"href":"https:\/\/e-bitco.in\/index.php\/wp-json\/wp\/v2\/media?parent=554801"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/e-bitco.in\/index.php\/wp-json\/wp\/v2\/categories?post=554801"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/e-bitco.in\/index.php\/wp-json\/wp\/v2\/tags?post=554801"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}