2026-11-17 10:50
Introduction: Avoiding KV Cache Migration, Attention Graph Reduces TTFT by 15%–20%. What is MapFlow? MapFlow is developed by PKU-DAIR team from Peking University.
What is MapFlow?
MapFlow is a cross-model LLM inference acceleration system introduced by the PKU-DAIR team at Peking University. When an Agent transitions between multiple models and shares long context, MapFlow avoids recomputing prefill and bypasses tightly-coupled KV Cache. Instead, it reuses more transferable "attention graphs." Under shared tokenizers, the receiving model recalculates only the initial layers, while subsequent layers leverage lightweight layer/head linear mappings to reuse token dependencies from the sending model. It employs block-sparse top-p compression, prefix hashing cache, and fused GPU kernels to manage memory footprint and latency. Experiments show a 15%–20% reduction in TTFT and up to 25% higher throughput in multi-round routing scenarios.

Key Features of MapFlow
- Cross-Model Attention Reuse: Transfers and reuses attention graphs across models of different scales and architectures, replacing hard-to-migrate KV Cache.
- Partial Recomputation: The receiving model computes its initial layers normally to establish internal hidden states before entering the reuse path, balancing accuracy and speed.
- Layer and Head Mapping: Aligns layers between sender and receiver models using learned linear weights to combine sender’s attention heads into receiver’s heads.
- Attention Graph Sparse Compression: Applies block-level top-p filtering based on cumulative attention probability, compressing dense attention matrices into block-sparse format to reduce storage overhead.
- Cache Management and Retrieval: Uses prefix hash indexing for sparse blocks, overlapping cache retrieval with early-layer recomputation to hide transmission latency.
- Fused GPU Kernel Execution: Combines cross-head linear transformation with Value multiplication and accumulation into a single kernel, avoiding materialization of dense intermediate tensors.
- Online Inference Acceleration: Lowers prefill latency and TTFT while maintaining generation quality, improving system throughput in multi-round routing scenarios.
Technical Principles of MapFlow
- Attention Graph as Reusable State: The bottleneck in large model prefill stems from attention computation growing quadratically with sequence length. Traditional prefix caching requires identical models and weights, making KV Cache unusable across models due to strong coupling with hidden dimensions, layers, and parameters. MapFlow’s key insight is that attention graphs capture token dependency structures—these structures exhibit stronger cross-model similarity than model-specific KV representations when sender and receiver share tokenizers and process identical contexts.
- Partial Recomputation and Linear Mapping: Attention graph transferability does not hold uniformly across all layers. Empirically, the receiving model’s initial layers have high fitting error, while middle and late layers can be approximated via linear mapping from the sender’s attention graph. MapFlow designs a three-tier mechanism: the receiver first fully recomputes the first r layers; layer mapping uses suffix alignment m(j)=j+L_S−L_R to align receiver layers with the sender’s more stable deep layers; head mapping applies simplex-constrained linear combinations to weight-average sender heads into each receiver head, preserving attention distribution structure. Mapping weights are trained offline via end-to-end response loss, with both large models frozen throughout, adding less than 0.001% of the receiver model’s parameters—calibration is performed once and amortized across all subsequent online requests.
- Block-Level Sparse Compression: Full attention matrices grow quadratically with context length, making direct caching infeasible. MapFlow leverages the heavy-tailed nature of attention distributions, applying top-p filtering at 64×64 block granularity: first, LogSumExp computes block-level importance scores for query-key block pairs; then, blocks are retained row-wise until cumulative probability reaches threshold p, forming a sparse mask; finally, only retained blocks undergo precise attention scoring, stored together with BSR-formatted indices in contiguous memory. This process avoids materializing full dense matrices, performs compression only once at the sender, and enables sharing of compressed state across multiple downstream receivers.

Follow WeChat and reply "open source" to join the AI Open Source Project Community
How to Use MapFlow
- Environment Setup: Create a Python 3.11 conda environment, install the vLLM fork and MapFlow from the repository in editable mode, then install dependencies such as nvtx, sacrebleu, and rouge_score.
- Launch MPS Service: Run
nvidia-cuda-mps-control -d to start NVIDIA MPS, preparing for GPU sharing among sender, receiver, and cache server.
- Start Cache Server: Execute the server script under
examples/bash/ (e.g., server_top_p.sh specifying model pairs and top-p), responsible for attention graph compression, prefix hash indexing, and block pool management.
- Start Sender: Run the sender script (e.g.,
sender_top_p.sh Qwen3-8B 0.7 0) in a separate terminal, letting the large model perform prefill on shared context and store generated attention graphs into cache.
- Start Receiver: Run the receiver script (e.g.,
receiver_top_p.sh Qwen3-1.7B 0.7 1 ./examples/weights/Qwen3-8B_to_Qwen3-1.7B/hotpotqa) in another terminal, loading pre-calibrated mapping weights and taking the reuse path upon prefix hash hit.
- Run Evaluation: Execute accuracy (
accuracy/hotpotqa.sh), TTFT (ttft/hotpotqa.sh), end-to-end throughput (e2e/multi_rounds_window_correct_log.sh), tri-model workflow, or RULER-20K long-context scripts as needed, and use tools like extract_ttft.py to parse log results.
- Tune Parameters by Scenario: Use default top-p=0.7 for short contexts; increase for long contexts (0.85–0.92) per README parameter table; modify trace file path in e2e scripts when changing routing thresholds.
Core Advantages of MapFlow
- Cross-Model Context Reuse: Breaks the traditional limitation of prefix caching restricted to identical models, enabling different-scale and architecture models to share computed token dependencies.
- Smarter State Selection: Replaces model-weight-coupled KV Cache with attention graphs—attention structures exhibit greater cross-model transferability than hidden representations.
- Minimal Overhead: Only trains simplex-constrained linear mapping weights; both large models remain frozen throughout, calibration done once for long-term benefit.
- Balances Quality and Speed: Partial recomputation preserves accuracy, linear mapping preserves structure—experiments show 15%–20% lower prefill latency without degradation in generation quality.
- Memory Controllable: Block-level top-p sparse compression reduces O(L²) attention graphs to block-sparse format; memory usage stabilizes beyond 12K tokens, enabling practical use in long-context scenarios.
- Engineering Efficiency: Fused GPU kernels complete cross-head weighting and Value multiply-add on-chip, avoiding dense intermediate tensor materialization; cache retrieval overlaps with early-layer recomputation, hiding transmission latency.
- Real System Gains: In multi-round routing, throughput increases up to 25% under same P90 latency constraint, primarily reducing tail TTFT for more stable online experience.
Project Links
- GitHub Repository: https://github.com/PKU-DAIR/MapFlow
- arXiv Technical Paper: https://arxiv.org/pdf/2610.14688
Competitive Comparison
| Comparison Dimension |
MapFlow (PKU-DAIR) |
RedKnot (Xiaohongshu / PKU / Huawei Cloud) |
| Problem Addressed |
Repeated prefill after model switching in multi-model routing or Agent workflows with long context |
Repeated prefill when same document segment appears non-prefix positions across requests in long-context RAG |
| Reuse Scope |
Cross-model (different layer counts, head counts, dimensions) |
Within-model, cross-request (position-agnostic caching, PIC) |
| Reusable State |
Attention Graph (token dependency structure) |
KV Cache (but managed decoupled by head dimension) |
| Core Mechanism |
Partial recomputation + layer/head linear mapping + block-sparse top-p compression + fused retrieval kernel |
Offline head classification (global head recompute/local head reuse) + RoPE position alignment + SegPagedAttention per-head paging + sparse FFN/MoE |
| Requires Training |
Yes: Offline calibration of linear mapping weights (< 0.001% of receiver model params; both LLMs frozen) |
No training: Needle-in-a-haystack offline head classification, zero overhead online |
| Quality Assurance |
Initial r layers recomputed for accuracy; attention graph structure transferable across models |
Head-level recovery superior to token-level; EM score outperforms CacheBlend/ProphetKV (e.g., Llama-3.3-70B HotpotQA EM 0.60→0.80) |
| Latency/CPU Gain |
TTFT reduced by 15%–20%, multi-round routing throughput up to +25% |
Prefill computation reduced by 3.9–4.3× (16K–24K), QPS/GPU ~4× higher than full recomputation at 64K |
| Inference Framework |
Integrated with vLLM (Triton fused kernels) |
Initially built on SGLang, now default branch shifts to DeepInfer; vLLM integration underway |
| Open Source Status |
GitHub open-source, includes pre-calibrated weights for four model pairs |
GitHub open-source (Apache-2.0); DeepSeek-V4-Flash has one-click repro config; others are experimental entry points |
| Applicable Scenarios |
Multi-model routing, cascade inference, multi-Agent workflows (database assistant, text-to-SQL, etc., prefix-intensive workloads) |
RAG services with repeated queries on long documents, ultra-long context (>128K) high-concurrency scenarios |
| Primary Limitations |
Requires shared tokenizer; gains limited by proportion of prefill in end-to-end latency |
Only effective within same model; does not solve cross-model handoff; limited gains on short contexts; some integrations still experimental |
Application Scenarios of MapFlow
- Multi-Model Routing Assistant: In systems routing complex queries to large models and simple ones to small models, the small model reuses the attention graph from the large model when inheriting long conversation history, avoiding redundant prefill.
- Cascade Inference: In cascade architectures where a large model handles initial processing and upgrades to a smaller one if dissatisfied, models share context computation results during handoff.
- Multi-Agent Workflows: In pipelines like “14B planning → 8B refinement → 3B summarization,” downstream models share attention states generated by upstream large models, saving 0.48–0.73 seconds per round.
- Database & Text-to-SQL Assistants: In scenarios repeatedly carrying the same database schema and system constraints as long prefixes, model switching avoids recalculation and directly hits cached results.
Source: AI Tools Collection