[RFC]: Graceful Error Handling for KV Connector Load Failures

[sdavidbd]

Motivation.

🧩 Summary

This RFC proposes adding fault recovery support to the KV connector infrastructure introduced in #15960. The goal is to make vLLM resilient to KV block load failures (e.g., due to eviction or network disconnections) by detecting failures and rescheduling only affected requests for recomputation.

🧠 Background

vLLM recently introduced the KV connector abstraction, providing a pluggable interface for integrating external solutions for KV cache offload and transfer. This enables supporting use cases like KV cache sharing and prefill-decode (P-D) disaggregation.
Notable examples include the LMCache connector (PR #16625) and NIXL connector (PR #17751).

The KV loading protocol involves:

• start_load_kv() → invoked before the forward pass to initiate (asynchronous) loading of all required KV blocks.
• wait_for_layer_load() → invoked before each layer’s attention kernel to ensure the relevant blocks are available.

⚠️ Problem

Currently, vLLM lacks a mechanism to detect or recover from KV load failures. If one or more KV blocks fail to load, it may lead to system instability (crash or hang) or incorrect outputs (silent corruption of the KV cache). This undermines reliability, especially in large-scale or disaggregated deployments.

---

Proposed Change.

Even if KV load failures occur, the forward pass should complete. Post-execution, vLLM will:

• Detect failed KV blocks, and
• Reschedule only the affected requests for recomputation, using the longest valid prefix.

💡 Key Considerations:

• Preserve progress for valid blocks in affected requests and for unaffected requests in the same batch.
• Prevent deadlocks in tensor-parallel setups (e.g., all-reduce ops).
• In some connector implementations attention kernels may have already been launched and cannot be "rolled back."

🛠️ Suggested Implementation

1. KV Connector API Extension

   • Add get_block_ids_with_load_errors() to KVConnectorBase_V1 to allow connectors to report failed block IDs.

2. Executor Reporting

   • GPUModelRunner retrieves the failed block IDs and includes them in ModelRunnerOutput (should be aggregated across all workers in TP setups).

3. Scheduler Recovery Logic

   • Detects affected requests that used failed blocks

   • Truncates the number of computed blocks for affected requests to their longest valid prefix

   • Reschedules affected requests for recomputation

---

Feedback Period.

No response

CC List.

No response

Any Other Things.

🔮 Future Enhancements

• Skip attention computations that depend on failed KV blocks (early detection)
• Make recovery behavior configurable (e.g., allow request failure instead of rescheduling for use cases like disaggregated P-D)

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[sdavidbd]

@robertgshaw2-redhat @KuntaiDu - would appreciate your thoughts when you get a chance!

[KuntaiDu]

Currently, vLLM lacks a mechanism to detect or recover from KV load failures. If one or more KV blocks fail to load, it may lead to system instability (crash or hang) or incorrect outputs (silent corruption of the KV cache). This undermines reliability, especially in large-scale or disaggregated deployments.

Just want to make sure I understand this correctly --- do you mean that the connector may fail to load the KV caches to GPU? If that's the case, will there be a detectable error raised by connector?

[sdavidbd]

Currently, vLLM lacks a mechanism to detect or recover from KV load failures. If one or more KV blocks fail to load, it may lead to system instability (crash or hang) or incorrect outputs (silent corruption of the KV cache). This undermines reliability, especially in large-scale or disaggregated deployments.

Just want to make sure I understand this correctly --- do you mean that the connector may fail to load the KV caches to GPU? If that's the case, will there be a detectable error raised by connector?

@KuntaiDu Yes, the connector may fail to load KV caches to GPU. In such cases, it should not raise an error and must allow the forward pass to complete normally. Importantly, the connector should detect these failures in a timely manner to prevent the forward pass from timing out. After the forward pass completes, we use the new get_block_ids_with_load_errors API to check whether any blocks failed to load.

[maobaolong]

We are using LMCache + remote Connector, and there are two cases that need to be covered. In both cases, when get_num_new_matched_tokens is called, everything works normally and returns the number x of hit tokens. However, when start_load_kv is called:

1. It gets stuck for a long time due to network issues or other conditions.
2. The start_load_kv fails to execute completely because the remote backend service of LMCache is abnormal or down. As a result, the GPU block of vLLM does not load x hit tokens from LMCache, which leads to accuracy degradation.

Possible solutions:

1. vLLM should set an expected time threshold for operations on the Connector. If the timeout threshold is exceeded, fallback to recompute or rescheduling.
2. After get_num_new_matched_tokens returns a number x, if start_load_kv encounters an exception, the lmcache Connector should throw an error or return loaded_tokens/loaded_blocks, this allows subsequent tokens to be recomputed or even rescheduled.

[sdavidbd]

Thank you @maobaolong, and apologies for the delay in responding.

What you described are exactly the problems this RFC is intended to address. Regarding your suggested solutions:

1. Setting a timeout threshold at the vLLM level would compromise completion of the forward step, impacting all scheduled requests in the batch and potentially forcing recomputation of all externally computed tokens. Instead, the approach proposed here requires that the connector guarantee that load/store operations from the external cache never hang. In case of a timeout or failure, the connector should internally track the failed blocks and report them back after the forward pass completes. The forward pass must always complete, as emphasized in the "Key Considerations" section above, to avoid deadlocks and preserve progress.
2. This aligns well with my suggested approach. While you proposed returning the loaded blocks, my approach focuses on reporting the failed blocks instead. Please see my PR: [V1] [P/D] Add Support for KV Load Failure Recovery #19330. It currently includes a toy example (based on disaggregated-prefill-v1) demonstrating successful recovery in a simple 1P1D setup when a load failure occurs on the decoder node. I’m also working on adding unit tests to improve coverage and reliability.

Will be glad to hear your thoughts!

[maobaolong]

@sdavidbd Thanks for your reply and explain.

I still don't know how to get what block ids occurred load error while invoke get_block_ids_with_load_errors method, if the start_load_kv crash, how lmcache connector or vllm know the invalid block ids?

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[markmc]

#19330 has been merged