[Bug] PD disaggregation Abort Issue

[whybeyoung]

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
• 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 5. Please use English, otherwise it will be closed.

Describe the bug

When I conducted a stability stress test with 400 concurrent requests of varying lengths, the unbootstrap queue in the prefill stage kept piling up. Additionally, the token_usage on the decode side approached 1. Due to the timeout mechanism in our link (e.g., automatically disconnecting when TTFT exceeds 60s), the business layer actively terminates the connection with the sglang HTTP server. However, it appears that the decode side fails to release resources properly, leading to the accumulation of unbootstrap tasks in the prefill stage and eventually causing the decode side to hang.

Decode

Prefill

Reproduction

Look Above

Environment

H20
Prefill 2-node
Decode 4-node

[mugglew]

@whybeyoung What's the model you're deploying btw?

[whybeyoung]

@whybeyoung What's the model you're deploying btw?

dsv3

[whybeyoung]

After Debugging , founding that waiting_queue is empty in this case. so it look likes that we should clear the req in the disagg_decode_prealloc_queue and disagg_decode_transfer_queue correctly.
@ShangmingCai

[datdo-msft]

@ShangmingCai @whybeyoung , do you have any insight into this? Are resources not actually being cleaned up when clients drop the requests?

[whybeyoung]

#8352 fix this , will close this soon.
BTW, can you help test this pr @datdo-msft

[whybeyoung]

@ShangmingCai @whybeyoung , do you have any insight into this? Are resources not actually being cleaned up when clients drop the requests?

yes, it happens on the decode node.

[datdo-msft]

@whybeyoung @ShangmingCai thanks for the quick fix! I was able to give this a test and so far have not seen this issue yet, will give it another day or two to monitor and see what happens. One thing I did notice though was that for requests which get aborted, it seems we're still returning 200? Shouldn't we be returning 500 instead?

[whybeyoung]

@whybeyoung @ShangmingCai thanks for the quick fix! I was able to give this a test and so far have not seen this issue yet, will give it another day or two to monitor and see what happens. One thing I did notice though was that for requests which get aborted, it seems we're still returning 200? Shouldn't we be returning 500 instead?

Good question. Error handling indeed needs to be redesigned at the system level. mayben we can draft another pr inluding this fix

[datdo-msft]

@whybeyoung should we track this in the roadmap here? #8210

cc @ShangmingCai