receive: remote write handler hangs when using "split-tenant-label-name"

[rvichery]

Thanos, Prometheus and Golang version used:

thanos, version 0.36.0 (branch: HEAD, revision: cfff5518d37756715a1cee43c42b21f4afccfbf2)
  build user:       root@317f3d9783e7
  build date:       20240731-15:23:35
  go version:       go1.21.12
  platform:         linux/amd64
  tags:             netgo

Object Storage Provider: None

What happened:
When using a single Thanos receive pod with --receive.split-tenant-label-name=<my tenant label> I noticed that all remote write calls coming from Prometheus server were timing out after N seconds when the number of tenant label values was greater than 1. Single tenant using split-tenant-label-name was actually not timing out.

What you expected to happen:
I expected to see the incoming remote write call being processed and split into several TSDB.

How to reproduce it (as minimally and precisely as possible):
Start a single instance of Thanos receive in RouterIngestor mode and with --receive.split-tenant-label-name=<my tenant label>. Send remote write API calls using a Prometheus server and have timeseries with at least two different tenant label values.

I have identified the issue in the original #7256 for this feature and have implemented a fix and unit test for this case.

[dosubot]

Hello @rvichery! I'm here to help you with any bugs, questions, or contributions you have. Let's work together to resolve the issue you're facing.

The issue you're experiencing with remote write calls timing out when using the --receive.split-tenant-label-name feature with multiple tenant label values is related to the changes introduced in GitHub pull request #7256. This pull request added the functionality to split remote-write HTTP requests based on tenant labels, allowing the handler to determine the tenant dynamically and remove the label from the series before processing ^[1][2].

Since you've identified the issue and implemented a fix along with a unit test, it would be beneficial to review the changes made in the pull request to ensure your fix aligns with the intended functionality. If your fix resolves the timeout issue, it might be worth considering contributing it back to the Thanos project to help others facing similar issues.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}