You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The specified endpoints will be used if the tenant is set to `foobar`. It is possible to use glob matching through the parameter `tenant_matcher_type`. It can have the value `glob`. In this case, the strings inside the array are taken as glob patterns and matched against the `tenant` inside of a remote-write request. For instance:
@@ -172,34 +160,34 @@ In order to ensure even spread for replication over nodes in different availabil
172
160
173
161
```json
174
162
[
175
-
{
176
-
"endpoints": [
177
-
{
178
-
"address": "127.0.0.1:10907",
179
-
"az": "A"
180
-
},
181
-
{
182
-
"address": "127.0.0.1:11907",
183
-
"az": "B"
184
-
},
185
-
{
186
-
"address": "127.0.0.1:12907",
187
-
"az": "C"
188
-
},
189
-
{
190
-
"address": "127.0.0.1:13907",
191
-
"az": "A"
192
-
},
193
-
{
194
-
"address": "127.0.0.1:14907",
195
-
"az": "B"
196
-
},
197
-
{
198
-
"address": "127.0.0.1:15907",
199
-
"az": "C"
200
-
}
201
-
]
202
-
}
163
+
{
164
+
"endpoints": [
165
+
{
166
+
"address": "127.0.0.1:10907",
167
+
"az": "A"
168
+
},
169
+
{
170
+
"address": "127.0.0.1:11907",
171
+
"az": "B"
172
+
},
173
+
{
174
+
"address": "127.0.0.1:12907",
175
+
"az": "C"
176
+
},
177
+
{
178
+
"address": "127.0.0.1:13907",
179
+
"az": "A"
180
+
},
181
+
{
182
+
"address": "127.0.0.1:14907",
183
+
"az": "B"
184
+
},
185
+
{
186
+
"address": "127.0.0.1:15907",
187
+
"az": "C"
188
+
}
189
+
]
190
+
}
203
191
]
204
192
```
205
193
@@ -282,7 +270,7 @@ These limits are applied per request and can be configured within the `request`
282
270
- `series_limit`: the maximum amount of series in a single remote write request.
283
271
- `samples_limit`: the maximum amount of samples in a single remote write request (summed from all series).
284
272
285
-
Any request above these limits will cause an 413 HTTP response (*Entity Too Large*) and should not be retried without modifications.
273
+
Any request above these limits will cause an 413 HTTP response (_Entity Too Large_) and should not be retried without modifications.
286
274
287
275
Currently a 413 HTTP response will cause data loss at the client, as none of them (Prometheus included) will break down 413 responses into smaller requests. The recommendation is to monitor these errors in the client and contact the owners of your Receive instance for more information on its configured limits.
288
276
@@ -296,6 +284,7 @@ By default, all these limits are disabled.
296
284
### Remote write request gates
297
285
298
286
The available request gates in Thanos Receive can be configured within the `global` key:
287
+
299
288
- `max_concurrency`: the maximum amount of remote write requests that will be concurrently worked on. Any request request that would exceed this limit will be accepted, but wait until the gate allows it to be processed.
300
289
301
290
## Active Series Limiting (experimental)
@@ -307,14 +296,17 @@ Every Receive Router/RouterIngestor node, queries meta-monitoring for active ser
307
296
To use the feature, one should specify the following limiting config options:
308
297
309
298
Under `global`:
299
+
310
300
- `meta_monitoring_url`: Specifies Prometheus Query API compatible meta-monitoring endpoint.
311
301
- `meta_monitoring_limit_query`: Option to specify PromQL query to execute against meta-monitoring. If not specified it is set to `sum(prometheus_tsdb_head_series) by (tenant)` by default.
312
302
- `meta_monitoring_http_client`: Optional YAML field specifying HTTP client config for meta-monitoring.
313
303
314
304
Under `default` and per `tenant`:
305
+
315
306
- `head_series_limit`: Specifies the total number of active (head) series for any tenant, across all replicas (including data replication), allowed by Thanos Receive. Set to 0 for unlimited.
316
307
317
308
NOTE:
309
+
318
310
- It is possible that Receive ingests more active series than the specified limit, as it relies on meta-monitoring, which may not have the latest data for current number of active series of a tenant at all times.
319
311
- Thanos Receive performs best-effort limiting. In case meta-monitoring is down/unreachable, Thanos Receive will not impose limits and only log errors for meta-monitoring being unreachable. Similarly to when one receiver cannot be scraped.
320
312
- Support for different limit configuration for different tenants is planned for the future.
@@ -331,7 +323,7 @@ Please see the metric `thanos_receive_forward_delay_seconds` to see if you need
331
323
332
324
The following formula is used for calculating quorum:
0 commit comments