grpc_server_handled_total metric issue

[tomas-stary]

Hi,
can someone help me? The grpc_server_handled_total metric returns errors as shown below. I also have an alert for this metric and it is of course constantly active.

However, I do not register any problem other than the active alert, which is strange.

I checked all logs of the thanos-sidecar and prometheus containers and found nothing.

Do you know what could be causing it?

Screenshot of Thanos:

[yeya24]

@tomas-stary Can you please give more info about your setup? What thanos version are you using?
I don't have any clue about it because this call should be rate ideally. 3.5m is a very large value. Can you check the thanos query http requests metrics to see is the query exemplars handler's rate?

[tomas-stary]

Hi, thank you very much for the reply. I am using the latest version of Thanos v0.24.0. I use deployment with sidecar. My setup looks the same as in the picture:

I'm not sure what metrics you meant. I am attaching a statement from the metric grpc_client_handled_total:
OK Series - 94.4 req / s
OK Info - 1.4 req / s
OK LabelNames - 0.0882 req / s
Ok Exemplars - 0.0882 req / s
Unavailable Series, Canceled Series, DeadlineExceeded Info and Aborted Series - 0 req / s

The second metric is grpc_client_handling_seconds_bucket:
I used this promql:
histogram_quantile (0.9999, sum (rate (grpc_client_handling_seconds_bucket {pod ="prometheus-thanos-querier-79cd6c9b8-nkw4l"} [1m])) by (grpc_method, pod))
99.99 Series - 8.76s
99.99 MetricMetadata - 600 ms
99.99 Info - 10 ms
99.99 Exemplars - 99.8 ms

Did you mean a specific metric?