Skip to content

feat: Split +Inf overflow alerts by label-set#5

Merged
vikin91 merged 7 commits intomainfrom
feat/inf-bucket-respects-labels
Mar 13, 2026
Merged

feat: Split +Inf overflow alerts by label-set#5
vikin91 merged 7 commits intomainfrom
feat/inf-bucket-respects-labels

Conversation

@vikin91
Copy link
Collaborator

@vikin91 vikin91 commented Mar 13, 2026

+Inf overflow alerts were hard to act on because they only reported the base histogram metric, not the specific label combinations causing the overflow. In practice, one problematic series was hidden among many label variants, so operators could not quickly identify the offending path.

For example, the issue in:

rox_central_k8s_event_processing_duration_count{Action="SYNC_RESOURCE",Dispatcher="unstructured.Unstructured",Resource="ComplianceOperatorRule"}

was being reported as an issue with

rox_central_k8s_event_processing_duration_count

however, the metric file contained over 1k lines for the metric rox_central_k8s_event_processing_duration_count and the particular set of labels was difficult to find.

We fixed this by evaluating histogram series per label-set and changing reporting behavior:

  • RED/YELLOW: emit one alert per problematic label-set, with labels included in the title:
    • <metric>{...labels...} (+Inf overflow check)
  • GREEN: keep output concise by emitting a single metric-level summary when all label-sets are healthy.
  • Add and render Prometheus HELP as Metric description before Message in markdown/console/TUI.

This makes overflow alerts actionable (exact offending series is visible) without adding noise when metrics are healthy.

Moreover, we will now try to guess the histogram unit based on the metric name and the help text.

vikin91 added 7 commits March 13, 2026 12:14
Emit one RED/YELLOW result per offending histogram series (labels in title),
keep one GREEN summary per metric, and render HELP as Metric description before Message for clearer triage.

AI-generated: evaluator/report/template/TUI/test updates.
User-provided/corrected: output contract (title format, section order,
green-collapsing rule).
@vikin91 vikin91 merged commit 438dc67 into main Mar 13, 2026
7 checks passed
@vikin91 vikin91 deleted the feat/inf-bucket-respects-labels branch March 13, 2026 13:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant