🤖 sippy load: add --prow-load-since and --skip-matview-refresh options

[sosiouxme]

These are helpful for retroactively loading job runs for new releases.

🤖 Assisted by Claude Code

Summary by CodeRabbit

New Features

• Added --prow-load-since flag to specify a custom start time for loading Prow data
• Added --skip-matview-refresh flag to optionally skip materialized view refresh operations

[openshift-ci-robot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

[coderabbitai]

Walkthrough

Implements time-bounded Prow data loading by adding a --prow-load-since CLI flag to specify custom data import start times and a --skip-matview-refresh flag to optionally skip materialized view refresh. Updates the Prow loader to accept and apply manual load-since timestamps, with fallback logic for existing behavior.

Changes

Cohort / File(s)	Summary
CLI flags and load logic cmd/sippy/load.go	Adds ProwLoadSince and SkipMatviewRefresh fields to LoadFlags struct. Introduces --prow-load-since and --skip-matview-refresh CLI flags. Adds helper function to parse load-since value from RFC3339 or duration strings. Conditionally skips materialized view refresh and propagates parsed loadSince through to Prow loader.
Prow loader initialization pkg/dataloader/prowloader/prow.go	Adds unexported loadSince *time.Time field to ProwLoader struct. Updates New constructor to accept loadSince parameter and assign it to the struct field.
BigQuery job loader logic pkg/dataloader/prowloader/bigqueryjobs.go	Adds conditional logic to use manual loadSince value when provided, bypassing DB query. Retains existing behavior when loadSince is nil. Applies -12 hour adjustment only when DB query succeeds with non-zero value; falls back to two-week default on failure or zero result.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Go Error Handling	✅ Passed	All modified files implement proper Go error handling patterns with correct error returns, nil checks before pointer dereferences, and appropriate error wrapping using %w format specifiers.
Sql Injection Prevention	✅ Passed	The PR safely implements the --prow-load-since option without SQL injection vulnerabilities through validated time.Time parsing and parameterized BigQuery binding.
Excessive Css In React Should Use Styles	✅ Passed	This custom check focusing on excessive inline CSS in React components is not applicable to this pull request which modifies Go backend source files.
Single Responsibility And Clear Naming	✅ Passed	New fields use action-oriented names (ProwLoadSince, SkipMatviewRefresh) without generic terms, and constructor signatures are focused and clear.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

📝 Coding Plan

• Generate coding plan for human review comments

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 golangci-lint (2.11.3)

Error: can't load config: unsupported version of the configuration: "" See https://golangci-lint.run/docs/product/migration-guide for migration instructions
The command is terminated due to an error: can't load config: unsupported version of the configuration: "" See https://golangci-lint.run/docs/product/migration-guide for migration instructions

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🧹 Nitpick comments (1)

cmd/sippy/load.go (1)

476-487: Consider validating that duration results in a past time.

If a user accidentally passes a negative duration like "-72h", the result would be a timestamp in the future, which could cause unexpected behavior. This is an edge case but worth noting.

🔧 Optional validation

 func parseProwLoadSince(val string) (time.Time, error) {
 	if t, err := time.Parse(time.RFC3339, val); err == nil {
 		return t, nil
 	}
 	d, err := time.ParseDuration(val)
 	if err != nil {
 		return time.Time{}, fmt.Errorf("must be an RFC3339 timestamp (e.g. 2024-01-15T00:00:00Z) or a duration (e.g. 72h)")
 	}
+	if d < 0 {
+		return time.Time{}, fmt.Errorf("duration must be positive (e.g. 72h for 72 hours ago)")
+	}
 	return time.Now().Add(-d), nil
 }

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@cmd/sippy/load.go` around lines 476 - 487, The parseProwLoadSince function
currently accepts negative durations which produce future timestamps; after
parsing the duration in parseProwLoadSince, validate that d is positive (or that
the computed time t := time.Now().Add(-d) is strictly before time.Now()) and
return a clear error if not (e.g., "duration must be positive and produce a time
in the past"); reference the function parseProwLoadSince and the parsed duration
variable d when adding this validation and error return.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@cmd/sippy/load.go`:
- Around line 476-487: The parseProwLoadSince function currently accepts
negative durations which produce future timestamps; after parsing the duration
in parseProwLoadSince, validate that d is positive (or that the computed time t
:= time.Now().Add(-d) is strictly before time.Now()) and return a clear error if
not (e.g., "duration must be positive and produce a time in the past");
reference the function parseProwLoadSince and the parsed duration variable d
when adding this validation and error return.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: 3084b989-7123-485c-a087-1adf87dc7341

📥 Commits

Reviewing files that changed from the base of the PR and between 0344658 and 73984cb.

📒 Files selected for processing (3)

• cmd/sippy/load.go
• pkg/dataloader/prowloader/bigqueryjobs.go
• pkg/dataloader/prowloader/prow.go

[openshift-ci-robot]

Scheduling required tests:
/test e2e

[stbenjam]

E2E failure: TestReleaseHealth — OOMKilled load pod, not caused by this PR 🤖

The TestReleaseHealth test failed with zero counts for all health categories because materialized views were never populated.

Root cause: The first load pod was OOMKilled (3G limit). The job retried and the second pod completed, but matviews remained empty:

sippy-load-job-86kk5   0/1     OOMKilled   0          8m56s
sippy-load-job-bxvx4   0/1     Completed   0          7m52s

This PR is not responsible. The new flags (--prow-load-since, --skip-matview-refresh) aren't used in the e2e, so loadSince is nil and SkipMatviewRefresh is false — the code follows the same paths as before. The same test passed on PR #3346 with the same base SHA. A /retest should resolve this.

Prow job | Analysis by Claude

[stbenjam]

/test e2e

[stbenjam]

/test e2e

[stbenjam]

Sorry for AI slop above, was seeing if openshift/release#76493 is useful

[stbenjam]

/lgtm
/hold

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sosiouxme, stbenjam

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [sosiouxme,stbenjam]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[stbenjam]

Something weird is going on, does this PR make us load more data somehow? We're OOMing a lot.

[sosiouxme]

This PR makes absolutely no difference to any code path currently in use. I just added the flags to enable backfilling new releases.

e2e job seems to have periods where it does this, e.g. over the weekend it was solid red

[stbenjam]

/test e2e

[stbenjam]

/hold cancel

[openshift-ci]

@sosiouxme: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.