Skip to content

🌱 E2E serial logs per test #2911

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
lentzi90 wants to merge 4 commits into
base: main
Choose a base branch
from
Open

🌱 E2E serial logs per test #2911

lentzi90 wants to merge 4 commits into from

Conversation

@lentzi90
Copy link
Member

@lentzi90 lentzi90 commented Jan 15, 2026

What this PR does / why we need it:

We collect serial logs from the VMs, but these are reused for all the tests so it is hard to tell which part of the logs belong to what test. This PR aims to fix that by extracting the part of the logs that were created during the test, for each test. We still collect the complete logs at the end.

Fixes #

Checklist:

  • Documentation has been updated, if necessary.
  • Unit tests have been added, if necessary.
  • E2E tests have been added, if necessary.
  • Integration tests have been added, if necessary.

This is a draft for now. It is meant as a follow up to #2905.

@metal3-io-bot
Copy link
Contributor

metal3-io-bot commented Jan 15, 2026

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@metal3-io-bot metal3-io-bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 15, 2026
@metal3-io-bot
Copy link
Contributor

metal3-io-bot commented Jan 15, 2026

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign honza for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@metal3-io-bot metal3-io-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jan 15, 2026
@lentzi90
Copy link
Member Author

lentzi90 commented Jan 15, 2026

/test metal3-bmo-e2e-test-optional-pull

@lentzi90 lentzi90 force-pushed the lentzi90/e2e-serial-logs branch from c1d9e74 to 907f681 Compare January 15, 2026 10:57
@lentzi90
Copy link
Member Author

lentzi90 commented Jan 15, 2026

/test metal3-bmo-e2e-test-optional-pull

@metal3-io-bot metal3-io-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jan 15, 2026
@lentzi90 lentzi90 force-pushed the lentzi90/e2e-serial-logs branch from 907f681 to f686aa1 Compare January 15, 2026 11:21
@lentzi90
Copy link
Member Author

lentzi90 commented Jan 15, 2026

/test metal3-bmo-e2e-test-optional-pull

@lentzi90 lentzi90 force-pushed the lentzi90/e2e-serial-logs branch from f686aa1 to c8321b4 Compare January 15, 2026 12:50
@lentzi90
Copy link
Member Author

lentzi90 commented Jan 15, 2026

/test metal3-bmo-e2e-test-optional-pull

@lentzi90 lentzi90 force-pushed the lentzi90/e2e-serial-logs branch from c8321b4 to eea9d30 Compare January 15, 2026 13:13
@lentzi90
Copy link
Member Author

lentzi90 commented Jan 15, 2026

/test metal3-bmo-e2e-test-optional-pull

@lentzi90 lentzi90 force-pushed the lentzi90/e2e-serial-logs branch from eea9d30 to 3f72a37 Compare January 15, 2026 13:59
@lentzi90
Copy link
Member Author

lentzi90 commented Jan 15, 2026

/test metal3-bmo-e2e-test-optional-pull

@lentzi90 lentzi90 force-pushed the lentzi90/e2e-serial-logs branch from 3f72a37 to 01defea Compare January 16, 2026 06:43
@lentzi90
Copy link
Member Author

lentzi90 commented Jan 16, 2026

/test metal3-bmo-e2e-test-optional-pull

1 similar comment
@lentzi90
Copy link
Member Author

lentzi90 commented Jan 16, 2026

/test metal3-bmo-e2e-test-optional-pull

@lentzi90 lentzi90 force-pushed the lentzi90/e2e-serial-logs branch from a5d69f4 to d82632f Compare January 16, 2026 10:16
@lentzi90
Copy link
Member Author

lentzi90 commented Jan 16, 2026

/test metal3-bmo-e2e-test-optional-pull

@lentzi90
Copy link
Member Author

lentzi90 commented Jan 16, 2026

Deduplication did not work well so that has been removed here now. I have also excluded the broken test (ironic 29.0 -> latest with BMO 0.10).
There are still some flaky behavior so I am not sure if this will pass, but I have high hopes.

@lentzi90 lentzi90 force-pushed the lentzi90/e2e-serial-logs branch from d82632f to d32a38d Compare January 16, 2026 10:40
@lentzi90
Copy link
Member Author

lentzi90 commented Jan 16, 2026

/test metal3-bmo-e2e-test-optional-pull

@lentzi90
Copy link
Member Author

lentzi90 commented Jan 16, 2026

Ok, finally redfish-virtualmedia upgrade jobs are green. IPMI is failing randomly. I'll start cleaning this up and splitting into separate PRs.
The fixes that should make the periodics green have been pushed to #2910. I am planning to extract some log improvements into #2905 and then keep the final piece here.

This was referenced Jan 16, 2026
Merged
Merged
@lentzi90 lentzi90 force-pushed the lentzi90/e2e-serial-logs branch from d32a38d to 4d98445 Compare January 22, 2026 09:39
@metal3-io-bot metal3-io-bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jan 22, 2026
@lentzi90 lentzi90 force-pushed the lentzi90/e2e-serial-logs branch from 4d98445 to 0abfad7 Compare January 23, 2026 07:48
@lentzi90
Copy link
Member Author

lentzi90 commented Jan 23, 2026

/test metal3-bmo-e2e-test-optional-pull

@lentzi90 lentzi90 marked this pull request as ready for review January 23, 2026 11:24
@metal3-io-bot metal3-io-bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jan 23, 2026
@lentzi90
Copy link
Member Author

lentzi90 commented Jan 29, 2026

/hold
Testing a few things...

@metal3-io-bot metal3-io-bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 29, 2026
@lentzi90 lentzi90 force-pushed the lentzi90/e2e-serial-logs branch from 0abfad7 to 1589cb9 Compare January 29, 2026 07:49
@lentzi90
E2E: Use per-test cluster for upgrades
a23442d 
Avoid cleanup issues by using a fresh cluster for each upgrade test.
We create a separate kind cluster for each upgrade test. These "upgrade
clusters" use different port mappings than the normal kind cluster in
order to be able to co-exist without issues. However, they all use the
same set of ports, so the upgrade tests cannot run in parallel with each
other.

Signed-off-by: Lennart Jern <[email protected]>
@lentzi90
E2E: Serial logs per test
8ae20ec 
Signed-off-by: Lennart Jern <[email protected]>
@lentzi90
WIP: dnsmasq ready?
94b3017 
Signed-off-by: Lennart Jern <[email protected]>
@lentzi90 lentzi90 force-pushed the lentzi90/e2e-serial-logs branch from 1589cb9 to ae3fc37 Compare January 29, 2026 13:19
@metal3-io-bot metal3-io-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jan 29, 2026
@lentzi90
Copy link
Member Author

lentzi90 commented Jan 29, 2026

/test metal3-bmo-e2e-test-optional-pull

@lentzi90
Improve error handling in E2E artifact dump helpers
fda3da0 
Replace Gomega Expect calls with graceful error logging in dumpObj,
dumpCRDS, and dumpIronicNodes to prevent test interruption and improve
diagnostics during artifact collection.

Signed-off-by: Lennart Jern <[email protected]>
@lentzi90 lentzi90 force-pushed the lentzi90/e2e-serial-logs branch from ae3fc37 to fda3da0 Compare January 29, 2026 13:35
@lentzi90
Copy link
Member Author

lentzi90 commented Jan 29, 2026

/test metal3-bmo-e2e-test-optional-pull

@metal3-io-bot
Copy link
Contributor

metal3-io-bot commented Jan 29, 2026

@lentzi90: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
metal3-bmo-e2e-test-optional-pull fda3da0 link false /test metal3-bmo-e2e-test-optional-pull
Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@metal3-io-bot
Copy link
Contributor

metal3-io-bot commented Jan 29, 2026

@lentzi90: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
metal3-bmo-e2e-test-optional-pull fda3da0 link false /test metal3-bmo-e2e-test-optional-pull

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@honza honza Awaiting requested review from honza

@kashifest kashifest Awaiting requested review from kashifest

@Rozzii Rozzii Awaiting requested review from Rozzii

@zhouhao3 zhouhao3 Awaiting requested review from zhouhao3

Assignees

No one assigned

Labels

do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@lentzi90 @metal3-io-bot