Issue #237 parse flakyFailure and rerunFailure from surefire report files #765

olamy · 2025-11-17T06:43:12Z

Part of #237

Support <flakyFailure> and <flakyError> in junit xml report #237

Testing done

Submitter checklist

Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
Ensure that the pull request title represents the desired changelog entry
Please describe what you did
Link to relevant issues in GitHub or Jira
Link to relevant pull requests, esp. upstream and downstream changes
Ensure you have provided tests that demonstrate the feature works or the issue is fixed

Signed-off-by: Olivier Lamy <[email protected]>

olamy · 2025-11-17T06:46:11Z

I have only added the datas as I'm not sure how to do this with the UI.
Well TBH my first intention is to have those datas available for the mcp server plugin :)

timja · 2025-11-17T09:42:06Z

src/main/java/hudson/tasks/junit/CaseResult.java

    private String errorStackTrace;
    private String errorDetails;
    private Map<String, String> properties;
+    private FlakyFailure[] flakyFailures;


any reason these are an array rather than a list?

for some reasons, using List is creating some empty elements in xml and fail TestResultTest#bigResultReadWrite

Because you are using records, I think. XStream does not support them well. x-stream/xstream#345 If you switch to final classes (yes it is irritating) then you should be able to make this a List<FlakyFailure> and XStream should produce (semi-)attractive and functional XML.

Just be sure the concrete type is ArrayList! Do not try to use List.of(), Collections.unmodifiableList, etc. in the field itself. You can always clone lists in a getter and/or setter or use an unmodifiable collection in a getter.

The problem was more about returning an empty liist which create an empty element in the xml and so was breaking the test TestResultTest#bigResultReadWrite
Now by using List but returning null instead of empty list the test is passing.

OK, just check the format carefully to make sure the records work here.

unit test is reading correct values from parsed files.
So I suppose it's fine.

Yeah just double-check the XML format on disk by inspection.

Signed-off-by: Olivier Lamy <[email protected]>

src/main/java/hudson/tasks/junit/CaseResult.java

Signed-off-by: Olivier Lamy <[email protected]>

timja

I think fine as a start, not too useful for most people without the UI but that can be added on top, what do others think?

uhafner · 2025-11-18T08:42:37Z

src/main/java/hudson/tasks/junit/CaseResult.java

        this.parent = parent;
    }

+    public @CheckForNull List<FlakyFailure> getFlakyFailures() {


The @CheckForNull is not required now.

I wonder, why the field itself still is nullable? All those ugly if (x != null) guards could be eliminated by working with an empty collection. You can add a readResolve method to ensure that the field is set in old serializations.

because this

[ERROR] Tests run: 18, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.579 s <<< FAILURE! -- in hudson.tasks.junit.TestResultTest [ERROR] hudson.tasks.junit.TestResultTest.bigResultReadWrite -- Time elapsed: 0.298 s <<< FAILURE! org.opentest4j.AssertionFailedError: Forgot to implement XML parsing for something? ==> expected: <true> but was: <false> at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:158) at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:139) at org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:69) at org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:41) at org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:228) at hudson.tasks.junit.TestResultTest.bigResultReadWrite(TestResultTest.java:464) [INFO] [INFO] Results: [INFO] [ERROR] Failures: [ERROR] TestResultTest.bigResultReadWrite:464 Forgot to implement XML parsing for something? ==> expected: <true> but was: <false>

so it looks parsing files needs to be done with XMLStreamReader as well,
Hopefully I don't have to do with SAX.. (at least I hope :))

Interesting test, it is totally unclear what the test is doing. Would be helpful when the variables would use meaningful names...

Without using XmlUnit it is hard too see what is wrong in the document.

olamy · 2025-11-18T09:33:36Z

back to draft to implement all possible xml parsers :)

timja · 2025-11-19T16:05:05Z

FYI there seems to be some way to show flakiness as this screenshot has it: #766

papajulio · 2025-11-19T16:08:14Z

@timja that's this plugin https://plugins.jenkins.io/test-stability/

Signed-off-by: Olivier Lamy <[email protected]>

olamy · 2025-11-21T09:11:55Z

FYI there seems to be some way to show flakiness as this screenshot has it: #766

this plugin is not taking into account the fact that surefire may eventually rerun test multiple times before getting a success (which means it's flaky :))

uhafner

I'm not a maintainer of the plugin so I don't want to create too much overhead for you.

So only a non-binding comment (as another Jenkins plugin developer):
I am wondering why the same code is used for flakyFailures and rerunFailures. Even the entities are identical. Wouldn't it simplify everything when the values are stored in an entity Failure? One is created for the flakyFailures element and one for the rerunFailures element?

Signed-off-by: Olivier Lamy <[email protected]>

olamy · 2025-11-22T01:35:54Z

@uhafner yes good point. I have refactored a bit to simplify. back to record and use a single entity.

olamy · 2025-11-22T01:38:11Z

src/test/java/hudson/tasks/junit/TestResultTest.java

                        .getProperties()
                        .size());

+        Diff fileDiff = DiffBuilder.compare(f.getFile())


well we can keep to use only the simple String diff option.
but this one gives more details when failing

olamy · 2025-11-22T02:07:22Z

@timja back to review and ready

timja

LGTM

olamy added 2 commits November 17, 2025 16:25

fix this test

710aafd

Signed-off-by: Olivier Lamy <[email protected]>

Issue jenkinsci#237 parse flakyFailure and rerunFailure

d0887a5

Signed-off-by: Olivier Lamy <[email protected]>

olamy requested a review from a team as a code owner November 17, 2025 06:43

timja reviewed Nov 17, 2025

View reviewed changes

olamy added 2 commits November 18, 2025 07:55

use List but returns null rather than empty list

2705286

Signed-off-by: Olivier Lamy <[email protected]>

Collections.unmodifiableList only for getters

b3566a3

Signed-off-by: Olivier Lamy <[email protected]>

uhafner reviewed Nov 17, 2025

View reviewed changes

src/main/java/hudson/tasks/junit/CaseResult.java Outdated Show resolved Hide resolved

do not return null

a5c266c

Signed-off-by: Olivier Lamy <[email protected]>

olamy requested review from jglick, timja and uhafner November 18, 2025 00:17

timja approved these changes Nov 18, 2025

View reviewed changes

uhafner reviewed Nov 18, 2025

View reviewed changes

olamy marked this pull request as draft November 18, 2025 09:33

fix some missing parsing

89dbd72

Signed-off-by: Olivier Lamy <[email protected]>

olamy marked this pull request as ready for review November 21, 2025 09:06

remove commented code

dca551b

Signed-off-by: Olivier Lamy <[email protected]>

olamy requested review from timja and uhafner November 21, 2025 12:13

uhafner reviewed Nov 21, 2025

View reviewed changes

olamy added 2 commits November 22, 2025 10:45

back to record as this was not the problem

0a33e5e

Signed-off-by: Olivier Lamy <[email protected]>

simplify code

f58164a

Signed-off-by: Olivier Lamy <[email protected]>

olamy commented Nov 22, 2025

View reviewed changes

olamy requested a review from uhafner November 22, 2025 01:38

timja added the enhancement label Nov 22, 2025

timja approved these changes Nov 22, 2025

View reviewed changes

timja merged commit 491ff05 into jenkinsci:master Nov 23, 2025
17 checks passed

Issue #237 parse flakyFailure and rerunFailure from surefire report files #765

Issue #237 parse flakyFailure and rerunFailure from surefire report files #765

Uh oh!

Conversation

olamy commented Nov 17, 2025 • edited by timja Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing done

Submitter checklist

Uh oh!

olamy commented Nov 17, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

timja left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

olamy commented Nov 18, 2025

Uh oh!

timja commented Nov 19, 2025

Uh oh!

papajulio commented Nov 19, 2025

Uh oh!

olamy commented Nov 21, 2025

Uh oh!

uhafner left a comment

Choose a reason for hiding this comment

Uh oh!

olamy commented Nov 22, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

olamy commented Nov 22, 2025

Uh oh!

timja left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

olamy commented Nov 17, 2025 •

edited by timja

Loading