Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions src/main/java/hudson/tasks/junit/CaseResult.java
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,8 @@
private String errorStackTrace;
private String errorDetails;
private Map<String, String> properties;
private FlakyFailure[] flakyFailures;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any reason these are an array rather than a list?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for some reasons, using List is creating some empty elements in xml and fail TestResultTest#bigResultReadWrite

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because you are using records, I think. XStream does not support them well. x-stream/xstream#345 If you switch to final classes (yes it is irritating) then you should be able to make this a List<FlakyFailure> and XStream should produce (semi-)attractive and functional XML.

Just be sure the concrete type is ArrayList! Do not try to use List.of(), Collections.unmodifiableList, etc. in the field itself. You can always clone lists in a getter and/or setter or use an unmodifiable collection in a getter.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem was more about returning an empty liist which create an empty element in the xml and so was breaking the test TestResultTest#bigResultReadWrite
Now by using List but returning null instead of empty list the test is passing.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, just check the format carefully to make sure the records work here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unit test is reading correct values from parsed files.
So I suppose it's fine.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah just double-check the XML format on disk by inspection.

private RerunFailure[] rerunFailures;

@SuppressFBWarnings(value = "SE_TRANSIENT_FIELD_NOT_RESTORED", justification = "Specific method to restore it")
private transient SuiteResult parent;
Expand Down Expand Up @@ -247,6 +249,8 @@
}
this.properties = properties;
this.keepTestNames = keepTestNames;
this.flakyFailures = parseFlakyFailures(testCase);
this.rerunFailures = parseRerunFailures(testCase);
}

public CaseResult(CaseResult src) {
Expand All @@ -269,6 +273,38 @@
return Math.min(365.0f * 24 * 60 * 60, Math.max(0.0f, d));
}

private static FlakyFailure[] parseFlakyFailures(Element testCase) {
List<FlakyFailure> flakyFailures = new ArrayList<>();
List<Element> flakyFailuresElements = testCase.elements("flakyFailure");
if (flakyFailuresElements != null) {

Check warning on line 279 in src/main/java/hudson/tasks/junit/CaseResult.java

View check run for this annotation

ci.jenkins.io / Code Coverage

Partially covered line

Line 279 is only partially covered, one branch is missing
for (Element flakyFailuresElement : flakyFailuresElements) {
String message = flakyFailuresElement.attributeValue("message");
String type = flakyFailuresElement.attributeValue("type");
String stackTrace = flakyFailuresElement.elementText("stackTrace");
String stdout = flakyFailuresElement.elementText("system-out");
String stderr = flakyFailuresElement.elementText("system-err");
flakyFailures.add(new FlakyFailure(message, type, stackTrace, stdout, stderr));
}
}
return flakyFailures.isEmpty() ? null : flakyFailures.toArray(new FlakyFailure[0]);
}

private static RerunFailure[] parseRerunFailures(Element testCase) {
List<RerunFailure> rerunFailures = new ArrayList<>();
List<Element> rerunFailureElements = testCase.elements("rerunFailure");
if (rerunFailureElements != null) {

Check warning on line 295 in src/main/java/hudson/tasks/junit/CaseResult.java

View check run for this annotation

ci.jenkins.io / Code Coverage

Partially covered line

Line 295 is only partially covered, one branch is missing
for (Element rerunFailureElement : rerunFailureElements) {
String message = rerunFailureElement.attributeValue("message");
String type = rerunFailureElement.attributeValue("type");
String stackTrace = rerunFailureElement.elementText("stackTrace");
String stdout = rerunFailureElement.elementText("system-out");
String stderr = rerunFailureElement.elementText("system-err");
rerunFailures.add(new RerunFailure(message, type, stackTrace, stdout, stderr));
}
}
return rerunFailures.isEmpty() ? null : rerunFailures.toArray(new RerunFailure[0]);
}

static CaseResult parse(SuiteResult parent, final XMLStreamReader reader, String context, String ver)
throws XMLStreamException {
CaseResult r = new CaseResult(parent, null, null, null);
Expand Down Expand Up @@ -1040,6 +1076,14 @@
this.parent = parent;
}

public @CheckForNull FlakyFailure[] getFlakyFailures() {
return flakyFailures;
}

public @CheckForNull RerunFailure[] getRerunFailures() {
return rerunFailures;
}

/**
* Constants that represent the status of this test.
*/
Expand Down
6 changes: 6 additions & 0 deletions src/main/java/hudson/tasks/junit/FlakyFailure.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
package hudson.tasks.junit;

import java.io.Serializable;

public record FlakyFailure(String message, String type, String stackTrace, String stdout, String stderr)
implements Serializable {}
6 changes: 6 additions & 0 deletions src/main/java/hudson/tasks/junit/RerunFailure.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
package hudson.tasks.junit;

import java.io.Serializable;

public record RerunFailure(String message, String type, String stackTrace, String stdout, String stderr)
implements Serializable {}
49 changes: 49 additions & 0 deletions src/test/java/hudson/tasks/junit/TestResultTest.java
Original file line number Diff line number Diff line change
Expand Up @@ -463,4 +463,53 @@ void bigResultReadWrite() throws Exception {
boolean isTwoEqual = FileUtils.contentEquals(f.getFile(), f2.getFile());
assertTrue(isTwoEqual, "Forgot to implement XML parsing for something?");
}

@Issue("GH-237")
@Test
void includeFlakyAndRerun() throws IOException, URISyntaxException {
TestResult testResult = new TestResult();
testResult.parse(getDataFile("gh-237/TEST-io.olamy.AlwaysFailTest.xml"), null);
testResult.parse(getDataFile("gh-237/TEST-io.olamy.FlakyTest.xml"), null);
testResult.tally();

assertEquals(2, testResult.getSuites().size(), "Wrong number of testsuites");
assertEquals(2, testResult.getTotalCount(), "Wrong number of test cases");

{ // assert on flaky
SuiteResult flakySuiteResult = testResult.getSuite("io.olamy.FlakyTest");
assertNotNull(flakySuiteResult);
assertEquals(
2,
flakySuiteResult.getCase("io.olamy.FlakyTest.testApp").getFlakyFailures().length,
"Wrong number of flayfailures");

FlakyFailure flakyFailure =
flakySuiteResult.getCase("io.olamy.FlakyTest.testApp").getFlakyFailures()[0];
assertNotNull(flakyFailure);
assertEquals("junit.framework.AssertionFailedError", flakyFailure.type());
assertEquals("obvious fail", flakyFailure.message());
assertTrue(flakyFailure.stackTrace().contains("at io.olamy.FlakyTest.testApp(FlakyTest.java:27)"));
assertEquals("this will fail maybe", flakyFailure.stdout().trim());
assertEquals("this will maybe fail", flakyFailure.stderr().trim());
}

{ // assert on rerun failures
SuiteResult rerunSuite = testResult.getSuite("io.olamy.AlwaysFailTest");
assertNotNull(rerunSuite);
assertEquals(
3,
rerunSuite.getCase("io.olamy.AlwaysFailTest.testApp").getRerunFailures().length,
"Wrong number of rerun failures");

RerunFailure rerunFailure =
rerunSuite.getCase("io.olamy.AlwaysFailTest.testApp").getRerunFailures()[0];
assertNotNull(rerunFailure);
assertEquals("junit.framework.AssertionFailedError", rerunFailure.type());
assertEquals("built to fail", rerunFailure.message());
assertTrue(
rerunFailure.stackTrace().contains("at io.olamy.AlwaysFailTest.testApp(AlwaysFailTest.java:23)"));
assertEquals("this will fail for real", rerunFailure.stdout().trim());
assertEquals("this will really fail", rerunFailure.stderr().trim());
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,188 @@
<?xml version="1.0" encoding="UTF-8"?>
<testsuite xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="https://maven.apache.org/surefire/maven-surefire-plugin/xsd/surefire-test-report.xsd" version="3.0.2" name="io.olamy.AlwaysFailTest" time="0.002" tests="4" errors="0" skipped="0" failures="4">
<properties>
<property name="java.specification.version" value="17"/>
<property name="sun.jnu.encoding" value="UTF-8"/>
<property name="java.class.path" value="/home/olamy/test/test-junit/target/test-classes:/home/olamy/test/test-junit/target/classes:/home/olamy/.m2/repository/junit/junit/4.12/junit-4.12.jar:/home/olamy/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar:"/>
<property name="java.vm.vendor" value="Eclipse Adoptium"/>
<property name="sun.arch.data.model" value="64"/>
<property name="java.vendor.url" value="https://adoptium.net/"/>
<property name="os.name" value="Linux"/>
<property name="java.vm.specification.version" value="17"/>
<property name="sun.java.launcher" value="SUN_STANDARD"/>
<property name="user.country" value="AU"/>
<property name="sun.boot.library.path" value="/home/olamy/.sdkman/candidates/java/17.0.11-tem/lib"/>
<property name="sun.java.command" value="/home/olamy/test/test-junit/target/surefire/surefirebooter-20251117145123698_3.jar /home/olamy/test/test-junit/target/surefire 2025-11-17T14-51-23_625-jvmRun1 surefire-20251117145123698_1tmp surefire_0-20251117145123698_2tmp"/>
<property name="jdk.debug" value="release"/>
<property name="surefire.test.class.path" value="/home/olamy/test/test-junit/target/test-classes:/home/olamy/test/test-junit/target/classes:/home/olamy/.m2/repository/junit/junit/4.12/junit-4.12.jar:/home/olamy/.m2/repository/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar:"/>
<property name="sun.cpu.endian" value="little"/>
<property name="user.home" value="/home/olamy"/>
<property name="user.language" value="en"/>
<property name="java.specification.vendor" value="Oracle Corporation"/>
<property name="java.version.date" value="2024-04-16"/>
<property name="java.home" value="/home/olamy/.sdkman/candidates/java/17.0.11-tem"/>
<property name="file.separator" value="/"/>
<property name="basedir" value="/home/olamy/test/test-junit"/>
<property name="java.vm.compressedOopsMode" value="Zero based"/>
<property name="line.separator" value="&#10;"/>
<property name="java.vm.specification.vendor" value="Oracle Corporation"/>
<property name="java.specification.name" value="Java Platform API Specification"/>
<property name="surefire.real.class.path" value="/home/olamy/test/test-junit/target/surefire/surefirebooter-20251117145123698_3.jar"/>
<property name="sun.management.compiler" value="HotSpot 64-Bit Tiered Compilers"/>
<property name="java.runtime.version" value="17.0.11+9"/>
<property name="user.name" value="olamy"/>
<property name="path.separator" value=":"/>
<property name="os.version" value="6.16.3-76061603-generic"/>
<property name="java.runtime.name" value="OpenJDK Runtime Environment"/>
<property name="file.encoding" value="UTF-8"/>
<property name="java.vm.name" value="OpenJDK 64-Bit Server VM"/>
<property name="java.vendor.version" value="Temurin-17.0.11+9"/>
<property name="localRepository" value="/home/olamy/.m2/repository"/>
<property name="java.vendor.url.bug" value="https://github.com/adoptium/adoptium-support/issues"/>
<property name="java.io.tmpdir" value="/tmp"/>
<property name="java.version" value="17.0.11"/>
<property name="surefire.rerunFailingTestsCount" value="3"/>
<property name="user.dir" value="/home/olamy/test/test-junit"/>
<property name="os.arch" value="amd64"/>
<property name="java.vm.specification.name" value="Java Virtual Machine Specification"/>
<property name="native.encoding" value="UTF-8"/>
<property name="java.library.path" value="/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib"/>
<property name="java.vm.info" value="mixed mode, sharing"/>
<property name="java.vendor" value="Eclipse Adoptium"/>
<property name="java.vm.version" value="17.0.11+9"/>
<property name="sun.io.unicode.encoding" value="UnicodeLittle"/>
<property name="java.class.version" value="61.0"/>
</properties>
<testcase name="testApp" classname="io.olamy.AlwaysFailTest" time="0.001">
<failure message="built to fail" type="junit.framework.AssertionFailedError"><![CDATA[junit.framework.AssertionFailedError: built to fail
at junit.framework.Assert.fail(Assert.java:57)
at junit.framework.Assert.assertTrue(Assert.java:22)
at junit.framework.TestCase.assertTrue(TestCase.java:192)
at io.olamy.AlwaysFailTest.testApp(AlwaysFailTest.java:23)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at junit.framework.TestCase.runTest(TestCase.java:176)
at junit.framework.TestCase.runBare(TestCase.java:141)
at junit.framework.TestResult$1.protect(TestResult.java:122)
at junit.framework.TestResult.runProtected(TestResult.java:142)
at junit.framework.TestResult.run(TestResult.java:125)
at junit.framework.TestCase.run(TestCase.java:129)
at junit.framework.TestSuite.runTest(TestSuite.java:252)
at junit.framework.TestSuite.run(TestSuite.java:247)
at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:86)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:316)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:240)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:214)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:155)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495)
]]></failure>
<system-out><![CDATA[this will fail for real
]]></system-out>
<system-err><![CDATA[this will really fail
]]></system-err>
<rerunFailure message="built to fail" type="junit.framework.AssertionFailedError">
<stackTrace><![CDATA[junit.framework.AssertionFailedError: built to fail
at junit.framework.Assert.fail(Assert.java:57)
at junit.framework.Assert.assertTrue(Assert.java:22)
at junit.framework.TestCase.assertTrue(TestCase.java:192)
at io.olamy.AlwaysFailTest.testApp(AlwaysFailTest.java:23)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at junit.framework.TestCase.runTest(TestCase.java:176)
at junit.framework.TestCase.runBare(TestCase.java:141)
at junit.framework.TestResult$1.protect(TestResult.java:122)
at junit.framework.TestResult.runProtected(TestResult.java:142)
at junit.framework.TestResult.run(TestResult.java:125)
at junit.framework.TestCase.run(TestCase.java:129)
at junit.framework.TestSuite.runTest(TestSuite.java:252)
at junit.framework.TestSuite.run(TestSuite.java:247)
at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:86)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:316)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:257)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:214)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:155)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495)
]]></stackTrace>
<system-out><![CDATA[this will fail for real
]]></system-out>
<system-err><![CDATA[this will really fail
]]></system-err>
</rerunFailure>
<rerunFailure message="built to fail" type="junit.framework.AssertionFailedError">
<stackTrace><![CDATA[junit.framework.AssertionFailedError: built to fail
at junit.framework.Assert.fail(Assert.java:57)
at junit.framework.Assert.assertTrue(Assert.java:22)
at junit.framework.TestCase.assertTrue(TestCase.java:192)
at io.olamy.AlwaysFailTest.testApp(AlwaysFailTest.java:23)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at junit.framework.TestCase.runTest(TestCase.java:176)
at junit.framework.TestCase.runBare(TestCase.java:141)
at junit.framework.TestResult$1.protect(TestResult.java:122)
at junit.framework.TestResult.runProtected(TestResult.java:142)
at junit.framework.TestResult.run(TestResult.java:125)
at junit.framework.TestCase.run(TestCase.java:129)
at junit.framework.TestSuite.runTest(TestSuite.java:252)
at junit.framework.TestSuite.run(TestSuite.java:247)
at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:86)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:316)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:257)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:214)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:155)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495)
]]></stackTrace>
<system-out><![CDATA[this will fail for real
]]></system-out>
<system-err><![CDATA[this will really fail
]]></system-err>
</rerunFailure>
<rerunFailure message="built to fail" type="junit.framework.AssertionFailedError">
<stackTrace><![CDATA[junit.framework.AssertionFailedError: built to fail
at junit.framework.Assert.fail(Assert.java:57)
at junit.framework.Assert.assertTrue(Assert.java:22)
at junit.framework.TestCase.assertTrue(TestCase.java:192)
at io.olamy.AlwaysFailTest.testApp(AlwaysFailTest.java:23)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at junit.framework.TestCase.runTest(TestCase.java:176)
at junit.framework.TestCase.runBare(TestCase.java:141)
at junit.framework.TestResult$1.protect(TestResult.java:122)
at junit.framework.TestResult.runProtected(TestResult.java:142)
at junit.framework.TestResult.run(TestResult.java:125)
at junit.framework.TestCase.run(TestCase.java:129)
at junit.framework.TestSuite.runTest(TestSuite.java:252)
at junit.framework.TestSuite.run(TestSuite.java:247)
at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:86)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:316)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:257)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:214)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:155)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385)
at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495)
]]></stackTrace>
<system-out><![CDATA[this will fail for real
]]></system-out>
<system-err><![CDATA[this will really fail
]]></system-err>
</rerunFailure>
</testcase>
</testsuite>
Loading
Loading