The jenkins test result GUI shows skipped or expectedFail tests as skipped. The individual test view for skipped or expectedFail tests shows "Skipped Message" and "Standard Output"
e.g. "Skipped Message" can be:
a custom skip message
* e.g. from python @unittest.skip("some reason") tag or * e.g. raise unittest.SkipTest("thing not found.")- "expected test failure"
- "xfail-marked test passes unexpectedly"
We are using groovy script to generate test reports. We would like to include more information about skipped tests rather than just "skipped". How can we get info on the skipped test like in the "Skipped Message" in the GUI view?
The jenkins API is documented here:
https://javadoc.jenkins.io/plugin/junit/hudson/tasks/junit/TestResult.html
There is no specific call for getting information on the skipped or expectedFail tests. I am hoping with some experimentation that it will be possible to get expectedFail information through this testResult API. Starting with these API calls:
String getErrorDetails()
If there was an error or a failure, this is the text from the message.
String getErrorStackTrace()
If there was an error or a failure, this is the stack trace, or otherwise null.
String getName()
Gets the name of this object.
String getStderr()
The stderr of this test.
String getStdout()
The stdout of this test.
TestResult getTestResult()
Returns the top level test result data.
String getTitle()
Gets the human readable title of this result object.
In the GUI:
- A normal passed test just has "Standard Output".
- A normal failed test has "Error Message" and "Stacktrace" and "Standard Output".
- A skipped or expectedFail tests shows "Skipped Message" and "Standard Output".
We are using python unittest outputting junit test result files. Loading that into jenkins using junit test result plugin.
Have I missed something in the jenkins test results API that would give more information on expected fail or skipped tests ? I hope to find the info through experimentation using the API. And document it in an answer here.
Here is the guts of test report groovy script (used in jenkins Execute Groovy Script plugin after jUnit result plugin has harvested test results):
import hudson.model.*
def build = Thread.currentThread().executable
workspace = build.getEnvVars()["WORKSPACE"]
reportfilename = workspace + "/testreport.html"
rf = new File(reportfilename);
def testCount = "0"
def testPassed = "0"
def testFailed = "0"
def testSkipped = "0"
def buildDuration = "0"
def workspace = "unknown"
def buildName = "unknown"
def BUILD_STATUS = ""
def BUILD_URL = ""
def testResult = null
def testResult1 = null
def testResult2 = null
def testDuration = ""
def caseResult = null
def buildNumber = 0
def buildNumHash = ""
def buildTimeString = ""
def rooturl = ""
try {
buildNumber = build.number
buildNumHash = build.getDisplayName()
//currentBuildNumber = manager.build.number
buildTimeString = build.getTime().format("YYYY-MMM-dd HH:mm:ss")
if(build.testResultAction) {
testResult = build.testResultAction
testCount = String.format("%d",(testResult.totalCount))
testPassed = String.format("%d",(testResult.result.passCount))
testFailed = String.format("%d",(testResult.result.failCount))
testSkipped = String.format("%d",(testResult.result.skipCount))
testDuration = String.format("%.2f",(testResult.result.duration ))
}
workspace = build.getEnvVars()["WORKSPACE"]
buildName = build.getEnvVars()["JOB_NAME"]
BUILD_STATUS = build.getEnvVars()["BUILD_STATUS"]
BUILD_URL = build.getEnvVars()["BUILD_URL"]
testResult1 = hudson.tasks.junit.TestResult
testResult2 = build.getAction(hudson.tasks.junit.TestResultAction.class)
caseResult = hudson.tasks.junit.CaseResult
rooturl = manager.hudson.rootUrl
} catch(Exception ex) {
rf << "exception accessing build.testResultAction object.";
//rf << ex;
}
// in groovy the write RE-creates the file, rf << "whatever" is used to append.
rf.write "<html><head><title>testreport.groovy #$buildNumber $buildName</title></head><body>"
rf << "Summary test report <br><br>\n\
<b>TEST RESULT:</b> $testCount total, <b>$testPassed pass</b>, <b>$testFailed fail</b>, $testSkipped skip.<br>\n\
Workspace : $workspace<br>\n\
Project Name : $buildName $buildNumHash<br><br>\n\
"
if (build) {
rf << """<!-- GENERAL INFO -->\n\
\n\
<TABLE>\n\
<TR><TD align=\"right\">\n\
<j:choose>\n\
<j:when test=\"${build.result=='SUCCESS'}\">\n\
<IMG SRC=\"${rooturl}static/e59dfe28/images/32x32/blue.gif\" />\n\
</j:when>\n\
<j:when test=\"${build.result=='FAILURE'}\">\n\
<IMG SRC=\"${rooturl}static/e59dfe28/images/32x32/red.gif\" />\n\
</j:when>\n\
<j:otherwise>\n\
<IMG SRC=\"${rooturl}static/e59dfe28/images/32x32/yellow.gif\" />\n\
</j:otherwise>\n\
</j:choose>\n\
</TD><TD valign='center'><B style='font-size: 200%;'>BUILD ${build.result}</B></TD></TR>\n\
<TR><TD>Build URL</TD><TD><A href=\"${rooturl}${build.url}\">${rooturl}${build.url}</A></TD></TR>\n\
<TR><TD>Project:</TD><TD>${buildName}</TD></TR>\n\
<TR><TD>Date of build:</TD><TD>${buildTimeString}</TD></TR>\n\
<TR><TD>Build duration:</TD><TD>${build.durationString}</TD></TR>\n\
<TR><TD>Test duration:</TD><TD>${testDuration}</TD></TR>\n\
</TABLE>\n\
<BR/>\n\
"""
}
if(!testResult) {
rf << "<br>No test result<br>"
rf << "</body></html>"
return ("No test result")
}
def junitResultList = [];
junitResultList.add(testResult.getResult())
if (junitResultList.size() > 0) {
rf << "<br>test result from build.testResultAction"
} else {
junitResultList.add(testResult2.getResult())
if (junitResultList.size() > 0) {
rf << "<br>test result from build.getAction"
} else {
rf << "<br>No results in 'testResult2'<br>\n"
junitResultList.add(testResult1.getResult())
}
}
//rf << "<br>DEBUG" + junitResultList.size() + " test items"
// API: http://hudson-ci.org/javadoc/hudson/tasks/junit/PackageResult.html
rf << "<!-- JUnit TEMPLATE: all tests PASS FAIL SKIP -->\n"
if (junitResultList.size() > 0) {
rf << '<TABLE width="100%">\n'
rf << "<TR><TD class='bg1' colspan='2'><B>${junitResultList.first().displayName}</B></TD></TR>\n"
junitResultList.each { junitResult ->
junitResult.getChildren().each { packageResult ->
rf << "<TR><TD class='bg2' colspan='2'> <B>TEST SUITE: ${packageResult.getName()} Failed: ${packageResult.getFailCount()} test(s), Passed: ${packageResult.getPassCount()} test(s)</B>, Skipped: ${packageResult.getSkipCount()} test(s), Total: ${packageResult.getPassCount()+packageResult.getFailCount()+packageResult.getSkipCount()} test(s)</TD></TR>\n"
packageResult.getChildren().each { suite ->
suite.getChildren().each { test ->
def colour = "lightgreen"
def highlight1=""
def highlight2=""
RESULT = test.getStatus().name() // FAILED or PASSED or SKIPPED (.name() not .value)
// hudson.tasks.junit.CaseResult.Status.FAILED
if (RESULT == "FAILED" || RESULT == "REGRESSION") {
colour = "#ffcccc"
highlight1="<B>"
highlight2="</B>"
}
if (RESULT == "SKIPPED") { colour = "#ffffb3" }
rf << "<TR bgcolor='${colour}'><TD class='test' colspan='2'>${highlight1}<li>${RESULT}: ${test.getFullName()} </li>${highlight2}</TD></TR>\n"
}
}
}
}
rf << '</TABLE><BR/>\n'
}
rf << "testreport.groovy</body></html>\n"