Efficient report generation with XSLT and PowerShell
In my blog post "Simple Test Report from Tosca via Jenkins Test Execution," I described, in an initial version, how to create a basic test report from the results of tests conducted with Jenkins using simple methods. Experience gained since then shows that the approach described there is likely insufficiently efficient in most cases. Using Tosca test cases for report generation is relatively time-consuming, and the approach doesn't scale. The effort required for implementation, maintenance, and execution increases at least linearly with the number of test cases that need to be included in the report. Initially, I considered a different method for extracting the relevant data from the XML files and using XML modules in Tosca. However, it has since become clear that the detour via Tosca is unnecessary. Below, I will explain how the same test report can be generated much more efficiently using a PowerShell script.
1. General
For reasons of efficiency and purpose, report generation is faster if Tosca test cases are not required. Instead, the process for creating a test report is as shown in Figure 1.
The XSL transformation of the Jenkins result files from XML to CSV format is performed sequentially using a PowerShell script that is executed at the end of the test run. A test report in HTML format is then generated from this raw data.
2. Creating the test report in CSV format
An example XML file resulting from the Jenkins test execution looks like this (see Listing 1):
Listing 1: Example Jenkins result file in XML format
The relevant data is taken from the beginning of each element.<testcase> extracted, specifically from this part:
testcase name="Financing Smoke Tests – 01_Property Recording" time="168.5116337" timestamp="2020-02-10T16:13:05.6008759+01:00" log="+ Passed …"
The XML transformation to CSV format is based on the following XSL stylesheet (see Listing 2):
<?xml version="1.0" encoding="UTF-8"?><xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"><xsl:output method="text"/><xsl:template match="/"><xsl:for-each select="testsuites/testsuite/testcase"><xsl:value-of select="@name"/><xsl:text> ;</xsl:text><xsl:choose><xsl:when test="@timestamp = ''"><xsl:text> Not executed; </xsl:text></xsl:when><xsl:otherwise><xsl:value-of select="substring(@timestamp,0,11)"/><xsl:text></xsl:text><xsl:value-of select="substring(@timestamp,12,8)"/><xsl:text> ;</xsl:text></xsl:otherwise></xsl:choose><xsl:choose><xsl:when test="not(@log)"><xsl:text> -- </xsl:text></xsl:when><xsl:otherwise><xsl:value-of select="substring(@log,3,6)"/><xsl:text></xsl:text></xsl:otherwise></xsl:choose></xsl:for-each></xsl:template></xsl:stylesheet>
Listing 2: XSL stylesheet for XML transformation
The following PowerShell script extracts the data relevant for the test report from this Jenkins result file in XML format and saves it in CSV format (see Listing 3). It is based on a similar script described under XSL Transformation of XML Documents via PowerShell . Only the content-relevant functions are listed (e.g., without path checking or error handling).
# XML transformation to CSV and HTML format using an XSLT file # Script call: *.ps1 [XML source file(s)] [XSLT file] [Subject area] # Parameter initialization param ( $xml = @("D:\Tosca\Resultat\Jenkins\Finanzieren\Eigenheim_Resultat.xml","D:\Tosca\Resultat\Jenkins\Finanzieren\ Smoketests_Resultat.xml"), $xslt = "D:\Tosca\Resultat\XSL\XSL-Stylesheet.xsl", $area = "Finanzieren") # Function for the XML transformation function XML-Transformation($xml, $xsltFile, $area){ # Create timestamp $timestamp = ([datetime]::now).tostring("yyyy-MM-dd HH-mm") # Define temporary CSV output file $csv = "D:\Tosca\Resultat\XSL\Testreport.csv" # Initialize result files with path $resultCSV = "D:\Tosca\Resultat\XSL\Report_" + $area + "_$timestamp.csv" $resultHTML = "D:\Tosca\Resultat\XSL\Report_" + $area + "_$timestamp.html" # Write header to CSV file Add-Content -Path $resultCSV -Value 'Test case;Timestamp;Test result' # Process array of XML input files ForEach ($xmlFile in $xml) { $xmlFile = resolve-path $xmlFile # Create XSL object $script:xslt = new-object system.xml.xsl.xslcompiledtransform $script:xslt = new-object system.xml.xsl.xsltransform $xslt.Load($xsltFile) # Load XSLT file $xslt.Transform($xmlFile, $csv) # Perform XSLT transformation # Append result to result file Add-Content -Path $resultCSV -Value (Get-Content $csv) -Encoding UTF8 } # Delete temporary output file Remove-Item $csv}# Function call for XML transformation XML transformation $xml $xslt $area
Listing 3: PowerShell script for transforming XSL into CSV format
The following parameters can be passed when calling the script:
-xmlJenkins result files in XML format as a comma-separated list-xslPath to XSLT stylesheet file (optional, if specified in the script)-areaDepartment (e.g., Finance, Payments)
Another parameter could be used to determine the storage location of the generated report files.
The result in CSV format looks like this, for example:
Finance smoke tests – Record property; 2020-02-10 16:13:05; Passed
Finance smoke tests – Record insurance; 2020-02-10 16:16:58; Passed
Finance smoke tests – Assemble financing; 2020-02-10 16:19:37; Failed
Review loan portfolio; Not executed;–
3. Transformation to HTML format
Since importing into Microsoft Excel's binary format is not a simple process, and the test report is primarily intended to provide information about the test results, the HTML format is used. As mentioned in the previous blog post, the Excel import could be performed using Tosca. However, this will not be done here.
The transformation of the raw CSV data into an HTML test report is carried out using the second part of the PowerShell script at the end of the XML transformation function (see Listing 4; reduced to the essential content):
$head = @"<title> Test report</title><meta charset="utf-8"><style type="text/css"> BODY{background-color:#FFFFFF;color:black;font-family:Arial,sans-serif;font-size:14px;margin:25pt;} TABLE{border-width:2px;border-style:solid;border-color:black;border-collapse:collapse;} TH{border-width:1px;padding:8px;border-style:solid;border-color:black;background-color:#D5D8DC; font-size:18px;text-align:left;} TD{border-width:1px;padding:8px;border-style:solid;border-color:black;background-color:#F9F9F9;}</style><script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/1.4.4/jquery.js"></script><script type="text/javascript"> jQuery(document).ready(function(){ jQuery('td:contains("Passed")').css('background-color','#2ECC71'); jQuery('td:contains("Failed")').css('background-color','#EC7063'); jQuery('td:contains("--")').css('background-color','#D0D3D4'); });</script> "@ $body = "<h1> Test report $area</h1> " Import-Csv -Delimiter ";" -Path $resultCSV -Encoding UTF8 | ConvertTo-Html -Head $head -Body $body | Out-File $resultHTML
Listing 4: PowerShell script for creating the HTML test report from the CSV file
The test report in HTML format can be designed, for example, as shown in Fig. 2:
Fig. 2: Test report in HTML format
4. Jenkins Configuration
The configuration for running the Jenkins test is shown in Listing 5:
pipeline { agent any options { timeout(time: 120, unit: 'MINUTES') } parameters { choice(name: 'ENV', choices: ['CT', 'INT'], description: 'Select Environment') } environment { TRICENTIS_CI_HOME = 'C:\\Program Files (x86)\\TRICENTIS\\Tosca Testsuite\\ToscaCommander\\ToscaCI\\Client' TRICENTIS_CI_CONFIG = 'D:\\Tosca\\Configurations\\' TRICENTIS_CI_ENVIRONMENT = "${params.ENV}" JUNIT_RESULT_FILE = 'D:\\Tosca\\Result\\Jenkins\\Financing' } stages { stage('Create Loan Application') { steps { parallel("Home Ownership": { echo "Starting Tosca tests for Financing: Home Ownership" bat label: 'Homeownership', script: '"%TRICENTIS_CI_HOME%\\ToscaCIClient.exe" -t junit -r "%JUNIT_RESULT_FILE%\\Homeownership_Result.xml" -m distributed -c "%TRICENTIS_CI_CONFIG%\\%TRICENTIS_CI_ENVIRONMENT%_Financing_Homeownership.xml"' } ) } } stage('Process-Neutral Tests') { steps { parallel("Smoketests": { echo "Starting the Tosca tests for financing: Smoketests" bat label: 'Smoketests', script: '"%TRICENTIS_CI_HOME%\\ToscaCIClient.exe" -t junit -r "%JUNIT_RESULT_FILE%\\Smoketests_Result.xml" -m distributed -c "%TRICENTIS_CI_CONFIG%\\%TRICENTIS_CI_ENVIRONMENT%_Financing_Smoketests.xml"' } ) } } stage('Testreport') { steps { parallel("Testreport": { echo "Starting the test report" bat label: 'Testreport', script: '"C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe" -ExecutionPolicy Bypass -Command "D:\\Tosca\\Resultat\\XSL\\XML-Transformation.ps1" -xml "%JUNIT_RESULT_FILE%\\Eigenheim_Resultat.xml,%JUNIT_RESULT_FILE%\\Smoketests_Resultat.xml" -area "Financing"' } ) } } }}
Listing 5: Example Jenkins configuration
5. Delivery via email
Using PowerShell, this report file can be automatically sent as an HTML attachment to the desired recipients (see Sending emails with PowerShell). The test report can also be integrated directly into the email if the HTML format is enabled. With the HTML generation method shown in Listing 4, the color highlighting of the test result is lost, as JavaScript execution is usually disabled for HTML emails for security reasons. However, CSS classes can be used in the "Test result" column when converting the CSV report to HTML code for visual differentiation. The script definitions using jQuery functions are then unnecessary. The correspondingly adapted script can be found in Listing 6.
$head = @"<title> Test report</title><meta charset="utf-8"><style type="text/css"> BODY{background-color:#FFFFFF;color:black;font-family:Arial,sans-serif;font-size:14px;margin:25pt;} TABLE{border-width:2px;border-style:solid;border-color:black;border-collapse:collapse;} TH{border-width:1px;padding:8px;border-style:solid;border-color:black;background-color:#D5D8DC; font-size:18px;text-align:left;} TD{border-width:1px;padding:8px;border-style:solid;border-color:black;background-color:#F9F9F9;} TD.passed {background-color:#58D68D} TD.failed {background-color:#EC7063} TD.unexecuted {background-color:#D0D3D4}</style> "@ $body = "<h1> Test report $area</h1> " Import-Csv -Delimiter ";" -Path $resultCSV -Encoding UTF8 | ConvertTo-Html -Head $head -Body $body | ForEach { if ($_ -like "*<td> Passed*") {$_ -replace "<td> Passed", "<td class=`"passed`"> Passed"} elseif ($_ -like "*<td> Failed*") {$_ -replace "<td> Failed","<td class=`"failed`"> Failed"} else {$_ -replace "<td> --", "<td class=`"unexecuted`"> --"}} | Out-File $resultHTML
Listing 6: Adapted PowerShell script for creating the HTML test report from the CSV file
Another option could be to send an email via a Jenkins plugin after the test run (or build) has been executed. However, this option was not investigated further, as the chosen method allows for more direct control of the process.
The password for accessing the mail server should be stored encrypted. This can be done in PowerShell using the code in Listing 7:
$KeyFile = "C:\Temp\AES256.key"$PasswordFile = "C:\Temp\Password.txt"$Password = "Password"# Create key file with AES-256$CryptoKey = New-Object Byte[] 32[Security.Cryptography.RNGCryptoServiceProvider]::Create().GetBytes($CryptoKey)$CryptoKey | out-file $KeyFile# Encrypt password$CryptoKey = Get-Content $KeyFile$SecurePassword = $Password | ConvertTo-SecureString -AsPlainText -Force$SecurePassword | ConvertFrom-SecureString -key $CryptoKey | Out-File $PasswordFile
Listing 7: Generating an encrypted password in PowerShell
The test report can be included in the main body of the email (see Listing 8 – reduced to the essential content, without error handling).
# Access data$KeyFile = "C:\Temp\AES256.key"$PasswordFile = "C:\Temp\Password.txt"$MailUser = "Username"# SMTP settings$PSEmailServer = "mail.example.com"$smtpFrom = "sender@example.com"$smtpTo = "recipient@example.com"$messageSubject = "Test report " + $area$messageBody = (Get-Content ".\Report_Finanzieren_2020-02-10 16-32.html" | Out-String)# Generate access data$CryptoKey = Get-Content $KeyFile$creds = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $MailUser, (Get-Content $PasswordFile | ConvertTo-SecureString -Key $CryptoKey)# Send emailSend-MailMessage -Credential $creds -From $smtpFrom -To $smtpTo -Subject $messageSubject -Body $messageBody -BodyAsHtml -Encoding ([System.Text.Encoding]::UTF8) -port 587 -UseSsl
Listing 8: Automatic email sending using PowerShell
Depending on the configuration of the specific environment, for Send-MailMessage, only -port 25 may need to be used instead of the last two parameters, e.g. on the Jenkins server.
5. Using Jenkins plugins
The use of Jenkins plugins for the same purpose of report generation and distribution based on XML result files, such as HTML Publisher or Test Results Aggregator. However, depending on the specific use case, the approach described here allows for more targeted use, as the configuration is easier to adjust or can even be adjusted at all (e.g., when sending email attachments).
6. Conclusion
This explanation has demonstrated how a report on test execution with Jenkins can be generated in just a few seconds. Compared to using Tosca test cases, report generation takes a fraction of the time. Configuration effort is significantly reduced, as structures no longer need to be defined for each test case, but only once per domain, according to the structure of the Jenkins result files. This method of report generation allows for more versatile use. To include newly added test cases, only the Jenkins configuration files need to be adjusted as needed. Furthermore, distributing the report via email is an efficient way to directly inform relevant stakeholders.
