1. Initial situation
In a project at a major Swiss cantonal bank to automate the testing of the Avaloq banking system using Tricentis Tosca, reporting is a key issue. Currently, releases are made four times a year. Due to the decreasing effort required for manually performing regression tests, a monthly release cycle is planned. Despite intensive efforts, Tosca Analytics has not yet been successfully implemented for test reporting. This situation led to the idea of implementing a basic, automatically generated test report from within Tosca using simple technical means.
2. Objectives
In the current project, relevant information such as the test case name, order number(s), verification results, etc., is written to a text file during the execution of each test case (based on an idea by my colleague Michael Weiglein). This file is made available to the department head so they can more easily track the test execution. However, this file does not indicate whether a test case actually ran successfully or whether, for example, the verification of a certain condition was unsuccessful. Since this text file is written before the end of the test execution, its result does not yet exist at that point.
Therefore, the department heads should be given a simple way to check, at any time and in a directly accessible location, whether and when a test case last ran successfully. Furthermore, the time required to generate a simple test report should be significantly reduced.
3. Technical requirements
A fundamental prerequisite for implementation is running tests with Jenkins, as the system relies on the resulting XML-format files. By default, Jenkins writes these files to a directory on the relevant server corresponding to the specific test task, e.g., C:\jenkins\workspace\Financing. A Tosca agent for Distributed Execution running on a different machine may not be able to easily access this directory due to potentially insufficient permissions. Therefore, the Jenkins configuration must be adjusted so that the result files are saved on a network-accessible drive (see Fig. 1). The directories used to store these files should be created before running the tests; for example, "Create Loan Application".
Fig. 1: Example Jenkins configuration file for conducting tests in a specific department.
4. General Process
Among other things, regular expressions are used to filter the following relevant data from a Jenkins result file (see Fig. 2):
- Test case name
- Timestamp
- Result (Passed, Failed)
Fig. 2: Extract from an example Jenkins result file
This information is compiled into a string in CSV format, which is then written to a file of the same format in the target directory. The following algorithm is used in a more general way (see Fig. 3):
Fig. 3: Algorithm for extracting the relevant data from the Jenkins result file
Sometimes a test case isn't executed but is included in the Jenkins results file. Therefore, the system first checks whether the initial search for the relevant text string yields any results. If the search is empty, a default text string containing the timestamp and test result is written to the target file. This immediately identifies that this test case wasn't executed during the last test run.
5. Concrete implementation with Tosca
For variable implementation, Tosca uses a test sheet with the following structure (see Fig. 4):
Fig. 4: Tosca test sheet
In the current project, a separate test sheet is used for each department, such as finance or payment transactions, to ensure that the list of test cases is not too extensive and that the result is tailored to the target audience.
Some of the relevant steps in the TestTemplate are discussed in more detail below (see Fig. 5).
Fig. 5: Tosca test template
The content of the XML result file generated by Jenkins can be loaded from the respective directory based on the Tosca test sheet via a path as follows: \\Tosca\Resultat\Jenkins\{XL[Name of Stage]}\{XL[Name of Parallel]}_Resultat.xml.
The search for the environment string is implemented in such a way that it searches directly for the appropriate name for the test environment (e.g. INT for an integration test environment).
The following regular expression is used to search for the source string, from which the desired data is then extracted: ({XL[Instance.Name]}).* log="[-+] (Failed|Passed)" (where the XL reference is resolved to the specific name of a test case). This test case name corresponds exactly to the respective instance name in the test sheet, so it cannot be extracted from the source string.
The text between the test case name and the timestamp is found using the following regex expression and then removed: .*".*timestamp=".
The UTC timestamp has the following form: 2020-01-31T12:49:54.1985841+01:00. It is converted to the format YYYY-MM-DD HH:MM. The timestamp is transformed into the target format in the following steps:
- Extract date
- Extract time
- Assemble timestamps in the intended format
The test result itself is extracted as text via a direct search: (Failed|Passed).
If this entire process were not implemented with Tosca, but instead, for example, using a script, the search for all result strings could be performed in one step using specific test case lists per department, instead of using concrete Tosca test cases.
The current result string is then written to the end of a CSV file. This file is saved with a name using the structure {Test Environment_Department_Date}.csv in a suitable directory, for example, INT_Financing_2020-01-31.csv (or in variable form: {B[AVQ_Test Environment]}_{XL[Department]}_{DATE[][][yyyy-MM-dd]}.csv). If this text file does not yet exist, the header containing the string "Test Case;Timestamp;Result" is first written to an empty file. Each time the process is run, the system checks whether the CSV file already exists. If it does, the existing content is copied, the new text string is appended to the end, and the new content is written back to the same file. The following text is used as a placeholder string: {XL[Instance.Name]};Not executed;–.
The process thus produces, for example, a CSV file with the following content:
Test case; Timestamp; Result:
Finance smoke tests – record loan; 2020-01-31 16:38:35; Passed:
Finance create loan application – home; Not executed; –
Finance create loan application – business credit limit; 2020-01-31 16:52:10; Failed
6. Execution
The test report can be generated manually using an ExecutionList in Tosca. For automated execution immediately following a test run, a dedicated stage set up in the Jenkins configuration, containing the necessary elements for distributed test execution in Tosca.
7. Benefits
This automatically generated, simple test report can provide individual functions with benefits such as:
- The test manager needs less time for manually creating test reports. Failed test cases can be annotated with a comment in an additional column in an Excel file before being distributed to a department head.
- The test engineer maintains an overview of the specific test case instances by continuously updating and configuring the test sheets with newly implemented test cases. The test instances themselves are quickly created with Tosca.
- The subject matter expert has constant access to the test results without having to explicitly distribute them. This leads to better visibility of the test results.
The actual benefit, of course, depends on the specific environment and the type of test reporting required.
8. Further expansion
8.1 Improved readability with an Excel or PDF file
If Microsoft Excel is installed on the computer running the Tosca agent for testing with Jenkins, the CSV file can be made more readable. First, an Excel template with conditional formatting is created so that the test result is easier to identify through green or red coloring. The contents of the CSV file are then imported into cells A4:C54 and similar. This Excel file could also be saved as a PDF.
Fig. 6: Excel template with conditional formatting of test results
8.2 Distribution via email
The generated test reports can be sent to the desired recipients using a PowerShell script of the following type:
Send-MailMessage -to @(“test@example.com”) -from “Regression Tests<no-reply@example.com> " -SmtpServer "servername.example.com" -Subject "Test result" -Body "The test result can be found in the attachment." -Attachments "Test result.csv" -Encoding ([System.Text.Encoding]::UTF8)')
This task will be executed at the desired time via the Windows Task Scheduler (or a cron job in Unix-based environments).
9. Conclusion
This description demonstrates how automated reporting can be implemented from within Tricentis Tosca using simple technical means when testing is performed with Jenkins. The design and distribution of the test reports can be quickly adapted to the specific needs of each context. Of course, these basic reports are not a substitute for comprehensive reporting to management. The focus is on providing an automatically generated test run result that business users can access as needed.
February 3, 2020, Bernhard Fuchs
Update from March 2nd, 2020:
In the original version, the date format, e.g., 2020-01-31T12:49:54.1985841+01:00, was not interpreted correctly. Since there is no time difference compared to Central European Time, the section on time conversion has been removed from the above representation.
I describe a significantly more elegant way to generate a test report from the Jenkins result files in the blog post " More Elegant Ways to Generate a Simple Report on Jenkins Test Execution."
