We released the last update for our V2 line in April, and then jumped into preparing for the next major release: V3. On this occasion, I would like to share some updates on our roadmap and some good news about the HTML report.
TL;DR
Native HTML report generation is coming in V3! This will work out of the box, without additional conversion tools, and will be free and open source!
V2 was about test execution
The V2 versions of Reqnroll focused on stabilizing the Reqnroll codebase and the test execution workflow. The SpecFlow codebase was forked at version v4-beta. That version was never officially released and therefore was only used by a few SpecFlow projects. It contained many changes, especially related to asynchronous execution of tests. This was a significant change affecting many aspects of execution, and because it was only tried by a limited set of users, it was not “battle tested.” We could say that Reqnroll V2 is the “proper SpecFlow v4” release.
Another important and outstanding need that we inherited from SpecFlow was to clean up the execution workflow. Probably the most notable element of that was supporting scenario-level parallelization, but in the background we also had to redesign the entire resource handling, hook execution, and error handling infrastructure.
V3 is about SpecFlow ecosystem parity
Users have seen the SpecFlow ecosystem as a homogeneous set of frameworks and supporting tools that consisted of the SpecFlow core framework, the Visual Studio and Rider extensions, the SpecFlow plugins, and the SpecFlow+ toolset — particularly the SpecFlow+ LivingDoc Generator. In reality, each of these parts and tools has its own story, and most importantly, the SpecFlow+ toolset was never open source, so we could not fork it as we did with the other assets.
This has been a pain point for many users, because LivingDoc Generator was the go-to tool for generating nice HTML reports about their specifications and execution results. And we could not take that over.
Although there are alternative solutions for generating HTML reports (notably Allure Report, one of the most popular open-source reporting tools, which was one of the first to integrate with Reqnroll), we felt that we needed to provide a built-in solution as well.
Our strategy has been to adopt the concept of “Cucumber Messages” from the Cucumber tool family. Cucumber Messages is basically a standardized file format that can capture all execution details of a BDD framework, including step or hook executions and attachments. Our idea was that if we were able to provide a Cucumber Messages report, we would open up visualization possibilities that have already been available to Cucumber tools. We cannot talk about this feature without mentioning the project contributors, who made an incredible effort to bring this strategy to reality. Thank you!
While V3 will contain many other interesting updates, most importantly, it will include the first version of our native Reqnroll HTML report generation! With this step, we will finally reach parity with the SpecFlow ecosystem.
Reqnroll HTML Report
While this is still in the beta phase, I would like to share some details about Reqnroll HTML report generation. The beta phase in this case means that the code has been implemented and is working, but we are in the middle of the review process and making some small fixes.
As mentioned above, our strategy was to be able to produce Cucumber Messages so that we could use existing tooling to turn that into HTML. We were aware of the Cucumber React Components project, which provides HTML visualization for Cucumber Messages, but in the end, we found another important open-source component, Cucumber HTML Formatter, which provides an end-to-end wrapper of the react-components for producing a complete HTML report from Cucumber Messages. Exactly what we needed!
Have you heard of “Cucumber Formatters”? That is a feature and extensibility point in Cucumber tools that can be used to produce reports, outputs, or other results in a specific format. Since Cucumber started as a command-line tool, the concept of formatters has existed there for a long time — basically, when you executed Cucumber, it was a formatter that produced the console output and another formatter that generated the test result JSON file. As SpecFlow/Reqnroll has been executed by the .NET test execution hosts (e.g., dotnet test or Visual Studio Test Explorer), we had not implemented this particular concept before.
During the implementation of the Cucumber Messages output, we realized that what we were doing was essentially the same as what Cucumber calls “formatters,” so we reframed our solution and implemented the formatter concept from Cucumber into Reqnroll.
And if we have formatters in Reqnroll, then we can implement the Cucumber Messages output as one formatter and the HTML output as another! That means we will not need any additional tool to convert Cucumber Messages to HTML (as was needed for “SpecFlow+ LivingDoc Generator”), but it will generate the HTML file out of the box! Yes, even if you run the tests from Visual Studio!
It is important to mention that all parts of the HTML generation toolchain (Reqnroll, formatters, Cucumber HTML formatter, Cucumber React Components) are free and open source.
The formatter concept has two important characteristics that make it even more appealing for Reqnroll:
- You can register multiple formatters. If you need to have both Cucumber Messages and an HTML report, you can register both!
- You can write your own formatter if you need a customized report or output format! This is all integrated into the Reqnroll Plugin infrastructure.
Currently, the Cucumber Messages and HTML formatters will be packaged with Reqnroll, so these can be used out of the box. Later, we might extend the set of formatters (built-in or as plugins), and I’m pretty sure there will be formatters from the community as well.
To use the formatters, you simply need to add a formatters setting to your reqnroll.json config file (you can also enable it with environment variables). With the current beta, this is how a sample config file looks that produces both HTML and Cucumber Messages reports:
{
"$schema": "https://schemas.reqnroll.net/reqnroll-config-latest.json",
"formatters": {
"html": {
"outputFilePath": "report/reqnroll_report.html"
},
"messages": {
"outputFilePath": "messages/reqnroll_report.ndjson"
}
}
}
This is how the report looks for a Calculator example. Pretty!
V4 is about structural changes
After V3, we will not stop, but continue with further structural changes that will hopefully make everyone’s life easier. We have plans to consolidate the dependency injection support plugins, the configuration infrastructure, and we also plan to use the Roslyn infrastructure to improve the overall workflow.
For V4, we will also review the range of different targets we support and deprecate those that require high maintenance costs and are not broadly used. The support for adding feature files to VB.NET projects is one candidate for deprecation.
TUnit and xUnit V3 support coming too
There are also two new unit test frameworks we plan to support: TUnit and xUnit v3. As these are additional features and not breaking changes, they can probably be included as a minor update of Reqnroll V3.
Your feedback counts
Thank you for reading through this long post. Please share your ideas in the Reqnroll Discussion Board or on our Discord server.