How to use code coverage in your Mendix team

Start with a product risk analysis

A product risk analysis helps you to identify, assess, and prioritize the potential risks that may affect the quality, functionality, or usability of your Mendix app. A product risk  analysis is a qualitative analysis that can help you to identify which parts of your Mendix app should benefit from automated tests. However, it cannot measure the actual coverage of the automated tests.

Code coverage on the other hand provides a quantitative measure of which parts of the code have been tested and, more importantly, which not. Code coverage and product risk analysis can complement each other by providing different perspectives and insights on the quality and testing process of your Mendix app.

Keep your coverage goals up to date

Once you have defined the important areas for which you want to set code coverage goals you need to define a strategy on how to keep the goals up to date. There is no definitive answer to how often you need to update code coverage goals, as it may depend on various factors such as the size, complexity, and scope of your Mendix app, the frequency and type of your releases, the expectations and requirements of your customers and stakeholders, and the standards and best practices of your industry or domain. However, some general guidelines that can help you decide when to update your code coverage goals are:

• Update goals for each release: Update your code coverage goals whenever you introduce a major change or feature to your Mendix app, such as adding new functionality, modifying existing functionality, fixing bugs, or refactoring the code. This can help you ensure that your automated tests cover the new or changed code adequately, and that your Mendix app maintains its quality and performance after the change.
• Update goals when the trend is breaking: Update your code coverage goals whenever you observe a significant deviation or variation in your code coverage trend, such as a sudden increase or decrease in code coverage, or a fluctuation or inconsistency in code coverage over time. This can help you identify and address any issues or challenges in your Mendix app development or testing process, such as insufficient or outdated test cases, code duplication or complexity, technical debt, or regression errors.
• Update goals by updating your product risk analysis: Update your code coverage goals whenever you receive feedback or input from your customers or stakeholders, such as user reviews, complaints, suggestions, or feature requests. This can help you align your code coverage goals with the needs and expectations of your target audience, and enhance the usability and satisfaction of your Mendix app. User feedback is actually helping you to improve your product risk analysis.
• Update your goals periodically:  Update your code coverage goals periodically, such as every month, quarter, or year, depending on the duration and cycle of your Mendix app. This can help you review and evaluate your code coverage progress and achievements, and adjust your code coverage goals accordingly to reflect the current state and direction of your Mendix app.
• Do not update goals too often: Updating your code coverage goals regularly can help you keep your quality and performance of your Mendix app at an optimal level, and deliver software that meets or exceeds the expectations and requirements of your customers and stakeholders. However, updating your code coverage goals too frequently or arbitrarily can also have negative effects, such as creating unrealistic or unattainable expectations, wasting time and resources, or demotivating your Mendix team. Therefore, you should update your code coverage goals carefully and strategically, based on the evidence and data from your code coverage reports and analyses, and the feedback and input from your customers and stakeholders.

Use code coverage results in your release criteria

Code coverage can be used in a release strategy for your Mendix app  to determine the readiness and quality of the Mendix app for deployment. Code coverage measures how much of the source code is executed by the automated tests, and can indicate the effectiveness and completeness of the testing process. By setting a minimum code coverage threshold for each release, the Mendix team can ensure that the Mendix app meets a certain standard of quality and functionality before it is delivered to the customers. Additionally, code coverage can help identify the areas of the code that need more testing or improvement, and guide the development of new test cases or refactoring of the code. Code coverage can also be used to compare different versions or branches of the Mendix app, and evaluate the impact of new features or changes on the overall quality of the Mendix app.

Conclusion

Once you can measure code coverage of your automated tests, you need to figure out how to use coverage to improve the quality of your automated tests and thereby the long term quality of your Mendix app. Measuring code coverage is a means to an end, but setting and maintaining code coverage goals is a dynamic process that is part of your quality assurance strategy.

Code coverage goals can help you focus your efforts and can provide a stimulation for the entire team to reach them. Make sure to make the goals as transparent and straightforward as possible and keep the entire team involved in both setting and maintaining them. In that way code coverage can provide a good way of measuring the success of your test automation efforts.