Array
(
)

‘Direct Model Testing’, a disruptive new paradigm for test automation

woensdag september 22, 2021
Markus Travaille, Rolf Bolt

In the previous blog Desired test automation properties for testing Mendix apps we addressed criteria that should be met in order to make your test automation investments returning profits. This blog introduces Menditect’s totally different approach to test automation that incorporates these criteria.

 

The current test automation paradigm is flawed

When you think of it, test automation by itself is actually a strange phenomenon. First, you create business software that contains a lot of functionality, logic and rules in order to meet customer demands. Secondly, you create another application that is checking if the first application is working properly. To ridicule it, why not make a third application that is testing if the second application is doing its job right…and a fourth application

The main problem of the test automation paradigm is that test software contains logic by itself which needs maintenance and scrutiny for correctness. And where the business application is dependent for its logic on business requirements, so is the test automation software solely dependent for its logic on the logic of the business application. In essence, you are coding the business logic twice, first in your business application and secondly in your test application.

Negative side effects of current test automation paradigm

Most test automation implementations are adopting two approaches that have negative side effects. Let’s evaluate some of them.

Black box testing 

By definition, black box testing is banging on the outside of the application in order to find out if the inner workings of the application are correct. Test automation based on page-testing or API testing  are examples of this approach. The disadvantages of this approach are:

  • You need a lot of banging in order to get sufficient test coverage. If the tester has no deep insight in the internal workings of the app or thorough documentation about the business requirements, no guarantees about test coverage can be given this way. For example, imagine testing the Google search engine by providing data to the available user interface…;
  • The outside interfaces (like pages or API’s) which are used by black box testing do have their own logic and rules. This prevents easy testing low level components or units and is forcing large scripts to be made in order to test that functionality;
  • The correctness of designed tests is not checked ‘design time’ but ‘run time’ which is quite late in the process. For instance, adding a string to a number field in page-test automation can be done but leads to an error during run-time execution.

Unit testing

Unit tests are usually implemented by adding extra code to your business application. It gives the opportunity to test the inner workings of the application logic by running the logic. Some disadvantages of this way of testing are:

  • The business logic of your application is expanded with extra code solely for test purposes which in essence is code pollution;
  • It is hard to implement a structure of unit tests to form modules and even processes. Unit testing often lacks a proper test framework such as the ISTQB framework;
  • Combining unit tests with several sets of data (data driven testing) is often hard to implement and manage;

‘Direct Model Testing’, a new test paradigm

The ideal test automation does not contain logic by itself but derives its logic completely from the business application logic. That is, the test automation is ‘knowing’ the inner workings without ‘programming’ this logic. On top of that, it would be great if the test can execute this logic directly. Just run the business application logic in the right order, provide the necessary test data and evaluate the correctness of the outcome.

Of course, when a change of business logic invokes a different data need, the tester must be alerted ‘design time’ to offer a different data source. This way of test automation and execution leads to seamless coupling between business logic on the one hand and the design and operation of tests on the other.

In addition to that, ideally it must be possible to create a ‘building’ of small executable tests into comprehensive all logic encompassing tests in a well structured and maintainable test framework such as the ISTQB framework. In that way, you start by defining unit tests for calling ‘low-level’ business logic  and composing them into encompassing module- process- and API tests. Of course, this ‘building of tests’ must be checked ‘design time’ as well for consistency in order to keep maintenance issues to the minimum;

Last but not least, the ideal test solution must be capable of defining tests that support the testing of a business process that touches more than one business app during a test run. This prevents the necessity of a multitude of test automation environments.

This new paradigm will enhance test capabilities without writing test code and without invoking a maintenance burden. We call this way of testing ‘Direct Model Testing’.

How does Menditect apply ‘Direct Model Testing’ to Mendix apps?

First of all, it should be noted that Mendix models are key in the paradigm. These models contain all the business logic and business rules in the form of microflows and entities and are perfect touchpoints for the test application.

The Menditect Test Automation solution (MTA) utilizes the Mendix model by retrieving model components (microflows and entities) and exposing them to the test automator as testable components. The test automator creates teststeps from these components by selecting a model component and defining the desired data input. The picture below shows the selection page for finding a microflow to be turned into a teststep.

No-code-testing

After configuring, the teststep can be executed directly on the backend server. When an assert is configured for the teststep, this will be evaluated on the fly. In other words, you can execute microflows by providing data to the microflow parameters, run them, collect the results and evaluate if the outcome is ok. In the same way you can select entities in order to inject, delete, retrieve and evaluate data objects directly. This way, all existing business logic can be re-used for test automation purposes for which no extra logic needs to be programmed.

Consistency checks

On top of easy configuration, MTA has unique consistency checking capabilities, like the Mendix Studio model consistency checker. You  receive an alert in the test modeler immediately during ‘design time’ whenever your designed test is not conforming to the data types or logic in the business apps that are connected to your test definition.

Re-use of tests

Moreover, you can use the microflow result or the manipulated object in the next part of your test script. This enhances the possibility to re-use test components by chaining and organizing them into hierarchically structured module- and process tests.

Normally, re-use of tests by ‘chaining’ them into larger scripts is undesired because the implicit dependencies result in bad maintainability. But MTA supports consistency checking of the coherence of scripts as well. You get an alert when a previously designed test case is not delivering the correct information to a dependent test case (see picture below). This way the dependencies become explicit and are easy to maintain.

Automatic-consistency-check

In order to speed up test automation even further, MTA supports data driven testing. For each model interaction, (microflow or object manipulation), a dataset can be configured that is used during test execution. By that, each test can be executed numerous times with different data without the need to copy teststeps.

Some other advantages of ‘Direct Model Testing’

The focus on utilizing the business logic directly in test configuration often leads to a couple of advantages.

Lower maintenance costs

First of all, you will experience a big drop in maintenance efforts and costs. The freed time and money can be spent on enhancing the application test coverage making the overall return of investment a better one.

Better collaboration between testers and developers

Secondly, developers and testers are in close contact with each other about testing the app. Not to say that it is now possible to enhance the T-shaped developer with easy to maintain test capabilities or the other way around, engaging citizen testers for designing and realizing test automation.

Third, experience learned that the re-use of models during ‘Direct Model Testing’ is a natural way of having code reviewing embedded in your development cycle. By this, the test actions will deliver a quality improvement of your app even before a test is executed.

Finally, executing tests directly on the model instead by using a page interface leads to fast test execution.

Conclusion

Changing the test automation paradigm to a ‘Direct Model Testing’ delivers exciting new ways of testing Mendix apps. We will continue to blog about utilizing ‘Direct Model Testing’ with subjects like ‘security handling of MTA’, ‘how to build process and component tests on top of unit tests’, ‘small steps to speed up the ROI of your test investment’ and so on. Please leave a comment on LinkedIn if you like us to write about some specific aspect of test automation with MTA.

Markus Travaille, Chief Vision officer van Menditect en co-founder van EGALiT

Contact the author

Enter your contact details

Menu