Stable and scalable frontend test automation

Flakiness of a test run

Flakiness in frontend test automation refers to a test run that produces inconsistent results—sometimes passing, sometimes failing—without any alterations to the Mendix application or the test code itself. These tests are unreliable and unpredictable, undermining developer trust in test results.

Flaky test runs often indicate underlying issues in the test environment or the test code’s inability to gracefully handle the dynamic nature of Mendix applications.

• Timing issues: This is the most prevalent cause. Mendix apps are highly asynchronous, with elements loading at varying times due to API calls, animations, or dynamic rendering. A test might attempt to interact with an element (e.g., click a button or verify text) before it has fully rendered, leading to a failure. Even minor network latency can determine a passing or failing test.
• Race conditions: When tests execute in parallel, they may contend for shared resources, leading to unpredictable outcomes. For example, two tests attempting to log in with the same user credentials could cause one to fail because the user session is already active.
• Environmental instability: The test environment itself can introduce flakiness. A test that passes on a fast, local machine might fail in a continuous integration (CI) pipeline due to a slower network connection, a different operating system, or variations in available system resources. Running tests on local Mendix applications can also fail due to Mendix’s user session limitations in “trial mode.”

Maintenance overhead

The necessity for constant maintenance is a direct consequence of fragility and flakiness. The more brittle and unreliable a test suite becomes, the more time and effort is required to keep it functional.

• Continuous updates: As development teams iterate on a frontend, they constantly make small changes to the UI, microflows, and domain model. Each of these changes, even minor, can cause fragile tests to fail, demanding manual debugging and test updates from developers or QA engineers.
• Debugging time: When a test fails, the immediate question is: “Is this a genuine bug, or is it a flaky test?” This uncertainty forces teams to dedicate significant time investigating false negatives, a task that provides no value to the user and impedes the entire development and release process.
• Long scripts: Frontend tests often require extensive data setup and cleanup of previous test results. This can lead to relatively long and complex test scripts, increasing maintenance. Similarly, asserting data stored in the database via the frontend requires multiple steps that inflate frontend test script size.
• High cost: The cumulative effect of continuously fixing fragile and flaky tests results in a substantial maintenance overhead. The time invested in maintenance can quickly surpass the time saved by initial test automation, diminishing the return on investment of the test automation effort.
• Cross browser compatibility: Building and maintaining a cross-browser testing tool is a major undertaking. It requires a dedicated infrastructure with real devices and browser versions, which is expensive and difficult to keep up to date. For a test tool to be successful, it must handle frequent browser updates and new device releases, ensuring its tests remain relevant and accurate.