Without test automation no low-code success
OutSystems enables low-code and high velocity software development. In order to keep the low-code advantages, quality must be ensured on a daily basis. The most effective way to ensure quality on a daily basis is to integrate nightly test automation. Without test automation there is no way testing can keep up with the OutSystems velocity and bugs will be introduced or not detected. Buggy low-code means slow code.
In order to conserve the quality of the Apps, the agility of the organization and in order to conserve the ROI of the development effort a continuous focus on quality is needed. Due to the fact that OutSystems teams can deliver a very high velocity there is a big part to play for test automation. This implies both functional testing of new and existing functions, i.e.: regression testing, as well as performance/load testing and validation of the application security. All of these tests must be run quickly and in a flexible manner in order to create a short ‘feedback loop’ so that developers can quickly remedy any errors. In that way it is possible to avoid slowing down the software delivery process and avoid slow code.
Test automation is the only option
In view of the workload for testing and the fact that testing and test results needs to be available (nearly) every day the only possible option is to use automated testing.
Effective test automation must:
- Run completely automated without any human intervention.
- Be completely OutSystems compatible and can handle the changes each OutSystems deploy generates on the App level.
- Run tests which are understandable to the business and product owners.
- Run comprehensive reports which support a flexible decision track in the CI/CD process.
- Be easily maintainable.
- Be based on industry standards.
In this paper we will give some practical examples of a test automation solution which delivers on the requirements above and is completely compatible with OutSystems.
Functional test script: Validate login
Note that the test is set up using business identifiers for the fields involved such as Username and Password, also the Login button is identified by the text Login. Due to the highly structured HTML OutSystems generates it is not necessary to use technical identifiers such as CSS, XREF, Class of ID. However, technical identifiers can be used if needed. Business identifiers and multiple technical identifiers can be mixed and matched as needed.
This test script performs the following actions:
- Open the application using the specified URL.
- Enter the credentials of user Eddie R. Riedel in the login form fields. Note that these fields are selected using their on-screen names. In this particular case, the test software finds the fields using their placeholder attributes.
- Click the Login button, again using a non-technical identifier.
- Wait until the full name of the user (‘Eddie R. Riedel’) is visible. If the login fails, this name will not appear on the screen, so the test will fail as well.
Overview test runs
Individual tests or test suites can be monitored in a simple highly efficient overview. This overview enables direct entry to the individual tests for closer inspection by clicking on the actual test line. From there on the test results are immediately available. The same info can also be automatically pushed into the build/deployment process for automated build/deployment purposes. In many CI/CD environments the overview as seen above is used as the primary interface on the success or failure of individual test(suites), providing a simple ‘traffic light’ list overview of daily/nightly test automation runs.
High-level KPI’s, statistical information and history, is also available on separate pages. This way the test automation framework can also be used to highlight overall quality and progress made. Individual test results are still available and are only a couple of clicks away.
Data-driven test script: Check accounts
This test is set up to fail in order to demonstrate the way in which failures of data-driven tests are reported. This test runs in a data-driven mode, repeatedly executing an identical script using multiple rows of test data as illustrated above. The script validates multiple instances of account data. When all is well, FitNesse keeps the actions collapsed. When an action fails, the actions are shown by default so the failing instance can be examined readily.
In this case, the data row for ‘GROSELLA-Restaurante’ generated a failure and is expanded by default.
As we can see in the table above, the test performs the following actions for each data row:
- Click the company name to go to the company page.
- Check if the company name is visible on the page.
- Check if the company code is visible on the page.
- Check if the contact name is visible on the page. The red background of the ensure word in this row indicates that this action failed.
- Check if the contact title is visible on the page.
Functional test script: create order
This test is set up to fail in order to demonstrate the way in which failures are reported. This test script shows an example of a failing action stopping the test and automatically generating a screenshot of the application at the time of the failure. The action ‘wait for visible’ fails because the phrase ‘Eastern Trade Co.’ is not visible on the page after searching for companies with ‘eastern’ in their name, so the test is aborted. In the screen shot above a standard failure detail page is shown. On this page the failure is documented in business terms including a screen shot so that both programmer, scrum master and product owner can evaluate the failure.
For posterity, here are the actions of the script explained:
- Click the Accounts button in the menu on the left side of the page.
- Enter ‘eastern’ in the search bar. Note that ‘company name’ appears in the placeholder of the field, and therefore the software selects this field for the phrase ‘company name’.
- Press the Enter key to perform the search.
- Wait for ‘Eastern Trade Co.’ to become visible. This fails, and therefore shows a screenshot and aborts the test, as explained in the paragraph above.
- Click the company name of ‘Eastern Trade Co.’ Note that this action was never performed because the previous action aborts the test on failure.