By José Díaz & Björn Lemke, Díaz & Hilterscheid GmbH
Several years ago, the sale of UMTS licenses brought in millions of euros for the German federal government. The telecommunication companies spent a lot of money obtaining these licenses. Some of them, for example Mannesmann Telekom, had run up such high debts that they did not survive the first phase of the mobile future. They were simply bought up by the major players.
In economic terms, the early days of UMTS were extremely painful for the telcos and of little interest to the end customers. UMTS was initially only affordable for well-heeled customers and business customers who were often out of the office. The general public was deterred by the prices. There were also no smartphones as we know them today.
Blackberry, from Research in Motion, was the first good solution for mobile e-mail clients. Everyone who thought a lot of themselves had a Blackberry. Nowadays that’s history. Blackberry recently made losses amounting to billions, and was sold. Blackberry, however, is not the only casualty of the dynamic mobile market. The Nokia mobile phone business was only recently sold to Microsoft. The former market leaders in the mobile phones sector have apparently failed to keep up with the market.
Momentum came with the iPhone. The iPhone not only revolutionized the telecommunications sector – it changed the entire world of IT. In the meantime, however, Google with its Android operating system took over market leadership for mobile devices. These two operating systems are currently engaged in fierce competition, and are far ahead of all other competitors. Currently, however, significantly more Android devices are being rolled out.
With all mobile platforms, the focus is on the user.The objective was and continues to be to offer the customer convenient mobility, allowing him to do everything that he wants.Nowadays the customer is mobile! Nowadays just about everybody has a smartphone. We use apps for just about everything. Up to May 2013, for example, 50 billion apps were downloaded via iTunes. By 2016 it is expected that approx. 290 billion apps will be downloaded. There is a similar picture at Google – in April 2013,1.5 million new devices were registered daily with Google, and approx. 1 million applications are available to download from Google’s Play Store. These are new dimensions.
Figure 1. Fragmentation of the market for Android devices. Source:
Android Fragmentation Visualized; http://opensignal.com/reports/fragmentation-
This trend towards mobility is also affecting banks. In the past, the customer transferred money via a branch office of his bank, via an ATM, or via the internet from his home PC or laptop. The parameters were straightforward and thus relatively easy to control. Nowadays transactions can be handled via apps that can run on approx. 20 different Apple products with two different operating system versions, and on more than 10,000 (yes, ten thousand) different mobile devices running under Android with currently eight different operating system versions.
This all means that companies must develop strategies that not only affect development and testing. They must also address how their own apps are managed, marketed and, last but not least, designed in order to be able to hold their own against so much competition.
In the remainder of this article, however, the focus lies on the aspects of quality assurance and testing.
The following factors influence testing:
- User experience/emotional experience with the application
- Networks and providers
- Interruptions of the application
- (incoming call, text message, alarm, etc.)
- Platform diversity
- iOS 6 and 7, iPhone and iPad
- Blackberry OS 9 and 10
- Windows Phone 7 and 8
- Mobile Web
- Device manufacturers, display sizes
- Use of resources (memory, power, bandwidth)
- Sensor inputs and control capabilities
- The regression and automation portfolio
- Application types
- Web application – responsive design
- Hybrid application – cross-platform development
- Hybrid application with native elements
- Native application
- The development model – agile or traditional
- The frequency of provision of the test objects
- Test cycles and testing levels.
The complexity exists, will remain, and will, in all probability, even increase. From the testers’ point of view there is a lot to be considered and, in particular, a lot to learn.
User expectations in terms of comfort and convenience
Irrespective of the operating system and the device being used, the user expects the app to operate correctly. If the app is not loaded within 2 to 3 seconds, it will be deleted. No chance for a second attempt!
Figure 2. Display sizes of iOS and Android devices. Source: Android Fragmentation Visualized;
All objects must be available and react the way the user expects them to react. When we solely consider the various display sizes and resolutions, we can see that we will have a lot to do in terms of testing, since mobile applications always run in full-screen mode and therefore have to look good at all display resolutions and display sizes.
During functional testing we are challenged to address a wide range of different aspects. It is not sufficient to systematically define and execute test cases. We have to carry out exploratory testing and perform tests in both stationary and mobile environments. Above all, we have to develop unorthodox test scenarios that reflect the behavior of the real users as authentically as possible. A combination of real devices and emulators is indispensable. Manual testing alone, however, is insufficient; a further essential element is the automation of selected test cases.
Involvement of real users at an early stage is unavoidable in order to ensure that using the app is a positive experience for the user. If the end user does not like the app, it will be deleted and a competitor’s product will be installed in its place. The customer has gone and will never come back.
Interoperability with the different versions of the operating systems, and interruptions by other applications such as the telephone or text messages, represent an additional challenge that must also be addressed when planning and executing the testing of mobile applications.
This type of testing focuses on the interaction between the user and the application, and on the design of the application. The mobile user is a special type of user. He wants everything to be fast and uncomplicated, he expects the app to operate exactly as he envisaged it would, and his time is limited. The number of apps that can do the same thing is huge. Specialists should be called in for this type of testing. Usability studies should also be carried out as early as possible with representatives of the end users at whom the app is targeted, to determine as early as possible whether the development is moving in the right direction or whether changes need to be made to the design of the GUI or the user guidance.
Accessibility testing is a subset of usability testing. It must be ensured that the apps can be used by persons with, for example,impaired sight or hearing. These aspects have to be addressed during the design and testing of the application. As is the case with usability testing, accessibility testing must also be planned, assigned sufficient resources, and, ultimately, executed at an early stage.
Security tests are context-specific. Different apps demand different security concepts. A banking app is not the same as an event app. This applies to both the implementation and the testing of the app. It is therefore particularly important to define the requirements on the security of the application. Only when the security requirements are known can they be economically taken into account during design, implementation, and testing. Merely attempting to ensure security via tests carried out subsequent to the development and implementation is not an expedient approach and usually results in the need for extensive changes, which in turn result in extensive effort for regression tests to ensure that the security changes do not impact the functionality or other characteristics of the software.
During security testing, an analysis of the architecture is indispensable. Modelling of threats and derivation of measures for coping with these threats are further important elements in the process of bringing a secure app onto the market. Additional aspects to be addressed are data security and integrity. Not just the data used by the app itself need to be addressed here, but also all other data available on the mobile device. For example, it would be disastrous if the app were to modify or delete the data in the user’s contacts list. The app should also be subjected to penetration tests.
As already mentioned, the performance of an application is a central element in its success. The overall performance consists of various elements – on the one hand the performance of the app itself, and on the other hand the performance of the backend with which the app communicates. The users will very soon stop using an app that loads quickly but requires what seems like an eternity for every step in the process of communicating with the server. The strategy in this case should be to carry out specific tests in the backend, while at the same time closely monitoring the behavior of the app. This can be planned and executed extremely well using commercial tools such as SOASTA CloudTest and TouchTest. For the apps themselves, the initial loading time and the loading time between individual process steps must be tested. What processing steps are required? What are the resource consumption levels? In tests such as these, it is not so much the actually measured times that are relevant but the emotionally perceived times.
The network plays an additional important role here. Are we connected via WiFi, LTE, G3, or GSM? In the case of a mobile app, the impact on performance of a change of network during use is also relevant.
It is necessary to test apps for robustness, particularly in the case of Android devices that, despite having the same operating system version, behave differently due to different hardware and manufacturer-specific extensions. How does the app react in the case of known faults/errors in the devices?
Portability tests – installation and compatibility
The worst thing that can happen is that the customer never has an opportunity to use the app. The challenge in this respect is to know the necessary target platforms and to ensure that the app can be installed, executed, and used on these platforms. Compatibility with the devices must be ensured.
Internationalization and localization testing
Some apps use different languages, GUI elements, or even utilization processes, depending on the country of use. If this is the case, an appropriate strategy must be followed. It must be determined whether the regional adaptations can be tested locally using simulators or whether it is necessary to use selected device types for testing by local staff in the respective countries. This type of testing can be extremely expensive.
Apps usually have a short version life, meaning they are updated more frequently than other software. Before the app is rolled out following testing of error corrections or new functionalities, a regression test must be carried out. Where possible, the depth of the regression testing should be defined by means of an impact assessment.
Manual testing of the apps is unavoidable. Test automation, on the other hand, is also extremely important. A number of tools with different areas of focus are available on the market. Selection of a tool without a Proof of Concept is next to impossible since the concepts behind the tools vary considerably. In addition to tools from commercial vendors such as SOASTA or Ranorex, there are also open-source alternatives such as Selendroid and ios-driver for automated testing using Selenium.
Mobile app testing involves significantly more than was previously the case within the scope of software quality assurance. The complexity and diversity resulting from operating systems, application types, devices, networks, display sizes, user experience, interoperability, security, performance, etc. must be taken into account during the planning and execution of the testing process. A company’s success is becoming increasingly linked to how it operates with mobile customers and to the customers’ level of satisfaction with the app.We must adapt our capabilities as testers to these new circumstances.
Díaz & Hilterscheid GmbH will be exhibiting at Apps World Germany on April 22-23 at the CityCube, Berlin; visit them on booth #422.