Before we understand what are the challenges faced in Mobile Testing, Lets first focus on what is the scope of Mobile Testing and then we will dig deeper in the world of Mobile Testing.
Scope of Mobile Testing
Mobile Testing covers following Apps:
- Web Apps: Web applications opened on mobile browsers (chrome, safari, opera etc.)
- Native Apps: application program that has been developed for use on a particular platform or device (android, iOS etc.)
- Hybrid Apps : Native Apps + Web Apps
Challenges in Mobile Testing
The objective of testing is same for both website and mobile i.e. if the AUT meets the quality criteria or not, but there are some issues/challenges/factors that are only relevant to Mobiles and are of no importance to traditional Website Testing.
Here in my blog, I have tried to list down possible challenges we face in Mobile Testing and the factors that constitute that challenge. So it becomes very important to specially deal with these challenges to come up with an effective Mobile Test Strategy.
Hardware Challenge
Mobile application testing is difficult because you need to test for various hardware configurations which is a result of different combinations of the factors mentioned below:
- Various Manufacturers (Samsung, Apple etc.)
- Various Screen Sizes (Phone and Tablet etc.)
- Keypad – (Virtual or hardware)
- Variations in RAM, CPU, OS utilization etc.
- Touch Screen Responsiveness.
Did you know: – There is No Guarantee that application tested on one device will run successfully run on other device also.
Software Challenge
Mobile application testing is difficult because you need to test for various software configurations which is a result of different combinations of the factors mentioned below:
- Various OS (iOS, android, Windows etc.)
- Various Versions of OS (iOS 4 to iOS 7.1, android 4.2.2 to android 4.4 etc.)
- Also need to test on upgraded OS versions( i.e. from iOS 6 to iOS 7)
Did you know:-OS like iOS does not allow the device to be downgraded to lower iOS once upgraded to higher iOS. So if do not keep this in mind, it can exponentially increase the testing cost.
Network Challenge
Mobile application testing is difficult because you need to test for various Network configurations which is a result of different combinations of the factors mentioned below:
- Network Carrier (Airtel , Vodafone etc.)
- Bandwidth (2G, 3G, Broadband etc.)
- No Network, Airplane Mode
- Network Connectivity (low, high etc.)
- Replicate diverse Geo-graphical locations?
- Test GPS-based Operations at real Site?
Did you know:- Network Simulators can be used for creating artificial network of varying speeds 2G,3G etc.
Interrupts Challenge
This is one area which is quite often missed by testing teams until the alpha or beta testing and this is one area which is also most often missed by developers also. So as soon as you start testing for interrupts, be ready for some crazy behavior.
But creating the real life scenarios for this kind of testing is a real head-ache for the testing team. So you need to innovative in creating real life scenarios for:
- Phone Calls
- Text Messages
- Push Notifications from other apps
- Push Notifications from AUT
- Battery/OS level interrupts
Did you know:-While accessing internet if a calls comes than Internet connection is interrupted.
Data Security
This is also one area that is quite often not given the importance by testing teams and also not understood fully also. Data Security is a big challenge for the testing teams because now a days all sorts of user data is present on Mobile Handsets so security of that data is big daunted task.
- Confidentiality: Does your app keep your private data private?
- Integrity: Can the data from your app be trusted and verified?
- Authentication: Does your app verify you are who you say you are?
- Authorization: Does your application properly limit user privileges?
- Availability: Can an attacker take the app offline?
- Non-Repudiation: Does your app keep records of events?
Did you know:- Unintended use of user personal data is against the App Store Guidelines.
Shankar Garg 
Hi Shankar,
I contradict your comment on : “OS like iOS does not allow the device to be downgraded to lower iOS once upgraded to higher iOS. So if do not keep this in mind, it can exponentially increase the testing cost.”
When a new version is released for iOS after that till 15-20 days you can downgrade your device. But for some limited periods only. Here is the details: http://www.iphonehacks.com/2016/08/downgrade-ios-9-3-4-ios-9-3-3-jailbreak-iphone-ipad.html
Let me know if it is helpful.
LikeLike
Hi Jayabrata,
The link you have provided, talks about JailBreaking the device(because you are downloading the firmware from an unknown resource and not apple). Two main issues with this approach:
1. At first place makes my device will be not eligible for testing my(client/own) app.
2. No other client will accept this device for testing their apps. So making the device unusable forever.
My original statement meant “Not possible to downgrade iOS version without Jailbreaking” still holds true.
Please confirm if their is any other way to downgrade iOS.
LikeLike
Hi Shankar,
For Downgrading iOS 9.3.x it does not require to Jailbreak the device. The only condition that hold is: The downgrade can be done as long as Apple is signing iOS 9.3.x. Once the signing is stopped by apple we will not be able to downgrade without Jailbreak.. & why I am so much confident on that because I have downgraded my own iPhone & it is not jailbroken yet 🙂
But yes I am not sure if we can downgrade from 10 .x to 9 versions..
LikeLike
Ok, lets not call it a Jailbreak, but are you sure about the .ipsw file you are downloading from iPhoneHacks website and run it on a device which is connected to your organisation/clients network? kind of a Risk that should be noted.
Also, as you mentioned in last sentence still no way to downgrade from version like 10 to 9.
Appreciate the thought shared in this thread.
LikeLike
Yes there might be a risk but it should not be as much vulnerable which can damage any personal info.. I am still using it without any such data leakage..
Also i have not upgraded to iOS 10 so did not required it for a downgrade.
LikeLike
if the organisation is ok with this risk you can go ahead.
Personally i would still avoid, the risk of an unknown OS is more than having an older version of OS available for few days.
LikeLike