How can I test my responsive mobile app - Part 2
The manual testing series
In this series I’ll be uncovering the various techniques and tools that can help you get started to manually test a mobile web or hybrid app. Manual testing is a great way for verifying layouts and basic functionality. Automated tests afford us more verification but require a different set of tools and deeper understanding that’s not part of this series.
In [part 1](how-can-i-test-my-responsive-mobile-app-part-1), we tested with online and browser plug-ins to capture bugs.
Part 2, delves into testing on emulators specifically for mobile web apps running from the web. We’ll cover,
- Screenshot based tests.
- Testing with cloud based emulators.
- Testing with a local iOS based emulator aka simulator.
- Testing with a local Android emulator
The layout of a page is influenced by more than just the width settings assigned to a media query. Testing on an emulator or on a physical device can highlight hiccups caused by how a specific version of a mobile browser renders an element before applying a media query. When you stick to basic tests you can never be sure how your app will truly render and function. More often than not, it will definitely be broken.
Also, there’s a breed of mobile apps like Forecast.io that render different layouts by detecting whether a user is visiting from a mobile browser. For these, emulators or devices are the only testing solution.
Hybrid mobile apps add their own complication, some have device dependencies that stop us dead in our tracks when testing on a desktop browser.
Emulators versus physical devices
Emulators are great for sanity checks. They’re brilliant for quick feedback loops that can optimise your work-flow. Moreover, they give you access to a browser environment that is a one to one match with your target device. Bugs encountered here are not an approximation - these are real issues.
But, watch out! An emulator will mostly be running on either a server or a desktop machine. This gives it far more resources than you’ll ever get on the physical device. It allows emulators to be more forgiving. I have caught many a bugs on a physical device that were never obvious on emulator tests.
I want emulators
If you’re working on a mobile web app, you might not have any emulators set up at all. It all depends on your development environment and the resources available. But its easy to get set up. Although, your access might be restricted to a smaller range.
Cloud based testing sites offer a wider range of emulators. Its very quick to get going and legacy emulators will always be available - something very hard on a personal development machine.
Let’s start by looking at cloud based emulators and then plunge into running our own.
Emulators on the cloud
Some of the services include BrowserStack, Sauce Labs and Cross Browser Testing to name a few. They’re essentially the same, but provide different UI testing interfaces. My personal preference is BrowserStack but feel free to use any solution you (or your pocket) prefers.
Starting With Screen-shots
You can start things off quick and dirty. Just take a screen-shot of a single screen across all available devices.
BrowserStack offers this,
- Navigate to their Responsive UI testing screen.
- Enter the URL of the page you’d like to test, I’m entering http://aishy.github.io/UpApp/
- Click (or tap) the View button.
It can take a few seconds. BrowserStack loads Up App’s homepage on a super huge range of mobile browsers, all processing screenshots and then displaying them.
Since I haven’t fixed the issues from part 1, you can see these cropping up again with the iPad 3 on portrait mode.
But there are other peculiarities, the animated sun illustration is broken on all Android devices from the 2.1 Droid Razr (320x640) to the 4.1 Nexus 7 (603x966). The layout renders well while the illustration positioning is slightly off or in case of the Droid Razr the illustration is bigger than it needs to be.
There’s great value in this exercise. Very quickly, we’ve realised bugs that were in no away apparent from simple plug-in or online testing.
In fact, here’s a screen-shot using the Droid Razr dimensions as rendered by the Viewport-Resizer browser plug-in. According to the plug-in, there are no issues whatsoever (but we know better).
Probing further with live testing
Now that we have a bug benchmark for faulty OS and screen sizes, we don’t have to waste our time testing things that already work. Instead, we can hone in on those problems that need fixing straight away.
Most cloud based testing tools have a live environment where you can select an emulator and use it just as if it was running on your machine. We know that we’re having some serious problems running on Android.
To start testing on BrowserStack you need to be signed up and logged in (there’s a trial subscription).
Then, navigate to the Live testing section.
- Enter the URL of the page you’d like to test, I’m entering http://aishy.github.io/UpApp/
- Select Android as the OS.
- Select Google Nexus 7 as the device.
- Click (or tap) the Start Testing button.
A few seconds go by while the environment is starting. Once complete, you’ll be connected to a server running a browser on a Google Nexus 7 emulator with your target URL loaded up.
Note: Your environment may never load if you’re browser does not have a recent version of the flash plugin installed - I found that the hard way.
Using the app, I can see the sun illustration dimensions are completely wrong. However, the layout is correct and the css3 animation works. This suggests that only a few simple css tweaks are necessary to support the Google Nexus 7 default browser for Android 4.1 - Yay!
Browser, browser, browser
All this testing highlights that the peculiarities are browser centric. In the Android world there’s a high spread of browsers all supporting variations of CSS and the latest HTML5 capabilities. The Cross Browser Testing site has a good list of the different browser configurations for mobile devices. I particularly like their detailed listings for Android.
Coupling this knowledge with tools like Modernizr allow us to smooth out the discrepancies and build a unified set of instructions that can be applied to more browsers. Basically, less frustration, more fun and speedier development!
Running your own emulators
We’ll attack iOS and Android emulators separately. Each platform requires some software installation before you can start. Once you’re up and running you can use the same setup over and over again with no hassle!
Bear in mind that iOS emulators are only available for Mac OSX operating systems. This pretty much means that you have to be running on a Mac. Android is more widespread and you can run the emulator on multiple operating systems.
Local iOS emulators
To avoid confusion, I’ve been constantly talking about emulators but in iOS world they’re referred to as simulators. So, on a mac desktop if you’d like to access an iPhone simulator then you need to,
- Ensure you have Xcode installed or install Xcode 5 from the App Store.
- Launch Xcode.
- Choose Xcode > Open Developer Tool > iOS Simulator.
This gives you access to the latest iOS7 simulator. You can also access older simulators which would simulate an older browser environment. Here are more details.
To test an app on Safari,
- From the emulator’s Home screen click Safari.
- In the address field type the address of your app, in my case http://aishy.github.io/UpApp/
Local Android emulators
For Android, the installation is slightly more cumbersome but definitely doable with speed improvements here. Also, a few vendors have popped up to help get you get up and running in no time.
I like Genymotion simulator. Although slightly long-winded because of the dependencies, it is simple to set-up and they have a free personal licence.
Genymotion relies on having a Virtual Box VM installed to run the different configurations for an Android Simulator. Its very similar to the emulator configurations seen on cloud based services.
Let’s install this first,
- Download an installation for your OS from here.
- Install Virtual Box on your machine.
You also need to ensure that you have Android SDK for your OS installed including the platform-tools directory. On Mac OSX,
- Navigate to the Android downloads page.
- Select Use an Existing IDE section.
- Click Download the SDK tools for Mac.
- After the SDK is downloaded, move the folder (android-sdk-macosx) to your home folder using Finder.
- In the (android-sdk-macosx) folder, open the tools folder.
- In the tools folder right click the android Unix executable file and select open.
- The Android SDK Manager should appear.
- Under the Tools listing, select the Android SDK Tools + Android SDK Platform-tools and Android SDK Build-tools.
- Click Install … Packages.
Your done once you can now see a new folder called ‘platform-tools’ in the (android-sdk-macosx) folder.
To get Genymotion up and running,
- Start using instructions here.
- Run Genymotion and click on Settings.
- In Settings, select the ADB tab and ensure that Path to Android SDK points to the (android-sdk-macosx) folder.
- Add a device using instructions here - I added a Nexus 4 - 4.3 768x1280.
- Play your device.
After the device is up and running,
- Navigate to the Built in browser by clicking Browser icon on the homepage.
- In the address field type the address of your app, in my case http://aishy.github.io/UpApp/ 4.3 Nexus 4 768x1280
There you have it, all the emulator love we can handle!
Part 3 will showcase how to manually test hybrid mobile apps using both cloud services and local emulators mainly for the Cordova ecosystem.