Question

Recently in QCon, Gerard Meszaros said that we should run automation tests only on simulators to improve efficiency. We are also considering about the mobile automation test. But I’m not sure if there will be some issues that can only found in a real device? Or some components like camera, gravity sensors could not be tested in a simulator/emulator? Thanks!

Edit:

You can find Gerard’s slides here: http://www.slideshare.net/pinot_noir/ensuring-quality-in-cloud-and-mobile-applications.

I read through the slides again and I found that I misunderstood his point. What he really means is that we should try to avoid debugging on the hardware (page 41). And he also mentioned we could trap outgoing requests to the sensors and other services, then inject the response from them. Maybe we can pick some cases that couldn’t be fully tested in simulator and then test it with real devices.

Thank you for all your comments! 🙂

Answer 1

Recently in QCon, Gerard Meszaros said that we should run automation tests only on simulators to improve efficiency.

This was odd advice, if that is really what Mr. Meszaros said. Running tests on the emulator is fine, but “only” is an excessive recommendation. There is no harm in running automated tests on devices, and you can learn a lot from doing so.

But I’m not sure if there will be some issues that can only found in a real device?

Of course.

• Many devices have multi-core CPUs, whereas the emulator only emulates a single core at this time
• Device storage tends to run a lot slower than does storage on the emulator
• Device manufacturers tinker with Android in ways that will not appear on an emulator running stock Android
• The emulator only loosely emulates hardware related to power, Internet (e.g., no mobile data, no WiFi), GPS, sensors, camera, etc.
• The emulator does not support some device capabilities, like the new V2 version of Maps, the Play Store, multiple accounts on Android 4.2, etc.

And so on.

Or some components like camera, gravity sensors could not be tested in a simulator/emulator?

Those ones are difficult to test in an automated fashion, period.