Android Auto & CarPlay are automotive platforms that present unique challenges to Quality Assurance. Many people may be unfamiliar with these platforms, what they are, how they work, and especially how to test them. In this blog, I will share my experience working with a customer to help them test and deliver a radio and podcast app to Android Auto & CarPlay.
An introduction to Android Auto & CarPlay
What is Android Auto and Apple CarPlay? At their core, they are a means to bring the iOS and Android experience to in-car dashboards. They are designed to display information from a phone to a car’s built-in head unit, enabling drivers a much safer way to use apps, listen to music, make phone calls, send text messages and much more. Many manufacturers have been building Android Auto and CarPlay support into cars since. However, what if your car is older than that? Luckily there’s also a way to get Android Auto and CarPlay into existing vehicles. Aftermarket systems from a wide range of notable companies are also compatible with Android Auto and CarPlay. Both platforms are designed to be hands-free, assisting in making as little driver distraction as possible.
The challenges of testing on Android Auto & CarPlay platforms
The solution to the inherent challenges of testing these platforms began by asking a number of questions. Do we have access to testing vehicles that support Carplay or Android Auto? Do the devs or PM’s test along side QA? What about phones and tablets? The first step was to determine which combination of device and OS versions are compatible with CarPlay and Android Auto. With those devices in mind, I worked with the client to decide which specific phones and tablets would best represent their user base and provide adequate test coverage. However, there was still one big problem: how do we test the app on an actual car stereo? The solution we arrived at was to build a car stereo system that worked independently of a vehicle.
So the process began! We needed to find a car stereo system that had all the functionality needed for the project. This included: Carplay, Android Auto, Bluetooth, Touch Screen, and Voice Controls. Next was finding a way to make it all work as if it was still in a car.
My test solutions and workarounds
With the platform built and ready to test the application on, the next question arose. What tests do we actually need to run? When starting a new testing phase for a given platform, the test cases have been written for the chosen platform either by the client, yourself or a fellow team member. In this case, the platform was new to everyone and very little information was available. We approached writing test cases by first examining similarities with the mobile and tablet experience and finding which existing test cases were still relevant to the in-car platform. We then went on to power the head unit up and start writing some generic platform test cases. Some examples below:
- Plugging and unplugging phones in
- Understanding how the device functions from each of the platforms’ view. This is mainly to see if content continues to play or pause when the devices are plugged in or unplugged.
- Working with the Head Unit’s GUI
- See what functionality is available to the user when the devices are plugged into the Head Unit
- Interactions with other apps
- It is important to see how the application under test behaves when switching to and from other apps
- Voice control
- With the devices plugged in, test how the head unit responds to user voice commands.
With the groundwork laid out, let’s build on the foundation
Now that the basic test cases have been written, we need to dive into the complexity of app specific test cases. How will testing on the head unit differ from testing on a phone? Do the same features translate over to the car’s Head Unit? The simplest answer is, no. Android Auto and CarPlay both function very differently then their phone counterparts due to the limitations that are put on a vehicle’s stereo systems. They are designed to be simplistic and easy to use with very little interactions to ensure a safe driving experience.
We built two sets of test cases, one for the mobile platform and one for the car platform. The mobile platform test cases were relatively simple as these platforms have been around for a long time. The CarPlay and Android Auto specific test cases were more challenging as they require new ways of thinking. Features of the app within the context of the platform, thinking of a number of user scenarios specific to a car were things that had to be considered. This resulted in tests like:
- Navigation
- How users might navigate the app while in a car. This step was incredibly important to think about as people will be on the road when operating the app and it’s imperative that the steps to navigate were as simplistic as possible.
- Test the UI against standard overlay for car features
- Certain features required testing within an actual car. For example, back up cameras and side cameras were not something we were able to get functioning with our custom built Head Unit.
- Suspended app, incoming call, cameras
- What happens if a user gets a call while navigating the app. How does it function when the call ends.
- Resume points, create, destroy, etc…
- What happens when a user plugs the app in with content playback already in progress or when a user leaves the car with playback in progress. Does the app maintain those playback resume points? What is the desired user experience for these scenarios?
Additional findings
One thing that we didn’t anticipate was how a minor update to the phone’s OS would impact the behavior of the app on the Head Unit. After the update, we had unfortunately discovered several broken features within the app on the Head Unit. This taught the team that any changes, large or small, to the phone’s OS or to the mobile app, can cause big changes to the app on the car Head Unit.
and tips!
One additional thing to consider when writing tests for a car system is, how you would interact with it if it was in your own personal car? Could you drive carefree with the app functioning in the background with little to no need for interactions? What if you did have to interact with it? Are the features simple enough to provide quick access with little to no hassle? Keeping these things in mind will help not only test case development but also the ability to provide better feedback on the entire user experience.
You will find some more interesting story on our various performance testing here.
Conclusion
One of the core values of Quality Assurance is to not just provide surface level testing but a more in depth look into the product as a whole. As Quality Engineers, we likely spend more time interacting with the app than anyone else on the team. It is of great value to provide feedback to the team and the client. If we are able to test as close to the end user experience as possible, then as Quality Engineers we can better replicate real world scenarios. Which in turn will ultimately help uncover user pain points and find ways to improve the app.
Learn about our unique approached to Quality engineering here.
