Ensuring app quality for media streaming products

The importance of understanding customer demands to improve the quality of over-the-top media solutions

The media industry has been undergoing a major transformation over the past couple of years. In particular we have seen the core trend of cord cutting where traditional media providers have been losing subscribers. As a result they have needed to move to a more flexible model, with a better and wider variety of offerings.

This scenario creates two key challenges for media providers to tackle:

1. How can traditional media providers transform their business to stay relevant in the industry?
2. How can providers position and maintain an over-the-top (OTT) product in a vast market with many options, and that is rapidly evolving?

Underlying these two points we have to explore the third question, what do customers want? And this question takes us directly to the quality required for this kind of product.

How do we define quality in an OTT product or service?

When we want to talk about the quality of a software product there are some elements which are applicable for every industry – for example reliability, business compliance, performance. But we also have some specific quality aspects that customers consider when choosing which media platform they want. These include:

• Portability. Customers want to see and listen to their favorite series, movies or podcasts everywhere and not be tied to a unique device. That’s the reason why it is important to have a wide spectrum of platforms covered by applications. Some of the favorites nowadays include iOS, Android, Roku, Amazon Fire, Android TV, Apple TV, Chromecast and even smart TVs like Samsung TV or LG webOS. When people want to buy a TV they’re not only looking at the screen size and resolution, they’re also looking for its smart features and the capability to have specific applications. When developing an OTT application it’s important to consider this vast ecosystem and also the way you’re going to offer a unified experience.

• Content. Customers will often be looking at the content offered, rather than the usability or features. An application’s success increasingly depends on the strength of its video-on-demand (VOD) or live catalog offering. Conversely, if content cannot be watched, or the streaming is corrupted or with bad quality, there is a much higher risk of business failure.

• App simplicity and its attractiveness for users. People don’t want to navigate through several menus, carousels of content or even search options to see the content they want. They’re looking to find content through simple navigation which shows the different content options available. This is also tied to new features with smart assistants, where they ensure content automatically starts playing after a simple voice command.

• Personalization. This is linked to the previous point of creating an app that is attractive to customers. Having a single subscription for the whole family with different profiles and segmented content for each person has proven to be well liked. Smart recommendations of the most popular content, recently added titles, and content related to peoples’ interests and consumption habits have all become popular ways to create a more personalized experience.

• Performance. These kinds of applications must have the robustness and capacity to respond to hundreds of thousand users (even millions) spread across the globe. Let’s suppose the application has the new release of a top series or will stream the final match of an important sports tournament, customers will have little tolerance if they are unable to watch due to high demand.

These are just some of the key aspects to consider when we talk about the quality of a media OTT product, beyond just its functionality.

Best practices to ensure application quality

The key question here is, how to deal with all these different aspects and ensure the application’s quality? That’s why we are defining a new way to validate quality for such applications and are adapting traditional practices with new approaches in a unified service. Some of the key points of our approach are:

• Shift-left quality. Injecting quality from the beginning of the development lifecycle, and defining quality gates for each stage (for example definition, design, development, QA) are at the core of our approach. In addition we recommend including automation in your CI/CD pipeline, implementing techniques such as test driven development (TDD) and behavior driven development (BDD), and even static analysis techniques that will help you reduce extra costs derived from the expansion of defects across multiple stages and platforms – especially for these kind of products that usually have several platforms and devices with the same definition and code base. Based on our experience you will be able to reduce testing time by up to 30% by implementing in an efficient (an agile) way some of these techniques, thus improving the time to market of your products.

• Content testing. As we highlighted earlier, content is particularly important for the customer experience. This means you need to ensure content quality – this requires checking elements such as the metadata, video resolution, image cards, dubs and subs playback, buffering, entitlements, blackouts. There are several ways to check those characteristics but the challenge is to have a test automation strategy to validate thousands of assets while still maintaining the dynamism to be uploading new content every week. This automation strategy will be more effective if you can complement it with the manual execution of some complex automated scenarios, such as lip sync or video accuracy (for example if an individual wants to see a Star Wars movie, it really is Star Wars and not another movie which has been wrongly tagged). It also requires a deep understanding of the backend calls and responds that allow you to validate video resolution, metadata, dubs and subs, digital rights management (DRM), entitlement and even playback consistency, without using the front end (this considering it takes longer to execute one test escenario using the front end than checking directly the endpoint call).

• Performance. Of course you have to validate it, but the question here, is how? This does not just involve the traditional approach of simulating concurrent users using the most common features, but here the challenge is to simulate real events like the season premier of a series or the big game of a sports event. It involves simulating users from different platforms distributed across different countries and performing tasks such as signing up, logging-in in short periods of time and then streaming for several hours. This involves checking not only the infrastructure, but also the handling of the sessions and the buffering, among other things.

• API validations. These types of applications need to be spread through different platforms. It’s necessary for the architecture of the system to involve a common middleware that works as a bridge between them, unifying the business logic. Here it becomes important to have the capacity to have an automation framework that validates that the middleware is integrated into the CI/CD pipeline and validates not only individual behaviors of each endpoint, but also simulates end to end scenarios through the different endpoints. This will ensure testing coverage through the development cycle.

• Cross device testing. It is important to validate the front end in different platforms, but how can you define the scope for this? Is it the same to test one or another model of a smart TV? What are the differences between different versions or brands? What is the target market and what are the most used devices? Don’t underestimate low usage percentages, 5% of 1 million users with a bad experience on a specific platform or model implies 50,000 unsatisfied users. There are several topics to consider here:

1. What is the scope of devices/models/brands to cover?
2. What are the minimum scenarios that can only be validated on the front end and that differentiate one platform from another, and that must be validated?
3. How can I have a unified automation framework to allow the regression of those scenarios across the different platforms, thus avoiding large maintenance times of the scripts for each platform?
4. Should I use a cloud devices farm? There are services, especially for smartphones and tablets that allow you to increase your coverage range. But for 10ft UI devices of smart TVs there are fewer options. There are therefore extra service costs and it’s important to compare this with the cost of having your own devices, not only for one release, but for the planned program roadmap of the app.
5. Should you use crowdtesting services? This is a good option if you have a beta version of the application and want to check the user experience through different countries and devices. Another good option is to do a “friends and family” validation with people in your company. Both of these options are complementary actions to consider in a UAT phase looking to expand the devices’ coverage.

• Billing and analytics validations. You have to ensure that the application is monetized effectively, which implies at least two things: 
  • Are all the billing transactions performing well?
  • Are the analytics generated through the user’s usage reflecting the user’s consumption that will help in defining marketing strategies?

As many of the other characteristics, these two uses have a high dynamism in production, with price changes, promotions, new plans, upgrades or downgrades options and many more. Plus, of course, analytics to track those behaviors in production. 

It’s important to have a clear strategy on how to test the billing transactions, and how to validate the different scenarios that can trigger a pixel or action in your analytics. And yes, you’ll also need to consider these through the different platforms and countries the application is available.

With regards to the analytics, you have to consider implementing specialized test automation frameworks that allow you to validate these features in an efficient way. To be effective with these frameworks it’s important to consider their integration with the functional frameworks at the execution level, to have a better coverage of validations with different points of views. For example, if you are validating the signup of a new user, if you integrate the functional framework that allows you to see the user has successfully registered and can access to the platform, you can also trigger the event in the billing framework to check the right charge in the credit card, and also with the analytic framework checking the analytic for the signup is triggered and the pixel for a specific campaign was also generated. This is how to get efficient executing testing.

There are many other aspects to consider here, such as the UX, AI, big data, geolocation to mention just a few. But at the end the important point is to have a clear and efficient strategy of how to validate the different aspects of these kinds of applications, allowing the dynamism and short time to market that organizations in the industry require.

Here at Globant we have developed the Media OTT service framework to help media organizations create powerful experiences. We have built this based on our experience supporting many of the top players in the industry. Are you ready for the next step?