Performance Testing Mobile Networks

There has been a huge surge in the demand to access a variety of popular applications via mobile devices. This article discusses 2 tools The demand for access to business information and websites through mobile technologies has caused the business managers to drive mobility as part of their business technology. It is becoming imperative for companies to have their content effectively available on the mobile devices.

By Shashank Bhardwaj January 9, 2015 for Akamai

Performance testing to help maximize the end user experience has become a key component in the drive to have an effective mobile-based application or website.

I work as a performance engineer for Akamai and the key question we face is “How does the website perform on mobile networks?”

Advertisement: Gravitylab: Mobile video experts.

Challenges for Testing Mobile Applications

Before I asses the testing tools available, I would want to address the challenges present when accessing applications and web sites via mobile devices.

Poor Network Quality, High Latency

Network characteristics inherent in today’s mobile infrastructure have a significant impact on the response times for the client. Mobile networks can have poor quality, can be extremely “lossy” and experience high latency when compared to regular “wired” networks. These issues cause the response time at the client to be high. One should always consider these “mobile challenges” while testing. Response times of small objects will be masked by the inherent network timings. We should look for bigger objects.


There has been a rapid growth in the mobile market and we have different mobile devices with different sizes and different operating systems. Modern day websites adapt themselves based on the device characteristics and hence this must be accounted for when testing. This is a challenge since there is an ever-increasing change in the development of mobile devices and the testing platform needs reflect the current technological landscape.

Testing tools

I will keep the scope of this blog limited to testing of web applications, focusing on the factors one should keep in mind when evaluating testing platforms. . Factors to consider include:

  • Testing methodology.
  • One should be clear how the tests would be run and how often.
  • Devices used.
  • We should be aware if the testing tool accurately simulates the web application over the mobile network.
  • Environment configuration.
  • How are the testing agents configured? Do they actually test mobile networks? How are they connected to the ISP?
  • Geo spread.
  • What is the deployment of the testing agents? If I want to test globally will the tool have sufficient testing platforms?

Ease of use.

How easy is it to create a test script? How easy is it deploy the tests and how quickly can meaningful results be generated?How easy is it to review and interpret data?

Two tools which meet all of the above needs are Dynatrace (previously known as Gomez) and keynote. These are industry standard tools and used for a significant amount of performance testing. Why do I like these platforms?

  • Testing scenarios range from a single object to a click based interactive transaction.
  • The tests are run from various testing agents across the globe. Testing frequency is easily configurable from one test run per hour to 12 tests per hour.
  • The testing environment is consistent.
  • They testing agents are configured to use appropriate name servers.
  • They use actual mobile networks across various carriers.
  • They have a variety of mobile devices which can be tested.
  • They provide various ways to report and interpret data.
  • There is flexibility to see numbers by region/country/city/ISP etc.
  • The times report can be divided on first byte time, connection time, content download time etc.

These results provide a great insight into the performance of website on mobile networks.

The above factors set them apart from many other tools present in the market. The results from these tools are trusted and are used in website ranking. NPS scores also take these rankings into consideration.

Apart from the above I have listed the below key advantages and limitations of each of them.

Easy to create test scripts.
Features such as domain filtering and host to IP binding as available.
Extensive deployment of testing agents globally
Ability to simulate mobile and Wi-Fi networks.

SMS service not supported
Does not use actual browser. Browsers are simulated
Does not support flash

Uses actual browsers.
Extensive support for a wide variety of mobile devices

Less deployment of testing agents as compared to Dynatrace.
Advanced features such as host – host binding not present.
Seeing the waterfall costs extra.
The benefits of adding RUM

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Real User Monitoring is an Akamai tool, which provides performance monitoring of web pages based on the data collected from real end users. With RUM we are able to gather data about an individual page request.

For Akamai RUM becomes a critical tool to:

  • Monitor consumer experience when interacting with the website.
  • Verify the results of the synthetic tests.
  • Adjust synthetic tests based on real end-user data

While synthetic testing simulates users, and is repeatable and often reliable, it does not provide data on actual end-user experiences. As RUM measures performance directly, the data RUM collects can be filtered by geography, browser, and platform, one can drill down to see what subset of end users are experiencing problems. As one collects and analyze RUM data, one may notice trends may cause one to alter the synthetic tests to better reflect average customer’s experience with the site.


At last I want to conclude by saying that each of these tools have a wide range of testing and reporting capabilities. One should understand how to make best use of them. Going for the cheaper option should not be selection criteria. The tool should be selected which best fits the need of the application. Getting accurate data should be the highest priority. Only then can one understand the true behavior of their application over the mobile network.

This article is written by Shashank Bhardwaj for AkamaiMobile Networks – Performance Testing