How Accurate Are Studies Cited By Wireless Carriers?

10083494704_fb1ea93467_nVerizon Wireless has touted the results of a new study published by wireless analytics company RootMetrics as proof that they are ahead of the competition. The study, which occurred during the latter half of 2013, showed that when looking at network reliability, call completion and quality, test message service, and data speeds, Verizon was more successful than other wireless networks.

According to a CNET post, “the reality is that whether Verizon, AT&T, or T-Mobile are touting test results, it’s difficult to compare because the actual tests used to boast about these claims aren’t measuring the exact same things.” There were several examples in the article, including the fact that the study examined performance across the entire country (including 4 LTE and 3G networks), yet Verizon has most 4G LTE deployment and therefore had the advantage in testing. So how much weight should consumers give to such studies- by any wireless provider.

Aleksandar Koprivica is a core network engineer for a mobile phone company. “I would have to agree with the CNET article stating that the coverage, or rather lack of it, can lead to misinterpretation of the results,” Koprivica says. “That is how I understood the methodology used in this testing. So if a network does not have coverage in the area tested its score would be zero for that area, thus lowering the final score. From the customer’s point of view the quality (reliability and speed) of the network is important in the area(s) customer is visitingand the broad results presented here are not compatible with the real user experience. I can only imagine a sales person traveling throughout the country would find these scores helpful when choosing an operator.”

Debashish Bhattacharya, an independent market and technology consultant for policy and regulatory issues in South Asia, believes it is a battle for No. 1 ranking amongst carriers in terms of reliability, call completion & quality, and data speeds for the LTE network. “Verizon Wireless cites the results of RootMetrics, which is an analytics firm that tests the quality of the network, in claiming the most reliable network, while AT&T points to its results as one of the tests confirming its speedier network,” explains Bhattacharya. “The world of network testing is anything but transparent. Some research firms will compile their data anonymously, which means they don’t publicly disclose the results or how they came upon them. The different firms vary on how public they want to be, partially because they don’t want to be seen publicly endorsing one carrier over another.”

There does not appear any one test considered to be the industry standard. “Each of the wireless carriers subscribes to individual network drive tests, partially to augment and verify their internal tests, but also to see where they stack up with each other,” continues Bhattacharya. “Firms such as J.D. Power, Nielsen, and RootMetrics conduct such tests, while publications such as PCMag also conduct their own speed tests. So when carriers do make their claims, they aren’t idle boasts — they are in fact backed by studies. While RootMetrics is a younger firm that’s more eager to generate a little public buzz, the large carriers take its results seriously.

“Many analytic firms use data from publicly downloaded apps that allow anyone to download and run tests in their own neighborhoods and use these hundreds of millions of data points gathered from hundreds of thousands of individuals in their result formulations. These results are also not accurate either. That’s because the crowdsourced data is “messier” since there’s no way of telling if a person is inside, outside, or in a crowd. Without phones from other carriers, there’s no way to get an accurate comparison. The network-testing industry lacks consistency in measuring performance. Given that each firm has their own proprietary method of conducting tests, it’s unlikely that any one will come to an agreement. There has not been and there needs to be a standard.”
Zintro has experts in every industry sector, across every job function, in every geographic region. Recently, some of the following topics have seen inquiry activity: