Notice: Global Information Announces Listing on JASDAQ Standard Market of the Tokyo Stock Exchange

Cover Image
Market Research Report

5G: The Greatest Show on Earth - Volume 16, Apples to Apples (5G Benchmark Study, with a Focus on Evaluating Overall Network Performance and its Impact on User Experience)

Published by Signals Research Group Product code 990212
Published Content info 58 Pages
Delivery time: 1-2 business days
Price
Back to Top
5G: The Greatest Show on Earth - Volume 16, Apples to Apples (5G Benchmark Study, with a Focus on Evaluating Overall Network Performance and its Impact on User Experience)
Published: February 17, 2021 Content info: 58 Pages
Description

SRG just completed its sixteenth 5G benchmark study, this time with a focus on evaluating overall network performance and its impact on the user experience, based on ETSI TR 103 559. Testing took place in the Dallas metro area, but the analysis of the data and the performance KPIs that really influence the user experience are applicable on a global basis.

Highlights of the Report include the following:

  • Our Thanks. We did this study in collaboration with Rohde & Schwarz. R&S has implemented ETSI TR 103 559 into its Network Performance Score (NPS) framework, thereby providing us with a mechanism to thoroughly analyze how three operator networks in North America perform when evaluated against the ETSI recommendations. SRG is solely responsible for the analysis of the data and the commentary in the report.
  • Background. ETSI has published technical recommendations on best practices for benchmarking mobile networks. These recommendations are based on global operator inputs which define how to collect and score the data, as well as the performance indicators that really influence the customer experience.
  • Our Goal. Although testing occurred in the US, the results and analysis in this report are global in nature. Max speeds are nice for marketing purposes, but they have little bearing in determining the user experience with typical applications and use cases. We prove this point and show why an industry-approved and fully-disclosed approach makes sense for benchmarking purposes. In our test results, the operator with the "slowest network" finished first and the operator with the greatest 5G NR network coverage finished last.
  • Testing Scope. The data collection process involved 115 hours of testing, 5,071 kilometers of driving, 1,300+ voice calls per network, and 60,000+ data-related tests (HTTP browsing, video, social media, HTTP file transfers and capacity tests, as well as interactivity tests).
  • Deep Analysis. In addition to providing the overall scores and the individual scores for each test (all scoring and weighting in compliance with the TR) we delve into why the results varied by operator. We also look at 5G NR/LTE coverage and performance differences between radio access technologies (by operator), including an analysis of underlying RF parameters. Finally, we analyze IP/session information to discuss other critical factors which influenced performance (video MOS, webpage load times, latency, packet loss, etc.).
  • The Results are in. For reasons explained in the report, AT&T came out on top overall (voice and data), followed by Verizon and T-Mobile. Voice results showed very little differentiation between operators, but there were big differences in data performance.
Table of Contents

Table of Contents

1.0 Executive Summary

2.0 Key Observations

3.0 Network Performance Score - Details by Test Type

  • 3.1 Voice
  • 3.2 Data
    • 3.2.1 HTTP Transfer
    • 3.2.2 HTTP Browsing
    • 3.2.3 Social Media
    • 3.2.4 Video

4.0 Data Results and Analysis

  • 4.1 HTTP File Transfers
  • 4.2 HTTP Capacity Tests
  • 4.3 HTTP Browsing Tests
  • 4.4 Video Tests
  • 4.5 Interactivity Tests

5.0 Network Analysis

  • 5.1 LTE Analysis
  • 5.2 5G NR Analysis
  • 5.3 YouTube Analysis

6.0 Test Methodology

7.0 Final Thoughts

Index of Figures & Tables

  • Figure 1. Overall Network Performance Score
  • Figure 2. Voice Performance Score
  • Figure 3. Data Performance Score
  • Figure 4. Points - Call Setup Success Ratio
  • Figure 5. Points - Call Drop Ratio
  • Figure 6. Points - Average Call Setup Time
  • Figure 7. Points - Call Setup Time 10th Percentile
  • Figure 8. Points -Call Setup Time Excess Ratio
  • Figure 9. Points - Voice MOS
  • Figure 10. Points - Voice MOS 90th Percentile
  • Figure 11. Points - Voice MOS Bad Sample Ratio
  • Figure 12. Points - Voice Call LTE Band
  • Figure 13. Points - HTTP Uplink/Downlink Success Ratio
  • Figure 14. Points - HTTP Downlink Average Throughput
  • Figure 15. Points - HTTP Downlink Throughput - 10th Percentile
  • Figure 16. Points - HTTP Downlink Throughput - 90th Percentile
  • Figure 17. Points - Points - HTTP Uplink Average Throughput
  • Figure 18. Points - HTTP Uplink Throughput - 10th Percentile
  • Figure 19. Points - HTTP Uplink Throughput - 90th Percentile
  • Figure 20. Points - Browsing Success Ratio
  • Figure 21. Points - Browsing Duration Average
  • Figure 22. Points - Browsing Duration Excess Ratio
  • Figure 23. Points - Social Media Success Ratio
  • Figure 24. Points - Social Media Duration Average
  • Figure 25. Points - Social Media Duration Excess Ratio
  • Figure 26. Points - Video Success Ratio
  • Figure 27. Points - Average Video MOS
  • Figure 28. Points - Video MOS - 10th Percentile Ratio
  • Figure 29. Points - Average Video Setup
  • Figure 30. Points - Video Setup Excess Ratio
  • Figure 31. HTTP File Transfer Distribution by Technology
  • Figure 32. HTTP File Transfer Failure Ratio
  • Figure 33. HTTP File Transfer Downlink Throughput by Technology
  • Figure 34. HTTP File Transfer Uplink Throughput by Technology
  • Figure 35. HTTP File Transfer Downlink Throughput CDF
  • Figure 36. HTTP File Transfer Uplink Throughput CDF
  • Figure 37. HTTP File Transfer Downlink Throughput CDF by Technology
  • Figure 38. Throughput
  • Figure 39. Distribution by Technology
  • Figure 40. Throughput by Operator and by Technology
  • Figure 41. Browser Test Count by Technology
  • Figure 42. Browser Failure Rate by Technology
  • Figure 43. Browser Webpage Load Times by Operator
  • Figure 44. T-Mobile Browser Webpage Load Times by Technology
  • Figure 45. T-Mobile Relative Webpage Load Times by Technology
  • Figure 46. T-Mobile Browser IP Service Access Times by Technology
  • Figure 47. T-Mobile Relative IP Service Access Times by Technology
  • Figure 48. T-Mobile Browser Payload Downloaded in One Second by Technology
  • Figure 49. T-Mobile Relative Payload Downloaded in One Second by Technology
  • Figure 50. Web Browsing Latency by Operator
  • Figure 51. Video Test Count by Operator and Technology
  • Figure 52. Video Success Ratio by Operator and Technology
  • Figure 53. Video MOS by Operator and Technology
  • Figure 54. Video Setup Time by Operator and Technology
  • Figure 55. Video Setup Time Greater than Ten Seconds by Operator and Technology
  • Figure 56. Video Resolution by Operator and Technology
  • Figure 57. Video MOS Versus Video Resolution by Operator
  • Figure 58. Interactivity Test Count by Operator and Technology
  • Figure 59. Interactivity Average Score by Operator and Technology
  • Figure 60. Interactivity 3GPP QoS Target Application Reached
  • Figure 61. Interactivity Latency - constant long profile
  • Figure 62. Interactivity Latency Distribution - Overall Results
  • Figure 63. Interactivity Packet Error Rate
  • Figure 64. 5G NR and LTE Carrier Aggregation Utilization
  • Figure 65. Aggregate LTE Carrier Bandwidth
  • Figure 66. LTE Frequency Band Count
  • Figure 67. Modulation Type by Operator
  • Figure 68. RB Utilization by Operator
  • Figure 69. SINR CDF for Band 66 by Operator
  • Figure 70. SINR CDF for Band 2 by Operator
  • Figure 71. Application Layer Throughput by Operator
  • Figure 72 .5G NR Throughput by Band and by Operator
  • Figure 73. FR1 and FR2 5G NR Utilization by Operator
  • Figure 74. T-Mobile 5G NR Band n41 Coverage
  • Figure 75. T-Mobile LTE Band 41 Coverage
  • Figure 76. T-Mobile LTE Band n71 Coverage
  • Figure 77. Verizon 5G NR mmWave Coverage
  • Figure 78. 5G NR MIMO Rank by Operator
  • Figure 79. EN-DC Activation Time
  • Figure 80. YouTube 5G NR and LTE PDSCH Data Traffic - AT&T
  • Table 1. YouTube Session Events - AT&T
  • Figure 81. YouTube 5G NR and LTE PDSCH Data Traffic - T-Mobile
  • Table 2. YouTube Session Events - T-Mobile
  • Figure 82. YouTube 5G NR and LTE PDSCH Data Traffic - Verizon
  • Table 3. YouTube Session Events - Verizon
  • Figure 83. Drive Test Route
  • Table 4. NPS Scoring Insights
  • Table 5. NPS Scoring and Weighting Insights - Voice
  • Table 6. NPS Scoring and Weighting Insights - Data
  • Figure 84. NPS Test Plan
  • Table 7. Interactivity KPIs
  • Figure 85. Smart Analytics - YouTube Drill Down with Event Information
  • Figure 86. Smart Analytics - YouTube Drill Down with Enhanced View of Timeline and
  • Figure 87. Smart Analytics - YouTube Drill Down with IP Trace and System Trace Information
  • Figure 88. Smart Analytics - YouTube Drill Down with 5G NR UE Information
  • Figure 89. Smart Analytics - YouTube Drill Down with 5G NR UE Information II
  • Figure 90. Smart Analytics - YouTube Drill Down with GPS Insight
Back to Top