New Ambitious Performance Testing Framework For Fixed Broadband Providers Receiving USF Support
The Federal Communications Commission’s Wireline Competition Bureau, Wireless Telecommunications Bureau, and Office of Engineering and Technology have released an Order setting a framework for measuring speed and latency performance that applies to broadband providers receiving high-cost universal service fund (USF) support to serve fixed locations.[1] The testing scheme applies to “price cap carriers, rate-of-return carriers with mandatory buildout obligations, rural broadband experiment support recipients, Alaska Plan carriers, and Connect America Fund (CAF) Phase II auction winners.[2] These broadband providers are required to submit their test results annually, and are subject to audit. Failure to comply will result in a loss of USF support. The first set of testing data, however, is not due until at least July 1, 2020.
Without a doubt, this Order will result in numerous petitions for reconsideration and clarification. There may even be some applications for review of the bureau-level Order. Expect small rural broadband providers to have major concerns with the FCC’s testing framework. Some are obvious – cash-strapped rural providers will be required to expend financial resources on testing instead of deploying infrastructure. Rural providers will have deeper problems with parts of the testing methodology, especially the decision to test connections from customer locations to FCC-designated Internet exchange points. For the largest broadband providers though, the new testing requirements shouldn’t be a problem.
Will anything positive come from the new broadband testing? Yes. It will produce a ton of data. But what will the FCC do with it? What could the FCC do with it? Will it be more than a tool to reduce USF support? Could the data be used to create a new broadband map? Will USAC actually look at the data? Will the data be made public? If so, will anyone and everyone be able to parse the data? These are all interesting questions, but they are all high-level and don’t speak to the problems the testing framework will have getting off the ground. Here is a summary of the FCC’s Order.
Quick Background – There Are Strings Attached To USF Support
In 2011, the FCC required eligible telecommunications carriers that receive high-cost universal service fund support to offer broadband service in their supported areas that meets certain basic performance requirements.[3] Fixed providers must offer broadband service with latency suitable for real-time applications, such as VoIP, and meet a minimum speed standard of 10 Mbps downstream and 1 Mbps upstream. Broadband providers receiving USF support to provide service with higher speeds, such as winning bidders in the CAF Phase II auction, are obligated to meet those speeds. Since adopting performance requirements, the FCC has been developing the specific methodology for broadband providers to use to measure performance, and most recently sought public comment on various related issues in late 2017.[4]
Testing Broadband Performance – Three Options
Broadband providers that receive USF support have three options for testing the performance of their broadband services:
(1) Measuring Broadband America testing infrastructure (MBA testing). A broadband provider may use hire an entity that manages and performs testing for the MBA program to implement the provider’s performance testing. The broadband provider is responsible for all costs required to implement testing of its network, including any costs associated with obtaining and maintaining Whiteboxes, if used.
(2) Existing network management systems and tools (off-the-shelf testing). A broadband provider may use existing network management systems and tools, ping tests, and other commonly available performance measurement and network management tools to implement performance testing.
(3) Provider-developed self-testing configuration (self-testing). A broadband provider may develop its own self-testing configuration using software installed on residential gateways or in equipment attached to residential gateways to implement performance testing.
What is a Test & The Measurement Span
Broadband providers that receive high-cost USF support must measure service performance – speed and latency – from customer premises to a remote test server located at or reached by passing through an FCC-designated Internet Exchange Point (IXP).[5] This distance is referred to as the span of the measurement. Here is what must be tested for speed and latency:
A speed test is a single measurement of download or upload speed of 10 to 15 seconds duration between a specific consumer location and a specific remote server location.
A latency test is a single measurement of latency, often performed using a single User Datagram Protocol (UDP) packet or a group of three Internet Control Message Protocol (ICMP) or UDP packets sent at essentially the same time, as is common with ping tests.
The FCC has designated 16 IXPs: : New York City, NY; Washington, DC; Atlanta, GA; Miami, FL; Chicago, IL; Dallas-Fort Worth, TX; Los Angeles, CA; San Francisco, CA; Seattle, WA; Denver, CO; Salt Lake City, UT; St. Paul, MN; Helena, MT; Kansas City, MO; Phoenix, AZ; and Boston, MA.
The span may include facilities not owned or controlled by the broadband provider conducting the testing. This will be common for small broadband providers, who advocated against such an approach. The FCC concluded testing should include transport – backhaul or transit – that a provider does not control because large and small providers alike are able to negotiate for and purchase “a level of service that will enable it to meet the Commission’s performance requirements.”[6]
There is an exception for broadband providers outside the continental U.S. “[A]ll providers serving non-contiguous areas greater than 500 air miles from an FCC-designated IXP [may] conduct all required latency and speed testing between the customer premises and the point at which traffic is aggregated for transport to the continental U.S.”[7]
Timing: When Must Broadband Providers Conduct Performance Testing
When do broadband providers have to test the performance of their broadband services? The FCC has set requirements for the number of tests per day, the time of day providers must accomplish test, and the number of weeks testing must be conducted.
Tests Per Hour. For latency, a minimum of one discrete test per minute (i.e., 60 tests per hour), for each of the testing hours, at each subscriber test location, is required with the results of each discrete test recorded separately.[8] There are consumer load thresholds to prevent cross-talk – intensive consumer use of the network (such as streaming video) during testing. For latency testing, if the consumer load exceeds 64 Kbps downstream, the provider may cancel the test and reevaluate whether the consumer load exceeds 64 Kbps downstream before retrying the test in the next minute. Providers who elect to do more than the minimum required number of latency tests at subscriber test locations must include the results from all tests performed during testing periods in their compliance calculations.
For speed, broadband providers must conduct a minimum of one download test and one upload test per testing hour at each subscriber test location. Providers must start separate download and upload speed tests at the beginning of each test hour window. There are cross-talk thresholds for speed testing as well. If the consumer load is greater than 64 Kbps downstream for download tests or 32 Kbps upstream for upload tests, the provider may defer the affected download or upload test for one minute and reevaluate whether the consumer load exceeds the relevant 64 Kbps or 32 Kbps threshold before retrying the test.[9] This load check-and-retry must continue at one-minute intervals until the speed test can be run or the one-hour test window ends and the test for that hour is canceled. Providers who elect to do more than the minimum required number of speed tests at subscriber test locations must include the results from all tests performed during testing periods for compliance calculations.
Testing Hours. Broadband providers must conduct tests between 6:00 pm and 12:00 am (testing hours), including on weekends.
One Week of Testing. To capture any seasonal effects on broadband performance, broadband providers must conduct one week of speed and latency testing in each quarter of the calendar year. Testing for all locations in a single speed tier in a single state must be done during the same week. For a provider serving multiple states, testing of each service tier does not need to be done during the same week (i.e., a provider may test its 10/1 Mbps customers in New York one week and in Pennsylvania during a different week).
How Many Locations Must Be Tested – Which Locations Get Tested
The number of locations that must be tested is based on a provider’s total number of supported locations per state. More specifically, broadband providers will be required to test up to 50 locations per CAF-required service tier offering per state, depending on the number of subscribers a provider has in a state. Providers are to test only locations that are reported in the HUBB where there is an active subscriber. Broadband providers must comply with the following scaled requirements for each state and service tier combination:
Subscribers at CAF-Supported Locations per State and Service Tier Combination With Corresponding Number of Test Locations
50 or Fewer – 5 Test Locations
51-500 –10 % of Total Subscribers = Number of Test Locations
Over 500 – 50 Test Locations
If a broadband provider serving 50 or fewer subscribers in a state and particular service tier cannot find five active subscribers to test, the provider may test existing, non-CAF-supported active subscriber locations within the same state and service tier to satisfy the requirement to test five active subscriber locations. The FCC has provided the following examples to help explain the testing requirements:
· A carrier with 2,300 customers subscribed to a single service tier of 10/1 Mbps in one state must test 50 locations in that state, while a carrier providing solely 25/3 Mbps service to over 2,500 subscribers in each of three states must test 50 locations in each state.
· A carrier providing 10/1 Mbps service and 25/3 Mbps service to 100 subscribers each in a single state must test 10 locations for each of the two service tiers—20 locations in total.
· A carrier providing solely 10/1 Mbps service to 30 subscribers must test five locations, and if that carrier is only able to test three CAF-supported locations, that carrier must test two non-CAF-supported locations receiving 10/1 Mbps service in the same state.
· A carrier with 2,000 customers subscribed to 10/1 Mbps in one state through CAF Phase II funding and 500 RBE customers subscribed to 10/1 Mbps in the same state, and no other high-cost support with deployment obligations, must test a total of 50 locations in that state for the 10/1 Mbps service tier.
Broadband providers must randomly select test locations every two years from among their active subscribers in each service tier in each state. Subscribers for latency testing may be randomly selected from those subscribers being tested for speed at all speed tiers or randomly selected from all CAF-supported subscribers, every two years. Any sample location lacking an active subscriber 12 months after that location was selected must be replaced by an actively subscribed location, randomly selected.[10]
High-Latency Providers (a/k/a Satellite Service) – Performance Testing Requirements
Broadband providers (satellite Internet access providers) that cannot meet the 100 ms latency standard must meet an alternative standard. These high-latency providers must demonstrate a Mean Opinion Score (MOS) of four or higher. They must conduct an ITU-T Recommendation P.800 conversational-opinion test. For high-latency providers, the number of testing locations is based upon the number of subscribers nationally for which CAF-supported service is provided. If a provider has 3500 or fewer subscribers, it must test 100 locations. If a provider has more than 3500, it must test 370 locations.
Standards: Compliance Framework Based On Percentage Of Successful Tests
There are specific latency and speed standards – each a percentage of successful tests – that broadband providers must meet to be in full compliance with the FCC’s broadband performance requirements. Broadband providers that do not comply with these standards will be subject to a reduction in USF support, commensurate with their level of noncompliance.
Latency. To be in full compliance, high-cost support recipients serving fixed locations must certify that 95 percent or more of all testing hours measurements of network round-trip latency are at or below 100 ms. High-latency providers that will receive CAF Phase II auction support, must certify that 95 percent or more of all testing hours measurements are at or below 750 ms. All broadband providers must record the observed latency for all latency test measurements, including all lost packet tests. Providers may not discard lost-packet tests from their test results; these tests count as discrete tests not meeting the standard.
Speed. To be in full compliance for speed, 80 percent of a broadband provider’s download and upload measurements must be at or above 80 percent of the CAF-required speed tier (i.e., an 80/80 standard). The FCC provides this example: if a provider receives high-cost support for 10/1 Mbps service, 80 percent of the download speed measurements must be at or above 8 Mbps, while 80 percent of the upload speed measurements must be at or above 0.8 Mbps.
Providers must meet and test to their CAF obligation speeds regardless of whether their subscribers actually purchase service with speeds matching the CAF-required speeds at CAF-eligible locations. The FCC provides the following example: a broadband provider may be required to deploy and offer 100/20 Mbps service, but only 5 of its 550 subscribers at CAF-supported locations take 100/20 Mbps service, with the remainder taking 20/20 Mbps service. To satisfy its testing obligations, the provider would be required to (1) test all 5 of the 100/20 Mbps subscribers and (2) randomly select 45 of its other CAF-supported subscribers, raise those subscribers’ speed to 100/20 Mbps, at least temporarily, and test those 45 subscribers.[12]
Standards: The Penalty For Non-Compliance Is Loss Of Support
There is a four-level compliance framework based on the extent to which a broadband provider’s meets the latency and speed standards in each state and high-cost support program. Each standard will be determined separately. For latency and speed, the FCC will divide the percentage of a provider’s measurements meeting the relevant standard by the required percentage of measurements to be in full compliance. Here is how to calculate latency and speed.
Latency. The FCC will calculate the percentage of compliance using the 95-percent standard. The FCC will divide the percentage of the provider’s testing hours’ latency measurements at or below the required latency (i.e., 100 ms or 750 ms) by 95. For example, if a low-latency provider observes that 90 percent of all its testing hours measurements are at or below 100 ms, then that provider’s latency compliance percentage would be 90/95 = 94.7 percent in that state.
Speed. For each speed tier and state, the FCC will calculate the percentage of compliance relative to the 80-percent-based standard. The FCC will divide the percentage of the provider’s testing hours speed measurements at or above 80 percent of the target speed by 80. For instance, if a provider observes that 65 percent of its testing hours speed measurements meet 80 percent of the required speed, the provider’s compliance percentage would be 65/80 = 81.25 percent for the relevant speed tier in that state.
After calculating the percentage, a provider must determine its level of compliance. There are four levels of compliance, each with a different loss of support penalty.
Level 1 – providers with compliance percentages at or above 85 but below 100 percent. USAC will withhold 5 percent of a Level 1-compliant provider’s monthly support.
Level 2 – providers with compliance percentages at or above 70 but below 85 percent. USAC will withhold 10 percent of a Level 2-compliant provider’s monthly support.
Level 3 – providers with compliance percentages at or above 55 but below 70 percent. USAC will withhold 15 percent of a Level 3-compliant provider’s monthly support.
Level 4 – providers with compliance percentages below 55 percent. USAC will withhold 25 percent of a Level 4-compliant provider’s monthly support. The FCC will also refer Level 4-compliant providers to USAC for an investigation into the extent to which the provider has actually deployed broadband in accordance with its deployment obligations.[13]
To avoid penalizing a provider for failing to meet multiple standards for the same locations, the lowest of a provider’s separate latency and speed compliance percentages determines its obligations. All providers not fully compliant in a particular state must submit quarterly reports providing one week of testing hours test results and describing steps taken to resolve the compliance gap. Whenever a provider in Levels 1 through 3 comes into a higher level of compliance, that level’s requirements will apply, and USAC will return withheld support up to an amount reflecting the difference between the levels’ required withholding but not including any support withheld by USAC for more than 12 months.
There is an exception for providers that serve 50 or fewer subscribers in a state and particular service tier but cannot find five active subscribers for conducting the required testing and instead include non-CAF-supported active subscriber locations in their tests. Any such providers not fully complying with latency and speed standards will be referred to USAC for further investigation of the level of performance at CAF-supported locations.[14]
Annual Data Submission & Certification
Each year, broadband providers must submit speed and latency test results, including data on the technologies used to provide service at the tested locations, for each state and speed tier combination, along with a certification (in a format to be determined). USAC will calculate a provider’s compliance percentages using the provider’s submitted data. Broadband providers will be required to submit the first set of testing data and accompanying certification by July 1, 2020. The first submission should include data for at least the third and fourth quarters of 2019. After that, data and certifications will be due by July 1 of each year for the preceding calendar year.
********************************************
[1] Connect America Fund, WC Docket No. 10-90, Order, DA 18-710 (July 6, 2018) (Order).
[2] Order at Appendix A.
[3] See Connect America Fund et al., WC Docket No. 10-90 et al., Report and Order and Further Notice of Proposed Rulemaking, FCC 11-161, ¶ 86 (Nov. 18, 2011).
[4][4] Comment Sought On Performance Measures For Connect America High-Cost Universal Service Support Recipients, WC Docket No. 10-90, Public Notice, DA 17-1085 (Nov. 6, 2017).
[5] Order at ¶ 18.
[6] Order at ¶ 19. The FCC also notes that the 80/80 speed measurement standard allows sufficient leeway to providers that do not own or control backhaul facilities. See Order at ¶ 53.
[7] Order at ¶ 21.
[8] Order at ¶ 27.
[9] Order at ¶ 28.
[10] Order at ¶ 40.
[11] Order at ¶ 45.
[12] Order at ¶ 51.
[13] Order at ¶ 64. The noncompliance framework resembles the one applicable to interim deployment milestones in section 54.320(d) of the FCC’s rules.
[14] Order at ¶ 66.
