key: cord-0665373-09ccqw0a
authors: Flickinger, Allen; Klatsky, Carl; Ledesma, Atahualpa; Livingood, Jason; Ozer, Sebnem
title: Improving Latency with Active Queue Management (AQM) During COVID-19
date: 2021-07-29
journal: nan
DOI: nan
sha: d0a1860263d49f5953340539c65d2c9fec285b7c
doc_id: 665373
cord_uid: 09ccqw0a

During the COVID-19 pandemic the Comcast network performed well in response to unprecedented changes in Internet usage and video communications applications that are sensitive to network latency have exploded in popularity. However, in today's typical networks-such as a home network-those applications often degrade if they are sharing a network link with other network traffic. This is a problem caused by a network design flaw often described using the term 'buffer bloat'. Several years ago, Comcast helped to fund research and development in the technical community into new Active Queue Management (AQM) techniques to eliminate this issue and AQM was later built into Data Over Cable Service Interface Specification (DOCSIS) standards. Just prior to the pandemic, Comcast also deployed a large-scale network performance measurement system that included a latency under load test. In addition, Comcast happened to deploy two otherwise identical cable modems; one with upstream AQM enabled, and the other without. This fortuitous confluence of events has enabled Comcast to perform a comparative analysis of the differences between cable modem gateways using AQM with those that lack that enhancement, at a unique level of scale across many months of time and millions of devices. The data reveals significantly better upstream latency under load performance when AQM is used. For the device with AQM, most of the latency under load tests resulted in 15-30 milliseconds of latency. In comparison, the device without AQM averaged roughly 250 milliseconds of latency, between 8-16 times higher, a highly significant difference to the end user quality of experience. These large-scale measurement comparisons should provide additional data to justify accelerated deployment of AQM in DOCSIS and other Internet Service Provider networks and user-purchased home network equipment.

driving directions, etc. Even video streaming applications benefit from lower network delay (latency) because they need less initial buffering time before a video can begin playing. These large-scale measurement comparisons should provide additional data to justify accelerated deployment of AQM in DOCSIS and other Internet Service Provider (ISP) networks, particularly in the equipment installed in customer homes, including user-purchased home network equipment.

In 2020, the COVID-19 pandemic emerged as an extremely disruptive global event, leading to seismic changes in everyone's day-to-day lives. In particular, the pandemic prompted a sudden, massive shift to home-based work & school. This led to unprecedented increases in network usage for ISPs across the world, including significant new demands on home networks and local access network segments.

The key application to emerge and achieve mainstream adoption was real-time video conferencing, such as Zoom, which on the Comcast network increased in volume by 316% between February and October 2020 8 . As opposed to other types of applications, such as overnight online backups or background downloads of video game updates, the quality of the end user experience is highly delay-sensitive -meaning it is significantly affected by any network latency. Even simple web browsing is improved by lower network delays, though in this case users have been trained over the last 25 years to simply to be more patient when web pages are sometimes slow to load for no apparent reason.

While it has always been the case that real-time communications traffic is latency-sensitive, widespread adoption of video conferencing in parallel with significantly increased competing LAN traffic in a home during the COVID-19 pandemic has created a perfect storm of using latency-sensitive applications while other Internet traffic is sharing the home network. This can lead users to experience a degraded video conferencing experience -because of "latency under load" or higher "working latency" -during times when more than one kind of traffic is sharing the network.

High latency under load and buffer bloat 9 , are issues that the networking community has been aware of for many years and Comcast and other ISPs and software developers have been working on solutions to this challenge. In the case of DOCSIS networks, the solution has been to shift from legacy Tail-Drop First-In First-Out (FIFO) queues and configurable buffer control in DOCSIS 3.0, to Active Queue Management (AQM) 10 -a feature that is implemented DOCSIS and many other networks and software systems. Enabling AQM on a network interface that suffers from poor responsiveness under normal working conditions improves the performance of all user-facing network applications, and especially real-time, latency-sensitive applications such 8 Based on analysis of aggregate network changes from NetFlow, interconnection traffic exchange, and similar data sources. 9 See Buffer Bloat article, "Buffer Bloat: Dark Buffers in the Internet" by Jim Gettys in the November 29, 2011 issue of Communications of the ACM at https://queue.acm.org/detail.cfm?id=2071893 10 RFC 7567 at https://www.rfc-editor.org/rfc/rfc7567.html is a great background to why AQM is necessary, so is an excellent starting point for readers learning about the AQM and the context into which it fits in the history of the Internet.

as video conferencing, without any negative effects on overall throughput or on delay-tolerant applications.

This paper explains how and where AQM was deployed in the Comcast network. It also compares latency under load performance measurements for those devices that utilized AQM to those without AQM support.

In early 2020 Comcast implemented a new cable modem-based Quality of Experience (QoE) measurement system. This QoE measurement system, notably for this report, included a latency under load measurement 11 . The measurement system was deployed in early 2020 prior to the pandemic 12 13 . It consisted of millions of devices able to run performance tests, thus creating a baseline of latency under load measurements prior to the pandemic.

In addition, of the range of Comcast cable modem gateways and retail cable modems, the Comcast XB6 device has two distinct model variants -one with upstream AQM enabled during the measurement period in 2020 and one without. This provided a fortuitous opportunity for a uniquely high scale side-by-side performance comparison for a specific model of Comcast home gateway at a time of high demand for latency-sensitive applications. Comcast also in late 2020 deployed and optimized downstream AQM on all Arris E6000 Cable Modem Termination Systems (CMTSes) 14 .

Looking at the events and changes of the year 2020, it is clear that latency-sensitive applications such as video conferencing are here to stay. As well, implementing AQM and other low latency improvements can very positively impact end user QoE. Such improvements can also be deployed independently by ISPs and equipment manufacturers, with no need to coordinate with application developers 15 . This should provide motivation for network operators to deploy AQM and other low latency improvements and we believe that latency under load performance will emerge as a competitive differentiator -perhaps eventually on equal footing as connection speed in the minds of consumers once they learn how critical latency under load is to their perception of performance.

Latency is the delay that can occur on a network in sending or receiving packets and this can be caused by a wide range of factors 16 , from the distance traversed on the network or the way that applications and protocols are designed, to load on a home or access network, to the way that operating systems, application platforms, datacenters, network interface cards, broadband gateways, network switches, and network routers function.

Different types or classes of applications have widely differing sensitivities to latency. Generally though, applications can be grouped into two major latency categories: delay sensitive and delay tolerant. Delay sensitive applications include video conferencing, voice over IP (VoIP), online video games, web browsing, desktop remote control, live streaming, instant messaging, streaming video. Delay sensitive applications also include anything where a human user is waiting to see the result of some operation when interacting with a screen or device, such as web browsing, stock quotes, weather forecasts, maps, driving directions, etc. Delay tolerant applications include cloud-based hard drive backups, large game, or other background file downloads such as operating system updates, and peer to peer file sharing (P2P).

Most users can relate to the experience of using a delay sensitive application and then encountering latency. For example, participating in a video conference call trying to sing or play music together 17 or having audio or video intermittently freeze up, or playing an online game and a sudden increase in latency leading to a slow play reaction and lost game.

What users experience is their so-called Quality of Experience (QoE); a subjective measure of the customer's level of satisfaction when using the Internet and various applications. Traditionally, the Mean Opinion Score (MOS) is used as an arithmetic mean of all the individual QoE scores. With advanced applications of Machine Learning, new systems have been developed to predict the customer's QoE based on the measurable and objective QoE metrics and other factors including human perception, past experiences, expectations, and service price. Hence, QoE is a multidisciplinary concept involving engineering, cognitive neuroscience, economics, statistics, and social psychology 18 .

For instance, human reaction time is a combination of different components of mental processing including sensory perception, receipt of input into our consciousness, context applied to the input and decision made based on processing output 19 is perceptible and thus matters to the user's QoE 20 21 22 23 . While mean and median latency is important, for latency sensitive applications even minor increases at the 99 th percentile can affect a user's QoE. In addition, it is also important for applications to have not just low latency but also consistently low latency. Note however, that it is a rare networked application that is not improved by consistent lower network delay; improving latency improves more than just video conferencing and online gaming.

As explained earlier, latency under load can occur at any place along a network path, typically at the entrance of the most constrained link in the path (a.k.a. the bottleneck link). By design, the likely place a user may experience latency under load is in their Internet access device because this is the device where their advertised speed is provisioned. In the Comcast network, like many other ISP networks, the wired and wireless (WiFi) local area network (LAN) functions and routing functions are most often performed by one integrated gateway device. In the Comcast network, that is the cable modem gateway, though customers can instead install their own modem and a separate home router and/or WiFi access point(s) customer premises equipment (CPE).

Comcast's Internet service uses the Data Over Cable Service Interface Specifications (DOCSIS) standard as the data link layer protocol. This enables a network connection from the home's cable modem (CM) to Comcast's Cable Modem Termination System (CMTS). This provides the connection between the customer's home network and the Internet.

The DOCSIS suite of specifications are publicly available 24 , allowing various Original Equipment Manufacturers (OEMs) to design & develop cable modems. Customers have a choice for their cable modem; they can purchase one themselves (e.g., from Best Buy or Amazon) or they can lease one from their ISP (e.g., Comcast). When a customer purchases their own cable modem, they are responsible for administering it, updating the software, configuring it, replacing it if it fails, and so on. These modems are generally referred to as Consumer Owned And Managed (COAM) devices.

An important distinction between leased and COAM modems is support for the operating firmware. For COAM devices, the modem's operating firmware is provided by the modem's manufacturer, who controls the feature set, bug fixes, and firmware release schedule (to the extent that there even are any post-sale software updates). In contrast, leased devices are generally remotely administered, configured, and regularly updated by the ISP, which can bring a range of practical benefits, especially relating to performance and security. The decision to include support for AQM for COAM devices is in the hands of the device manufacturer, whereas in the case of leased devices the operating firmware and configuration of that firmware is controlled by Comcast.

In earlier generations of Comcast's leased devices, the manufacturer still played a prominent role in the development of the operating firmware. But this approach did not allow much flexibility, software release schedules were heavily influenced by the manufacturers, and those release cycles were quite long in duration.

As a result, starting around 2011, Comcast and other cable-based ISPs began to take a more active role in the development of cable modem software. Many cable ISPs have embraced a more open source-based methodology for cable modem firmware development which has led to the current generation of DOCSIS 3.0 and DOCSIS 3.1 cable modem software to being based on the open source "Reference Design Kit -Broadband" (RDK-B) 25 software stack. As the RDK website explains 26 , "The RDK is a standardized software stack with localization plugins created to accelerate the deployment of next-gen broadband products and services by broadband service providers."

With the adoption of the RDK-B software stack, Comcast and other cable ISPs have become the software developers for the cable modems, running on hardware supplied by the Original Equipment Manufacturer (OEM) partners. The cable ISP development team makes the decisions on which software features are included in each operating firmware release and has fine-grained control over which devices to deploy updates, when, and in what numbers. RDK also makes it possible for a range of parties in the ecosystem to develop and release software, but even if a given software module is developed by the OEM, the cable ISP determines if and when it is included in a future software release and when it is deployed to customer devices.

In the Comcast network, RDK-B devices include the Xfinity XB3, XB6 and XB7 devices 27 , with hardware produced by a range of OEMs and System on Chip (SOC) suppliers. The XB3 is based on DOCSIS 3.0, while XB6 and XB7 are based on DOCSIS 3.1. As a result of the new capability to manage and update software that is made possible by RDK-B, as well as the deployment of DOCSIS 3.1, Comcast was able to introduce AQM into the software of one variant of the XB6 gateway beginning in 2017.

Comcast engineers Rich Woundy and Jason Livingood started discussing the concept of latency under load with Jim Gettys, who coined the term "buffer bloat" in 2010. This later expanded to include Dave Täht, one of Gettys' colleagues. In 2011, Gettys began an industry-wide roadshow to explain the problem and possible solutions 28 which was quite influential and helped motivate the technical community to study this issue in greater detail.

The following year, in 2012, Comcast entered into a consulting agreement with Dave Täht to help underwrite development of an implementation of the "CoDel" (Controlled Delay) AQM algorithm 29 . In parallel, several CableLabs engineers including Greg White and Joey Padden started work to assess the impact of the CoDel AQM in DOCSIS networks, noting a 2.7-point improvement 30 38 , along with Dave Täht as well. That same month, in advance of the IETF meeting, this team also conducted a series of executive demos with and without AQM, using a CeroWrt-based router connected to a cable modem, running the Quake online game, running the Chrome browser performance benchmark test, and running a video conference session. This implementation and associated testing demonstrated meaningful quantitative performance improvements, such as probability of gaming latency under network load of roughly 10 milliseconds with AQM compared to 1 second without AQM (100x worse).

In 2014, CableLabs included PIE as the official AQM in the DOCSIS 3.1 specifications as a mandatory feature for the CMTS and cable modems 39 (PIE was also added as an optional feature for earlier generation DOCSIS 3.0 cable modems). At the same time, CableLabs required that DOCSIS 3.1 CMTS equipment support AQM as well but left the algorithm choice to the vendor.

That same year, Comcast provided further financial support for Dave Täht to perform research and development to improve buffer bloat in WiFi network links and equipment, a project he called "Make WiFi Fast" that focused on achieving "airtime fairness" and improved latency under load on the WiFi link.

In 2015, led by Carl Klatsky, with data analytics support from Yana Kane-Esrig, engineering support from Andrew Mulhern and Drew Taylor, as well as guidance from Chae Chung, Hal Kim, and Jason Livingood, Comcast performed limited field trials of a low latency service, using a commonly deployed modem, the Technicolor TC8305C, which supported adjustments to the modem's buffer size through configuration updates. Adjusting the modem's buffer size through this "buffer control" feature served as a pre-cursor technique for latency mitigation prior to AQM implementations being added to the modems. This field trial validated real-world performance differences using latency mitigation techniques in a DOCSIS network and cable modem buffer control configurations were eventually deployed into the Comcast network in 2019.

Comcast has also funded research & development via the Comcast Innovation Fund 40 that was focused on additional improvements in latency under load performance, as summarized below.

• In summary, the deployment of AQM into the Comcast DOCSIS network did not happen easily or by chance. The first step was to get the technical community to recognize that the Internet had a problem and to focus on likely solutions, and that critical early work was done brilliantly and passionately by Jim Gettys. This led to theorizing about, developing, and testing potential solutions over the course of many years of work and experimentation. It would not have been successful without the deep thinking and research of a handful of highly knowledgeable, committed, and passionate engineers and developers, of whom we have mentioned just a few. The success of this effort also is due substantially to work within the open source software community, the involvement of many Internet end users that were willing to do their part to help test, to collaborative development of CableLabs DOCSIS standards (spanning CableLabs staff and contractors, ISPs, and vendors), and finally to implementation support by both DOCSIS vendors and the RDK-B software community.

In DOCSIS 3.0 there was an optional "buffer control" parameter for the cable modem and CMTS that could be used but it was difficult to configure and was a static buffer setting rather than a dynamic AQM 44 . As configured in the Comcast network, when using buffer control with DOCSIS 3.0 devices, this would typically result in a maximum upstream latency of roughly 250 milliseconds.

But buffer control is not as flexible as AQM, which has the advantage of being able to absorb temporary traffic bursts while supporting smaller average latency and maintaining high throughput. After DOCSIS 3.0 was initially developed, a specification change was issued that optionally added the PIE AQM to DOCSIS 3.0 in 2014 45 for cable modems, but by that time development focus had largely shifted to DOCSIS 3. Of the Comcast leased cable modem gateway devices, RDK-B software is running on XB3, XB6 and XB7 50 . AQM development has been focused on DOCSIS 3.1-based devices, such as the greater than 10 million XB6 devices and the latest generation XB7 device. As well, customerowned DOCSIS 3.1 cable modem devices also should support AQM.

For the XB6, there are two different hardware manufacturers, which means two variants of XB6. Those are the Technicolor CGM4140COM and Arris (now CommScope) TG3482G. The XB6 is the focus of this analysis because one of these variants -the Technicolor device -had AQM enabled from its introduction in October 2017, whereas the other -the CommScope device -did not during the study period 51 . This is simply a result of differences in the chipsets used and the readiness and ability to add AQM code 52 .

There were roughly 4.3M of the AQM-enabled Technicolor devices deployed as of December 2020, in comparison to 2M non-AQM-enabled CommScope devices. In the early part of 2021, AQM was enabled on the remaining variant of XB6.

This unique situation of two variants of the same device, deployed to millions of homes in the United States, one of which has AQM and one that does not, presented the opportunity for an extremely interesting comparative case study in the performance impacts of AQM. This direct comparison is much more compelling than contrasting a newer XB7 with an older XB6, given greatly differing CPU, memory, radio, and network interface specifications. It provided an unparalleled opportunity to directly compare latency under load with and without AQM at a time when latency sensitive video conferencing and other Internet traffic had dramatically increased.

Previous latency testing options either introduced too many variables to make any definitive conclusions or did not offer the ability to scale in a way that would offer insight to ongoing network trends. As a result, Comcast uses a custom Network Performance Measurement System that leverages open-source software. This system aims to eliminate as many user device and WiFi-related variables as possible, focusing on those conditions and variables over which Comcast has direct control, while also being able to work at high levels of scale 53 . Figure 1 shows a high-level view of the system with the following components:

• Measurement Orchestration (orchestration) maintains a holistic view of the network between the client and the servers and initiates measurements accordingly. • Measurement Client (client) will, when instructed by the orchestration, run upstream and downstream measurements to and from the Measurement Servers. • Access Network is the internal Comcast network, from the DOCSIS portion of the network to which users connect as well as regional and backbone portions of the network. • Measurement Server (server) dedicated hardware servers responding to measurement requests. The platform actively introduces test traffic rather than passively measuring user-generated traffic.

The client is built into the RDK-B firmware that is running on the customer premise-based cable modem gateway. The orchestration function takes in measurement requests and determines whether the measurement will have enough available measurement system resources to sustain the peak load and duration of a given test. That resource check is performed on both the client and the server.

If either of those checks fails, the measurement request is rejected, and the test will be tried again later. If the measurement is accepted, the orchestration function will send a request to the client with all the parameters needed to perform the measurement, such as the type of measurement, duration of measurement, and server(s) to use. The orchestration function also instructs the server to reserve those resources for the client and then the client will start its measurement test cycle. Once the measurement test cycle is complete, the client will request that the server reallocate the reserved resources back into the server's available resource pool. The server then communicates any allocation or deallocation of resources back to the orchestration function.

The first measurement implemented at scale was a network capacity measurement, more generally known as a "speed test". Following NetForecast's design audit of the system, they also performed an audit of the results of speed test measurements 54 .

Latency is one of the biggest influencers of end user application performance but can vary greatly based on routine network usage (when a network flow is filling a queue on the network path between two hosts). To determine the impact to latency during periods when a data flow is causing queues to be filled to capacity the system uses a measurement that will simultaneously load the network and measure latency. This measurement, aptly named latency under load, utilizes iperf3 55 to load the network in either the downstream or upstream direction between the client and a server. Figure 2 shows a high-level view of this test.

Latency testing begins as the TCP ramp-up of the bandwidth measurement reaches its peak steady state and finishes before the bandwidth measurement connection is closed. The latency measurement sends UDP datagrams back and forth between the client and the server and measures the round-trip latency multiple times over a set interval.

54 See NetForecast report at https://netforecast.com/wp-content/uploads/Comcast-Phase-II-Audit-Report-NFR5134F.pdf 55 For more information on iperf3, see https://software.es.net/iperf/ This test uses netperf to measure the delay after every UDP request/response, defined as "UDP_RR" 56 . The duration of the test is configurable and in this case is set at 11 seconds for the downstream measurement and 7 seconds for upstream 57 .

As explained earlier, for two variants of XB6 cable modem gateway, upstream DOCSIS-PIE AQM was enabled on the CGM4140COM (experiment) variant but was not available on the TG3482G (control) variant during the measurement period. The TG3482G variant used a buffer control configuration that predated AQM in DOCSIS 58 . While measurements were performed throughout the course of 2020, the figures that follow represent over 26,000 tests run using random samples of RDK-B cable modems across the Comcast network in the United States between October 1 and December 31, 2020.

At a high level, when a device had AQM it consistently experienced between 15-30 milliseconds of latency under load. In contrast, without AQM but using the older buffer control configuration, a device would experience significantly higher and more variable of working latency. Those non-AQM devices experienced in many cases 250 milliseconds or higher latency under load. This demonstrates two key points: not only does AQM enable significantly better latency under load performance, but that latency performance is much more uniform and consistently (e.g., relatively lower jitter). Both are excellent outcomes that should improve end user quality of experience for all Internet applications.

Below are two Cumulative Distribution Functions (CDFs), comparing both mean and max latency under load for the two XB6 variants. The CDFs and the bar chart that follows are comprised presents the variability across all devices of a given type and compares the two device variants.

These two CDFs in Figures 3 and 4 show that, whether on average or in the worst case, nearly all devices with AQM will experience 15-30 milliseconds of latency under load. At the same time, without AQM, will experience much higher latency under load. When comparing the mean to the maximum in Figures 3 and 4 , respectively, it is also apparent that the devices with AQM perform quite similarly. This suggests that the latency performance is consistently good.

56 See netperf documentation at https://hewlettpackard.github.io/netperf/doc/netperf.html#UDP_005fRR 57 This period is after TCP ramp-up. The downstream interval is longer as it includes some omit intervals. Please refer to the netperf documentation for additional information 58 So, when comparing with and without AQM, it is really with AQM or with buffer control. Figure 4 below shows the CDF of the maximum (worst-case) latency under load performance. The device that lacked AQM, but had an older buffer control configuration, experienced much higher latency compared to its mean in Figure 3 , whereas the mean and max for the AQMenabled device is quite similar. It is also interesting to observe that there was a much longer tail of performance of the non-AQM device, trailing off from a roughly 0.7 cumulative frequency to performance levels that stretch over 1 full second in some cases. The following Figures 5 and 6 show the distribution of mean upstream latency for each of the two device variants under study. Figure 5 shows the variant without AQM, while figure 6 shows the variant that has AQM. The differences between the two distributions are quite interesting. In Figure 5 without AQM, there is a broad distribution from 15 milliseconds to several hundred milliseconds, with only 1% of results in the 15-30 millisecond range. In contrast, in Figure 6 with AQM, the distribution is tightly grouped between 15-45 milliseconds, with 77% of the results in the 15-30 millisecond range. As with the prior CDFs in Figures 4 and 5 , this view of the data demonstrates not only substantially better latency performance with AQM but also much more consistent performance. 

Just prior to the start of the pandemic, Comcast deployed a new Network Performance Measurement System that included a test for latency under load (working latency). That type of latency is central to an end user's Internet quality of experience (QoE), as became quite clear during the pandemic as a significant increase in video conferencing and other interactive Internet applications took place. Comcast moved quickly to optimize Active Queue Management (AQM) on the CMTS affecting downstream latency as a first step to ensure best QoE performance during COVID.

In addition, AQM functionality was deployed in cable modems in the Comcast network affecting upstream latency, though not at the same time on all CPE devices. That difference in deployment timing enabled us to study the performance of two variants of the same RDK-B-based cable modem gateway, one with and one without AQM for a period of time.

The data show that the latency under load performance differences between CPE devices with and without AQM is significant. At a time when users are more intensively using and depending upon latency-sensitive applications such as video conferencing, remote learning, and gaming, AQM is clearly a key tool to maximize their quality of experience. AQM also improves the user experience for all Internet applications where a human being is involved, such as web browsing, stock quotes, weather forecasts, maps, driving directions, etc. This improvement in QoE is also possible without any coordination with application developers or others, taking advantage of the loosely coupled architecture of the Internet and greatly simplifying any ISP and equipment manufacturer deployment of these performance improvements.

All Comcast DOCSIS 3.1 RDK-B-based gateway models have now been updated with DOCSIS-PIE AQM 59 and all are achieving dramatically improved working latency.

Latency under load is highly likely to emerge as an aspect of Internet service that is marketed to end users and which differentiates both ISPs and user-purchased CPE devices, particularly given end users' increasing adoption of latency sensitive applications. ISPs and CPE manufacturers should therefore continue to invest or start investing in AQM deployments, as Comcast continues to do, which will have an increasingly important and positive effect on end user performance now and in the future.

Beyond this type of AQM deployment, CableLabs has released the Low Latency DOCSIS (LLD) specifications for DOCSIS 60 61 . This may significantly improve latency under load as compared to earlier implementations of AQM but is not yet fully implemented in cable modem and CMTS software deployed by Comcast and other DOCSIS-based ISP networks. Additionally, some of the enhancements rely on standardization work that is currently ongoing in the IETF 62 . But the prospect of reducing network delay to below 5 milliseconds at the 99 th percentile could well enable the creation of entirely new classes of applications that were previously impossible or unimaginable, which is incredibly exciting for the future of the Internet.

Greg White, and Rich Woundy. We also wish to thank Aaron Tunstall for his contribution to the collection and analysis of the data, as well as Olakunle Ekundare for additional data to help contextualize these findings and Mulbah Dolley for performing analysis of related third-party datasets that helped to validate the latency trends we observed over the course of 2020.

Document Update Log: -14 February 2022 -After publishing the first version on 30 July 2021, the document was updated to correct the labeling of Figures 5 and 6 . The cable modem model numbers were correct, but the manufacturer names were transposed. In addition, the parenthetical was added to the X-axis to note that another name for latency under load was working latency.

Approaches to Latency Management: Combining Hopby-Hop and End-to-End Networking

A Non-Queue-Building Per-Hop Behavior (NQB PHB) for Differentiated Services

Some DOCSIS 3.1 features such as AQM had to be disabled shortly after deployment began due to performance issues. The cause was eventually traced to DOCSIS 3.1 PHY configuration issues and resolved

The CommScope XB7 (TG4482A) was fully enabled by February 1, 2021. AQM enablement for the TG3482G version of XB6 began the week of

Including assignment of a DiffServ code point and an Explicit Congestion Notification (ECN) codepoint

The authors wish to thank the following individuals for reviewing drafts of this paper and providing valuable feedback: Bob Briscoe, Stuart Cheshire, Jim Partridge, Dan Rice, Dave Täht,