Revisiting DNS and UDP truncation

The choice of User Datagram Protocol (UDP) as the default transport for the DNS was not a completely unqualified success. On the positive side, UDP's stateless query/response model has been a good fit for the stateless query/response model of DNS transactions between a client and a server. The use of a UDP transport enabled the implementation of highly efficient DNS server engines that managed high peak query rates.

On the other hand, these same minimal overheads imply that DNS over UDP cannot perform prompt detection of packet loss, and cannot efficiently defend itself against various approaches to tampering with the DNS, such as source address spoofing, payload alteration and third-party packet injection. Perhaps most importantly, the way UDP handles large payloads is a problem.

The DNS avoids IP fragmentation by restricting the maximum payload size carried over UDP. RFC 1035 contains the directive: "Messages carried by UDP are restricted to 512 octets (not counting the IP or UDP headers). Longer messages are truncated, and the TC bit is set in the header".

The intent of setting the truncation (TC) bit in the DNS response was to indicate that the receiver should discard this UDP response and perform the same DNS query over a TCP transport. This is not exactly a highly efficient measure. The DNS query now takes an additional two Round-Trip Time (RTT) intervals (one for the DNS truncated response and a further exchange for the TCP handshake), and the server also needs to maintain a TCP state, which limits the server's query processing capability. This option to re-query using TCP is preferably avoided but limiting DNS responses to, at most, 512 octets is not always feasible.

This limit of 512 octets becomes problematic in several scenarios. For example, a DNSSEC-signed query for the DNSSEC public keys of the root zone produces a response of 1,169 octets. It's not just DNSSEC that is the issue here. We use the DNS for various forms of authentication, and it's a common practice to load authentication codes into the DNS as TXT records. Multiple TXT records will all be packed into a response, which can lead to quite large responses. For example, a query for the TXT record for elicits a DNS response of 1,666 octets in size.

The workaround for this issue of a very conservative selection of the maximum UDP payload for the DNS was the use of a so-called pseudo-Resource Record, the OPT record. This is the general extension mechanism for DNS or EDNS. The specification for EDNS(0), RFC 6891, includes the option to use a UDP message size in the query to allow the querier to inform the responder of its capability to handle UDP responses greater than 512 octets, thereby avoiding some level of re-querying over TCP when the response is larger than this default size. RFC 6891 also contains the following advice: "Due to transaction overhead, it is not recommended to advertise an architectural limit as a maximum UDP payload size. … A good compromise may be the use of an EDNS maximum payload size of 4096 octets as a starting point."

The IPv6 specification requires IPv6 networks and hosts to be capable of handling an IPv6 packet of up to 1,280 octets in size without fragmentation. The IPv4 specification has an unfragmented packet size of 68 octets, and IPv4 hosts are required to be capable of reassembling IP packets of up to 576 octets in length.

In practice, the original Ethernet packet sizes (64 to 1,500 octets) have been largely adopted by the Internet, and in most cases (where no encapsulation tunnels exist) packets of up to 1,500 octets will pass through the public Internet without triggering packet fragmentation. What this implies is that in proposing a UDP buffer size of 4,096 octets, then IP fragmentation of large DNS responses will be triggered, and all the issues raised relating to the use of UDP fragments may surface as a consequence.

If a primary objective is to avoid IP packet fragmentation, then a UDP buffer size of 4,096 octets is just too large. The advice in DNS Flag Day 2020 proposed the use of an EDNS(0) buffer size of 1,232 octets as a minimum safe size, based on the 1,280-octet unfragmented IPv6 packets, and making allowance for the IPv6 and UDP packet headers. However, this is a very conservative choice, and the downside is potentially unnecessary re-queries in TCP.

A current work in progress in the IETF, draft-dnsop-avoid-fragmentation, proposes that the EDNS buffer size should reflect not only the requestor's maximum packet reassembly buffer size, but also the inbound network interface Maximum Transfer Unit (MTU), and where known, the network path MTU value. This working draft currently recommends a maximum buffer size of 1,400 octets.

Measuring EDNS buffer sizes

Which brings us to our first measurement question. To what extent do recursive resolvers follow this advice? What are the UDP buffer sizes used in queries from recursive resolvers to nameservers?

We have looked at the UDP buffer size in queries in June 2024, as shown in Table 1 and a cumulative distribution of this data is plotted in Figure 1. This is a query-weighted data set, using data from some 356,939,321 query sample points, collected over nine days from 25 June 2024 to 3 July 2024.

Table 1 shows the top 10 buffer sizes used in queries.

Just under 20% of these queries use a UDP buffer size greater than 1,472, which would appear to allow a responder to generate a fragmented UDP response unless, of course, it applies its own more stringent size restraints to the UDP response.

Measuring truncated responses

The next question is: 'How effective is truncation in today's DNS?'

Do DNS recursive resolvers always ignore the answer and additional sections of a DNS response if the truncated bit is set? And what proportion of resolvers can perform a DNS query over UDP in response to a truncated UDP response?

In this case, we have modified the behaviour of the DNS server for the second zone (oob.example.net), such that all UDP responses to queries for nameserver names in this zone are truncated. We also use a categorization of these names such that one half of the experiment's unique name set causes the nameserver to generate a truncated response (TC=1) with an empty answer section and the other half of the query names generates a perfectly normal complete DNS response with an intact and complete answer section, but with the TC bit set indicating (incorrectly in this case) that the UDP response has been truncated.

If the DNS resolver is using the contents of a truncated UDP response, then it will be able to obtain the IP address of the nameserver and make the third query without needing to re-query using TCP. A standards-compliant resolver will ignore the answer section of the UDP response that had the truncated bit set and re-query using TCP and use the TCP response to make the third query.

The behaviours are determined by performing a full recording of all packets that arrive at and leave our servers, and then analysing these packets to determine the DNS resolution query and response sequences for each individual experiment.

In a test conducted over June 2024, we found results as shown in Table 2

Across some 394M sample points, 197M tests were provided with a complete answer section as well as having the truncated bit set. Of these, 96,401 tests did not re-query using TCP, but performed the third target query, evidently using the contents of the answer section in the truncated response. The ten networks with the highest proportional use of truncated answers is shown in Table 3.

These results suggest that globally this aspect of DNS conformance to standards-specified behaviour is not a severe problem, and the incidence of the use of the answer section contained in truncated responses is just 0.05% of all samples. However, as shown in Table 3, the incidence of this DNS resolver behaviour in specific networks is not so small, and this table lists the 10 networks with the highest incidence of the use of truncated responses where more than 100 samples were gathered over the month of June 2024.

Measuring TCP re-query

The second part of a resolver's actions when receiving a DNS response over UDP that has the truncated bit set is to re-query using TCP. The related measurement question is: 'What proportion of resolvers are incapable of performing a DNS query over TCP? '

The overall results are shown in Table 4.

There is a visible level of use of TCP-only here. 439,900 tests performed the DNS resolution by asking the initial query over TCP rather than UDP. This represents 0.11% of the total count of 394M tests. The remaining tests were initiated over UDP and given a truncated response.

Of these 394M tests, 10.5M failed to re-query using TCP, or 2.67% of all such tests. The 10 largest TCP failure rates for networks with at least 500 sample points are shown in Table 5.

Once more, these results suggest that an Internet-wide behaviour to fail when performing a TCP re-query at a rate of a little under 3% of users is not a major issue for the DNS, yet there are individual networks, both large and small where there is a far higher TCP failure rate. Such high failure rates, more than 90%, of tests for users within each of these networks, suggest that the issue is likely to lie in the DNS resolver infrastructure operated by these networks rather than end clients performing their own DNS recursive resolver functions.

Conclusions

The DNS is attempting to steer a careful path between the issues associated with response loss associated with UDP and packet fragmentation and response loss with truncation and re-querying with TCP. Previous measurements of fragmented UDP failure rates in the DNS between recursive resolvers and authoritative nameservers showed a fragmented response failure rate of around 15%, while the current failure rate of truncation and TCP is far lower at some 3%. To improve robustness with large DNS responses, it appears that using a smaller maximum UDP size to avoid fragmentation and re-querying with TCP would be preferred due to lower potential loss rates.

This measurement exercise does not attempt to identify individual recursive resolvers. Modern high-capacity recursive resolver systems are compound systems composed of several DNS resolution engines. Some DNS resolver engines may use only UDP, where TCP tasks may be handed to other resolver engines that are specifically configured to manage the somewhat different TCP load profile. Without undertaking an effort to identify the modes of behaviour of these compound systems, identifying individual resolver systems by their IP address is not overly useful when trying to identify systemic behaviour anomalies.

A report on DNS re-query TCP failure rates can be found at APNIC Labs.

The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.

Public link

Please use the above public link if you want to share this noodl on another website.