The last year of IPv6 HEPix testbed operation

A summary of encountered technical issues

Francesco Prelz

INFN, sezione di Milano

(and members of the IPv6 group)

Summary

Just a progress log of various issues we have on our table:

1. Globus.
2. UberFTP.
3. SL6, glibc, RFC3484 (6724) and listening sockets.
4. GNOME NetworkManager.
5. Rogue RAs, RFC6104 and ramond.
6. Conclusions, bigger picture.

Globus/GridFTP

• The main IPv6 issue that was found so far in Globus is that the I/O library and the FTP client library disallow IPv6 by default and, up to Globus Toolkit v5.2.3 included, allow to override this via an API call only.
• As reported at the last HEPix spring meeting, we have been having a GGUS ticket open all along to track this issue.
• Something started moving for real in GT 5.2.4 (February 5^th, 2013), but, as reported in both the GGUS and the Globus tickets, it looks that only the Globus IO library was changed by adding an IPv6 preference environment variable (GLOBUS_IO_IPV6), while the FTP client was so far left as-is.

FTP at large

In case your mind started objecting that IPv6 should be purely a transport issue let me remind that:

1. Postel et al.'s RFC959/765 FTP is possibly the earliest service that implied the underlying IPv4 address structure in its PASV and PORT commands.
2. These were "fixed" in RFC 2428 (Sep. 1998). For example:
   PORT 132,235,1,2,24,131
   becomes EPRT |1|132.235.1.2|6275|
   allowing EPRT |2|2001:123::200C:417A|6275|.
3. Additionally, the Gridftp protocol includes commands for striped transfers (SPAS, SPOR), that do handle both address/port syntaxes in a possibly non-documented fashion.

UberFTP

• UberFTP is a GridFTP client tool.
• Still no IPv6 support, either at the transport or at the 'extended' FTP command level, in the last release (2.7, December 2012).
• We wrote and submitted an IPv6 patch on April 6^th, 2012.
• Jason Alt, UberFTP's author, then moved the code to github.
• A fork was created with the IPv6 patch: prelz/UberFTP. The fork was so far kept in sync with the upstream code.
• Is the best next step a friendly poke or a GIT pull request ?
• Note: as UberFTP always does a reverse DNS resolution of the target host (and uses the name it finds) there's no way to force it to use IPv4 when the canonical name of a host resolves both in IPv4 and IPv6.

IPv6 RFCs, glibc and listening sockets (1)

• When we first upgraded a testbed node to SL(C) 6, we were surprised to find that the GridFTP server stopped listening on IPv6.
• While this sounds like one of those backsliding cases we are most afraid of, it is rather a sophisticated by-product of RFC3484 (Default Address Selection for IPv6) and RFC4291 (IPv6 Addressing Architecture).
• In order to find a sockaddr where to bind, listening services call getaddrinfo() with ai_req.ai_flags = AI_PASSIVE; ai_req.ai_family = PF_UNSPEC;.
• With the glibc version in SL5, the IPv6 'unspecified' address (::) is returned: this causes binding on both stacks.
• With the glibc version in SL6 (any glibc version from 2.9 on) the IPv4-only address is returned.

IPv6 RFCs, glibc and listening sockets (2)

• A May 14^th 2008 glibc commit fixed, according to RFC4291, section 2.5.3, an asymmetry, where the IPv6 loopback address wasn't being handled as a link-local address.
• This exposed another IPv4/IPv6 asymmetry, in RFC 3484, section 2.1, where the loopback and unspecified IPv6 addresses are assigned different labels while all of the IPv4 space, including loopback, is assigned the same label.
• As getaddrinfo currently applies the RFC3484 preferences to unspecified/loopback address pairs in both IPv6 and IPv4, this leads to preferring IPv4!
• IPv6/IPv4 symmetry can be restored by adding the /etc/gai.conf configuration file, where only one extra label statement is added to the RFC3484 default:
  label ::ffff:7f00:0001/128 8.
• More details are in the working group Wiki.

IPv6 RFCs, glibc and listening sockets (2)

• If we agree that IPv6 preference should be maintained throughout unless specifically removed, how/where should we best address this issue ?
1. Ask all software providers to reorder/scan the list obtained from getaddrinfo() ?
2. Work around the issue in glibc as suggested in (open, unassigned) glibc Bugzilla ticket #14967 ?
3. Amend RFC 3484 ? Actually RFC3484 was already amended last September as RFC 6724. There's no change in the default labels, but this note was added in Section 2:
   Finally, a third application scenario is that of a listening application choosing on what local addresses to listen. This third scenario is out of the scope of this document.
   Does this allow glibc to freely restore the IPv6 precedence ?
4. Install /etc/gai.conf in our Linux distributions ? A post on the scientific-linux-devel mailing list has no answer as of today.
5. Any other idea ?

Gnome networkmanager

• That we like it or not, in many installations with GNOME the network configuration is being taken care of by the Networkmanager process. Here's how the way of parsing the static route configuration file was ported straight from the IPv4 case:
  #define IPV6_ADDR_REGEX "[0-9A-Fa-f:.]+"
(...)
const char *pattern_to1 = "^\\s*(" IPV6_ADDR_REGEX "|default)" /* IPv6 or 'default' keyword */
• Here's the result of a hopeful attempt to set the default route:
  NetworkManager[2236]: ifcfg-rh: error: Invalid IP6 route destination address 'defa'
  Should use ::/0 instead...
• A networkmanager-list post (with an attached patch) of last January has no answer as of today.
• How many other untested cases like this do we have under our feet ?

Rogue RAs (1)

• So we now have five officially dead IETF drafts to add a default route option to DHCPv6:
  1. draft-droms-dhc-dhcpv6-default-router
  2. draft-sun-mif-address-policy-dhcp6
  3. draft-sarikaya-mif-dhcpv6solution
  4. draft-dec-dhcpv6-route-option
  5. draft-ietf-mif-dhcpv6-route-option
• A look at the mif working group active issues shows the most serious objection to be that the option would violate proper fate sharing by advertising a possibly dead router.
• BTW: most of the objections seem to come from Google employees. Their needs may be different than ours.

Rogue RAs (2)

• But being directed by DHCP(v6) to a dead router isn't as bad as having the routing table of all nodes on a link disrupted by a rogue (unsolicited multicast) Router Advertisement message.
• Problem statement is in RFC 6104.
• The fundamental difference here is that the DHCP response is always unicast and solicited, making it somewhat easier to identify/tell rogue DHCP servers, and disallowing changes while the lease is valid. Current RA design allows, but makes it impractical to implement, unicast RAs in response to Router Solicitation (RS) messages.
• So RAs, as currently specified and as implemented by most OSes by default, require some action, preferably via ACLs on the switches (e.g. Cisco's RA-guard).
• The development of ramond, a rogue-RA squash daemon originally written by one of the authors of RFC 6104, is stalled and the current version on sourceforge needs patches. ramond wouldn't help in a DOS scenario, though.

Conclusions / bigger picture thoughts

• There are various cases where IPv6 seems still not to be taken seriously (i.e. with enough care/priority) when it comes to writing/patching software. Even recently.
• As a working group, and within available resources, we try not to limit ourselves to identifying issues, but tracking and keeping traction on their resolution has proved to be slow and expensive.
• Does the reality of IPv6 still have to strike many as a shock ?
• IPv6, in the current state of the (complex) protocol specification mesh, keeps forcing some level of infrastructure rethink (e.g. to filter rogue RAs).
  How much is this in the way of wider adoption ?