Diagnosing IPv4 traffic at dual-stack IPv6 sites.


HEPix IPv6 WG meeting - CERN, October 26th, 2022.
Francesco Prelz, francesco.prelz@mi.infn.it
⇔ Outline (short!)
⇔ Why are we so curious?
- Basic premise: yes, we still believe that the transition to IPv6 may approach completion one day, so we need to pave that way and not another. This is not the right forum for other systems of belief...
- A long (1, 2) effort was needed to bring our software universe to be IPv6-friendly. Residual components that require IPv4, especially those that break with DNS64/NAT64 (e.g. where IPv4 literal addresses are stored or signalled) may be equally hard to `befriend'. The earlier they are found, though, the better.
- Every now and then some macroscopic effect that doesn't match the working assumption that IPv6 be preferred on dual-stack systems is noticed. These 'puzzles' have wide ramifications, and we are still lacking higher level network diagnostic tools that work across our sites/countries/institutions.
⇔ What we found one day @KIT (1)
- So, one beautiful day Bruno H. kindly agreed to share a few snippets of network traffic (PCAP) from the Tier-1 center at KIT.
- Once the data are hashed and sorted, natural curiosity makes one
rush to the list of top IPv4 talkers...:
# of bytes | Source IP:port -> Destination IP:port 448513480 | 10.1.16.130:50448 -> 10.65.120.184:23056 334913896 | 10.65.120.127:47656 -> 10.1.15.112:46419 324002192 | 10.1.15.133:49580 -> 10.65.120.184:23717 314253816 | 10.1.15.115:46756 -> 10.65.120.128:21147 235081168 | 10.65.152.189:38858 -> 10.1.16.175:60059 199643078 | 10.65.120.126:51444 -> 10.1.15.180:49710 (... etc. etc. ...)
- Ehm... Er... Not much info there! But Bruno comes to the rescue: "10.1.x.x/16 are worker nodes and 10.65.x.x/16 are (dCache) storage servers". All properly dual-stack, of course.
⇔ What we found one day @KIT (2)
| Moving to the perspective of one worker-node helps in sorting the haystack...: |
|
⇔ What we found one day @KIT (3)
- Single-WN data reveals that all activity, even communication to services that are known to work in a dual-stack environment, and down to data exchange with local storage occurs on IPv4.
- Could it be some issue with the node (IPv4 accidentally disabled?). Bruno's help is again needed to find out which IPv6 address corresponds to a given IPv4 host (all hosts are dual-stack but they are not registered in the DNS with A and AAAA records for the same name): plenty of IPv6 traffic (mostly contacting ALICE services) is found to originate from the same worker node...
- So there must be some other reason! Carmen will continue telling this story...
⇔ What can help with this effort?
- Make it easy (e.g. via DNS) to associate the IPv4 and IPv6 addresses assigned to the same host (pass from one to the other and v/v). Are there concerns about this (e.g. attack surface discovery) ?
- DNS names "may" also help in understanding the role of nodes (in particular: bulk storage vs. compute nodes).
- In case captures of short squirts of data cannot be shared with outside observers (they do help, statistically, as we've seen), monitoring data should be collected and monitored regularly on-site.
- Site security and data privacy are important, but remote communication on technical matters is known to be difficult.
- Any observation of unexplained IPv4/IPv6 asymmetries can (should?) be bounced onto the IPv6 group.
⇔ Thank you!
Questions and comments are welcome.
Goto Page: