A short introduction to IPv6. For those of you who are somewhat new to IPv6 I would like to introduce some terms. The world in IPv6 is a little bit different to IPv4. Overall I have to say I like it much better. Yet there are some caveats I feel like sharing so you can benefit from my experience.
In IPv6 there are several address types, the first we look at is ULA. It stands for Unique Local Address. These addresses range is fc00::/7 (fc00:: to fdff:ffff:ffff:ffff:ffff:ffff:ffff:ffff) and is somewhat similar to what RFC1918 addresses in IPv4 were used. There is this IPv6 Buzz talk about ULA addresses and why it is bad to use them. Their main argument is that in DNS everything takes precedence before ULA addresses, even IPv4. I do not agree to ULA being bad, at all. Read the full article to understand why.
GUA is short for Glocal Unicast Address. Those addresses form the Internet, as they are publicly routed. Here's a popular comparison between IPv6 and IPv4 Address space (rounded):
IPv6 (2^128): 34,000,000,000,000,000,000,000,000,000,000,000,000,000
IPv4 (2^32): 4,200,000,000
A typical static prefix is of the size 2^48. So one single network has hugely more address space available, than all of the legacy Internet. Even a smaller but also rather commong prefix size for dialups is /56. Some more rounded numbers:
One /48 (2^80): 1,200,000,000,000,000,000,000,000
One /56 (2^72): 4,700,000,000,000,000,000,000
Link Local addresses to the user are mostly irrelevant. They are being used for an automated setup so that hosts within the same broadcast domain can reach eath other. Mechanisms for finding each other but also for finding special devices like gateways are part of that address type and of the Neighbor Discovery Protocol (NDP).
To get a deeper understanding of IPv6, everybody should know about those addresses and about Neighbor Discovery, but to this article this is not relevant.
Currently I run VDSL (NetAachen) with Vectoring at 100/40 Mbit/s together with static IPv4 and IPv6 (/48) prefix. It's about 55 € per month. The Deutsche Telekom will soon provide me with fiber, thus I can get faster line speeds. There are only two products, that will improve speed, especially in terms of upstream:
MagentaZuhause XXL: 500/100 Mbit/s for 59,95 € per month or
MagentaZuhause Giga: 1000/200 Mbit/s for 79,90 per month
If I wanted to have a static IP I would need to get a business rate:
CompanyPro 500: 500/100 Mbit/s for 99,95 € per month or
CompanyPro 1000: 1000/200 Mbit/s for 129,95 € per month
I decided to go for MagentaZuhause Giga which will provide me with a dynamic prefix. If I wanted the same speed but with static prefix it would result in an extra of 50 € per month. 130 € per month for an Internet uplink? No, not as long as I won't gain income from it. That's just too expensive. Even though I really wanted the comfort of a static prefix I thought the extra money is not worth it, so I adjusted to the idea of having a dynamic prefix.
My network environment relies heavily on IPv6. There are hardly any A records in my DNS zones as only hosts that really don't do IPv6 will get an A record. All the AAAA records point to GUA addresses from my public prefix. I'm also making use of the fact that I got a static /48 prefix quite intensively. I'm using several different VLANs, but there's more. I also run downstream routers below my main uplink firewall. They provide addresses to clients in simulated environments. So there are a couple of options how to make the transition:
On top of my home network there is a server I run in a datacenter that uses virtualization and has a virtualized firewall for all the different guests. My home site is interconnected to the datacenter's firewall with a wireguard tunnel.
My first thought was to go all in on dynamic IP configuration. As I don't want to and cannot have a dynamic dns clients on each of my devices, I thought I switch back from OpenWrt to OPNsense. OPNsense has a builtin feature so that every DHCPv6 client will leave an updated DNS AAAA record in my authoritative DNS servers once they obtain an IP. So even if I get a new prefix, together with RFC2136 and a very low TTL it would at least feel like having a static prefix.
I decided against that approach due to several reasons. One was that not every device I run has a DHCPv6 client builtin. Also I have had bad experience with DHCP in general when it comes to the DHCP server dying. I want my network to keep running, even if there is a disruption of some kind.
Another thought of mine was to just tunnel static IP addresses I run in a data center. There I have a routed /48 prefix which is more than enough for two or maybe even thousands of sites. While this idea looks appealing at first, there are some hefty caveats to this.
I would never run a whole site on IP addresses from another site. Maybe I will provide a single host or two with IP addresses from the remote site, but running the whole network on IP addresses from another site just doesn't make sense. It is a dependency that might break your network, if the datacenter isn't reachable or maybe just the router in the datacenter is failing or on maintenance. So: No, I won't follow that approach.
ULA addresses are a good utility in your home network but also in companies. You could even provide ULA if a static prefix is available. This would help if you'd move the server from one hoster to another. I started to deploy ULA in the datacenter as well, but I'm not quite sure if I will ever updated my DNS records to the ULA address. I just like the idea of the IP addresses being the same, no matter if I address them remotely through the tunnel, from my Roadwarrior VPN or even from public Internet.
Assuming you have a static prefix and using only that, you'll face the dilemma I'm facing right now. An exception of course is if you run your own AS via BGP either as RIR member or via your own PI prefix In that case your IPs will remain the same, even if you switch providers or datacenters.
With ULA you can get some of independence. Say you switch to another ISP or you will downgrade from a static prefix to a dynamic. Your ULA addresses will remain the same. Your services will stay reachable and the amount of work resulting from such a migration is little.
Yet ULA addresses alone will not suffice as they are not being routed on the Internet. You could use IP masquerading but no one in their right might actually wants that. So on top of a static ULA I will also use autoconf and/or DHCP for getting public address.
That's why I've come to the conclusion to use a combination of stateful DHCP plus static ULA addresses.
GUA addresses CAN talk to ULA addresses, as long as they don't traverse public routers, so through a VPN this works perfectly fine.
Regarding the IPv6 buzz argument. Well that's a no brainer. In a decent network you don't use A records. Even if you do and IPv4 takes precedence over ULA? So what? It will give you just another failsafe just in case v4 doesn't work for whatever reason. Yet I don't see a good reason why you would implement A records in the first place. We ran fine for decades without AAAA so now we can live without A. The long term goal of the Internet and also of local networks is indisputably IPv6 only. Adjust your strategy to that! You don't want to manage your firewall rules, your DNS records etc. all in both IPv4 and IPv6.
For most Linux hosts the approached type of configuration is unproblematic. Static addressing of course is no problem and DHCPv6 most of the times is no problem, too. Yet the situation sometimes is frustrating. There are so many differences of how Operating Systems implement those techniques. Android for example doesn't do DHCPv6 at all. While that's not too bad, as I don't care what the address of a dynamic Android client looks like.
The situation on other devices is rather problematic. Proxmox VE for example uses cumulus' ifupdown2 framework which doesn't seem to work with DHCPv6. Autoconf is broken for bridges alltogether. On plain interfaces autoconf works, but DHCPv6 just doesn't work at all. How can they fail such easy things?
Other devices like printers and switches sometimes don't do DHCPv6, but mostly they don't support a mixture of static ULA with autoconf/dhcp. That's why I choose to just autoconf or dhcp for them, as a static ULA alone won't get them outside of my local LAN. Together with the device's DUID I am able to provide them with a static ULA address, similar to static leases in DHCPv4. In v4 they were related to the MAC address. IN DHCPv6 DUIDs are being used which gives you more flexibility, but sometimes it's a bit hard to find the devices DUID.
OpenWrt is a good example of how it's done right. They support everything, like dynamic stateless, dynamic stateful, static and also a combination of either. If you run multiple different Linux OSes it's also quite annoying. Everyone is doing this stuff differently. Raspian for example uses dhcpcd, which works quite well. It implements v4 and v6 in one client. Other OSes use ISC DHCP, others don't come with DHCPv6 client by default. You will have to choose one, install and configure. There is for example the Dibbler or wide DHCPv6. The latter is quite common in BSD environments. What Gnome's Network Manager does in the background I could not find out.
As I'm switching from a static to a dynamic prefix I needed to do some renumbering. Over the last couple of years I memorized IPv6 addresses and prefixes. YES, that's possible. I didn't want to start all over, that's why I choose to make the ULA identical to my old static prefix, well of course the first 7 bits are an exception.
GUA: 2001:4dd0:28d4:7381::10
ULA: fd01:4dd0:28d4:7381::10
That way I can keep my old memories and the transition will get a little bit easier.
For my VPN connections I also had to make some adjustments. I'm using wireguard for a Site to Site (S2S) tunnel between my home network and a datacenter. Wireguard also is being used for Roadwarriors. Those connect to the datacenter and over the S2S they also will get to the home site.
The Adjustmens for the S2S were easy. At the datacenters side I just had to add the my new ULA prefix to the allowed ips. The corresponding option in OpenWrt looks like that
config wireguard_S2S
option description "my S2S"
...
list allowed_ips "fd01:4dd0:28d4::/48"
From the peer definition I had to remove the endpoint_host and endpoint_port directive as only the firewall at my home site will be able to establish the connection.
option endpoint_host "2a01:f20:1:4f::b1b1"
option endpoint_port "51821"
In the Roadwarrior's configuration I had to add the target to the AllowedIPs like this:
[Interface]
Address = ...
PrivateKey = ...
[Peer]
PublicKey = ...
PresharedKey = ...
AllowedIPs = ... fd01:4dd0:28d4::/48
For the tunnel to keep working I had to adjust the firewall rules, of course.
Those were the changes for VPN I thought would suffice. They did not. Keep reading.
I'm using Ansible to manage my firewalls and servers. Good thing is there are variables which I can define once and use many. Sadly I realized there were way too many IP addresses defined manually in my var definitions. That opportunity I took to update them and minimize the amount of hard coded IP addresses and networks. Beforehand I just added the ULA IPs to the rule. As soon as the transition to the new fiber link is done, the old GUA addresses will be removed.
Not only do I run centrlized firewalls but on the servers I run nftables, also managed by my very own Ansible role. Those also needed some updates. In those scenarios you can be very lucky, if you run Ansible, because the amount of work needed for such a change is minimal in contrast to doing it manually.
My hosts at home now have ULA addresses on top of the GUA ones. I started experimenting and not after long I realized there was an annoying problem.
There are several scenarios of who initiates communication with whom. Only one particular case is actually problematic. A host at home is trying to access a host in the datacenter via VPN. The Source Address Selection method reliably uses a GUA address at the home computer, rather than the ULA address. This is probably the best choice in general, but in this scenario it prevents successful communication.
A packet will arrive at the VM in the datacenter. As the tunnel between the sites has no information about the GUA prefix from home - as it is dynamic - the return traffic will never arrive. It will be routed through the internet which fails, as the firewall at home doesn't have rules for that. Apart from that the session information stored at the home firewall doesn't match, too.
Communication breakdown
This sequence diagrams demonstrates the problem with a site to site tunnel with one static and one dynamic prefix.
When a host in the datacenter iniates traffic the problem does not occur.
There are several methods to circumvent that problem.
I could manually or with some automation modify the tunnel specs every time a new prefix is deployed at home. This is obviously the worst option.
I could masquerade the traffic originating at home going to the data center. Just no.
I also could setup ULA in the datacenter. That way source and and destination would be ULA, well at least I assume it will. Either way, I don't want to change the DNS records in the datacenter, as I like them to be pointing to GUA addresses.
I tried an approach with dynamic routing and experimented with Quagga, FRR, babeld and bird. In the end I went with babeld and here is what I did. Using the wireguard tunnel between the two sites for exchanging routing information. In the end that worked just fine but the road was bumpy.
At first I had to change the wireguard peer definition. It consisted of a directive called allowed_ips which I had to modify. This directive has two purposes. One: It installs routes to the remote. Two: It limits what IP addresses can be used for data passing through the tunnel.
I changed it from naming all my remote nets to just
config wireguard_RXFORELLE
option description "fw.libcom.de"
option public_key "sjdfklsjdkfljdsklfjsdklfjkldsjfkldsjklfjdsklf"
option preshared_key "sdjfklsjdkfljskldfjsdklfjkdsljfkldsjfklsdj"
list allowed_ips "0.0.0.0/0"
list allowed_ips "::/0"
option route_allowed_ips "0"
option endpoint_host "2a00:fee:fee:4f::b1b1"
option endpoint_port "51821"
As you can see I have set the option route_allowed_ips to 0. Before it was set to 1. Now every IP may pass the tunnel, but no route will be installed by wireguard whatsoever. This is where babeld comes into game.
Babel s a modern and very fun dynamic routing solution. FRR has an implementation for it but there is the reference implementation called babled chose. My wireguard interface on both ends is called RXFORELLE and here is the sample configuration of one side. Remember, this is OpenWrt, so this is a UCI configuration file.
config general
option ipv6_subtrees 'true'
option ubus_bindings 'true'
option random_id 'true'
option local_port '33123'
config interface
option ifname "RXFORELLE"
option type "tunnel"
config filter
option type "redistribute"
option action "deny"
option local "true"
config filter
option type "redistribute"
option ip "fd01:4dd0:28d4::/48"
config filter
option type "redistribute"
option ip "2001:4dd0:28d4::/48"
config filter
option type "redistribute"
option ip "10.10.128.0/20"
config filter
option type "redistribute"
option ip "2a03:fe3:fe3::/48"
What this does is using the RXFORELLE interface for exchanging routing information. Be aware that babeld uses IPv6 multicast for that. So this needs to be allowed in the tunnel. Both systems register each other as neighbor and start exchanging routes, but limited to spaces given in the filters. Right now of course there is my actual static prefixes in there. When I switch to the dynamic prefix of Deutsche Telekom I will adapt to using the (probably) /32 prefis they use to provide me from. Example:
config filter
option type "redistribute"
option ip "2a00:afe::/32"
With all these fixes traffic again flow successfully when a system at home initiates traffic.
I run several services in my home network, that can be reached from the outside Internet. Some I run only for me. They continue to work though, as soon as I switched over to ULA addresses in DNS and the VPN was modified. Publicly reachable services will need to make use of dynamic DNS. If the prefix changes, the AAAA records will be updated and the downtimes will be minimal.
I'm hosting the DNS service for my domains myself. I spread it on three synced servers, one of which is at my home network. This one will be removed from the public directory. Yet I will still let it run locally under an ULA address, as I will configure domain overrides at my local resolvers, so that the resolution of my devices via DNS keeps working, even if my uplink fails.
The servers all are configured to listen on ::, so I don't have to change the binding IPs. There are several packet filters in place that will make sure unauthorized IPs won't be able to connect.
In the end this transition from static to dynamic prefix resulted in about ~ 12 hours of work. Don't make the same mistakes I did. Use ULA!