Dealing with Cellular Broadband CGNAT
We recently switched over to LTE broadband at home. Our previous provider (British Telecom) struggled to offer anything faster than ~35Mbps download & ~4Mbps upload, perhaps owing to the particularly long run of
wet string copper cable between our property and our VDSL DSLAM.
A few speed tests with SIMs from different providers showed that Vodafone were able to offer at least ~80Mbps download and 30Mbps upload most of the time, which suited us fine. If you are considering LTE broadband, I’d strongly suggest buying a cheap data card & USB adapter and a variety of Pay As You Go SIM cards from different providers to experiment with. Try different locations around your property - particularly upstairs windows to see which provider and installation location provides the best performance.
LTE (4G) broadband has a few downsides to consider too. Principally - latency (the time taken for a packet to complete a round-trip from one host to another) & jitter (the variation in latency over a period of time). Due to the modulation techniques and media-sharing inherent in their design, wireless links are nowhere near as performant or as stable in the latency department as typical wired connections. This can have an impact on jitter-sensitive applications such as video calling and VoIP, so if this is the sort of application you’ll be primarily using, LTE broadband might not be the best choice.
There is another thorny issue with LTE broadband that presented a much greater challenge…
CGNAT stands for Carrier-Grade NAT. It is a type of large-scale NAT (Network Address Translation) used by carriers to deliver Internet service to large numbers of users. One of the primary uses of CGNAT is to limit the number of public IPv4 addresses that are issued to subscribers. This has become a more pressing issue in recent years owing to the emergent IPv4 address exhaustion issue.
Most LTE broadband providers use CGNAT. The IP address subscribers receive is a “private” IP address (typically within
100.64.0.0/10, which is specifically designated by IANA for this purpose, however the
172.16.0.0/12 block is also sometimes used owing to it being uncommon in consumer private networks).
As you can see from the diagram above, end-users on the customer’s network effectively have representation in three different IP networks - there is the local
192.168.1.0/24 network most users are familiar with; the
100.64.0.0/10 network common to many customers’ CPEs (Customer Premises Equipment - typically combined modem/routers); and finally, the public Internet network.
This is effectively an additional “layer” of NATting - most users are used to having a private IP address assigned to their computer and outbound NAT (sometimes called “Source NAT”) being performed for them by their router/CPE. In this new scenario, this “inner” NAT is still present, but their router/CPE has another (effectively) private IP address instead of a publicly-routable one, and another “outer” level of translation is performed by the LTE carrier’s infrastructure to a final public IP address. For this reason, CGNAT is sometimes referred to as “444” (3 x IPv4) NAT.
For “most users”, CGNAT will most likely not pose any problems. Web browsing, video calling and other applications are generally tolerant of NAT, and as such are unlikely to be adversely affected by an additional level of translation.
The problems start to occur when a user needs to use applications that perform inbound NAT (sometimes called port forwarding, destination NAT, or simply
ipfilter speak). This is where the user wants to configure connections to the public IP address on specific protocol & port to always translate to a specific IP address & port on the local side of the NAT. A typical example of a scenario that requires this setup would be a CCTV camera on the customer’s network (say
192.168.1.2 in the diagram above) that a remote user needs to connect to and view an RTSP video stream.
Normally, this is achieved by configuring these NAT rules on the router/CPE. With CGNAT, this becomes impossible. The end-user has no way to configure these rules at the “outer” NAT boundary. Even if they could, the potential for conflicts would be too great (since many users are ultimately “sharing” the same public IP address).
There is no pretty way around this. That being said, there are solutions that can be implemented for specific applications.
The principal requirement in any solution to the CGNAT problem is that communication must happen via a connection that was ultimately initiated from the customer side of the network. There are several ways to realise this requirement, the most obvious being some form of tunneling protocol. The solution that I will detail here is just one of many ways in which you can work around CGNAT.
I should also mention that while the setup here isn’t insecure; better (but more complex) security can (and should) be implemented through the use of digital certificates.
Here’s a high-level diagram of what we are aiming to achieve:
We have a host with one or more public IP addresses outside of the customer-side network. An IP tunnel initiated from the customer-side of the network needs to be configured. For the purposes of this example, we’ll use an L2TP tunnel. The two ends of the tunnel have will IP addresses
192.168.100.2 in this example. Since L2TP doesn’t provide any kind of encryption, we’ll also configure IPsec in Encapsulating Security Payload (ESP) mode for the two hosts - again, initiated by the customer-side host.
Any traffic arriving at the Tunnel Exit host on
203.0.113.1 needs to be Destination NAT’d to the Tunnel Router at
192.168.100.2 (via the L2TP tunnel). This is an unusual Destination NAT configuration - normally we’d only be forwarding connections on specific ports - but in this case we want all traffic to be forwarded via the tunnel. Once these connections reach our Tunnel Router (
192.168.100.2), they can be Destination NAT’d again as appropriate. Note that with this setup, we still have the “444” scenario (our traffic still passes through 3 IPv4 networks), but now we have full control over the NAT rules at every stage.
Note that we have an additional public IP address assigned to our Tunnel Exit host -
203.0.113.2 - this is to facilitate the L2TP connection and any management SSH connections. Once the aforementioned NAT rule is configured, the
203.0.113.1 IP address will not be usable for connections intended for the Tunnel Exit. An alternative approach might be to create a rule earlier in the
PREROUTING chain to exempt connections on some ports (a port for an SSH server and the L2TP server, for instance) from the catch-all
Setting up the Tunnel
In this example, we’ll run Linux on the Remote Tunnel Exit and the Local Tunnel Exit1. We’ll use Strongswan for IPsec and xl2tpd for the L2TP tunnel termination.
We’ll start with the L2TP server (“LNS”), since that can be tested in isolation before setting up the arguably more involved IPsec. Setting up the L2TP server is fairly straightforward - our
/etc/xl2tpd.conf should look like this:
[lns default] exclusive = yes assign ip = no local ip = 192.168.100.1 require chap = yes refuse pap = yes require authentication = yes length bit = yes name = l2tpd pppoptfile = /etc/ppp/options.xl2tpd
This configuration permits a single tunnel only (
exclusive), and we won’t perform any dynamic IP assignments since we’ll be using two static IP addresses for the two ends of the tunnel (
assign ip = no). We set the Tunnel Exit side to be
192.168.100.2 and configure some authentication parameters for CHAP. We’ll also set up some PPP options for the underlying
pppd server in the file specified by
require-mschap-v2 auth lock
These parameters enforce the use of MS-CHAPv2 authentication and instruct
pppd to write a lockfile on startup.
We’ll also need a CHAP secrets file to specify a pre-shared key for the tunnel authentication -
# client server secret allowed addresses tunneluser l2tpd mysecurepsk *
You can start the tunnel server simply by starting the
xl2tpd service in the appropriate way for your Linux distribution - typically:
Next, on the Local Tunnel Exit, we’ll set up the L2TP Client (“LAC”). On the client, our
/etc/xl2tpd.conf should look like this:
[lac tunnel-connection] lns = 203.0.113.2 local ip = 192.168.100.2 pppoptfile = /etc/ppp/options.xl2tpd length bit = yes
lns directive instructs
xl2tpd to initiate a connection to the L2TP server at
local ip directive specifies the Local Tunnel Exit’s tunnel IP address. Again we have another
pppd options file -
require-mschap-v2 noccp noauth defaultroute connect-delay 5000 name tunneluser password mysecurepsk
connect-delay 5000 directive - this directive makes
pppd wait for 5 seconds before bringing up the tunnel - this delay gives the IPsec SAs time to become established before the L2TP tunnel is initialised.
You can test the tunnel connection without IPsec configured by starting
xl2tpd in the usual way then writing a control command to
xl2tpd’s configuration FIFO:
You should be able to ping the other end of the tunnel from either end (
192.168.100.2 in our example) and receive ICMP responses.
Setting up the NAT Rules
Now we’ll configure the NAT rules on the Remote Tunnel Exit:
This command adds an
ipfilter rule to the
PREROUTING chain of the
nat table that will Destination NAT all traffic arriving with destination
192.168.100.2 (our Local Tunnel Exit).
An easy way to test this configuration is by
pinging the Remote Tunnel Exit host (
203.0.113.1) and running
tcpdump | grep ICMP on the Local Tunnel Exit host and confirm that you can see the ICMP Ping traffic being correctly NAT’d and routed across the tunnel.
Securing the Tunnel
The next stage is to secure the tunnel with IPsec. This is a little more involved than the L2TP setup.
A quick primer on IPsec on Linux. Setting up a connection occurs in two phases. In Phase 1, a keying daemon (
charon - part of Strongswan) negotiates and sets up an IKE (Internet Key Exchange) Security Association (IKE SAs) with the opposite peer (via the IKEv2 protocol).
A Security Association defines behaviour for handling packets destined for, or arriving from a given network/host. That behaviour could include encrypting/decrypting or authenticating the origin of the packets using a key that is part of the Security Association. Security Associations periodically expire and are renegotiated by
charon. This initial IKE Security Association is used to protect the subsequent negotiations during the next stage (Phase 2).
Once an IKE SA is established, Child SAs (also called traffic SAs) are negotiated for each traffic direction (A → B and B → A). These SAs also have a limited lifetime (usually shorter than their IKE SA counterpart) and their own keying information.
Remote Tunnel Exit IPsec Configuration
Before we begin configuration, it’s worth mentioning some of the terminology used by Strongswan. Strongswan configuration always refers to this side (meaning the side with the configuration file on it) as
left and the remote side(s) as
We’ll configure the Remote Tunnel Exit side first.
/etc/ipsec.conf should contain the following directives:
conn tunnel left=203.0.113.2 leftprotoport=17/1701 right=%any rightprotoport=17/%any rightallowany=yes authby=secret ike=aes256-sha256-modp2048! esp=aes256-sha256-modp2048! ikelifetime=1d lifetime=1h auto=route type=transport
The configuration specifies that the
left (Remote Tunnel Exit) IP address as the public
203.0.113.2 address. The
right (Local Tunnel Exit) is permitted to be any address. The
rightprotoport directives specify a traffic selector to identify traffic that should be processed by IPsec. In this case, inbound UDP (IP protocol #17) traffic on port 1701 (L2TP) and outbound UDP traffic on any port should be processed.
We authenticate using a PSK (
authby=secret) and specify the encryption/digest/Diffe-Hellman schemes that are acceptable to us. Additionally, the maximum lifetime for the IKE SAs is defined as 1 day (
ikelifetime); and for the Child SAs as 1 hour (
We then define the startup behaviour and packet specificity for the
tunnel connection. The
auto=route directive specifies that the connection should be loaded but only established as soon as traffic is detected that demands IPsec processing.
We’ll also define the PSK used to secure the IPsec connection in
/etc/ipsec.secrets. If you wish, you could use an RSA certificate or other key material instead - for this example, we’ll just use a simple pre-shared key though:
%any %any : PSK "mysecureipsecpsk"
Local Tunnel Exit IPsec Configuration
On the Local Tunnel Exit side, we’ll use the following configuration in
config setup nat_traversal=yes conn tunnel left=192.168.100.2 leftprotoport=17/%any right=203.0.113.2 rightprotoport=17/1701 authby=secret ike=aes256-sha256-modp2048! esp=aes256-sha256-modp2048! ikelifetime=1d lifetime=1h auto=route type=transport
First we enable the NAT traversal features, since our Local Tunnel Exit will definitely be connecting from behind a NAT. Next, the left and right sides of the tunnel are defined, as well as the traffic specifiers to identify traffic that will be processed. All other configuration options match those on the other side of the tunnel.
Again, the pre-shared key is defined in
%any %any : PSK "mysecureipsecpsk"
We can test the IPsec configuration by starting Strongswan:
Then starting the L2TP tunnel as before in order to prompt the IPsec connection to be established. Afterwards, you should be able to see some
ESTABLISHED IPsec Security Associations in the output of
Configuring Port Forwards
Once everything is up and running, you’re able to add Destination NAT (
ipfilter rules on the Local Tunnel Exit to forward traffic arriving at
203.0.113.1 to any host inside your private network - just as you would if you were using a regular, non-CGNAT Internet connection.
An example rule to direct traffic to a web server (TCP port 80) running on
192.168.1.2 might look like this:
Going further, you could also configure Source NAT (
SNAT) rules on both the Local and Remote Tunnel Exits to allow outbound connections from inside the network that are transported over the tunnel. This is less relevant to the CGNAT situation we’re dealing with here, but still a useful side-effect of having a tunnel in place. One reason that this can be particularly useful is evading traffic shaping/management imposed by the LTE broadband provider.
In my home setup, I’m actually using a MikroTik RouterBoard as my Local Tunnel Exit instead of a Linux machine running
xl2tpd. Let me know if you’d like a description of that setup and I’ll add one. ↩