Dealing with Cellular Broadband CGNAT

We recently switched over to LTE broadband at home. Our previous provider (British Telecom) struggled to offer anything faster than ~35Mbps download & ~4Mbps upload, perhaps owing to the particularly long run of wet string copper cable between our property and our VDSL DSLAM.

Considering LTE

A few speed tests with SIMs from different providers showed that Vodafone were able to offer at least ~80Mbps download and 30Mbps upload most of the time, which suited us fine. If you are considering LTE broadband, I’d strongly suggest buying a cheap data card & USB adapter and a variety of Pay As You Go SIM cards from different providers to experiment with. Try different locations around your property - particularly upstairs windows to see which provider and installation location provides the best performance.

LTE (4G) broadband has a few downsides to consider too. Principally - latency (the time taken for a packet to complete a round-trip from one host to another) & jitter (the variation in latency over a period of time). Due to the modulation techniques and media-sharing inherent in their design, wireless links are nowhere near as performant or as stable in the latency department as typical wired connections. This can have an impact on jitter-sensitive applications such as video calling and VoIP, so if this is the sort of application you’ll be primarily using, LTE broadband might not be the best choice.

There is another thorny issue with LTE broadband that presented a much greater challenge…

CGNAT

CGNAT stands for Carrier-Grade NAT. It is a type of large-scale NAT (Network Address Translation) used by carriers to deliver Internet service to large numbers of users. One of the primary uses of CGNAT is to limit the number of public IPv4 addresses that are issued to subscribers. This has become a more pressing issue in recent years owing to the emergent IPv4 address exhaustion issue.

Most LTE broadband providers use CGNAT. The IP address subscribers receive is a “private” IP address (typically within 100.64.0.0/10, which is specifically designated by IANA for this purpose, however the 172.16.0.0/12 block is also sometimes used owing to it being uncommon in consumer private networks).

As you can see from the diagram above, end-users on the customer’s network effectively have representation in three different IP networks - there is the local 192.168.1.0/24 network most users are familiar with; the 100.64.0.0/10 network common to many customers’ CPEs (Customer Premises Equipment - typically combined modem/routers); and finally, the public Internet network.

This is effectively an additional “layer” of NATting - most users are used to having a private IP address assigned to their computer and outbound NAT (sometimes called “Source NAT”) being performed for them by their router/CPE. In this new scenario, this “inner” NAT is still present, but their router/CPE has another (effectively) private IP address instead of a publicly-routable one, and another “outer” level of translation is performed by the LTE carrier’s infrastructure to a final public IP address. For this reason, CGNAT is sometimes referred to as “444” (3 x IPv4) NAT.

Problems

For “most users”, CGNAT will most likely not pose any problems. Web browsing, video calling and other applications are generally tolerant of NAT, and as such are unlikely to be adversely affected by an additional level of translation.

The problems start to occur when a user needs to use applications that perform inbound NAT (sometimes called port forwarding, destination NAT, or simply dstnat in ipfilter speak). This is where the user wants to configure connections to the public IP address on specific protocol & port to always translate to a specific IP address & port on the local side of the NAT. A typical example of a scenario that requires this setup would be a CCTV camera on the customer’s network (say 192.168.1.2 in the diagram above) that a remote user needs to connect to and view an RTSP video stream.

Normally, this is achieved by configuring these NAT rules on the router/CPE. With CGNAT, this becomes impossible. The end-user has no way to configure these rules at the “outer” NAT boundary. Even if they could, the potential for conflicts would be too great (since many users are ultimately “sharing” the same public IP address).

Workarounds

There is no pretty way around this. That being said, there are solutions that can be implemented for specific applications.

The principal requirement in any solution to the CGNAT problem is that communication must happen via a connection that was ultimately initiated from the customer side of the network. There are several ways to realise this requirement, the most obvious being some form of tunneling protocol. The solution that I will detail here is just one of many ways in which you can work around CGNAT.

I should also mention that while the setup here isn’t insecure; better (but more complex) security can (and should) be implemented through the use of digital certificates.

Here’s a high-level diagram of what we are aiming to achieve:

We have a host with one or more public IP addresses outside of the customer-side network. An IP tunnel initiated from the customer-side of the network needs to be configured. For the purposes of this example, we’ll use an L2TP tunnel. The two ends of the tunnel have will IP addresses 192.168.100.1 and 192.168.100.2 in this example. Since L2TP doesn’t provide any kind of encryption, we’ll also configure IPsec in Encapsulating Security Payload (ESP) mode for the two hosts - again, initiated by the customer-side host.

Any traffic arriving at the Tunnel Exit host on 203.0.113.1 needs to be Destination NAT’d to the Tunnel Router at 192.168.100.2 (via the L2TP tunnel). This is an unusual Destination NAT configuration - normally we’d only be forwarding connections on specific ports - but in this case we want all traffic to be forwarded via the tunnel. Once these connections reach our Tunnel Router (192.168.100.2), they can be Destination NAT’d again as appropriate. Note that with this setup, we still have the “444” scenario (our traffic still passes through 3 IPv4 networks), but now we have full control over the NAT rules at every stage.

Note that we have an additional public IP address assigned to our Tunnel Exit host - 203.0.113.2 - this is to facilitate the L2TP connection and any management SSH connections. Once the aforementioned NAT rule is configured, the 203.0.113.1 IP address will not be usable for connections intended for the Tunnel Exit. An alternative approach might be to create a rule earlier in the PREROUTING chain to exempt connections on some ports (a port for an SSH server and the L2TP server, for instance) from the catch-all DNAT rule.

Setting up the Tunnel

In this example, we’ll run Linux on the Remote Tunnel Exit and the Local Tunnel Exit¹. We’ll use Strongswan for IPsec and xl2tpd for the L2TP tunnel termination.

L2TP Server

We’ll start with the L2TP server (“LNS”), since that can be tested in isolation before setting up the arguably more involved IPsec. Setting up the L2TP server is fairly straightforward - our /etc/xl2tpd.conf should look like this:

[lns default]
exclusive = yes
assign ip = no
local ip = 192.168.100.1
require chap = yes
refuse pap = yes
require authentication = yes
length bit = yes
name = l2tpd
pppoptfile = /etc/ppp/options.xl2tpd

This configuration permits a single tunnel only (exclusive), and we won’t perform any dynamic IP assignments since we’ll be using two static IP addresses for the two ends of the tunnel (assign ip = no). We set the Tunnel Exit side to be 192.168.100.2 and configure some authentication parameters for CHAP. We’ll also set up some PPP options for the underlying pppd server in the file specified by pppoptfile - /etc/ppp/options.xl2tpd:

require-mschap-v2
auth
lock

These parameters enforce the use of MS-CHAPv2 authentication and instruct pppd to write a lockfile on startup.

We’ll also need a CHAP secrets file to specify a pre-shared key for the tunnel authentication - /etc/ppp/chap-secrets:

# client        server      secret              allowed addresses
tunneluser      l2tpd       mysecurepsk         *

You can start the tunnel server simply by starting the xl2tpd service in the appropriate way for your Linux distribution - typically:

sudo service xl2tpd start

L2TP Client

Next, on the Local Tunnel Exit, we’ll set up the L2TP Client (“LAC”). On the client, our /etc/xl2tpd.conf should look like this:

[lac tunnel-connection]
lns = 203.0.113.2
local ip = 192.168.100.2
pppoptfile = /etc/ppp/options.xl2tpd
length bit = yes

The lns directive instructs xl2tpd to initiate a connection to the L2TP server at 203.0.113.2. The local ip directive specifies the Local Tunnel Exit’s tunnel IP address. Again we have another pppd options file - /etc/ppp/options.xl2tpd:

require-mschap-v2
noccp
noauth
defaultroute
connect-delay 5000
name tunneluser
password mysecurepsk

Note the connect-delay 5000 directive - this directive makes pppd wait for 5 seconds before bringing up the tunnel - this delay gives the IPsec SAs time to become established before the L2TP tunnel is initialised.

You can test the tunnel connection without IPsec configured by starting xl2tpd in the usual way then writing a control command to xl2tpd’s configuration FIFO:

sudo service xl2tpd start
sudo sh -c 'echo c tunnel-connection > /var/run/xl2tpd/l2tp-control'

You should be able to ping the other end of the tunnel from either end (192.168.100.1 or 192.168.100.2 in our example) and receive ICMP responses.

Setting up the NAT Rules

Now we’ll configure the NAT rules on the Remote Tunnel Exit:

iptables \
    -t nat -A PREROUTING -d 203.0.113.1 -j DNAT --to-destination=192.168.100.2

This command adds an ipfilter rule to the PREROUTING chain of the nat table that will Destination NAT all traffic arriving with destination 203.0.113.1 to 192.168.100.2 (our Local Tunnel Exit).

An easy way to test this configuration is by pinging the Remote Tunnel Exit host (203.0.113.1) and running tcpdump | grep ICMP on the Local Tunnel Exit host and confirm that you can see the ICMP Ping traffic being correctly NAT’d and routed across the tunnel.

Securing the Tunnel

The next stage is to secure the tunnel with IPsec. This is a little more involved than the L2TP setup.

A quick primer on IPsec on Linux. Setting up a connection occurs in two phases. In Phase 1, a keying daemon (charon - part of Strongswan) negotiates and sets up an IKE (Internet Key Exchange) Security Association (IKE SAs) with the opposite peer (via the IKEv2 protocol).

A Security Association defines behaviour for handling packets destined for, or arriving from a given network/host. That behaviour could include encrypting/decrypting or authenticating the origin of the packets using a key that is part of the Security Association. Security Associations periodically expire and are renegotiated by charon. This initial IKE Security Association is used to protect the subsequent negotiations during the next stage (Phase 2).

Once an IKE SA is established, Child SAs (also called traffic SAs) are negotiated for each traffic direction (A → B and B → A). These SAs also have a limited lifetime (usually shorter than their IKE SA counterpart) and their own keying information.

Remote Tunnel Exit IPsec Configuration

Before we begin configuration, it’s worth mentioning some of the terminology used by Strongswan. Strongswan configuration always refers to this side (meaning the side with the configuration file on it) as left and the remote side(s) as right.

We’ll configure the Remote Tunnel Exit side first. /etc/ipsec.conf should contain the following directives:

conn tunnel
    left=203.0.113.2
    leftprotoport=17/1701
    right=%any
    rightprotoport=17/%any
    rightallowany=yes
    authby=secret
    ike=aes256-sha256-modp2048!
    esp=aes256-sha256-modp2048!
    ikelifetime=1d
    lifetime=1h
    auto=route
    type=transport

The configuration specifies that the left (Remote Tunnel Exit) IP address as the public 203.0.113.2 address. The right (Local Tunnel Exit) is permitted to be any address. The leftprotoport and rightprotoport directives specify a traffic selector to identify traffic that should be processed by IPsec. In this case, inbound UDP (IP protocol #17) traffic on port 1701 (L2TP) and outbound UDP traffic on any port should be processed.

We authenticate using a PSK (authby=secret) and specify the encryption/digest/Diffe-Hellman schemes that are acceptable to us. Additionally, the maximum lifetime for the IKE SAs is defined as 1 day (ikelifetime); and for the Child SAs as 1 hour (lifetime).

We then define the startup behaviour and packet specificity for the tunnel connection. The auto=route directive specifies that the connection should be loaded but only established as soon as traffic is detected that demands IPsec processing.

We’ll also define the PSK used to secure the IPsec connection in /etc/ipsec.secrets. If you wish, you could use an RSA certificate or other key material instead - for this example, we’ll just use a simple pre-shared key though:

%any    %any    : PSK "mysecureipsecpsk"

Local Tunnel Exit IPsec Configuration

On the Local Tunnel Exit side, we’ll use the following configuration in /etc/ipsec.conf:

config setup
    nat_traversal=yes

conn tunnel
    left=192.168.100.2
    leftprotoport=17/%any
    right=203.0.113.2
    rightprotoport=17/1701
    authby=secret
    ike=aes256-sha256-modp2048!
    esp=aes256-sha256-modp2048!
    ikelifetime=1d
    lifetime=1h
    auto=route
    type=transport

First we enable the NAT traversal features, since our Local Tunnel Exit will definitely be connecting from behind a NAT. Next, the left and right sides of the tunnel are defined, as well as the traffic specifiers to identify traffic that will be processed. All other configuration options match those on the other side of the tunnel.

Again, the pre-shared key is defined in /etc/ipsec.secrets:

%any    %any    : PSK "mysecureipsecpsk"

We can test the IPsec configuration by starting Strongswan:

sudo service strongswan start

Then starting the L2TP tunnel as before in order to prompt the IPsec connection to be established. Afterwards, you should be able to see some ESTABLISHED IPsec Security Associations in the output of ipsec statusall.

Configuring Port Forwards

Once everything is up and running, you’re able to add Destination NAT (DNAT) ipfilter rules on the Local Tunnel Exit to forward traffic arriving at 203.0.113.1 to any host inside your private network - just as you would if you were using a regular, non-CGNAT Internet connection.

An example rule to direct traffic to a web server (TCP port 80) running on 192.168.1.2 might look like this:

iptables \
    -t nat -A PREROUTING -d 192.168.100.2 -p tcp --dport 80 \
    -j DNAT --to-destination=192.168.1.2

Going further, you could also configure Source NAT (SNAT) rules on both the Local and Remote Tunnel Exits to allow outbound connections from inside the network that are transported over the tunnel. This is less relevant to the CGNAT situation we’re dealing with here, but still a useful side-effect of having a tunnel in place. One reason that this can be particularly useful is evading traffic shaping/management imposed by the LTE broadband provider.