IPsec VPNs: remote access and site-to-site, protocols and modes

Why this matters

Companies with more than one building, and staff who work from home, need their private traffic to cross the public internet without anyone on the way reading or changing it. Leased private circuits do this but cost a great deal. A virtual private network (VPN) does it over ordinary internet links by wrapping each packet in protection before it leaves and unwrapping it at the far end. A network technician meets VPNs at every branch office and in every remote-working ticket: "the tunnel to the warehouse is down", "the VPN client connects at home but not at the hotel".

Most of those tickets come down to a few facts about how IPsec works: which protocol carries the protected data, which ports a firewall must allow, what happens when NAT sits in the path, and why routing protocols sometimes refuse to cross a tunnel. You do not need to build a VPN from memory for this topic, but you do need to describe its parts precisely enough to see which one has failed.

The lesson

Remote-access and site-to-site IPsec VPNs: protocols and modes

IPsec is a framework of open standards for protecting IP packets. It can provide four services:

• Confidentiality: the payload is encrypted, so a capture shows only scrambled data. AES is the usual cipher.
• Integrity: a keyed hash, such as HMAC-SHA-256, proves the packet was not changed on the way.
• Origin authentication: the receiver knows the packet came from its real peer.
• Anti-replay: sequence numbers let the receiver drop copies of old packets that an attacker resends.

It does not retransmit lost packets, which is TCP's job, and it does not translate addresses.

The two kinds of VPN

A site-to-site VPN joins whole networks. Each site has a VPN gateway, usually its edge router or firewall, and the gateways build the tunnel between their public addresses. Hosts on each LAN need nothing: they send ordinary traffic to their default gateway, and the gateway encrypts whatever matches its VPN policy. A PC at a branch never knows its traffic to head office was protected.

 Branch LAN                                                      HQ LAN
10.20.0.0/24 --[R1]==== IPsec tunnel across the internet ====[R2]-- 10.10.0.0/24
               203.0.113.2                          198.51.100.2

A remote-access VPN connects one user's device to the company. The laptop runs VPN client software, such as Cisco Secure Client, and builds its own tunnel to a headend at head office, for example a Cisco Secure Firewall. Remote-access clients can tunnel with IKEv2 IPsec, or with TLS (and DTLS, its UDP form, for better performance), whichever the headend allows. TLS-based VPNs use TCP 443, the same port as HTTPS, which is why they work from most hotel and café networks. Users authenticate with credentials or certificates, often checked by a RADIUS server (lesson 17).

IKE: agreeing how to protect traffic

Before any data is protected, the two peers must agree on algorithms and keys. IKE (Internet Key Exchange) does this. It runs over UDP port 500. IKE:

1. Authenticates the peers to each other, with a pre-shared key or with certificates and RSA signatures.
2. Runs a Diffie-Hellman (DH) exchange. Each peer combines its own private value with the other's public value, and both arrive at the same shared secret. The secret itself never crosses the network, so someone capturing the exchange cannot work it out.
3. Negotiates the security associations (SAs): the agreed set of cipher, hash, keys and lifetime for the traffic. IPsec SAs are one-way, so a working tunnel has at least one pair; the IKE SA itself is two-way.

IKE only negotiates. It does not encrypt each data packet; that is ESP's job. IKEv1 does its work in two phases, using exchanges called main mode or aggressive mode for phase 1 and quick mode for phase 2. Those names are IKE exchanges, not IPsec modes. IKEv2 is the newer, simpler version and is the default on current Cisco devices.

ESP and AH: protecting the packets

IPsec has two protocols for the data itself. Each is its own IP protocol, not TCP or UDP.

ESP (Encapsulating Security Payload) is IP protocol 50. It provides encryption, integrity, origin authentication and anti-replay. Almost every VPN today uses ESP.

AH (Authentication Header) is IP protocol 51. It provides integrity, origin authentication and anti-replay, but never encrypts. Its integrity check also covers the fixed fields of the outer IP header, including the addresses. That is why AH fails through NAT: when a NAT device rewrites the source address, the check no longer matches, and the receiver drops the packet. AH is rarely deployed.

So a firewall between two site-to-site peers with no NAT between them must permit UDP 500 for IKE and IP protocol 50 for the ESP data.

NAT traversal

ESP has no port numbers, so a PAT device has nothing to track and cannot translate it. NAT traversal (NAT-T) solves this. During IKE, the peers detect that a NAT device sits between them. From then on, they wrap ESP inside UDP port 4500, giving the NAT device a port it can translate, and IKE moves to port 4500 as well. A home user behind a broadband router relies on this every day. NAT-T is defined for ESP, not for AH.

Tunnel mode and transport mode

IPsec has two modes, which decide how much of the original packet is protected.

In tunnel mode, the whole original IP packet, header included, is protected and placed behind a new outer IP header. The outer header carries the gateways' public addresses; the inner header keeps the hosts' private addresses, hidden by encryption. Site-to-site VPNs use tunnel mode, because the hosts' private addresses could not be routed across the internet anyway.

Tunnel mode:    [New IP: 203.0.113.2 > 198.51.100.2][ESP][Original IP: 10.20.0.5 > 10.10.0.9][TCP][Data][ESP trailer]
                                                         |<--------------- encrypted ---------------->|
Transport mode: [Original IP header][ESP][TCP][Data][ESP trailer]
                                         |<---- encrypted ---->|

In transport mode, the original IP header stays in place and only the payload is protected. It suits traffic between the two IPsec endpoints themselves, where the original header already carries the right addresses.

GRE over IPsec

An IPsec tunnel defined by a crypto map protects only unicast traffic that matches its access list. OSPF, which sends multicast hellos (lesson 15), cannot form neighbors through it. GRE (Generic Routing Encapsulation), IP protocol 47, solves this. GRE builds a virtual point-to-point tunnel that can carry multicast and routing protocol packets. GRE on its own adds no encryption at all, so designers run GRE over IPsec: GRE carries the routing traffic, and IPsec encrypts the GRE packets. IKE is still needed to build the SAs.

GRE already adds an outer IP header between the two routers' public addresses. If IPsec then used tunnel mode, it would add yet another header. Transport mode keeps GRE's header and protects the rest, saving 20 bytes in every packet.

An outline of the configuration on R1 (the IKEv2 settings are left out to keep it short):

R1(config)#crypto ipsec transform-set TS esp-aes 256 esp-sha256-hmac
R1(cfg-crypto-trans)#mode transport
R1(cfg-crypto-trans)#exit
R1(config)#crypto ipsec profile GRE-PROT
R1(ipsec-profile)#set transform-set TS
R1(ipsec-profile)#exit
R1(config)#interface Tunnel0
R1(config-if)#ip address 172.16.99.1 255.255.255.252
R1(config-if)#tunnel source GigabitEthernet0/0/1
R1(config-if)#tunnel destination 198.51.100.2
R1(config-if)#tunnel protection ipsec profile GRE-PROT

The transform set names the ESP cipher and hash: AES-256 for confidentiality and HMAC-SHA-256 for integrity. Checking the session shows IKEv2 on UDP 500 and an IPsec flow for protocol 47, the GRE traffic:

R1#show crypto session
Crypto session current status

Interface: Tunnel0
Session status: UP-ACTIVE
Peer: 198.51.100.2 port 500
  Session ID: 1
  IKEv2 SA: local 203.0.113.2/500 remote 198.51.100.2/500 Active
  IPSEC FLOW: permit 47 host 203.0.113.2 host 198.51.100.2
        Active SAs: 2, origin: crypto map

Two active SAs are the inbound and outbound pair. If a NAT device sat between the routers, the ports would read 4500. Modern designs often use a virtual tunnel interface (VTI), which protects everything routed into a tunnel interface without GRE, but the exam topic here is the protocols and modes, not each design.

Reading a VPN fault from these facts

The protocol facts above are what turn a vague "the tunnel is down" into a short list of checks.

1. No IKE SA at all. The peers never agreed. Check that UDP 500 reaches each peer, that the pre-shared keys or certificates match, and that both sides offer at least one common set of algorithms.
2. IKE up, but no traffic passes. The SAs exist but the data is blocked or not selected. Check that a firewall in the path permits IP protocol 50, and that the traffic actually matches the VPN policy, such as the crypto map's access list.
3. Works from the office, fails from home or a hotel. A NAT device is in the path. Check that UDP 4500 is allowed, or use a TLS-based remote-access connection on TCP 443.
4. Pings cross the tunnel, but OSPF never forms a neighbor. The tunnel carries only unicast. Move to GRE over IPsec or a VTI.

What the exam is really asking

• Who needs software? Remote access: a client on every device. Site-to-site: nothing on the hosts, the gateways do it all.
• ESP encrypts, AH does not. ESP is protocol 50; AH is protocol 51 and breaks through NAT because it checks the addresses NAT changes.
• IKE negotiates, ESP protects. IKE on UDP 500 authenticates peers, runs Diffie-Hellman and builds SAs.
• Diffie-Hellman derives a shared key without sending it. RSA signatures prove identity; AES encrypts; HMAC checks integrity.
• NAT in the path means UDP 4500.
• Tunnel mode wraps the whole packet in a new header; transport mode keeps the original header. Site-to-site uses tunnel mode; GRE over IPsec usually uses transport mode.
• Main, aggressive and quick are IKEv1 exchanges, not IPsec modes.
• GRE carries multicast but does not encrypt. That is why OSPF across a VPN usually means GRE over IPsec.
• Remote-access clients can use IKEv2 IPsec or TLS. TLS VPNs use TCP 443.