IPsec (AH, ESP, ISAKMP, IKE, SA, Transport & Tunnel Modes)

Published by Darron Toy on


hi and welcome. Internet Protocol
Security (IPsec) is a suite of network security protocols which are
used to provide authentication and encryption services to Internet Protocol (IP). In this video I will discuss Internet Protocol Security (IPSec)
including its various sub protocols and components i.e. Encapsulating
Security Payload (ESP), Authentication Header (AH), Internet Security Association & Key
Management Protocol (ISAKMP), Security Associations (SAs) and Internet Key Exchange (IKE). Before moving further please turn on the
subtitles for this video. Internet Protocol (IP) operates at layer 3 (Network Layer) of OSI model. once the Internet Protocol was designed, it was designed for the government networks, therefore security was not a
consideration since this government network was physically secure. Later on, Internet Protocol Security (IPSec) was introduced as an extension of
Internet Protocol (IP) to provide security services to Internet Protocol (IP) packets. IPsec also operates at layer 3 of OSI model. There are certain other
protocols which operates at higher layers of OSI model i.e. Layer 7 (Application Layer). These protocols are Secure
Shell (SSH) and Transport Layer Security (TLS) but IPSec can also provide security to the
higher layer data. since it operates at layer 3. Once we talk about the history of Internet Protocol Security (IPsec), National Security
Agency in 1986 introduced a program called Secure Data Network System (SDNS) program and they run this program from 1986 to 1991. Purpose of this program was to design security protocols for the security of Internet. A part of these security protocols i.e. certain security protocols at Layer 3 were
further enhanced and improved by various stakeholders from 1992 to 1995. In 1995, Internet Engineering Task Force (IETF) standardized these protocols and
published an RFC which was the implementation of Naval Research
Laboratory (NRL) of United States, so the first implementation of IPsec was from Naval
Research Laboratory (NRL) in 1995. IPSec is an open standard and
it is modular, it provide encryption and connectionless authentication services
to IP packets. Internet Protocol Security (IPsec) also used to establish Virtual Private Network (VPN) between 2 x Gateways (networks), between two hosts, or between gateway & host. Virtual Private Networks (VPNs)
are used to extend the private network over the Internet. IPSec operates at two modes of operations i.e. Transport mode and Tunnel mode. so once we talk about end-to-end communication then it
is the transport mode and once we talk about the security/ communication
between two networks then it is normally Tunnel mode. IPsec also used to provide security/ confidentiality to Layer 2 Tunneling Protocol (T2TP). so IPSec
provides confidentiality, Peer authentication (connection-less), data & source integrity (connection-less), replay protection and
access control services at layer 3. There are certain issues with IPSec that it
reduces the throughput of the network up to 80%. Network Address Translation (NAT) has some issues with the IPSec therefore we use NAT Traversal (NAT-T) to overcome these issues. IPSec can also creates Iterative Tunneling, it means that to IPSec tunnels are created inside each other e.g. one tunnel
may use the transport protocol and other tunnels may encrypt this transport mode
into tunnel mode. NSA has developed a device which is
encryption gateway and it is known as High Assurance IP Encryptor (HAIPE) device and this device uses IPSec. In 2013, Edward Snowden revealed that NSA had cracked IPSec protocol. There are 2 x sub protocols which are used inside IPSec i.e. Encapsulating Security payload (ESP), which provide encryption
services to IPSec/ IP and second is Authentication Header (AH), which
provides connection-less integrity. I will show you the structure of a
packet. so this is application layer data, and this is the TCP header which
operates at layer 4 and this is the IP header which operates at
layer 3 and this is the Ethernet header which operates at layer 2 now ESP header
is added/ inserted between Layer 4 and Layer 3 and it
provides encryption services to the data after it i.e. Layer 4 data & above till Layer 7. so ESP Header does not encrypts IP header but it only encrypts IP payload/ data (upper layers data). Authentication Header (AH) connectionless data integrity. This is the application data and this is the TCP header which operates at layer 4 and
this is the IP header which operates at layer 3. this is Ethernet Header which operates at layer 2. now AH header is added after the IP header and before the Ethernet Header, so it provides data integrity and non-repudiation to data above it so it also adds a new IP header after it encrypts data above it (IP header + payload). so ESP does not do anything with the IP header but AH also
includes the IP header. Encapsulating Security Payload (ESP) is used to provide confidentiality or encryption and
limited non-repudiation, since it does not encrypts the IP header and it only
encrypts the IP payload/data (layers above IP). ESP encrypts the IP payload but not the IP header and it is mainly used to provide
end-to-end security that is between a client and a server and it can also be
used to provide link encryption that is between two networks and two sites but
mainly once we refer to ESP it is used to provide end-to-end security ESP can be used in tunnel mode or it can be used in transport mode. so once ESP or encapsulating security payload is used in Tunnel mode, it is used
between two gateways or between a gateway/ host, so either side should be
Gateway. In Tunnel mode, it encrypts all the data that is the IP header and all
the data above it, but once it is used in Transport mode,
it only encrypts the IP payload not the IP header. so it is used for peer-to-peer
that is between two hosts or between a host and a gateway, so either side
should be a peer/ host/ gateway. In tunnel mode, either side should be a
gateway. In transport mode, either side should be a host. Network Intrusion Detection System (NIDS) can monitors the traffic before it is tunneled,
so it can monitor the traffic on the LAN before it is tunneled. After it is tunnel, then NIDS cannot monitor this traffic. In Transport mode, since it is end-to-end encryption, therefore we use host-based intrusion
detection system (HIDS) on the host, after the traffic is decrypted at the
host. ESP can be used with/ without Authentication Header (AH). once it is used with AH, it is used in Tunnel mode. since authentication header (AH) only works
in tunnel mode and once it is used without authentication header it is used
in transport mode Authentication Header (AH) provides connection less data and origin
integrity and we can also say that these both data integrity and source integrity
combines into non-repudiation in which the sender of the packet cannot deny
that he has not sent this packet. AH does not provide confidentiality but it provides Replay protection and it also tunnel the
data and it provides signing services to the data In Authentication Header (AH), we also
encrypt the IP header and also the IP payload and a new header is added, which I have showed you earlier. So AH provides link encryption that is between two sites (networks)
and authentication header is only used in Tunnel mode and it’s not used in
Transport mode. Tunnel Mode is used for communication between gateway-gateway, between gateway-host, so either side should be a gateway. AH is not used in transport mode, in which either side should be a host (Client/Server), or host can be a gateway. so there are certain
issues with authentication header since it encrypts the IP header, therefore
routing encryption and decryption creates some issues for the devices
which are between the source and the destination. it is very rare to see
that the authentication header is used without Encapsulating security
payload (ESP) and encapsulating security payload (ESP) provides confidentiality to
authentication header and authentication header will provide data/ source integrity There is an issue of network sniffing on the
LAN since the tunnel is established between Gateways, so after the tunnel is
terminated at the gateway then the traffic can be sniffed by the hacker on
the LAN. Internet Security Association and Key Management Protocol (ISAKMP) is a framework for creation of security associations (SAs) and
key exchanges (encryption). It provide background security services for these purposes to IPsec. It is used for authentication and denial of service protection and a replay protection. ISAKMP is a framework and actually key exchanges can be pre-shared, through Internet Key Exchange (IKE), through Kerberized Internet Negotiation of Keys (KINK), where we use a third party i.e. Key Distribution Center (KDC) for distribution of keys between two endpoints. We can also exchange keys through IPSECKEY Resource Record (RR) where public keys of domains are published, it uses DNS Security (DNSSEC) for
communicating the public keys of domains. Security Associations (SAs) are
session parameters for authentication header (AH) and
encapsulating security payload (ESP). Security Parameter Index (SPI) which is of 32-bit and SPI denotes the Security Association (SA). Security Associations (SA) is
simplex i.e. we have to create two Security Associations (SAs) for ESP (one for each direction) and then 2 x SAs for Aauthentication Header (one for each direction), so if we are using
AH and ESP both in IPSec then we need to create four SAs (2 x AH for inbound/outbound, 2 x ESP for inbound/ outbound). we can also filter IPsec traffic based on Security Association (SAs) and we can
filter various services and protocols. Internet Key Exchange (IKE) is used to setup security associations (SAs) and these are based on X.509 digital certificate
for authentication and this can be pre-shared or it can be shared through Domain Name System (DNS). There are 2 x Phases in Internet Key Exchange (IKE), In phase-1, ISAKMP SAs (Internet Security Association & Key Management Protocol’s Security Associations) are established i.e. Diffie-Hellman (DH) for key exchange and Phase-I is actually the management traffic to
establish a policy for IPsec devices and this security association (SAs) is
bi-directional i.e. we have to create only a single Security Association (SA) for both side. In phase-2 of Internet Key Exchange (IKE),
IPSec SAs are established for data protection and
these SAs are simplex (uni-directional) that I have explained earlier i. e.Two x SAs (one for each direction). Highest protocol (most secure) will be selected by Internet Key Exchange (IKE) e.g. Data Encryption Standard (DES) will be
superseded by Advanced Encryption Standard (AES)


Leave a Reply

Your email address will not be published. Required fields are marked *