Network Security: Limitations Of The Internet Protocol

And The IPSec Protocol Suite

Ivan Dimitrov

    Department of Computing and Software, McMaster University, Hamilton, Ontario


    Over the past decade, the Internet has grown from a small network connecting a community of researchers to its present state - a big global network, connecting people and organizations all over the world. The role of the Internet has also drastically changed with the help of technological advances. It is no longer a big research project, but used as a general-purpose tool. Individuals use the Internet for on-line banking and other personal transactions, and large organizations, such as business corporations use it constantly to carry out tasks such as ordering materials from suppliers, or communicating information back and forth with other businesses.

    Many organizations have large Local Area Networks (LAN) with thousands of computers, which are open to remote employees, suppliers and business partners. While this speeds communication and makes most business operations effortless, it provides numerous people with access to confidential information, such as personnel records. With the growing amount of sensitive and valuable information that is transferred across the Internet continuously, protection from unauthorized access has become a major concern. Understanding the underlying network technology is important to understanding issues with Internet security.

    The Transmission Control Protocol (TCP) and Internet Protocol (IP) are the basis for today's Internet. The two protocols were developed in the early 1980's under the sponsorship of the US Department of Defense. At that time the Internet was small, and its users generally trusted each other. Because of this the TCP/IP protocol suite lacks even the most basic mechanism for security, encryption. This means that any person who has even limited understanding of the TCP/IP protocol suite can obtain access to any information traveling on a network.

    The following two sections describe in more detail the security problems associated with the TCP and IP protocols, and discuss an alternative protocol suite, IPSec, developed by the Internet Engineering Task Force (IETF) that solves these problems.


    As mentioned above the IP protocol underlies the vast majority of large corporate and academic networks as well as the Internet. Its strength lies in its easily and flexibly routed packets. IP's job is to deliver a packet from one machine, the source machine, to another, the destination machine. It provides a connection-less service; each packet is independent of the other, in that each packet of information can take a different path to the destination machine. It is also very common for packets arrive out of sequence.

    IP's strength, however, is also its weakness. The IP protocol uses unencrypted packet headers, so that any host along the path of a packet can read the source and destination addresses within the IP header. This makes IP networks vulnerable to a number of security risks, such as IP Spoofing, Packet Sniffing, and Session Hijacking.

    IP Spoofing is an attack where an attacker pretends to be sending data from an IP address other than the address of the source machine. In turn the destination machine will send information to the IP address specified as the source address, which has been changed. IP Spoofing is possible for two reasons. The first is the lack of encryption, which permits any host along the path of a packet to read and change the information within the IP Header. The second is that the IP protocol does not provide a method for authenticating packets. The receiver has no way of verifying that the packet has not been altered along the way.

    The second security risk is Packet Sniffing, also known as Electronic Eavesdropping. This is another attack that is possible on Ethernet-based networks. Packets traversing a LAN are available to every node on the network, and each node's network interface card is capable of collecting all packets. Normally, network interface cards only listen and respond to packets addressed to them, however a card can be put in what is known as "promiscuous mode". This enables a card to collect every single packet that it sees.

    Another security risk with IP-networks is Session Hijacking or Man-in-the-Middle attack. This is a variant of IP Spoofing, where a host inserts itself in the middle of a connection, acquiring access to information transmitted between the other hosts.


    The IETF is a large community of network designers, administrators, vendors and researchers, concerned with evolution of the Internet architecture and technology. The IETF has developed the IPSec protocol suite as an extension to the basic IP protocol. It is based on modern cryptographic technologies making possible strong data authentication and encryption. The IPSec eliminates all of the network security problems associated with the IP protocol. It works on the network level, Layer 3 on the protocol stack. Because of this it is invisible to applications. This is a feature that sets IPSec apart from other Internet security technologies that run at other layers, such as e-mail and web browser encryption schemes. IPSec is compatible with current Internet standards (Ipv4 and Ipv6), and makes the whole network secure. This means that any applications running on the network will also benefit.

    The IPSec protocol suite has two modes of operation, Tunnel mode and Transport mode, and includes three IP extensions:

    ESP and AH are the basic building blocks of IPSec. Both employ cryptographic techniques to ensure data confidentiality and digital signatures for source authentication. ESP encrypts the TCP packet using a symmetric encryption algorithm. Symmetric encryption is an encryption scheme that uses the same key to encrypt and decrypt the data.

    ESP is made up of six parts:

    The following four parts are all encrypted, when on the network.

    When the IPSec packet is created, the ESP is inserted after the TCP header, but before the IP header. This is because ESP relies on the IP header for routing. An optional component of ESP is the authentication field, which is simply a digital signature computed over the other parts of the ESP after encryption. It is added at the end of the ESP packet and allows the receiver to verify that the packet has not been changed and it came from the expected source.

    The AH provides different authentication service than the ESP. It protects both the external IP header and the entire contents of the packet, and contains the following:

    The third Internet Protocol extension is IKE. It enables the use of the authentication and encryption services provided by AH and ESP. It specifies how protocol negotiations and key exchanges between the parties are carried out. IKE accomplishes this in two phases. It first establishes a secure channel between the two parties, and then Security Associations (SAs) are exchanged. An SA is a set of information that is needed for successful communication. It includes things such as authentication algorithms to be used and the keys to these algorithms, how to authenticate the communication (what protocols to use), how often to exchange keys, the key life time, the life time of the SA, etc. Once negotiation between the two parities is complete encryption algorithms are determined. Two different algorithms are generated, one for inbound and one for outbound communication.

    As mentioned earlier IPSec has two modes of operation. The Transport mode is used for peer-to-peer communication to provide LAN security. Because the IP header is not encrypted it can be read and used by any of the standard network devices, such as routers. However, ESP encrypts and authenticates the packets, so that they are protected.

    The second mode is Tunnel mode and is used for site-to-site security. The whole packet is encapsulated in the ESP with a new IP header, protecting the topology of the site. While traveling on the Internet, the packet appears as a regular IP packet.

    When in Tunnel mode, IPSec can be used to create a Virtual Private Network (VPN), which is as safe as isolated office LANs, but is build on an insecure public network. A VPN is a system in which a public network is used to transmit the data of a private network. For example, a VPN can exist between a company and its supplier that is located on another continent. The company is one host and the supplier the other. By abstracting from the many networks that lie between the two and viewing the connection as a point-to-point, the result is a VPN. It is far more cost effective to use public lines than to lease a line. The IPSec protocol suite makes communicating over public networks secure, and as long as the company and its supplier are IPSec-compliant, the transmitted information is secured.


Brien M. Posey, Understanding the IPSec protocol. Published August 24, 2000.

Intel Corporation, 2000. IP Security: Building Block for the Trusted Virtual Network.

Alcatel. Understanding the IPSec Protocol Suite, March 2000.