Various VPN Encryption Schemes

Various VPN Encryption Schemes

As you may have figured out by now, not all VPNs are created equal and this applies to their encryption techniques as well. In the Internet world of competing standards, VPNs are no exception — the vendors all tout their solutions as the best. However, some standards have become standard options. These are the VPN protocols that handle authentication, tunneling, and encryption.


Point To Point Tunneling Protocol (PPTP) started life as the Point To Point Protocol (PPP). PPP was, and is still used in some places, to allow dial-up users to connect to the Internet. Many ISPs that sell dial up accounts are still using PPP for their connections. but it is not used for VPNs.

PPTP was developed by Microsoft and U.S. Robotics, using PPP as the basis from which to start. The authentication portion of PPTP used Microsoft’s authentication protocol, Challenge Handshake Authentication Protocol (CHAP) which was a very bad product, indeed. Today’s implementations of PPTP should be using CHAP v2 which has closed some of the gaping holes in the original version. (Be sure to check which version of CHAP your product uses!)

PPTP does not encrypt the traffic by default because its main job is to create a virtual tunnel in which to transport the data and to authenticate users. But, it can be configured to encrypt the tunnel as well using the RC4 algorithm. The default configuration for RC4 is 40 bits, which is horribly weak, but it can also be set to run with 128-bit encryption.

Because RC4 is a symmetric algorithm, both the user and the VPN appliance with share the same secret key. That key is thrown away as soon as the session is over. Most Cisco routers are able to handle PPTP but you’ll need to check that your particular vendor and model number are compatible with PPTP.

PPTP VPNs are normally used in client-to-gateway or network-to-network connections or for legacy Windows clients. If you have a mix of operating systems, one of the other VPN protocols might work better for you.


Layer 2 Tunneling Protocol (L2TP) is actually a combination of the Layer 2 Forwarding (L2F) protocol that was developed by Cisco and PPTP which was developed by Microsoft and US Robotics. In the early days of VPNs, it looked as if these two protocols (L2F and PPTP), which were not at all compatible, would duke it out for prevalence in the market. But, the IETF (Internet Engineering Task Force) asked that MS and Cisco play nicely and see if they could work together on a product that gave us the best of both their worlds. Thus, L2TP was born.

L2TP is able to authenticate users, create a virtual tunnel, and both encrypt and compress the data traveling within that tunnel. Please note, however, that the tunnel itself is not encrypted; only the data is encrypted. I say that with one caveat: this is only true if you are using a non-Microsoft L2TP set up. If you are using Microsoft products for your L2TP, you must also include IPsec in your configuration because MS’s L2TP product does not support encryption.

One of the beauties of L2TP is that it can be used in networks where more than one network operating system is in use. Yes, this means it will operate over IPX, NetBEUI, and IP networks.


I’m sure you’re familiar with TCP/IP and are aware of the fact that this suite of protocols was developed without a mind towards security. TCP/IP was developed with “openness” in mind. None of the people who worked on its development could conceive of all the different ways it would be used to circumvent security. With security in mind, IPsec was developed. IPsec is an upgrade or enhancement to TCP/IP that provides security features, including encryption capabilities. It’s a very good system to use in conjunction with VPNs.

The end-user wouldn’t notice the difference between TCP/IP and IPsec, but there are a lot of differences to network administrators. The set up and configuration is not exactly straightforward and the installer needs to have a good understanding of both IPsec and IKE (Internet Key Exchange).

If the installer does not have a good understanding of IPsec, he won’t understand what the various configuration options mean. In addition, some of the IPsec terminology has not been standardized and it’s easy to misunderstand what a certain configuration mode means.

The Internet Key Exchange (IKE) protocol is an encryption key management protocol which is used in conjunction with the IPsec. IPsec can be configured without IKE, but IKE enhances IPsec by providing additional security features.

IPsec requires the use of either keys or Digital Certificates to be able to work its magic. If you plan to use digital certificates, you’ll have to set up a PKI system to handle the creation and dissemination of the digital certificates. On the other hand, if you plan to use keys, these will be symmetric keys and you have to come up with a plan to safely exchange the keys. When an IPsec connection is initiated, the computers exchange information about themselves first. After that information has been passed (and it happens very quickly), the person on the computer originating the connection has to type in the encryption key information. Once this information has been correctly transferred, the data traveling between the systems will be encrypted.

There are also two modes IPsec may use: tunnel and transport. Transport mode would typically be used where communications terminate at IPsec servers, e.g. an IPsec server talking to another IPsec server. Tunnel mode is more often used when you have desktop computers running IPsec communicating with IPsec servers. Tunnel mode encrypts the header and the payload of each packet while transport mode only encrypts the payload. Only systems that are IPsec-compliant can take advantage of this protocol. For all these systems to be able to communicate with one another, all computers must use a common encryption key or Digital Certificate. Additionally, computers must also have very similar security policies set up.

Combining IPsec with the normal VPN tunneling protocols provides the best solution for most companies because you get a secure, encrypted tunnel as well as encrypted data transport.

 Python   SQL   Java   php   Perl 
 game development   web development   internet   *nix   graphics   hardware 
 telecommunications   C++ 
 Flash   Active Directory   Windows