While server virtualization is exploding into data centers around the world, the technology for securing virtual environments is lagging behind. It's a fact likely to give some organizations that have gone down the virtualization path some very acute server security headaches, according to James Collinge, product line management director at network security solution provider TippingPoint, a division of HP. Collinge expressed this view at the Infosecurity Europe 2010 conference in London earlier this year.

To get some perspective, at least 16 percent of all enterprise workloads were running in virtual servers in the latter part of 2009 according to Gartner, but this number is predicted to grow to 50 percent -- or around 58 million x86-based virtual servers -- by 2012. But Gartner predicts that some 60 percent of those virtual servers will be less secure than the physical machines they replace.

Forrester Research also points out that 98 percent of organizations that use some form of virtualization are using VMware virtualization technologies. That means that if a zero-day exploit is discovered for VMware, it is likely to be more interesting to hackers than a similar one for a given mail, web, or DNS server would be.

Why will virtualized servers be less secure than the physical machines they replace? Some of the reasons for this lower level of server security that Collinge mentioned include:

• Security considerations are not taken into account from the very beginning in many server virtualization projects
• All the virtualized workloads have the potential to be compromised by a single compromise of the virtualization layer
• Virtualized workloads that have different trust levels are often consolidated onto a single physical host without sufficient separation
• Many organizations lack adequate controls for administrative access to the hypervisor/virtual machine monitor layer and to administrative tools

In terms of actual threats to a virtualized environment, these fall into a number of categories -- centered on the hypervisor -- such as:

• Hyperjacking: This involves subverting the hypervisor or inserting a rogue hypervisor. Since hypervisors run at the most privileged ring level on a processor, it would be hard or even impossible for any OS running on the hypervisor to detect. In theory, a hacker with control of the hypervisor could control any virtual machine running on the physical server.
• VM Escape: As the name suggests, an exploit that enables VM Escape allows a hacker who compromises a specific virtual server to escalate his attack from the virtual server to take control of the underlying hypervisor.
• VM Hopping: Similar to VM Escape, VM Hopping allows an attack to move from one virtual server to compromise other virtual server on the same physical hardware
• VM Theft: This is the ability to steal a virtual machine file electronically, which can then be mounted and run elsewhere. It is an attack that is the equivalent of stealing a complete physical server without having to enter a secure data center and remove a piece of computing equipment.

What all this goes to show is that when organizations introduce a virtualized environment, they introduce a new mission-critical element: the hypervisor. Since successful attacks on hypervisors can lead to the compromise of all the hosted workloads -- and since successful attacks on individual virtualized workloads can also lead to a compromise of the hypervisor -- the organization's hypervisors should be considered mission-critical and secured appropriately, Collinge said.

In a traditional IT environment, network traffic can be monitored, inspected and filtered using a range of server security systems to try to detect malicious activity. But a problem with virtualized environments is that local communications between virtual servers that run through a virtual switch is largely invisible: It never "hits the wire" where it can be monitored in the normal way. There's only one solution to that, Collinge said he believes. "Visibility and control of VM-to-VM traffic flows must be established."

A compounding problem is the separation of duties that often occurs in a virtualized data center. Server and operations teams are often responsible for the provisioning and management of virtual switches. Little or no integration with tools and security controls is implemented. For the network and security teams, this leads to a lack of visibility to perform configuration auditing, and it makes it difficult to detect topology and configuration changes, Collinge said, noting, "Network and security teams must regain visibility at the access layer."

Collinge cited three ways to achieve this:

1. Hardware-Based Approach

This involves forcing traffic between ESX hosts to be inspected by an intrusion prevention system (IPS). The system Collinge describes has each ESX host configured with a unique ingress/egress VLAN pair, with the IPS configured with VLAN translation to configure each ingress VLAN and egress VLAN pair. This ensures all VM-to-VM traffic is sent out "over the wire" to the IPS for inspection, and only clean traffic is allowed to travel between each ingress/egress VLAN pair. A disadvantage of this approach is that it can be very costly to replicate this at multiple data centers and disaster recovery sites.

2. Fully Virtualized Approach

With this method, a virtual IPS and firewall is implemented on each ESX host, with policy configured on each virtual machine to decide what traffic should be inspected. This "bump in the wire" approach ensures all allowed intra-virtual machine traffic is inspected, and it has the bonus that when virtual machines are moved between physical hosts, the security policies move with them The downside, however, is there can be significant performance penalties with this architecture.

3. Hybrid Approach

This is an alternative way of doing things that largely mitigates the performance issues of the fully virtualized approach. It involves running a virtual redirector on each virtual machine, configured with a policy on what traffic should be redirected -- to a physical IPS -- for inspection. The IPS allows only inspected and clean re-directed traffic to travel between virtual machines.

So which is the best approach? Collinge says that what best suits a given organization will depend on its goals and budget, as well as its attitude to risk. Some successful solutions will probably involve a combination of two of these three approaches.

The good news is that building and implementing these types of solutions is likely to become easier in the near future, Collinge said, as security companies develop a larger range of products that provide the necessary functionality.