The OpenContrail solution uses overlays for network virtualization.
An earlier blog evaluated the performance of the OpenContrail virtual router (vrouter) when run as a kernel module on Linux. As described there, the vrouter module is able to fill a 10G link with TCP traffic from a virtual machine (VM) on one server to a VM on another server without making any assumptions about hardware capabilities in the server NICs. Also, in order to support interoperability and use a standards-based approach, vrouter does not use new protocols/encapsulations. However, in network function virtualization (NFV) scenarios, other performance metrics such as packets-per-second (pps) and latency are as important as TCP bandwidth. With a kernel module, the pps number is limited by various factors such as the number of VM exits, memory copies and the overhead of processing interrupts. In order to optimize performance for NFV use cases, vrouter has now been integrated with the Intel DPDK (Data Plane Development Kit).
DPDK is a set of libraries and drivers that perform fast packet processing by allowing NICs to DMA packets directly into an application’s address space and having the application poll for packets, thereby avoiding the overhead of interrupts from the NIC. To integrate with DPDK, the vrouter can now run in a user process instead of a kernel module. This process links with the DPDK libraries and communicates with the vrouter host agent, which runs as a separate process. Figure 1 compares the architecture when vrouter runs as a kernel module versus running as a user process. The application inside the guest VM can be written to use the DPDK API or it can use the traditional socket API. However, for NFV applications such as vMX, which require high performance, it would be preferable to use the DPDK API inside the VM.
Vrouter running as a kernel module
Vrouter running in user space – Figure 1
To evaluate the performance of vrouter running in user space, a setup with 2 servers connected back to back by an Intel 10G interface was used. The servers have 2 CPU sockets each, with 6 cores per socket and 2 threads per core. The processor is an Intel Xeon running at 2.5GHz. The servers have 128GB of memory each and run Ubuntu 12.4.3 as the host operating system. A virtual network is created and a VM is instantiated on each server in this virtual network. Each VM has 4 VCPUs, 8GB of memory and runs Ubuntu 12.04 as the guest operating system. VM1 runs an application (pktgen), which continuously sends 64 byte packets to VM2. VM2 also runs pktgen, which consumes the packet and displays the number of packets thus received. Figure 2 illustrates the setup used to measure performance.
Figure 2
On this setup, vrouter running in user space is able to forward 4.15 million packets per second from VM1 to VM2. Packets sent by VM1 are encapsulated by vrouter in MPLS-over-UDP before they are sent on the wire. On the receiving server, vrouter decapsulates the packets and sends them to VM2. On the same setup, VMs connected by Linux bridge are able to forward about 500,000 packets per second using the netperf application, so the performance is significantly higher when using vrouter integrated with DPDK. Another benefit of integrating vrouter with DPDK is that a kernel module is not required any more to perform the overlay function. The use of a user-space vrouter is oblivious to other components of the OpenContrail solution, so the control plane, analytics and UI function the same as before. Also, the VMs can communicate with any hardware gateway router (such as the Juniper MX) as before. The only change required is to specify that DPDK based vrouter should be used on the compute servers when a cluster is provisioned. A video of the above performance test can be found below and also on the videos section.
[video_lightbox_youtube video_id=”ZGiQJrKoDQM” width=”720″ height=”540″ auto_thumb=”1″]
In summary, the OpenContrail solution achieves significantly better packets-per-second numbers when vrouter is integrated with DPDK, thereby enabling NFV applications with high performance requirements.
