For an overview of this blogpost, check the below video. Also you can find similar videos in the Video section of this site.
[video_lightbox_youtube video_id=”B8aHoY–1Zs” width=”720″ height=”540″ auto_thumb=”1″]
Openstack networking can help us deliver the benefits of network virtualization to applications – lower lead times for deploying services, better resilience to failures and easier scaling. But virtualization presents a new paradigm for visibilty and debugging of network elements, physical and virtual. Applications live their lives in the overlay network. Sometimes they face connectivity, latency and other performance problems. Is it a network problem? How do we isolate it and solve it?
To do this, we really need to map what’s happening in the overlay network with the networking entities that are operating in the physical network. In this presentation, we will demonstrate an Analytics application using the OpenContrail system’s open API’s to look at a traffic flow in the overlay, and examine what physical routers and network interfaces it is traversing in the underlay. This can be co-related to interface statistics and other health indications in the underlay network. We can also look at the physical routers being traversed by tunnels that go across from one underlay entry point to another. Correlating physical and virtual network information is these ways is a big step towards identifying, solving and even anticipating the connectivity problems of applications in OpenStack.
Collecting Information from Physical Nodes
Discovered topology information and interface statistics are ingested via SNMP. We rely on lldpTable, ipMib, ifTable and ifXTable. Each physical router is represented by the pRouter UVE (see Operational State in the OpenContrail system: UVE – User Visible Entities through Analytics API), which exposes these MIBs. We also correlate this information against vRouter UVEs to present physical topology in the pRouter UVE – a list of all other pRouters and vRouters that are connected this pRouter, and the links between them.
OpenContrail Analytics’ collector can take in flow information from physical nodes in the form of sFlow and IPFIX. The collected records can then be accessed via the Query REST API (see OpenContrail Analytics Query API) , with the following schema:
http://127.0.0.1:8081/analytics/table/StatTable.UFlowData.flow/schema
Physical Topology
Using the information listed above, the OpenContrail UI presents an interactive view of the Phyisical Topology of the network. Along with the switches/routers, we display details about the links between them, including traffic statistics along that link.
For TORs, we show the vRouters hanging off them, and the VM instances that run on those vRouters. The interfaces that connect them and their traffic statistics can be shown as well.
Search Flows
We can list historical overlay flows records seen on the system, according to the flow parameters used for REST API Flow Record queries, i.e. Source-VN/Source-IP, Dest-VN/Dest-IP, vRouter, Protocol/Source-Port and Protocol/Dest-Port.
Then we can map these flows onto the Physical Nodes to see what path they took. The underlay flow parameters used for a given overlay flow depend on the type of encapsulation being used. MPLS-over-UDP and VXLAN encapsulations try to add entropy (for better load-balancing between paths) by varying the UDP source-port based on overlay flow parameters. Multiple overlay flows are expected to hash onto the same underlay flow. When looking at sFlow or IPFIX information of underlay flows to infer the path of a given overlay flow, we can exploit all samples of the underlay flow, even if they were actually due to other overlay flows.
Trace Path
The vRouterAgent supports an active probe-based Trace Path mechanism via its HTTP Introspect API.
http://127.0.0.1:8085/diag.xml#Snh_TraceRouteReq
The UI displays all recently active overlay flows for a given vRouter or Virtual Machine. We can then trace the corresponding underlay flow using active probes that have the same outer header as regular packets of this underlay flow. The active probes are sent out using escalating TTLs, to elicit ICMP packets from the physical routers that the underlay flow would traverse. The destination vRouter detects this as a special probe packet based on the inner header, and replies back to the originating vRouter, thus completing the illumination of the path.
Conclusion
OpenContrail Analytics exposes detailed information about overlay flows in the network, and operational information about vRouters. We have also seen the physical network information that can be ingested into OpenContrail Analytics and presented northbound. Using all these APIs, an application can be built that displays the physical topology, and shows the physical path taken by an overlay flow. We have built one such application – other applications can be built on top of these APIs as well.
