This is a guest blog by Johnny Chen from AT&T with contributions by Qasim Arham, Henon Huang, Vijay Kamisetty from Juniper Networks.
Overview
The purpose of this is to lay out a method of tracing flows through OpenStack & Contrail environment with a scaled-out Service Instance (virtual firewall) in the path.
Environment Versions:
OpenStack == Juno (2014.2.4)
Contrail == 3.0.0
Table of Contents
Step 1: On the SDNGW, find the NEXT-HOP (NH) to the vFW VM in the routing table
Step 2: Verify Traffic to vFW Service Instance VM Compute
Step 3: Trace Flow into vFW Service Instance VM
Step 4: Verify Traffic is leaving vFW towards DST VM
Step 5: Verify Traffic is getting to DST VM TAP Interface
Step 6: On DST VM, Verify Traffic Inbound
Step 1: On DST VM Compute, Trace Return Flow
Step 2: Verify Return Traffic hitting same vFW as Inbound Path
Background & Drawings
Access to some useful elements for this exercise:
|
RESOURCE
|
DESCRIPTION
|
|
OpenStack Horizon
|
OpenStack Horizon Web GUI
|
|
Juniper Contrail
|
Contrail Web GUI
|
|
SDNGW
|
SDN Gateway
|
|
Compute
|
Contrail vRouter Commands and tcpdump
|
|
Splunk or Juniper JSA/STRM Console
|
SEIM Log and Correlation (Optional)
|
Caveat: There is often more than one way to get to the end result so while this is an attempt to document one method, there could be other ways of achieving to a diagnosis. For this example, we will use a lab network depicted in the drawing below as a reference and trace the packet flow between 12.12.0.102 (SRC) and 10.10.0.100 (DST) where the destination is a VM in the Overlay Network.
The flow path is as follows:
12.12.0.102 <> Physical Network <> SDN-GW <> VN_A <> FW Service Instance <> VN_B <> 10.10.0.100
Tracing: Outside => In
STEP 1: On the SDNGW, find the NEXT-HOP (NH) to the vFW VM in the routing table.
1a. Single VM or Compute (Single Next-Hop)
|
In a single VM architecture for FW, you would see a single next-hop in the table for that routing-instance. However, in the case of a scaled-out service instance, there might be multiple next-hops pointing to the computes where the FW VM’s instantiated on. When dealing with Composite Next-Hop, identifying which FW on which Compute can get tricky (either log into all the FW’s and look through the session tables for the flow in question or use a SEIM or Log Correlator like Splunk or STRM/JSA). Here, we will focus on Single VM or Compute (Single Next-Hop).
|
On SDNGW:
SDNGW> show route 10.10.0.100 vntest.inet.0: 50 destinations, 150 routes (50 active, 0 holddown, 0 hidden) @ = Routing Use Only, # = Forwarding Use Only + = Active Route, - = Last Active, * = Both 10.10.0.0/24 *[BGP/170] 1w1d 21:30:43, MED 100, localpref 200, from 172.20.0.5 AS path: ?, validation-state: unverified > via gr-0/1/0.19283, Push 31 [BGP/170] 1w1d 21:30:43, MED 100, localpref 200, from 172.20.0.6 AS path: ?, validation-state: unverified > via gr-0/1/0.19283, Push 31 SDNGW> show interfaces gr-0/1/0.19283 Logical interface gr-0/1/0.19283 (Index 345) (SNMP ifIndex 526) Flags: Up Point-To-Point SNMP-Traps 0x4000 IP-Header 172.20.0.23:10.10.10.10:47:df:64:0000000800000000 Encapsulation: GRE-NULL Gre keepalives configured: Off, Gre keepalives adjacency state: down Input packets : 4332961 Output packets: 7699862 Protocol inet, MTU: 9062 Flags: None Protocol mpls, MTU: 9050, Maximum labels: 3 Flags: None
Useful Information from Output:
– 172.20.0.23: NEXT-HOP Compute
– 31: Label of NEXT-HOP on Compute
What we find is the GRE interface (Label 31) for the NEXT-HOP points to the COMPUTE at 172.20.0.23. You can find which COMPUTE this is via the Contrail WebGUI (Monitor => Virtual Routers => [Search Field]172.20.0.23 or search for the VHOST0 IP’s of your computes. In this case, the COMPUTE “Hostname” matches with the NEXT-HOP IP 172.20.0.23 matches with COMPUTE cmpt001. cmpt001 hosts the VM of the vFW Service Instance that is the next element in the path to the DST VM. From here, go to Step 2.
1b. Multiple VM / Compute (Composite Next-Hop)
|
Here, we will focus on Multiple VM’s or Computes (Composite Next-Hop).
|
On SDNGW:
SDNGW> show route 10.10.0.100 vntest.inet.0: 70 destinations, 150 routes (70 active, 0 holddown, 0 hidden) @ = Routing Use Only, # = Forwarding Use Only + = Active Route, - = Last Active, * = Both 10.10.0.0/24 @[BGP/170] 1w1d 21:11:22, MED 100, localpref 200, from 172.20.0.4 AS path: ?, validation-state: unverified > via gr-0/1/0.74659, Push 62 [BGP/170] 1w1d 21:14:54, MED 100, localpref 200, from 172.20.0.5 AS path: ?, validation-state: unverified > via gr-0/1/0.19283, Push 31 [BGP/170] 1w1d 21:11:22, MED 100, localpref 200, from 172.20.0.6 AS path: ?, validation-state: unverified > via gr-0/1/0.74659, Push 62 [BGP/170] 1w1d 21:14:54, MED 100, localpref 200, from 172.20.0.6 AS path: ?, validation-state: unverified > via gr-0/1/0.19283, Push 31 #[Multipath/255] 1w1d 21:13:23, metric 100, metric2 0 via gr-0/1/0.74659, Push 62 > via gr-0/1/0.19283, Push 31 SDNGW> show interfaces gr-0/1/0.19283 Logical interface gr-0/1/0.19283 (Index 345) (SNMP ifIndex 526) Flags: Up Point-To-Point SNMP-Traps 0x4000 IP-Header 172.20.0.23:10.10.10.10:47:df:64:0000000800000000 Encapsulation: GRE-NULL Gre keepalives configured: Off, Gre keepalives adjacency state: down Input packets : 4332961 Output packets: 7699862 Protocol inet, MTU: 9062 Flags: None Protocol mpls, MTU: 9050, Maximum labels: 3 Flags: None
If your environment utilizes logging correlation software like the Splunk or Juniper JSA/STRM, that tool would be useful to figure out which “next-hop” scaled-out service instance vFW. Otherwise, we will have to log into each vFW and look at the flow or session table entries to figure out which compute is next in path.
Based on the above information, we can see the traffic transit a particular vFW in the scaled-out Service Instance and we can go directly to Compute where that vFW VM resides. If you know which vFW is taking the traffic (we will assume VFW001 in this case), you can look up with COMPUTE that VM is instantiated on and go from there.
Figure out which COMPUTE the ServiceInstance vFW is on:
toolsvm:~$ nova list | grep -i vfw | a1b2c3d4e5f6g-7h8i-9j0k-1l2m3n4o5p6q | VFW001 | ACTIVE | - | Running | ADMIN=192.168.0.194; VN_A=11.11.0.64; LOGNET=192.168.11.65; VN_B=10.10.0.64 | | z1y2x3w4v5u6t-7s8r-9q0p-1o2n3m4l5k6j | VFW002 | ACTIVE | - | Running | ADMIN=192.168.0.196; VN_A=11.11.0.66; LOGNET=192.168.11.67; VN_B=10.10.0.65 | toolsvm:~$ nova show a1b2c3d4e5f6g-7h8i-9j0k-1l2m3n4o5p6q +------------------------------------------+------------------------------------------------------------+ | Property | Value | +------------------------------------------+------------------------------------------------------------+ | VN_A network | 11.11.0.64 | | LOGNET network | 192.168.11.65 | | ADMIN network | 192.168.0.194 | | VN_B network | 10.10.0.64 | | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | default | | OS-EXT-SRV-ATTR:host | cmpt001 | | OS-EXT-SRV-ATTR:hypervisor_hostname | cmpt001.test.net | | OS-EXT-SRV-ATTR:instance_name | instance-00000eef | | OS-EXT-STS:power_state | 1 | | OS-EXT-STS:task_state | - | | OS-EXT-STS:vm_state | active | | OS-SRV-USG:launched_at | 2016-06-01T00:35:36.000000 | | OS-SRV-USG:terminated_at | - | | accessIPv4 | | | accessIPv6 | | | config_drive | | | created | 2016-06-01T00:35:36Z | | flavor | flv_vsrx (12345678-012a-3bcd-e4f5-6789g01h2345) | | hostId | asdflkijhbalk1h23k132427896r98a7asfhio1u241234iuhewnoafs | | id | a1b2c3d4e5f6g-7h8i-9j0k-1l2m3n4o5p6q | | image | vsrx15.1x49d30.3 (6baishdu-ewiu-9238-sibh-02394isfoayx) | | key_name | - | | name | VFW001 | | os-extended-volumes:volumes_attached | [] | | progress | 0 | | security_groups | default | | status | ACTIVE | | tenant_id | 9asdf2983riahdf9987sdf9ya9sya9sy | | updated | 2016-06-01T00:35:36Z | | user_id | d8923923hisuadfhf893923h2hfdasfh | +------------------------------------------+------------------------------------------------------------+ toolsvm:~$ for x in $(nova list | grep -i foam | awk {'print $4'}); do nova show $x | sed -n '10p;24p;28p' | awk {'print $4'} | xargs | sed 's/ / || /g'; done cmpt001 || a1b2c3d4e5f6g-7h8i-9j0k-1l2m3n4o5p6q || VFW001 cmpt003 || z1y2x3w4v5u6t-7s8r-9q0p-1o2n3m4l5k6j || VFW002
We see that VFW001 with ID a1b2c3d4e5f6g-7h8i-9j0k-1l2m3n4o5p6q is on COMPUTE cmpt001.
STEP 2: Verify Traffic is making it into vFW ServiceInstance VM Compute
2a. Go to COMPUTE
toolsvm ~]$ ssh user@cmpt001 password for [user]: Welcome to Ubuntu 14.04.2 LTS (GNU/Linux 3.13.0-61-generic x86_64)
2b. Find COMPUTE interfaces
cmpt001:~# cat /etc/contrail/contrail-vrouter-agent.conf | grep -A13 -i virtual-host-interface
[VIRTUAL-HOST-INTERFACE]
# Everything in this section is mandatory
# name of virtual host interface
name=vhost0
# IP address and prefix in ip/prefix_len format
ip=172.20.0.23/32
# Gateway IP address for virtual host
gateway=172.20.0.1
# Physical interface name to which virtual host interface maps to
physical_interface=p1p1
cmpt001:~# ifconfig vhost0
vhost0 Link encap:Ethernet HWaddr b0:ob:ab:ba:0a:a0
inet addr:172.20.0.23 Bcast:172.20.0.31 Mask:255.255.255.240
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:84487487 errors:0 dropped:182627 overruns:0 frame:0
TX packets:82063519 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:253984497954 (253.9 GB) TX bytes:67502412941 (67.5 GB)
cmpt001:~# ifconfig p1p1
p1p1 Link encap:Ethernet HWaddr b0:ob:ab:ba:0a:a0
UP BROADCAST RUNNING MULTICAST MTU:9000 Metric:1
RX packets:194126327 errors:0 dropped:0 overruns:0 frame:0
TX packets:125130748 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:286638778868 (286.6 GB) TX bytes:94956347917 (94.9 GB)
Interrupt:40 Memory:f3000000-f37fffff
2c. Verify vRouter is exchanging XMPP information with the CONTRAIL CONTROLLER(s)
cmpt001:~# tcpdump -D | grep -i vhost0 1.vhost0 cmpt001:~# tcpdump -nei 1 port xmpp-server tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vhost0, link-type EN10MB (Ethernet), capture size 65535 bytes 12:04:23.420090 c9:36:dd:24:po:p0 > b0:ob:ab:ba:0a:a0, ethertype IPv4 (0x0800), length 66: 172.20.0.5.5269 > 172.20.0.23.58699: Flags [.], ack 3731677826, win 8748, options [nop,nop,TS val 1268013680 ecr 1267113680], length 0 12:04:23.420120 b0:ob:ab:ba:0a:a0 > c9:36:dd:24:po:p0, ethertype IPv4 (0x0800), length 66: 172.20.0.23.58699 > 172.20.0.5.5269: Flags [.], ack 1, win 6792, options [nop,nop,TS val 1267114931 ecr 1268012429], length 0 ^C 2 packets captured 4 packets received by filter 0 packets dropped by kernel
2d. Verify ICMP packets from example (12.12.0.102 => 10.10.0.100) is making it into the COMPUTE cmpt001
cmpt001:~# tcpdump -D | grep p1p1 7.p1p1 cmpt001:~# tcpdump -nei 7 proto 47 | grep 11.11.0.60 tcpdump: WARNING: bond0.2004: no IPv4 address assigned tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on bond0.2004, link-type EN10MB (Ethernet), capture size 65535 bytes 12:07:02.093007 00:1d:c2:8f:22:db > 0a:36:cc:61:aa:l0, ethertype IPv4 (0x0800), length 126: 172.20.0.25 > 172.20.0.23: GREv0, proto MPLS unicast (0x8847), length 92: MPLS (label 31, exp 0, [S], ttl 255) 12.12.0.102 > 10.10.0.100: ICMP echo request, id 15119, seq 20030, length 64 12:07:03.094702 00:1d:c2:8f:22:db > 0a:36:cc:61:aa:l0, ethertype IPv4 (0x0800), length 126: 172.20.0.25 > 172.20.0.23: GREv0, proto MPLS unicast (0x8847), length 92: MPLS (label 31, exp 0, [S], ttl 255) 12.12.0.102 > 10.10.0.100: ICMP echo request, id 15119, seq 20031, length 64 ^C 2 packets captured 4 packets received by filter 0 packets dropped by kernel
|
We see that ICMP is making it into the Compute in the above TCPDUMP output. (TCPDUMP can process PROTO 47 for MPLSoGRE but if the tunnels are MPLSoUDP, you’ll have to decode this in another packet analyzer like WireShark).
|
STEP 3: Trace the flow into vFW Service Instance VM
3a. Find NEXT-HOP (mpls –dump will output the entire Label => NextHop table) – we found the MPLS Label 31 from the Next-Hop from the SDNGW
cmpt001:~# mpls --get 31 MPLS Input Label Map Label NextHop ------------------- 31 49
3b. Find NEXT-HOP Interface and VRF that the FW VM interface that is created on for this flow on VN_A.
cmpt001:~# nh --get 49 Id:49 Type:Encap Fmly: AF_INET Rid:0 Ref_cnt:8 Vrf:11 Flags:Valid, Policy, EncapFmly:0806 Oif:24 Len:14 Encap Data: 02 95 3d 1d c9 69 00 00 5e 00 01 00 08 00 cmpt001:~# vif --get 24 Vrouter Interface Table Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2 D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload vif0/24 OS: tap93d2dja8-22 Type:Virtual HWaddr:bk:2a:n6:00:02:00 IPaddr:0 Vrf:11 Flags:PL3L2D MTU:9160 Ref:6 RX packets:728634 bytes:71113479 errors:0 TX packets:756736 bytes:89113687 errors:0
3c. [IGNORE FOR NOW] On Vrf:11 (VN_A), we see the routing-table point to the FW’s “RIGHT” interface as the NEXT-HOP to 10.10.0.0/24 Network (VN_B)
cmpt001:~# rt --dump 11 | grep 10.10.0
Vrouter inet4 routing table 0/13/unicast
Flags: L=Label Valid, P=Proxy ARP, T=Trap ARP, F=Flood ARP
Destination PPL Flags Label Nexthop Stitched MAC(Index)
10.10.0.0/24 24 P - 164 -
cmpt001:~# nh --get 164
Id:164 Type:Composite Fmly: AF_INET Rid:0 Ref_cnt:2 Vrf:11
Flags:Valid, Policy, Ecmp,
Sub NH(label): 95(31) -1 14(62)
root@cmpt001:~# nh --get 95
Id:95 Type:Encap Fmly: AF_INET Rid:0 Ref_cnt:25 Vrf:11
Flags:Valid,
EncapFmly:0806 Oif:24 Len:14
Encap Data: 02 95 3d 1d c9 69 00 00 5e 00 01 00 08 00
cmpt001:~# vif --get 24
Vrouter Interface Table
Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload
vif0/24 OS: tap93d2dja8-22
Type:Virtual HWaddr:bk:2a:n6:00:02:00 IPaddr:0
Vrf:11 Flags:PL3L2D MTU:9160 Ref:6
RX packets:733959 bytes:71635018 errors:0
TX packets:762144 bytes:89646655 errors:0
cmpt001:~# cat /var/lib/nova/instances/a1b2c3d4e5f6g-7h8i-9j0k-1l2m3n4o5p6q/libvirt.xml | grep -i tap
<target dev="tapc218da9d-32"/>
<target dev="tap13a2d42c-5d"/>
<target dev="tap93d2dja8-22"/>
<target dev="tapa8d2gf3a-k4"/>
(Interfaces are enumerated in the order MGMT, LEFT, RIGHT, OTHER… so the TAP interface above it the “RIGHT” interface in VN_A)
3d. Check for “Discards” on Vrf:11
cmpt001:~# watch vrfstats --get 11
Every 2.0s: vrfstats --get 11 Fri Jun 03 19:13:13 2016
Vrf: 11
Discards 0, Resolves 0, Receives 0, L2 Receives 7783, Vrf Translates 0, Unknown Unicast Floods 0
Ecmp Composites 0, L2 Mcast Composites 26818, Fabric Composites 24593, Encap Composites 24593, Evpn Composites 0
Udp Tunnels 0, Udp Mpls Tunnels 4000360, Gre Mpls Tunnels 7780, Vxlan Tunnels 0
L2 Encaps 3321946, Encaps 32480
GROs 2662900, Diags 0
Arp Virtual Proxys 654, Arp Virtual Stitchs 1539, Arp Virtual Floods 600, Arp Physical Stitchs 0, Arp Tor Proxys 0, Arp Physical Flo
ods 0
Based on the above output, it does not appear that there are any discards on Vrf:11 that would be causing the connectivity failure (Counter at “0” and not incrementing).
3e. Look for “Discards” via dropstats and check relevant counters (“Flow Action Drop” & “Discards” in this case) to see if they increment
cmpt001:~# watch dropstats Every 2.0s: dropstats Fri Jun 03 19:16:59 2016 GARP 0 ARP no where to go 0 Invalid ARPs 0 Invalid IF 0 Trap No IF 0 IF TX Discard 0 IF Drop 0 IF RX Discard 0 Flow Unusable 0 Flow No Memory 0 Flow Table Full 0 Flow NAT no rflow 0 Flow Action Drop 12040 Flow Action Invalid 0 Flow Invalid Protocol 0 Flow Queue Limit Exceeded 132 Discards 8372 TTL Exceeded 0 Mcast Clone Fail 0 Cloned Original 348762341 Invalid NH 20637113 Invalid Label 0 Invalid Protocol 0 Rewrite Fail 0 Invalid Mcast Source 0 Push Fails 0 Pull Fails 0 Duplicated 0 Head Alloc Fails 0 Head Space Reserve Fails 0 PCOW fails 0 Invalid Packets 0 Misc 764 Nowhere to go 0 Checksum errors 0 No Fmd 0 Invalid VNID 0 Fragment errors 0 Invalid Source 12 Jumbo Mcast Pkt with DF Bit 0 ARP No Route 0 ARP Reply No Route 0 No L2 Route 136423
“Discards” and “Flow Action Drop” counters stayed steady at “12040” and “8372” respectively and did not increment.
3f. Find which VM is associated with TAP Interface tap93d2dja8-22 (there is more than one way to get this information)
CLI Method:
– From previous discovery, we know that the ID of the VM VFW001 on cmpt001 is a1b2c3d4e5f6g-7h8i-9j0k-1l2m3n4o5p6q. Map the TAP Interface to the VM:
cmpt001:~# cat /var/lib/nova/instances/a1b2c3d4e5f6g-7h8i-9j0k-1l2m3n4o5p6q/libvirt.xml | grep -i tap <target dev="tapc218da9d-32"/> <target dev="tap13a2d42c-5d"/> <target dev="tap93d2dja8-22"/> <target dev="tapa8d2gf3a-k4"/>
Contrail GUI Method: Find the Compute where the VM resides (cmpt001) via Monitor => Virtual Routers = cmpt001 // Interfaces. From there, do a search for the interface tap93d2dja8-22. From there, you should be able to find which VM the TAP Interface is mapped to.
3g. From the above information, we know that this is mapped to the vFW”RIGHT” interface which is on the VN_A Network. We can do a TCPDUMP to verify if the traffic is making it to the FW’s Ingress/VN_A interface:
cmpt001:~# tcpdump -nei tap93d2dja8-22 tcpdump: WARNING: tap93d2dja8-22: no IPv4 address assigned tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on tap93d2dja8-22, link-type EN10MB (Ethernet), capture size 65535 bytes 12:09:41.995714 bk:2a:n6:00:02:00 > 02:95:3d:1d:c9:69, ethertype IPv4 (0x0800), length 74: 12.12.0.102 > 10.10.0.100: ICMP echo request, id 61, seq 10575, length 40 12:09:42.995769 bk:2a:n6:00:02:00 > 02:95:3d:1d:c9:69, ethertype IPv4 (0x0800), length 74: 12.12.0.102 > 10.10.0.100: ICMP echo request, id 61, seq 10593, length 40 ^C 2 packets captured 4 packets received by filter 0 packets dropped by kernel
3h. From the above, we see ICMP echo requests going into the FW, but no replies. Let’s check for what the vFW sees:
toolsvm:~$ ssh 192.168.0.194 Password: --- JUNOS 15.1X49-D30.3 built 2015-12-17 04:39:24 UTC VFW001> VFW001> show security flow session destination-prefix 10.10.0.100 Session ID: 206346, Policy name: allow_restricted/6, Timeout: 2, Valid In: 12.12.0.102/34054 --> 10.10.0.100/61;icmp, If: ge-0/0/1.0, Pkts: 1, Bytes: 60, Out: 10.10.0.100/61 --> 12.12.0.102/34054;icmp, If: ge-0/0/0.0, Pkts: 1, Bytes: 0, Session ID: 206347, Policy name: allow_restricted/6, Timeout: 4, Valid In: 12.12.0.102/34058 --> 10.10.0.100/61;icmp, If: ge-0/0/1.0, Pkts: 1, Bytes: 60, Out: 10.10.0.100/61 --> 12.12.0.102/34058;icmp, If: ge-0/0/0.0, Pkts: 1, Bytes: 0, Session ID: 206350, Policy name: allow_restricted/6, Timeout: 2, Valid In: 12.12.0.102/34057 --> 10.10.0.100/61;icmp, If: ge-0/0/1.0, Pkts: 1, Bytes: 60, Out: 10.10.0.100/61 --> 12.12.0.102/34057;icmp, If: ge-0/0/0.0, Pkts: 1, Bytes: 0, Total sessions: 3
The vFW sees packets inbound through it to the VM 10.10.0.100 but no return traffic (which matches what we see on the TCPDUMP on the FW’s GE-0/0/1 or “RIGHT” interface). At this point, we need to look at the traffic leaving the FW on the “LEFT”/VN_B interface.
3i. Check the COMPUTE vRouter Flow Table to see if the traffic is being forwarded.
cmpt001:~# flow --match 10.10.0.100 Flow table(size 68157440, entries 542386) Entries: Created 898146 Added 898144 Processed 898146 Used Overflow entries 0 (Created Flows/CPU: 368496 140472 62462 39956 28374 22527 25634 19742 20169 15999 6767 6693 6514 6322 7041 6249 6200 6272 6324 6235 7985 9035 7329 9146 6741 6608 5981 7374 10025 8246 663 941 721 734 700 671 771 837 1120 4070)(oflows 0) Action:F=Forward, D=Drop N=NAT(S=SNAT, D=DNAT, Ps=SPAT, Pd=DPAT, L=Link Local Port) Other:K(nh)=Key_Nexthop, S(nh)=RPF_Nexthop Flags:E=Evicted, Ec=Evict Candidate, N=New Flow, M=Modified TCP(r=reverse):S=SYN, F=FIN, R=RST, C=HalfClose, E=Established, D=Dead Listing flows matching ([10.10.0.100]:*) Index Source:Port/Destination:Port Proto(V) ----------------------------------------------------------------------------------- 168624<=>23140 12.12.0.102:1393 1 (2->3) 10.10.0.100:0 (Gen: 5, K(nh):33, Action:F, Flags:, S(nh):94, Stats:257760/25260480, SPort 53824) -- 430164<=>119628 12.12.0.102:1393 1 (13->5) 10.10.0.100:0 (Gen: 11, K(nh):49, Action:F, Flags:, S(nh):88, Stats:257760/21651840, SPort 60610) --
STEP 4: Verify traffic is leaving vFW Service Instance VM towards 10.10.0.100
4a. Verify next TAP Interface for FW (“LEFT” / VN_B) sees traffic exit the vFW destined for the VM at 10.10.0.100
cmpt001:~# nh --get 33
Id:93 Type:Encap Fmly: AF_INET Rid:0 Ref_cnt:5 Vrf:12
Flags:Valid, Policy,
EncapFmly:0806 Oif:12 Len:14
Encap Data: 02 31 e6 d2 4c 7d 00 00 5e 00 01 00 08 00
cmpt001:~# vif --get 12
Vrouter Interface Table
Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror
Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2
D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged
Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored
Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload
vif0/12 OS: tap13a2d42c-5d
Type:Virtual HWaddr:bk:2a:n6:00:02:00 IPaddr:0
Vrf:12 Flags:PL3L2D MTU:9160 Ref:6
RX packets:1236404 bytes:135113415 errors:0
TX packets:1231952 bytes:120251878 errors:0
cmpt001:~# cat /var/lib/nova/instances/a1b2c3d4e5f6g-7h8i-9j0k-1l2m3n4o5p6q/libvirt.xml | grep -i tap
<target dev="tapc218da9d-32"/>
<target dev="tap13a2d42c-5d"/>
<target dev="tap93d2dja8-22"/>
<target dev="tapa8d2gf3a-k4"/>
cmpt001:~# tcpdump -nei tap13a2d42c-5d
tcpdump: WARNING: tap13a2d42c-5d: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tap13a2d42c-5d, link-type EN10MB (Ethernet), capture size 65535 bytes
12:24:31.175890 02:dd:26:2b:24:p2 > 7b:23:a9:tu:c3:p0, ethertype IPv4 (0x0800), length 74: 12.12.0.102 > 10.10.0.100: ICMP echo request, id 61, seq 11410, length 40
12:24:32.176227 02:dd:26:2b:24:p2 > 7b:23:a9:tu:c3:p0, ethertype IPv4 (0x0800), length 74: 12.12.0.102 > 10.10.0.100: ICMP echo request, id 61, seq 11413, length 40
^C
2 packets captured
4 packets received by filter
0 packets dropped by kernel
4b. Traffic (ICMP echo requests) is leaving vFW egress interface (GE-0/0/0) “LEFT” or VN_B interface destined towards the VM 10.10.0.100. Check to see if there are any discards on the VRF.
cmpt001:~# watch vrfstats --get 12
Every 2.0s: vrfstats --get 12 Fri Jun 03 19:13:13 2016
Vrf: 12
Discards 0, Resolves 0, Receives 0, L2 Receives 12369, Vrf Translates 0, Unknown Unicast Floods 0
Ecmp Composites 0, L2 Mcast Composites 18742, Fabric Composites 12190, Encap Composites 12168, Evpn Composites 0
Udp Tunnels 0, Udp Mpls Tunnels 17, Gre Mpls Tunnels 4010, Vxlan Tunnels 0
L2 Encaps 71982, Encaps 1302371
GROs 2687, Diags 0
Arp Virtual Proxys 8658, Arp Virtual Stitchs 4858, Arp Virtual Floods 3748, Arp Physical Stitchs 0, Arp Tor Proxys 0, Arp Physical Floods 30
No discards.
4c. Verify routing on Vrf:10 (VN_B) FIB table for 10.10.0.100
cmpt001:~# rt --dump 12 | grep 10.10.0.100
Vrouter inet4 routing table 0/2/unicast
Flags: L=Label Valid, P=Proxy ARP, T=Trap ARP, F=Flood ARP
Destination PPL Flags Label Nexthop Stitched MAC(Index)
10.10.0.100/32 32 LP 31 88 2:38:c8:ea:9a:21(211664)
4d. Find the NEXT-HOP COMPUTE to get to the VM on VN_B at 10.10.0.100
cmpt001:~# nh --get 88 Id:88 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:144 Vrf:0 Flags:Valid, MPLSoUDP, Oif:0 Len:14 Flags Valid, MPLSoUDP, Data:8c dc d4 10 74 c0 8c dc d4 10 75 a0 08 00 Vrf:0 Sip:172.20.0.23 Dip:172.20.0.24
Next-Hop 88 points to a destination IP of 172.20.0.24 (vhost0 Interface IP of the destination Compute on Default Vrf0) with MPLS Label 31
STEP 5: Verify traffic is getting to Destination VM at 10.10.0.100 on TAP Interface
5a. Find NEXT-HOP COMPUTE at 172.29.2.82
CLI Method (From toolsvm – use the IP of the Destination VM to find Compute):
toolsvm:~$ for x in $(nova list | grep -i 10.10.0.100 | awk {'print $4'}); do nova show $x | sed -n '7p;21p;25p' | awk {'print $4'}; done
cmpt002
98asdfhjkn213-sdai-ncxv-3421987kjlsm
testvm-VN_B_1
toolsvm:~$ nova show 98asdfhjkn213-sdai-ncxv-3421987kjlsm
+------------------------------------------+-------------------------------------------------------------+
| Property | Value |
+------------------------------------------+-------------------------------------------------------------+
| MNS_shared_OAM_PROTECTED_NET_1_1 network | 10.10.0.100 |
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | cmpt002 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | cmpt002.test.net |
| OS-EXT-SRV-ATTR:instance_name | instance-00000f28 |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2016-06-01T00:35:36.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2016-06-01T00:35:36Z |
| flavor | m1.tiny (2) |
| hostId | a0s9dfuwqehrlkqwerl89ydfkj1394874329y2nlkadfasygas98ydfs |
| id | 98asdfhjkn213-sdai-ncxv-3421987kjlsm |
| image | ubuntu-14 (1f062a45-4a90-437d-a2b8-b2b5a565d95a) |
| key_name | - |
| metadata | {} |
| name | testvm-VN_B_1 |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| status | ACTIVE |
| tenant_id | 9asdf2983riahdf9987sdf9ya9sya9sy |
| updated | 2016-06-01T00:35:36Z |
| user_id | d8923923hisuadfhf893923h2hfdasfh |
+------------------------------------------+-------------------------------------------------------------+
GUI Method (From Contrail GUI)
Monitor => Infrastructure => Virtual Routers => Search
5b. Go to COMPUTE cmpt002. Based on Label 31 for NEXT-HOP on this COMPUTE, we can find where the traffic is destined to.
cmpt002:~# mpls --get 31 MPLS Input Label Map Label NextHop ------------------- 31 73 root@cmpt002:~# nh --get 73 Id:73 Type:Encap Fmly: AF_INET Rid:0 Ref_cnt:6 Vrf:13 Flags:Valid, Policy, EncapFmly:0806 Oif:14 Len:14 Encap Data: 02 38 c8 ea 9a 21 00 00 5e 00 01 00 08 00 cmpt002:~# vif --get 14 Vrouter Interface Table Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2 D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload vif0/14 OS: tap83f2ki4w-72 Type:Virtual HWaddr:bk:2a:n6:00:02:00 IPaddr:0 Vrf:13 Flags:PL3L2D MTU:9160 Ref:6 RX packets:2272968 bytes:214007093 errors:0 TX packets:2294863 bytes:287437540 errors:0
5c. Verify the above output matches the VM libvert.xml
cmpt002:~# cat /var/lib/nova/instances/98asdfhjkn213-sdai-ncxv-3421987kjlsm/libvirt.xml | grep -i tap83f2ki4w-72 <target dev="tap83f2ki4w-72"/>
5d. Check for Vrf:13 discards
cmpt002:~# watch vrfstats --get 13
Every 2.0s: vrfstats --get 13 Fri Jun 03 19:33:33 2016
Vrf: 13
Discards 0, Resolves 0, Receives 0, L2 Receives 3299, Vrf Translates 0, Unknown Unicast Floods 0
Ecmp Composites 0, L2 Mcast Composites 3364, Fabric Composites 330, Encap Composites 739, Evpn Composites 0
Udp Tunnels 0, Udp Mpls Tunnels 4, Gre Mpls Tunnels 0, Vxlan Tunnels 0
L2 Encaps 3338, Encaps 2421364
GROs 40513, Diags 0
Arp Virtual Proxys 2906, Arp Virtual Stitchs 0, Arp Virtual Floods 0, Arp Physical Stitchs 0, Arp Tor Proxys 0, Arp Physical Floods 0
5e. Check vRouter Flow Table for “Action:F”
cmpt002:~# flow --match 10.10.0.100
Flow table(size 68157440, entries 532480)
Entries: Created 2101736 Added 2100485 Processed 2101736 Used Overflow entries 0
(Created Flows/CPU: 367906 175642 111664 82241 69167 55275 51234 46499 43885 43268 16593 19378 19266 18389 18108 17846 17739 18159 17939 18510 31225 119065 99542 54060 37515 32004 30342 31058 41341 28021 7123 86430 156010 50283 20184 11714 9180 8086 7844 12001)(oflows 0)
Action:F=Forward, D=Drop N=NAT(S=SNAT, D=DNAT, Ps=SPAT, Pd=DPAT, L=Link Local Port)
Other:K(nh)=Key_Nexthop, S(nh)=RPF_Nexthop
Flags:E=Evicted, Ec=Evict Candidate, N=New Flow, M=Modified
TCP(r=reverse):S=SYN, F=FIN, R=RST, C=HalfClose, E=Established, D=Dead
Listing flows matching ([10.10.0.100]:*)
Index Source:Port/Destination:Port Proto(V)
-----------------------------------------------------------------------------------
14325<=>438965 12.12.0.102:61 1 (6->5)
10.10.0.100:0
(Gen: 10, K(nh):73, Action:F, Flags:, S(nh):22, Stats:1166/69960, SPort 61800)
--
5f. Verify Traffic is being forwarded on TAP Interface on the VM
cmpt002:~# tcpdump -nei tap83f2ki4w-72 tcpdump: WARNING: tap83f2ki4w-72: no IPv4 address assigned tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on tap83f2ki4w-72, link-type EN10MB (Ethernet), capture size 65535 bytes 13:02:32.939608 bk:2a:n6:00:02:00 > 7b:23:a9:tu:c3:p0, ethertype IPv4 (0x0800), length 74: 12.12.0.102 > 10.10.0.100: ICMP echo request, id 64, seq 59313, length 40 13:02:33.935583 bk:2a:n6:00:02:00 > 7b:23:a9:tu:c3:p0, ethertype IPv4 (0x0800), length 74: 12.12.0.102 > 10.10.0.100: ICMP echo request, id 64, seq 59317, length 40 13:02:34.952637 bk:2a:n6:00:02:00 > 7b:23:a9:tu:c3:p0, ethertype IPv4 (0x0800), length 74: 12.12.0.102 > 10.10.0.100: ICMP echo request, id 64, seq 59321, length 40 13:02:35.960168 bk:2a:n6:00:02:00 > 7b:23:a9:tu:c3:p0, ethertype IPv4 (0x0800), length 74: 12.12.0.102 > 10.10.0.100: ICMP echo request, id 64, seq 59322, length 40 ^C 4 packets captured 6 packets received by filter 0 packets dropped by kernel
The traffic flow packets are making it all the to the TAP Interface on the vRouter of the Compute that is assigned to the VM 10.10.0.100. We need to get the DST VM owner to verify that they can see the traffic and at this point it could be any number of problems:
– VM Host Routing Table
– VM IPTables or similar service filtering inbound/outbound traffic
– OpenStack Security Group applied to VM
– Controller/Compute Routing Issue
– Etc.,
STEP 6: On VM, run TCPDUMP to inspect inbound traffic
6a. Access VM (this example will utilize Link-Local Address from Compute)
Find Link-Local Address of TAP interface of VM via Contrail Web GUI
On Compute, SSH to Link-Local Address 169.254.0.14
cmpt002:/etc# ssh 169.254.0.14 169.254.0.14's password: Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 3.13.0-32-generic x86_64) testvm-vn_b:~$
6b. Verify Traffic is making it to the VM
testvm-vn_b:~$ ifconfig eth0 Link encap:Ethernet HWaddr 7b:23:a9:tu:c3:p0 inet addr:10.10.0.100 Bcast:172.20.50.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:2298122 errors:0 dropped:0 overruns:0 frame:0 TX packets:2276178 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:287697225 (287.6 MB) TX bytes:214259840 (214.2 MB) eth1 Link encap:Ethernet HWaddr 02:8a:db:64:fb:45 inet addr:192.168.0.58 Bcast:192.168.0.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:2298122 errors:0 dropped:0 overruns:0 frame:0 TX packets:2276178 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:287697225 (287.6 MB) TX bytes:214259840 (214.2 MB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:34 errors:0 dropped:0 overruns:0 frame:0 TX packets:34 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:7130 (7.1 KB) TX bytes:7130 (7.1 KB) testvm-vn_b:~# tcpdump -D 1.eth0 2.eth1 3.any (Pseudo-device that captures on all interfaces) 4.lo testvm-vn_b:~$ tcpdump -nei 1 proto 1 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes 13:16:15.056859 bk:2a:n6:00:02:00 > 7b:23:a9:tu:c3:p0, ethertype IPv4 (0x0800), length 74: 12.12.0.102 > 10.10.0.100: ICMP echo request, id 65, seq 109, length 40 13:16:15.056963 7b:23:a9:tu:c3:p0 > bk:2a:n6:00:02:00, ethertype IPv4 (0x0800), length 74: 10.10.0.100 > 12.12.0.102: ICMP echo reply, id 65, seq 109, length 40 ^C 2 packets captured 4 packets received by filter 0 packets dropped by kernel testvm-vn_b:~$
|
At this point, we can safely say that the traffic flow packets are making it all the to the TAP Interface on the vRouter of the Compute that is assigned to the VM 10.10.0.100. There are any number of reasons why traffic may fail to/from a VM:
|
– VM Host Routing Table
– VM IPTables or similar service filtering inbound/outbound traffic
– OpenStack Security Group applied to VM
– Controller/Compute Routing Issue
– Etc.,
Tracing: Inside => Out
STEP 1: On Compute where VM is being hosted on, Trace flow back out to Source Host
>>*Assume traffic is fixed on the DST VM and now we are tracing the return flow back to the SRC IP
1a. Get vRouter Flow Table information for return traffic from 10.10.0.100 to 12.12.0.102
cmpt002:~# flow --match 12.12.0.102 Flow table(size 68157440, entries 532480) Entries: Created 2105344 Added 2104093 Processed 2105344 Used Overflow entries 0 (Created Flows/CPU: 368319 176011 111801 82325 69226 55341 51282 46532 43914 43306 16597 19384 19280 18394 18114 17851 17745 18169 17948 18521 31269 119556 99662 54124 37564 32051 30401 31112 41397 28061 7134 86678 156797 50372 20220 11723 9190 8099 7854 12020)(oflows 0) Action:F=Forward, D=Drop N=NAT(S=SNAT, D=DNAT, Ps=SPAT, Pd=DPAT, L=Link Local Port) Other:K(nh)=Key_Nexthop, S(nh)=RPF_Nexthop Flags:E=Evicted, Ec=Evict Candidate, N=New Flow, M=Modified TCP(r=reverse):S=SYN, F=FIN, R=RST, C=HalfClose, E=Established, D=Dead Listing flows matching ([12.12.0.102]:*) Index Source:Port/Destination:Port Proto(V) ----------------------------------------------------------------------------------- 463620<=>89580 10.10.0.100:65 1 (6->5) 12.12.0.102:0 (Gen: 14, K(nh):73, Action:F, Flags:, E:0, S(nh):73, Stats:3/222, SPort 51745) --
|
E:0 == ECMP Index 0 (for ECMP Multi-Path, Next-Hop starts and increments from 0)
|
1b. Find Return Next-Hop Information
cmpt002:~# nh --get 73 Id:73 Type:Encap Fmly: AF_INET Rid:0 Ref_cnt:6 Vrf:13 Flags:Valid, Policy, EncapFmly:0806 Oif:14 Len:14 Encap Data: 02 38 c8 ea 9a 21 00 00 5e 00 01 00 08 00 cmpt002:/etc# rt --dump 13 | grep 172.20.212 Vrouter inet4 routing table 0/6/unicast Flags: L=Label Valid, P=Proxy ARP, T=Trap ARP, F=Flood ARP Destination PPL Flags Label Nexthop Stitched MAC(Index) 172.20.212.0/24 0 P - 102 - cmpt002:~# nh --get 102 Id:102 Type:Composite Fmly: AF_INET Rid:0 Ref_cnt:6601 Vrf:13 Flags:Valid, Policy, Ecmp, Sub NH(label): 22(27) 22(31)
|
Here we see 2 Next-Hops and since we saw E:0 in the vRouter Flow Table, it means we will take the first Next-Hop(Label) of 22(27). Remember this MPLS Label 22 as we will need this to get the VHOST0 IP of the Compute where the vFW resides on.
|
cmpt002:/etc# nh --get 22 Id:22 Type:Tunnel Fmly: AF_INET Rid:0 Ref_cnt:100 Vrf:0 Flags:Valid, MPLSoUDP, Oif:0 Len:14 Flags Valid, MPLSoUDP, Data:8c dc d4 10 75 a0 8c dc d4 10 74 c0 08 00 Vrf:0 Sip:172.20.0.24 Dip:172.20.0.23
|
172.20.0.23 (cmpt001) is the Compute where the vFW VM resides.
|
STEP 2: On Compute where Next-Hop vFW VM is being hosted on, verify return flow is hitting same egress vFW as ingress
2a. Trace path of return flow back (MPLS Label 27) to vFWand verify it is the same vFW that was utilized for the inbound traffic.
The COMPUTE with Vhost0 IP of 172.20.0.23 == cmpt001 and VFW001 has a ID of a1b2c3d4e5f6g-7h8i-9j0k-1l2m3n4o5p6q from previous discovery.
cmpt001:~# mpls --get 27 MPLS Input Label Map Label NextHop ------------------- 27 65 root@cmpt001:~# nh --get 65 Id:65 Type:Encap Fmly: AF_INET Rid:0 Ref_cnt:5 Vrf:12 Flags:Valid, Policy, EncapFmly:0806 Oif:29 Len:14 Encap Data: 02 31 e6 d2 4c 7d 00 00 5e 00 01 00 08 00 cmpt001:~# vif --get 29 Vrouter Interface Table Flags: P=Policy, X=Cross Connect, S=Service Chain, Mr=Receive Mirror Mt=Transmit Mirror, Tc=Transmit Checksum Offload, L3=Layer 3, L2=Layer 2 D=DHCP, Vp=Vhost Physical, Pr=Promiscuous, Vnt=Native Vlan Tagged Mnp=No MAC Proxy, Dpdk=DPDK PMD Interface, Rfl=Receive Filtering Offload, Mon=Interface is Monitored Uuf=Unknown Unicast Flood, Vof=VLAN insert/strip offload vif0/29 OS: tap13a2d42c-5d Type:Virtual HWaddr:bk:2a:n6:00:02:00 IPaddr:0 Vrf:12 Flags:PL3L2D MTU:9160 Ref:6 RX packets:1311527 bytes:145819268 errors:0 TX packets:1305396 bytes:125789289 errors:0 cmpt001:~# cat /var/lib/nova/instances/a1b2c3d4e5f6g-7h8i-9j0k-1l2m3n4o5p6q/libvirt.xml | grep -i tap <target dev="tapc218da9d-32"/> <target dev="tap13a2d42c-5d"/> <target dev="tap93d2dja8-22"/> <target dev="tapa8d2gf3a-k4"/>
|
This is the same vFWas was used for the Ingress flow so this symmetric flow is now verified. If traffic was SOURCED from VM out, then you would follow the same procedure outbound the rest of the way as we used to trace the flow inbound into the VM 10.10.0.100.
|
Continue the same process to trace the flows outbound from the vFW to the VM.
References
|
RESOURCE
|
DESCRIPTION
|
| Juniper Contrail – Life of Packet | Life of packet Blog |
| Juniper vRouter Commands | Juniper GitHub for vRouter CLI Commands |
| Juniper vRouter Utilities |
Juniper TechPubs vRouter CLI Utilities
|
| OpenStack CLI Reference |
OpenStack CLI Reference
|
