Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

[Mike Hammett <nanog () ics-il net>]

Via all mechanisms I could find in the router, it thinks the best path is the direct path, the packets just don't go 
that way. 


The in traffic isn't a concern at this time, just the out (from my perspective). 




----- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

----- Original Message -----

From: "David Bass" <davidbass570 () gmail com> 
To: "Mike Hammett" <nanog () ics-il net> 
Cc: "Matthew Huff" <mhuff () ox com>, "NANOG" <nanog () nanog org> 
Sent: Tuesday, April 4, 2023 11:39:26 AM 
Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 


If you are both connected to the same upstream, but the customer wants traffic destined to the upstream to go through 
you (in and out), then they need to do something on their devices to try and affect the inbound path to their AS. From 
the upstream carrier in question they’ll take the best path to a prefix, which direct connection is generally going to 
be preferred over a transit AS (basic BGP best path algorithm stuff) unless there is some manipulation of the prefix 
advertisement happening. 


To confirm the path being taken you should be able to do a few trace routes from various locations as well as use 
looking glasses. 


Now the sflow data is an entirely different thing to analyze. 



On Tue, Apr 4, 2023 at 7:45 AM Mike Hammett < nanog () ics-il net > wrote: 









sh ip bgp neighbor advertised-routes shows the only routes being advertised to Y are the routes that should be 
advertised to them. I checked a variety of other peers and have the expected results. 






> From my perspective: 


Packets come in on port A, supposed to leave on port X, but they leave on port Y. All of the troubleshooting steps I've 
done on my own (or suggested by mailing lists) say the packets should be leaving on the desired port X. 




> From the customer's perspective, they're supposed to be coming from me on port X, but they're arriving on port Y, 
> another network . 


Port X in both scenarios is our direct connection, while port Y is a mutual upstream provider. 




Without knowing more about the specific platform, it seems to me like a bug in the platform. If all indicators (not 
just configurations, but show commands as well) say the packet should be leaving on X and it leaves on Y, then I'm not 
sure what else it could be, besides a bug. 





----- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 



From: "David Bass" < davidbass570 () gmail com > 
To: "Mike Hammett" < nanog () ics-il net > 
Cc: "Matthew Huff" < mhuff () ox com >, "NANOG" < nanog () nanog org > 
Sent: Monday, April 3, 2023 9:12:52 PM 




Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 


You said that they are seeing traffic from another upstream…are you advertising the prefix to them? Are you advertising 
their prefix to your upstream? 


Looks like the route maps are involved in some dual redistribution…might want to make sure everything is matching 
correctly, and being advertised like you want. 



On Mon, Apr 3, 2023 at 4:20 PM Mike Hammett < nanog () ics-il net > wrote: 

<blockquote>



I don't see any route-maps applied to interfaces, so there must not be any PBR going on. I only see ACLs, setting 
communities, setting local pref, etc. in the route maps that are applied to neighbors. 




----- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 



From: "Mike Hammett" < nanog () ics-il net > 
To: "Matthew Huff" < mhuff () ox com > 
Cc: "NANOG" < nanog () nanog org > 
Sent: Monday, April 3, 2023 8:26:30 AM 



Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 




Only two VRFs, default and manangement. IIRC, everything I saw before mentioned the default VRF. 

I do see a ton of route-maps. It's mostly Greek to me, so I'll have to dig through this a bit to see what's going on. 




----- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 



From: "Matthew Huff" < mhuff () ox com > 
To: "Mike Hammett" < nanog () ics-il net > 
Cc: "NANOG" < nanog () nanog org > 
Sent: Monday, April 3, 2023 8:06:51 AM 
Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 

What about VRFs and/or policy based routing? 

switch-core1# show vrf 
VRF-Name VRF-ID State Reason 
default 1 Up -- 
management 2 Up -- 

switch-core1# show route-map 
route-map rmap_bgp_to_eigrp_b2b, permit, sequence 10 
Match clauses: 
interface: Ethernet1/33 
route-type: internal 
Set clauses: 
metric 40000000 10 255 1 1500 
route-map rmap_bgp_to_eigrp_b2b, permit, sequence 20 
Match clauses: 
interface: Ethernet1/34 
route-type: internal 
Set clauses: 
metric 40000000 30 255 1 1500 
route-map rmap_static_to_eigrp, permit, sequence 10 
Match clauses: 
ip address prefix-lists: prefix_static_to_eigrp 
Set clauses: 
route-map rmap_static_to_eigrp_v6, permit, sequence 10 
Match clauses: 
ipv6 address prefix-lists: prefix_ipv6_static_to_eigrp 
Set clauses: 



From: Mike Hammett < nanog () ics-il net > 
Sent: Monday, April 3, 2023 9:00 AM 
To: Matthew Huff < mhuff () ox com > 
Cc: NANOG < nanog () nanog org > 
Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 

It could be an sFlow bug, but I come at this from a reported problem and gathering data on that problem as opposed to 
looking at data for problems. 

The snmp if index reported by the Nexus matches the if index in ElastiFlow. 




----- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

________________________________________ 
From: "Matthew Huff" <mailto: mhuff () ox com > 
To: "Mike Hammett" <mailto: nanog () ics-il net > 
Cc: "NANOG" <mailto: nanog () nanog org > 
Sent: Monday, April 3, 2023 7:50:08 AM 
Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 
SFlow misconfiguration or bug on either the nexus or the sflow monitor? On the monitor, can you verify that the snmp 
interfaces are mapped to the correct ones on the nexus? 





From: Mike Hammett <mailto: nanog () ics-il net > 
Sent: Monday, April 3, 2023 8:47 AM 
To: Matthew Huff <mailto: mhuff () ox com > 
Cc: NANOG <mailto: nanog () nanog org > 
Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 

It shows the desired result. 


----- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

________________________________________ 
From: "Matthew Huff" <mailto: mhuff () ox com > 
To: "Mike Hammett" <mailto: nanog () ics-il net >, "NANOG" <mailto: nanog () nanog org > 
Sent: Monday, April 3, 2023 5:38:23 AM 
Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 

switch-core1# sh forwarding route x.x.x.x 

slot 1 
======= 


IPv4 routes for table default/base 

------------------+-----------------------------------------+----------------------+-----------------+----------------- 
Prefix | Next-hop | Interface | Labels | Partial Install 
------------------+-----------------------------------------+----------------------+-----------------+----------------- 
x.x.x.x/24 x.x.x.250 Ethernet1/29 


switch-core1# show routing hash x.x.x.x y.y.y.y 
Load-share parameters used for software forwarding: 
load-share mode: address source-destination port source-destination 
Hash for VRF "default" 
Hashing to path *y.y.y.y Eth1/29 
For route: 
y.y.y.0/24, ubest/mbest: 1/0 
*via z.z.z.z, Eth1/29, [90/3072], 1w2d, eigrp-100, internal 




From: NANOG <mailto: nanog-bounces+mhuff = ox.com () nanog org > On Behalf Of Mike Hammett 
Sent: Monday, April 3, 2023 1:21 AM 
To: NANOG <mailto: nanog () nanog org > 
Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 

We have a Nexus 3064 that is setup with partial BGP tables and is routing based on that. 


I've done a show ip bgp for an IP of interest and it has an expected next hop IP. I show ip arp on that next hop IP and 
it has the expected interface. 


However, sFlows show the packets leaving on a different interface, the one that would carry the default route for 
routes not otherwise known. 


If the next hop IP is expected and the ARP of that next hop IP is expected, why are packets leaving out an unexpected 
interface? 


----- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 







</blockquote>