Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

[Nick Olsen <nick () 141networks com>]

Not that it's a "Fix" but have you tried rebooting the box? If this is a
bug in the forwarding plane that might clear/rebuild it. And maybe it works
correctly after that.

Friend saw something similar on a Juniper MX with DPC cards that had run
out of FIB space. It would show correctly in all places, but was failing to
install in FIB (Router cut a error about it in the log). So the actual
traffic didn't follow the same path. I saw you specifically referenced
"forwarding" in one of your copy pastes. And it looked right. But I don't
know enough about Cisco to say if that's really what is in FIB. Or just
what it thinks is in FIB.

On Tue, Apr 4, 2023 at 5:47 PM Mike Hammett <nanog () ics-il net> wrote:

> Via all mechanisms I could find in the router, it thinks the best path is
> the direct path, the packets just don't go that way.
>
> The in traffic isn't a concern at this time, just the out (from my
> perspective).
>
>
>
> -----
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
> Midwest-IX
> http://www.midwest-ix.com
>
> ------------------------------
> *From: *"David Bass" <davidbass570 () gmail com>
> *To: *"Mike Hammett" <nanog () ics-il net>
> *Cc: *"Matthew Huff" <mhuff () ox com>, "NANOG" <nanog () nanog org>
> *Sent: *Tuesday, April 4, 2023 11:39:26 AM
> *Subject: *Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
>
> If you are both connected to the same upstream, but the customer wants
> traffic destined to the upstream to go through you (in and out), then they
> need to do something on their devices to try and affect the inbound path to
> their AS. From the upstream carrier in question they’ll take the best path
> to a prefix, which direct connection is generally going to be preferred
> over a transit AS (basic BGP best path algorithm stuff) unless there is
> some manipulation of the prefix advertisement happening.
>
> To confirm the path being taken you should be able to do a few trace
> routes from various locations as well as use looking glasses.
>
> Now the sflow data is an entirely different thing to analyze.
>
> On Tue, Apr 4, 2023 at 7:45 AM Mike Hammett <nanog () ics-il net> wrote:
>
> > sh ip bgp neighbor advertised-routes shows the only routes being
> > advertised to Y are the routes that should be advertised to them. I checked
> > a variety of other peers and have the expected results.
> >
> >
> >
> > From my perspective:
> >
> > Packets come in on port A, supposed to leave on port X, but they leave on
> > port Y. All of the troubleshooting steps I've done on my own (or suggested
> > by mailing lists) say the packets should be leaving on the desired port X.
> >
> >
> > From the customer's perspective, they're supposed to be coming from me on
> > port X, but they're arriving on port Y, another network.
> >
> > Port X in both scenarios is our direct connection, while port Y is a
> > mutual upstream provider.
> >
> >
> > Without knowing more about the specific platform, it seems to me like a
> > bug in the platform. If all indicators (not just configurations, but show
> > commands as well) say the packet should be leaving on X and it leaves on Y,
> > then I'm not sure what else it could be, besides a bug.
> >
> >
> >
> > -----
> > Mike Hammett
> > Intelligent Computing Solutions
> > http://www.ics-il.com
> >
> > Midwest-IX
> > http://www.midwest-ix.com
> >
> > ------------------------------
> > *From: *"David Bass" <davidbass570 () gmail com>
> > *To: *"Mike Hammett" <nanog () ics-il net>
> > *Cc: *"Matthew Huff" <mhuff () ox com>, "NANOG" <nanog () nanog org>
> > *Sent: *Monday, April 3, 2023 9:12:52 PM
> >
> > *Subject: *Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
> >
> > You said that they are seeing traffic from another upstream…are you
> > advertising the prefix to them?  Are you advertising their prefix to your
> > upstream?
> >
> > Looks like the route maps are involved in some dual redistribution…might
> > want to make sure everything is matching correctly, and being advertised
> > like you want.
> >
> > On Mon, Apr 3, 2023 at 4:20 PM Mike Hammett <nanog () ics-il net> wrote:
> >
> > > I don't see any route-maps applied to interfaces, so there must not be
> > > any PBR going on. I only see ACLs, setting communities, setting local pref,
> > > etc. in the route maps that are applied to neighbors.
> > >
> > >
> > >
> > > -----
> > > Mike Hammett
> > > Intelligent Computing Solutions
> > > http://www.ics-il.com
> > >
> > > Midwest-IX
> > > http://www.midwest-ix.com
> > >
> > > ------------------------------
> > > *From: *"Mike Hammett" <nanog () ics-il net>
> > > *To: *"Matthew Huff" <mhuff () ox com>
> > > *Cc: *"NANOG" <nanog () nanog org>
> > > *Sent: *Monday, April 3, 2023 8:26:30 AM
> > >
> > > *Subject: *Re: Cisco Nexus 3k Route Selection\Packet Forwarding
> > > Debugging
> > >
> > > Only two VRFs, default and manangement. IIRC, everything I saw before
> > > mentioned the default VRF.
> > >
> > > I do see a ton of route-maps. It's mostly Greek to me, so I'll have to
> > > dig through this a bit to see what's going on.
> > >
> > >
> > >
> > > -----
> > > Mike Hammett
> > > Intelligent Computing Solutions
> > > http://www.ics-il.com
> > >
> > > Midwest-IX
> > > http://www.midwest-ix.com
> > >
> > > ------------------------------
> > > *From: *"Matthew Huff" <mhuff () ox com>
> > > *To: *"Mike Hammett" <nanog () ics-il net>
> > > *Cc: *"NANOG" <nanog () nanog org>
> > > *Sent: *Monday, April 3, 2023 8:06:51 AM
> > > *Subject: *RE: Cisco Nexus 3k Route Selection\Packet Forwarding
> > > Debugging
> > >
> > > What about VRFs and/or policy based routing?
> > >
> > > switch-core1# show vrf
> > > VRF-Name                           VRF-ID State   Reason
> > >
> > > default                                 1 Up      --
> > >
> > > management                              2 Up      --
> > >
> > >
> > > switch-core1# show route-map
> > > route-map rmap_bgp_to_eigrp_b2b, permit, sequence 10
> > >   Match clauses:
> > >     interface: Ethernet1/33
> > >     route-type: internal
> > >   Set clauses:
> > >     metric 40000000 10 255 1 1500
> > > route-map rmap_bgp_to_eigrp_b2b, permit, sequence 20
> > >   Match clauses:
> > >     interface: Ethernet1/34
> > >     route-type: internal
> > >   Set clauses:
> > >     metric 40000000 30 255 1 1500
> > > route-map rmap_static_to_eigrp, permit, sequence 10
> > >   Match clauses:
> > >     ip address prefix-lists: prefix_static_to_eigrp
> > >   Set clauses:
> > > route-map rmap_static_to_eigrp_v6, permit, sequence 10
> > >   Match clauses:
> > >     ipv6 address prefix-lists: prefix_ipv6_static_to_eigrp
> > >   Set clauses:
> > >
> > >
> > >
> > > From: Mike Hammett <nanog () ics-il net>
> > > Sent: Monday, April 3, 2023 9:00 AM
> > > To: Matthew Huff <mhuff () ox com>
> > > Cc: NANOG <nanog () nanog org>
> > > Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
> > >
> > > It could be an sFlow bug, but I come at this from a reported problem and
> > > gathering data on that problem as opposed to looking at data for problems.
> > >
> > > The snmp if index reported by the Nexus matches the if index in
> > > ElastiFlow.
> > >
> > >
> > >
> > >
> > > -----
> > > Mike Hammett
> > > Intelligent Computing Solutions
> > > http://www.ics-il.com
> > >
> > > Midwest-IX
> > > http://www.midwest-ix.com
> > >
> > > ________________________________________
> > > From: "Matthew Huff" <mailto:mhuff () ox com>
> > > To: "Mike Hammett" <mailto:nanog () ics-il net>
> > > Cc: "NANOG" <mailto:nanog () nanog org>
> > > Sent: Monday, April 3, 2023 7:50:08 AM
> > > Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
> > > SFlow misconfiguration or bug on either the nexus or the sflow monitor?
> > > On the monitor, can you verify that the snmp interfaces are mapped to the
> > > correct ones on the nexus?
> > >
> > >
> > >
> > >
> > >
> > > From: Mike Hammett <mailto:nanog () ics-il net>
> > > Sent: Monday, April 3, 2023 8:47 AM
> > > To: Matthew Huff <mailto:mhuff () ox com>
> > > Cc: NANOG <mailto:nanog () nanog org>
> > > Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
> > >
> > > It shows the desired result.
> > >
> > >
> > > -----
> > > Mike Hammett
> > > Intelligent Computing Solutions
> > > http://www.ics-il.com
> > >
> > > Midwest-IX
> > > http://www.midwest-ix.com
> > >
> > > ________________________________________
> > > From: "Matthew Huff" <mailto:mhuff () ox com>
> > > To: "Mike Hammett" <mailto:nanog () ics-il net>, "NANOG" <mailto:
> > > nanog () nanog org>
> > > Sent: Monday, April 3, 2023 5:38:23 AM
> > > Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
> > >
> > > switch-core1# sh forwarding route x.x.x.x
> > >
> > > slot  1
> > > =======
> > >
> > >
> > > IPv4 routes for table default/base
> > >
> > >
> > > ------------------+-----------------------------------------+----------------------+-----------------+-----------------
> > > Prefix            | Next-hop                                | Interface
> > >            | Labels          | Partial Install
> > >
> > > ------------------+-----------------------------------------+----------------------+-----------------+-----------------
> > > x.x.x.x/24      x.x.x.250                            Ethernet1/29
> > >
> > >
> > > switch-core1# show routing hash x.x.x.x y.y.y.y
> > > Load-share parameters used for software forwarding:
> > > load-share mode: address source-destination port source-destination
> > > Hash for VRF "default"
> > > Hashing to path *y.y.y.y Eth1/29
> > > For route:
> > > y.y.y.0/24, ubest/mbest: 1/0
> > >     *via z.z.z.z, Eth1/29, [90/3072], 1w2d, eigrp-100, internal
> > >
> > >
> > >
> > >
> > > From: NANOG <mailto:nanog-bounces+mhuff=ox.com () nanog org> On Behalf Of
> > > Mike Hammett
> > > Sent: Monday, April 3, 2023 1:21 AM
> > > To: NANOG <mailto:nanog () nanog org>
> > > Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
> > >
> > > We have a Nexus 3064 that is setup with partial BGP tables and is
> > > routing based on that.
> > >
> > >
> > > I've done a show ip bgp for an IP of interest and it has an expected
> > > next hop IP. I show ip arp on that next hop IP and it has the expected
> > > interface.
> > >
> > >
> > > However, sFlows show the packets leaving on a different interface, the
> > > one that would carry the default route for routes not otherwise known.
> > >
> > >
> > > If the next hop IP is expected and the ARP of that next hop IP is
> > > expected, why are packets leaving out an unexpected interface?
> > >
> > >
> > > -----
> > > Mike Hammett
> > > Intelligent Computing Solutions
> > > http://www.ics-il.com
> > >
> > > Midwest-IX
> > > http://www.midwest-ix.com