Re: constant FEC errors juniper mpc10e 400g

[Tom Beecher <beecher () beecher cc>]

I'm being sloppy with my verbiage, it's just been a long time since
I thought about this in detail, sorry.

The MAC layer hands bits to the Media Independent Interface, which connects
the MAC to the PHY. The PHY converts the digital 1/0 into the form required
by the media transmission type; the 'what goes over the wire' L1 stuff. The
method of encoding will always add SOME number of bits as overhead. Ex,
64b/66b means that for every 64 bits of data to transmit, 2 bits are added,
so 66 actual bits are transmitted. This encoding overhead is what I meant
when I said 'ethernet control frame juju'. This starts getting into the
weeds on symbol/baud rates and stuff as well, which I dont want to do now
cause I'm even rustier there.

When FEC is enabled, the number of overhead bits added to the transmission
increases. For 400G-FR4 for example, you start with 256b/257b , which is
doubled to 512b/514b for ($reason I cannot remember), then RS-FEC(544,514)
is applied, adding 30 more bits for FEC. Following the example, this means
544 bits are transmitted for every 512 bits of payload data. So , more
overhead. Those additional bits can correct up to 15 corrupted bits of the
payload.

All of these overhead bits are added in the PHY on the way out, and removed
on the way in. So you'll never see them on a packet capture unless you're
using something that's actually grabbing the bits off the wire.

( Pretty sure this is right, anyone please correct me if I munged any of it
up.)


On Thu, Apr 18, 2024 at 2:45 PM Aaron Gould <aaron1 () gvtc com> wrote:

> Thanks.  What "all the ethernet control frame juju" might you be referring
> to?  I don't recall Ethernet, in and of itself, just sending stuff back and
> forth.  Does anyone know if this FEC stuff I see concurring is actually
> contained in Ethernet Frames?  If so, please send a link to show the
> ethernet frame structure as it pertains to this 400g fec stuff.  If so, I'd
> really like to know the header format, etc.
>
> -Aaron
> On 4/18/2024 1:17 PM, Tom Beecher wrote:
>
> FEC is occurring at the PHY , below the PCS.
>
> Even if you're not sending any traffic, all the ethernet control frame
> juju is still going back and forth, which FEC may have to correct.
>
> I *think* (but not 100% sure) that for anything that by spec requires FEC,
> there is a default RS-FEC type that will be used, which *may* be able to be
> changed by the device. Could be fixed though, I honestly cannot remember.
>
> On Thu, Apr 18, 2024 at 1:35 PM Aaron Gould <aaron1 () gvtc com> wrote:
>
> > Not to belabor this, but so interesting... I need a FEC-for-Dummies or FEC-for-IP/Ethernet-Engineers...
> >
> > Shown below, my 400g interface with NO config at all... Interface has no traffic at all, no packets at all....  BUT, 
> > lots of FEC hits.  Interesting this FEC-thing.  I'd love to have a fiber splitter and see if wireshark could read it 
> > and show me what FEC looks like...but something tells me i would need a 400g sniffer to read it, lol
> >
> > It's like FEC (fec119 in this case) is this automatic thing running between interfaces (hardware i guess), with no 
> > protocols and nothing needed at all in order to function.
> >
> > -Aaron
> >
> >
> > {master}
> > me@mx960> show configuration interfaces et-7/1/4 | display set
> >
> > {master}
> > me@mx960>
> >
> > {master}
> > me@mx960> clear interfaces statistics et-7/1/4
> >
> > {master}
> > me@mx960> show interfaces et-7/1/4 | grep packet
> >     Input packets : 0
> >     Output packets: 0
> >
> > {master}
> > me@mx960> show interfaces et-7/1/4 | grep "put rate"
> >   Input rate     : 0 bps (0 pps)
> >   Output rate    : 0 bps (0 pps)
> >
> > {master}
> > me@mx960> show interfaces et-7/1/4 | grep rror
> >   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU Error: None, Loop Detect PDU Error: None, 
> > Loopback: Disabled, Source filtering: Disabled,
> >     Bit errors                             0
> >     Errored blocks                         0
> >   Ethernet FEC statistics              Errors
> >     FEC Corrected Errors                28209
> >     FEC Uncorrected Errors                  0
> >     FEC Corrected Errors Rate            2347
> >     FEC Uncorrected Errors Rate             0
> >
> > {master}
> > me@mx960> show interfaces et-7/1/4 | grep packet
> >     Input packets : 0
> >     Output packets: 0
> >
> > {master}
> > me@mx960> show interfaces et-7/1/4 | grep "put rate"
> >   Input rate     : 0 bps (0 pps)
> >   Output rate    : 0 bps (0 pps)
> >
> > {master}
> > me@mx960> show interfaces et-7/1/4 | grep rror
> >   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU Error: None, Loop Detect PDU Error: None, 
> > Loopback: Disabled, Source filtering: Disabled,
> >     Bit errors                             0
> >     Errored blocks                         0
> >   Ethernet FEC statistics              Errors
> >     FEC Corrected Errors                45153
> >     FEC Uncorrected Errors                  0
> >     FEC Corrected Errors Rate              29
> >     FEC Uncorrected Errors Rate             0
> >
> > {master}
> > me@mx960> show interfaces et-7/1/4 | grep packet
> >     Input packets : 0
> >     Output packets: 0
> >
> > {master}
> > me@mx960> show interfaces et-7/1/4 | grep "put rate"
> >   Input rate     : 0 bps (0 pps)
> >   Output rate    : 0 bps (0 pps)
> >
> > {master}
> > me@mx960> show interfaces et-7/1/4 | grep rror
> >   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU Error: None, Loop Detect PDU Error: None, 
> > Loopback: Disabled, Source filtering: Disabled,
> >     Bit errors                             0
> >     Errored blocks                         0
> >   Ethernet FEC statistics              Errors
> >     FEC Corrected Errors                57339
> >     FEC Uncorrected Errors                  0
> >     FEC Corrected Errors Rate            2378
> >     FEC Uncorrected Errors Rate             0
> >
> > {master}
> > me@mx960>
> >
> >
> > On 4/18/2024 7:13 AM, Mark Tinka wrote:
> >
> >
> >
> > On 4/17/24 23:24, Aaron Gould wrote:
> >
> > Well JTAC just said that it seems ok, and that 400g is going to show 4x
> > more than 100g "This is due to having to synchronize much more to support
> > higher data."
> >
> >
> > We've seen the same between Juniper and Arista boxes in the same rack
> > running at 100G, despite cleaning fibres, swapping optics, moving ports,
> > moving line cards, e.t.c. TAC said it's a non-issue, and to be expected,
> > and shared the same KB's.
> >
> > It's a bit disconcerting when you plot the data on your NMS, but it's not
> > material.
> >
> > Mark.
> >
> > --
> > -Aaron
> >
> > --
> -Aaron