Re: Strange behavior on the Juniper MX240

[Nehul Patel <nehul.patel () gmail com>]

Thank you Saku and the warren  Here is the requested output


show route summary

inet.0: 879635 destinations, 879649 routes (879634 active, 0 holddown, 1
hidden)
              Direct:      9 routes,      8 active
               Local:      8 routes,      8 active
                OSPF:    928 routes,    925 active
                 BGP: 878686 routes, 878678 active
              Static:      2 routes,      2 active
           Aggregate:     15 routes,     12 active

inet.3: 718 destinations, 718 routes (718 active, 0 holddown, 0 hidden)
                 LDP:    718 routes,    718 active

Test_VRF.inet.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
               Local:      1 routes,      1 active

mpls.0: 390 destinations, 390 routes (390 active, 0 holddown, 0 hidden)
                MPLS:      3 routes,      3 active
                 LDP:    387 routes,    387 active

inet6.0: 143065 destinations, 286099 routes (143065 active, 0 holddown, 0
hidden)
              Direct:     13 routes,      9 active
               Local:     10 routes,     10 active
               OSPF3:     16 routes,     15 active
                 BGP: 286060 routes, 143030 active
              Static:      1 routes,      1 active

show chassis fpc
                     Temp  CPU Utilization (%)   Memory    Utilization (%)
Slot State            (C)  Total  Interrupt      DRAM (MB) Heap     Buffer
  0  Online            31     11          0       1024       37         29
  1  Online            31     11          0       1024       45         29
  2  Online            30      4          0       1024       36         29

request pfe execute target fpc0 command "show jtree 0 memory extensive"
SENT: Ukern command: show jtree 0 memory extensive
GOT:
GOT: Jtree memory segment 0 (Context: 0x44817bb0)
GOT: -------------------------------------------
GOT: Memory Statistics:
GOT:    16777216 bytes total
GOT:    16715144 bytes used
GOT:       56880 bytes available (7168 bytes from free pages)
GOT:        3024 bytes wasted
GOT:        2168 bytes unusable
GOT:       32768 pages total
GOT:       32519 pages used (2568 pages used in page alloc)
GOT:         235 pages partially used
GOT:          14 pages free (max contiguous = 6)
GOT:
GOT:  Partially Filled Pages (In bytes):-
GOT:       Unit    Avail Overhead
GOT:          8    26256        0
GOT:         16    14320        0
GOT:         24     8352     2040
GOT:         32      352        0
GOT:         48      432      128
GOT:
GOT:  Free Page Lists(Pg Size = 512 bytes):-
GOT:            Page Bucket Avail(Bytes)
GOT:                    1-1     2560
GOT:                    3-3     1536
GOT:                    6-6     3072
GOT:
GOT:  Fragmentation Index = 0.946, (largest free = 3072)
GOT:  Counters:
GOT:     2643777 allocs (0 failed)
GOT:           0 releases(partial 0)
GOT:     1095040 frees
GOT:           0 holds
GOT:           7 pending frees(pending bytes 56)
GOT:           0 pending forced
GOT:           0 times free blocked
GOT:           0 sync writes
GOT:  Error Counters:-
GOT:           0 bad params
GOT:           0 failed frees
GOT:           0 bad cookie
GOT:
GOT: Jtree memory segment 1 (Context: 0x448997f0)
GOT: -------------------------------------------
GOT: Memory Statistics:
GOT:    16777216 bytes total
GOT:     4589552 bytes used
GOT:    12185384 bytes available (12183552 bytes from free pages)
GOT:        2248 bytes wasted
GOT:          32 bytes unusable
GOT:       32768 pages total
GOT:        8967 pages used (8967 pages used in page alloc)
GOT:           5 pages partially used
GOT:       23796 pages free (max contiguous = 23793)
GOT:
GOT:  Partially Filled Pages (In bytes):-
GOT:       Unit    Avail Overhead
GOT:          8     1416        0
GOT:         16       80        0
GOT:         48      336       32


request pfe execute target fpc1 command "show jtree 0 memory extensive"
SENT: Ukern command: show jtree 0 memory extensive
GOT:
GOT: Jtree memory segment 0 (Context: 0x447cc698)
GOT: -------------------------------------------
GOT: Memory Statistics:
GOT:    16777216 bytes total
GOT:    16715840 bytes used
GOT:       56184 bytes available (8192 bytes from free pages)
GOT:        3024 bytes wasted
GOT:        2168 bytes unusable
GOT:       32768 pages total
GOT:       32533 pages used (2568 pages used in page alloc)
GOT:         219 pages partially used
GOT:          16 pages free (max contiguous = 5)
GOT:
GOT:  Partially Filled Pages (In bytes):-
GOT:       Unit    Avail Overhead
GOT:          8    25544        0
GOT:         16    13312        0
GOT:         24     8352     2040
GOT:         32      352        0
GOT:         48      432      128
GOT:
GOT:  Free Page Lists(Pg Size = 512 bytes):-
GOT:            Page Bucket Avail(Bytes)
GOT:                    1-1     2048
GOT:                    2-2     1024
GOT:                    5-5     5120
GOT:
GOT:  Fragmentation Index = 0.954, (largest free = 2560)
GOT:  Counters:
GOT:     2645725 allocs (0 failed)
GOT:           2 releases(partial 0)
GOT:     1096891 frees
GOT:           0 holds
GOT:           0 pending frees(pending bytes 0)
GOT:           0 pending forced
GOT:           0 times free blocked
GOT:           0 sync writes
GOT:  Error Counters:-
GOT:           0 bad params
GOT:           0 failed frees
GOT:           0 bad cookie
GOT:
GOT: Jtree memory segment 1 (Context: 0x4484e2d8)
GOT: -------------------------------------------
GOT: Memory Statistics:
GOT:    16777216 bytes total
GOT:     4589504 bytes used
GOT:    12185432 bytes available (12184576 bytes from free pages)
GOT:        2248 bytes wasted
GOT:          32 bytes unusable
GOT:       32768 pages total
GOT:        8967 pages used (8967 pages used in page alloc)
GOT:           3 pages partially used
GOT:       23798 pages free (max contiguous = 23798)
GOT:
GOT:  Partially Filled Pages (In bytes):-
GOT:       Unit    Avail Overhead
GOT:          8      424        0
GOT:         16       96        0
GOT:         48      336       32
GOT:
GOT:  Free Page Lists(Pg Size = 512 bytes):-
GOT:            Page Bucket Avail(Bytes)
GOT:               27-32768 12184576
GOT:
GOT:  Fragmentation Index = 0.000, (largest free = 12184576)
GOT:  Counters:
GOT:          45 allocs (0 failed)
GOT:           0 releases(partial 0)
GOT:           0 frees
GOT:           0 holds

On Wed, May 4, 2022 at 1:39 PM Saku Ytti <saku () ytti fi> wrote:

> 'show route summary'
> 'start shell pfe network fpcX'
> 'show jnh N pool summary'
> 'show jnh N pool usage'
>
> On Wed, 4 May 2022 at 23:31, Nehul Patel <nehul.patel () gmail com> wrote:
> > Hi NANOG,
> >
> > We are seeing some strange behavior on our Juniper MX240 Chassis it is
> randomly dropping the routes to the certain destination IP address getting
> the following errors on the MX240 Chassis
> > If Someone has seen these errors before please suggest how to resolve it
> >
> >
> > May  4 12:42:00 cr01 newsyslog[44735]: logfile turned over due to
> size>1024K
> > May  4 12:42:01   /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err
> 5 (Invalid)
> > May  4 12:42:01   /kernel: RT_PFE: RT msg op 3 (PREFIX CHANGE) failed,
> err 5 (Invalid)
> > May  4 12:42:01   last message repeated 4 times
> > May  4 12:42:01   fpc0 RT: IPv6:0 - 2600:40fc:1011::/48 (add rt entry
> into jtree failed)
> > May  4 12:42:01   fpc0 RT-HAL,rt_entry_add_msg_proc,2028:
> rt_halp_vectors->rt_create failed
> > May  4 12:42:01   fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv6,len
> 48 prefix 2600:40fc:1011::/48 nh 1048576
> > May  4 12:42:01   fpc0 RT-HAL,rt_msg_handler,540: route process failed
> > May  4 12:42:01   fpc0 RT: Failed prefix add IPv6 - 2001:67c:20fc::/48
> (No memory) on FE 0
> > May  4 12:42:01   fpc0 RT: IPv6:0 - 2001:67c:20fc::/48 (add rt entry
> into jtree failed)
> > May  4 12:42:01   fpc0 RT-HAL,rt_entry_add_msg_proc,2028:
> rt_halp_vectors->rt_create failed
> > May  4 12:42:01   fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv6,len
> 48 prefix 2001:67c:20fc::/48 nh 1048576
> > May  4 12:42:01   fpc0 RT-HAL,rt_msg_handler,540: route process failed
> > May  4 12:42:01   fpc0 RT: Failed prefix add IPv6 - 2606:2800:e004::/48
> (No memory) on FE 0
> > May  4 12:42:01   fpc0 RT: Failed prefix add IPv6 - 2a05:3181:ffff::/48
> (No memory) on FE 0
> > May  4 12:42:01   /kernel: RT_PFE: RT msg op 3 (PREFIX CHANGE) failed,
> err 5 (Invalid)
> > May  4 12:42:01   /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err
> 5 (Invalid)
> > May  4 12:42:01   /kernel: RT_PFE: RT msg op 2 (PREFIX DELETE) failed,
> err 5 (Invalid)
> > May  4 12:42:02   fpc0 RT: Failed prefix add IPv4 - 79.120.22/24 (No
> memory) on FE 0
> > May  4 12:42:02   fpc0 RT: IPv4:0 - 79.120.22/24 (add rt entry into
> jtree failed)
> > May  4 12:42:02   fpc0 RT-HAL,rt_entry_add_msg_proc,2028:
> rt_halp_vectors->rt_create failed
> > May  4 12:42:02   fpc0 RT-HAL,rt_entry_add_msg_proc,2092: proto ipv4,len
> 24 prefix 79.120.22/24 nh 1048583
> > May  4 12:42:02   /kernel: RT_PFE: RT msg op 1 (PREFIX ADD) failed, err
> 5 (Invalid)
> > May  4 12:42:02   fpc0 RT-HAL,rt_msg_handler,540: route process failed
> >
> > May  4 09:33:17   fpc0 RSMON: Resource Category:jtree
> Instance:jtree2-seg0 Type:free-pages Available:20 is less than LWM
> limit:1638, rsmon_syslog_limit()
> > May  4 09:33:17   fpc0 RSMON: Resource Category:jtree
> Instance:jtree2-seg0 Type:free-dwords Available:1280 is less than LWM
> limit:104857, rsmon_syslog_limit()
> > May  4 09:33:18   fpc0 RSMON: Resource Category:jtree
> Instance:jtree3-seg0 Type:free-pages Available:19 is less than LWM
> limit:1638, rsmon_syslog_limit()
> > May  4 09:33:18   fpc0 RSMON: Resource Category:jtree
> Instance:jtree3-seg0 Type:free-dwords Available:1216 is less than LWM
> limit:104857, rsmon_syslog_limit()
> > May  4 09:33:18   fpc1 RSMON: Resource Category:jtree
> Instance:jtree0-seg0 Type:free-pages Available:16 is less than LWM
> limit:1638, rsmon_syslog_limit()
> > May  4 09:33:18   fpc1 RSMON: Resource Category:jtree
> Instance:jtree0-seg0 Type:free-dwords Available:1024 is less than LWM
> limit:104857, rsmon_syslog_limit()
> > May  4 09:33:18   fpc1 RSMON: Resource Category:jtree
> Instance:jtree1-seg0 Type:free-pages Available:15 is less than LWM
> limit:1638, rsmon_syslog_limit()
> > May  4 09:33:18   fpc1 RSMON: Resource Category:jtree
> Instance:jtree1-seg0 Type:free-dwords Available:960 is less than LWM
> limit:104857, rsmon_syslog_limit()
> > May  4 09:33:18   fpc1 RSMON: Resource Category:jtree
> Instance:jtree2-seg0 Type:free-pages Available:19 is less than LWM
> limit:1638, rsmon_syslog_limit()
> > May  4 09:33:19   fpc1 RSMON: Resource Category:jtree
> Instance:jtree2-seg0 Type:free-dwords Available:1216 is less than LWM
> limit:104857, rsmon_syslog_limit()
> > May  4 09:33:19   fpc1 RSMON: Resource Category:jtree
> Instance:jtree3-seg0 Type:free-pages Available:17 is less than LWM
> limit:1638, rsmon_syslog_limit()
> > May  4 09:33:19   fpc1 RSMON: Resource Category:jtree
> Instance:jtree3-seg0 Type:free-dwords Available:1088 is less than LWM
> limit:104857, rsmon_syslog_limit()
> > May  4 09:33:19   fpc2 RSMON: Resource Category:jtree
> Instance:jtree0-seg0 Type:free-pages Available:15 is less than LWM
> limit:1638, rsmon_syslog_limit()
> > May  4 09:33:19   fpc2 RSMON: Resource Category:jtree
> Instance:jtree0-seg0 Type:free-dwords Available:960 is less than LWM
> limit:104857, rsmon_syslog_limit()
> > May  4 09:33:19   fpc2 RSMON: Resource Category:jtree
> Instance:jtree1-seg0 Type:free-pages Available:15 is less than LWM
> limit:1638, rsmon_syslog_limit()
> > May  4 09:33:19   fpc2 RSMON: Resource Category:jtree
> Instance:jtree1-seg0 Type:free-dwords Available:960 is less than LWM
> limit:104857, rsmon_syslog_limit()
> > Any suggestions will be helpful
> >
> >
> > Please do let me know if you have any questions.
> >
> >
> >
> >
> > Regards and thanks,
> >  Nehul
>
>
> --
>   ++ytti