[c-nsp] IOS-XR and interface discards (input)

Discussion:

Hank Nussbacher

2015-05-14 19:04:35 UTC

We have an ASR 9010 running IOS-XR v 5.1.3. We see a high level of input
discards:

TenGigE0/1/1/7 is up, line protocol is up
Interface state transitions: 25
Layer 1 Transport Mode is WAN
MTU 9028 bytes, BW 10000000 Kbit (Max: 10000000 Kbit)
reliability 255/255, txload 13/255, rxload 99/255
Encapsulation ARPA,
Full-duplex, 10000Mb/s, link type is force-up
output flow control is on, input flow control is on
Carrier delay (up) is 100 msec, Carrier delay (down) is 4000 msec
loopback not set,
ARP type ARPA, ARP timeout 04:00:00
Last input 00:00:00, output 00:00:00
Last clearing of "show interface" counters 1d13h
30 second input rate 3886868000 bits/sec, 392410 packets/sec
30 second output rate 527057000 bits/sec, 154557 packets/sec
55223825999 packets input, 63505398737673 bytes, 116320028 total
input drops
0 drops for unrecognized upper-level protocol
Received 111 broadcast packets, 1499584 multicast packets
0 runts, 0 giants, 0 throttles, 0 parity
10 input errors, 5 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

The config on the interface looks like this:

interface TenGigE0/1/1/7
mtu 9028
ipv4 unreachables disable
ipv6 nd dad attempts 5
ipv6 address 2001:798:28:20aa::6/126
monitor-session No1 ethernet
flow-control bidirectional
carrier-delay up 100 down 4000
load-interval 30
flow ipv4 monitor fmp-nfsen sampler fsm-nfsen ingress
transport-mode wan
ipv4 access-group catch13-ing ingress
!

Any clue would be helpful for us to begin to debug this issue.

Thanks,
Hank

_______________________________________________
cisco-nsp mailing list cisco-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Mikael Abrahamsson

2015-05-14 19:31:42 UTC

Permalink

Post by Hank Nussbacher
interface TenGigE0/1/1/7
mtu 9028
ipv4 unreachables disable
ipv6 nd dad attempts 5
ipv6 address 2001:798:28:20aa::6/126
monitor-session No1 ethernet
flow-control bidirectional
carrier-delay up 100 down 4000
load-interval 30
flow ipv4 monitor fmp-nfsen sampler fsm-nfsen ingress
transport-mode wan
ipv4 access-group catch13-ing ingress
!
Any clue would be helpful for us to begin to debug this issue.

I would look at this:

http://www.cisco.com/c/en/us/td/docs/routers/asr9000/software/asr9k_r4-0/troubleshooting/guide/tr40asr9kbook.pdf
, especially the "show controller interface TenGigE0/1/1/7 stats" output.

I don't remember exactly how this works, it was two years since I last did
this, but I seem to remember that on some plattforms if an ACL causes a
packet loss, it's seen as "input drop".

Also, I am curious behind your reasoning for the carrier delay settings
(I'd say they are backwards to what I would configure), and why are you
using flow control?.

--
Mikael Abrahamsson email: ***@swm.pp.se
_______________________________________________
cisco-nsp mailing list cisco-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Hank Nussbacher

2015-05-14 19:53:42 UTC

Permalink

Post by Mikael Abrahamsson

http://www.cisco.com/c/en/us/td/docs/routers/asr9000/software/asr9k_r4-0/troubleshooting/guide/tr40asr9kbook.pdf
, especially the "show controller interface TenGigE0/1/1/7 stats" output.

Thanks. The stats looks clear:
RP/0/RSP0/CPU0:GP1#show controller TenGigE0/1/1/7 stats
Thu May 14 22:47:24.492 IDT
Statistics for interface TenGigE0/1/1/7 (cached values):

Ingress:
Input total bytes = 64749515187975
Input good bytes = 64749515187975

Input total packets = 56263865420
Input 802.1Q frames = 0
Input pause frames = 0
Input pkts 64 bytes = 3669376552
Input pkts 65-127 bytes = 6922264753
Input pkts 128-255 bytes = 1119116057
Input pkts 256-511 bytes = 848490198
Input pkts 512-1023 bytes = 1106852656
Input pkts 1024-1518 bytes = 42597682434
Input pkts 1519-Max bytes = 82772

Input good pkts = 56263865410
Input unicast pkts = 56262343834
Input multicast pkts = 1521473
Input broadcast pkts = 113

Input drop overrun = 0
Input drop abort = 0
Input drop invalid VLAN = 0
Input drop invalid DMAC = 0
Input drop invalid encap = 0
Input drop other = 0

Input error giant = 0
Input error runt = 0
Input error jabbers = 0
Input error fragments = 0
Input error CRC = 5
Input error collisions = 0
Input error symbol = 0
Input error other = 5

Input MIB giant = 82772
Input MIB jabber = 0
Input MIB CRC = 5

Egress:
Output total bytes = 16487074186716
Output good bytes = 16487074186716

Output total packets = 34302407278
Output 802.1Q frames = 0
Output pause frames = 53469
Output pkts 64 bytes = 9983012845
Output pkts 65-127 bytes = 12091864296
Output pkts 128-255 bytes = 994597516
Output pkts 256-511 bytes = 839278845
Output pkts 512-1023 bytes = 814935894
Output pkts 1024-1518 bytes = 9578715883
Output pkts 1519-Max bytes = 2000

Output good pkts = 34302407278
Output unicast pkts = 34302336577
Output multicast pkts = 68701
Output broadcast pkts = 0

Output drop underrun = 0
Output drop abort = 0
Output drop other = 0

Output error other = 0

Another issue might be SPAN ports. We do:

TenGigE0/0/1/2 -> TenGigE0/1/1/4
TenGigE0/1/1/7 -> TenGigE0/1/1/4

Looking at:

http://www.cisco.com/c/en/us/td/docs/routers/asr9000/software/asr9k_r4-2/interfaces/configuration/guide/hc42asr9kbook/hc42span.html#wp1395653

Specifically:

"Performance Impact with Traffic Mirroring

It is recommended that you do not mirror more than 15% of your total
transit traffic. On the Cisco ASR 9000 Ethernet Line Card, that uses Ten
Gigabit Ethernet interfaces or bundle interfaces there is a limit of
1.5G of data on each of the ingress and egress traffic that can be
mirrored. This limitation is not applicable on the Cisco ASR 9000
Enhanced Ethernet Line Card. - See more at:
https://supportforums.cisco.com/document/59721/asr9000xr-troubleshooting-
packet-drops-and-understanding-np-drop-counters#sthash.Vltc5vMM.dpuf "

Enhanced Ethernet Line cards are Typhoon type cards and we ordered
A9K-MOD160-TR
which according to Cisco falls into the Enhanced category so we should be ok:
http://www.cisco.com/c/en/us/support/docs/routers/asr-9000-series-aggregation-services-routers/116726-qanda-product-00.html

Could all these input discards be just mirrored pkts being dropped to the
SPAN port?

Post by Mikael Abrahamsson
I don't remember exactly how this works, it was two years since I last did
this, but I seem to remember that on some plattforms if an ACL causes a
packet loss, it's seen as "input drop".

Really!!!???? I would love to see that reference for that.

Post by Mikael Abrahamsson
Also, I am curious behind your reasoning for the carrier delay settings
(I'd say they are backwards to what I would configure), and why are you
using flow control?.

The reason for the carrier delay settings was due to frequent line
transitions that caused us to lose our BGP peer. The trade off is to live
for a brief hiccup and not drop the link vs dropping the link quickly
but then suffering a BGP reset and reload of the full routing tables.

Thanks,
Hank

Post by Mikael Abrahamsson
--

_______________________________________________
cisco-nsp mailing list cisco-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Nikolay Shopik

2015-05-14 22:19:23 UTC

Permalink

Search list for details, you need to look for output
show controllers np ports all location 0/0/CPU0
show controllers np counters np1 location 0/0/CPU0 | i "DROP|DISCARD|NOT"

Post by Hank Nussbacher
TenGigE0/1/1/7 is up, line protocol is up
Interface state transitions: 25
Layer 1 Transport Mode is WAN
MTU 9028 bytes, BW 10000000 Kbit (Max: 10000000 Kbit)
reliability 255/255, txload 13/255, rxload 99/255
Encapsulation ARPA,
Full-duplex, 10000Mb/s, link type is force-up
output flow control is on, input flow control is on
Carrier delay (up) is 100 msec, Carrier delay (down) is 4000 msec
loopback not set,
ARP type ARPA, ARP timeout 04:00:00
Last input 00:00:00, output 00:00:00
Last clearing of "show interface" counters 1d13h
30 second input rate 3886868000 bits/sec, 392410 packets/sec
30 second output rate 527057000 bits/sec, 154557 packets/sec
55223825999 packets input, 63505398737673 bytes, 116320028 total input drops
0 drops for unrecognized upper-level protocol
Received 111 broadcast packets, 1499584 multicast packets
0 runts, 0 giants, 0 throttles, 0 parity
10 input errors, 5 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
interface TenGigE0/1/1/7
mtu 9028
ipv4 unreachables disable
ipv6 nd dad attempts 5
ipv6 address 2001:798:28:20aa::6/126
monitor-session No1 ethernet
flow-control bidirectional
carrier-delay up 100 down 4000
load-interval 30
flow ipv4 monitor fmp-nfsen sampler fsm-nfsen ingress
transport-mode wan
ipv4 access-group catch13-ing ingress
!
Any clue would be helpful for us to begin to debug this issue.
Thanks,
Hank
_______________________________________________
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

_______________________________________________
cisco-nsp mailing list cisco-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/piperma

John Neiberger

2015-05-14 22:19:52 UTC

Permalink

You can check the NP drop counters using the "show drops" command on ASR.
You can get basically the same data by doing "show controller np ports all
location <location>" to find the relevant NP then do "show controllers np
counters <np> <location>". There's usually a lot of data there and it can
be hard to wade through, but that's the best place to start, IMO.

John

Post by Hank Nussbacher
We have an ASR 9010 running IOS-XR v 5.1.3. We see a high level of input
TenGigE0/1/1/7 is up, line protocol is up
Interface state transitions: 25
Layer 1 Transport Mode is WAN
MTU 9028 bytes, BW 10000000 Kbit (Max: 10000000 Kbit)
reliability 255/255, txload 13/255, rxload 99/255
Encapsulation ARPA,
Full-duplex, 10000Mb/s, link type is force-up
output flow control is on, input flow control is on
Carrier delay (up) is 100 msec, Carrier delay (down) is 4000 msec
loopback not set,
ARP type ARPA, ARP timeout 04:00:00
Last input 00:00:00, output 00:00:00
Last clearing of "show interface" counters 1d13h
30 second input rate 3886868000 bits/sec, 392410 packets/sec
30 second output rate 527057000 bits/sec, 154557 packets/sec
55223825999 packets input, 63505398737673 bytes, 116320028 total
input drops
0 drops for unrecognized upper-level protocol
Received 111 broadcast packets, 1499584 multicast packets
0 runts, 0 giants, 0 throttles, 0 parity
10 input errors, 5 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
interface TenGigE0/1/1/7
mtu 9028
ipv4 unreachables disable
ipv6 nd dad attempts 5
ipv6 address 2001:798:28:20aa::6/126
monitor-session No1 ethernet
flow-control bidirectional
carrier-delay up 100 down 4000
load-interval 30
flow ipv4 monitor fmp-nfsen sampler fsm-nfsen ingress
transport-mode wan
ipv4 access-group catch13-ing ingress
!
Any clue would be helpful for us to begin to debug this issue.
Thanks,
Hank
_______________________________________________
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

brad dreisbach

2015-05-15 06:48:11 UTC

Permalink

https://supportforums.cisco.com/document/59721/asr9000xr-troubleshooting-packet-drops-and-understanding-np-drop-counters

Post by Hank Nussbacher
Thanks,
Hank
_______________________________________________
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Alexandr Gurbo

2015-05-15 08:02:13 UTC

Permalink

Hello,

Check you fiber connection.

On Thu, 14 May 2015 22:04:35 +0300

--
Alexandr Gurbo
_______________________________________________
cisco-nsp mailing list cisco-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Andrew Miehs

2015-05-15 18:30:01 UTC

Permalink

Discards is probably acl/ control plane acl, full buffers on egress port/ fabric.

Not familiar with the platform so cant provide more info.

Sent from a mobile device

Post by Alexandr Gurbo
Hello,
Check you fiber connection.
On Thu, 14 May 2015 22:04:35 +0300

--
Alexandr Gurbo
_______________________________________________
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Hank Nussbacher

2015-05-16 18:01:45 UTC

Permalink

At 11:30 15/05/2015 -0700, Andrew Miehs wrote:

Thanks for all the feedback and links. The person who came up with the

" ipv4 unreachables disable”
I believe that XR counts packets that it needs to respond to with an
unreachable as a input drop, since these are disabled.
Once I disabled this command I stopped seeing input drop counters on our
interface increment.

We disabled this command and the discards disappeared immediately.

Regards,
Hank

Discards is probably acl/ control plane acl, full buffers on egress port/
fabric.
Not familiar with the platform so cant provide more info.
Sent from a mobile device

Post by Alexandr Gurbo
Hello,
Check you fiber connection.
On Thu, 14 May 2015 22:04:35 +0300

--
Alexandr Gurbo
_______________________________________________
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/