[c-nsp] Storm-Control on server switch uplinks.

[Christina Klam]

All,

A couple weeks ago, I added storm-control for all of the (two 1-gig
fiber)uplinks to our Cat6500 gateway switch. Because of the large
amounts of drops from the data center switch uplinks, I removed unicast
storm-control. What are your thoughts on using storm-control, in
general, on switch uplinks?

Port UcastSupp % McastSupp % BcastSupp %
TotalSuppDiscards

Po25 90.00 90.00 90.00
21710

Po33 90.00 90.00 90.00
44700

Po38 90.00 90.00 90.00
2629797

Thank you,
-- Christina
Christina Klam
Network Administrator

[Peter Rathlev]

Post by Christina Klam
A couple weeks ago, I added storm-control for all of the (two 1-gig
fiber)uplinks to our Cat6500 gateway switch. Because of the large
amounts of drops from the data center switch uplinks, I removed unicast
storm-control. What are your thoughts on using storm-control, in
general, on switch uplinks?

We use broadcast og multicast storm-control on downlinks towards access
switches, generally at 50% just to make sure a broadcast storm doesn't
spread too much.

I'm not sure I understand unicast suppresion; it sometimes triggers even
when I would think it shouldn't. I haven't been able to grab the data
(via SPAN) so I can't say exactly why it happens. The scenario was
without any L2 redundancy, which could otherwise explain unknown
unicast.

I'm not sure why one would use it on uplinks, i.e. on the access switch
side of things. AFAIK storm-control is an ingress thing.

[Saku Ytti]

Post by Peter Rathlev
We use broadcast og multicast storm-control on downlinks towards access
switches, generally at 50% just to make sure a broadcast storm doesn't
spread too much.

I would run <1% broadcast storm control, preferably entered in pps and
rather low number. On 10GE 7600 receiving <0.34% bps broadcast (L3 or L2
port, doesn't matter) it can still drop IGP on unrelated interface and
cause serious downtime.
But of course you can use 50% too if you have 50% excess capacity to handle
the storm and you know your devices can handle it. In typical network 1kpps
of broadcast is very high number for even 10GE L2, unless there is some
special application.

Post by Peter Rathlev
I'm not sure I understand unicast suppresion; it sometimes triggers even
when I would think it shouldn't. I haven't been able to grab the data

In some switches (e.g. 3550) if multicast triggers, all traffic is
suppressed, not just multicast.
Otherwise, I've not seen unicast blocked unless unicast is exceeded. I
think that the unicast storm-control is quite useless, unknown unicast
storm control would be much more useful.
I've used unicast storm-control in office LAN, to stop infected PC's from
congesting firewall. But I wouldn't run it on my customers as standard, as
our products do not dictate pps limit.

Post by Peter Rathlev
I'm not sure why one would use it on uplinks, i.e. on the access switch
side of things. AFAIK storm-control is an ingress thing.

If storm happens behind many edge ports, you could get aggregated rate
multiple times higher than individual access port limit. So you might want
to limit edge port to 1 unit and core to say 5 units.

[Phil Mayers]

Post by Peter Rathlev
We use broadcast og multicast storm-control on downlinks towards access
switches, generally at 50% just to make sure a broadcast storm doesn't
spread too much.

I would run<1% broadcast storm control, preferably entered in pps and
rather low number. On 10GE 7600 receiving<0.34% bps broadcast (L3 or L2
port, doesn't matter) it can still drop IGP on unrelated interface and
cause serious downtime.

Doubtless you know this because you specified 0.34, but for other
readers, a cautionary tale: We recently deployed an edge 10gig port on a
6716, and I set the broadcast storm level to 0.10 as per our standard
config:

storm-control broadcast level 0.10

Unfortunately on this linecard, anything <0.34 == 0, i.e. all broadcasts
trigger it. This makes things like ARP rather unhappy! So beware the
varying linecard limits.

It's a real shame the broadcast limiter (along with all the other limits
on this platform) aren't more granular e.g. per-vlan broadcast and glean
limits on a trunk port, etc.
_______________________________________________
cisco-nsp mailing list cisco-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

[Peter Rathlev]

Post by Saku Ytti
But of course you can use 50% too if you have 50% excess capacity to
handle the storm and you know your devices can handle it. In typical
network 1kpps of broadcast is very high number for even 10GE L2,
unless there is some special application.

We chose the 50% based on the fact that we didn't exactly know how much
broadcast/multicast was "normal"; we have since been measuring (via
SNMP/Cacti) this level and will probably adjust downwards.

You're right about the spare capacity needing to be there. We naively
concluded that a loop induced storm would tend to settle down as long as
there's some kind of limit imposed.

I guess it's time to adjust things now. :-)

Post by Saku Ytti
I think that the unicast storm-control is quite useless, unknown
unicast storm control would be much more useful.

Hm... I thought "storm-control unicast" was exactly for _unknown_
unicast. Otherwise it seems a little retarded (excuse my french).
Blocking/policing unicast as such isn't really helpful, is it?

If it really is just unicast (and not unknown unicast) I understand now
why the 3560G blocked traffic.

It would like the 6500 to support "storm-control action shutdown" by the
way. It can be EEM scripted of course, but that's a kludge.

Post by Saku Ytti
If storm happens behind many edge ports, you could get aggregated rate
multiple times higher than individual access port limit. So you might
want to limit edge port to 1 unit and core to say 5 units.

Agreed.


_______________________________________________
cisco-nsp mailing list cisco-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

[Saku Ytti]

Post by Peter Rathlev
Hm... I thought "storm-control unicast" was exactly for _unknown_
unicast. Otherwise it seems a little retarded (excuse my french).
Blocking/policing unicast as such isn't really helpful, is it?
If it really is just unicast (and not unknown unicast) I understand now
why the 3560G blocked traffic.

First CSCO box to support policing unknown unicast is EARL7.5 but it is
per chassis instead of per port. I'm not sure if any Cisco can support
per port unknown unicast policing, but if Nexus7k/EARL8 doesn't do it,
I'm betting there isn't any box which does it.
It is one of the two big WTFs I have with Cisco switches, the 2nd is
inability to limit port MAC count without also employing port-security,
which murders convergency budget.

[Lincoln Dale]

Post by Saku Ytti
First CSCO box to support policing unknown unicast is EARL7.5 but it is
per chassis instead of per port. I'm not sure if any Cisco can support
per port unknown unicast policing, but if Nexus7k/EARL8 doesn't do it,
I'm betting there isn't any box which does it.

generally speaking, cisco-nsp is not really the forum where we talk about internal details of internal implementation details of packet/frame forwarding within silicon, but...


N7K M1 (EARL8) I/O modules do not currently do unknown unicast policing (UUFB aka unknown unicast flood blocking) at this point in time in any shipping release of NX-OS.

do we plan to enable it? yes.
can we do it per port? yes
will we do so? yes.

Post by Saku Ytti
It is one of the two big WTFs I have with Cisco switches, the 2nd is
inability to limit port MAC count without also employing port-security,
which murders convergency budget.

historically, enabling port security resulted in L2 (MAC) learning in h/w being disabled on many platforms, which would make MAC learning behave like many other vendors' switches that don't do h/w L2 learning.

speaking for N7K M1 (EARL8) I/O modules, we can and still do h/w L2 learning with Port Security enabled on a port provided you use the "protect" port security method as outlined in <http://www.cisco.com/en/US/docs/switches/datacenter/sw/5_x/nx-os/security/configuration/guide/Cisco_Nexus_7000_NX-OS_Security_Configuration_Guide__Release_5.x_chapter16.html#con_1210940>.


cheers,

lincoln.


_______________________________________________
cisco-nsp mailing list cisco-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

[schilling]

We just had a incident that a person plugged two ends of a long wire
into two ports of a gig mini switch. The miniswitch then comes to our
access, distribution switch, then comes on layer 2 to our catalyst
6500. That crashed our catalyst 6500 w/ sup720 3bxl, IOS 12.2(33)SXI3.

Now we are reviewing the whole process, the access switch uplink was
only 730000 bits/sec and 8900
packets/sec. This kind of traffic was enough to crash the catalyst 6500.

During the storm, we enabled storm-control broadcast 0.01, the
catalyst 6500 was running at 99% CPU. we shutdown that port afraid of
crash it again.

We were thinking of implement storm-control broadcast 0.04 on our
WS-X6724-SFP gig ports.
Use 64 bytes packet and 576 bytes as references
64 byte
576bytes
0.04%*1000,1000,1000/(64*8) = 780 pps 87
0.08%*1000,1000,1000/(64*8) = 1562 pps 174
0.16%*1000,1000,1000/(64*8) = 3152 pps 350
0.48% 9456 pps
1051pps

Which one do you think make more sense?

Schilling

Post by Saku Ytti

Post by Peter Rathlev
We use broadcast og multicast storm-control on downlinks towards access
switches, generally at 50% just to make sure a broadcast storm doesn't
spread too much.

I would run <1% broadcast storm control, preferably entered in pps and
rather low number. On 10GE 7600 receiving <0.34% bps broadcast (L3 or L2
port, doesn't matter) it can still drop IGP on unrelated interface and
cause serious downtime.
But of course you can use 50% too if you have 50% excess capacity to handle
the storm and you know your devices can handle it. In typical network 1kpps
of broadcast is very high number for even 10GE L2, unless there is some
special application.

Post by Peter Rathlev
I'm not sure I understand unicast suppresion; it sometimes triggers even
when I would think it shouldn't. I haven't been able to grab the data

In some switches (e.g. 3550) if multicast triggers, all traffic is
suppressed, not just multicast.
Otherwise, I've not seen unicast blocked unless unicast is exceeded. I
think that the unicast storm-control is quite useless, unknown unicast
storm control would be much more useful.
I've used unicast storm-control in office LAN, to stop infected PC's from
congesting firewall. But I wouldn't run it on my customers as standard, as
our products do not dictate pps limit.

Post by Peter Rathlev
I'm not sure why one would use it on uplinks, i.e. on the access switch
side of things. AFAIK storm-control is an ingress thing.

If storm happens behind many edge ports, you could get aggregated rate
multiple times higher than individual access port limit. So you might want
to limit edge port to 1 unit and core to say 5 units.
--
 ++ytti
_______________________________________________
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

_______________________________________________
cisco-nsp mailing list cisco-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

[Saku Ytti]

Post by schilling
Now we are reviewing the whole process, the access switch uplink was
only 730000 bits/sec and 8900
packets/sec. This kind of traffic was enough to crash the catalyst 6500.

It shouldn't crash, if you meant there was software forced crash followed
by reload.
Storm-control isn't only thing you need to consider, also mls-ratelimits
and CoPP.

Post by schilling
During the storm, we enabled storm-control broadcast 0.01, the
catalyst 6500 was running at 99% CPU. we shutdown that port afraid of
crash it again.

It would be important to know what the box was receiving, to know how to
protect it.

[Christina Klam]

Thank you everyone. I will set the broadcast and multicast
storm-control to 0.35 (35%) on the interinks between the distribution
and access layer switches. At a later time, I will add the filters to
all edge ports as well.

Thank you for your help,
Christina

1. Re: Storm-Control on server switch uplinks. (Saku Ytti)
2. Re: Storm-Control on server switch uplinks. (Phil Mayers)
3. Re: Storm-Control on server switch uplinks. (Peter Rathlev)
4. 3750 stack (James Greig)
5. Re: Storm-Control on server switch uplinks. (Saku Ytti)
6. Re: Cogent IOS upgrade == BGP-3, "update malformed" (Tima Maryin)
7. Re: 3750 stack (Matt Bennett)
8. Re: Juniper M20 to Cisco 2600 Multilink Frame Relay & FRF.16
issues (Keegan Holley)

[Saku Ytti]

Post by Christina Klam
Thank you everyone. I will set the broadcast and multicast
storm-control to 0.35 (35%) on the interinks between the distribution
and access layer switches. At a later time, I will add the filters to
all edge ports as well.

0.35 in IOS config is 0.35% not 35%. 0.35 would be 3.5Mbps on 1GE, which in
turn could be 300pps to 10kpps, assuming MTU of 1500.

It is shame not all platforms allow setting the limit as pps.

[Jens S Andersen]

Hi

I just found out I can't set different levels for broadcast and multicast
storm control

I tried this on a C6503-E/Sup32/WS-X6516A running 12.2(33)SXI4a
and a C6506-E/VS-S720-10G/WS-X6724-SFP running 12.2(33)SXI3

Looks like a bug.

-Jens

Post by Christina Klam
Thank you everyone. I will set the broadcast and multicast
storm-control to 0.35 (35%) on the interinks between the distribution
and access layer switches. At a later time, I will add the filters to
all edge ports as well.
Thank you for your help,
Christina

1. Re: Storm-Control on server switch uplinks. (Saku Ytti)
2. Re: Storm-Control on server switch uplinks. (Phil Mayers)
3. Re: Storm-Control on server switch uplinks. (Peter Rathlev)
4. 3750 stack (James Greig)
5. Re: Storm-Control on server switch uplinks. (Saku Ytti)
6. Re: Cogent IOS upgrade == BGP-3, "update malformed" (Tima Maryin)
7. Re: 3750 stack (Matt Bennett)
8. Re: Juniper M20 to Cisco 2600 Multilink Frame Relay & FRF.16
issues (Keegan Holley)

Jens S Andersen Email: ***@adm.aau.dk
Aalborg University Telf: 9940 9464
Selma Lagerlöfs Vej 300, 4.2.59 Fax: [phone]
9220 Aalborg
Denmark
_______________________________________________
cisco-nsp mailing list cisco-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

[Peter Rathlev]

Post by Jens S Andersen
I just found out I can't set different levels for broadcast and multicast
storm control

Cisco hints at this in the documentation, e.g. for the "storm-control
broadcast level" command:

"Enables broadcast traffic storm control on the interface, configures
the traffic storm control level, and applies the traffic storm control
level to all traffic storm control modes enabled on the interface. "

http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SXF/native/configuration/guide/storm.html#wp1021340

Post by Jens S Andersen
Looks like a bug.

Rather than a "bug" they probably call it a "hardware limitation". :-)

[Jon Lewis]

Post by Peter Rathlev

Post by Jens S Andersen
I just found out I can't set different levels for broadcast and multicast
storm control

Cisco hints at this in the documentation, e.g. for the "storm-control
"Enables broadcast traffic storm control on the interface, configures
the traffic storm control level, and applies the traffic storm control
level to all traffic storm control modes enabled on the interface. "

Even clearer than that:

"Each port has a single traffic storm control level that is used for all
types of traffic (broadcast, multicast, and unicast).

Traffic storm control monitors the level of each traffic type for which
you enable traffic storm control in 1-second traffic storm control
intervals."

So it seems there's one storm-control threshold per interface, and you
decide which types of traffic (unicast/broadcast/multicast) have that
threshold applied.

It then gets a little murky:

"Traffic storm control on the Catalyst 6500 series switches is implemented
in hardware. The traffic storm control circuitry monitors packets passing
from a LAN interface to the switching bus. Using the Individual/Group bit
in the packet destination address, the traffic storm control circuitry
determines if the packet is unicast or broadcast, keeps track of the
current count of packets within the 1-second interval, and when a
threshold is reached, filters out subsequent packets.

Because hardware traffic storm control uses a bandwidth-based method to
measure traffic, the most significant implementation factor is setting the
percentage of total available bandwidth that can be used by controlled
traffic. Because packets do not arrive at uniform intervals, the 1-second
interval during which controlled traffic activity is measured can affect
the behavior of traffic storm control."

Here, they first say storm control keeps track of the "count of packets"
which implies to me "number of packets" or PPS, but then they say it's
bandwidth based. I think I'd actually prefer if it were simply based on
PPS or if configuring it as PPS was at least an option. We had a recent
event in which a few VMs started sending an excessive rate of both
broadcast and multicast. The traffic was arriving on 1gig interfaces on a
pair of 6509s, and at a traffic rate of about 55mbit/s, we were seeing 78k
PPS, and the 6509s were not amused. This got me looking at storm-control
again. We'd experimented with it years ago, but never fully implemented
it.

One thing I'm curious about...if I have an interface with storm-control
configured, and that interface is the source for a monitor session, during
a traffic storm, will I see the dropped packets on the monitor session
destination?

----------------------------------------------------------------------
Jon Lewis, MCP :) | I route
Senior Network Engineer | therefore you are
Atlantic Net |
_________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
_______________________________________________
cisco-nsp mailing list cisco-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

[Tim Durack]

Post by Jon Lewis
"Each port has a single traffic storm control level that is used for all
types of traffic (broadcast, multicast, and unicast).
Traffic storm control monitors the level of each traffic type for which you
enable traffic storm control in 1-second traffic storm control intervals."
So it seems there's one storm-control threshold per interface, and you
decide which types of traffic (unicast/broadcast/multicast) have that
threshold applied.
"Traffic storm control on the Catalyst 6500 series switches is implemented
in hardware. The traffic storm control circuitry monitors packets passing
from a LAN interface to the switching bus. Using the Individual/Group bit in
the packet destination address, the traffic storm control circuitry
determines if the packet is unicast or broadcast, keeps track of the current
count of packets within the 1-second interval, and when a threshold is
reached, filters out subsequent packets.
Because hardware traffic storm control uses a bandwidth-based method to
measure traffic, the most significant implementation factor is setting the
percentage of total available bandwidth that can be used by controlled
traffic. Because packets do not arrive at uniform intervals, the 1-second
interval during which controlled traffic activity is measured can affect the
behavior of traffic storm control."
Here, they first say storm control keeps track of the "count of packets"
which implies to me "number of packets" or PPS, but then they say it's
bandwidth based.  I think I'd actually prefer if it were simply based on PPS
or if configuring it as PPS was at least an option.  We had a recent event
in which a few VMs started sending an excessive rate of both broadcast and
multicast.  The traffic was arriving on 1gig interfaces on a pair of 6509s,
and at a traffic rate of about 55mbit/s, we were seeing 78k PPS, and the
6509s were not amused.  This got me looking at storm-control again.  We'd
experimented with it years ago, but never fully implemented it.

I'm glad it's not just me that thinks this way. Looking at other
vendors, most seem to imitate Cisco and do percentage based, which is
almost pointless. pps would make this a useful protection.

Interestingly NX-OS allows a decimal point:

"storm-control {broadcast | multicast | unicast} level percentage[.fraction]"

Wonder what the "technical" reason is for not allowing this to be
configured with pps as well.

[Peter Rathlev]

Post by Tim Durack
"storm-control {broadcast | multicast | unicast} level percentage[.fraction]"

So does the 6500 actually. The fraction can be specified with two
decimal digits. :-)

(It'll be many years before I'll accept that NX-OS is an improvement
over IOS!)

[Jon Lewis]

Back on the topic of storm-control, I recently deployed some new 3560G
switches and configured them with storm-control. There are some
interesting differences in storm-control on the 3560G and 6500.

The 3560G allowd me to configure storm-control using either bps or pps
rates. I chose to use pps. My only complaint so far is that on the 6500
I could "show int count storm" to see both the per-port config, and more
importantly, the per-port TotalSuppDiscards, which lets me see which ports
have been having traffic storm issues.

On the 3560G "show int count storm" isn't valid. I can "show storm" to
see the per-port config, interface filter state, and current
broadcast/multicast traffic level, but there's no history, so I won't know
if an interface dropped some packets due to storm-control.

Searching the documentation, I found

show interfaces [interface-id] counters multicast: Displays the
storm-control multicast suppression discard counter with the number of
packets discarded for all interfaces or the specified interface.

CLI online help / command completion doesn't seem to know about this, but
it does work (I think):

#sh int count ?
errors Show interface error counters
etherchannel Show interface etherchannel counters
module Limit display to interfaces on module
protocol interface protocol counters
trunk Show interface trunk counters
| Output modifiers
<cr>

#sh int count multi

Port McastSuppDiscards
Gi0/1 0
Gi0/2 0
Gi0/3 0
...

cloud-uplink-sw-1#sh int count broad

Port BcastSuppDiscards
Gi0/1 0
Gi0/2 0
Gi0/3 0
...

The switch is running the latest code (12.2(55)SE). All the counters are
0...which makes me wonder if they "work" because the servers on this
switch were generating occasional broadcast/multicast storms before the
3560G's were put in place.

----------------------------------------------------------------------
Jon Lewis, MCP :) | I route
Senior Network Engineer | therefore you are
Atlantic Net |
_________ http://www.lewis.org/~jlewis/pgp for PGP public key_________
_______________________________________________
cisco-nsp mailing list cisco-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

[Peter Rathlev]

Post by Jon Lewis
cloud-uplink-sw-1#sh int count broad
Port BcastSuppDiscards
Gi0/1 0
Gi0/2 0
Gi0/3 0
...
The switch is running the latest code (12.2(55)SE). All the counters are
0...which makes me wonder if they "work" because the servers on this
switch were generating occasional broadcast/multicast storms before the
3560G's were put in place.

FWIW we were recently lucky enough to experience a broadcast storm (no
real ill effects though), and the involved switches actually increased
their counters:

rhzzz-xfm-asw-106#sh int gi0/28 cou bro

Port BcastSuppDiscards
Gi0/28 358304
rhzzz-xfm-asw-106#

I can't assess if the numbers are totally correct, but they do increase.
This one is a WS-C3560G-24PS running 12.2(50)SE3.

Might the servers not just stay below the configured limit?

Measuring IF-MIB::ifInNUcastPkts from e.g. Cacti of course gives you the
counts even when nooen violates the threholds.