[c-nsp] Best practice, MPLS and MTU settings

[Eric A Louie]

I'm preparing to recommend how my company should engineer the MPLS connections within our backbone.  I'd like some feedback.

Right now, I have a suggested workaround that reduces the MSS on the upstream Internet interfaces, so that we keep the TCP payload at 1400.  This is due to asymmetrical traffic flows (Internet BGP preferred routes to my ISPs) and DF bits set by servers in web-land.  When one of these 1500 byte packets enters an MPLS interface, the label causes the packet to be 4 bytes too big, and the packets are dropped and the user is left with an incomplete webpage or a blank page.  I tested raising the MTU on the MPLS interfaces and found that it worked as expected.  So my permanent solution is to raise the MTU on the MPLS interfaces to some reasonable number of bytes.

I have routers that have a maximum IP MTU of 1530, and other that have maximums of 9980 and 9216.

The 1530 byte MTU devices are not in the core of the backbone and would not transport any tagged traffic.  I'm not going to create any MPLS tunnels throughout the network - all MPLS traffic will be with MPLS-capable switches point to point.


So my recommendation would be a 1600 byte MTU on MPLS interfaces.  I have packet size value information to support my recommendation.  It also seems to be the value that Cisco recommends.


How have you solved this fragmentation problem?  If you used the method of increasing the MTU, what value did you use?

Thanks, I'll summarize or share the data numbers if you are interested.
_______________________________________________
cisco-nsp mailing list cisco-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

[Nick Hilliard]

Post by Eric A Louie
If you used the method of increasing the MTU, what value did you use?

I aim towards a 9100 byte ip mtu, in order to guarantee a 9000 byte MTU for
customers with plenty of overhead on our side. Sometimes this isn't
possible due to third party provider core link restrictions.

Nick

_______________________________________________
cisco-nsp mailing list cisco-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

[Mack McBride]

I concur with 9100.
Which providers have you run into problems with 9100?
I know certain cisco gear doesn't support packets that big for certain links.
The 3750/3560 series for example only support 9000 on gigabit interfaces and much smaller on 10/100 ports.

LR Mack McBride
Network Architect

-----Original Message-----
From: cisco-nsp [mailto:cisco-nsp-***@puck.nether.net] On Behalf Of Nick Hilliard
Sent: Thursday, October 24, 2013 8:40 PM
To: Eric A Louie; Cisco NSP
Subject: Re: [c-nsp] Best practice, MPLS and MTU settings

Post by Eric A Louie
If you used the method of increasing the MTU, what value did you use?

I aim towards a 9100 byte ip mtu, in order to guarantee a 9000 byte MTU for customers with plenty of overhead on our side. Sometimes this isn't possible due to third party provider core link restrictions.

Nick

_______________________________________________
cisco-nsp mailing list cisco-***@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

_______________________________________________
cisco-nsp mailing list cisco-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

[Aaron]

I set all my mpls interfaces to....

9216 - ios xr (9k's)
9202 - ios (901's, 3600's)

Aaron

-----Original Message-----
From: cisco-nsp [mailto:cisco-nsp-***@puck.nether.net] On Behalf Of Mack
McBride
Sent: Friday, October 25, 2013 4:01 PM
To: Nick Hilliard; Eric A Louie; Cisco NSP
Subject: Re: [c-nsp] Best practice, MPLS and MTU settings

I concur with 9100.
Which providers have you run into problems with 9100?
I know certain cisco gear doesn't support packets that big for certain
links.
The 3750/3560 series for example only support 9000 on gigabit interfaces and
much smaller on 10/100 ports.

LR Mack McBride
Network Architect

-----Original Message-----
From: cisco-nsp [mailto:cisco-nsp-***@puck.nether.net] On Behalf Of Nick
Hilliard
Sent: Thursday, October 24, 2013 8:40 PM
To: Eric A Louie; Cisco NSP
Subject: Re: [c-nsp] Best practice, MPLS and MTU settings

Post by Eric A Louie
If you used the method of increasing the MTU, what value did you use?

I aim towards a 9100 byte ip mtu, in order to guarantee a 9000 byte MTU for
customers with plenty of overhead on our side. Sometimes this isn't
possible due to third party provider core link restrictions.

Nick

_______________________________________________
cisco-nsp mailing list cisco-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

_______________________________________________
cisco-nsp mailing list cisco-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

_______________________________________________
cisco-nsp mailing list cisco-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

[Mark Tinka]

Post by Aaron
I set all my mpls interfaces to....
9216 - ios xr (9k's)
9202 - ios (901's, 3600's)

Cisco's highest value is 9,216 bytes, while Juniper
generally tops out at 9,192 bytes.

We alway match them (which means we never really top out the
Cisco's) so that the IGP's (and other protocols) match
automatically without having to manually setup each
protocol, unless really necessary.

We've been in situations where we've had to modify customer-
facing protocol MTU because of "strange" hardware on their
side that can't modify MTU, and only support 1,500 bytes,
making the transfer of a full BGP table an issue (which
could be fixed by setting the MSS in BGP itself a la
Juniper, but only if that's what is facing the customer).

Mark.

[Mark Tinka]

Post by Nick Hilliard
I aim towards a 9100 byte ip mtu, in order to guarantee a
9000 byte MTU for customers with plenty of overhead on
our side. Sometimes this isn't possible due to third
party provider core link restrictions.

We are a Cisco and Juniper house.

To match things, we run 9,192 bytes on Juniper and IOS XR-
based Cisco platforms, as well as 9,178 bytes on IOS and IOS
XE-based Cisco systems.

We have noted that some Juniper SDH/SONET line cards require
you to set the MTU at the address family level (inet, inet6,
iso and mpls) for the MTU to take effect. You'd normally
apply it at the global interface level and each address
family would infer what it needs, but not on SDH/SONET line
cards. This issue does not affect Ethernet line cards.

Cisco FE ports tend to not support Jumbo frames (except
SPA's on the XR 12000 and CRS systems), but Juniper have no
issue supporting Jumbo frames on FE ports. Suffice it to
say, I haven't used FE ports anywhere for nearly 5x years
now.

Several Cisco switches will top out at 9,000 bytes. Not much
you can do about that if you have them. However, newer
platforms shipping now do up that to be inline with what we
prefer now.

Mark.

[Xu Hu]

any particular reason for this?
I meant you set 9192 & 9178? did you consider the mpls stack labeles or
Layer 2 headers? or whatever, then finally you made this recommendation
value?

Post by Mark Tinka

Post by Nick Hilliard
I aim towards a 9100 byte ip mtu, in order to guarantee a
9000 byte MTU for customers with plenty of overhead on
our side. Sometimes this isn't possible due to third
party provider core link restrictions.

We are a Cisco and Juniper house.
To match things, we run 9,192 bytes on Juniper and IOS XR-
based Cisco platforms, as well as 9,178 bytes on IOS and IOS
XE-based Cisco systems.
We have noted that some Juniper SDH/SONET line cards require
you to set the MTU at the address family level (inet, inet6,
iso and mpls) for the MTU to take effect. You'd normally
apply it at the global interface level and each address
family would infer what it needs, but not on SDH/SONET line
cards. This issue does not affect Ethernet line cards.
Cisco FE ports tend to not support Jumbo frames (except
SPA's on the XR 12000 and CRS systems), but Juniper have no
issue supporting Jumbo frames on FE ports. Suffice it to
say, I haven't used FE ports anywhere for nearly 5x years
now.
Several Cisco switches will top out at 9,000 bytes. Not much
you can do about that if you have them. However, newer
platforms shipping now do up that to be inline with what we
prefer now.
Mark.

_______________________________________________
cisco-nsp mailing list cisco-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

[Mark Tinka]

Post by Xu Hu
any particular reason for this?
I meant you set 9192 & 9178? did you consider the mpls
stack labeles or Layer 2 headers? or whatever, then
finally you made this recommendation value?

On Ethernet interfaces, IOS and IOS XE automatically account
for the 14-byte Layer 2 Ethernet overhead when you set an
MTU value.

Junos and IOS XR do not, and any MTU value defined is
considered, by the router or switch, to be the "true" MTU of
the interface.

So whatever you set on Junos and IOS XR system, subtract 14
bytes (for Ethernet) for IOS and IOS XE systems and apply
that.

Also, at this MTU level, you should have no issues
supporting various MPLS application, be it simple IP
encapsulation, or a label stack that signals advanced
services like MPLS-based VPNS, BGP-MVPN's, MPLS-TE, e.t.c.

If your customers were requesting for Jumbo frames with
their (on-net) service, they should be fine too. We normally
guarantee 9,000 bytes to customers using this for on-net
traffic, and offer no guarantees for anything higher (even
if, technically, we can support it edge-to-edge).

Cheers,

Mark.

[Adam Vitkovsky]

The higher you can get the better.

You don't want to restrict your MPLS pipes unnecessarily.
In case your platform counts L2 overhead into MTU leave some space for
future L2 overheads (802.1ah) / label stacks (Segment Routing) and leave the
rest for your customers.

See in case you'll provide carrier-ethernet services for Nick or myself we
would require MTU 9100 so that we can still offer MTU 9000 to our customers
in that particular region.


adam

_______________________________________________
cisco-nsp mailing list cisco-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/