[c-nsp] MPLS VPN with PE over GRE tunnels

[Ross Halliday]

Hi list,

This one just popped up recently. I'm wondering if I'm missing some sort of platform caveat as I can't find a routing issue.

We have a customer with multi-site IP VPN. The whole thing previously was done in VRF-lite-style with VRFs, VLANs, and separate OSPF. We've since start cutting over to MPLS land. Since then there has been a problem with one remote site that connects over GRE tunnels.

Currently our network has one switch that is at the hub of our transition to MPLS as we cut various devices over and wait for maintenance windows. It has:

- Sites connected via MPLS
- Sites connected directly
- Sites connected via OSPF to an un-MPLSed box
- Sites connected via GRE tunnels

I've tried every combination I can think of, but any MPLS-transported site to the GRE-tunneled sites don't pass traffic in one direction. I started out with OSPF running over GRE tunnels from the PE router to the CE routers and switched to BGP with no dice. Route tables on every device are correct.

After trying to pass pings between loopback interfaces I have found:

Packets from MPLS site to GRE site work
Packets from GRE site to MPLS site do NOT work
interface Loopback2210
description Customer VPN
ip vrf forwarding vpn-customer
ip address 10.17.10.17 255.255.255.255
!
interface Tunnel23
description Data VLAN 573
ip vrf forwarding vpn-customer
ip address 10.17.10.64 255.255.255.254
ip tcp adjust-mss 1400
tunnel source 10.10.3.5
tunnel destination 10.10.3.6
tunnel path-mtu-discovery
tunnel vrf vpn-customer-dsl
service-policy output Data-Tunnel-3meg
!
interface Tunnel24
description Data VLAN 574
ip vrf forwarding vpn-customer
ip address 10.17.10.66 255.255.255.254
ip tcp adjust-mss 1400
tunnel source 10.10.4.5
tunnel destination 10.10.4.6
tunnel path-mtu-discovery
tunnel vrf vpn-customer-dsl
service-policy output Data-Tunnel-3meg
!
router ospf 2210 vrf vpn-customer
router-id 10.17.10.17
log-adjacency-changes
area 2210 authentication message-digest
redistribute connected subnets
redistribute static subnets
redistribute bgp 64723 metric 20 subnets
network 10.17.10.0 0.0.0.15 area 2210
network 10.17.10.64 0.0.0.1 area 2210
network 10.17.10.66 0.0.0.1 area 2210
network 10.17.10.68 0.0.0.1 area 2210
network 10.17.10.70 0.0.0.1 area 2210
network 10.17.10.72 0.0.0.1 area 2210
!
!
router bgp 64723
!
address-family ipv4 vrf vpn-customer
redistribute connected
redistribute ospf 2210 vrf vpn-customer metric 20 match internal external 1 external 2
neighbor 10.17.10.65 remote-as 65171
neighbor 10.17.10.65 activate
neighbor 10.17.10.65 as-override
neighbor 10.17.10.67 remote-as 65171
neighbor 10.17.10.67 activate
neighbor 10.17.10.67 as-override
maximum-paths 4
default-information originate
no synchronization
exit-address-family


core-kgtn-c6509#sh ip route vrf vpn-customer

Routing Table: vpn-customer
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route

Gateway of last resort is 10.17.10.1 to network 0.0.0.0

172.19.0.0/27 is subnetted, 2 subnets
O E2 172.19.10.32 [110/1] via 10.17.10.1, 00:05:15, Vlan2210
O E2 172.19.10.0 [110/1] via 10.17.10.1, 00:05:15, Vlan2210
10.0.0.0/8 is variably subnetted, 19 subnets, 5 masks
C 10.17.10.17/32 is directly connected, Loopback2210
O E2 10.17.10.16/32 [110/20] via 10.17.10.1, 00:05:15, Vlan2210
O E2 10.17.10.19/32 [110/20] via 10.17.10.4, 00:05:15, Vlan2210
B 10.17.10.18/32 [200/0] via 172.18.3.240, 01:32:11 <-------- MPLS-connected site
B 10.17.10.21/32 [20/0] via 10.17.10.67, 00:06:47 <-------- GRE site
[20/0] via 10.17.10.65, 00:05:20 <-------- GRE site
O E2 10.17.10.24/32 [110/20] via 10.17.10.73, 00:05:15, Tunnel27 <-------- GRE site
[110/20] via 10.17.10.71, 00:05:15, Tunnel26 <-------- GRE site
[110/20] via 10.17.10.69, 00:05:15, Tunnel25 <-------- GRE site
C 10.17.10.0/28 is directly connected, Vlan2210
O E2 10.17.10.48/30 [110/20] via 10.17.10.73, 00:05:15, Tunnel27 <-------- GRE site
[110/20] via 10.17.10.71, 00:05:15, Tunnel26 <-------- GRE site
[110/20] via 10.17.10.69, 00:05:15, Tunnel25 <-------- GRE site
B 10.17.10.52/30 [20/0] via 10.17.10.67, 00:06:47 <-------- GRE site
[20/0] via 10.17.10.65, 00:05:20 <-------- GRE site
O E2 10.17.10.56/29 [110/20] via 10.17.10.1, 00:05:15, Vlan2210
O E2 10.17.10.32/30 [110/20] via 10.17.10.4, 00:05:15, Vlan2210
B 10.17.10.36/30 [200/0] via 172.18.3.240, 01:32:11 <-------- MPLS-connected site
C 10.17.10.40/30 is directly connected, Vlan2213
C 10.17.10.44/30 is directly connected, Vlan2214
C 10.17.10.68/31 is directly connected, Tunnel25 <-------- GRE site
C 10.17.10.70/31 is directly connected, Tunnel26 <-------- GRE site
C 10.17.10.64/31 is directly connected, Tunnel23 <-------- GRE site
C 10.17.10.66/31 is directly connected, Tunnel24 <-------- GRE site
C 10.17.10.72/31 is directly connected, Tunnel27 <-------- GRE site
O*E2 0.0.0.0/0 [110/1] via 10.17.10.1, 00:05:15, Vlan2210
interface Loopback1
ip vrf forwarding vpn-uscu
ip address 10.17.10.21 255.255.255.255
!
interface Tunnel23
description Data VLAN 573
ip vrf forwarding vpn-customer
ip address 10.17.10.65 255.255.255.254
ip virtual-reassembly
ip tcp adjust-mss 1400
tunnel source 10.10.3.6
tunnel destination 10.10.3.5
tunnel path-mtu-discovery
tunnel vrf DoNotRoute
service-policy output Data-Tunnel
!
interface Tunnel24
description Data VLAN 574
ip vrf forwarding vpn-customer
ip address 10.17.10.67 255.255.255.254
ip virtual-reassembly
ip tcp adjust-mss 1400
tunnel source 10.10.4.6
tunnel destination 10.10.4.5
tunnel path-mtu-discovery
tunnel vrf DoNotRoute
service-policy output Data-Tunnel
!
router bgp 65171
no synchronization
bgp router-id 10.17.10.21
bgp log-neighbor-changes
no auto-summary
!
address-family ipv4 vrf vpn-uscu
redistribute connected
neighbor 10.17.10.64 remote-as 64723
neighbor 10.17.10.64 activate
neighbor 10.17.10.66 remote-as 64723
neighbor 10.17.10.66 activate
maximum-paths 2
no synchronization
exit-address-family
!


ce-CUSTOMER-vd-c1811#sh ip route vrf vpn-customer

Routing Table: vpn-customer
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route

Gateway of last resort is 10.17.10.66 to network 0.0.0.0

172.19.0.0/27 is subnetted, 2 subnets
B 172.19.10.32 [20/20] via 10.17.10.66, 00:00:01
[20/20] via 10.17.10.64, 00:00:44
B 172.19.10.0 [20/20] via 10.17.10.66, 00:00:01
[20/20] via 10.17.10.64, 00:00:44
10.0.0.0/8 is variably subnetted, 19 subnets, 5 masks
B 10.17.10.17/32 [20/0] via 10.17.10.66, 00:00:01
[20/0] via 10.17.10.64, 00:00:44
B 10.17.10.16/32 [20/20] via 10.17.10.66, 00:00:01
[20/20] via 10.17.10.64, 00:00:44
B 10.17.10.19/32 [20/20] via 10.17.10.66, 00:00:01
[20/20] via 10.17.10.64, 00:00:44
B 10.17.10.18/32 [20/0] via 10.17.10.66, 00:00:01 <-------- MPLS-connected site
[20/0] via 10.17.10.64, 00:00:44 <-------- MPLS-connected site
C 10.17.10.21/32 is directly connected, Loopback1
B 10.17.10.24/32 [20/20] via 10.17.10.66, 00:00:01
[20/20] via 10.17.10.64, 00:00:44
B 10.17.10.0/28 [20/0] via 10.17.10.66, 00:00:01
[20/0] via 10.17.10.64, 00:00:44
B 10.17.10.48/30 [20/20] via 10.17.10.66, 00:00:01
[20/20] via 10.17.10.64, 00:00:44
C 10.17.10.52/30 is directly connected, Vlan2216
B 10.17.10.56/29 [20/20] via 10.17.10.66, 00:00:01
[20/20] via 10.17.10.64, 00:00:44
B 10.17.10.32/30 [20/20] via 10.17.10.66, 00:00:01
[20/20] via 10.17.10.64, 00:00:44
B 10.17.10.36/30 [20/0] via 10.17.10.66, 00:00:01 <-------- MPLS-connected site
[20/0] via 10.17.10.64, 00:00:44 <-------- MPLS-connected site
B 10.17.10.40/30 [20/0] via 10.17.10.66, 00:00:01
[20/0] via 10.17.10.64, 00:00:44
B 10.17.10.44/30 [20/0] via 10.17.10.66, 00:00:01
[20/0] via 10.17.10.64, 00:00:44
C 10.17.10.64/31 is directly connected, Tunnel23
C 10.17.10.66/31 is directly connected, Tunnel24
B 10.17.10.68/31 [20/0] via 10.17.10.66, 00:00:01
[20/0] via 10.17.10.64, 00:00:44
B 10.17.10.70/31 [20/0] via 10.17.10.66, 00:00:01
[20/0] via 10.17.10.64, 00:00:44
B 10.17.10.72/31 [20/0] via 10.17.10.66, 00:00:01
[20/0] via 10.17.10.64, 00:00:44
B* 0.0.0.0/0 [20/20] via 10.17.10.66, 00:00:01
[20/20] via 10.17.10.64, 00:00:44


I've tried IPIP tunnels but they don't pass any traffic at all. Tomorrow I'll be ripping out the GRE config from the 6509 and pushing them to an outboard 1811 as it's the GRE tunnels that seem to stick out in my mind.

Any help or suggestions would be very appreciated!

Thanks
Ross Halliday


_______________________________________________
cisco-nsp mailing list cisco-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

[Gert Doering]

Hi,

Post by Ross Halliday
Currently our network has one switch that is at the hub of our transition to

This "switch" would be a 6500 with all these protocols being enabled, and
the problem spot is "packet comes in MPLS-encapsulated and needs to leave
GRE-encapsulated" (or return path)?

Post by Ross Halliday
Any help or suggestions would be very appreciated!

There was something about the 6500 architecture that certain combinations
of ingress and egress need packets to go through the forwarding plane twice,
and you need to enable "packet recirculation" for it to do that.

The command I could find for that is "mls mpls tunnel-recir", but you
might want to verify with the docs whether this is what you want.

Cisco(config)#mls mpls ?
...
tunnel-recir Recirculate Tunnel packets

gert

[Christophe Fillot]

Gert Doering wrote:

Hello,

Post by Gert Doering
There was something about the 6500 architecture that certain combinations
of ingress and egress need packets to go through the forwarding plane twice,
and you need to enable "packet recirculation" for it to do that.

http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SXF/native/configuration/guide/pfc3mpls.html

Especially:

In certain cases, the PFC3BXL or PFC3B provides the capability to
recirculate the packets. Recirculation can be used to perform additional
lookups in the ACL or QoS TCAMs, the NetFlow table, or the FIB TCAM
table. Recirculation is necessary in these situations:

- To push more than three labels on imposition

- To pop more than two labels on disposition

- To pop an explicit null top label

- When the VPN Routing and Forwarding (VRF) number is more than 511

- For IP ACL on the egress interface (for nonaggregate (per-prefix)
labels only)

Packet recirculation occurs only on a particular packet flow; other
packet flows are not affected.The rewrite of the packet occurs on the
modules; the packets are then forwarded back to the PFC3BXL or PFC3B for
additional processing.


_______________________________________________
cisco-nsp mailing list cisco-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

[Arie Vayner (avayner)]

On 6500 if you want to use MPLS over GRE, you would need to have your
core facing links (through which the GRE packets are sent/received) on a
SIP400 card...

Alternatively, SUP2T can support this natively.
http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/white_p
aper_c11-652042.html#wp9000959


Arie

-----Original Message-----
From: cisco-nsp-***@puck.nether.net
[mailto:cisco-nsp-***@puck.nether.net] On Behalf Of Gert Doering
Sent: Tuesday, September 20, 2011 10:55
To: Ross Halliday
Cc: cisco-***@puck.nether.net
Subject: Re: [c-nsp] MPLS VPN with PE over GRE tunnels

Hi,

Post by Ross Halliday
Currently our network has one switch that is at the hub of our
transition to MPLS as we cut various devices over and wait for

maintenance windows. It has:

This "switch" would be a 6500 with all these protocols being enabled,
and the problem spot is "packet comes in MPLS-encapsulated and needs to
leave GRE-encapsulated" (or return path)?

Post by Ross Halliday
Any help or suggestions would be very appreciated!

There was something about the 6500 architecture that certain
combinations of ingress and egress need packets to go through the
forwarding plane twice, and you need to enable "packet recirculation"
for it to do that.

The command I could find for that is "mls mpls tunnel-recir", but you
might want to verify with the docs whether this is what you want.

Cisco(config)#mls mpls ?
...
tunnel-recir Recirculate Tunnel packets

gert

--
USENET is *not* the non-clickable part of WWW!

//www.muc.de/~gert/
Gert Doering - Munich, Germany
***@greenie.muc.de
fax: [phone]
***@net.informatik.tu-muenchen.de

_______________________________________________
cisco-nsp mailing list cisco-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

[Arie Vayner (avayner)]

Sorry for double posting... This seems to be a good reference:
http://www.cisco.com/en/US/prod/collateral/routers/ps9343/Deploying_and_
Configuring_MPLS_Virtual_Private_Networks_In_IP_Tunnel_Environment.pdf

Arie

-----Original Message-----
From: Arie Vayner (avayner)
Sent: Tuesday, September 20, 2011 17:18
To: 'Gert Doering'; Ross Halliday
Cc: cisco-***@puck.nether.net
Subject: RE: [c-nsp] MPLS VPN with PE over GRE tunnels

On 6500 if you want to use MPLS over GRE, you would need to have your
core facing links (through which the GRE packets are sent/received) on a
SIP400 card...

Alternatively, SUP2T can support this natively.
http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/white_p
aper_c11-652042.html#wp9000959


Arie

-----Original Message-----
From: cisco-nsp-***@puck.nether.net
[mailto:cisco-nsp-***@puck.nether.net] On Behalf Of Gert Doering
Sent: Tuesday, September 20, 2011 10:55
To: Ross Halliday
Cc: cisco-***@puck.nether.net
Subject: Re: [c-nsp] MPLS VPN with PE over GRE tunnels

Hi,

Post by Ross Halliday
Currently our network has one switch that is at the hub of our
transition to MPLS as we cut various devices over and wait for

maintenance windows. It has:

This "switch" would be a 6500 with all these protocols being enabled,
and the problem spot is "packet comes in MPLS-encapsulated and needs to
leave GRE-encapsulated" (or return path)?

Post by Ross Halliday
Any help or suggestions would be very appreciated!

There was something about the 6500 architecture that certain
combinations of ingress and egress need packets to go through the
forwarding plane twice, and you need to enable "packet recirculation"
for it to do that.

The command I could find for that is "mls mpls tunnel-recir", but you
might want to verify with the docs whether this is what you want.

Cisco(config)#mls mpls ?
...
tunnel-recir Recirculate Tunnel packets

gert

--
USENET is *not* the non-clickable part of WWW!

//www.muc.de/~gert/
Gert Doering - Munich, Germany
***@greenie.muc.de
fax: [phone]
***@net.informatik.tu-muenchen.de

_______________________________________________
cisco-nsp mailing list cisco-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

[Ross Halliday]

It seems I made an error in the subject of my message, should read "MPLS VPN with CE over GRE tunnels"... Looks like a few people didn't read far beyond the subject line :P

Thank you Gert and Cristophe, I will give that a test tonight. Does the same sort of gotcha exist on the 7200 platform? I moved the interfaces over to that router, which also runs MPLS, and before I corrected the VPNv4 iBGP relationships the traffic worked fine when the 7204 sent packets out labeled for that default route (which caused them to be sent back via OSPF into an SVI). Once I fixed the BGP peering so that the 7204 learned the far VPNv4 route properly it exhibited the same problem as the 6509. The 7204 is a dinky ol' NPE-225 running 12.4(22)T.

Reading that page that Cristophe linked, I'm curious why this isn't default behavior. Is it just some magic knob to stump people on a CCIE exam or is there some performance impact or other side effects?


Thanks
Ross Halliday

Post by Arie Vayner (avayner)
-----Original Message-----
Sent: Tuesday, September 20, 2011 10:19 AM
To: Gert Doering; Ross Halliday
Subject: RE: [c-nsp] MPLS VPN with PE over GRE tunnels
http://www.cisco.com/en/US/prod/collateral/routers/ps9343/Deploying_and
_
Configuring_MPLS_Virtual_Private_Networks_In_IP_Tunnel_Environment.pdf
Arie
-----Original Message-----
From: Arie Vayner (avayner)
Sent: Tuesday, September 20, 2011 17:18
To: 'Gert Doering'; Ross Halliday
Subject: RE: [c-nsp] MPLS VPN with PE over GRE tunnels
On 6500 if you want to use MPLS over GRE, you would need to have your
core facing links (through which the GRE packets are sent/received) on a
SIP400 card...
Alternatively, SUP2T can support this natively.
http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/white_
p
aper_c11-652042.html#wp9000959
Arie
-----Original Message-----
Sent: Tuesday, September 20, 2011 10:55
To: Ross Halliday
Subject: Re: [c-nsp] MPLS VPN with PE over GRE tunnels
Hi,

Post by Ross Halliday
Currently our network has one switch that is at the hub of our
transition to MPLS as we cut various devices over and wait for

This "switch" would be a 6500 with all these protocols being enabled,
and the problem spot is "packet comes in MPLS-encapsulated and needs to
leave GRE-encapsulated" (or return path)?

Post by Ross Halliday
Any help or suggestions would be very appreciated!

There was something about the 6500 architecture that certain
combinations of ingress and egress need packets to go through the
forwarding plane twice, and you need to enable "packet recirculation"
for it to do that.
The command I could find for that is "mls mpls tunnel-recir", but you
might want to verify with the docs whether this is what you want.
Cisco(config)#mls mpls ?
...
tunnel-recir Recirculate Tunnel packets
gert
--
USENET is *not* the non-clickable part of WWW!
//www.muc.de/~gert/
Gert Doering - Munich, Germany
fax: [phone]

_______________________________________________
cisco-nsp mailing list cisco-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

[Bruce Pinsky]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by Ross Halliday
It seems I made an error in the subject of my message, should read "MPLS
VPN with CE over GRE tunnels"... Looks like a few people didn't read far
beyond the subject line :P
Thank you Gert and Cristophe, I will give that a test tonight. Does the
same sort of gotcha exist on the 7200 platform? I moved the interfaces
over to that router, which also runs MPLS, and before I corrected the
VPNv4 iBGP relationships the traffic worked fine when the 7204 sent
packets out labeled for that default route (which caused them to be sent
back via OSPF into an SVI). Once I fixed the BGP peering so that the
7204 learned the far VPNv4 route properly it exhibited the same problem
as the 6509. The 7204 is a dinky ol' NPE-225 running 12.4(22)T.
Reading that page that Cristophe linked, I'm curious why this isn't
default behavior. Is it just some magic knob to stump people on a CCIE
exam or is there some performance impact or other side effects?

No, the recirculation issue is only related to the 6500. It has to do with
what operations can be performed in the hardware in a single cycle. The
7200 is a software switching platform and would not have such a limitation.
If you are having a problem on the 7200, there is something else going on
here. We terminate DMVPN in VRFs for L3VPN on 7200's all the time without
issue.

Some additional troubleshooting info would be useful such as traceroutes in
both directions, ping tests from the CE site to the PE itself, etc to see
where the transit path is breaking down. Also, complete configs would be
useful to see if anything jumps out.

- --
=========
bep

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk54w0cACgkQE1XcgMgrtyYubQCg93E8VwIUKVuy6+CDg/5AHqxq
bY8AnAxA0DZ951Nju4LkJD78h6QxiH18
=yrvU
-----END PGP SIGNATURE-----
_______________________________________________
cisco-nsp mailing list cisco-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

[Gert Doering]

Hi,

Post by Ross Halliday
Thank you Gert and Cristophe, I will give that a test tonight. Does the
same sort of gotcha exist on the 7200 platform?

As it doesn't have hardware forwarding to circulate packets through, no.

Post by Ross Halliday
I moved the interfaces over to that router, which also runs MPLS,
and before I corrected the VPNv4 iBGP relationships the traffic
worked fine when the 7204 sent packets out labeled for that default
route (which caused them to be sent back via OSPF into an SVI).
Once I fixed the BGP peering so that the 7204 learned the far VPNv4
route properly it exhibited the same problem as the 6509. The 7204
is a dinky ol' NPE-225 running 12.4(22)T.

In that case, it would be "something else", though I have no idea what
it could be.

Post by Ross Halliday
Reading that page that Cristophe linked, I'm curious why this
isn't default behavior. Is it just some magic knob to stump people
on a CCIE exam or is there some performance impact or other side
effects?

recirculation reduces forwarding performance, as (some) packets are
using twice as many cycles...

gert