[c-nsp] Problem encountered while securing NTP

[Justin Shore]

Given the recent NTP PSIRT from Cisco (cisco-sa-20090923-ntp) I decided
to spend the morning revisiting my NTP practices to lessen the chance of
getting kicked in the teeth by this router-crashing bug later on. In my
networks I usually have a pair (or more sometimes) of border routers as
local NTP servers, advertising themselves internally as stratum-3. They
use stratum-1s on the Internet in geographically diverse areas. The
local network devices are configured to use both of the local NTP
servers with authentication.

Here's a border router's config:

ntp authentication-key 1 md5 BLAHBLAHBLAH 7
ntp authenticate
ntp trusted-key 1
ntp source Loopback1
ntp access-group peer 5
ntp access-group serve-only 6
ntp master 3
ntp update-calendar
ntp server a.a.a.a prefer
ntp server b.b.b.b
ntp peer LOCAL.BORDER.RTR.TWO key 1
ntp server c.c.c.c
!
access-list 5 remark NTP Peers
access-list 5 permit 127.127.7.1
access-list 5 permit a.a.a.a.
access-list 5 permit b.b.b.b
access-list 5 permit c.c.c.c
access-list 5 permit LOCAL.BORDER.RTR.ONE
access-list 5 permit LOCAL.BORDER.RTR.TWO
access-list 5 deny any log
!
access-list 6 remark NTP Serve-Only
access-list 6 permit LOCAL.NETWORK.MANAGEMENT.PREFIXES x.x.x.mask
access-list 6 permit LOCAL.RIR.ASSIGNED.PREFIXES x.x.x.mask
access-list 6 deny any log

Here's a client device's config:

ntp authentication-key 1 md5 105D020D0619061B4851 7
ntp authenticate
ntp trusted-key 1
ntp clock-period 17179513
ntp source Vlan224
ntp access-group serve-only 6
ntp server 64.71.98.51 key 1
ntp server 64.71.98.59 key 1
!
access-list 6 remark NTP Serve-Only
access-list 6 permit LOCAL.NETWORK.MANAGEMENT.PREFIXES x.x.x.mask
access-list 6 permit LOCAL.RIR.ASSIGNED.PREFIXES x.x.x.mask
access-list 6 deny any log
(Same as above for simplicity's sake)

That's my standard NTP config that I run on several networks without any
real problems, normally. There may be other or better ways to help
secure NTP that I either haven't implemented yet (iACLs, CoPP) or don't
know about but I'm always open to suggestions.

The problem I'm running into today is that the 'access-group peer'
statements on the NTP servers are matching local clients with ACL 6 as
well as configured stratum-1 peers (successfully matching the peers at
that). The local clients should be matched with the 'access-group
serve-only' ACL 6, but for some reason they are not.

Strangely enough I have ACE counters increasing on ACL 6's lines that
correspond to my network infrastructure, even though the 'deny any log'
ACE in ACL 5 is also reporting denied hits for the very same subnets.
It's as if I've freaked out NTP. I made changes to the ACL earlier
today as part of my initial review. Then NTP for the network
infrastructure stopped working. Could this be an IOS bug by chance? My
next step is to blow away the NTP config from the borders, hopefully
killing the NTP process at the same time, and re-add the config.

Thoughts before I blow away the config and start over?
Justin

_______________________________________________
cisco-nsp mailing list cisco-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

[Kevin Graham]

The problem I'm running into today is that the 'access-group peer' statements on
the NTP servers are matching local clients with ACL 6 as well as configured
stratum-1 peers (successfully matching the peers at that). The local clients
should be matched with the 'access-group serve-only' ACL 6, but for some reason
they are not.

CSCsw79186. Its broken more than the bug suggests; both v3 and v4 clients are
get applied only to the 'peer' access-group. I had meant to bring this to
PSIRT's attention when the advisory went out, but got distracted by something
shiny.
_______________________________________________
cisco-nsp mailing list cisco-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

[Justin Shore]

Post by Kevin Graham
CSCsw79186. Its broken more than the bug suggests; both v3 and v4 clients are
get applied only to the 'peer' access-group. I had meant to bring this to
PSIRT's attention when the advisory went out, but got distracted by something
shiny.

Excellent catch. I tried to search the BugToolkit today for anything
related to NTP and couldn't get it to work. I rebooted the router
tonight and bumped the rev to 24T1 in hopes that it would fix the issue.
It didn't. Clearly this problem isn't fixed as the BugToolkit
indicates since there isn't a T-train release with the alleged fix in
it. I'll hammer on them later today about this. I don't think that the
severity of the problem is moderate as the bug notes indicate. For me
it's severe since it's affecting NTP from our VoIP phones and soft
switch. I think the PSIRT folks would also disagree with a failure of
NTP being only a moderate issue too since logging with accurate
timestamps is essential to any security model.

Justin

_______________________________________________
cisco-nsp mailing list cisco-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

[Paul Oxman (poxman)]

Hello,
Cisco PSIRT will be amending the NTP advisory to reflect the
encountered bug, update should be available in about 24 hours.

CSCsw79186 will be available in 12.4(24)T2, which according to
our knowledge, should be available for download on Cisco.Com
mid-October/2009. It is also currently fixed in 15.0(1)M.

Regards

Name: Paul Oxman
Phone: [phone]
Mobile: [phone]
Title: PSIRT Incident Manager
PGP Key: 0x6EA839A6

Have you seen the Security Intelligence Operations Portal
http://www.cisco.com/security

Have you seen the new Cisco Security Blog:
http://blogs.cisco.com/security
_______________________________________________
cisco-nsp mailing list cisco-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

[Jeff Kell]

This post might be inappropriate. Click to display it.

[Phil Mayers]

Post by Jeff Kell
While we're on the subject, I came in this morning to find our core 6500
out of NTP sync. Checking the associations, a local host was in the
list as a "dynamic" association, with an invalid time.

Yeah, we've seen that. You need an ACL. This is dumb. To anyone from
cisco reading - this is dumb, fix it.

Post by Jeff Kell
I was under the (apparently incorrect) assumption that IOS would not
accept unsolicited/unconfigured NTP control requests from anyone... as I
haven't revisited my NTP configuration in years.
The IOS in question (12.2(33)SXI2) does not have a "ntp broadcast
client" option I can simply turn off, as the generic NTP configuration
suggests.
The access-group documentation is a bit confusing...
I'd like to have control requests restricted to my configured 'ntp
server' list, but allow queries from anyone, and certainly not accept
NTP updates from unsolicited sources.
Does anyone have a nice, canned NTP config to accomplish this goal they
would care to share?

Ours does not do what you need; it restricts NTP querying to the NTP
servers & our management network (nagios checks the routers are in-sync
with an NTP query packet).

Give the caveat mentioned earlier in the thread (CSCsw79186) it's not
clear to me that you *can* do what you want.

Any reason you can't just run an NTP server?
_______________________________________________
cisco-nsp mailing list cisco-***@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/