Cisco's Control Plane Policing (CoPP) Implementation Notes


March 15, 2005

Edited by John Kristoff

Table of Contents
Introduction
Getting Started
Classes and Policies
Access Lists
  1. Critical
  2. Important
  3. Normal
  4. Undesirable
  5. Default
Enable CoPP
Troubleshooting
Submitted Feedback
References
Acknowledgments
Revision History

Introduction

This document was written in order to share some implementation experience and configuration examples with Cisco's new Control Plane Policing (CoPP) feature. Since this is relatively new stuff and based on what little feedback I'd gotten when I asked about it on the cisco-nsp mailing list in December 2004 and in other forums, I suspect it hasn't yet gotten much if any deployment in production networks. So in in the hope that others will find it useful as I might have a few months ago, this report was written up detailing some example configurations for CoPP.

CoPP was recently added to the select IOS train for specific hardware platforms. If you are familiar with Juniper routers, you may know that they allow you to filter packets that are destined to the box (via any interface) through the use of firewall filters on the loopback interface. CoPP is really just a way of doing the same kind of thing on Cisco routers. In my opinion the Cisco implementation is not quite as elegant as Juniper's, but we're not here to complain, rather this comment is just to give you some perspective if you are already familiar with the other vendor.

One caveat, I'm not recommending people go deploy CoPP. It may be worth considering, but it is sufficiently new, complicated and limited that it may be unnecessary and risky in your environment. If you are considering using CoPP, hopefully these notes will help you make an informed decision not only on whether you should proceed, but also on how to proceed.

This report is based on experience with a 6509/720b running native IOS 12.2(18)SXD2.

Specific configuration examples will be documented in-line using HTML's preformatted (<pre>) tags. Each configuration section will contain a brief description. In-line comments are included where they are most appropriate. Options that are site specific will be noted in brackets and italicized.

The best place to submit questions or feedback regarding any general configuration information in this text is to the cisco-nsp mailing list. For document editing or style comments, feel free to contact the document editor directly.

Getting Started

Cisco's Deploying Control Plane Policing White Paper is a relatively good way to get started. I used that as my guide. However, as I recall you almost assuredly cannot use the examples it provides in your actual configs. For instance, Rodney Dunn pointed out to me on the mailing list that the default class is not currently supported. CoPP is likely to be enhanced quickly over the next few IOS releases, check your software train release notes for any changes that may render this document obsolete or incorrect.

Classes and Policies

The simplest use of CoPP would be to simply discard or rate limit away specific traffic you do not want to see. We will use a more complex set of rules in order to explore CoPP as described by Cisco more fully. Here we will use five unique classes of traffic that the router may receive. They are named (our names, you can name them anything you want of course), in descending order of importance cp-critical-in, cp-important-in, cp-normal-in, cp-default-in and cp-undesirable-in. You use Cisco's class-map notation to define these classes and in each you can apply a standard or extended IP access list, which associates packets with the class. For all classes except the default, the access list is named the same as the class in our example. We will define the access lists later.

class-map match-all cp-critical-in
  description Control plane critcal traffic
  match access-group name cp-critical-in
class-map match-all cp-important-in
  description Control plane important traffic
  match access-group name cp-important-in
class-map match-all cp-normal-in
  description Control plane normal traffic
  match access-group name cp-normal-in
class-map match-any cp-undesirable-in
  description Control plane undesirable traffic
  match access-group name cp-undesirable-in
class-map match-all cp-default-in
  description Control plane default traffic
  match access-group 2

Then we associate any needed policies with these class-maps. In this case we use rate limiters to restrict how much traffic the CPU will see for each class of traffic. The cp-critical-in class does not have a rate limit defined, which means there is none. Also note there is an implied default-class, which is used for things like ARP, which cannot be changed and thus cannot be rate limited using CoPP. The values for these rate limits are site specific, but these have been found to be pretty good generally. However, if you have a lot of Microsoft Windows hosts, or other types of hosts that generate a lot of IP broadcast traffic, the default rate may easily fill and you may wish to raise that limit higher. Please note that the undesirable class has both a drop action for conforming and non-conforming traffic. This is effectively a complete deny filter for that category of traffic.

policy-map control-plane-in
  class cp-critical-in
  class cp-important-in
     police 128000 24000 48000 conform-action transmit exceed-action drop
  class cp-normal-in
     police 32000 6000 12000 conform-action transmit exceed-action drop
  class cp-undesirable-in
     police 32000 1000 1000 conform-action drop exceed-action drop
  class cp-default-in
     police 32000 6000 12000 conform-action transmit exceed-action drop

Access Lists

An ACL permit statement causes the associated policy to be applied to the packet (it will be policed), whereas a deny statement will ignore the packet for the associated policy (but it should be evaluated elsewhere). These are example access lists and are by no means meant to be ideal for the packet types listed or for general networking environments. Simpler, relaxed rules may make things more manageable while sacrificing some potential for unwanted packets hitting the CPU. More complex rules may reduce the noise seen by the CPU, but at the expense of making the filters more cumbersome to manage, especially if router configurations change frequently. Hopefully these rules show both some complex and simple examples that help you figure out how to do your own CoPP implementation.

Critical

The following access list is used to categorize packets that are critical to the operation of the router. These packets will not be policed at all as defined by the class and policy above. The OSPF rules are specific to the neighboring IP address adjacencies. As are the PIM addresses. However, if this router is a PIM RP, you may find yourself creating a large list of downstream PIM routers that send you register packets. You may wish to simplify that rule if so, otherwise you'll be creating a lot of specific rules if you have a lot of PIM speaking routers. IGMP, a multicast control protocol can be seen on any usable multicast group address so here we allow the entire group address space. If the router is a DHCP relay, you allow broadcast packets from clients, but also responses back from the DHCP server to the router's BOOTPS (67) port. Finally, BGP sessions are protected regardless of which side is doing the active open.

ip access-list extended cp-critical-in
 remark Control plane critical traffic - inbound
 remark OSPF
 permit ospf host [OSPF neighbor A addr] any
 permit ospf host [OSPF neighbor B addr] any
 remark PIM
 permit pim host [PIM neighbor A addr] any
 permit pim host [PIM neighbor B addr] any
 permit pim host [PIM RP A addr] any
 permit pim host [PIM RP B addr] any
 remark IGMP
 permit igmp any 224.0.0.0 15.255.255.255
 remark DHCP
 permit udp host 0.0.0.0 host 255.255.255.255 eq bootps
 permit udp host [DHCP server addr] eq bootps any eq bootps
 remark BGP
 permit tcp host [BGP neighbor addr] eq bgp host [local BGP addr]
 permit tcp host [BGP neighbor addr] host [local BGP addr] eq bgp
 deny   ip any any

Important

The following access list is used to categorize packets that are important to the operation of the router, but if for some reason they weren't being processed, the network could still be operational. The rules here are mainly associated with packets that help manage the router.

ip access-list extended cp-important-in
 remark Control plane important traffic - inbound
 remark TACACS
 permit udp host [TACACS server addr] eq tacacs any
 permit tcp host [TACACS server addr]eq tacacs any
 remark SSH/TELNET
 permit tcp [remote vty mgmt subnet] 0.0.0.255 any range 22 telnet
 remark SNMP
 permit udp  host [SNMP manager addr] any eq snmp
 remark NTP
 permit udp host [NTP server addr] eq ntp any
 deny   ip any any

Normal

The following access list is used to categorize packets that are normally seen by the router, but aren't necessary to the operation of the network. This might include ICMP and traceroute packets, which you normally allow some subset of. Note, in this example, we do not try to rate limit traceroute packets, because anything could be a traceroute packet (not just typical ICMP or UDP traceroute packets).

ip access-list extended cp-normal-in
 remark Control plane normal traffic - inbound
 remark ICMP
 permit icmp any any echo
 permit icmp any any echo-reply
 permit icmp any any parameter-problem
 permit icmp any any time-exceeded
 permit icmp any any unreachable
 deny   ip any any

Undesirable

The following access list is used to categorize packets that should never be processed by the router, because they are just a waste of CPU resources. Between various types of traceroutes and router initiated router initiated connections, you have to be careful about what you block. In this example, we just block ports that I've seen open on a Cisco router, that generally shouldn't need to be. You may wish to block other ports that you know the router won't listen to, but it's not clear there is much value in bothering to drop them rather than just sending an unreachable or a TCP reset. Note, the policy here is that packets matching this access list will never be processed by the CPU. If you are unsure, you might just let the packets fall through to the default class, which throttles all remaining traffic.

ip access-list extended cp-undesirable-in
 remark Control plane undesirable traffic - inbound
 remark NTP
 permit udp any any eq ntp
 remark SNMPTRAP
 permit udp any any eq snmptrap
 deny   ip any any

Default

This is a catchall access list for packets that do not match any permit rule in other CoPP access list statements. Packets matching this rate limiter all fight for remaining CPU access. It is important to match on packets in the other classes for traffic that you generally want the router to process, because this class can become a bit of a no man's land, with no expectation of service under adverse conditions.

access-list 2 remark utility ACL to allow everything
access-list 2 permit any

Enable CoPP

Once the class maps, policies and ACLs have been defined you need to enter two global configuration commands for CoPP to begin acting on packets to the processor. Note, we've used control-plane-in as a policy name in our examples, but you could name it anything you want. Enable mls qos to ensure CoPP is performed in hardware.

mls qos
control-plane
service-policy input control-plane-in

Troubleshooting

Unfortunately, troubleshooting CoPP configurations can be pretty difficult. The primary reason for this is the lack of logging support on access list rules used with CoPP. In my experience, I have so far misclassified two types of DHCP packets that a router should see, by having them land in the default class. The symptoms of which, results in clients have sporadic problems in getting DHCP addresses due to the capacity starvation they got in the default rate limiter. One command that can give you a general idea of what is happening is the show policy-map control-plane input command. This will just show you counters associated with average rate and packet drops for each class of traffic so the usefulness of this command is limited. Perhaps a bit more useful for getting an idea about which ACL rules are being used is with the show access-list command. This will show each ACL and after each rule in each ACL the number of packets that have matched a particular rule.

You should have out-of-band access to the router if you are not local to it, otherwise if you configure CoPP incorrectly, you can completely lock yourself out of the router. As a last resort, you can schedule a reload at command before making changes and then cancel it if you changes prove to work. In your initial testing, if you use a configuration like the one above, start with rates set high and bring them down as necessary instead of the reverse. It is useful to have a default deny as the last statement in each class except for the default class. In addition, the default class should have a high enough rate so that if you remove or change any of the other classes, packets should fall through and still get to the router under normal conditions. This helps prevent lock-out.

Submitted Feedback

Rodney Dunn says that you should be able to match arp with the following:

class-map match-all arp
 match protocol arp

In addition, he and others reminded me that there are various mls rate-limit knobs that can be used to protect the processor as well. Phil Rosenthal and I carried on a private conversation about how he is using CoPP to not only address DoS attacks, but also any magic packet attack that may cause problems for the router. This includes any known or unknown attacks based on bugs that have yet to be found. Phil prefers the default deny approach in his CoPP implementations. One good example Phil demostrated, which I will likely add to the notes above at some point is to completely block IP protocols that are not supported by the router. So for instance, the cp-undesirable-in policy might also include the following:

 deny icmp any any
 deny igmp any any
 deny tcp any any
 deny udp any any
 deny ospf any any
 deny pim any any
 permit ip any any

The idea above is to ignore protocols that the router may in some form wish to receive, but then all others should be explicitly denied. Phil and I also discussed the issue allowing certain protocols to be allowed to hit the router for troubleshooting purposes (e.g. in the case of traceroute packets). The stance taken in this document was to be more conservative in what you accept, but you may wish to be even more restrictive. It is a trade-off and each operator will have to decide for themselves what makes sense for their own environment. ISPs for example may make an entirely different choice than an end institution like a research university.

Phil also pointed out that some packets through the box and not necessarily to the box can attack the control plane. For example, IP datagrams with options such as record route and ICMP unreachables that are generated when a host or network is not available can be the cause or symptom of control plane performance degradation.

References

Deploying Control Plane Policing White Paper
Cisco presentation on Control Plane Policing
cisco-nsp mailing list

Acknowledgments

All @cisco.com people participating on the cisco-nsp list, thanks!
Phil Rosenthal, ISPrime

Revision History

2005-02-21,jtk: initial public document release
2005-02-28,jtk: Submitted Feedback section, minor edits
2005-03-04,jtk: mls qos parameter
2005-03-15,jtk: use of the show access-list command, minor edits
2005-03-15,jtk: fixed order for enabling CoPP