Introduction to IP Addressing Types
Internet Protocol (IP) addresses are often overloaded to serve
multiple purposes, such as a unique host identifier and locator. Most
people are familiar with an IP address that is used as a unique
identifier such as one assigned to a local PC's Ethernet or dial-up PPP
interface. IP addresses assigned as a unique identifier to a host
interface are referred to as unicast addresses. A unicast address can
be used either as the source or destination address in an IP datagram.
A sender inserts an assigned unicast address into the source address
field of an IP datagram for packets sent onto the network. When
packets are destined for a single host, a sender inserts a unicast
address into the destination address field of an IP datagram,
indicating who (and where) the packet should be delivered to. In some
cases, hosts have multiple unicast address assignments. Often times in
those cases, it usually means that the host has multiple network
attachments with a unicast address assigned to each. Although in some
special circumstances, multiple unicast addresses can be assigned to a
single interface. An example of a unicast address is 18.104.22.168.
This is the unicast IP address that, at least as of this writing, is
associated with the hostname www.kuro5hin.org.
Unicast addresses have historically been used to identify unique
hosts or host interfaces. Knowing a unicast address usually meant
knowing the actual, physical host the unicast address was associated
with. This changes with anycast (and also with things like load
balancers or IETF RFC 1918 addresses), but we'll come back to that part of the story shortly.
Broadcast addresses have been used to support discovery of services
or hosts. Packets to an IP broadcast address are delivered to all
hosts on a particular physical data link network or IP subnet.
Broadcast addresses should only be used in the destination address
field of an IP header, because it would be illogical for a unique
packet to be sourced by a group of hosts or interfaces. IP broadcast
addresses take the form of either the limited broadcast address
(255.255.255.255) or the directed broadcast address (the network ID
plus all 1's in the host ID portion of the intended subnet).
There is also the concept of a group address and for this the term
multicast is used. A multicast address is associated with a group of
interested receivers. A group can consist of any number of hosts,
including all hosts on the network. Therefore, a broadcast address is
a subset address type within the concept of multicast addressing. Like
the broadcast address, multicast addresses can only be used in the
destination address field of an IP header. IP multicast addresses are
allocated from the historic class D range of addresses (22.214.171.124/4 in
CIDR notation, or an address between 126.96.36.199 and 188.8.131.52
inclusive, for those of you unfamiliar with IP address netblock
Anycast addresses can be used as either a source or destination
address, but no longer uniquely identify a single host or service.
However, they do continue to act as a locator. Anycast addresses come
from or are sent to the "nearest" host or service. The definition of
"nearest" a) depends on the network topology, b) protocols used to make
forwarding decisions and c) the associated administrative policies
within the internetwork. Sometimes anycast addresses are referred to
as group addresses, but unlike multicast, packets are delivered only to
one member within the anycast group. Except for the 6to4 relay router
specification, there is not an address range set aside for anycast use
on the public IPv4 Internet today (see IETF RFC 3330).
Individual organizations must allocate anycast addresses from an
available unicast address netblock they have administrative control of.
By definition, IPv4 anycast addressing assigns a common unicast
address to multiple interfaces, hosts or services on an internetwork.
In other words, duplicate IP addresses are purposefully configured into
the IP internetwork. Assigning duplicate IP addresses can be a
dangerous affair and in fact will often break (catastrophically in some
cases) communications when used on the same physical data link network.
Without special load balancing protocols or hardware, anycast IP
addresses should not be duplicated on any one data link network.
While some may argue that any use of duplicate IP addressing is
fundamentally a bad idea, for our purposes in this article we will
ignore the philosophical debates. So how then, does the network figure
out where to send packets to an anycast address when that address can
be multiple places at once?
Routers advertise netblocks (also known as prefixes) serving
anycast address space just as they would advertise netblocks for
unicast or multicast prefixes. With anycast address space, routers
advertise the address netblock from multiple origin points in the
internetwork. From a routing topology perspective, a common anycast
address netblock looks as if it is multi-homed to all points of origin.
In practice, an anycast netblock is often just a host route (/32 in
CIDR notation) within an autonomous system (AS). To network routers
and routing protocols, nothing changes about their operation and there
are no new anycast specific routing concepts to learn.
To illustrate how anycast address netblocks are used in an
internetwork, we will use an example. Imagine an internetwork with
routers in three cities, Chicago, Los Angeles and New York. Presume
there are anycast service instances attached to the Los Angeles and New
York routers. A client of the anycast service is attached to the
Chicago router. Packets from the host in Chicago to the anycast address
will be forwarded towards either New York or Los Angeles, arriving at a
unique instance of the service at one city, but not both. The packets
will be forwarded based on the routing topology as viewed from Chicago
(and other routers in between).
Presume packets from Chicago to the anycast address in the previous
example normally travel to the New York instance. What happens if the
route to New York from Chicago changes such that Los Angeles looks
closer from a topology perspective (as might happen when a link or
router goes down)? As soon as the routing topology converges due to
the change, packets would then transition to the Los Angeles path.
As an aside, one may be thinking, if the path changes in the middle
of a conversation between the Chicago source and the anycast service in
New York or Los Angeles, won't communications fail? Without complex
mechanisms to maintain synchronization, the other anycast service
instance will not know about the session that existed previously, right?
Correct. This is why anycast is not well suited for communications
that involve "state" or long-lived flows of traffic. This eliminates
TCP-based protocols and and even some UDP-based applications from being
deployed with anycast addressing reliably. We will come back to
specific applications where anycast is most applicable in the next
How anycast address netblocks populate the routing topology and how
the routing "metrics" are calculated can vary from site to site. In
the simplest form, a static route can be set in a local router,
pointing to an instance of an anycast service host. However, if the
anycast host fails, the static route has to be manually removed so that
packets to the anycast netblock can be forwarded to other instances on
the network. The most popular way to inject and remove anycast address
netblocks from the network routing tables is by running a routing
process directly on the anycast service host. This enables the anycast
service host to automatically control whether it is available or not if
downtime or maintenance is to occur. It also allows the network to
automatically age out routes, which may happen if an anycast host fails.
Packets travel to an anycast address based on the locality, from a
routing perspective, between a client and anycast service host. This
tends to keep network traffic within a well defined coverage area as
long as anycast service hosts and the routing infrastructure remains
stable. Through careful network design, traffic patterns and link
loads can be greatly influenced by the deployment of anycast
addressing. By leveraging the unicast routing system, redundancy, load
balancing and attack resiliency can be designed into services utilizing
anycast addressing. In the next section, we examine some of the
applications that can take advantage of these properties.
While there is nothing to prevent someone from running any service
using anycast addressing, even TCP-based ones, it may be sufficiently
unwise to do so. If routing instabilities or host failures occur,
communications will fail as they are re-routed to another anycast
service instance. The new anycast service instance will not recognize
session in progress without complex state-sharing mechanisms. In
addition, it would be foolish to begin transitioning services to run as
anycast services without thoroughly understanding all of the nuances
that come into managing applications and hosts with non-unique IP
addresses. In other words, anycast addressing is not for everyone and
everything. It often doesn't make sense to go through the extra
complexity in deploying a service with anycast addressing if it doesn't
justify the benefit.
That said, anycast addressing is currently used successfully by a
number of organizations and for a number of applications. We will
focus our discussion of application usage on DNS and multicast
rendezvous points (RPs), while offering a glimpse into other
applications that are most likely candidates for anycast addressing
Domain Name System (DNS)
Perhaps the most widely publicized use of anycast addressing today is in
large DNS deployments, most notably by a handful of the Internet's
root DNS server operators. Nearly all exchanges between a DNS client
and a DNS server are a single UDP client query request and sole UDP
message server response. In this mode of operation, anycast addressing
is well suited. Its been measured that this simple exchange accounts for
over 99% of all legitimate DNS traffic between DNS clients and root DNS
servers. As a result of the enormous load required by DNS root servers
and the stateless nature of DNS communications, root servers are perfect
candidates for leveraging anycast addressing.
Those familiar with DNS protocol operations may know that DNS
messages can and sometimes do use TCP instead of UDP. Isn't this going to
be a problem if anycast routes or anycast services are unstable? Potentially
yes, but practically no. Root server operators have measured typical,
legitimate DNS traffic and found very little TCP-based communications.
In addition, since the routes and hosts are generally stable, risk due to
transition to other instances of an anycast address are very low. Even
if there was a transition, it would have to fall within a very short
window, since even TCP-based DNS queries are relatively short-lived.
The F root nameserver operated
by ISC was the first root nameserver to
deploy anycast addressing and they have been doing so since November, 2002.
Root nameservers such as F root have been on the receiving end of many powerful
denial of service attacks. It is widely recognized that anycasting of
some servers has been glaringly helpful in mitigating the impact of those
attacks. Primarily to deal with offered load, due to valid requests,
misconfigured or attack traffic, anycast addressing has proven to greatly
enhance DNS root server survivability.
Multicast Rendezvous Points (RPs)
One of the first and most widely used applications of anycast addressing
was for protocol independent multicast sparse-mode (PIM-SM) rendezvous
focal point, bringing together multicast senders and interested
receivers within a multicast domain.
IETF RFC 2362, the
PIM-SM specification originally stipulated a single RP for any
particular group. For large and geographically diverse networks, this
can imply a significant single point of failure, an unruly traffic load
aggregation point, large amounts of group state maintenance or all of
the above. Anycast addressing has been used to create multiple
instances of RPs for common groups within a PIM domain.
IETF RFC 3446 has
recently been written to update the original PIM-SM specification,
allowing for multiple RPs per group through the use of anycast
In anycast RP environments, multicast group registration is delivered
to the nearest RP. However, because an RP must know about all sources
for any particular group it is an RP for, multiple anycast RPs must
share group registrations they each know about. This is currently
achieved through the use of multicast source discovery protocol (MSDP)
peering between the RPs. Anycast RP is now widely believed
to be the "current best practice" for PIM-SM deployments.
Other Anycast Applications
Dozens of other custom or standard applications can potentially be
deployed with anycast addressing techniques. Syslog servers, which collect
UDP based messages from remote hosts are a good example, because the
traditional Syslog server implementations never send messages in response to
received log messages. Traffic flow collectors, such as Cisco's Netflow
architecture or the freely implemented collectors like
are also candidates for anycast addressing
deployments. Sinkhole networks, which are designed to route "bogon"
netblocks off production networks and are sometimes used to monitor bogus
address space usage for later analysis are also good candidates for anycast deployment.
The Simple Network Time Protocol (SNTP) would work
with anycast, but NTP could be a problem. SNTP uses a single request
and reply message exchange, but NTP generally requires at least two
packets from both the client and server to exchange time information
properly. IPv6 to IPv4 relay routers as defined in
IETF RFC 3068 have their own netblock, which acts as an anycast service for IPv6 networks to talk
to IPv4 networks. These 6to4 relay routers use anycast to help ease
network management of IPv4 to IPv6 protocol transition. SNMP trap managers
could also be configured for anycast addressing, because SNMP managers
do not generally respond to received traps. Although if used as a full
two-way SNMP manager, the SNMP manager should use a globally unique unicast
address for SNMP management.
It is possible to configure any application using anycast
addressing, but practically it may be unwise to do so. An unstable
network may interrupt long lived sessions. Coupled with the difficulty
in troubleshooting broken connections, many applications will often
call for alternative solutions.
Anycast Service Management
When duplicate IP addresses are used, how does a network manager
monitor each instance of the anycast service? There are at least two
ways. First, the monitoring tools could be placed in proximity to
each instance of the anycast service, so that probe packets will arrive
at each anycast service instance. Second, a remote monitoring station
could be located remotely, but could tunnel management packets to a
unique address in close proximity to the anycast service to be managed.
In this way, when the probe packets reach the end of the tunnel, they
will be forwarded to the nearest anycast service. However, when the
response is sent back to the centralized management station, which by the
way does not need to be tunneled, how does the central management station know which
instance of the anycast service is responding? This is left as an
exercise for the reader.
Most anycast services are deployed along with a global, unique
unicast address for management and state-based communications. So for
instance, a DNS server may receive and reply to DNS queries using the
anycast address, but it will receive and reply to zone transfers and
remote management messages on a separate, unique unicast address
(possibly on the same interface as that of the anycast address).
Implementing a unique unicast address for management purposes isn't
strictly required, but it is such a good idea that it should be.
Usually the anycast address is implemented as a virtual interface on
the anycast service host, such as in the form of a loopback address and
the unique unicast management address is associated with an actual
physical data link interface.
Since an anycast service, such as DNS, is separate from the underlying
routing infrastructure that makes the service available through anycast
netblock routing updates, what happens
when the anycast service fails, but the route to its anycast address is
still in the routing topology? Typically there is no strong coupling
of application service to address route announcement, which means a failure in application service
does not induce a route withdrawal to its instance of the anycast address
from the network's routing tables. To get closer to a solution, the
host running the anycast service can run a distributed routing protocol
process locally and inject route updates to its local router for inclusion
in the local AS or global Internet routing tables. If the host crashes,
its anycast address route will eventually time-out, resulting in a route withdrawal
for that instance of the anycast address netblock. However, what if only
the application fails, but not the entire host? How does the local routing
process know to withdraw the route for the unavailable anycast service?
Today, the answer is "it probably doesn't". For most anycast applications
anyway, there is typically not going to be a strong coupling between the
application service and the local routing process. However, the host
can be carefully configured to monitor its running anycast service. If
the anycast service exits, crashes or becomes unresponsive, a local
process can send an alert to an administrator and even automatically
restart the anycast process. If the process exits and cannot be restarted,
a local process can be configured to shutdown the local routing
process or withdraw the anycast address netblock.
As used in IPv4 networks, anycast addressing is the strategic use of
duplicate unicast IP addressing to provide localization for services and
hosts in order to obtain robustness, redundancy and resiliency.
While anycast addressing is a useful technique, it is not suitable for all applications or environments.
Anycast addressing is not a panacea to scaling, redundancy or security
problems in the IPv4 Internet. It, like many other techniques, is just
one tool that if carefully considered and applied can be advantageous
in certain scenarios. For many problems, other solutions may be
preferable and any network designer or engineer should carefully
consider options and trade-offs.
Full discussion: http://www.kuro5hin.org/story/2003/12/31/173152/86