What is going on with BGP

[Ignas Bagdonas <ibagdona.nog () gmail com>]

A brief overview of developments happening in the IETF working groups
related to BGP evolution. The view is current as of mid-2023, in the
timeframe between IETF meetings 116 and 117, and looking back several years
to cover the recently published documents. The overview is given from the
perspective of development of the protocol mechanics and recommended
operational considerations, and is not directly related to specific
implementation aspects of specific platforms – for that you would need to
consult your vendors’ documentation. It is not expected that all of the
functionality described here will be universally productized, as well as
there will be specific deviations and extensions to functionality
implemented by different vendors as seen required by the market. This is
not an end to end overview of BGP, instead it focuses on specific protocol
changes and therefore it is assumed that a reader has a sufficient
understanding of foundations of BGP and its supporting machinery. It is a
high level overview and does not go deep into the specifics, pointers to
documents are provided for further and more detailed view into the topics
under the discussion. This part covers the core protocol part and
mechanisms specific to IPv4 unicast and IPv6 unicast AFs.



Deprecation of AS path aggregation sets
(draft-ietf-idr-deprecate-as-set-confed-set). When aggregating multiple
prefixes with different ASNs into a shorter covering prefix, besides other
aggregation related path attributes, the ASN identifiers of component
prefixes are contained in an unordered structure which has a different type
than the ordered ASN sequence of the AS path attribute. The need for such a
structure is to avoid possible propagation loops due to missing information
on which ASNs the update has traversed previously. However such an approach
obfuscates the real origin and prefix lengths of component prefixes and as
a result is directly incompatible with the developments in global routing
security mechanisms. Also it is yet another influencing aspect into
possible attribute packing conflicts due to different interpretations of
what an unordered set of ASNs in fact means. Therefore aggregation
resulting in generation of ASN sets (AS_SET and AS_CONFED_SET segments in
AS path attribute) is deprecated and should not be used. Receipt of an
update carrying such segments should be treated as a withdraw due to a
recoverable error (RFC7606), and no announcements carrying ASN set segments
can be advertised. This does not deprecate the aggregation of component
prefixes as such, but only the generation of ASN set segments. Both
aggregation within an AS and proxy aggregation can be deployed as designed,
and the fact of aggregation is indicated by the AGGREGATOR path attribute
carrying the ASN and RID of the node that performed the aggregation – same
as before, just omitting the addition of ASN set segment. The loop
avoidance is ensured by controlling the advertisement of the resulting
covering aggregate prefix – it must not be advertised to any of the origins
of component prefixes.

Overall this is not a new concept, RFC6472 recommended against advertising
ASN sets but did not change the behaviour of the receiving side. The amount
of ASN sets seen in the global routing system is small enough (such cases
do exist, but they are a clear exception or simply a neglect to clean the
configuration up) to justify a more strict set of rules that would remove
the ambiguity of interpretation of prefix origin.



Extended messages (RFC 8654). BGP message size is limited to 4096 octets
(PDU size should not be confused with link and packet layer MTU sizes and
transport window size), and that might not be enough in some cases. BGP
PDUs do not have any mechanism for fragmentation, and therefore a set of
path attributes that does not fit into the message cannot be advertised at
all. In addition, attribute packing is an efficient way of speeding up the
convergence of BGP, and the PDU size limitation puts an upper bound on the
balance of how many NLRIs can be advertised together with the path
attributes. New address families may carry larger NLRI elements and contain
more or larger attributes, and therefore 4K octet limit may be not enough.
BGP message encoding allows for a larger size, it is a historical limit
that is now being lifted. Extended messages mechanism defines a new
capability that needs to be configured and exchanged between the peers and
if both sides agree, they can use messages up to 65535 octets for BGP
signalling. Open message is excluded for backwards compatibility reasons,
and a large keepalive message just does not make practical sense. Update,
notification, refresh signalling, and potentially other newly defined
messages may use this mechanism for exchanging both larger size objects and
larger amounts of objects. Of particular note is the handling of
notification message – while the overall size of the message now may be
larger, so can be the size of notification data, and it cannot exceed the
negotiated total limit of 4096 or 65535 octets.



Extended optional parameters (RFC9072). BGP Open message carries
capabilities – a set of parameters that are exchanged between peers for
finding out common operational modes and their corresponding parameters.
The encoding used historically had a single octet length field for both
total length of all parameters and for a length of each individual
parameter, and therefore practically limited the amount of capabilities and
their parameters that can be exchanged. Given the trends in increasing the
amount of address families and their configuration parameters, the
usability features carrying human readable information such as host names
and version information, and supported BGP feature indicators, a mechanism
that would allow for a larger capability size is needed. The overall
approach of this extension is by defining a new optional parameter having a
specific length which acts as an indicator that a new format of container
is used for encoding individual optional parameters (BGP capability is a
type of optional parameter). No changes are made to the actual optional
parameters carried, it is just a container that is larger. This change is
unambiguous to a speaker that does not understand the new encoding – it
will result in an error indicating a presence of an unsupported optional
parameter. This mechanism allows for slightly above 4K octets of usable
space for optional open parameters – should be enough for everyone.



Dynamic capabilities (draft-ietf-idr-dynamic-cap). Capabilities provide an
ability for parametrizing a set of BGP operational parameters during
initial session startup. Once negotiated, capabilities and their
corresponding parameters stay constant for the duration of the BGP session
without any protocol-level ability to adjust it if needed. Dynamic
capabilities introduce a two-way capability parameter synchronization
mechanism without requiring a session bringdown. This is implemented by
means of a new protocol message that carries a list of potentially
negotiable capabilities and a request-response type of negotiation between
the peers. While this mechanism by itself allows for renegotiation of any
BGP capabilities after the initial session establishment, not all of such
negotiations would be practical or even technically feasible. Enabling a
new address family without bringing down an already established session
might be both practical and easy to implement, while disabling an already
established address family may result in logical dependency conflicts that
would render the remaining address families unusable. Changing already
negotiated timer parameters is easy, while enabling functionality such as
additional paths may not be technically feasible – those all are
implementation dependent aspects and would limit the practical breadth of
applicability of dynamic capabilities.



Send hold timer (draft-ietf-idr-bgp-sendholdtimer). BGP transport session
termination results in withdrawal of all received updates – which is a
practical way of clearing out the state that has become stale. However,
transport session liveness is controlled by the operation of the receive
side of the connection – a timeout in receiving of any message from the
remote peer is treated as transport failure. It is not the case for the
transmit side of the connection though, and a scenario where a remote peer
happily sends out periodic keepalives but fails to process any incoming
messages from the local peer would result in keeping the potentially stale
received information on a local peer. The concept of a send hold timer
tracks the local transport endpoint activity on the transmit side and if no
locally generated messages can be sent for a timer interval, local peer
will initiate a session teardown and clear out all the state received from
the remote peer.



Optimal route reflection (RFC 9107). Reflectors represent a universally
used mechanism for reducing the amount of state to be transferred
throughout the AS, their usage patterns are well understood, as well as
some limitations. This proposal addresses one of them – the path selection
as performed by the reflector is not necessarily an optimal one if that
same selection were to be performed by the reflector client itself. If BGP
and forwarding topologies are not congruent (which is the case in many
reflector deployments), the path selection will be influenced by the
metrics relevant and observed by the reflector itself and not by the
reflector client. Given that the number of paths to be selected from is
limited by a reflector, a client has no means for choosing what would have
been a more optimal path from its own perspective. From the perspective of
a reflector the differences would lie in the per-client path selection
(which may be different due to different policies being used for different
clients), and interpretation of IGP metric from the perspective of a
particular client and not of a reflector itself. There are no changes
required to be performed on the client side for this mechanism to work,
while a reflector would need to have a more detailed visibility into the
IGP topology for deriving a proper next hop cost for a prefix from a
perspective of a reflector client. In addition a path selection process on
a reflector would need to be performed individually per client or per group
of clients sharing the same reflection policy.



Wide BGP communities (draft-ietf-idr-wide-bgp-communities). Standard
communities, extended communities, large communities – haven’t we got
enough of different flavours of communities for expressing the policy
constructs? It appears that we in fact haven’t. The limiting factor appears
to be the ability to express actions and parameters for those actions in a
reasonably scalable and extensible way. Standard and large communities form
a functionally equivalent pair, with large communities primarily addressing
the 32 bit ASN clean signalling; extended communities mostly deal with
address families other than global unicast and also lack sufficient
flexibility for carrying information elements that have semantic
interpretation differing from a single plain 32 bit field.

The underlying base for wide communities is a new BGP path attribute that
acts as a container for community sub-objects, and wide communities are the
first actual user of this container infrastructure. Container attribute
itself is optional transitive, and individual sub-objects may have a finer
level of transitivity control, thus allowing for more controllable
attribute propagation.

The overall format of wide communities is <Community>:<Source ASN>:<Target
ASN>:<variable length parameters>, with Community being the actual
community value, Source ASN indicating the AS that is originating this
community, Target ASN indicating the AS that is supposed to interpret and
react on the community, and then followed by a variable length and format
set of parameters that are interpreted in the context of a community and
target ASN namespaces. Wide communities also define a parameterized
matching mechanism for indicating whether the community should or should
not be acted upon based on a set of criteria matching or not matching upon
the specific parameters. The actual parameters used for wide community
policy include ASN lists, IPv4 and IPv6 prefix lists, uint32 and IEEE 754
fp32 number lists, neighbour class list, an UTF-8 string, and also a
user-defined binary object having a free interpretation.



Extended admin shutdown communication (RFC 9003). RFC8302 defined a
mechanism for sending a human-readable message for several Notification
Cease subcodes, for a form of “in-band” message channel for BGP session
shutdown. The initial message length of 128 octets appears to be too
limited in the context of multibyte encodings, therefore this extension
lifts the limitation to 255 octets (but not necessarily characters!). The
message must be carried in a valid UTF-8 encoding, and it is not for the
receiving router to try to make sense of it – it is to be presented to the
operator as is.



Cease notification due to BFD session going down (RFC9384). BFD is a
universally deployed mechanism for tracking data plane liveness, and BFD
session state can also be propagated into control plane protocols for
speeding up their reaction to data plane failures. BGP is not an exception
to this mechanism, and it just works. When a BGP session goes down for some
reason, peer sends out a Cease message that provides some information on
the reason why the session is going down. This extension defines a subcode
indicating that the session was brought down due to the underlying BFD
session going down too. It may well be the case that a remote peer will not
be able to receive this notification, but the local peer will contain an
indication that the session was brought down due to BFD failure.



BGP graceful restart for notification messages (RFC 8538). BGP graceful
restart (RFC4724) allows for retaining the forwarding state while the
actual BGP session is restarting – except for when the session was brought
down due to a notification. Having an ability to retain the state in case
of protocol errors that are recoverable in some way appears to be of value
for maintaining the stability and reducing the amount of state that needs
to be propagated around. The core of this extension is exactly this – for
some error scenarios GR retains the forwarding state while BGP sessions
recover after the error that resulted in notification being sent. Not all
errors can be treated this way – only those that are of a temporary nature,
such as a remote peer running out of resources or transport having
intermittent problems.



Long-lived BGP graceful restart (draft-ietf-idr-long-lived-gr). BGP
graceful restart functionality (RFC4724) has been around for a while and is
widely deployed. There is a limit of time for which the stale routing
information can be retained by the BGP speaker, and that limit has the
upper bound of 4K seconds – for a simple reason that a field used for
signalling that value has got 12 bits. While an interval of more than an
hour for a remote peer to come back up might seem reasonably long, there
are use cases where it would be beneficial to retain a stale routing
information for a longer time, and primarily to avoid propagating the
withdrawals for the prefixes affected by the GR, at a cost and risk of
potential blackholing of a traffic destined to those prefixes. Extending
the timer value namespace is trivial, and that is precisely what is done by
this specification. Keeping stale routing information for extended periods
of time might be a bit more dangerous, therefore prefixes covered by the
long lived GR are also marked by a specific community value indicating that
they should be treated as a last resort for best path selection and also
can be propagated only to peers that also support long lived GR.



Peer roles (RFC9234). BGP does not define a semantic relationship between
peers – that is just a session over which prefixes can be advertised. What
is the relationship of those peers and whether they in fact are allowed to
exchange any routes learned from various sources historically was not tied
to the actual BGP session and was left to the domain of policy
implementation. This extension defines a set of mechanisms for specifying
and validating the roles of BGP peers and the prefixes they are allowed to
announce. Peers have roles defined based on their type and place in the
network (a customer, a provider, a route server, a generic peer among the
others) which get exchanged during the session establishment. If roles
disagree the session is not allowed to come up in the first place; if roles
allow for session to be established then depending on the actual role
routes advertised get an additional path attribute that indicates the class
of an advertising peer. Quite a similar mechanism for route leak detection
can be implemented by using community tagging – and while the overall logic
of peer roles is the same, the implementation is different: community needs
to be acted upon by the policy, and if the policy is not configured
properly or simply does not exist at all then such method will not work at
all. A dedicated path attribute can be advertised and acted upon by the
implementation not based on a user configurable policy and be active all
the time regardless of the policy configuration. Policy can be applied on
top of what is allowed in based on the peer roles, but peer roles have
priority. This mechanism is implemented as a capability for session
establishment with a set of role pairs that are allowed, and also as an
additional path attribute that gets added to advertised routes and can be
acted upon by the policy of the receiving peer.



Well-known large communities (draft-heitz-idr-wklc). Large communities
enjoy a wide deployment, and that same wide deployment starts to bring in
some requirements for an additional functionality, specifically limiting
propagation scope and avoiding conflicts in assigning function identifiers.
Large communities are a direct replacement of standard BGP communities and
as a result they do not have any structure of interpretation of the values
carried – it is just an AS number in the Global Administrator field. The
proposal is to reserve a set of ASNs for the purpose of being used as
well-known large community identifiers and also for control of propagation
scope, leaving a total of 10 octets for use within a context of a
particular WKLC. This appears to be quite controversial, both on the aspect
of reserving a significant portion of 32 bit ASN space, and also trying to
control the propagation of a transitive path attribute based on a value of
that same attribute. However, the functionality appears to be needed, and
it might be a right time to restart a discussion on defining an equivalent
of extended communities for 32 bit ASN clean operation.



AS path prepending (draft-ietf-grow-as-path-prepending). More does not
necessarily mean better. AS path prepending as a policy mechanism to
influence path selection is well known and is universally deployed.
Increasing the length of AS path by prepending ASN more than once decreases
the probability of such path being selected as best. The whole question is
how much to prepend in order to achieve the intended depreference of a
path, especially in the conditions of everyone else effectively doing the
same, and not to become too vulnerable to intentionally crafted shorter AS
paths. It is not uncommon to see paths with multiples of tens of prepends
in the global routing table – that is not surprising as AS path is a
transitive attribute and it is strictly not allowed to remove ASNs from the
received path. This can lead to a situation when any other – including
intentionally crafted – AS path is seen as more preferred due to having a
shorter length, and prepending yet more times only makes such types of
attacks easier. Signalling of the policy intent to the remote peers is
recommended to be implemented via specifically allocated communities or
other path attributes, and prepending should be used only when there are no
other alternatives available.



Maximum prefix limits (draft-sas-idr-maxprefix-inbound,
draft-sas-idr-maxprefix-outbound). The number of prefixes received by the
peer and advertised to other peers depends on the role and the place in the
network of a particular BGP speaker. Generally those numbers are quite
specifically bounded – a leaf site is not supposed to advertise a full
global table, and a peer in the exchange should not advertise substantially
larger amounts or prefixes than what it originates. A counter based
mechanism for controlling the number of received prefixes is supported by
virtually all BGP implementations. There are some deficiencies though – it
may be that the large amount of prefixes received from the remote peer will
be rejected by the inbound policy, and while keeping such rejected prefixes
is a handy optimization for speeding up the convergence in case of inbound
policy changes, the storage and processing of such rejected prefixes does
not come for free. In addition, refresh signalling could be used for
requesting only a specific set of prefixes to be readvertised if needed.
Another aspect is the number of prefixes accepted by the inbound policy –
for various reasons, including operator errors, the number of such prefixes
may end up being larger than it should be. On the sending side, there is
also a need for a mechanism to limit the number of prefixes past the
outbound policy that will get advertised to a remote peer. This results in
three separate counters – inbound pre-policy, inbound post-policy, and
outbound post-policy - that together can control the tolerable amount of
prefixes at various points of the BGP speaker processing. A twin set of
documents defines a recommended session shutdown and notification of the
remote side with an appropriate cease error to indicate the specific reason.



BMP support for local RIB monitoring (draft-ietf-grow-bmp-local-rib). BMP
provides a view into input and output BGP RIBs but lacks a mechanism to
transport all local RIB routes (all local routes meaning the full view of
the local RIB after the path selection process, not only the routes
imported into BGP context from other sources). Portions of such view can be
derived from the information available via input RIB monitoring (likely
requiring a coordinated monitoring of multiple nodes), however that may
result in a notable amount of data to be filtered through, would require
access to entities and state outside of BGP context, and still may lack to
provide a specific sequence of events that happened during topology
convergence. A dedicated mechanism for exporting information about all the
prefixes contained in the local RIB is defined by means of a new type of
BMP peer logically representing the contents of a local RIB. From the
perspective of BMP protocol processing there are no logical changes to the
usual operation of BMP route monitoring functionality – local RIB routes
would be represented as being received from an emulated peer bound to a
specific instance of a local RIB on a node.



More BMP TLVs (draft-ietf-grow-bmp-tlv). This is a framework document for
defining an extensibility mechanism for BMP route monitoring messages in
order to be able to carry additional and structured information within BMP.
Initially BMP defined a fixed packet format for route monitoring messages,
and deployment experience and evolving uses of BMP have indicated a need to
convey additional information elements that were not thought of at the
initial design time of BMP or may be specific to a particular
implementation. BMP has a TLV based encoding mechanism from the start for
most of its messages, but not for route monitoring. Therefore this simple
extension mechanism allows for a TLV based encoding to be used for all BMP
messages.



Autonomous System Provider Authorization (draft-ietf-sidrops-aspa-profile,
draft-ietf-sidrops-aspa-verification). Origin validation provides a
practical and reasonable level of verification of origination of prefixes,
but the propagation path of those prefixes once originated is difficult to
validate and protect from both unintended errors and malicious attacks. The
tree-like nature of BGP peerings may be used for building sets of adjacency
lists, treating one AS as a customer, and its peers as providers. This
provides a 1:n relationship between a particular AS and its peers, and a
distributed collection of such relationships forms a foundation for AS path
validation in terms of checking whether adjacent pairs of AS numbers in
fact have a customer-provider relationship. ASPA object is yet another
object type in the RPKI infrastructure, and therefore existing mechanisms
of RPKI used for origin validation would require only modest extensions in
order to distribute path validation information. The validation outcome
from the perspective of routing policy would result in already familiar
states of “valid”, “invalid”, and “unknown” related to AS path attributes
contained in received announcements.



RTR extensions (draft-ietf-sidrops-8210bis). RTR is a protocol used between
a router and a cache for distribution of RPKI data objects. ASPA brings in
a requirement for distributing AS path attestation objects to routers, and
it is a simple extension to RTR for carrying yet another object type – and
this set of extensions defines RTR protocol version 2. The overall protocol
mechanics is equivalent to the lower versions (version 0 is specified in
RFC6810, and version 1 is specified in RFC8210), and is based on the
concept of a router controlling the pull in of the validation information
instead of a cache pushing it out towards the router. Operation starts with
a maximum supported version and can negotiate a lower backwards compatible
protocol version if required during the session startup, and stays at the
negotiated version for a lifetime of a session. Deployment experience has
also identified several synchronization corner cases in the cache content
transfer in the presence of dynamic changes of that content. ROAs for
longer prefixes should be advertised before ROAs for corresponding shorter
covering prefixes, and multiple ROAs for the same prefix should be
advertised consecutively.