Alibaba is deploying SRv6 VPN with SONiC based white box router at its edge of the network. The Alibaba's SRv6 VPN use case has been described in muliple conferences. The basic idea follows RFC 8986, which we use SRv6 as a network programming framework for applications. VPN information is used as service anchor points for identifying service instances in the service plane. SRv6 policy could be used to provide paths to let applications reach service instances. The following diagram provides a summary on what Alibaba would like to achieve via SRv6 VPN.
To help promoting SRv6's adopting in the industry, Alibaba team decides to upstream its deployed SRv6 VPN solution to SONiC community via 202305 release. We expect more contributors would join us to complete SRv6 VPN related features.
The SONiC SRv6 VPN deployable solution requires changes from the following three communities.
Besides recent FRR changes for SRv6 VPN contributed by Cisco team, Alibaba team needs to bring in the following FRR changes for a deployable solution.
-
VRF routes leaking and performance tuning
-
BGP advertise delay
-
Conditional advertisement
-
SRv6 Policy handling
FRR changes would be commited to FRR community. SONiC 202305 release would pull in the corresponding version of FRR release.
The minimal FRR version for SRv6 VPN is 8.4, which you could get SRv6 VPN PoC functionalities. FRR version 8.5 or 9.0 would provide Alibaba deployment feature parity.
Current SRv6 SAI doesn't have VPN support. SRv6 VPN SAI would be contributed by Cisco team. The SAI object model for SRv6 VPN is shown in the following diagram. The detail information about SRv6 VPN SAI APIs could be found in SAI's HLD document. (https://github.com/rameshms-work/SAI/blob/rameshms/srv6_vpn_enhancements/doc/SAI-IPv6-Segment-Routing-VPN.md)
The main SRv6 SWSS design is documented via SRv6 HLD. This HLD focuses on the changes needed to enable SRv6 VPN on the top of existing SRv6 changes.
The overall work flow diagram is shown in the diagram below.
SRv6_SID_LIST_TABLE and SRv6_MY_SID_TABLE have already been supported. The detail information could be found in SRv6 HLD. For SRv6 traffic steering, we add a new SRv6 POLICY table.
Description : new table to hold SRv6 policy
Schema:
; New table
; holds SRv6 policy
key = SRV6_POLICY_TABLE|policy_name ;
segment = SRv6_SID_LIST.key, ; List of segment names, separated by ','
weight = weight_list ; List of weights, separated by ','
Description: vpn_sid and policy fields are added to existing route table schema.
Schema:
;Stores a list of routes
;Status: Mandatory
key = ROUTE_TABLE:VRF_NAME:prefix ;
nexthop = prefix, ; IP addresses separated ',' (empty indicates no gateway). May indicate VXLAN endpoint if vni_label is non zero.
intf = ifindex? PORT_TABLE.key ; zero or more separated by ',' (zero indicates no interface)
vni_label = VRF.vni ; zero or more separated by ',' (empty value for non-vxlan next-hops). May carry MPLS label in future.
router_mac = mac_address ; zero or more remote router MAC address separated by ',' (empty value for non-vxlan next-hops)
blackhole = BIT ; Set to 1 if this route is a blackhole (or null0)
segment = SRV6_SID_LIST.key ; Optional field. List of segment names, separated by ','
seg_src = address ; Optional field. Source addrs for sid encap
vpn_sid = vpn_sid ; Optional field to add vpn_sid to learnt BGP routes
policy = policy ; Optional field to add a list of policy names, separated by ','
In Alibaba's deployment, anycast routes are used as MY_SID. This approach would help us to reduce the outage from BGP NH's convergencs, when BGP prefix independent convergence (PIC) is not supported in FRR.
In the current FRR approach, zebra sends the SRV6_MY_SID information to the kernel, not the fpm server. This is due to SRV6_MY_SID field has not been defined in netlink message yet. There is a workaround used to pass MY SID via route info. When adding routes, SRv6 routes can be distinguished from ordinary routes via route's nexthop information. But this workaround has a limitation in the deleting path. SRv6_MY_SID can't be distinguished from ordinary routes, since prefixes are the only information provided to fpmserver. Since Alibaba uses anycast routes as MY_SID, we have to find an alternative way to solve this limitation. We add a simple workaround which is described in "MY_SID enhancement" section to overcome this limitation.
In the current support for SRV6_MY_SID, End.X action has been implemented in srv6orch, but the processing of nexthop id is missing. We added the support of nexthop id for End.X action in srv6orch.
Schema:
; holds local SID to behavior mapping
; Status: Mandatory
key = SRV6_MY_SID_TABLE:block_len:node_len:func_len:arg_len:ipv6address
; field = value
action = behavior ; behaviors defined for local SID
vrf = VRF_TABLE.key ; VRF name for END.DT46, can be empty
adj = address, ; List of adjacencies for END.X, can be empty
segment = SRv6_SID_LIST.key, ; List of segment names for END.B6.ENCAP, can be empty
source = address, ; List of src addrs for encap for END.B6.ENCAP
There is an enhancement added for handling segments. When the first sid in a segment matches some SRv6 MY_SID and the MY_SID's action is End.X, an attribute SAI_SRV6_SIDLIST_ATTR_NEXT_HOP_ID will be set. The value of this attribute is set to MY_SID's nexthop id. The SAI_SRV6_SIDLIST_ATTR_SEGMENT_LIST of this segment will not contain the first sid.
SRv6 VPN routes handling is added on the top of existing SRv6 routes handling. When a SRv6 VPN routes are created, SRv6 VPN routes contains VPN_SID. SRv6Orch would check if it needs to create a corresponding SRv6 tunnel for the given BGP NH, and create the SRv6 tunnel if needed. Each SRv6 Tunnel points to an encap mapper.
The encap mapper is used to store all the VPN_SIDs, which SRv6 VPN routes would use via this BGP NH. The key for this tunnel map is Prefix AGG_ID. SRv6Orch allocates a Prefix AGG_ID for a given BGP NH, and store Prefix AGG_ID with VPN_SID binding in encap mapper table. SRv6 tunnel would be used in SRv6 NH. SRv6 NH also points to created SID list if a set of selected IGP paths are specified via SRv6 policy. The detail SAI forwarding chain information for SRv6 VPN could be found in the above SAI section.
SONiC's current design focuses on data center switch's use cases. The number of routes are limited on these types of devices. Some inefficient memory usage exists in current code base. SRv6 VPN would bring in VPN routes to the million level. These inefficient memory usage would not only bring up device's memory usage, but also slow down the BGP loading time for bringing up million routes. Therefore, we need to reevaluate the memory usage for route objects and reduce the overall (SONiC/FRR/SAI) memory footprint per route to below 2KB as a starting point.
We found that when sending 2M routes to orchagent, the orchagent will use about 3.5G of memory. Most of the memory is used by saimeta. the memory in saimeta is used for routing checks. In order to reduce memory usage and improving performance, we added a flag in the saimeta to decide whether to skip this routing check. By skipping this check, the memory usage will be reduced to 1.4G with 2M routes.
Segment Routing policy is defined in RFC 9256. SRv6 Policy's definition follows the same RFC. SRv6 Policy related information would be provided from FRR's pathd. Currently, we take a workaround approach which is described in "SRv6 policy download" section.
The SRv6-policy information is only stored in the orchagent and won't be distributed to the ASIC_DB. Therefore, when SRv6-policy are added and deleted, SAI entry will not be involved if the policy is not in use. SRv6 policy information would be consumed when a SRv6 policy route is added.
The SRv6-policy content could be changed at any time. The changes could include segment adding/removing or weight updating. When a SRv6-policy content is updated, a backwalk is triggered to walk through all corresponding nexthop groups. Corresponding hardware information would be updated during the backwalk based on updated policy information.
In SRv6 deployment, we rely on SBFD (RFC 7880, RFC7881) to protect SRv6 policy defined paths. Due to fast convergence and scale 's requirements, we would like to offload SBFD to hardware. The following changes are made for offloading SBFD sessions.
-
Add a new bfd encapsulation type enum SAI_BFD_ENCAPSULATION_TYPE_SRV6. When the SRv6 policy bfd session type is SBFD Initiator or SBFD echo, set SAI_BFD_SESSION_ATTR_BFD_ENCAPSULATION_TYPE attribute to SAI_BFD_ENCAPSULATION_TYPE_SRV6 if SRv6 is used for this SBFD session.
-
Add a new bfd session attr enum SAI_BFD_SESSION_ATTR_SRV6_SIDLIST_ID. When the SRv6 policy bfd session type is SBFD Initiator or SBFD echo, we must set the sidlist id in SAI_BFD_SESSION_ATTR_SRV6_SIDLIST_ID to provide path information for SBFD SRv6 session. The sidlist OID would be obtained via getBfdSidListOID in SRv6Orch.
There are three changes maded in BFD_PEER schema.
-
"local-discriminator" and "remote-discriminator" are added in BFD_PEER schema. Different bfd sessions may have the same local IP address and remote IP address. We need use local and remote discriminator to differentiate these BFD sessions.
-
Three new BFD types are added, SBFD initiator , SBFD reflector and SBFD echo.
-
Two extra fields "endpoint" and "seglist-name" are added
The SBFD_PEER schema is shown as the following
community:
SBFD_SESSION:{{vrf}}:{{local_discriminator}}:{{ipaddr)}}
"type": {{string}} (sbfd initiator/sbfd reflector/sbfd echo)
"multiplier": {{integer}}
"tx_interval": {{interval}}
"rx_interval": {{interval}}
"remote_discriminator": {{integer}}
"local_addr": {{ipaddr}}
"udp_sport": {{integer}}
"udp_dport": {{integer}}
"ttl" : {{integer}}
"endpoint": {{ipaddr}}
"seglist_name": {{string}}
When BFD orch receives a BFD session state notification from hardware, BFD orch updates the state information in state db. Other SONiC modules could subscribe the BFD session notification based on their needs. bfdd also needs to be informed. But bfdd belongs to FRR code base, currently, there is no infrastructure to let SONiC generated states be propogated to FRR related processes. Therefore, we take a workaround approach which is described in "Notify BFD state change events to FRR bfdd" section.
The following schema is added for describing bfd_hw_state in STATE_DB. Syncd would monitor hardware asic notification and update the session state in ASIC DB. BFD orch monitors ASIC DB and set corresponding hw offloaded BFD session state in the state DB with the following schema.
BFD_PEER|{vrfname}|{alias}|{peer_address}|{local-discriminator}
"bfd_hw_state": {{string}} (up/down)
-
Add SRv6 route with single nexthop. Verify the following items
-
SAI_OBJECT_TYPE_TUNNEL_MAP with type SAI_TUNNEL_MAP_TYPE_PREFIX_AGG_ID_TO_SRV6_VPN_SID is created
-
SAI_OBJECT_TYPE_TUNNEL with mode SAI_TUNNEL_PEER_MODE_P2P is created
-
SAI_OBJECT_TYPE_TUNNEL_MAP_ENTRY with vpn sid value in SRv6 route is created
-
SAI_OBJECT_TYPE_NEXT_HOP with type SAI_NEXT_HOP_TYPE_SRV6_SIDLIST is created
-
SAI_OBJECT_TYPE_ROUTE_ENTRY with attribute SAI_ROUTE_ENTRY_ATTR_PREFIX_AGG_ID is created
-
-
Remove SRv6 route with single nexthop. Verify the following items
- Verify that the SAI object created in the above step are deleted
-
Add SRv6 route with multi nexthops. Verify the following items
-
Verify SAI object in step 1
-
SAI_OBJECT_TYPE_NEXT_HOP_GROUP is created
-
SAI_OBJECT_TYPE_NEXT_HOP_GROUP_MEMBER is created
-
-
Remove SRv6 route with multi nexthops. Verify the following items
- Verify that the SAI object created in the above step are deleted
-
Switch SRv6 route from single nexthop to nexthopgroup. Verify the following items
-
SAI_OBJECT_TYPE_TUNNEL_MAP_ENTRY with a new SAI_TUNNEL_MAP_ENTRY_ATTR_PREFIX_AGG_ID_KEY is created
-
SAI_OBJECT_TYPE_NEXT_HOP_GROUP is created
-
SAI_OBJECT_TYPE_NEXT_HOP_GROUP_MEMBER is created
-
SAI_OBJECT_TYPE_ROUTE_ENTRY with a new SAI_ROUTE_ENTRY_ATTR_PREFIX_AGG_ID and a new SAI_ROUTE_ENTRY_ATTR_NEXT_HOP_ID is seted
-
-
Add two SRv6 route with same nexthopgroup. Verify the following items
-
Add first one, Verify SAI object in step 3
-
Add second one, Verify SAI_OBJECT_TYPE_ROUTE_ENTRY is created, with same SAI_ROUTE_ENTRY_ATTR_PREFIX_AGG_ID and SAI_ROUTE_ENTRY_ATTR_NEXT_HOP_ID as first one
-
-
Remove two SRv6 route one by one. Verify the following items
-
Delete first route, Verify SAI_OBJECT_TYPE_ROUTE_ENTRY is deleted
-
Delete second one, Verify that the SAI object created in the step 6 are deleted
-
-
Add and remove ipv6 seamless BFD session (sbfd initiator and sbfd reflector). Verify the asci db especially SAI_BFD_SESSION_ATTR_BFD_ENCAPSULATION_TYPE and SAI_BFD_SESSION_ATTR_SRV6_SIDLIST_ID and state db.
-
Add and remove multiple ipv6 seamless BFD session . Verify same as last one.
-
Add and remove ipv6 seamless BFD echo . Verify the asci db especially SAI_BFD_SESSION_ATTR_BFD_ENCAPSULATION_TYPE, SAI_BFD_SESSION_ATTR_ECHO_ENABLE , SAI_BFD_SESSION_ATTR_SRV6_SIDLIST_ID.
-
Add and remove multiple ipv6 seamless BFD echo . Verify same as last one.
-
Handle the notify of bfd hardware state's change. Verify the state db.
RFC 8986 Segment Routing over IPv6 (SRv6) Network Programmin
RFC 9256 Segment Routing Policy Architecture
RFC 7880 Seamless Bidirectional Forwarding Detection (S-BFD)
RFC 7881Seamless Bidirectional Forwarding Detection (S-BFD) for IPv4, IPv6, and MPLS
We don't have an efficient infrastructure to communicate between FRR related processes and SONiC related processes due to current SONiC design limitation. This area needs to be redesigned in the future. Alibaba's business can't wait for the community agreed solution to be ready, therefore we put in the following workaround for an overall solution in the cloud environment. These workarounds would NOT be commited in 202305 release. We will raise PRs and provide diff for referencing purposes only.
For a simple workaround, FRR pathd would add, update and delete SRv6 policy related information by writing to APP_DB directly.
For a simple workaround, Zebra would add, update and delete SRV6_MY_SID by writing to APP_DB directly.
For notifying bfdd in FRR, we introduce a new bfdsyncd process in our code base which would set up a TCP socket connection between FRR's bfdd process and SWSS's bfd syncd process. This new process monitors state db for hardware offloaded bfd sessions' state change events and relay these state change events to FRR's bfdd.
These PRs are for referencing only.
TODO, will be added once PRs are raised.