Architecture and Components
The Cisco SD-WAN solution is comprised of separate orchestration, management, control, and data planes.
- The orchestration plane assists in the automatic onboarding of the SD-WAN routers into the SD-WAN overlay.
- The management plane is responsible for central configuration and monitoring.
- The control plane builds and maintains the network topology and makes decisions on where traffic flows.
- The data plane is responsible for forwarding packets based on decisions from the control plane.
Figure 9. Overview of Cisco SD-WAN solution planes
Components
The primary components for the Cisco SD-WAN solution consist of the vManage network management system (management plane), the vSmart controller (control plane), the vBond orchestrator (orchestration plane), and the WAN Edge router (data plane).
- vManage – This centralized network management system is software-based and provides a GUI interface to easily monitor, configure, and maintain all Cisco SD-WAN devices and their connected links in the underlay and overlay network. It provides a single pane of glass for Day 0, Day 1, and Day 2 operations.
- vSmart controller – This software-based component is responsible for the centralized control plane of the SD-WAN network. It maintains a secure connection to each WAN Edge router and distributes routes and policy information via the Overlay Management Protocol (OMP), acting as a route reflector. It also orchestrates the secure data plane connectivity between the WAN Edge routers by reflecting crypto key information originating from WAN Edge routers, allowing for a very scalable, IKE-less architecture.
- vBond orchestrator – This software-based component performs the initial authentication of WAN Edge devices and orchestrates vSmart, vManage, and WAN Edge connectivity. It also has an important role in enabling the communication between devices that sit behind Network Address Translation (NAT).
- WAN Edge router – This device, available as either a hardware appliance or software-based router, sits at a physical site or in the cloud and provides secure data plane connectivity among the sites over one or more WAN transports. It is responsible for traffic forwarding, security, encryption, quality of service (QoS), routing protocols such as Border Gateway Protocol (BGP) and Open Shortest Path First (OSPF), and more.
The following diagram demonstrates several aspects of the Cisco SD-WAN solution. This sample topology depicts two WAN Edge sites, each directly connected to a private MPLS transport and a public Internet transport. The cloud-based SD-WAN controllers (the two vSmart controllers, the vBond orchestrator, along with the vManage server) are reachable directly through the Internet transport. In addition, the topology also includes cloud access to SaaS and IaaS applications.
Figure 10. Example SD-WAN topology
The WAN Edge routers form a permanent Datagram Transport Layer Security (DTLS) or Transport Layer Security (TLS) control connection to the vSmart controllers and connect to both of the vSmart controllers over each transport. The routers also form a permanent DTLS or TLS control connection to the vManage server, but over just one of the transports. The WAN Edge routers securely communicate to other WAN Edge routers using IPsec tunnels over each transport. The Bidirectional Forwarding Detection (BFD) protocol is enabled by default and runs over each of these tunnels, detecting loss, latency, jitter, and path failures.
Site ID
A site ID is a unique identifier of a site in the SD-WAN overlay network with a numeric value 1 through 4294967295 (2^32-1) and it identifies the source location of an advertised prefix. This ID must be configured on every WAN Edge device, including the controllers, and must be the same for all WAN Edge devices that reside at the same site. A site could be a data center, a branch office, a campus, or something similar. By default, IPsec tunnels are not formed between WAN Edge routers within the same site which share the same site-id.
System IP
A System IP is a persistent, system-level IPv4 address that uniquely identifies the device independently of any interface addresses. It acts much like a router ID, so it doesn’t need to be advertised or known by the underlay. It is assigned to the system interface that resides in VPN 0 and is never advertised. A best practice, however, is to assign this system IP address to a loopback interface and advertise it in any service VPN. It can then be used as a source IP address for SNMP and logging, making it easier to correlate network events with vManage information.
Organization Name
Organization Name is a name that is assigned to the SD-WAN overlay. It is case-sensitive and must match the organization name configured on all the SD-WAN devices in the overlay. It is used to define the Organization Unit (OU) field to match in the Certificate Authentication process when an SD-WAN device is brought into the overlay network.
Public and Private IP Addresses
Private IP Address
On WAN Edge routers, the private IP address is the IP address assigned to the interface of the SD-WAN device. This is the pre-NAT address, and despite the name, can be a public address (publicly routable) or a private address (RFC 1918).
Public IP Address
The Post-NAT address detected by the vBond orchestrator. This address can be either a public address (publicly routable) or a private address (RFC 1918). In the absence of NAT, the private and public IP address of the SD-WAN device are the same.
TLOC
A TLOC, or Transport Location, is the attachment point where a WAN Edge router connects to the WAN transport network. A TLOC is uniquely identified and represented by a three-tuple, consisting of system IP address, link color, and encapsulation (Generic Routing Encapsulation [GRE] or IPsec).
Color
The color attribute applies to WAN Edge routers or vManage and vSmart controllers and helps to identify an individual TLOC; different TLOCs are assigned different color labels. The example SD-WAN topology in figure 10 uses a public color called biz-internet for the Internet transport TLOC and a private color called mpls for the other transport TLOC. You cannot use the same color twice on a single WAN Edge router.
Overlay Management Protocol (OMP)
The OMP routing protocol, which has a structure similar to BGP, manages the SD-WAN overlay network. The protocol runs between vSmart controllers and between vSmart controllers and WAN Edge routers where control plane information, such as route prefixes, next-hop routes, crypto keys, and policy information, is exchanged over a secure DTLS or TLS connection. The vSmart controller acts similar to a BGP route reflector; it receives routes from WAN Edge routers, processes and applies any policy to them, and then advertises the routes to other WAN Edge routers in the overlay network.
Virtual private networks (VPNs)
In the SD-WAN overlay, virtual private networks (VPNs) provide segmentation, much like Virtual Routing and Forwarding instances (VRFs) that many are already familiar with. Each VPN is isolated from one another and each have their own forwarding table. An interface or subinterface is explicitly configured under a single VPN and cannot be part of more than one VPN. Labels are used in OMP route attributes and in the packet encapsulation, which identifies the VPN a packet belongs to.
The VPN number is a four-byte integer with a value from 0 to 65535, but several VPNs are reserved for internal use, so the maximum VPN that can or should be configured is 65527. There are two main VPNs present by default in the WAN Edge devices and controllers, VPN 0 and VPN 512. Note that VPN 0 and 512 are the only VPNs that can be configured on vManage and vSmart controllers. For the vBond orchestrator, although more VPNs can be configured, only VPN 0 and 512 are functional and the only ones that should be used.
- VPN 0 is the transport VPN. It contains the interfaces that connect to the WAN transports. Secure DTLS/TLS connections to the controllers are initiated from this VPN. Static or default routes or a dynamic routing protocol needs to be configured inside this VPN in order to get appropriate next-hop information so the control plane can be established and IPsec tunnel traffic can reach remote sites.
- VPN 512 is the management VPN. It carries the out-of-band management traffic to and from the Cisco SD-WAN devices. This VPN is ignored by OMP and not carried across the overlay network.
In addition to the default VPNs that are already defined, one or more service-side VPNs need to be created that contain interfaces that connect to the local-site network and carry user data traffic. It is recommended to select service VPNs in the range of 1-511, but higher values can be chosen as long as they do not overlap with default and reserved VPNs. Service VPNs can be enabled for features such as OSPF or BGP, Virtual Router Redundancy Protocol (VRRP), QoS, traffic shaping, or policing. User traffic can be directed over the IPsec tunnels to other sites by redistributing OMP routes received from the vSmart controllers at the site into the service-side VPN routing protocol. In turn, routes from the local site can be advertised to other sites by advertising the service VPN routes into the OMP routing protocol, which is sent to the vSmart controllers and redistributed to the other WAN Edge routers in the network.
The following figure demonstrates VPNs on a WAN Edge router. The interfaces, Int0 and Int2, are part of the transport VPN; Int1 and Int3 are part of the service VPN, which is attached to the local network at the site; and the mgmt0 port is part of VPN 512.
| Tech tip |
| Note that any interface could also be a subinterface. In that case, the main (or parent) physical interface that the subinterface belongs to must be configured in VPN 0. The subinterface MTU also must be 4 bytes lower than the physical interface due to the 802.1Q tag. To fulfill this requirement, set the main interface MTU to 1504 and leave the subinterface MTU at default (1500). |
Figure 11. VPNs on a WAN Edge router
Note: The above illustrates how VPNs are represented directly on the vEdge router and through the vManage configuration. When configurations get pushed from vManage to the IOS XE SD-WAN routers, they are automatically converted into a format accepted by the IOS XE SD-WAN software parser. Some differences include:
- VRF terminology is used instead of the VPN keyword
- The global table is used to represent VPN 0
- VRF Mgmt-intf is enabled by default on the management interface and is used to represent VPN 512
| Tech tip |
| While IOS XE routers accept names for VRF definitions, with IOS XE SD-WAN code, VRF definitions must be numbers only. |
Control Plane
Control Connections
Cisco SD-WAN vManage and vSmart controllers initially contact and authenticate to the vBond controller, forming persistent DTLS connections, and then subsequently establish and maintain persistent DTLS/TLS connections with each other. WAN Edge devices onboard in a similar manner, but drop the transient vBond connection and maintain DTLS/TLS connections with the vManage and vSmart controllers. The following diagram illustrates this:
Figure 12. SD-WAN control connections
| Tech tip |
| Control connections to vBond are always DTLS. By default, connections to vManage and vSmart are DTLS as well, but this can be changed on any device by configuring TLS for the security control protocol. If one device is configured for TLS and another device is configured for DTLS, TLS is chosen for the control connection between the two devices. TLS is recommended since it uses TCP, which uses acknowledgments for greater reliability. TCP is also connection-oriented, so firewalls can maintain the state of the connections and allow return traffic without explicitly having to allow the traffic. |
Note: Each core (up to a maximum of 8) on the vManage and vSmart initiates and maintains a control connection to each vBond (which has a single core), while a single connection is maintained between the vManage and each vSmart controller. If a vSmart has 2 vCPUs (which translates into 2 cores), for example, there will be 2 total control connections maintained from the vSmart to each vBond, one from each core. If a vManage has 4 vCPUs (which translates to 4 cores), there will be 4 total control connections maintained from the vManage to each vBond, one from each core. Only one control connection is formed between each vSmart to vSmart controller, and vManage to vManage server. No control connections are formed between redundant vBond orchestrators.
WAN Edge Control Connections
The WAN Edge router tries to establish control connections over all provisioned transports by default, first initiating contact with the vBond orchestrator over each transport before attempting to connect to the other controllers. Only one vBond control connection is made per transport when multiple vBond orchestrators exist. Transports are tried one at a time, typically starting with the transport connected to the lowest port number. The WAN Edge router establishes a permanent connection to the vSmart controller over each transport, and establishes a single, permanent connection to vManage over only one transport, the first one which establishes a connection. The vBond connection is then terminated. Note that a WAN Edge router does not have to connect to every vSmart controller, it depends on the network redundancy design and configurations. Technically, a single connection to a vSmart controller over one transport is sufficient for a WAN Edge router to receive control plane information, but for redundancy purposes, additional vSmart controllers over multiple transports are typically configured. When a WAN Edge router connects to a vManage cluster, the control connection is hashed to one vManage instance and does not need to establish connections with all members.
| Tech tip |
| If all vSmart controller connections are lost, the WAN Edge router continues to operate with the latest control plane information for the length of the OMP graceful restart timer (12 hours by default). |
Once a secure connection is built, NETCONF is used by the vManage to provision the WAN Edge device, and OMP peering is established between the vSmart and WAN Edge. Note that OMP peering is established using the system IPs and only one peering session is established between a WAN Edge device and a vSmart controller, even if multiple DTLS/TLS connections exist.
| Tech tip |
| Since there is only one transport used for the connection to vManage, you can influence the transport preference by setting the vmanage-connection-preference parameter to a higher value under the tunnel interface. The default value is 5. The value 0 is used to indicate that a connection is never made to vManage over the tunnel. This is often implemented on metered links, like LTE. |
Figure 13. WAN Edge control connections
Control Connection Summary
The following summarizes the control connections for controllers and WAN Edge routers:
- Permanent DTLS connections between each vSmart core (up to 8) and each vBond orchestrator
- Permanent DTLS connections between each vManage core (up to 8) and each vBond orchestrator
- A permanent TLS or DTLS connection between each vManage and each vSmart controller
- Full mesh of TLS or DTLS connections between vSmart controllers (1 connection between each pair)
- Full mesh of TLS or DTLS connections between vManage cluster instances (1 connection between each pair)*
- Temporary DTLS connection between each WAN Edge and one vBond – one connection on each transport
- Permanent TLS or DTLS connection between each WAN Edge and one vManage instance – only one connection over one transport is chosen
- Permanent TLS or DTLS connections between each WAN Edge and two vSmart controllers by default – connections to each over each transport**
*For vManage cluster instances, some instances that are dedicated to statistics as an example and do not handle WAN Edge devices can be configured without tunnel interfaces and thus, no control connections are built to those instances.
**For vSmart controllers, the number of connections depend on the max-control-connections and max-omp-sessions configurations on the WAN Edge router.
Authorized List Model
All WAN Edge devices and controllers mutually authenticate each other using an authorized list model, where the devices have to be authorized before establishing connections and being allowed access onto the network.
There are two authorized lists that are distributed by vManage, one for the controllers and one for WAN Edge devices.
- Authorized controller list: The authorized controller list is a result of the administrator adding the controllers manually into the vManage user interface. This list can be distributed from the vManage to the controllers and subsequently, from the vBond to the vSmart controllers.
- Authorized serial number list for WAN Edge devices: The digitally-signed authorized serial number list for the WAN Edge devices can be modified and retrieved from the Plug and Play Connect portal at http://software.cisco.com. The list can be retrieved manually or synced automatically from vManage by a user with a valid Cisco CCO account with access to the proper Smart Account and Virtual Account for the SD-WAN overlay. After the file is uploaded or synced to vManage, it is distributed by vManage to all the controllers.
With the WAN Edge authorized serial number list, the administrator can decide and configure the identity trust of each individual WAN Edge router. The options are:
◦ Valid: The router is authorized to be fully operational in the SD-WAN network.
◦ Invalid: The router is not authorized in the SD-WAN network, so no control connections form with the controllers.
◦ Staging: The router can authenticate and form control connections with the controllers, but OMP does not send any routes, data policies, or TLOCs to the WAN Edge router, so traffic is not forwarded. This state allows you to provision and test a router before allowing it to join the production SD-WAN network.
When the WAN Edge authorized serial number list is loaded or synced to vManage, there is an option to validate devices. If you select the checkbox to validate the devices before the list is imported, all devices are Valid by default. If you do not select the checkbox to validate, all devices are Invalid by default, and you must configure each to Valid before a router can form control connections with the controllers and join the SD-WAN network.
Figure 14. Authorized controller and WAN Edge serial number lists
Identity
Authentication between devices involves validating device identity via certificates.
How device certificate validation works:
- The client device presents a CA-signed device certificate to the server.
- The server validates the certificate signature by
- Running a hash algorithm on the certificate data to get a value, and
- Decrypting the certificate signature with the public key obtained from the CA Root certificate to get a second value
If both values are equal, then the signature is valid.
- The client device is now trusted and the client public key can be trusted for use in encryption.
Figure 15. Validating device identity via certificates
Note that the corresponding root certificate is required in order to validate device certificates.
Controller Identity
Controller identity is provided by a Symantec/Digicert or Cisco-signed certificate, or alternatively, an Enterprise CA certificate. Each controller in the network must have a certificate signed and installed. In addition, the root certificate chain for the corresponding CA must also be installed for each controller before the controller certificates can be installed. Additional root chains are installed in order to validate the device certificate of devices that do not use the same CA root. Some root certificate chains are pre-loaded or automatically installed, and others, like the Enterprise root CA, must be installed by an administrator.
WAN Edge Router Identity
Identity for vEdge hardware routers is provided by a device certificate signed by Avnet, generated during the manufacturing process and burned into the Trusted Platform Module (TPM) chip. The Symantec/Digicert and Cisco root certificates are pre-loaded in software for trust for the controllers’ certificates. Additional root certificates may either be loaded manually, distributed automatically by vManage, or installed during the ZTP automatic provisioning process.
Identity for IOS XE SD-WAN hardware routers, with the exception of the ASR 1002-X, is provided by the Secure Unique Device Identifier (SUDI), which is an X.509v3 certificate associated with a key pair that is protected in hardware. The Symantec/Digicert and Cisco root certificates are pre-loaded in software for trust for the controllers’ certificates. Additional root certificates may either be loaded manually, distributed automatically by the vManage NMS, or installed during the Plug-and-Play (PnP) automatic provisioning process.
vEdge cloud routers, ISRv routers, CSR1000v routers, and Cisco ASR 1002-X routers do not have device certificates pre-installed. Each device uses a One Time Password (OTP)/Token that is generated by vManage and configured during device deployment for the purpose of a temporary identity. Once the device is temporarily authenticated, a permanent identity is provided by vManage, which can operate as a Certificate Authority (CA) to generate and install certificates for these devices.
The figure below shows:
- The vManage acting as a Certificate Authority (CA) for WAN Edge cloud routers and the ASR 1002-X.
- vManage distributes the Viptela root certificate to vBond and vSmart in order for them to validate the WAN Edge cloud identity.
- Once the WAN Edge routers are authenticated via OTP, the vManage CA issues them Viptela-signed certificates that are used from then on for authentication.
Note that if there is a vManage cluster, each vManage signs a certificate for the device and distributes the corresponding root certificate.
Figure 16. vManage root CA for WAN Edge cloud routers and the ASR1002-X
Certificates
The following illustrates the device and root certificates installed for authentication for various Cisco SD-WAN devices. In this example, Symantec/Digicert certificates are installed on the controllers. Cisco certificates or Enterprise CA certificates could alternatively be used. Using Enterprise CA certificates require the Enterprise CA root chain to be installed on all SD-WAN devices, which can be manually installed or distributed automatically to WAN Edge devices via ZTP or PnP.
Note that the following only illustrates certificates used in this authentication example. Additional root certificates, such as Cisco root certificates, may also be installed.
- Controllers: A device certificate signed by Digicert is installed for its own identity which uses the SHA 256 algorithm. In software, the following root certificates are present:
◦ Digicert (formerly Symantec) Root Chain: To trust other controller certificates
◦ Avnet Root Chain: To trust vEdge router certificates
◦ Cisco Root Chain: To trust Cisco SUDI router certificates
◦ Viptela Root Chain (vManage): To trust WAN Edge virtual routers and Cisco routers without SUDI certficates
- vEdge Router: A device certificate signed by Avnet is installed during the manufacturing process which uses the SHA 1 algorithm. In software, the Digicert root chain is present in order to trust controller certificates.
- Cisco Router (with SUDI): A device certificate signed by Cisco is installed during the manufacturing process which uses the SHA 256 algorthm. In software, the Digicert root chain is present in order to trust controller certificates.
- Cloud Router and Cisco Routers without SUDI (ASR 1002-X): A device certificate signed by vManage is installed after the One-Time-Password (OTP) authentication which uses the SHA 256 algorithm. In software, the Digicert root chain is present in order to trust controller certificates.
Figure 17. Certificates present for authentication purposes
Authentication/Authorization of SD-WAN Devices
When the controllers authenticate each other and WAN Edge devices, they generally:
- Validate the trust for the certificate root Certificate Authority (CA).
- Compare the organization name of the received certificate OU against the locally configured one (except when authenticating against WAN Edge hardware devices)
- Compare the certificate serial numbers against the authorized serial number list distributed from vManage (except when authenticating against vBond)
When WAN Edge devices authenticate to the controllers, they:
- Validate the trust for the certificate root Certificate Authority (CA).
- Compare the organization name of the received certificate OU against the locally configured one.
After authentication and authorization succeeds, a DTLS/TLS connection is established.
The following diagrams illustrate how different devices authenticate with each other using Symantec/Digicert or Cisco certificates. Enterprise CA certificates operate in the same manner. Note that typically:
- Controllers and WAN Edge devices act as clients to initiate connections with the vBond, which acts as a server
- vManage controllers act as clients to initiate connections with the vSmart, which acts as a server
- vSmart controllers act as clients to initiate connections with other vSmart controllers and the one with the highest public IP address acts as a server
- WAN Edge devices act as clients to initiate connections with vManage and vSmart controllers, which act as servers
Figure 18. Authentication and authorization of SD-WAN devices
For information on deploying certificates for the Cisco SD-WAN solution, refer to the Cisco SD-WAN Controller Certificates and Authorized Serial Number File Deployment Guide at https://www.cisco.com/c/dam/en/us/td/docs/solutions/CVD/SDWAN/cisco-sd-wan-certificates-deploy-2019sep.pdf