The Cisco ACI fabric bootstrap sequence begins when the Cisco ACI fabric is booted with factory-installed images on all the switches. Cisco Nexus 9000 switches running ACI firmware and Cisco APICs use a reserved overlay for the boot process. This infrastructure space is hard-coded on the switches. The Cisco APIC can connect to a leaf through the default overlay, or it can use a locally significant identifier. The Cisco ACI fabric is brought up in a cascading manner, starting with the leaf node directly attached to the Cisco APIC. Link Layer Discovery Protocol (LLDP) and control-plane IS-IS convergence occurs in parallel to this boot process.

All Cisco ACI fabric management communication within the fabric takes place in the infrastructure space using internal private IP addresses, the address assigned from an internal DHCP pool (TEP pool). Although Tunnel End Points (TEPs) are located inside the fabric, in some scenarios the TEP range may be extended beyond the fabric. Therefore, it is not advisable to use overlapping addresses between the internal TEP range and the external network in your data center. To avoid issues with address exhaustion in the future, Cisco strongly recommends that you allocate a /16 or /17 range for the TEP pool, if possible. This addressing scheme allows Cisco APICs to communicate with fabric nodes and other Cisco APICs in the cluster. The Cisco APICs discover the IP address and node information of other Cisco APICs in the cluster using the LLDP-based discovery process.

In this discovery process, a fabric node is considered active when the APIC and node can exchange heartbeats through the Intra-Fabric Messaging (IFM) process. The APIC also uses the IFM process to push policy to the fabric leaf nodes.

Fabric discovery happens in three stages, as shown in Figure 4-9. The leaf node directly connected to the APIC is discovered in the first stage. The second stage of discovery brings in the spines connected to that initial seed leaf. Then the third stage processes the discovery of the other leaf nodes and APICs in the cluster.

Figure 4-9 ACI Auto Fabric Discovery

Figure 4-9 illustrates the discovery process for switches that are directly connected to the APIC. The steps are as follows:

1. Link Layer Discovery Protocol (LLDP) Neighbor Discovery

2. Tunnel End Point (TEP) IP address assignment to the node from the TEP address pool (the default TEP pool is 10.0.0.0/16)

3. Node software upgraded if necessary, downloading the new software from APIC repository

4. Policy Element IFM setup

Prior to this automated process, a minimal bootstrap configuration must be performed on the Cisco APIC.