SAN Design: March 29, 2001 3:18 pm
53-0001575-01 BROCADE Technical Note Page: 21 of 31
3.0 Additional Considerations in Fabric Design and Implementation
3.1 FABRIC BRING UP
A large (greater than 8) switch fabric will require care and planning on initial power up and for expanding switch elements. Bro-
cade has developed an extensive planning document to allow SAN administrators to successfully bring up, add to, delete from, and gener-
ally maintain a large switch fabric. Consult the following reference document on the Brocade Web Site for these details:
•Building and Scaling BROCADE SAN Fabrics: Large Fabric Bring Up and Maintenance Guide, Doc. # 53-
0001573-01
3.2 INTER-SWITCH LINKS (ISL)
Inter-switch links (E-port connections) provide for connection between switches. Multiple links can be setup between switches. Any switch
in the fabric can have one or more links to another switch in the fabric. At initial start-up of the switch, the links are allocated in a round-
robin fashion to balance the load on the system. On addition or deletion of an ISL, the Dynamic Load Sharing capability is used to re-com-
pute routes in the traffic to balance ports across ISL paths. (In the current BROCADE switch versions, dynamic load balancing after path
initialization does not exist across these links.The BROCADE switch guarantees in-order delivery and dividing traffic from a given port
among available ISLs will violate the in-order delivery we enforce.)
•Some features of the SilkWorm can be used to increase inter-switch bandwidth. Adding an ISL between switches is dynamic and
can be done while the switch is active. Adding a new ISL will result in a routing re-computation and new allocation of ISL links
between source and destination ports. Similarly, removing a link will result in routing re-computation across the fabric and possible
fabric re-configuration. RSCNs are generated following these events sometimes causing I/O traffic to pause as nodes re-
authenticate with the fabric.
•Adding ISL’s will cause routing traffic/zoning data to be updated across each ISL. The number of ISL’s is not so relevant as changing
ISL configuration--as each change will result in a re-calculation of routes in the fabric. When numerous fabric reconfigurations occur
(removing or adding links, downing/upping a switch, etc.) the load on the switches CPUs will be increased and some fabric events
may time out waiting on CPU response. While fabric reconfiguration is taking place, data is held pending the completion of new
route calculations. Extensive changes on an ongoing basis of ISLs can effect data traffic. Adding or deleting ISL’s should be done
on a planned basis when SAN utilization is low and grouped in a manner to minimize impact on the fabric.
•No more than 8 ISLs between any two switches is supported. More than 8 ports can be used on a switch for ISL traffic as long as
no more than 8 go to a single adjacent switch. A sample of a switch star inter-connect is shown in Figure 1.