Escali 4.4 Network devices, Shared Memory Device, Ethernet Devices, Direct Access Transport DAT

Section: 2.2 SMC network devices

library, which in turn may (e.g. Myrinet or SCI) or may not require a kernel driver (e.g. TCP/IP). These provider libraries provide a network device to SMC.

2.2.1 Network devices

There are two basic types of network devices in SMC, native and DAT. The native devices are built-in and are neither replaceble nor upgradable without replacing the Scali MPI Connect package. There are currently five built in devices, SMP, TCP, IB, GM and SCI; the Release Notes included with the Scali MPI Connect package should have more details on this issue.

To find out what network device is used between two processes, set the environment variable SCAMPI_NETWORKS_VERBOSE=2. With value 2 the MPI library will print out during startup a table over every process and what device it's using to every other process.

2.2.1.1 Direct Access Transport (DAT)

The other type of devices use the DAT uDAPL API in order to have an open API for generic third party vendors. uDAPL is an abbrevation for User DAT Provider library. This is a shared library that SMC loads at runtime through the static DAT registry. These libraries are normally listed in /etc/dat.conf. For clusters using ‘exotic’ interconnects whose vendor provides a uDAPL shared object, these can be added to this file (if this isn’t done automatically by the vendor). The device name is given by the uDAPL, and the interconnect vendor must provide it.

Please note that Scali has a certification program, and may not provide support for unknown third party vendors.

The DAT header files and registry library conforming to the uDAPL v1.1 specification, is provided by the dat-registry package.

For more information on DAT, please refer to http://www.datcollaborative.org.

2.2.2 Shared Memory Device

The SMP device is a shared memory device that is used exclusively for intra-node communication and use SYS V IPC shared memory. Mulit CPU nodes are frequent in clusters, and SMP provide optimal communication between the CPUs. In cases where only one processor per node is used, SMP is not used.

2.2.3 Ethernet Devices

An Ethernet for networking is a basic requirement fpr a cluster. For some uses this also has enough performance for carrying application communication. To serve this Scali MPI Connect has a TCP device. In addition there are Direct Ethernet Transport (DET) devices which implement a protocol devised by Scali for aggregating multiple TCP-type interconnects.

2.2.3.1 TCP

The TCP device is really a generic device that works over any TCP/IP network, even WANs. This network device requires only that the node names given to mpimon map correctly to the nodes IP address. TCP/IP connectivity is required for SMC operation, and for this reason the TCP device is always perational.

Note: Users should always append the TCP device at the end of a devicelist as the device of last resort. This way communication will fall back to the management ethernet that anyway has to be present for the cluster to work.

Scali MPI Connect Release 4.4 Users Guide 13