Escali 4.4 2.2.4 Myrinet, 2.2.5 Infiniband

Section: 2.2 SMC network devices

Scali MPI Connect Release 4.4 Users Guide 15

• root# detstat -r det0 # reset statistics for the det0 device.

• root# detstat -r -a # resets statistics for all DET devices.

2.2.4 Myrinet

This is a RDMA capable device that uses the Myricom GM driver and library. A GM release above

2.0 is required. This device is straight forward and requires no configuration other than the

presence of the libgm.so library in the library path (see /etc/ld.so.conf).

Note:

Myricom GM software is not provided by Scali. If you have purchased a Myrinet interconnect

you have the right to use the GM source, and a source tar ball is available from Myricom. It is

necessary to obtain the GM source since it must be compiled per kernel version. Scali provides

tools for generating binary RPMs to ease installing and management. These tools are provided

in the scagmbuilder package; see the Release Notes/Readme file for detailed instructions.

If you used Scali Manage to install your compute nodes, and supplied it with the GM source tar

ball, the installation is already complete.

2.2.5 Infiniband

Infiniband is a relatively new interconnect that has been available since 2002, and became

affordable in 2003. On PCI-X based systems you can expect latencies around 5υS and

bandwidth up to 700-800Mb/s (please note that performance results may vary based on

processors, memory sub system, and the PCI bridge in the chipsets).

There are various Infiniband vendors that provide slightly different hardware and software

environments. Scali have established relationships with the following vendors: Mellanox,

Silverstorm, Cisco, and Voltaire.

See release notes on the exact versions of software stack that is supported. Scali provide a

utility known as ScaIBbuilder that does an automated install of some of these stacks. (See

IBbuilders release notes).

The different vendors’ InfiniBand switches vary in feature sets, but the most important

difference is whether they have a built in subnet manager or not. An InfiniBand network must

have a subnet manager (SM) and if the switches don't come with a builtin SM, one has to be

started on a node attached to the IB network. The SMs of choice for software SMs are OpenSM

or minism. If you have SM-less switches your vendor will provide one as part of their software

bundle.

SMC uses either the uDAPL (User DAT Provider Library) supplied by the IB vendor, or the low

level VAPI/IBA layer. DAT is an established standard and is guaranteed to work with SMC.

However better performance is usually achieved with the VAPI/IBT interfaces. However, VAPI

is an API that is in flux and SMC is not guaranteed to work with all (current nor future) versions

of VAPI.