Escali 4.4 manual Myrinet, Infiniband, 4.1 GM, 5.1 IB

Section: 2.2 SMC network devices

•root# detstat -r det0 # reset statistics for the det0 device.

•root# detstat -r -a # resets statistics for all DET devices.

2.2.4 Myrinet

2.2.4.1 GM

This is a RDMA capable device that uses the Myricom GM driver and library. A GM release above

2.0is required. This device is straight forward and requires no configuration other than the presence of the libgm.so library in the library path (see /etc/ld.so.conf).

Note:

Myricom GM software is not provided by Scali. If you have purchased a Myrinet interconnect you have the right to use the GM source, and a source tar ball is available from Myricom. It is necessary to obtain the GM source since it must be compiled per kernel version. Scali provides tools for generating binary RPMs to ease installing and management. These tools are provided in the scagmbuilder package; see the Release Notes/Readme file for detailed instructions.

If you used Scali Manage to install your compute nodes, and supplied it with the GM source tar ball, the installation is already complete.

2.2.5 Infiniband

2.2.5.1 IB

Infiniband is a relatively new interconnect that has been available since 2002, and became affordable in 2003. On PCI-X based systems you can expect latencies around 5υS and

bandwidth up to 700-800Mb/s (please note that performance results may vary based on processors, memory sub system, and the PCI bridge in the chipsets).

There are various Infiniband vendors that provide slightly different hardware and software environments. Scali have established relationships with the following vendors: Mellanox, Silverstorm, Cisco, and Voltaire.

See release notes on the exact versions of software stack that is supported. Scali provide a utility known as ScaIBbuilder that does an automated install of some of these stacks. (See IBbuilders release notes).

The different vendors’ InfiniBand switches vary in feature sets, but the most important difference is whether they have a built in subnet manager or not. An InfiniBand network must have a subnet manager (SM) and if the switches don't come with a builtin SM, one has to be started on a node attached to the IB network. The SMs of choice for software SMs are OpenSM or minism. If you have SM-less switches your vendor will provide one as part of their software bundle.

SMC uses either the uDAPL (User DAT Provider Library) supplied by the IB vendor, or the low level VAPI/IBA layer. DAT is an established standard and is guaranteed to work with SMC. However better performance is usually achieved with the VAPI/IBT interfaces. However, VAPI is an API that is in flux and SMC is not guaranteed to work with all (current nor future) versions of VAPI.

Scali MPI Connect Release 4.4 Users Guide 15