Escali 4.4 2.3 Communication protocols on DAT-devices

Section: 2.3 Communication protocols on DAT-devices

Scali MPI Connect Release 4.4 Users Guide 16

This is a built-in device that uses the Scali SCI driver and library (ScaSCI). This driver is for the

Dolphin SCI network cards. Please see the ScaSCI Release Notes for specific requirements. This

device is straight forward and requires no configuration itself, but for multi-dimensional toruses

(2D and 3D) the Scali SCI Management system (ScaConf) needs to be running somewhere in

your system. Refer to Appendix C for installation and configuration of the Scali SCI

Management software.

2.3 Communication protocols on DAT-devices

In SMC, the communication protocol used to transfer data between a sender and a receiver

depends on the size of the message to transmit, as illustrated in Figure 2-3:.

Figure 2-3: Thresholds for different communication protocol

The default thresholds that control whether a message belongs to the inlining, eagerbuffering

or transporter protocols can be controlled from the application launch program (mpimon)

described in chapter 3.

Figure 2-4: illustrates the node resources associated with communication and mechanisms

implemented in Scali MPI Connect for handling messages of different sizes. The three

communication protocols from Figure 2-3: rely on buffers located in the main memory of the

nodes. This memory is allocated as shared, i.e., it is not private to a particular process in the

node. Each process has one set of receiving buffers for of the processes it communicates with.

As the figure shows all communication relies on the sending process depositing messages

directly into the communication buffers of the receiver. For Inline and Eagerbuffering the

management of the buffer resources does not require participation from the receiving process,

because of their designs as ring buffers.

The Channel ringbuffer is divided into equally sized entries. The size varies differs for different

architectures and networks; see “Scali MPI Connect Release Notes” for details. An entry in the

ringbuffer, which is used to hold the information forming the message envelope, is reserved

each time a message is being sent, and is used by the inline protocol, the eagerbuffering

protocol, and the transporter protocol. In addition, one ore more entries are used by the inline

protocol for application data being transmitted.

Transporter protocol:

message size > eager_size

Eagerbuffering protocol:

channel_inline_threshold < message size <= eager_size

Inlining protocol

0 <= message size <= channel_inline_threshold

Increasing

message size