Escali 4.4 manual Controlling communication resources, Communication resources on DAT-devices

Section: 3.8 Controlling communication resources

3.8 Controlling communication resources

Even though it is normally not necessary to set buffer parameters when running applications, it can be done, e.g., for performance reasons. Scali MPI Connect automatically adjusts communication resources based on the number of processes in each node and based on pool_size and chunk_size.

The built-in devices SMP and TCP/IP use a simplified protocol based on serial transfers. This can be visualized as data being written into one end of a pipe and read from the other end. Messages arriving out-of-order are buffered by the reader. The names of these standard devices are SMP for intra-node-communication and TCP for node-to-node-communication.

The size of the buffer inside the pipe can be adjusted by setting the following environment variables:

•SCAFUN_TCP_TXBUFSZ - Sets the size of the transmit buffer.

•SCAFUN_TCP_RXBUFSZ - Sets the size of the receive buffer.

•SCAFUN_SMP_BUFSZ - Sets the size of the buffer for intranode-communication.

The ringbuffers are divided into equally sized entries. The size varies differs for different architectures and networks; see “Scali MPI Connect Release Notes” for details. An entry in the ringbuffer, which is used to hold the information forming the message envelope, is reserved each time a message is being sent, and is used by the inline protocol, the eagerbuffering protocol, and the transporter protocol. In addition, one ore more entries are used by the inline protocol for application data being transmitted.

mpimon has the following interface for the eagerbuffer and channel thresholds:

•Channel threshold definitions

-channel_inline_threshold <size> to set threshold for inlining

•Eager threshold definitions

-eager_threshold <size> to set threshold for eager buffering

3.8.1 Communication resources on DAT-devices

All resources (buffers) used by SMC reside in shared memory in the nodes. This way multiple processes (typically when a node has multiple CPUs) can share the communication resources.

SMC operates on a buffer pool. The pool is divided into equally sized parts called chunks. SMC uses one chunk per connection to other processes. The mpimon option “pool_ size” limits the total size of the pool and the “chunk_size” limits the block of memory that can be allocated for a single connection.

To set the pool size and the chunk size, specify:

-pool_size <size>	to set the buffer pool size
-chunk_size <size>	to set the chunk size

Scali MPI Connect Release 4.4 Users Guide 31