Escali 4.4 2.2 SMC network devices

Section: 2.2 SMC network devices

Scali MPI Connect Release 4.4 Users Guide 12

Figure 2-1: illustrates how applications started with mpimon have their communication system

established by a system of daemons on the nodes. This process uses TCP/IP communication

over the networking Ethernet, whereas optional high performance interconnects are used for

communication between processes.

Parameter control is performed by mpimon to check as many of the specified options and

parameters as possible. The user program names are checked for validity, and the nodes are

contacted (using sockets) to ensure they are responding and that mpid is running.

Via mpid mpimon establishes contact with the nodes and transfers basic information to enable

mpid to start the submonitor mpisubmon on each node. Each submonitor establishes a

connection to mpimon for exchange of control information between each mpisubmon and

mpimon to enable mpisubmon to start the specified userprograms (MPI-processes).

As mpisubmon starts all the MPI-processes to be executed they MPI_Init(). Once inside here

the user processes wait for all the mpisubmons inovolved to coordinate via mpimon. Once all

processes are ready mpimon will return a “start running” message to the processes. They will

then return from MPI_Init() and start executing the user code.

Stopping MPI application programs is requested by the user processes as they enter the

MPI_Finalize() call. The local mpisubmon will signal mpimon and wait for mpimon to return a

“all stopped message”. This comes when all processes are waiting in MPI_Finalize(). As the user

processes return from the MPI_Finalize() they release their resources and terminates. Then the

local mpisubmon terminates and eventuall mpimon terinates.

2.2 SMC network devices

Figure 2-2: Scali MPI Connect relies on DAT to interface to a number of interconnects

Beginning with SSP 4.0.0 and SMC 4.0.0 the Scali MPI offers generic support for interconnects.

This does not yet mean that every interconnect is supported out of the box, since SMC still

requires a driver for each interconnect. But from SMC's point of view, a driver is just a

Contents

Main Acknowledgement SCALI BRONZE SOFTWARE CERTIFICATE Page Page Page Page Page Page Page Page Page Table of contents Chapter 1 Introduction ....................................................................5 Chapter 2 Description of Scali MPI Connect ...................................11 Chapter 3 Using Scali MPI Connect ................................................21 Chapter 4 Profiling with Scali MPI Connect.................................... 37 Chapter 5 Tuning SMC to your application ..................................... 47 Page Page Chapter 1 Introduction 1.1 Scali MPI Connect product context 1.2 Support 1.2.1 Scali mailing lists 1.2.2 SMC FAQ 1.2.3 SMC release documents 1.2.4 Problem reports 1.3 How to read this guide 1.4 Acronyms and abbreviations Page 1.5 Terms and conventions 1.6 Typographic conventions Page Chapter 2 Description of Scali MPI Connect 2.1 Scali MPI Connect components 2.2 SMC network devices 2.2.1 Network devices 2.2.1.1 Direct Access Transport (DAT) 2.2.2 Shared Memory Device 2.2.3 Ethernet Devices 2.2.3.1 TCP 2.2.3.2 DET 2.2.3.3 Using detctl 2.2.3.4 Using detstat 2.2.4 Myrinet 2.2.4.1 GM 2.2.5 Infiniband 2.2.5.1 IB 2.2.6 SCI 2.3 Communication protocols on DAT-devices 2.3.1 Channel buffer Page 2.3.5 Zerocopy protocol 2.4 Support for other interconnects 2.5 MPI-2 Features Page Page Chapter 3 Using Scali MPI Connect 3.1 Setting up a Scali MPI Connect environment 3.1.1 Scali MPI Connect environment variables 3.2 Compiling and linking 3.2.1 Running 3.2.2 Compiler support 3.2.3 Linker flags 3.2.4 Notes on Compiling and linking on AMD64 and EM64T 3.2.5 Notes on Compiling and linking on Power series 3.2.6 Notes on compiling with MPI-2 features 3.3 Running Scali MPI Connect programs 3.3.1 Naming conventions 3.3.2 mpimon - monitor program 3.3.2.1 Basic usage 3.3.2.2 Identity of parallel processes 3.3.2.3 Controlling options to mpimon 3.3.2.4 Standard input 3.3.2.5 Standard output 3.3.2.6 How to provide options to mpimon 3.3.2.7 Network options 3.3.3 mpirun - wrapper script 3.3.3.1 mpirun usage 3.4 Suspending and resuming jobs 3.5 Running with dynamic interconnect failover capabilities 3.6 Running with tcp error detection - TFDR 3.7 Debugging and profiling 3.7.1 Debugging with a sequential debugger 3.7.2 Built-in-tools for debugging 3.7.2.1 Using built-in segment protect violation handler 3.7.3 Assistance for external profiling 3.7.4 Debugging with Etnus Totalview 3.8 Controlling communication resources 3.8.1 Communication resources on DAT-devices 3.9 Good programming practice with SMC 3.9.1 Matching MPI_Recv() with MPI_Probe() 3.9.2 Using MPI_Isend(), MPI_Irecv() 3.9.3 Using MPI_Bsend() 3.9.4 Avoid starving MPI-processes - fairness 3.10 Error and warning messages 3.10.1 User interface errors and warnings 3.10.2 Fatal errors 3.11 Mpimon options 3.11.1 Giving numeric values to mpimon Page Chapter 4 Profiling with Scali MPI Connect 4.1 Example 4.2 Tracing Page Page 4.3 Timing Page 4.4 Using the scanalyze Page 4.5 Using SMC's built-in CPU-usage functionality Page Chapter 5 Tuning SMC to your application 5.1 Tuning communication resources 5.2 How to optimize MPI performance 5.3 Benchmarking 5.4 Collective operations Page Appendix A Example MPI code A-1 Programs in the ScaMPItst package A-2 Image contrast enhancement Page Page Appendix B Troubleshooting B-1 When things do not work - troubleshooting B-1.1 Why does not my program start to run? B-1.2 Why can I not start mpid? B-1.3 Why does my program terminate abnormally? B-1.4 General problems Appendix C Install Scali MPI Connect C-1 Per node installation of Scali MPI Connect C-2 Install Scali MPI Connect for TCP/IP C-3 Install Scali MPI Connect for Direct Ethernet C-4 Install Scali MPI Connect for Myrinet C-5 Install Scali MPI Connect for Infiniband C-6 Install Scali MPI Connect for SCI C-7 Install and configure SCI management software C-8 License options C-9 Scali kernel drivers C-10 Uninstalling SMC C-11 Troubleshooting Network providers C-11.1 Troubleshooting 3rdparty DAT providers C-11.2 Troubleshooting the GM provider Page Appendix D Bracket expansion and grouping D-1 Bracket expansion D-2 Grouping Page Appendix E Related documentation Page Page Page Index