Escali 4.4 3.7 Debugging and profiling

Section: 3.7 Debugging and profiling

Scali MPI Connect Release 4.4 Users Guide 29

As this feature is limited to tcp communication only, it will not have any effect when using

native RDMA drivers such as Infiniband or Myrinet. Note that the combination of tfdr and

failover mode is not supported in this version of Scali MPI Connect.

Data errors will be logged using the standard syslog mechanism.

3.7 Debugging and profiling

The complexity of debugging programs can grow dramatically when going from serial to parallel

programs. So to assist in debugging MPI programs Scali MPI Connect has a nnumber of

features built-in, like starting processes directly in a debugger and tracing proceses’ traffic

patterns.

3.7.1 Debugging with a sequential debugger

SMC applications can be debugged using a sequential debugger. By default, the GNU debugger

gdb is invoked by mpimon. If another debugger is to be used, specify the debugger using the

mpimon option -debugger <debugger>.

To set debug-mode for one or more MPI-processes, specify the MPI-process(es) to debug using

the mpimon option -debug <select>. In addition, note that the mpimon option -display

<display> should be used to set the display for the xterm terminal emulator. An xterm

terminal emulator, and one debugger, is started for each of the MPI-processes being debugged.

For example, to debug an application using the default gdb debugger do:

user% mpimon -debug all <application +parameters> -- <node specification>

Initially, for both MPI-process 0 and MPI-process 1, an xterm window is opened. Next, in the

upper left hand corner of each xterm window, a message containing the application program’s

run parameter(s) is displayed. Typically, the first line reads Run parameters: run

<programoptions>. The information following the colon, i.e. run <programoptions>, is

needed by both the debugger and the SMC application being debugged. Finally, one debugger

is started for each session. In each debugger's xterm window, first input the appropriate

debugging action before the MPI-process is started. Then, when ready to run the MPI-process,

paste <programoptions> into the debugger to start running.

Figure 3-1: /opt/scali/bin/mpirun -debug all ./kollektive-8 ./ultrasound_fetus-256x256-8.pgm

Contents

Main Acknowledgement SCALI BRONZE SOFTWARE CERTIFICATE Page Page Page Page Page Page Page Page Page Table of contents Chapter 1 Introduction ....................................................................5 Chapter 2 Description of Scali MPI Connect ...................................11 Chapter 3 Using Scali MPI Connect ................................................21 Chapter 4 Profiling with Scali MPI Connect.................................... 37 Chapter 5 Tuning SMC to your application ..................................... 47 Page Page Chapter 1 Introduction 1.1 Scali MPI Connect product context 1.2 Support 1.2.1 Scali mailing lists 1.2.2 SMC FAQ 1.2.3 SMC release documents 1.2.4 Problem reports 1.3 How to read this guide 1.4 Acronyms and abbreviations Page 1.5 Terms and conventions 1.6 Typographic conventions Page Chapter 2 Description of Scali MPI Connect 2.1 Scali MPI Connect components 2.2 SMC network devices 2.2.1 Network devices 2.2.1.1 Direct Access Transport (DAT) 2.2.2 Shared Memory Device 2.2.3 Ethernet Devices 2.2.3.1 TCP 2.2.3.2 DET 2.2.3.3 Using detctl 2.2.3.4 Using detstat 2.2.4 Myrinet 2.2.4.1 GM 2.2.5 Infiniband 2.2.5.1 IB 2.2.6 SCI 2.3 Communication protocols on DAT-devices 2.3.1 Channel buffer Page 2.3.5 Zerocopy protocol 2.4 Support for other interconnects 2.5 MPI-2 Features Page Page Chapter 3 Using Scali MPI Connect 3.1 Setting up a Scali MPI Connect environment 3.1.1 Scali MPI Connect environment variables 3.2 Compiling and linking 3.2.1 Running 3.2.2 Compiler support 3.2.3 Linker flags 3.2.4 Notes on Compiling and linking on AMD64 and EM64T 3.2.5 Notes on Compiling and linking on Power series 3.2.6 Notes on compiling with MPI-2 features 3.3 Running Scali MPI Connect programs 3.3.1 Naming conventions 3.3.2 mpimon - monitor program 3.3.2.1 Basic usage 3.3.2.2 Identity of parallel processes 3.3.2.3 Controlling options to mpimon 3.3.2.4 Standard input 3.3.2.5 Standard output 3.3.2.6 How to provide options to mpimon 3.3.2.7 Network options 3.3.3 mpirun - wrapper script 3.3.3.1 mpirun usage 3.4 Suspending and resuming jobs 3.5 Running with dynamic interconnect failover capabilities 3.6 Running with tcp error detection - TFDR 3.7 Debugging and profiling 3.7.1 Debugging with a sequential debugger 3.7.2 Built-in-tools for debugging 3.7.2.1 Using built-in segment protect violation handler 3.7.3 Assistance for external profiling 3.7.4 Debugging with Etnus Totalview 3.8 Controlling communication resources 3.8.1 Communication resources on DAT-devices 3.9 Good programming practice with SMC 3.9.1 Matching MPI_Recv() with MPI_Probe() 3.9.2 Using MPI_Isend(), MPI_Irecv() 3.9.3 Using MPI_Bsend() 3.9.4 Avoid starving MPI-processes - fairness 3.10 Error and warning messages 3.10.1 User interface errors and warnings 3.10.2 Fatal errors 3.11 Mpimon options 3.11.1 Giving numeric values to mpimon Page Chapter 4 Profiling with Scali MPI Connect 4.1 Example 4.2 Tracing Page Page 4.3 Timing Page 4.4 Using the scanalyze Page 4.5 Using SMC's built-in CPU-usage functionality Page Chapter 5 Tuning SMC to your application 5.1 Tuning communication resources 5.2 How to optimize MPI performance 5.3 Benchmarking 5.4 Collective operations Page Appendix A Example MPI code A-1 Programs in the ScaMPItst package A-2 Image contrast enhancement Page Page Appendix B Troubleshooting B-1 When things do not work - troubleshooting B-1.1 Why does not my program start to run? B-1.2 Why can I not start mpid? B-1.3 Why does my program terminate abnormally? B-1.4 General problems Appendix C Install Scali MPI Connect C-1 Per node installation of Scali MPI Connect C-2 Install Scali MPI Connect for TCP/IP C-3 Install Scali MPI Connect for Direct Ethernet C-4 Install Scali MPI Connect for Myrinet C-5 Install Scali MPI Connect for Infiniband C-6 Install Scali MPI Connect for SCI C-7 Install and configure SCI management software C-8 License options C-9 Scali kernel drivers C-10 Uninstalling SMC C-11 Troubleshooting Network providers C-11.1 Troubleshooting 3rdparty DAT providers C-11.2 Troubleshooting the GM provider Page Appendix D Bracket expansion and grouping D-1 Bracket expansion D-2 Grouping Page Appendix E Related documentation Page Page Page Index