Escali 4.4 Built-in-tools for debugging, Assistance for external profiling, Legal options are

Page 42

Section: 3.7 Debugging and profiling

3.7.2 Built-in-tools for debugging

Built-in tools for debugging in Scali MPI Connect covers discovery of the MPI calls used through tracing and timing, and an attachment point to processes that fault with segmentation violation. The tracing and timing is covered in Chapter 4.

3.7.2.1 Using built-in segment protect violation handler

When running applications that terminate with a SIGSEGV-signal it is often useful to be able to freeze the situation instead of exiting, the default behavior. The built-in SIGSEGV-handler can be made to do this by defining the environment-variable

SCAMPI_INSTALL_SIGSEGV_HANDLER.

Legal options are:

6.The handler dumps all registers and starts looping. Attaching with a debugger will then make it possible to examine the situation which resulted in the segment protect violation.

7.The handler dumps all registers but all processes will exit afterwards.

All other values will disable the installation of the handler.

To attach to process <pid> on a machine with the GNU debugger (gdb) do;

user% gdb /proc/<pid>/exe <pid>

In general, this will allow gdb to inspect the stack trace and identify the functions active when the sigsegv occurred, and disssasemble the functions. If the application is compiled with debug info (-g) and the source code is available, then source level debugging can be carried out.

3.7.3 Assistance for external profiling

Profiling parallel applications is complicated by having multiple processes at the same time. But Scali MPI Connect comes to assistance; through the SCAMPI_PROFILE_APPLICATION environment variable together with the -separate_outputoption (SCAMPI_SEPARATE_OUTPUT) the output from the application runs is directed at one file per process for easier use.

The environment variables SCAMPI_PROFILE_APPLICATION_START and

SCAMPI_PROFILE_APPLICATION_END are also available for steering the range of memory addresses applicable to profiling.

3.7.4 Debugging with Etnus Totalview

SMC applications can be debugged using the Etnus Totalview, see http://www.etnus.com for more information about this product.

To start the Etnus Totalview debugger with a Scali MPI application, use the tvmpimon wrapper script. This wrapper script accepts the regular options that mpimon accepts, and sets up the environment for Totalview. The totalview binary must be in the search path when launching tvmpimon. If the mpirun script is the preferred way of starting jobs, it accepts the “standard” -tv option, however the same rules applies with regards to the search path.

Scali MPI Connect Release 4.4 Users Guide

30

Image 42
Contents Scali MPI ConnectTM Users Guide Acknowledgement Copyright 1999-2005 Scali AS. All rights reservedScali Bronze Software Certificate Maintenance II Software License Terms CommencementGrant of License Sub-license and distribution SupportLicense Manager Export RequirementsSCALI’s Obligations LICENSEE’s ObligationsTitle to Intellectual Property Rights TransferWarranty of Title and Substantial Performance Compliance with LicensesLimitation on Remedies and Liabilities Scali MPI Connect Release 4.4 Users Guide ViiProprietary Information MiscellaneousGoverning Law Scali MPI Connect Release 4.4 Users Guide Table of contents Profiling with Scali MPI Connect Appendix a Example MPI code Scali MPI Connect Release 4.4 Users Guide Chapter Scali MPI Connect product contextSupport Scali mailing lists SMC FAQ SMC release documentsProblem reports Platforms supportedLicensing How to read this guideAcronyms and abbreviations FeedbackNIC Terms and conventions Typographic conventionsGUI style font Typographic conventions Description of Scali MPI Connect Scali MPI Connect componentsSMC network devices Shared Memory Device Direct Access Transport DATNetwork devices Ethernet DevicesUsing detctl Using detstat3.2 DET 4.1 GM MyrinetInfiniband 5.1 IBCommunication protocols on DAT-devices 6 SCIChannel buffer Inlining protocol Eagerbuffering protocolTransporter protocol MPI-2 Features Support for other interconnectsZerocopy protocol Scali MPI Connect Release 4.4 Users Guide MPI-2 Features Scali MPI Connect environment variables Setting up a Scali MPI Connect environmentCompiling and linking RunningCompiler support Linker flagsRunning Scali MPI Connect programs Naming conventionsMpimon monitor program Basic usageIdentity of parallel processes Standard output Controlling options to mpimonStandard input Program specHow to provide options to mpimon Network optionsMpirun wrapper script Mpirun usageRunning with dynamic interconnect failover capabilities Running with tcp error detection TfdrSuspending and resuming jobs Part partDebugging and profiling Debugging with a sequential debuggerAssistance for external profiling Using built-in segment protect violation handlerBuilt-in-tools for debugging Debugging with Etnus TotalviewControlling communication resources Communication resources on DAT-devicesChannelinlinethreshold size to set threshold for inlining Good programming practice with SMC Using MPIIsend, MPIIrecvUsing MPIBsend Matching MPIRecv with MPIProbeFatal errors Error and warning messagesUser interface errors and warnings Unsafe MPI programsMpimon options Giving numeric values to mpimon PrefixPostfix Scali MPI Connect Release 4.4 Users Guide Profiling with Scali MPI Connect ExampleUsing Scali MPI Connect built-in trace TracingAbsRank MPIcallcommNamerankcall-dependant-parameters where +relSecs S eTime whereFeatures ExampleUsing Scali MPI Connect built-in timing TimingMPIcallDcallsDtimeDfreq TcallsTtimeTfreq Commrank send to toworldTocommonFields where Using the scanalyzeCommrank recv from fromworldFromcommonFields Count!avrLen!zroLen!inline!eager!transporter! whereFor timing Using SMCs built-in CPU-usage functionality This produces the following reportScali MPI Connect Release 4.4 Users Guide Tuning communication resources Automatic buffer managementCaching the application program on the nodes How to optimize MPI performanceBenchmarking First iteration is very slowCollective operations Memory consumption increase after warm-upFinding the best algorithm Appendix a Programs in the ScaMPItst packageImage contrast enhancement Scali MPI Connect Release 4.4 Users Guide File format OriginalWhen things do not work troubleshooting Why does not my program start to run?Appendix B Why can I not start mpid? Why does my program terminate abnormally?General problems Per node installation of Scali MPI Connect Appendix CInstall Scali MPI Connect for Myrinet Install Scali MPI Connect for TCP/IPInstall Scali MPI Connect for Direct Ethernet ExampleInstall and configure SCI management software Install Scali MPI Connect for InfinibandInstall Scali MPI Connect for SCI License optionsUninstalling SMC Troubleshooting Network providersScali kernel drivers Troubleshooting 3rdparty DAT providers Troubleshooting the GM providerScali MPI Connect Release 4.4 Users Guide Appendix D Bracket expansion and grouping Bracket expansionGrouping Scali MPI Connect Release 4.4 Users Guide Appendix E Related documentationScali MPI Connect Release 4.4 Users Guide List of figures Scali MPI Connect Release 4.4 Users Guide Index Transporter protocolSSP