Escali 4.4 manual Timing, Using Scali MPI Connect built-in timing

Page 53

From time to time it may be desirable or feasible to trace only one or a few of the processes. Specifying the "-p" options offers the ability to pick the processes to be traced.

All MPI-calls are enabled for tracing by default. To view only a few calls, specify a "-t <call- list>" option; to exclude some calls, add a "-x<call-list>" option. The "-t" will disable all tracing and then enable those calls that match the <call-list>. The matching is done using "regular-posix-expression"-syntax. "-x"will lead to the opposite; first enable all tracing and then disable those call matching <call-list>.

Examples:

"-t MPI_Irecv" : Trace only immediate recv (MPI_Irecv)

"-t isend;irecv;wait" :Trace only MPI_Isend, MPI_Irecv and MPI_Wait

"-t MPI_[b,r,s]*send" : Trace only send-calls (MPI_Send, MPI_Bsend, MPI_Rsend, MPI_Ssend)

"-t i[a-z]*" : Trace only calls beginning with MPI_I

4.3 Timing

Timing will give you information about which MPI routines were called and how long the MPI calls took. This information is printed at intervals set by the user with the “-s n” option, where n if the number of seconds.

4.3.1 Using Scali MPI Connect built-in timing

To use the built-in timing functionality in SMC, the mpimon-option -timing "<options>" must be set, specifying which options are to be applied.

The following options can be specified:

(<...-list> is a semicolon-separated list of Posix-regular-expressions.)

-s<seconds>

print for intervals of <seconds>seconds

-c<calls>

print for intervals of <calls>MPI_calls

-m<mode>

special mode for timing

 

<mode> = ”sync”: Synchronize with MPI_Barrier before starting

 

collective call

-p<selection>

enable for process(es) 'n,m,o..' = (list) or 'n-m' = (range) or 'all'

-f<call-list>

Print after MPI_calls in <call-list>: MPI_call = 'MPI_call' 'call'

-v

verbose

-h

print this list of options

Printing of timing-information can be either at a fixed time-interval, if "-s <seconds>" is specified, or for a fixed number-of-calls-interval, if "-c <calls>" ia used. It is also possible to obtain output after specific MPI-calls by using "-f <call-list>"; see above for details on how to write <call-list>.

The output has two parts: a timing-part and a buffer-statistics-part.

The first part has the following layout:

All lines start with <rank>: where <rank>: is rank within MPI_COMM_WORLD. This part is included to facilitate separation of output (grep).

Example:

user% SCAMPI_TIMING=”-s 1” mpimon ./kollektive-8 ./uf256-8.pgm -- r1 r2

where '<seconds>' is the number of seconds per printout from Scali MPI Connect produces:

1: 13.26.10

-------------#calls

Delta----------

tim/cal

---------Total----------

tim/cal

1: Init+0.002659 s

time

#calls

time

1: MPI_Bcast

2

169.0us

84.5us

2

169.0us

84.5us

Scali MPI Connect Release 4.4 Users Guide

41

Image 53
Contents Scali MPI ConnectTM Users Guide Copyright 1999-2005 Scali AS. All rights reserved AcknowledgementScali Bronze Software Certificate Grant of License MaintenanceII Software License Terms Commencement License Manager SupportSub-license and distribution Export RequirementsLICENSEE’s Obligations SCALI’s ObligationsTransfer Title to Intellectual Property RightsCompliance with Licenses Warranty of Title and Substantial PerformanceScali MPI Connect Release 4.4 Users Guide Vii Limitation on Remedies and LiabilitiesMiscellaneous Proprietary InformationGoverning Law Scali MPI Connect Release 4.4 Users Guide Table of contents Profiling with Scali MPI Connect Appendix a Example MPI code Scali MPI Connect Release 4.4 Users Guide Scali MPI Connect product context ChapterProblem reports Scali mailing lists SMC FAQ SMC release documentsSupport Platforms supportedAcronyms and abbreviations How to read this guideLicensing FeedbackNIC GUI style font Terms and conventionsTypographic conventions Typographic conventions Scali MPI Connect components Description of Scali MPI ConnectSMC network devices Network devices Direct Access Transport DATShared Memory Device Ethernet Devices3.2 DET Using detctlUsing detstat Infiniband Myrinet4.1 GM 5.1 IBChannel buffer Communication protocols on DAT-devices6 SCI Transporter protocol Inlining protocolEagerbuffering protocol Zerocopy protocol MPI-2 FeaturesSupport for other interconnects Scali MPI Connect Release 4.4 Users Guide MPI-2 Features Compiling and linking Setting up a Scali MPI Connect environmentScali MPI Connect environment variables RunningLinker flags Compiler supportNaming conventions Running Scali MPI Connect programsIdentity of parallel processes Mpimon monitor programBasic usage Standard input Controlling options to mpimonStandard output Program specNetwork options How to provide options to mpimonMpirun usage Mpirun wrapper scriptSuspending and resuming jobs Running with tcp error detection TfdrRunning with dynamic interconnect failover capabilities Part partDebugging with a sequential debugger Debugging and profilingBuilt-in-tools for debugging Using built-in segment protect violation handlerAssistance for external profiling Debugging with Etnus TotalviewChannelinlinethreshold size to set threshold for inlining Controlling communication resourcesCommunication resources on DAT-devices Using MPIBsend Using MPIIsend, MPIIrecvGood programming practice with SMC Matching MPIRecv with MPIProbeUser interface errors and warnings Error and warning messagesFatal errors Unsafe MPI programsMpimon options Postfix Giving numeric values to mpimonPrefix Scali MPI Connect Release 4.4 Users Guide Example Profiling with Scali MPI ConnectTracing Using Scali MPI Connect built-in trace+relSecs S eTime where AbsRank MPIcallcommNamerankcall-dependant-parameters whereExample FeaturesTiming Using Scali MPI Connect built-in timingMPIcallDcallsDtimeDfreq TcallsTtimeTfreq Commrank recv from fromworldFromcommonFields Using the scanalyzeCommrank send to toworldTocommonFields where Count!avrLen!zroLen!inline!eager!transporter! whereFor timing This produces the following report Using SMCs built-in CPU-usage functionalityScali MPI Connect Release 4.4 Users Guide Automatic buffer management Tuning communication resourcesBenchmarking How to optimize MPI performanceCaching the application program on the nodes First iteration is very slowMemory consumption increase after warm-up Collective operationsFinding the best algorithm Image contrast enhancement Appendix aPrograms in the ScaMPItst package Scali MPI Connect Release 4.4 Users Guide Original File formatAppendix B When things do not work troubleshootingWhy does not my program start to run? General problems Why can I not start mpid?Why does my program terminate abnormally? Appendix C Per node installation of Scali MPI ConnectInstall Scali MPI Connect for Direct Ethernet Install Scali MPI Connect for TCP/IPInstall Scali MPI Connect for Myrinet ExampleInstall Scali MPI Connect for SCI Install Scali MPI Connect for InfinibandInstall and configure SCI management software License optionsScali kernel drivers Uninstalling SMCTroubleshooting Network providers Troubleshooting the GM provider Troubleshooting 3rdparty DAT providersScali MPI Connect Release 4.4 Users Guide Grouping Appendix D Bracket expansion and groupingBracket expansion Scali MPI Connect Release 4.4 Users Guide Related documentation Appendix EScali MPI Connect Release 4.4 Users Guide List of figures Scali MPI Connect Release 4.4 Users Guide Transporter protocol IndexSSP