PAR Technologies V5 manual Problem cannot start parallel task, Problem bad performance

Page 34

Problem: cannot start parallel task

Or logged on to this node, run psiadmin which also starts up the ParaStation daemon psid. See Section 6.1, “ Problem:psiadmin returns error ” for more details.

Check the logfile /var/log/messages on this node for error messages. Verify that all nodes have an identical configuration (/etc/parastation.conf).

6.3. Problem: cannot start parallel task

Problem: a parallel task cannot be launched, an error is reported:

PSI: PSI_createPartition: Resource temporarily unavailable

Check for available nodes and active parallel tasks. Check for user or group restrictions.

If the error

PSI: dospawn: spawn to node 1 failed.

PSE: Could not spawn './mpi_latency' process 1, error = Bad \ file descriptor.

is reported, check if the current directory holding the program mpi_latency is accessible on all nodes. Verify that the program is executable on all nodes.

6.4. Problem: bad performance

Verify that the proper interconnect and/or transport is used: check for environment variables controlling transport (see Section 5.8, “ControllingParaStation5 communication paths” and ps_environment(5)).

Watch protocol counters, e.g. counters indicating timeouts, retries, errors or other bad conditions. For p4sock, check recv_net_data and recv_user. See Section 5.2, “ParaStation5 protocol p4sock”.

Look for a crystal bowl!

Or contact <support@par-tec.com>.

6.5.Problem: different groups of nodes are seen as up or down

Problem: depending on which node the psiadmin is run, different groups of nodes are seen as "up" or "down".

Check for identical configuration on each node, e.g. compare the configuration file /etc/ parastation.conf on each node.

6.6. Problem: cannot start process on frontend

Problem: Starting a job is canceled giving the error message

Connecting client 139.27.166.22:44784 (rank 6) failed : Network is unreachable

PSIlogger: Child with rank 12 exited with status 1.

30

ParaStation5 Administrator's Guide

Image 34
Contents Administrators Guide ParaStation5 Administrators Guide Info@par-tec.comTable of Contents Problem different groups of nodes are seen as up or down Introduction History of ParaStationAbout this document Libraries Technical overviewRuntime daemon Kernel modulesLicense Prerequisites InstallationHardware Software Directory structureKernel version Man Installation via RPM packagesGetting the ParaStation5 RPM packages Mpi2, mpi2-intel, mpi2-pgi, mpi2-pscCompiling the ParaStation5 packages from source Installing the RPMsFile Version Etc/init.d/xinetd reload Installing the documentationParaStation entries # rpm -Uv psdoc-5.0.0-1.noarch.rpm Installing MPIFurther steps # rpm -Uv psmpi2.5.0.0-1.i586.rpmUninstalling ParaStation5 ParaStation5 Administrators Guide Copy template ConfigurationConfiguration of the ParaStation system Define Number of nodes# /opt/parastation/bin/testconfig Enable optimized network driversHostname id HWType runJob starter accounter Testing the installation # /opt/parastation/bin/psiadmin -s -c list # /opt/parastation/bin/testnodes -np nodesInsight ParaStation5 ParaStation5 pscom communication libraryDirectory /proc/sys/ps4/state # echo 10 /proc/sys/ps4/state/ResendTimeout# cat /proc/sys/ps4/state/connections Controlling process placement Directory /proc/sys/ps4/localUsing non-ParaStationapplications Using the ParaStation5 queuing facilityExporting environment variables for a task Controlling ParaStation5 communication paths Export LDPRELOAD=/opt/parastation/lib64/libp4tcp.soPSPP4S or PSPP4SOCK Authentication within ParaStation5Pspshm or Pspsharedmem Export PSPLIB=/opt/parastation/lib64/libpscomopenib.soParallel shell tool Homogeneous user ID spaceSingle system view Nodes and CPUsTok2env Integration with AFSIntegrating external queuing systems PSIRARGPRE0=/some/path/env2tokMulticasts # UseMCast Using ParaStation accountingCopying files in parallel Route add -net 224.0.0.0 netmask 240.0.0.0 dev ethXChanging the default ports for psid8 Using ParaStation process pinningUsing memory binding Spawning processes belonging to all groupsPort Problem psiadmin returns error TroubleshootingProblem node shown as down Problem different groups of nodes are seen as up or down Problem cannot start parallel taskProblem bad performance Problem cannot start process on frontendProblem pssh fails Problem psid does not startup, reports port in useProblem processes cannot access files on remote nodes Reference Pages ParaStation5 Administrators Guide Description InstallDir inst-dir , InstallationDir inst-dirParastation.conf ParametersStopscript SetupscriptStartscript StatusscriptMvapi P4sockOpenib ElanAccounter NrOfNodes numNode node17 16 HWType ethernet p4sock starter yes runJobs no $GENERATE 1-96 node$0,2 $0LogLevel num SelectTime timeDeadInterval num MCastGroup group-numDataSize size Core sizeCPUTime time MemLock sizeProc Processes maxprocs CPUmap mapRdpClosedTimeout ms RdpTimeout msStatusTimeout ms RdpResendTimeout msErrors See alsoParaStation5 Administrators Guide Synopsis PsiadminOptions Standard Output Standard ErrorStandard Input Extended descriptionAll ExitAllproc cnt count Hardware Count hw hwDown LoadSummary max max RdpQuit Group nodes Accounters nodesUser nodes Maxproc nodesHandleOldBins nodes Master nodesFreeOnSuspend nodes NodesSort nodesCpumap nodes Rlrss nodesRdpClosedTimeout nodes RdpTimeout nodesStatusTimeout nodes RdpResendTimeout nodesResolve nodes Restart nodesSelecttime time nodes Psiddebug mask nodesPattern Name Description HandleOldBins 0 1 nodes Rdpmaxretrans val nodes RdpClosedTimeout ms nodes RdpTimeout ms nodesStatusTimeout ms nodes RdpResendTimeout ms nodesNormal FilesQuiet VerbosePsid Debug=level Configfile=fileLogfile=file Num TestconfigFilename ? , --usage Show a help messageParaStation5 Administrators Guide Cnt num TestnodesNp num MapParaStation5 Administrators Guide Testpse Testpse -npnumParaStation5 Administrators Guide Net P4statSock ?,--helpParaStation5 Administrators Guide Add P4tcpDelete ParaStation5 Administrators Guide Psaccounter Pattern Description?, --help DumpcoreCoredir=dir Var/account/yyyymmdd Accounting files, one per dayPsaccview Lg,--ltotgroup Lj,--ljobsLu,--ltotuser Ls,--ltotsumCputime CpuweightAqtime EndInitialization file Mlisten ParaStation5 Administrators Guide Appendix A. Quick Installation Guide # chkconfig -a /etc/init.d/parastation # /opt/parastation/bin/psiadmin psiadmin addTesting Appendix B. ParaStation license Page Page Page Appendix C. Upgrading ParaStation4 to ParaStation5 Building and installing ParaStation5 packages# psiadmin -s Changes to the runtime environmentPage Glossary ARPSee ParaStation Logger To share a common address space within a node ParaStation5 Administrators Guide