HP Insight Cluster Management Utility instruction

HP Insight Cluster Management Utility v7.1

User Guide

Abstract

This guide describes how to install, configure, and use HP Insight Cluster Management Utility (CMU) v7.1 on HP systems. HP Insight CMU is software dedicated to the administration of HPC and large Linux clusters. This guide is intended primarily for administrators who install and manage a large collection of systems. This document assumes you have access to the documentation that comes with the hardware platform where the HP Insight CMU cluster will be installed, and you are familiar with installing and administering Linux operating systems.

HP Part Number: 5900-2346

Published: April 2013

Edition: 1

Page 1

Image 1

HP manual HP Insight Cluster Management Utility

Contents

HP Insight Cluster Management Utility Copyright 2013 Hewlett-Packard Development Company, L.P Contents Provisioning a cluster with HP Insight CMU Defining a cluster with HP Insight CMU Monitoring a cluster with HP Insight CMU Advanced topics 112 Managing a cluster with HP Insight CMUActions Alerts Alert reactions Detailed installation instructions 131 Troubleshooting 126Support and other resources 123 HP Insight CMU manpages 139 Glossary 187 Index 189 Figures Tables Date command Dmidecode commandExamples HP Insight CMU configuration FeaturesOverview Compute node monitoring System disk replication Compute node administration Installing HP Insight CMU Installing and upgrading HP Insight CMUManagement node hardware requirements Firmware upgrade requirements Planning for compute node installationConfiguring the local smart array card Configuring the management cards NIC 1 PXE boot or PXE enabled NIC 2 Disabled Configuring the Bios7.1 DL3xx, DL5xx, DL7xx, Blades Speed 9600 Bd IDE 7.2 DL160 G5, DL165c G5, DL165c G6, and DL180 G5 Servers Dhcp Disabled Share NIC mode DisabledNIC1 control Enabled NIC1 PXE Enabled NIC2 on the SL2x170z G6 Server 7.4 SL2x170z G6 and DL170h G6 Servers Bios setting Preinstallation limitations Preparing for installationHP Insight CMU kit delivery HP Insight CMU CD-ROM directory structure Operating system supportRhel 6 support Login privileges HP Insight CMU installation checklistSELinux and HP Insight CMU Run /opt/cmu/bin/cmumgtconfig -c Installation procedures# chkconfig --add cmu Installing HP Insight CMU with high availability Installing and upgrading HP Insight CMU HA hardware requirements Installing HP Insight CMU under HASoftware prerequisites Overview HP Insight CMU HA service requirements Configuring HA control of HP Insight CMUInstalling and testing Start cmuserver Cmuadmin1# /etc/init.d/cmu start# /etc/init.d/cmuserver start Var/log/cmuservicehostname.log file for errors Cmu hacmu service needs restartCmuadmin2# /opt/cmu/tools/cmuhapostinstall # /etc/init.d/cmu setaudit # /etc/init.d/cmu stop Upgrading HP Insight CMU HA service HP Insight CMU configuration considerationsCmuha nothing to backup from the cmu HA share Run cmuhapostinstall on server Upgrading HP Insight CMU Stopping the HP Insight CMU serviceDependencies Upgrading Java Runtime Environment Restoring the HP Insight CMU database Saving the HP Insight CMU database Defining a cluster with HP Insight CMU HP Insight CMU service statusLaunching the HP Insight CMU GUI HP Insight CMU main window Quitting administrator mode Administrator modeClick Options→Unprivileged Mode Server requirements Cluster administration High-level checklist for building an HP Insight CMU cluster Node management window Node management Scan node dialog Scanning nodes Adding nodes Management card password window Modifying nodes Cluster Administration→Node Management Importing nodes Network entity managementDeleting nodes Exporting nodes Deleting network entities Adding network entities Logical group management Provisioning a cluster with HP Insight CMU Deleting logical groups Modifying logical groupsRenaming logical groups Autoinstall requirements AutoinstallAutoinstall templates Autoinstall calling methods Enabling autoinstall Using autoinstall from GUICreating an autoinstall logical group Restart cmuserver Registering compute nodes Autoinstall compute nodes Registering an autoinstall logical group Using autoinstall from CLIAdding nodes to autoinstall logical group Cmu addtologicalgroup node to logicalgroupname Customization Where nodes.txt is the list of nodes to autoinstall Restrictions Backing upBacking up a disk from a compute node in a logical group # /opt/cmu/image/logicalgroupname Cloning Cloning procedure Cloning status Preconfiguration Default content of prereconf.sh is Reconfiguration Rescan MAC Node static info Expanding an image HP Insight CMU image editorAfter editing the image, commit changes # cmuimagecommit -i rh5u4x8664 Modifying an image HP Insight CMU diskless environmentsSaving a modified cloning image # /opt/cmu/bin/cmuimagecommit -i rh5u4x8664 Operating systems supported Modifying the Tftp server configurationOn the management node On the golden node Populating the HP Insight CMU database Activating the diskless featureCreating a diskless image Creating a diskless logical group Adding a new logical group From the CLI Adding nodes into the logical group# /opt/cmu/cmucli Cmu probekernel Cmu boot net myTestImage node1 noden Booting the compute nodesUnderstanding the structure of a diskless image Customizing your diskless image Using reconf-diskless-image.sh Best practices for diskless clusters Do not update the /opt/cmu/image/imageName/root directoryTemplates and image file Node. /opt/cmu/image/imageName/snapshot/nodeName # chkconfig nfs on Configuring additional NFS serversOn Red Hat # chkconfig nfsserver on When a diskless logical group is createdOn Sles Sample file Comments on High Availability HA When a node is added to the diskless logical group Monitoring a cluster with HP Insight CMU Installing the HP Insight CMU monitoring clientDeploying the monitoring client Launch the HP Insight CMU GUI Monitoring the cluster Selecting the central frame display Node and group status Monitoring window Global cluster view in the central frame Resource view overview Resource view in the central frame Gauge widget Detail mode in resource viewNode view in the central frame Node details Using time view Tagging nodes Getting startedAdaptive stacking Mouse control Bindings and options Technical dependencies Troubleshooting Visualizing history data Archiving user groups Stopping HP Insight CMU monitoring Tuning HP Insight CMU monitoringAction and alert files Limitations Alerts Actions Alert reactions Alerts Level of the alert Name of the alert that caused the reactionText of the Description for this reaction Installing and starting collectl on compute nodes Using collectl for gathering monitoring data# chkconfig --add collectl # /etc/init.d/collectl start ActionAndAlertFile.txt file # collectl -c 1 -s+C --export lexpr Var/log/collectl *rw,sync,noallsquash,norootsquash # mkdir /var/log/collectl # vi /etc/exports# exportfs -r # cp -a /var/www/html/colplot /opt/cmu/www/colplot # vi /etc/collectl.conf # mkdir /var/log/collectl # vi /etc/fstabRestart collectl # /etc/init.d/collectl restart Monitoring Nvidia GPUs Monitoring GPUs and coprocessorsSelect plotting options, then click Generate Plot Monitoring AMD GPUs # cmuconfigamd Monitoring Intel coprocessors # /opt/cmu/bin/cmuconfigintel Review the results and verify no errors are reported HP Insight CMU alert converted to SIM event Extended metric support Opt/cmu/bin/cmusubmitextendedmetrics -f datafile Managing a cluster with HP Insight CMU Administrator menuUnprivileged user menu SSH connection Management card connection Power offVirtual serial port connection Shutdown Reboot BootChange UID LED status Single window pdsh Multiple windows broadcastDshbak Cmudiff examples Cmupdsh cmudiff -h Cmupdsh dmidecodeCmupdsh cmudiff -d Parallel distributed copy pdcp # vi /root/.ssh/config Adding user groups User group management HP Insight firmware management Viewing and analyzing Bios settingsDeleting user groups Renaming user groups Installing and upgrading firmware Checking Bios versionsCustomizing the GUI menu Basic commands Saving user settingsHP Insight CMU CLI Starting a CLI interactive session Getting help for a command Help commandsFor example, to get more information about the halt command Cmu help Specifying nodes Executing a command on one nodeDisplaying logical groups of a cluster Displaying nodes of a logical group Executing a command on a range of nodes Executing a command on a list of nodesUsing wildcards Executing a command on all nodes Executing a command on specific nodes of a logical group Administration and cloning commandsBooting a set of nodes Broadcasting commands to a set of nodes Powering off a set of nodes Rebooting a set of nodesHalting a set of nodes Cloning a set of nodes Setting the locator LED on or offCmu locate on o185i192 Cmu locate off o185i192 Adding nodes to a logical group Adding a new logical groupBacking up a node Cmu backup Cmu modifypassword ILOILOCMlo100i Modifying a management card passwordCmu modifypassword lo100i Discovering MAC address for new nodes HP Insight CMU Linux shell commands Administration utilities pdcp and pdsh# /opt/cmu/bin/pdcp -w cn0001,cn0002 source /tmp/dest # /opt/cmu/bin/pdsh -w cn0001,cn0002 ls Advanced topics Accessing the GUI for non-root users Custom menu options for non-root users Configuring sudo support Bstevens ALL = Nopasswd Cmupower Cmuimage Cjones ALL = CmupowerSbarney ALL = Nopasswd Cmupower Cmuimage Cmuetc HP Insight CMU diskless API Delete diskless image Build diskless imageName of the new logical group Unconfigure diskless node Configure diskless nodeBoot diskless node Diskless check HP Insight CMU remote hardware control APILo100i None Osoff OffUidoff Uidon CMUVALIDHARDWARETYPES=ILOlo100iILOCMIPMI CMUVALIDHARDWARETYPES=ILOlo100iILOCM Cloning mechanisms Support for ScaleMPCMUvSMPPREFIX=vSMP Cloning mechanisms Advanced topics Contacting HP Support and other resourcesRelated information Typographic conventions CommandComputer output User input Typographic conventions Network boot issues TroubleshootingHP Insight CMU logs Troubleshooting switch issues Backup issuesTroubleshooting network boot Administration command problems Cloning issuesGUI problems If not, restart the HP Insight CMU service Detailed Java exception is Certificate error Troubleshooting Install required RPMs Detailed installation instructionsActivating xinetd services # chkconfig nfs on # /etc/init.d/nfs start Java installation Firewall configurationVerifying the Dhcpd listen interface On the HP Insight CMU management node Setting the Java Path Installing HP Insight CMU licensingConfiguring the HP Insight CMU management server hostname Method Starting HP Insight CMU Cmu service needs restartEdit the /opt/cmu/etc/cmuserver.conf file # vi /opt/cmu/etc/cmuserver.conf Installing HP Insight CMU on the GUI client workstation Configuring HP Insight CMU to start automaticallyVerifying the HP Insight CMU state Cmuserver utility reports the state of the daemons Using an ssh tunnel Configuring the GUI client on Linux workstationsUsing an X Window server Activating the HP Insight CMU GUI Launching the HP Insight CMU GUI using a web browserLaunching the HP Insight CMU directly from the Java file HP Insight CMU GUI HP Insight CMU manpages Cmushownodes8 # /opt/cmu/bin/cmushownodes # /opt/cmu/bin/cmushownodes -a -o %n %i %k %m default %b %t # /opt/cmu/bin/cmushowlogicalgroups -h logicalgroupname Cmushowlogicalgroups8Help # /opt/cmu/bin/cmushowlogicalgroups # /opt/cmu/bin/cmushownetworkentities -h networkentity Cmushownetworkentities8# /opt/cmu/bin/cmushownetworkentities # /opt/cmu/bin/cmushownetworkentities rack1 # /opt/cmu/bin/cmushowusergroups -h usergroup Cmushowusergroups8# /opt/cmu/bin/cmushowusergroups # /opt/cmu/bin/cmushowusergroups user1 # /opt/cmu/bin/cmushowarchivedusergroups Cmushowarchivedusergroups8# /opt/cmu/bin/cmushowarchivedusergroups -p -f # /opt/cmu/bin/cmushowarchivedusergroups -f -s Mgtcardip -T--mgt-card ILOlo100iILOCM -R--arch architecture Cmuaddnode8Node-number num Processing 1 node Command-line mode# cat nodes.txt # /opt/cmu/bin/cmuaddnode -f nodes.txt Filename inputfile Cmuaddnetworkentity8# /opt/cmu/bin/cmuaddnetworkentity rack1 rack2 # /opt/cmu/bin/cmuaddnetworkentity -f networkentitylist Filename inputfile input logical groups from inputfile Name Cmuaddlogicalgroup8# /opt/cmu/bin/cmuaddlogicalgroup -n test -d cciss/c0d0 # /opt/cmu/bin/cmuaddlogicalgroup -f logicalgroupfile Logicalgroup Cmuaddtologicalgroupcandidates8Nodenamefile # /opt/cmu/bin/cmuaddusergroup user1 user2 Cmuaddusergroup8# /opt/cmu/bin/cmuaddusergroup -f usergrouplist Usergroup Cmuaddtousergroup8# /opt/cmu/bin/cmuaddtousergroup -t group1 cn0003 # /opt/cmu/bin/cmuaddtousergroup -t group1 -f nodenamefile To flag cn0001 active in logical group rh6u0x8664 Cmuchangeactivelogicalgroup8 Networkentity Cmuchangenetworkentity8# /opt/cmu/bin/cmuchangenetworkentity -t rack1 cn0001 Delete one or more nodes from a logical group Cmudelfromlogicalgroupcandidates8Delete nodes from this logical group To delete one node from a logical group rh6u0x8664 # /opt/cmu/bin/cmudelfromnetworkentity -t rack1 node1 Cmudelfromnetworkentity8 Cmudelarchivedusergroup8 # /opt/cmu/bin/cmudelarchivedusergroup -h -v-t timeout -dCmudelarchivedusergroup -- Delete an archived user group Delete an archived user group # /opt/cmu/bin/cmudelfromusergroup -t user1 node1 Cmudelfromusergroup8 # /opt/cmu/bin/cmudellogicalgroup rh5u5x8664 sles11sp1x8664 Cmudellogicalgroup8# /opt/cmu/bin/cmudellogicalgroup -f logicalgrouplist # /opt/cmu/bin/cmudelnetworkentity rack1 rack2 Cmudelnetworkentity8# /opt/cmu/bin/cmudelnetworkentity -f networkentitylist # /opt/cmu/bin/cmudelnode node1 node2 Cmudelnode8# /opt/cmu/bin/cmudelnode -f nodelist Delete monitoring snapshots from the history database Cmudelsnapshots8 # /opt/cmu/bin/cmudelusergroup user1 user2 Cmudelusergroup8# /opt/cmu/bin/cmudelusergroup -f usergrouplist # /opt/cmu/bin/cmudelusergroup -a -m 60 -f usergrouplist Cmuconsole -- Connect to compute node management ports Cmuconsole8# /opt/cmu/bin/cmuconsole computenodehostname Cmupower8 Cmupower -p OFF -u user1 Cmupower -p OFF -n cn0001Cmupower -p Boot -l rh6u0x8664 Cmupower -p Uidon -u user2 Title Cmucustomrun8# /opt/cmu/bin/cmucustomrun -l # /opt/cmu/bin/cmucustomrun -t auditlspci =node Cmuclone8# /opt/cmu/bin/cmuclone -f /tmp/nodelist -i sles11sp1x8664 # /opt/cmu/bin/cmuclone -n node1 -i rh6u2x8664 Cmubackup -l myimage -n node20 -p cciss/c0d0p3,cciss/c0d0p1 Cmubackup8Cmubackup -l myimage -n node20 -r 3 -e /tmp/err.log Options naming Cmuscanmacs8# /opt/cmu/bin/cmuscanmacs -h hostname -p hostnameprefix Options general Example node definitions Example 173 Cmurescanmac -- Rescan the MAC address of a node Cmurescanmac8# /opt/cmu/tools/cmurescanmac -n nodename N NICnum -h Node name in the HP Insight CMU database Filename Cmumodnode8Lg logicalgroup # /opt/cmu/bin/cmumodnode -f nodes.txt Cmumonstat8 # /opt/cmu/bin/cmumonstat --all-sensors --all-nodes # /opt/cmu/bin/cmumonstat --nodes=n0001,n0002,n0001-11# /opt/cmu/bin/cmumonstat --all-nodes --all-sensors --stats # /opt/cmu/bin/cmumonstat --all-nodes --color # /opt/cmu/bin/cmuimageopen -h -i imagename Cmuimageopen8To open the HP Insight CMU image rh5u5x8664 # /opt/cmu/bin/cmuimageopen -i rh5u5x8664 # /opt/cmu/bin/cmuimagecommit -i rh5u5x8664 Cmuimagecommit8 Cmuconfignvidia -- Configure Nvidia GPU monitoring Cmuconfignvidia8 Cmuconfigamd -- Configure AMD GPU monitoring Cmuconfigamd8 Cmuconfigintel -- Configure Intel coprocessor monitoring Cmuconfigintel8# /opt/cmu/bin/cmuconfigintel -h -r -n Automatically defaults when reasonable Cmumgtconfig8Eth num1num2bond num1 Num # /opt/cmu/bin/cmumgtconfig -c -s dhcp # /opt/cmu/bin/cmumgtconfig -c# /opt/cmu/bin/cmumgtconfig -t Cmufirmwaremgmt -- Verify and execute firmware Cmufirmwaremgmt8Ocmudiffparameters Specifies a text file listing compute nodes Ebipa Glossary Package management Index Index 191