Troubleshooting | HP Serviceguard Extension for RAC (SGeRAC)

6 Troubleshooting

Go to www.hp.com/go/hpux-serviceguard-docs, and then click HP Serviceguard . In the User Guide section, click on the latest Managing Serviceguard manual and see the “Troubleshooting your Cluster” chapter.

NOTE: All messages from cmgmsd log to /var/adm/syslog/syslog.log by default.

138 Troubleshooting

Image 138

HP Serviceguard Extension for RAC (SGeRAC) manual Troubleshooting

Contents

Using Serviceguard Extension for RAC Version A.11.20 Legal Notices Contents Contents Support of Oracle RAC ASM with SGeRAC SGeRAC Toolkit for Oracle RAC 10g or later Maintenance 113 Troubleshooting 138 Software Upgrades 139 Blank Planning Worksheets 151Index 153 Advantages of using SGeRAC User Guide Overview Problem Reporting Typographical Conventions Where to find Documentation on the Web Introduction to Serviceguard Extension for RAC What is a Serviceguard Extension for RAC Cluster? Using Packages in a Cluster Group Membership Serviceguard Extension for RAC Architecture Group Membership Daemon Package Dependencies Storage Configuration OptionsServiceguard Extension for RAC About Veritas CFS and CVM from Symantec Overview of SGeRAC and Oracle 10g, 11gR1, and 11gR2 RAC SMS bundle Overview of SGeRAC Cluster Interconnect Subnet Monitoring How Cluster Interconnect Subnet Works Configuring Packages for Oracle RAC Instances Configuring Packages for Oracle Listeners Node Failure Before Node Failure Larger Clusters Up to Four Nodes with Scsi Storage Point-to-Point Connections to Storage Devices Four-Node RAC Cluster GMS Authorization Eight-Node Cluster with EVA, XP or EMC Disk Array Monitoring Clusters with Serviceguard Manager Overview of Serviceguard ManagerStarting Serviceguard Manager Administering Clusters with Serviceguard Manager Configuring Clusters with Serviceguard Manager Interface Areas Cluster TimeoutsServiceguard Cluster Timeout Group Membership API NMAPI2 Oracle Cluster Software Shared Storage Listener RAC Instances Network MonitoringAutomated Startup and Shutdown Network Planning for Cluster Communication Manual Startup and Shutdown Planning Storage for Oracle Cluster Software Planning Storage for Oracle 10g/11gR1/11gR2 RAC Volume Planning with Slvm Volume Planning with CVMStorage Planning with CFS RAW Logical Volume Name Size MB Installing Serviceguard Extension for RAC About Device Special FilesAbout Multipathing Where cDSFs Reside About Cluster-wide Device Special Files cDSFs Configuration File Parameters Cluster Communication Network Monitoring Single Network for Cluster Communications Single Network for Cluster Communications SG-HB/RAC-IC Traffic Separation Alternate Configuration-Multiple RAC Databases When Cluster Interconnect Subnet Monitoring is used When Cluster Interconnect Subnet Monitoring is not UsedGuidelines for Changing Cluster Parameters Creating a Storage Infrastructure with LVM Limitations of Cluster Communication Network MonitorCluster Interconnect Monitoring Restrictions Building Volume Groups for RAC on Mirrored Disks Creating Volume Groups and Logical VolumesHP Serviceguard Extension for RAC # mknod /dev/vgrac/group c 64 0xhh0000 Building Mirrored Logical Volumes for RAC with LVM Commands# mkdir /dev/vgrac # ls -l /dev/*/group Creating Mirrored Logical Volumes for RAC Data Files # lvcreate -m 1 -M y -s g -n redo1.log -L 408 /dev/vgrac# lvcreate -m 1 -M y -s g -n system.dbf -L 408 dev/vgrac Creating RAC Volume Groups on Disk Arrays Creating Logical Volumes for RAC on Disk Arrays Oracle Demo Database Files Displaying the Logical Volume Infrastructure Exporting the Logical Volume InfrastructureExporting with LVM Commands Installing Oracle Real Application Clusters Creating a Storage Infrastructure with CFS Initializing the Veritas Volume Manager # cmapplyconf -C clm.asc# cmruncl # cmviewcl # /etc/vx/bin/vxdisksetup -i c4t4d0 # cfscluster config -s# vxdctl -c mode # vxdg -s init cfsdg1 c4t4d0 # cfsmntadm add cfsdg1 vol1 /cfs/mnt1 all=rw # newfs -F vxfs /dev/vx/rdsk/cfsdg1/vol2# newfs -F vxfs /dev/vx/rdsk/cfsdg1/vol3 #cfsmntadm add cfsdg1 vol2 /cfs/mnt2 all=rw # cfsmntadm delete /cfs/mnt1 Deleting CFS from the Cluster# bdf grep cfs # cfsmntadm delete /cfs/mnt2 Creating a Storage Infrastructure with CVM # cfscluster unconfig# vxinstall # cfsdgadm delete cfsdg1 Using CVM 5.x or later # vxdg -s init opsdg c4t4d0 # vxedit set diskdetpolicy=global DiskGroupName Using CVMPreparing the Cluster for Use with CVM # cmapplyconf -P /etc/cmcluster/cvm/VxVM-CVM-pkg.conf # vxdg -s init opsdg c0t3d2 # /usr/lib/vxvm/bin/vxdisksetup -i /dev/dsk/c0t3d2# cmruncl # vxdg list Mirror Detachment Policies with CVM Creating VolumesOracle Demo Database Files # vxassist -g opsdg make logfiles 1024m Adding Disk Groups to the Cluster Configuration # passwd oracle # groupadd oinstall # groupadd dba # groupadd oper# useradd -u 203 -g oinstall -G dba,oper oracle # mkdir -p /mnt/app/crs/oracle/product/10.2.0/crs # usermod -d /cfs/mnt1/oracle oracle # chmod 755 /dev/vgrac Mnt/app/oracle/oradata/ver10/ver10raw.conf Installing on Local File System Installing Oracle 10g, 11gR1, or 11gR2 Cluster SoftwareInstalling Oracle 10g/11gR1/11gR2 RAC Binaries Installing RAC Binaries on a Local File System Creating a RAC Demo Database on Slvm or CVM Installing RAC Binaries on Cluster File SystemCreating a RAC Demo Database $ netca Verifying Oracle Disk Manager is Configured Creating a RAC Demo Database on CFS#swlist VRTSodm #ll -L /opt/VRTSodm/lib/libodm.sl Configuring Oracle to Use Oracle Disk Manager LibraryVerify that Oracle Disk Manager is Running # cat /dev/odm/stats # kcmodule -P state odm $ cd $ORACLEHOME/lib $ORACRSHOME/bin/crsctl disable crs Preparing Oracle Cluster Software for Serviceguard PackagesConfigure Serviceguard Packages $ORACLEHOME/bin/srvctl modify database -d dbname -y manual Sbin/init.d/init.crs start SG/SGeRAC Support for ASM on HP-UX 11i Why ASM over SLVM?Introduction Overview Configuring Slvm Volume Groups for ASM Disk Groups Sample Command Sequence for Configuring Slvm Volume Groups Support of Oracle RAC ASM with SGeRAC ASM over Slvm ASM support with SG/SGeRAC A.11.17.01 or later Configuring Slvm Volume Groups for ASM Disk Groups Sample Command Sequence for Configuring Slvm Volume Groups ASM over Raw disk Additional Hints on ASM Integration with SGeRAC ASM may require Modified Backup/Restore Procedures Installation, Configuration, Support, and Troubleshooting Additional Documentation on the Web and Scripts SGeRAC Toolkit for Oracle RAC 10g or later Background Background SGeRAC Toolkit for Oracle RAC 10g or later Page Serviceguard Extension for RAC Toolkit operation Startup and shutdown of the combined Oracle RAC-SGeRAC stack SGeRAC Toolkit for Oracle RAC 10g or later Asmdg MNP Use Case 2 Setup Use case 3 Database storage in ASM over Slvm Use Case 1 Performing maintenance with Oracle Clusterware Use Case 2 performing maintenance with ASM disk groups SGeRAC Toolkit for Oracle RAC 10g or later Internal structure of SGeRAC for Oracle Clusterware Contents Support for the SGeRAC Toolkit Support for the SGeRAC Toolkit CFS-DG1-MNP CFS-DG2-MNP Readme SGeRAC Toolkit for Oracle RAC 10g or later DEPENDENCYCONDITIONDG-MNP-PKG=UP Dependencylocationsamenode ORACLEHOME, CHECKINTERVAL...MAINTENANCEFLAG Dependencyname DEPENDENCYCONDITIONOC-MNP-PKG=UP Dependencylocationsamenode Set to any name desired for the RAC MNP For a package using CFS Oracrshome Asmdisk Group Asmtkitdir SGeRAC Toolkit for Oracle RAC 10g or later Asmdg MNP SGeRAC Toolkit for Oracle RAC 10g or later Support for the SGeRAC Toolkit SGeRAC Toolkit for Oracle RAC 10g or later DEPENDENCYCONDITIONOC-DGMP-PKG=UP Dependencylocationsamenode Conclusion Additional Documentation on the Web Types of Cluster and Package States Examples of Cluster and Package States# cmviewcl -r A.11.16 Name Status State Types of Cluster and Package States Cluster Status Node Status and StatePackage Status and State Package Switching Attributes Service StatusStatus of Group Membership Failover and Failback Policies Examples of Cluster and Package StatesNetwork Status Normal Running Status Quorum Server Status CVM Package Status Output of the cmviewcl -vcommand is as follows Status After Moving the Package to Another Node Status After Package Switching is Enabled Status After Halting a NodeViewing Data on Unowned Packages Checking the Cluster Configuration and Components Status Nodename Name Checking Cluster Components Verifying Cluster Components Cmcheckconf -P Setting up Periodic Cluster Verification Example Online Reconfiguration LimitationsOnline Node Addition and Deletion Making LVM Volume Groups Shareable Managing the Shared Storage Making a Volume Group Unshareable Activating an LVM Volume Group in Shared ModeMaking Offline Changes to Shared Volume Groups Deactivating a Shared Volume Group On node 2, issue the following command Removing Serviceguard Extension for RAC from a System Adding Additional Shared LVM Volume GroupsChanging the CVM Storage Configuration Monitoring Hardware Using Event Monitoring ServiceUsing EMS Hardware Monitors Adding Disk Hardware Replacing Disks Replacing a Mechanism in a Disk Array Configured with LVM# vgcfgrestore /dev/vgsg01 /dev/dsk/c2t3d0 # lvreduce -m 0 /dev/vgsg01/lvolname /dev/dsk/c2t3d0 # lvsync /dev/vgsg01/lvolname # vgcfgrestore -n vg name pv raw path# lvextend -m 1 /dev/vgsg01 /dev/dsk/c2t3d0 # pvchange -a n pv path Online Hardware Maintenance with Inline Scsi Terminator Replacing a Lock Disk W Scsi Buses with Inline Terminators Offline Replacement Replacement of I/O CardsReplacement of LAN Cards Online Replacement Monitoring RAC Instances Troubleshooting Software Upgrades Rolling Software Upgrades Upgrading Serviceguard to SGeRAC cluster Cmhaltnode nodename Steps for Rolling Upgrades Autostartcmcld = Example of Rolling Upgrade Keeping Kernels Consistent Running Cluster Before Rolling Upgrade Step Node 1 Upgraded to SG/SGeRAC # cmhaltnode -f node2 Limitations of Rolling Upgrades Running Cluster After Upgrades Non-Rolling Software Upgrades Limitations of Non-Rolling UpgradesMigrating an SGeRAC Cluster with Cold Install Non-Rolling Upgrade Using DRD Upgrade Using DRDRolling Upgrade Using DRD Restrictions for DRD Upgrades Software Upgrades LVM Volume Group and Physical Volume Worksheet Oracle Logical Volume Worksheet Blank Planning Worksheets Index RAC 155