HP Serviceguard Extension for RAC (SGeRAC) instruction

Using Serviceguard Extension for RAC Version A.11.20

HP Part Number: 5900-1887

Published: August 2011

Image 1

HP Serviceguard Extension for RAC (SGeRAC) manual Using Serviceguard Extension for RAC Version A.11.20

Contents

Using Serviceguard Extension for RAC Version A.11.20 Legal Notices Contents Contents SGeRAC Toolkit for Oracle RAC 10g or later Support of Oracle RAC ASM with SGeRAC Maintenance 113 Blank Planning Worksheets 151 Troubleshooting 138 Software Upgrades 139Index 153 Advantages of using SGeRAC User Guide Overview Typographical Conventions Problem Reporting Where to find Documentation on the Web What is a Serviceguard Extension for RAC Cluster? Introduction to Serviceguard Extension for RAC Group Membership Using Packages in a Cluster Group Membership Daemon Serviceguard Extension for RAC Architecture Serviceguard Extension for RAC Storage Configuration OptionsPackage Dependencies About Veritas CFS and CVM from Symantec SMS bundle Overview of SGeRAC and Oracle 10g, 11gR1, and 11gR2 RAC How Cluster Interconnect Subnet Works Overview of SGeRAC Cluster Interconnect Subnet Monitoring Configuring Packages for Oracle Listeners Configuring Packages for Oracle RAC Instances Before Node Failure Node Failure Up to Four Nodes with Scsi Storage Larger Clusters Four-Node RAC Cluster Point-to-Point Connections to Storage Devices Eight-Node Cluster with EVA, XP or EMC Disk Array GMS Authorization Starting Serviceguard Manager Overview of Serviceguard ManagerMonitoring Clusters with Serviceguard Manager Administering Clusters with Serviceguard Manager Configuring Clusters with Serviceguard Manager Serviceguard Cluster Timeout Cluster TimeoutsInterface Areas Group Membership API NMAPI2 Shared Storage Oracle Cluster Software Listener Network Monitoring RAC InstancesAutomated Startup and Shutdown Manual Startup and Shutdown Network Planning for Cluster Communication Planning Storage for Oracle 10g/11gR1/11gR2 RAC Planning Storage for Oracle Cluster Software Volume Planning with CVM Volume Planning with SlvmStorage Planning with CFS RAW Logical Volume Name Size MB About Device Special Files Installing Serviceguard Extension for RACAbout Multipathing About Cluster-wide Device Special Files cDSFs Where cDSFs Reside Cluster Communication Network Monitoring Configuration File Parameters Single Network for Cluster Communications Single Network for Cluster Communications SG-HB/RAC-IC Traffic Separation Alternate Configuration-Multiple RAC Databases When Cluster Interconnect Subnet Monitoring is not Used When Cluster Interconnect Subnet Monitoring is usedGuidelines for Changing Cluster Parameters Limitations of Cluster Communication Network Monitor Creating a Storage Infrastructure with LVMCluster Interconnect Monitoring Restrictions Creating Volume Groups and Logical Volumes Building Volume Groups for RAC on Mirrored DisksHP Serviceguard Extension for RAC # mkdir /dev/vgrac Building Mirrored Logical Volumes for RAC with LVM Commands# mknod /dev/vgrac/group c 64 0xhh0000 # ls -l /dev/*/group # lvcreate -m 1 -M y -s g -n redo1.log -L 408 /dev/vgrac Creating Mirrored Logical Volumes for RAC Data Files# lvcreate -m 1 -M y -s g -n system.dbf -L 408 dev/vgrac Creating RAC Volume Groups on Disk Arrays Oracle Demo Database Files Creating Logical Volumes for RAC on Disk Arrays Exporting the Logical Volume Infrastructure Displaying the Logical Volume InfrastructureExporting with LVM Commands Creating a Storage Infrastructure with CFS Installing Oracle Real Application Clusters # cmapplyconf -C clm.asc Initializing the Veritas Volume Manager# cmruncl # cmviewcl # vxdctl -c mode # cfscluster config -s# /etc/vx/bin/vxdisksetup -i c4t4d0 # vxdg -s init cfsdg1 c4t4d0 # newfs -F vxfs /dev/vx/rdsk/cfsdg1/vol3 # newfs -F vxfs /dev/vx/rdsk/cfsdg1/vol2# cfsmntadm add cfsdg1 vol1 /cfs/mnt1 all=rw #cfsmntadm add cfsdg1 vol2 /cfs/mnt2 all=rw # bdf grep cfs Deleting CFS from the Cluster# cfsmntadm delete /cfs/mnt1 # cfsmntadm delete /cfs/mnt2 # vxinstall # cfscluster unconfigCreating a Storage Infrastructure with CVM # cfsdgadm delete cfsdg1 Using CVM 5.x or later # vxdg -s init opsdg c4t4d0 Preparing the Cluster for Use with CVM Using CVM# vxedit set diskdetpolicy=global DiskGroupName # cmapplyconf -P /etc/cmcluster/cvm/VxVM-CVM-pkg.conf # cmruncl # /usr/lib/vxvm/bin/vxdisksetup -i /dev/dsk/c0t3d2# vxdg -s init opsdg c0t3d2 # vxdg list Oracle Demo Database Files Creating VolumesMirror Detachment Policies with CVM # vxassist -g opsdg make logfiles 1024m Adding Disk Groups to the Cluster Configuration # useradd -u 203 -g oinstall -G dba,oper oracle # groupadd oinstall # groupadd dba # groupadd oper# passwd oracle # mkdir -p /mnt/app/crs/oracle/product/10.2.0/crs # chmod 755 /dev/vgrac # usermod -d /cfs/mnt1/oracle oracle Mnt/app/oracle/oradata/ver10/ver10raw.conf Installing Oracle 10g/11gR1/11gR2 RAC Binaries Installing Oracle 10g, 11gR1, or 11gR2 Cluster SoftwareInstalling on Local File System Installing RAC Binaries on a Local File System Creating a RAC Demo Database Installing RAC Binaries on Cluster File SystemCreating a RAC Demo Database on Slvm or CVM $ netca Creating a RAC Demo Database on CFS Verifying Oracle Disk Manager is Configured#swlist VRTSodm Verify that Oracle Disk Manager is Running Configuring Oracle to Use Oracle Disk Manager Library#ll -L /opt/VRTSodm/lib/libodm.sl # cat /dev/odm/stats $ cd $ORACLEHOME/lib # kcmodule -P state odm Configure Serviceguard Packages Preparing Oracle Cluster Software for Serviceguard Packages$ORACRSHOME/bin/crsctl disable crs $ORACLEHOME/bin/srvctl modify database -d dbname -y manual Sbin/init.d/init.crs start Introduction Why ASM over SLVM?SG/SGeRAC Support for ASM on HP-UX 11i Overview Configuring Slvm Volume Groups for ASM Disk Groups Sample Command Sequence for Configuring Slvm Volume Groups Support of Oracle RAC ASM with SGeRAC ASM support with SG/SGeRAC A.11.17.01 or later ASM over Slvm Configuring Slvm Volume Groups for ASM Disk Groups Sample Command Sequence for Configuring Slvm Volume Groups ASM over Raw disk Additional Hints on ASM Integration with SGeRAC Installation, Configuration, Support, and Troubleshooting ASM may require Modified Backup/Restore Procedures Additional Documentation on the Web and Scripts Background SGeRAC Toolkit for Oracle RAC 10g or later Background SGeRAC Toolkit for Oracle RAC 10g or later Page Serviceguard Extension for RAC Toolkit operation Startup and shutdown of the combined Oracle RAC-SGeRAC stack SGeRAC Toolkit for Oracle RAC 10g or later Asmdg MNP Use case 3 Database storage in ASM over Slvm Use Case 2 Setup Use Case 2 performing maintenance with ASM disk groups Use Case 1 Performing maintenance with Oracle Clusterware SGeRAC Toolkit for Oracle RAC 10g or later Internal structure of SGeRAC for Oracle Clusterware Support for the SGeRAC Toolkit Contents Support for the SGeRAC Toolkit CFS-DG1-MNP CFS-DG2-MNP Readme SGeRAC Toolkit for Oracle RAC 10g or later DEPENDENCYCONDITIONDG-MNP-PKG=UP Dependencylocationsamenode ORACLEHOME, CHECKINTERVAL...MAINTENANCEFLAG Dependencyname DEPENDENCYCONDITIONOC-MNP-PKG=UP Dependencylocationsamenode Set to any name desired for the RAC MNP For a package using CFS Oracrshome Asmdisk Group Asmtkitdir SGeRAC Toolkit for Oracle RAC 10g or later Asmdg MNP SGeRAC Toolkit for Oracle RAC 10g or later Support for the SGeRAC Toolkit SGeRAC Toolkit for Oracle RAC 10g or later DEPENDENCYCONDITIONOC-DGMP-PKG=UP Dependencylocationsamenode Additional Documentation on the Web Conclusion Examples of Cluster and Package States Types of Cluster and Package States# cmviewcl -r A.11.16 Name Status State Types of Cluster and Package States Node Status and State Cluster StatusPackage Status and State Service Status Package Switching AttributesStatus of Group Membership Network Status Examples of Cluster and Package StatesFailover and Failback Policies Normal Running Status CVM Package Status Quorum Server Status Status After Moving the Package to Another Node Output of the cmviewcl -vcommand is as follows Status After Halting a Node Status After Package Switching is EnabledViewing Data on Unowned Packages Status Nodename Name Checking the Cluster Configuration and Components Verifying Cluster Components Checking Cluster Components Cmcheckconf -P Example Setting up Periodic Cluster Verification Limitations Online ReconfigurationOnline Node Addition and Deletion Managing the Shared Storage Making LVM Volume Groups Shareable Making Offline Changes to Shared Volume Groups Activating an LVM Volume Group in Shared ModeMaking a Volume Group Unshareable Deactivating a Shared Volume Group On node 2, issue the following command Adding Additional Shared LVM Volume Groups Removing Serviceguard Extension for RAC from a SystemChanging the CVM Storage Configuration Using EMS Hardware Monitors Using Event Monitoring ServiceMonitoring Hardware Adding Disk Hardware # vgcfgrestore /dev/vgsg01 /dev/dsk/c2t3d0 Replacing a Mechanism in a Disk Array Configured with LVMReplacing Disks # lvreduce -m 0 /dev/vgsg01/lvolname /dev/dsk/c2t3d0 # lvextend -m 1 /dev/vgsg01 /dev/dsk/c2t3d0 # vgcfgrestore -n vg name pv raw path# lvsync /dev/vgsg01/lvolname # pvchange -a n pv path Replacing a Lock Disk Online Hardware Maintenance with Inline Scsi Terminator W Scsi Buses with Inline Terminators Replacement of LAN Cards Replacement of I/O CardsOffline Replacement Online Replacement Monitoring RAC Instances Troubleshooting Rolling Software Upgrades Software Upgrades Upgrading Serviceguard to SGeRAC cluster Cmhaltnode nodename Autostartcmcld = Steps for Rolling Upgrades Keeping Kernels Consistent Example of Rolling Upgrade Step Running Cluster Before Rolling Upgrade Node 1 Upgraded to SG/SGeRAC # cmhaltnode -f node2 Running Cluster After Upgrades Limitations of Rolling Upgrades Limitations of Non-Rolling Upgrades Non-Rolling Software UpgradesMigrating an SGeRAC Cluster with Cold Install Rolling Upgrade Using DRD Upgrade Using DRDNon-Rolling Upgrade Using DRD Restrictions for DRD Upgrades Software Upgrades Oracle Logical Volume Worksheet LVM Volume Group and Physical Volume Worksheet Blank Planning Worksheets Index RAC 155