HP Serviceguard Extension for RAC (SGeRAC) guide

Additional Documentation on the Web and Scripts

•Oracle Clusterware Installation Guide 11g Release 1 (11.1) for HP-UX at www.oracle.com/ pls/db111/portal.portal_db?selected=11&frame= → HP-UX Installation Guides → Clusterware Installation Guide for HP-UX

•ASM related sections in Oracle Manuals

◦Oracle® Database Administrator's Guide 10g R2 (10.2) at www.oracle.com/pls/db102/ portal.portal_db?selected=4 → Guides → Administrator's Guide

◦Oracle® Database Oracle Clusterware and Oracle Real Application Clusters

Administration and Deployment Guide10g R2 (10.2) at www.oracle.com/pls/db102/ portal.portal_db?selected=4→ High Availability → Oracle Clusterware and Oracle Real Application Clusters Administration and Deployment Guide

◦Oracle® Database Administrator's Guide 11g R1 at www.oracle.com/pls/db111/ portal.portal_db?selected=4&frame=→ Supporting Documentation → Administrator's Guide

◦www.oracle.com/technology/products/database/asm/index.html

◦www.oracle.com/pls/db111/portal.portal_db?selected=4&frame=→ Storage → Storage Administrator’s Guide

Additional Hints on ASM Integration with SGeRAC 79

Image 79

HP Serviceguard Extension for RAC (SGeRAC) manual Additional Documentation on the Web and Scripts

Contents

Using Serviceguard Extension for RAC Version A.11.20 Legal Notices Contents Contents SGeRAC Toolkit for Oracle RAC 10g or later Support of Oracle RAC ASM with SGeRAC Maintenance 113 Blank Planning Worksheets 151 Troubleshooting 138 Software Upgrades 139Index 153 Advantages of using SGeRAC User Guide Overview Typographical Conventions Problem Reporting Where to find Documentation on the Web What is a Serviceguard Extension for RAC Cluster? Introduction to Serviceguard Extension for RAC Group Membership Using Packages in a Cluster Group Membership Daemon Serviceguard Extension for RAC Architecture About Veritas CFS and CVM from Symantec Storage Configuration OptionsServiceguard Extension for RAC Package Dependencies SMS bundle Overview of SGeRAC and Oracle 10g, 11gR1, and 11gR2 RAC How Cluster Interconnect Subnet Works Overview of SGeRAC Cluster Interconnect Subnet Monitoring Configuring Packages for Oracle Listeners Configuring Packages for Oracle RAC Instances Before Node Failure Node Failure Up to Four Nodes with Scsi Storage Larger Clusters Four-Node RAC Cluster Point-to-Point Connections to Storage Devices Eight-Node Cluster with EVA, XP or EMC Disk Array GMS Authorization Administering Clusters with Serviceguard Manager Overview of Serviceguard ManagerStarting Serviceguard Manager Monitoring Clusters with Serviceguard Manager Configuring Clusters with Serviceguard Manager Group Membership API NMAPI2 Cluster TimeoutsServiceguard Cluster Timeout Interface Areas Shared Storage Oracle Cluster Software Listener Network Monitoring RAC InstancesAutomated Startup and Shutdown Manual Startup and Shutdown Network Planning for Cluster Communication Planning Storage for Oracle 10g/11gR1/11gR2 RAC Planning Storage for Oracle Cluster Software Volume Planning with CVM Volume Planning with SlvmStorage Planning with CFS RAW Logical Volume Name Size MB About Device Special Files Installing Serviceguard Extension for RACAbout Multipathing About Cluster-wide Device Special Files cDSFs Where cDSFs Reside Cluster Communication Network Monitoring Configuration File Parameters Single Network for Cluster Communications Single Network for Cluster Communications SG-HB/RAC-IC Traffic Separation Alternate Configuration-Multiple RAC Databases When Cluster Interconnect Subnet Monitoring is not Used When Cluster Interconnect Subnet Monitoring is usedGuidelines for Changing Cluster Parameters Limitations of Cluster Communication Network Monitor Creating a Storage Infrastructure with LVMCluster Interconnect Monitoring Restrictions Creating Volume Groups and Logical Volumes Building Volume Groups for RAC on Mirrored DisksHP Serviceguard Extension for RAC # ls -l /dev/*/group Building Mirrored Logical Volumes for RAC with LVM Commands# mkdir /dev/vgrac # mknod /dev/vgrac/group c 64 0xhh0000 # lvcreate -m 1 -M y -s g -n redo1.log -L 408 /dev/vgrac Creating Mirrored Logical Volumes for RAC Data Files# lvcreate -m 1 -M y -s g -n system.dbf -L 408 dev/vgrac Creating RAC Volume Groups on Disk Arrays Oracle Demo Database Files Creating Logical Volumes for RAC on Disk Arrays Exporting the Logical Volume Infrastructure Displaying the Logical Volume InfrastructureExporting with LVM Commands Creating a Storage Infrastructure with CFS Installing Oracle Real Application Clusters # cmapplyconf -C clm.asc Initializing the Veritas Volume Manager# cmruncl # cmviewcl # vxdg -s init cfsdg1 c4t4d0 # cfscluster config -s# vxdctl -c mode # /etc/vx/bin/vxdisksetup -i c4t4d0 #cfsmntadm add cfsdg1 vol2 /cfs/mnt2 all=rw # newfs -F vxfs /dev/vx/rdsk/cfsdg1/vol2# newfs -F vxfs /dev/vx/rdsk/cfsdg1/vol3 # cfsmntadm add cfsdg1 vol1 /cfs/mnt1 all=rw # cfsmntadm delete /cfs/mnt2 Deleting CFS from the Cluster# bdf grep cfs # cfsmntadm delete /cfs/mnt1 # cfsdgadm delete cfsdg1 # cfscluster unconfig# vxinstall Creating a Storage Infrastructure with CVM Using CVM 5.x or later # vxdg -s init opsdg c4t4d0 # cmapplyconf -P /etc/cmcluster/cvm/VxVM-CVM-pkg.conf Using CVMPreparing the Cluster for Use with CVM # vxedit set diskdetpolicy=global DiskGroupName # vxdg list # /usr/lib/vxvm/bin/vxdisksetup -i /dev/dsk/c0t3d2# cmruncl # vxdg -s init opsdg c0t3d2 # vxassist -g opsdg make logfiles 1024m Creating VolumesOracle Demo Database Files Mirror Detachment Policies with CVM Adding Disk Groups to the Cluster Configuration # mkdir -p /mnt/app/crs/oracle/product/10.2.0/crs # groupadd oinstall # groupadd dba # groupadd oper# useradd -u 203 -g oinstall -G dba,oper oracle # passwd oracle # chmod 755 /dev/vgrac # usermod -d /cfs/mnt1/oracle oracle Mnt/app/oracle/oradata/ver10/ver10raw.conf Installing RAC Binaries on a Local File System Installing Oracle 10g, 11gR1, or 11gR2 Cluster SoftwareInstalling Oracle 10g/11gR1/11gR2 RAC Binaries Installing on Local File System $ netca Installing RAC Binaries on Cluster File SystemCreating a RAC Demo Database Creating a RAC Demo Database on Slvm or CVM Creating a RAC Demo Database on CFS Verifying Oracle Disk Manager is Configured#swlist VRTSodm # cat /dev/odm/stats Configuring Oracle to Use Oracle Disk Manager LibraryVerify that Oracle Disk Manager is Running #ll -L /opt/VRTSodm/lib/libodm.sl $ cd $ORACLEHOME/lib # kcmodule -P state odm $ORACLEHOME/bin/srvctl modify database -d dbname -y manual Preparing Oracle Cluster Software for Serviceguard PackagesConfigure Serviceguard Packages $ORACRSHOME/bin/crsctl disable crs Sbin/init.d/init.crs start Overview Why ASM over SLVM?Introduction SG/SGeRAC Support for ASM on HP-UX 11i Configuring Slvm Volume Groups for ASM Disk Groups Sample Command Sequence for Configuring Slvm Volume Groups Support of Oracle RAC ASM with SGeRAC ASM support with SG/SGeRAC A.11.17.01 or later ASM over Slvm Configuring Slvm Volume Groups for ASM Disk Groups Sample Command Sequence for Configuring Slvm Volume Groups ASM over Raw disk Additional Hints on ASM Integration with SGeRAC Installation, Configuration, Support, and Troubleshooting ASM may require Modified Backup/Restore Procedures Additional Documentation on the Web and Scripts Background SGeRAC Toolkit for Oracle RAC 10g or later Background SGeRAC Toolkit for Oracle RAC 10g or later Page Serviceguard Extension for RAC Toolkit operation Startup and shutdown of the combined Oracle RAC-SGeRAC stack SGeRAC Toolkit for Oracle RAC 10g or later Asmdg MNP Use case 3 Database storage in ASM over Slvm Use Case 2 Setup Use Case 2 performing maintenance with ASM disk groups Use Case 1 Performing maintenance with Oracle Clusterware SGeRAC Toolkit for Oracle RAC 10g or later Internal structure of SGeRAC for Oracle Clusterware Support for the SGeRAC Toolkit Contents Support for the SGeRAC Toolkit CFS-DG1-MNP CFS-DG2-MNP Readme SGeRAC Toolkit for Oracle RAC 10g or later DEPENDENCYCONDITIONDG-MNP-PKG=UP Dependencylocationsamenode ORACLEHOME, CHECKINTERVAL...MAINTENANCEFLAG Dependencyname DEPENDENCYCONDITIONOC-MNP-PKG=UP Dependencylocationsamenode Set to any name desired for the RAC MNP For a package using CFS Oracrshome Asmdisk Group Asmtkitdir SGeRAC Toolkit for Oracle RAC 10g or later Asmdg MNP SGeRAC Toolkit for Oracle RAC 10g or later Support for the SGeRAC Toolkit SGeRAC Toolkit for Oracle RAC 10g or later DEPENDENCYCONDITIONOC-DGMP-PKG=UP Dependencylocationsamenode Additional Documentation on the Web Conclusion Examples of Cluster and Package States Types of Cluster and Package States# cmviewcl -r A.11.16 Name Status State Types of Cluster and Package States Node Status and State Cluster StatusPackage Status and State Service Status Package Switching AttributesStatus of Group Membership Normal Running Status Examples of Cluster and Package StatesNetwork Status Failover and Failback Policies CVM Package Status Quorum Server Status Status After Moving the Package to Another Node Output of the cmviewcl -vcommand is as follows Status After Halting a Node Status After Package Switching is EnabledViewing Data on Unowned Packages Status Nodename Name Checking the Cluster Configuration and Components Verifying Cluster Components Checking Cluster Components Cmcheckconf -P Example Setting up Periodic Cluster Verification Limitations Online ReconfigurationOnline Node Addition and Deletion Managing the Shared Storage Making LVM Volume Groups Shareable Deactivating a Shared Volume Group Activating an LVM Volume Group in Shared ModeMaking Offline Changes to Shared Volume Groups Making a Volume Group Unshareable On node 2, issue the following command Adding Additional Shared LVM Volume Groups Removing Serviceguard Extension for RAC from a SystemChanging the CVM Storage Configuration Adding Disk Hardware Using Event Monitoring ServiceUsing EMS Hardware Monitors Monitoring Hardware # lvreduce -m 0 /dev/vgsg01/lvolname /dev/dsk/c2t3d0 Replacing a Mechanism in a Disk Array Configured with LVM# vgcfgrestore /dev/vgsg01 /dev/dsk/c2t3d0 Replacing Disks # pvchange -a n pv path # vgcfgrestore -n vg name pv raw path# lvextend -m 1 /dev/vgsg01 /dev/dsk/c2t3d0 # lvsync /dev/vgsg01/lvolname Replacing a Lock Disk Online Hardware Maintenance with Inline Scsi Terminator W Scsi Buses with Inline Terminators Online Replacement Replacement of I/O CardsReplacement of LAN Cards Offline Replacement Monitoring RAC Instances Troubleshooting Rolling Software Upgrades Software Upgrades Upgrading Serviceguard to SGeRAC cluster Cmhaltnode nodename Autostartcmcld = Steps for Rolling Upgrades Keeping Kernels Consistent Example of Rolling Upgrade Step Running Cluster Before Rolling Upgrade Node 1 Upgraded to SG/SGeRAC # cmhaltnode -f node2 Running Cluster After Upgrades Limitations of Rolling Upgrades Limitations of Non-Rolling Upgrades Non-Rolling Software UpgradesMigrating an SGeRAC Cluster with Cold Install Restrictions for DRD Upgrades Upgrade Using DRDRolling Upgrade Using DRD Non-Rolling Upgrade Using DRD Software Upgrades Oracle Logical Volume Worksheet LVM Volume Group and Physical Volume Worksheet Blank Planning Worksheets Index RAC 155