Legal Notices | HP Serviceguard Extension for RAC (SGeRAC) specification

Legal Notices

© Copyright 2011 Hewlett-Packard Development Company, L.P.

Confidential computer software. Valid license from HP required for possession, use, or copying. Consistent with FAR 12.211 and 12.212, Commercial Computer Software, Computer Software Documentation, and Technical Data for Commercial Items are licensed to the U.S. Government under vendor’s standard commercial license.

The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements a-ccompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.

Oracle® is a registered trademark of Oracle Corporation.

UNIX® is a registered trademark in the United States and other countries, licensed exclusively through The Open Group.

VERITAS® is a registered trademark of VERITAS Software Corporation.

VERITAS File System™ is a trademark of VERITAS Software Corporation.

Image 2

HP Serviceguard Extension for RAC (SGeRAC) manual Legal Notices

Contents

Using Serviceguard Extension for RAC Version A.11.20 Legal Notices Contents Contents Support of Oracle RAC ASM with SGeRAC SGeRAC Toolkit for Oracle RAC 10g or later Maintenance 113 Index 153 Troubleshooting 138 Software Upgrades 139Blank Planning Worksheets 151 Advantages of using SGeRAC User Guide Overview Problem Reporting Typographical Conventions Where to find Documentation on the Web Introduction to Serviceguard Extension for RAC What is a Serviceguard Extension for RAC Cluster? Using Packages in a Cluster Group Membership Serviceguard Extension for RAC Architecture Group Membership Daemon Package Dependencies Storage Configuration OptionsServiceguard Extension for RAC About Veritas CFS and CVM from Symantec Overview of SGeRAC and Oracle 10g, 11gR1, and 11gR2 RAC SMS bundle Overview of SGeRAC Cluster Interconnect Subnet Monitoring How Cluster Interconnect Subnet Works Configuring Packages for Oracle RAC Instances Configuring Packages for Oracle Listeners Node Failure Before Node Failure Larger Clusters Up to Four Nodes with Scsi Storage Point-to-Point Connections to Storage Devices Four-Node RAC Cluster GMS Authorization Eight-Node Cluster with EVA, XP or EMC Disk Array Monitoring Clusters with Serviceguard Manager Overview of Serviceguard ManagerStarting Serviceguard Manager Administering Clusters with Serviceguard Manager Configuring Clusters with Serviceguard Manager Interface Areas Cluster TimeoutsServiceguard Cluster Timeout Group Membership API NMAPI2 Oracle Cluster Software Shared Storage Listener Automated Startup and Shutdown RAC InstancesNetwork Monitoring Network Planning for Cluster Communication Manual Startup and Shutdown Planning Storage for Oracle Cluster Software Planning Storage for Oracle 10g/11gR1/11gR2 RAC Storage Planning with CFS Volume Planning with SlvmVolume Planning with CVM RAW Logical Volume Name Size MB About Multipathing Installing Serviceguard Extension for RACAbout Device Special Files Where cDSFs Reside About Cluster-wide Device Special Files cDSFs Configuration File Parameters Cluster Communication Network Monitoring Single Network for Cluster Communications Single Network for Cluster Communications SG-HB/RAC-IC Traffic Separation Alternate Configuration-Multiple RAC Databases Guidelines for Changing Cluster Parameters When Cluster Interconnect Subnet Monitoring is usedWhen Cluster Interconnect Subnet Monitoring is not Used Cluster Interconnect Monitoring Restrictions Creating a Storage Infrastructure with LVMLimitations of Cluster Communication Network Monitor HP Serviceguard Extension for RAC Building Volume Groups for RAC on Mirrored DisksCreating Volume Groups and Logical Volumes # mknod /dev/vgrac/group c 64 0xhh0000 Building Mirrored Logical Volumes for RAC with LVM Commands# mkdir /dev/vgrac # ls -l /dev/*/group # lvcreate -m 1 -M y -s g -n system.dbf -L 408 dev/vgrac Creating Mirrored Logical Volumes for RAC Data Files# lvcreate -m 1 -M y -s g -n redo1.log -L 408 /dev/vgrac Creating RAC Volume Groups on Disk Arrays Creating Logical Volumes for RAC on Disk Arrays Oracle Demo Database Files Exporting with LVM Commands Displaying the Logical Volume InfrastructureExporting the Logical Volume Infrastructure Installing Oracle Real Application Clusters Creating a Storage Infrastructure with CFS # cmruncl # cmviewcl Initializing the Veritas Volume Manager# cmapplyconf -C clm.asc # /etc/vx/bin/vxdisksetup -i c4t4d0 # cfscluster config -s# vxdctl -c mode # vxdg -s init cfsdg1 c4t4d0 # cfsmntadm add cfsdg1 vol1 /cfs/mnt1 all=rw # newfs -F vxfs /dev/vx/rdsk/cfsdg1/vol2# newfs -F vxfs /dev/vx/rdsk/cfsdg1/vol3 #cfsmntadm add cfsdg1 vol2 /cfs/mnt2 all=rw # cfsmntadm delete /cfs/mnt1 Deleting CFS from the Cluster# bdf grep cfs # cfsmntadm delete /cfs/mnt2 Creating a Storage Infrastructure with CVM # cfscluster unconfig# vxinstall # cfsdgadm delete cfsdg1 Using CVM 5.x or later # vxdg -s init opsdg c4t4d0 # vxedit set diskdetpolicy=global DiskGroupName Using CVMPreparing the Cluster for Use with CVM # cmapplyconf -P /etc/cmcluster/cvm/VxVM-CVM-pkg.conf # vxdg -s init opsdg c0t3d2 # /usr/lib/vxvm/bin/vxdisksetup -i /dev/dsk/c0t3d2# cmruncl # vxdg list Mirror Detachment Policies with CVM Creating VolumesOracle Demo Database Files # vxassist -g opsdg make logfiles 1024m Adding Disk Groups to the Cluster Configuration # passwd oracle # groupadd oinstall # groupadd dba # groupadd oper# useradd -u 203 -g oinstall -G dba,oper oracle # mkdir -p /mnt/app/crs/oracle/product/10.2.0/crs # usermod -d /cfs/mnt1/oracle oracle # chmod 755 /dev/vgrac Mnt/app/oracle/oradata/ver10/ver10raw.conf Installing on Local File System Installing Oracle 10g, 11gR1, or 11gR2 Cluster SoftwareInstalling Oracle 10g/11gR1/11gR2 RAC Binaries Installing RAC Binaries on a Local File System Creating a RAC Demo Database on Slvm or CVM Installing RAC Binaries on Cluster File SystemCreating a RAC Demo Database $ netca #swlist VRTSodm Verifying Oracle Disk Manager is ConfiguredCreating a RAC Demo Database on CFS #ll -L /opt/VRTSodm/lib/libodm.sl Configuring Oracle to Use Oracle Disk Manager LibraryVerify that Oracle Disk Manager is Running # cat /dev/odm/stats # kcmodule -P state odm $ cd $ORACLEHOME/lib $ORACRSHOME/bin/crsctl disable crs Preparing Oracle Cluster Software for Serviceguard PackagesConfigure Serviceguard Packages $ORACLEHOME/bin/srvctl modify database -d dbname -y manual Sbin/init.d/init.crs start SG/SGeRAC Support for ASM on HP-UX 11i Why ASM over SLVM?Introduction Overview Configuring Slvm Volume Groups for ASM Disk Groups Sample Command Sequence for Configuring Slvm Volume Groups Support of Oracle RAC ASM with SGeRAC ASM over Slvm ASM support with SG/SGeRAC A.11.17.01 or later Configuring Slvm Volume Groups for ASM Disk Groups Sample Command Sequence for Configuring Slvm Volume Groups ASM over Raw disk Additional Hints on ASM Integration with SGeRAC ASM may require Modified Backup/Restore Procedures Installation, Configuration, Support, and Troubleshooting Additional Documentation on the Web and Scripts SGeRAC Toolkit for Oracle RAC 10g or later Background Background SGeRAC Toolkit for Oracle RAC 10g or later Page Serviceguard Extension for RAC Toolkit operation Startup and shutdown of the combined Oracle RAC-SGeRAC stack SGeRAC Toolkit for Oracle RAC 10g or later Asmdg MNP Use Case 2 Setup Use case 3 Database storage in ASM over Slvm Use Case 1 Performing maintenance with Oracle Clusterware Use Case 2 performing maintenance with ASM disk groups SGeRAC Toolkit for Oracle RAC 10g or later Internal structure of SGeRAC for Oracle Clusterware Contents Support for the SGeRAC Toolkit Support for the SGeRAC Toolkit CFS-DG1-MNP CFS-DG2-MNP Readme SGeRAC Toolkit for Oracle RAC 10g or later DEPENDENCYCONDITIONDG-MNP-PKG=UP Dependencylocationsamenode ORACLEHOME, CHECKINTERVAL...MAINTENANCEFLAG Dependencyname DEPENDENCYCONDITIONOC-MNP-PKG=UP Dependencylocationsamenode Set to any name desired for the RAC MNP For a package using CFS Oracrshome Asmdisk Group Asmtkitdir SGeRAC Toolkit for Oracle RAC 10g or later Asmdg MNP SGeRAC Toolkit for Oracle RAC 10g or later Support for the SGeRAC Toolkit SGeRAC Toolkit for Oracle RAC 10g or later DEPENDENCYCONDITIONOC-DGMP-PKG=UP Dependencylocationsamenode Conclusion Additional Documentation on the Web # cmviewcl -r A.11.16 Types of Cluster and Package StatesExamples of Cluster and Package States Name Status State Types of Cluster and Package States Package Status and State Cluster StatusNode Status and State Status of Group Membership Package Switching AttributesService Status Failover and Failback Policies Examples of Cluster and Package StatesNetwork Status Normal Running Status Quorum Server Status CVM Package Status Output of the cmviewcl -vcommand is as follows Status After Moving the Package to Another Node Viewing Data on Unowned Packages Status After Package Switching is EnabledStatus After Halting a Node Checking the Cluster Configuration and Components Status Nodename Name Checking Cluster Components Verifying Cluster Components Cmcheckconf -P Setting up Periodic Cluster Verification Example Online Node Addition and Deletion Online ReconfigurationLimitations Making LVM Volume Groups Shareable Managing the Shared Storage Making a Volume Group Unshareable Activating an LVM Volume Group in Shared ModeMaking Offline Changes to Shared Volume Groups Deactivating a Shared Volume Group On node 2, issue the following command Changing the CVM Storage Configuration Removing Serviceguard Extension for RAC from a SystemAdding Additional Shared LVM Volume Groups Monitoring Hardware Using Event Monitoring ServiceUsing EMS Hardware Monitors Adding Disk Hardware Replacing Disks Replacing a Mechanism in a Disk Array Configured with LVM# vgcfgrestore /dev/vgsg01 /dev/dsk/c2t3d0 # lvreduce -m 0 /dev/vgsg01/lvolname /dev/dsk/c2t3d0 # lvsync /dev/vgsg01/lvolname # vgcfgrestore -n vg name pv raw path# lvextend -m 1 /dev/vgsg01 /dev/dsk/c2t3d0 # pvchange -a n pv path Online Hardware Maintenance with Inline Scsi Terminator Replacing a Lock Disk W Scsi Buses with Inline Terminators Offline Replacement Replacement of I/O CardsReplacement of LAN Cards Online Replacement Monitoring RAC Instances Troubleshooting Software Upgrades Rolling Software Upgrades Upgrading Serviceguard to SGeRAC cluster Cmhaltnode nodename Steps for Rolling Upgrades Autostartcmcld = Example of Rolling Upgrade Keeping Kernels Consistent Running Cluster Before Rolling Upgrade Step Node 1 Upgraded to SG/SGeRAC # cmhaltnode -f node2 Limitations of Rolling Upgrades Running Cluster After Upgrades Migrating an SGeRAC Cluster with Cold Install Non-Rolling Software UpgradesLimitations of Non-Rolling Upgrades Non-Rolling Upgrade Using DRD Upgrade Using DRDRolling Upgrade Using DRD Restrictions for DRD Upgrades Software Upgrades LVM Volume Group and Physical Volume Worksheet Oracle Logical Volume Worksheet Blank Planning Worksheets Index RAC 155