HP Serviceguard Extension for RAC (SGeRAC) guide

Configuring Clusters with Serviceguard Manager

You can configure clusters and packages in Serviceguard Manager. You must have root (UID=0) access to the cluster nodes.

24 Introduction to Serviceguard Extension for RAC

Image 24

HP Serviceguard Extension for RAC (SGeRAC) manual Configuring Clusters with Serviceguard Manager

Contents

Using Serviceguard Extension for RAC Version A.11.20 Legal Notices Contents Contents Support of Oracle RAC ASM with SGeRAC SGeRAC Toolkit for Oracle RAC 10g or later Maintenance 113 Troubleshooting 138 Software Upgrades 139 Blank Planning Worksheets 151Index 153 Advantages of using SGeRAC User Guide Overview Problem Reporting Typographical Conventions Where to find Documentation on the Web Introduction to Serviceguard Extension for RAC What is a Serviceguard Extension for RAC Cluster? Using Packages in a Cluster Group Membership Serviceguard Extension for RAC Architecture Group Membership Daemon Storage Configuration Options Serviceguard Extension for RACPackage Dependencies About Veritas CFS and CVM from Symantec Overview of SGeRAC and Oracle 10g, 11gR1, and 11gR2 RAC SMS bundle Overview of SGeRAC Cluster Interconnect Subnet Monitoring How Cluster Interconnect Subnet Works Configuring Packages for Oracle RAC Instances Configuring Packages for Oracle Listeners Node Failure Before Node Failure Larger Clusters Up to Four Nodes with Scsi Storage Point-to-Point Connections to Storage Devices Four-Node RAC Cluster GMS Authorization Eight-Node Cluster with EVA, XP or EMC Disk Array Overview of Serviceguard Manager Starting Serviceguard ManagerMonitoring Clusters with Serviceguard Manager Administering Clusters with Serviceguard Manager Configuring Clusters with Serviceguard Manager Cluster Timeouts Serviceguard Cluster TimeoutInterface Areas Group Membership API NMAPI2 Oracle Cluster Software Shared Storage Listener RAC Instances Network MonitoringAutomated Startup and Shutdown Network Planning for Cluster Communication Manual Startup and Shutdown Planning Storage for Oracle Cluster Software Planning Storage for Oracle 10g/11gR1/11gR2 RAC Volume Planning with Slvm Volume Planning with CVMStorage Planning with CFS RAW Logical Volume Name Size MB Installing Serviceguard Extension for RAC About Device Special FilesAbout Multipathing Where cDSFs Reside About Cluster-wide Device Special Files cDSFs Configuration File Parameters Cluster Communication Network Monitoring Single Network for Cluster Communications Single Network for Cluster Communications SG-HB/RAC-IC Traffic Separation Alternate Configuration-Multiple RAC Databases When Cluster Interconnect Subnet Monitoring is used When Cluster Interconnect Subnet Monitoring is not UsedGuidelines for Changing Cluster Parameters Creating a Storage Infrastructure with LVM Limitations of Cluster Communication Network MonitorCluster Interconnect Monitoring Restrictions Building Volume Groups for RAC on Mirrored Disks Creating Volume Groups and Logical VolumesHP Serviceguard Extension for RAC Building Mirrored Logical Volumes for RAC with LVM Commands # mkdir /dev/vgrac# mknod /dev/vgrac/group c 64 0xhh0000 # ls -l /dev/*/group Creating Mirrored Logical Volumes for RAC Data Files # lvcreate -m 1 -M y -s g -n redo1.log -L 408 /dev/vgrac# lvcreate -m 1 -M y -s g -n system.dbf -L 408 dev/vgrac Creating RAC Volume Groups on Disk Arrays Creating Logical Volumes for RAC on Disk Arrays Oracle Demo Database Files Displaying the Logical Volume Infrastructure Exporting the Logical Volume InfrastructureExporting with LVM Commands Installing Oracle Real Application Clusters Creating a Storage Infrastructure with CFS Initializing the Veritas Volume Manager # cmapplyconf -C clm.asc# cmruncl # cmviewcl # cfscluster config -s # vxdctl -c mode# /etc/vx/bin/vxdisksetup -i c4t4d0 # vxdg -s init cfsdg1 c4t4d0 # newfs -F vxfs /dev/vx/rdsk/cfsdg1/vol2 # newfs -F vxfs /dev/vx/rdsk/cfsdg1/vol3# cfsmntadm add cfsdg1 vol1 /cfs/mnt1 all=rw #cfsmntadm add cfsdg1 vol2 /cfs/mnt2 all=rw Deleting CFS from the Cluster # bdf grep cfs# cfsmntadm delete /cfs/mnt1 # cfsmntadm delete /cfs/mnt2 # cfscluster unconfig # vxinstallCreating a Storage Infrastructure with CVM # cfsdgadm delete cfsdg1 Using CVM 5.x or later # vxdg -s init opsdg c4t4d0 Using CVM Preparing the Cluster for Use with CVM# vxedit set diskdetpolicy=global DiskGroupName # cmapplyconf -P /etc/cmcluster/cvm/VxVM-CVM-pkg.conf # /usr/lib/vxvm/bin/vxdisksetup -i /dev/dsk/c0t3d2 # cmruncl# vxdg -s init opsdg c0t3d2 # vxdg list Creating Volumes Oracle Demo Database FilesMirror Detachment Policies with CVM # vxassist -g opsdg make logfiles 1024m Adding Disk Groups to the Cluster Configuration # groupadd oinstall # groupadd dba # groupadd oper # useradd -u 203 -g oinstall -G dba,oper oracle# passwd oracle # mkdir -p /mnt/app/crs/oracle/product/10.2.0/crs # usermod -d /cfs/mnt1/oracle oracle # chmod 755 /dev/vgrac Mnt/app/oracle/oradata/ver10/ver10raw.conf Installing Oracle 10g, 11gR1, or 11gR2 Cluster Software Installing Oracle 10g/11gR1/11gR2 RAC BinariesInstalling on Local File System Installing RAC Binaries on a Local File System Installing RAC Binaries on Cluster File System Creating a RAC Demo DatabaseCreating a RAC Demo Database on Slvm or CVM $ netca Verifying Oracle Disk Manager is Configured Creating a RAC Demo Database on CFS#swlist VRTSodm Configuring Oracle to Use Oracle Disk Manager Library Verify that Oracle Disk Manager is Running#ll -L /opt/VRTSodm/lib/libodm.sl # cat /dev/odm/stats # kcmodule -P state odm $ cd $ORACLEHOME/lib Preparing Oracle Cluster Software for Serviceguard Packages Configure Serviceguard Packages$ORACRSHOME/bin/crsctl disable crs $ORACLEHOME/bin/srvctl modify database -d dbname -y manual Sbin/init.d/init.crs start Why ASM over SLVM? IntroductionSG/SGeRAC Support for ASM on HP-UX 11i Overview Configuring Slvm Volume Groups for ASM Disk Groups Sample Command Sequence for Configuring Slvm Volume Groups Support of Oracle RAC ASM with SGeRAC ASM over Slvm ASM support with SG/SGeRAC A.11.17.01 or later Configuring Slvm Volume Groups for ASM Disk Groups Sample Command Sequence for Configuring Slvm Volume Groups ASM over Raw disk Additional Hints on ASM Integration with SGeRAC ASM may require Modified Backup/Restore Procedures Installation, Configuration, Support, and Troubleshooting Additional Documentation on the Web and Scripts SGeRAC Toolkit for Oracle RAC 10g or later Background Background SGeRAC Toolkit for Oracle RAC 10g or later Page Serviceguard Extension for RAC Toolkit operation Startup and shutdown of the combined Oracle RAC-SGeRAC stack SGeRAC Toolkit for Oracle RAC 10g or later Asmdg MNP Use Case 2 Setup Use case 3 Database storage in ASM over Slvm Use Case 1 Performing maintenance with Oracle Clusterware Use Case 2 performing maintenance with ASM disk groups SGeRAC Toolkit for Oracle RAC 10g or later Internal structure of SGeRAC for Oracle Clusterware Contents Support for the SGeRAC Toolkit Support for the SGeRAC Toolkit CFS-DG1-MNP CFS-DG2-MNP Readme SGeRAC Toolkit for Oracle RAC 10g or later DEPENDENCYCONDITIONDG-MNP-PKG=UP Dependencylocationsamenode ORACLEHOME, CHECKINTERVAL...MAINTENANCEFLAG Dependencyname DEPENDENCYCONDITIONOC-MNP-PKG=UP Dependencylocationsamenode Set to any name desired for the RAC MNP For a package using CFS Oracrshome Asmdisk Group Asmtkitdir SGeRAC Toolkit for Oracle RAC 10g or later Asmdg MNP SGeRAC Toolkit for Oracle RAC 10g or later Support for the SGeRAC Toolkit SGeRAC Toolkit for Oracle RAC 10g or later DEPENDENCYCONDITIONOC-DGMP-PKG=UP Dependencylocationsamenode Conclusion Additional Documentation on the Web Types of Cluster and Package States Examples of Cluster and Package States# cmviewcl -r A.11.16 Name Status State Types of Cluster and Package States Cluster Status Node Status and StatePackage Status and State Package Switching Attributes Service StatusStatus of Group Membership Examples of Cluster and Package States Network StatusFailover and Failback Policies Normal Running Status Quorum Server Status CVM Package Status Output of the cmviewcl -vcommand is as follows Status After Moving the Package to Another Node Status After Package Switching is Enabled Status After Halting a NodeViewing Data on Unowned Packages Checking the Cluster Configuration and Components Status Nodename Name Checking Cluster Components Verifying Cluster Components Cmcheckconf -P Setting up Periodic Cluster Verification Example Online Reconfiguration LimitationsOnline Node Addition and Deletion Making LVM Volume Groups Shareable Managing the Shared Storage Activating an LVM Volume Group in Shared Mode Making Offline Changes to Shared Volume GroupsMaking a Volume Group Unshareable Deactivating a Shared Volume Group On node 2, issue the following command Removing Serviceguard Extension for RAC from a System Adding Additional Shared LVM Volume GroupsChanging the CVM Storage Configuration Using Event Monitoring Service Using EMS Hardware MonitorsMonitoring Hardware Adding Disk Hardware Replacing a Mechanism in a Disk Array Configured with LVM # vgcfgrestore /dev/vgsg01 /dev/dsk/c2t3d0Replacing Disks # lvreduce -m 0 /dev/vgsg01/lvolname /dev/dsk/c2t3d0 # vgcfgrestore -n vg name pv raw path # lvextend -m 1 /dev/vgsg01 /dev/dsk/c2t3d0# lvsync /dev/vgsg01/lvolname # pvchange -a n pv path Online Hardware Maintenance with Inline Scsi Terminator Replacing a Lock Disk W Scsi Buses with Inline Terminators Replacement of I/O Cards Replacement of LAN CardsOffline Replacement Online Replacement Monitoring RAC Instances Troubleshooting Software Upgrades Rolling Software Upgrades Upgrading Serviceguard to SGeRAC cluster Cmhaltnode nodename Steps for Rolling Upgrades Autostartcmcld = Example of Rolling Upgrade Keeping Kernels Consistent Running Cluster Before Rolling Upgrade Step Node 1 Upgraded to SG/SGeRAC # cmhaltnode -f node2 Limitations of Rolling Upgrades Running Cluster After Upgrades Non-Rolling Software Upgrades Limitations of Non-Rolling UpgradesMigrating an SGeRAC Cluster with Cold Install Upgrade Using DRD Rolling Upgrade Using DRDNon-Rolling Upgrade Using DRD Restrictions for DRD Upgrades Software Upgrades LVM Volume Group and Physical Volume Worksheet Oracle Logical Volume Worksheet Blank Planning Worksheets Index RAC 155