Using Serviceguard Extension for RAC Version A.11.20
 Legal Notices
 Contents
 Contents
 Support of Oracle RAC ASM with SGeRAC
SGeRAC Toolkit for Oracle RAC 10g or later
 Maintenance 113
 Blank Planning Worksheets 151
Troubleshooting 138 Software Upgrades 139
Index 153
 Advantages of using SGeRAC
 User Guide Overview
 Problem Reporting
Typographical Conventions
 Where to find Documentation on the Web
 Introduction to Serviceguard Extension for RAC
What is a Serviceguard Extension for RAC Cluster?
 Using Packages in a Cluster
Group Membership
 Serviceguard Extension for RAC Architecture
Group Membership Daemon
 Storage Configuration Options
Serviceguard Extension for RAC
Package Dependencies
About Veritas CFS and CVM from Symantec
 Overview of SGeRAC and Oracle 10g, 11gR1, and 11gR2 RAC
SMS bundle
 Overview of SGeRAC Cluster Interconnect Subnet Monitoring
How Cluster Interconnect Subnet Works
 Configuring Packages for Oracle RAC Instances
Configuring Packages for Oracle Listeners
 Node Failure
Before Node Failure
 Larger Clusters
Up to Four Nodes with Scsi Storage
 Point-to-Point Connections to Storage Devices
Four-Node RAC Cluster
 GMS Authorization
Eight-Node Cluster with EVA, XP or EMC Disk Array
 Overview of Serviceguard Manager
Starting Serviceguard Manager
Monitoring Clusters with Serviceguard Manager
Administering Clusters with Serviceguard Manager
 Configuring Clusters with Serviceguard Manager
 Cluster Timeouts
Serviceguard Cluster Timeout
Interface Areas
Group Membership API NMAPI2
 Oracle Cluster Software
Shared Storage
 Listener
 Network Monitoring
RAC Instances
Automated Startup and Shutdown
 Network Planning for Cluster Communication
Manual Startup and Shutdown
 Planning Storage for Oracle Cluster Software
Planning Storage for Oracle 10g/11gR1/11gR2 RAC
 Volume Planning with CVM
Volume Planning with Slvm
Storage Planning with CFS
 RAW Logical Volume Name Size MB
 About Device Special Files
Installing Serviceguard Extension for RAC
About Multipathing
 Where cDSFs Reside
About Cluster-wide Device Special Files cDSFs
 Configuration File Parameters
Cluster Communication Network Monitoring
 Single Network for Cluster Communications
Single Network for Cluster Communications
 SG-HB/RAC-IC Traffic Separation
 Alternate Configuration-Multiple RAC Databases
 When Cluster Interconnect Subnet Monitoring is not Used
When Cluster Interconnect Subnet Monitoring is used
Guidelines for Changing Cluster Parameters
 Limitations of Cluster Communication Network Monitor
Creating a Storage Infrastructure with LVM
Cluster Interconnect Monitoring Restrictions
 Creating Volume Groups and Logical Volumes
Building Volume Groups for RAC on Mirrored Disks
HP Serviceguard Extension for RAC
 Building Mirrored Logical Volumes for RAC with LVM Commands
# mkdir /dev/vgrac
# mknod /dev/vgrac/group c 64 0xhh0000
# ls -l /dev/*/group
 # lvcreate -m 1 -M y -s g -n redo1.log -L 408 /dev/vgrac
Creating Mirrored Logical Volumes for RAC Data Files
# lvcreate -m 1 -M y -s g -n system.dbf -L 408 dev/vgrac
 Creating RAC Volume Groups on Disk Arrays
 Creating Logical Volumes for RAC on Disk Arrays
Oracle Demo Database Files
 Exporting the Logical Volume Infrastructure
Displaying the Logical Volume Infrastructure
Exporting with LVM Commands
 Installing Oracle Real Application Clusters
Creating a Storage Infrastructure with CFS
 # cmapplyconf -C clm.asc
Initializing the Veritas Volume Manager
# cmruncl # cmviewcl
 # cfscluster config -s
# vxdctl -c mode
# /etc/vx/bin/vxdisksetup -i c4t4d0
# vxdg -s init cfsdg1 c4t4d0
 # newfs -F vxfs /dev/vx/rdsk/cfsdg1/vol2
# newfs -F vxfs /dev/vx/rdsk/cfsdg1/vol3
# cfsmntadm add cfsdg1 vol1 /cfs/mnt1 all=rw
#cfsmntadm add cfsdg1 vol2 /cfs/mnt2 all=rw
 Deleting CFS from the Cluster
# bdf grep cfs
# cfsmntadm delete /cfs/mnt1
# cfsmntadm delete /cfs/mnt2
 # cfscluster unconfig
# vxinstall
Creating a Storage Infrastructure with CVM
# cfsdgadm delete cfsdg1
 Using CVM 5.x or later
 # vxdg -s init opsdg c4t4d0
 Using CVM
Preparing the Cluster for Use with CVM
# vxedit set diskdetpolicy=global DiskGroupName
# cmapplyconf -P /etc/cmcluster/cvm/VxVM-CVM-pkg.conf
 # /usr/lib/vxvm/bin/vxdisksetup -i /dev/dsk/c0t3d2
# cmruncl
# vxdg -s init opsdg c0t3d2
# vxdg list
 Creating Volumes
Oracle Demo Database Files
Mirror Detachment Policies with CVM
# vxassist -g opsdg make logfiles 1024m
 Adding Disk Groups to the Cluster Configuration
 # groupadd oinstall # groupadd dba # groupadd oper
# useradd -u 203 -g oinstall -G dba,oper oracle
# passwd oracle
# mkdir -p /mnt/app/crs/oracle/product/10.2.0/crs
 # usermod -d /cfs/mnt1/oracle oracle
# chmod 755 /dev/vgrac
 Mnt/app/oracle/oradata/ver10/ver10raw.conf
 Installing Oracle 10g, 11gR1, or 11gR2 Cluster Software
Installing Oracle 10g/11gR1/11gR2 RAC Binaries
Installing on Local File System
Installing RAC Binaries on a Local File System
 Installing RAC Binaries on Cluster File System
Creating a RAC Demo Database
Creating a RAC Demo Database on Slvm or CVM
$ netca
 Creating a RAC Demo Database on CFS
Verifying Oracle Disk Manager is Configured
#swlist VRTSodm
 Configuring Oracle to Use Oracle Disk Manager Library
Verify that Oracle Disk Manager is Running
#ll -L /opt/VRTSodm/lib/libodm.sl
# cat /dev/odm/stats
 # kcmodule -P state odm
$ cd $ORACLEHOME/lib
 Preparing Oracle Cluster Software for Serviceguard Packages
Configure Serviceguard Packages
$ORACRSHOME/bin/crsctl disable crs
$ORACLEHOME/bin/srvctl modify database -d dbname -y manual
 Sbin/init.d/init.crs start
 Why ASM over SLVM?
Introduction
SG/SGeRAC Support for ASM on HP-UX 11i
Overview
 Configuring Slvm Volume Groups for ASM Disk Groups
 Sample Command Sequence for Configuring Slvm Volume Groups
 Support of Oracle RAC ASM with SGeRAC
 ASM over Slvm
ASM support with SG/SGeRAC A.11.17.01 or later
 Configuring Slvm Volume Groups for ASM Disk Groups
 Sample Command Sequence for Configuring Slvm Volume Groups
 ASM over Raw disk
 Additional Hints on ASM Integration with SGeRAC
 ASM may require Modified Backup/Restore Procedures
Installation, Configuration, Support, and Troubleshooting
 Additional Documentation on the Web and Scripts
 SGeRAC Toolkit for Oracle RAC 10g or later
Background
 Background
 SGeRAC Toolkit for Oracle RAC 10g or later
Page
 Serviceguard Extension for RAC Toolkit operation
 Startup and shutdown of the combined Oracle RAC-SGeRAC stack
 SGeRAC Toolkit for Oracle RAC 10g or later
 Asmdg MNP
 Use Case 2 Setup
Use case 3 Database storage in ASM over Slvm
 Use Case 1 Performing maintenance with Oracle Clusterware
Use Case 2 performing maintenance with ASM disk groups
 SGeRAC Toolkit for Oracle RAC 10g or later
 Internal structure of SGeRAC for Oracle Clusterware
 Contents
Support for the SGeRAC Toolkit
 Support for the SGeRAC Toolkit
 CFS-DG1-MNP CFS-DG2-MNP
 Readme
 SGeRAC Toolkit for Oracle RAC 10g or later
 DEPENDENCYCONDITIONDG-MNP-PKG=UP Dependencylocationsamenode
 ORACLEHOME, CHECKINTERVAL...MAINTENANCEFLAG
 Dependencyname
 DEPENDENCYCONDITIONOC-MNP-PKG=UP Dependencylocationsamenode
 Set to any name desired for the RAC MNP
 For a package using CFS
 Oracrshome
 Asmdisk Group
 Asmtkitdir
 SGeRAC Toolkit for Oracle RAC 10g or later
 Asmdg MNP
 SGeRAC Toolkit for Oracle RAC 10g or later
 Support for the SGeRAC Toolkit
 SGeRAC Toolkit for Oracle RAC 10g or later
 DEPENDENCYCONDITIONOC-DGMP-PKG=UP Dependencylocationsamenode
 Conclusion
Additional Documentation on the Web
 Examples of Cluster and Package States
Types of Cluster and Package States
# cmviewcl -r A.11.16
 Name Status State
 Types of Cluster and Package States
 Node Status and State
Cluster Status
Package Status and State
 Service Status
Package Switching Attributes
Status of Group Membership
 Examples of Cluster and Package States
Network Status
Failover and Failback Policies
Normal Running Status
 Quorum Server Status
CVM Package Status
 Output of the cmviewcl -vcommand is as follows
Status After Moving the Package to Another Node
 Status After Halting a Node
Status After Package Switching is Enabled
Viewing Data on Unowned Packages
 Checking the Cluster Configuration and Components
Status Nodename Name
 Checking Cluster Components
Verifying Cluster Components
 Cmcheckconf -P
 Setting up Periodic Cluster Verification
Example
 Limitations
Online Reconfiguration
Online Node Addition and Deletion
 Making LVM Volume Groups Shareable
Managing the Shared Storage
 Activating an LVM Volume Group in Shared Mode
Making Offline Changes to Shared Volume Groups
Making a Volume Group Unshareable
Deactivating a Shared Volume Group
 On node 2, issue the following command
 Adding Additional Shared LVM Volume Groups
Removing Serviceguard Extension for RAC from a System
Changing the CVM Storage Configuration
 Using Event Monitoring Service
Using EMS Hardware Monitors
Monitoring Hardware
Adding Disk Hardware
 Replacing a Mechanism in a Disk Array Configured with LVM
# vgcfgrestore /dev/vgsg01 /dev/dsk/c2t3d0
Replacing Disks
# lvreduce -m 0 /dev/vgsg01/lvolname /dev/dsk/c2t3d0
 # vgcfgrestore -n vg name pv raw path
# lvextend -m 1 /dev/vgsg01 /dev/dsk/c2t3d0
# lvsync /dev/vgsg01/lvolname
# pvchange -a n pv path
 Online Hardware Maintenance with Inline Scsi Terminator
Replacing a Lock Disk
 W Scsi Buses with Inline Terminators
 Replacement of I/O Cards
Replacement of LAN Cards
Offline Replacement
Online Replacement
 Monitoring RAC Instances
 Troubleshooting
 Software Upgrades
Rolling Software Upgrades
 Upgrading Serviceguard to SGeRAC cluster
 Cmhaltnode nodename
 Steps for Rolling Upgrades
Autostartcmcld =
 Example of Rolling Upgrade
Keeping Kernels Consistent
 Running Cluster Before Rolling Upgrade
Step
 Node 1 Upgraded to SG/SGeRAC
 # cmhaltnode -f node2
 Limitations of Rolling Upgrades
Running Cluster After Upgrades
 Limitations of Non-Rolling Upgrades
Non-Rolling Software Upgrades
Migrating an SGeRAC Cluster with Cold Install
 Upgrade Using DRD
Rolling Upgrade Using DRD
Non-Rolling Upgrade Using DRD
Restrictions for DRD Upgrades
 Software Upgrades
 LVM Volume Group and Physical Volume Worksheet
Oracle Logical Volume Worksheet
 Blank Planning Worksheets
 Index
 RAC
 155