Using Serviceguard Extension for RAC Version A.11.20
Legal Notices
Contents
Contents
SGeRAC Toolkit for Oracle RAC 10g or later
Support of Oracle RAC ASM with SGeRAC
Maintenance 113
Index 153
Troubleshooting 138 Software Upgrades 139
Blank Planning Worksheets 151
Advantages of using SGeRAC
User Guide Overview
Typographical Conventions
Problem Reporting
Where to find Documentation on the Web
What is a Serviceguard Extension for RAC Cluster?
Introduction to Serviceguard Extension for RAC
Group Membership
Using Packages in a Cluster
Group Membership Daemon
Serviceguard Extension for RAC Architecture
About Veritas CFS and CVM from Symantec
Storage Configuration Options
Serviceguard Extension for RAC
Package Dependencies
SMS bundle
Overview of SGeRAC and Oracle 10g, 11gR1, and 11gR2 RAC
How Cluster Interconnect Subnet Works
Overview of SGeRAC Cluster Interconnect Subnet Monitoring
Configuring Packages for Oracle Listeners
Configuring Packages for Oracle RAC Instances
Before Node Failure
Node Failure
Up to Four Nodes with Scsi Storage
Larger Clusters
Four-Node RAC Cluster
Point-to-Point Connections to Storage Devices
Eight-Node Cluster with EVA, XP or EMC Disk Array
GMS Authorization
Administering Clusters with Serviceguard Manager
Overview of Serviceguard Manager
Starting Serviceguard Manager
Monitoring Clusters with Serviceguard Manager
Configuring Clusters with Serviceguard Manager
Group Membership API NMAPI2
Cluster Timeouts
Serviceguard Cluster Timeout
Interface Areas
Shared Storage
Oracle Cluster Software
Listener
Automated Startup and Shutdown
RAC Instances
Network Monitoring
Manual Startup and Shutdown
Network Planning for Cluster Communication
Planning Storage for Oracle 10g/11gR1/11gR2 RAC
Planning Storage for Oracle Cluster Software
Storage Planning with CFS
Volume Planning with Slvm
Volume Planning with CVM
RAW Logical Volume Name Size MB
About Multipathing
Installing Serviceguard Extension for RAC
About Device Special Files
About Cluster-wide Device Special Files cDSFs
Where cDSFs Reside
Cluster Communication Network Monitoring
Configuration File Parameters
Single Network for Cluster Communications
Single Network for Cluster Communications
SG-HB/RAC-IC Traffic Separation
Alternate Configuration-Multiple RAC Databases
Guidelines for Changing Cluster Parameters
When Cluster Interconnect Subnet Monitoring is used
When Cluster Interconnect Subnet Monitoring is not Used
Cluster Interconnect Monitoring Restrictions
Creating a Storage Infrastructure with LVM
Limitations of Cluster Communication Network Monitor
HP Serviceguard Extension for RAC
Building Volume Groups for RAC on Mirrored Disks
Creating Volume Groups and Logical Volumes
# ls -l /dev/*/group
Building Mirrored Logical Volumes for RAC with LVM Commands
# mkdir /dev/vgrac
# mknod /dev/vgrac/group c 64 0xhh0000
# lvcreate -m 1 -M y -s g -n system.dbf -L 408 dev/vgrac
Creating Mirrored Logical Volumes for RAC Data Files
# lvcreate -m 1 -M y -s g -n redo1.log -L 408 /dev/vgrac
Creating RAC Volume Groups on Disk Arrays
Oracle Demo Database Files
Creating Logical Volumes for RAC on Disk Arrays
Exporting with LVM Commands
Displaying the Logical Volume Infrastructure
Exporting the Logical Volume Infrastructure
Creating a Storage Infrastructure with CFS
Installing Oracle Real Application Clusters
# cmruncl # cmviewcl
Initializing the Veritas Volume Manager
# cmapplyconf -C clm.asc
# vxdg -s init cfsdg1 c4t4d0
# cfscluster config -s
# vxdctl -c mode
# /etc/vx/bin/vxdisksetup -i c4t4d0
#cfsmntadm add cfsdg1 vol2 /cfs/mnt2 all=rw
# newfs -F vxfs /dev/vx/rdsk/cfsdg1/vol2
# newfs -F vxfs /dev/vx/rdsk/cfsdg1/vol3
# cfsmntadm add cfsdg1 vol1 /cfs/mnt1 all=rw
# cfsmntadm delete /cfs/mnt2
Deleting CFS from the Cluster
# bdf grep cfs
# cfsmntadm delete /cfs/mnt1
# cfsdgadm delete cfsdg1
# cfscluster unconfig
# vxinstall
Creating a Storage Infrastructure with CVM
Using CVM 5.x or later
# vxdg -s init opsdg c4t4d0
# cmapplyconf -P /etc/cmcluster/cvm/VxVM-CVM-pkg.conf
Using CVM
Preparing the Cluster for Use with CVM
# vxedit set diskdetpolicy=global DiskGroupName
# vxdg list
# /usr/lib/vxvm/bin/vxdisksetup -i /dev/dsk/c0t3d2
# cmruncl
# vxdg -s init opsdg c0t3d2
# vxassist -g opsdg make logfiles 1024m
Creating Volumes
Oracle Demo Database Files
Mirror Detachment Policies with CVM
Adding Disk Groups to the Cluster Configuration
# mkdir -p /mnt/app/crs/oracle/product/10.2.0/crs
# groupadd oinstall # groupadd dba # groupadd oper
# useradd -u 203 -g oinstall -G dba,oper oracle
# passwd oracle
# chmod 755 /dev/vgrac
# usermod -d /cfs/mnt1/oracle oracle
Mnt/app/oracle/oradata/ver10/ver10raw.conf
Installing RAC Binaries on a Local File System
Installing Oracle 10g, 11gR1, or 11gR2 Cluster Software
Installing Oracle 10g/11gR1/11gR2 RAC Binaries
Installing on Local File System
$ netca
Installing RAC Binaries on Cluster File System
Creating a RAC Demo Database
Creating a RAC Demo Database on Slvm or CVM
#swlist VRTSodm
Verifying Oracle Disk Manager is Configured
Creating a RAC Demo Database on CFS
# cat /dev/odm/stats
Configuring Oracle to Use Oracle Disk Manager Library
Verify that Oracle Disk Manager is Running
#ll -L /opt/VRTSodm/lib/libodm.sl
$ cd $ORACLEHOME/lib
# kcmodule -P state odm
$ORACLEHOME/bin/srvctl modify database -d dbname -y manual
Preparing Oracle Cluster Software for Serviceguard Packages
Configure Serviceguard Packages
$ORACRSHOME/bin/crsctl disable crs
Sbin/init.d/init.crs start
Overview
Why ASM over SLVM?
Introduction
SG/SGeRAC Support for ASM on HP-UX 11i
Configuring Slvm Volume Groups for ASM Disk Groups
Sample Command Sequence for Configuring Slvm Volume Groups
Support of Oracle RAC ASM with SGeRAC
ASM support with SG/SGeRAC A.11.17.01 or later
ASM over Slvm
Configuring Slvm Volume Groups for ASM Disk Groups
Sample Command Sequence for Configuring Slvm Volume Groups
ASM over Raw disk
Additional Hints on ASM Integration with SGeRAC
Installation, Configuration, Support, and Troubleshooting
ASM may require Modified Backup/Restore Procedures
Additional Documentation on the Web and Scripts
Background
SGeRAC Toolkit for Oracle RAC 10g or later
Background
SGeRAC Toolkit for Oracle RAC 10g or later
Page
Serviceguard Extension for RAC Toolkit operation
Startup and shutdown of the combined Oracle RAC-SGeRAC stack
SGeRAC Toolkit for Oracle RAC 10g or later
Asmdg MNP
Use case 3 Database storage in ASM over Slvm
Use Case 2 Setup
Use Case 2 performing maintenance with ASM disk groups
Use Case 1 Performing maintenance with Oracle Clusterware
SGeRAC Toolkit for Oracle RAC 10g or later
Internal structure of SGeRAC for Oracle Clusterware
Support for the SGeRAC Toolkit
Contents
Support for the SGeRAC Toolkit
CFS-DG1-MNP CFS-DG2-MNP
Readme
SGeRAC Toolkit for Oracle RAC 10g or later
DEPENDENCYCONDITIONDG-MNP-PKG=UP Dependencylocationsamenode
ORACLEHOME, CHECKINTERVAL...MAINTENANCEFLAG
Dependencyname
DEPENDENCYCONDITIONOC-MNP-PKG=UP Dependencylocationsamenode
Set to any name desired for the RAC MNP
For a package using CFS
Oracrshome
Asmdisk Group
Asmtkitdir
SGeRAC Toolkit for Oracle RAC 10g or later
Asmdg MNP
SGeRAC Toolkit for Oracle RAC 10g or later
Support for the SGeRAC Toolkit
SGeRAC Toolkit for Oracle RAC 10g or later
DEPENDENCYCONDITIONOC-DGMP-PKG=UP Dependencylocationsamenode
Additional Documentation on the Web
Conclusion
# cmviewcl -r A.11.16
Types of Cluster and Package States
Examples of Cluster and Package States
Name Status State
Types of Cluster and Package States
Package Status and State
Cluster Status
Node Status and State
Status of Group Membership
Package Switching Attributes
Service Status
Normal Running Status
Examples of Cluster and Package States
Network Status
Failover and Failback Policies
CVM Package Status
Quorum Server Status
Status After Moving the Package to Another Node
Output of the cmviewcl -vcommand is as follows
Viewing Data on Unowned Packages
Status After Package Switching is Enabled
Status After Halting a Node
Status Nodename Name
Checking the Cluster Configuration and Components
Verifying Cluster Components
Checking Cluster Components
Cmcheckconf -P
Example
Setting up Periodic Cluster Verification
Online Node Addition and Deletion
Online Reconfiguration
Limitations
Managing the Shared Storage
Making LVM Volume Groups Shareable
Deactivating a Shared Volume Group
Activating an LVM Volume Group in Shared Mode
Making Offline Changes to Shared Volume Groups
Making a Volume Group Unshareable
On node 2, issue the following command
Changing the CVM Storage Configuration
Removing Serviceguard Extension for RAC from a System
Adding Additional Shared LVM Volume Groups
Adding Disk Hardware
Using Event Monitoring Service
Using EMS Hardware Monitors
Monitoring Hardware
# lvreduce -m 0 /dev/vgsg01/lvolname /dev/dsk/c2t3d0
Replacing a Mechanism in a Disk Array Configured with LVM
# vgcfgrestore /dev/vgsg01 /dev/dsk/c2t3d0
Replacing Disks
# pvchange -a n pv path
# vgcfgrestore -n vg name pv raw path
# lvextend -m 1 /dev/vgsg01 /dev/dsk/c2t3d0
# lvsync /dev/vgsg01/lvolname
Replacing a Lock Disk
Online Hardware Maintenance with Inline Scsi Terminator
W Scsi Buses with Inline Terminators
Online Replacement
Replacement of I/O Cards
Replacement of LAN Cards
Offline Replacement
Monitoring RAC Instances
Troubleshooting
Rolling Software Upgrades
Software Upgrades
Upgrading Serviceguard to SGeRAC cluster
Cmhaltnode nodename
Autostartcmcld =
Steps for Rolling Upgrades
Keeping Kernels Consistent
Example of Rolling Upgrade
Step
Running Cluster Before Rolling Upgrade
Node 1 Upgraded to SG/SGeRAC
# cmhaltnode -f node2
Running Cluster After Upgrades
Limitations of Rolling Upgrades
Migrating an SGeRAC Cluster with Cold Install
Non-Rolling Software Upgrades
Limitations of Non-Rolling Upgrades
Restrictions for DRD Upgrades
Upgrade Using DRD
Rolling Upgrade Using DRD
Non-Rolling Upgrade Using DRD
Software Upgrades
Oracle Logical Volume Worksheet
LVM Volume Group and Physical Volume Worksheet
Blank Planning Worksheets
Index
RAC
155