Using Serviceguard Extension for RAC Version A.11.20
Legal Notices
Contents
Contents
SGeRAC Toolkit for Oracle RAC 10g or later
Support of Oracle RAC ASM with SGeRAC
Maintenance 113
Troubleshooting 138 Software Upgrades 139
Blank Planning Worksheets 151
Index 153
Advantages of using SGeRAC
User Guide Overview
Typographical Conventions
Problem Reporting
Where to find Documentation on the Web
What is a Serviceguard Extension for RAC Cluster?
Introduction to Serviceguard Extension for RAC
Group Membership
Using Packages in a Cluster
Group Membership Daemon
Serviceguard Extension for RAC Architecture
Serviceguard Extension for RAC
Storage Configuration Options
Package Dependencies
About Veritas CFS and CVM from Symantec
SMS bundle
Overview of SGeRAC and Oracle 10g, 11gR1, and 11gR2 RAC
How Cluster Interconnect Subnet Works
Overview of SGeRAC Cluster Interconnect Subnet Monitoring
Configuring Packages for Oracle Listeners
Configuring Packages for Oracle RAC Instances
Before Node Failure
Node Failure
Up to Four Nodes with Scsi Storage
Larger Clusters
Four-Node RAC Cluster
Point-to-Point Connections to Storage Devices
Eight-Node Cluster with EVA, XP or EMC Disk Array
GMS Authorization
Starting Serviceguard Manager
Overview of Serviceguard Manager
Monitoring Clusters with Serviceguard Manager
Administering Clusters with Serviceguard Manager
Configuring Clusters with Serviceguard Manager
Serviceguard Cluster Timeout
Cluster Timeouts
Interface Areas
Group Membership API NMAPI2
Shared Storage
Oracle Cluster Software
Listener
RAC Instances
Network Monitoring
Automated Startup and Shutdown
Manual Startup and Shutdown
Network Planning for Cluster Communication
Planning Storage for Oracle 10g/11gR1/11gR2 RAC
Planning Storage for Oracle Cluster Software
Volume Planning with Slvm
Volume Planning with CVM
Storage Planning with CFS
RAW Logical Volume Name Size MB
Installing Serviceguard Extension for RAC
About Device Special Files
About Multipathing
About Cluster-wide Device Special Files cDSFs
Where cDSFs Reside
Cluster Communication Network Monitoring
Configuration File Parameters
Single Network for Cluster Communications
Single Network for Cluster Communications
SG-HB/RAC-IC Traffic Separation
Alternate Configuration-Multiple RAC Databases
When Cluster Interconnect Subnet Monitoring is used
When Cluster Interconnect Subnet Monitoring is not Used
Guidelines for Changing Cluster Parameters
Creating a Storage Infrastructure with LVM
Limitations of Cluster Communication Network Monitor
Cluster Interconnect Monitoring Restrictions
Building Volume Groups for RAC on Mirrored Disks
Creating Volume Groups and Logical Volumes
HP Serviceguard Extension for RAC
# mkdir /dev/vgrac
Building Mirrored Logical Volumes for RAC with LVM Commands
# mknod /dev/vgrac/group c 64 0xhh0000
# ls -l /dev/*/group
Creating Mirrored Logical Volumes for RAC Data Files
# lvcreate -m 1 -M y -s g -n redo1.log -L 408 /dev/vgrac
# lvcreate -m 1 -M y -s g -n system.dbf -L 408 dev/vgrac
Creating RAC Volume Groups on Disk Arrays
Oracle Demo Database Files
Creating Logical Volumes for RAC on Disk Arrays
Displaying the Logical Volume Infrastructure
Exporting the Logical Volume Infrastructure
Exporting with LVM Commands
Creating a Storage Infrastructure with CFS
Installing Oracle Real Application Clusters
Initializing the Veritas Volume Manager
# cmapplyconf -C clm.asc
# cmruncl # cmviewcl
# vxdctl -c mode
# cfscluster config -s
# /etc/vx/bin/vxdisksetup -i c4t4d0
# vxdg -s init cfsdg1 c4t4d0
# newfs -F vxfs /dev/vx/rdsk/cfsdg1/vol3
# newfs -F vxfs /dev/vx/rdsk/cfsdg1/vol2
# cfsmntadm add cfsdg1 vol1 /cfs/mnt1 all=rw
#cfsmntadm add cfsdg1 vol2 /cfs/mnt2 all=rw
# bdf grep cfs
Deleting CFS from the Cluster
# cfsmntadm delete /cfs/mnt1
# cfsmntadm delete /cfs/mnt2
# vxinstall
# cfscluster unconfig
Creating a Storage Infrastructure with CVM
# cfsdgadm delete cfsdg1
Using CVM 5.x or later
# vxdg -s init opsdg c4t4d0
Preparing the Cluster for Use with CVM
Using CVM
# vxedit set diskdetpolicy=global DiskGroupName
# cmapplyconf -P /etc/cmcluster/cvm/VxVM-CVM-pkg.conf
# cmruncl
# /usr/lib/vxvm/bin/vxdisksetup -i /dev/dsk/c0t3d2
# vxdg -s init opsdg c0t3d2
# vxdg list
Oracle Demo Database Files
Creating Volumes
Mirror Detachment Policies with CVM
# vxassist -g opsdg make logfiles 1024m
Adding Disk Groups to the Cluster Configuration
# useradd -u 203 -g oinstall -G dba,oper oracle
# groupadd oinstall # groupadd dba # groupadd oper
# passwd oracle
# mkdir -p /mnt/app/crs/oracle/product/10.2.0/crs
# chmod 755 /dev/vgrac
# usermod -d /cfs/mnt1/oracle oracle
Mnt/app/oracle/oradata/ver10/ver10raw.conf
Installing Oracle 10g/11gR1/11gR2 RAC Binaries
Installing Oracle 10g, 11gR1, or 11gR2 Cluster Software
Installing on Local File System
Installing RAC Binaries on a Local File System
Creating a RAC Demo Database
Installing RAC Binaries on Cluster File System
Creating a RAC Demo Database on Slvm or CVM
$ netca
Verifying Oracle Disk Manager is Configured
Creating a RAC Demo Database on CFS
#swlist VRTSodm
Verify that Oracle Disk Manager is Running
Configuring Oracle to Use Oracle Disk Manager Library
#ll -L /opt/VRTSodm/lib/libodm.sl
# cat /dev/odm/stats
$ cd $ORACLEHOME/lib
# kcmodule -P state odm
Configure Serviceguard Packages
Preparing Oracle Cluster Software for Serviceguard Packages
$ORACRSHOME/bin/crsctl disable crs
$ORACLEHOME/bin/srvctl modify database -d dbname -y manual
Sbin/init.d/init.crs start
Introduction
Why ASM over SLVM?
SG/SGeRAC Support for ASM on HP-UX 11i
Overview
Configuring Slvm Volume Groups for ASM Disk Groups
Sample Command Sequence for Configuring Slvm Volume Groups
Support of Oracle RAC ASM with SGeRAC
ASM support with SG/SGeRAC A.11.17.01 or later
ASM over Slvm
Configuring Slvm Volume Groups for ASM Disk Groups
Sample Command Sequence for Configuring Slvm Volume Groups
ASM over Raw disk
Additional Hints on ASM Integration with SGeRAC
Installation, Configuration, Support, and Troubleshooting
ASM may require Modified Backup/Restore Procedures
Additional Documentation on the Web and Scripts
Background
SGeRAC Toolkit for Oracle RAC 10g or later
Background
SGeRAC Toolkit for Oracle RAC 10g or later
Page
Serviceguard Extension for RAC Toolkit operation
Startup and shutdown of the combined Oracle RAC-SGeRAC stack
SGeRAC Toolkit for Oracle RAC 10g or later
Asmdg MNP
Use case 3 Database storage in ASM over Slvm
Use Case 2 Setup
Use Case 2 performing maintenance with ASM disk groups
Use Case 1 Performing maintenance with Oracle Clusterware
SGeRAC Toolkit for Oracle RAC 10g or later
Internal structure of SGeRAC for Oracle Clusterware
Support for the SGeRAC Toolkit
Contents
Support for the SGeRAC Toolkit
CFS-DG1-MNP CFS-DG2-MNP
Readme
SGeRAC Toolkit for Oracle RAC 10g or later
DEPENDENCYCONDITIONDG-MNP-PKG=UP Dependencylocationsamenode
ORACLEHOME, CHECKINTERVAL...MAINTENANCEFLAG
Dependencyname
DEPENDENCYCONDITIONOC-MNP-PKG=UP Dependencylocationsamenode
Set to any name desired for the RAC MNP
For a package using CFS
Oracrshome
Asmdisk Group
Asmtkitdir
SGeRAC Toolkit for Oracle RAC 10g or later
Asmdg MNP
SGeRAC Toolkit for Oracle RAC 10g or later
Support for the SGeRAC Toolkit
SGeRAC Toolkit for Oracle RAC 10g or later
DEPENDENCYCONDITIONOC-DGMP-PKG=UP Dependencylocationsamenode
Additional Documentation on the Web
Conclusion
Types of Cluster and Package States
Examples of Cluster and Package States
# cmviewcl -r A.11.16
Name Status State
Types of Cluster and Package States
Cluster Status
Node Status and State
Package Status and State
Package Switching Attributes
Service Status
Status of Group Membership
Network Status
Examples of Cluster and Package States
Failover and Failback Policies
Normal Running Status
CVM Package Status
Quorum Server Status
Status After Moving the Package to Another Node
Output of the cmviewcl -vcommand is as follows
Status After Package Switching is Enabled
Status After Halting a Node
Viewing Data on Unowned Packages
Status Nodename Name
Checking the Cluster Configuration and Components
Verifying Cluster Components
Checking Cluster Components
Cmcheckconf -P
Example
Setting up Periodic Cluster Verification
Online Reconfiguration
Limitations
Online Node Addition and Deletion
Managing the Shared Storage
Making LVM Volume Groups Shareable
Making Offline Changes to Shared Volume Groups
Activating an LVM Volume Group in Shared Mode
Making a Volume Group Unshareable
Deactivating a Shared Volume Group
On node 2, issue the following command
Removing Serviceguard Extension for RAC from a System
Adding Additional Shared LVM Volume Groups
Changing the CVM Storage Configuration
Using EMS Hardware Monitors
Using Event Monitoring Service
Monitoring Hardware
Adding Disk Hardware
# vgcfgrestore /dev/vgsg01 /dev/dsk/c2t3d0
Replacing a Mechanism in a Disk Array Configured with LVM
Replacing Disks
# lvreduce -m 0 /dev/vgsg01/lvolname /dev/dsk/c2t3d0
# lvextend -m 1 /dev/vgsg01 /dev/dsk/c2t3d0
# vgcfgrestore -n vg name pv raw path
# lvsync /dev/vgsg01/lvolname
# pvchange -a n pv path
Replacing a Lock Disk
Online Hardware Maintenance with Inline Scsi Terminator
W Scsi Buses with Inline Terminators
Replacement of LAN Cards
Replacement of I/O Cards
Offline Replacement
Online Replacement
Monitoring RAC Instances
Troubleshooting
Rolling Software Upgrades
Software Upgrades
Upgrading Serviceguard to SGeRAC cluster
Cmhaltnode nodename
Autostartcmcld =
Steps for Rolling Upgrades
Keeping Kernels Consistent
Example of Rolling Upgrade
Step
Running Cluster Before Rolling Upgrade
Node 1 Upgraded to SG/SGeRAC
# cmhaltnode -f node2
Running Cluster After Upgrades
Limitations of Rolling Upgrades
Non-Rolling Software Upgrades
Limitations of Non-Rolling Upgrades
Migrating an SGeRAC Cluster with Cold Install
Rolling Upgrade Using DRD
Upgrade Using DRD
Non-Rolling Upgrade Using DRD
Restrictions for DRD Upgrades
Software Upgrades
Oracle Logical Volume Worksheet
LVM Volume Group and Physical Volume Worksheet
Blank Planning Worksheets
Index
RAC
155