Clusteralarm Seconds Notification | HP Serviceguard Metrocluster

CLUSTER_EVENT	cluster2/DOWN
MONITORING_CLUSTER			cluster3
CLUSTER_ALERT			30 SECONDS
NOTIFICATION	TEXTLOG /var/opt/resmon/log/CCTextlog
“DRT: (Ora-test)		DOWN alert”
NOTIFICATION		SYSLOG
“DRT: (Ora-test)		cluster2 DOWN alert”
CLUSTER_ALARM			30 SECONDS
NOTIFICATION	TEXTLOG /var/opt/resmon/log/CCTextlog
“DRT: (Ora-test) DOWN alarm”
NOTIFICATION		SYSLOG
“DRT: (Ora-test) cluster2			DOWN alarm”
CLUSTER_EVENT	cluster3/DOWN
MONITORING_CLUSTER			cluster1
CLUSTER_ALERT			30 SECONDS
NOTIFICATION		TEXTLOG /var/opt/resmon/log/logging
“DRT: (Ora-test)		DOWN alert”
NOTIFICATION		SYSLOG
“DRT: (Ora-test)		cluster3 DOWN alert”
CLUSTER_ALARM			30 SECONDS
NOTIFICATION	TEXTLOG /var/opt/resmon/log/CCTextlog
“DRT: (Ora-test) DOWN alarm”
NOTIFICATION		SYSLOG
“DRT: (Ora-test) cluster3			DOWN alarm”
CLUSTER_EVENT	cluster1/UP
MONITORING_CLUSTER			cluster3
CLUSTER_ALERT			30 SECONDS
NOTIFICATION	TEXTLOG /var/opt/resmon/log/CCTextlog
“DRT: (Ora-test)		UP alert”
NOTIFICATION		SYSLOG
“DRT: (Ora-test)		cluster1 UP alert”
CLUSTER_EVENT	cluster2/UP
MONITORING_CLUSTER			cluster3
CLUSTER_ALERT			30 SECONDS
NOTIFICATION	TEXTLOG /var/opt/resmon/log/CCTextlog
“DRT: (Ora-test)		UP alert”
NOTIFICATION		SYSLOG
“DRT: (Ora-test)		cluster2 UP alert”
CLUSTER_EVENT	cluster3/UP
MONITORING_CLUSTER			cluster1
CLUSTER_ALERT			30 SECONDS
NOTIFICATION	TEXTLOG /var/opt/resmon/log/CCTextlog
“DRT: (Ora-test)		UP alert”
NOTIFICATION		SYSLOG
“DRT: (Ora-test)		cluster3 UP alert”

122 Sample Continentalclusters ASCII configuration file

Image 122

HP Serviceguard Metrocluster manual Clusteralarm Seconds Notification

Contents

HP Part Number Published February Legal Notices Contents Disaster recovery rehearsal in Continentalclusters Administering Continentalclusters Troubleshooting Continentalclusters Legacy packages 100 Sample Continentalclusters Ascii configuration file 115Package attributes 114 Glossary 145 Index 150 Introduction Sample Continentalclusters Configuration Easy deployment method Building the Continentalclusters configurationCreating the Serviceguard clusters at both the sites Traditional deployment method Csshsetup -r -k rsa -f host-list-file Setting up security# cmapplyconf -v -C /etc/cmcluster/cluster.config # passwd conclusr Username conclusr Userhost Anyserviceguardnode Userrole MonitorCreating data replication between the clusters Using any other array based physical replication technology Using software based logical replication # vgchange -a n vgname Creating and Exporting LVM Volume Groups# mkdir /dev/vgxx # mknod /dev/vgxx/group c 64 0xnn0000 # vgexport -s -p -m mapfilename vgname Creating VxVM Disk Groups Dts/xpca/dtspkgdir pkgdirname # cmmakepkg -m dts/ccxpca temp.configDts/xpca/dtspkgdir Devicegroup # cmmakepkg -m dts/cccaeva temp.config # cmcheckconf -P temp.config# cmapplyconf -P temp.config Fence DC1STORAGEWORLDWIDENAME DtapplicationstartuppolicyDrgroupname DC1SMISLIST Rdfmode # cmmakepkg -m dts/ccsrdf temp.configDts/dts/dtspkgdir # cmhaltpkg packagename # cmmodpkg -e packagename # cmmakepkg -m dts/cc3parrc pkgName.config# cmrunpkg -n nodename packagename # cmapplyconf -P pkgName.config Configuring the monitor package# cmcheckconf -P pkgName.config Cluster information Creating a Continentalclusters configuration# cd /etc/cmcluster # cmqueryconcl -C cmconcl.config # cmcheckconf -P ccmonpkg.conf Recovery groups Recovery groups, the following parameters are available Monitoring definitions Monitoring definitions has the following parameters # cmcheckconcl -v -C cmconcl.config Checking and applying the Continentalclusters configurationVerify the Continentalclusters configuration # cmapplyconcl -v -C cmconcl.config Testing Continentalclusters Operations Testing the ContinentalclustersTesting Individual Packages Starting the Continentalclusters monitor package Testing the Continentalclusters # cmviewconcl Building the Continentalclusters configuration Preparing the storage manually in the recovery cluster Performing recovery in case of disasterVerifying that recovery is required Receiving notification # vxdg deport disk group name Recovering the entire cluster after a cluster alarm# cmrecovercl # cmrecovercl -f Recovering the entire cluster after a cluster alertRecovering a single cluster in an N-1 configuration Viewing the Continentalclusters status Restoring disaster recovery cluster after a disaster Switching the Primary and Recovery Cluster Roles Restart the monitor packages on every cluster Check and apply the Continentalclusters configurationRun this command Creating a new Primary Cluster Restart the monitor package on the recovery cluster Creating a new Recovery Cluster# cmapplyconf -p ccmonpkg.config # cmrunpkg ccmonpkg Configuring Continentalclusters Disaster Recovery rehearsal Disaster recovery rehearsal in ContinentalclustersOverview of Disaster Recovery rehearsal Configuring maintenance mode in Continentalclusters # lvcreate -L size vgname Mke2fs -j Lvol # cmgetconf -p ccmonpkg ccnew.config# vgcreate /dev/vgcc -f /dev/sda1 # mkdir directorypath # cmhaltpkg ccmonpkg # cmapplyconf -P ccnew.config Configuring Continentalclusters rehearsal packagesModifying Continentalclusters configuration # cmcheckconf -P packagename Cluster role switch during rehearsal Precautions to be taken while performing DR RehearsalClient access IP address at recovery cluster Prepare the replication environment for DR rehearsal Move the recovery group into maintenance modeVerify data replication environment Start rehearsal Restore replication environment for recovery Cleanup of secondary mirror copyRecovering the primary cluster disaster during DR Rehearsal Move the recovery group out of maintenance mode Limitations of DR rehearsal feature Run cmrrcovercl command Sadta Configuration in Continentalclusters Setting up replication Configuring the primary cluster with a single site # vxdg -s init cvmdgname replicateddisk1 replicateddisk2 Configuring the recovery cluster with a single siteSetting up the complex workload in the primary cluster # vxdg -g cvmdgname set activation=sw # cmcheckconf -P cfspkg1.ascii # newfs -F vxfs /dev/vx/rdsk/cvmdgname/cvmdgvolname# cmmakepkg -m sg/cfsall /etc/cmcluster/cfspkg1.ascii # cmapplyconf -P cfspkg1.ascii # lvcreate -L Xxxx /dev/vgname # cmapplyconf -v -P cvmdgpkgname.conf# mkdir /dev/vgname # mknod /dev/vgname/group c 64 0xnn0000 # vgimport -s -m mapfilename vgname Configuring the complex workload at the primary cluster # cmmakepkg -m dts/sc -m dts/ccsrdf cwsc.config # cmmakepkg -m dts/sc -m dts/ccxpca cwsc.config# cmmakepkg -m dts/sc -m dts/cccaeva cwsc.config # cmmakepkg -m dts/sc -m dts/cc3parrc cwsc.config # cmviewcl -v -p pkgname Suspending the replication to the recovery cluster# cmapplyconf -v -P /etc/cmcluster/cwsc/cwsc.config # cmrunpkg siteBcfspkgname Where node3 and node4 are the nodes at the recovery cluster# vxdg -stfC import cvmdgname # vgchange -c n -S n vgname # vgchange -c y -S y vgname Configuring Site Safety Latch dependencies# vgchange -c n vgname # vgchange -c y vgname Resuming the replication to the recovery cluster Configuring Continentalclusters Administering Continentalclusters Checking the status of clusters, nodes, and packages Configured Event Status Duration Last Notification Sent Package Status State Pkgswitch Node # cmrecovercl -d recoverygroup1 Enabling and disabling maintenance modeStartup and Switching Characteristics Network Attributes # cmrecovercl -e recoverygroup1 Recovering a cluster when the storage array or disks failStarting a recovery package forcefully # cmrecovercl -g RecoveryGroupName Adding or Removing a Node from a Cluster Adding a Recovery Group to Continentalclusters# cmapplyconf -C cluster.config # cmforceconcl cmrunpkg -n node3 Pkg1 Removing a recovery group from the Continentalclusters Modifying a package in a recovery groupModifying Continentalclusters configuration Behavior of Serviceguard commands in Continentalclusters Removing a rehearsal package from a recovery groupModifying a recovery group with a new rehearsal package Changing monitoring definitions # ps -ef grep cmclsentryd Verifying the status of Continentalclusters daemonsRenaming the Continentalclusters # ps -ef grep cmclapplyd # what /usr/sbin/cmviewconcl Deleting the Continentalclusters configurationMaintaining the data replication environment Using the pairresync command Continuous Access EVA Link Suspend and Resume Modes Maintaining EMC Srdf data replication environment Maintaining 3PAR Remote Copy data replication environment Maintaining the Site Administering Continentalclusters using Sadta configurationMaintaining a Node Maintaining Site Controller Package # cfsdgadm deleteems pkg1dg /dts/mcsc/cwsc Deleting the Site Controller PackageStarting a Complex Workload Start the Site Controller package by enabling it Restarting a Failed Site Controller Package Moving a Complex Workload to the Recovery ClusterShutting Down a Complex Workload # cmviewcl -l package Reviewing Messages and Log Files Troubleshooting ContinentalclustersTroubleshooting Continentalclusters Error Messages Reviewing Messages and Log Files of Monitoring Daemon Troubleshooting Continentalclusters Error Messages PackageName on cluster ClusterName Verify the configuration and the status of the cluster Migrating to Continentalclusters A.08.00 Data Center Worksheet Recovery Group WorksheetContinentalclusters Worksheets Recovery Checklist Cluster Event Worksheet Continentalclusters Site configuration Replication configuration CRS Sub-cluster configuration using CFS Configuring a CRS sub-cluster using CFS RAC database configuration RAC database configuration Site Controller package configuration RAC MNP Site Controller package configuration Configuration file parameters for Continentalclusters Datasenderpackage clustername/packagename Primarypackage Clustername/Packagename Continentalclusters Command and Daemon Reference Cmrecovercl -f Cmrunpkg or cmmodpkg commandCmcluster/cmoncl.config Cmdeleteconcl -f Cmviewconcl Cmrecovercl -r-gRecovery group name Package attributes Autofencedatasplit Autopsuepsus Autopsusssws Autosvolpfus Autosvolpsue Autosvolpsus Clustertype Pkg Startup Timeout Horctimeout HorctimeoutHorctakeover command, -t timeout. Continuous Access link timeout value Monpollinterval DTS PKG DIRWaittime Monnotificationfrequency Autoresync MonnotificationsyslogMonnotificationconsole If Waittime is 0 default value, and if the state of DR Before startup of the package on destination volumeStart with an error Fails to start with an error Etc/cmcluster/ packagename/FORCEFLAG Querytimeout DefaultPackage Attributes for Continentalcluster with EMC Srdf Consistencygroups is set to Where the Metrocluster Srdf environment file is located ConfigurationsGroup in an M by N configuration Supported mode are sync for synchronous and async for Migrating legacy monitor package Legacy packagesMigrating legacy to modular packages # cmapplyconf -P modularccmonpkg.conf # cmmigratepkg -p packagename -o modularccmonpkg.conf# cmcheckconf -P modularccmonpkg.conf # cmmigratepkg -p packagename -s -o modularsgconf # cmcheckconf -P modularsgecmcc.conf Validate the new modular package configuration filePathtoenvfile modularsgecmcc.conf # cmapplyconf -P modularsgecmcc.conf # cmmigratepkg -p packagename -s -o modularsg.conf Validate the new package configuration file Configuring legacy packagesConfiguring the monitor package in legacy style # mkdir /etc/cmcluster/ccmonpkg # mkdir /etc/cmcluster/pkgname Use the cmcheckconf command to validate the package# cmcheckconf -P ccmonpkg.config # cd /etc/cmcluster/pkgname # cmmakepkg -p pkgname.ascii Create a package control script # cmmakepkg -s pkgname.cntl EVA Legacy packages # mkdir /etc/cmcluster/pkgname Example 1 Example Set Pkgdir to /etc/cmcluster/pkgname # vxdg -s init cvmdgname replicateddisk1\ replicateddisk2 Configuring storage devices for complex workloadEtc/cmcluster/pkgname # cfsdgadm add cvmdgnamecvmdgpkgname all=sw\ node1node2 # newfs -F vxfs \ Dev/vx/rdsk/cvmdgname/cvmdgvolname Where node1 and node2 are the nodes in the Source Disk Site# cfsdgadm activate cvmdgname # mkdir /cfs # chmod 775 /cfs # mkdir /cfs/cvmdgname # cfsmount /cfs/cvmdgname EVA Sample Continentalclusters Ascii configuration file Sample Continentalclusters Ascii configuration fileContinentalclusters Ascii configuration file Clustername #### #### Recoverygroupname Datasenderpackage Notification Console #### #### Clusterevent # Cluster Information Clusteralarm Seconds Notification Sample input and output files for cmswitchconcl command Recoverygroupname RG2 Recoverygroupname RG2 Primarypackage Oracle RAC instances in a Continentalclusters environment Supported Continentalclusters and RAC configuration # cfsmntadm add racdgl vol4 /cfs/mntl all=rw # vxdctl -c mode# cfsdgadm add disk group name all=sw # cfsdgadm activate diskgroupPage Copy the file # cp /opt/cmconcl/scripts/ccrac.config ccrac.config.mycopy# cd your own directory # cp ccrac.config.mycopy /etc/cmconcl/ccrac/ccrac.config Copy the edited file to the final directoryPage Failover of Oracle RAC instances to the recovery site Oracle RAC instance package can be started in sequence Failback of Oracle RAC instances after a failover # cfsdgadm deactivate Disk Group Name Rehearsing Oracle RAC databases in Continentalclusters# cfsumount Mount Point Name # /opt/cmconcl/bin/ccracmgmt.ksh -i indexNumber \ start Sample Oracle RAC database with ASM in Sadta Configure a primary cluster with a single site Installing Oracle Real Application Clusters RAC software Configure a recovery cluster with a single siteInstalling and configuring Oracle Clusterware Configuring the ASM disk group in the primary cluster Creating the Oracle RAC database in the primary cluster Halting the RAC database in the primary cluster # chown -R oracleoinstall /opt/app/oracle/admin Page # srvctl status Run the following command at the remote site# chown -R oracleoinstall /opt/app/oracle/admin/hrdb # cmviewcl -v -p ASMDGpkgname # cmapplyconf -v -P sitecontrollerconfigurationfile Site cc2site1 # cmrunpkg siteController1 Glossary See also quorum server Span 100s or 1000s of kilometers Or system availability Format /dev/devtype/filename Svol Symbols Index 151