HP Serviceguard Metrocluster manual

PRIMARY CLUSTER		STATUS	EVENT LEVEL			POLLING INTERVAL
PTST_sanfran		Unmonitored		unmonitored		1 min
CONFIGURED EVENT		STATUS	DURATION		LAST NOTIFICATION SENT
alert	unreachable		1 min		--
alert		unreachable	2	min		--
alarm		unreachable	3	min		--
alert		down	1	min		--
alert		down	2	min		--
alarm		down	3	min		--
alert		error	0	sec		--
alert		up	1	min		--
RECOVERY CLUSTER		PTST_sanfran
PRIMARY CLUSTER		STATUS		EVENT LEVEL		POLLING INTERVAL
PTST_dts1		Unmonitored		unmonitored		1 min
CONFIGURED EVENT		STATUS	DURATION		LAST NOTIFICATION SENT

alert	unreachable	1	min	--
alert	unreachable	2	min	--
alarm	unreachable	3	min	--
alert	down	1	min	--
alert	down	2	min	--
alarm	down	3	min	--
alert	error	0	sec	--
alert	up	1	min	--

PACKAGE RECOVERY GROUP hpgroup10
PACKAGE	ROLE	STATUS
PTST_sanfran/PACKAGE1	primary	down
TST_dts1/PACKAGE1	recovery	down

PACKAGE RECOVERY GROUP hpgroup20
PACKAGE	ROLE	STATUS
PTST_dts1/PACKAGE1x_ld	primary	down
PTST_sanfran/PACKAGE1x_ld	recovery	down

For a more comprehensive status of component clusters, nodes, and packages, use the cmviewcl command on both the clusters. On each cluster, note the nodes on which the primary packages are running on, as well as data sender and data receiver packages, if they are being used for logical data replication. Verify that the monitor is running on every cluster on which it is configured.

The following is an example of output of the cmviewcl command for a cluster (nycluster) that is running a monitor package. Note that the recovery package salespkg_bak is not running, and is shown as an unowned package. This is the expected display while the other cluster is running salespkg.

CLUSTER	STATUS
nycluster	up
NODE	STATUS	STATE
nynode1	up	running

Network Parameters:
INTERFACE		STATUS	PATH	NAME
PRIMARY		up		12.1	lan0
PRIMARY		up		56.1	lan2
NODE	STATUS		STATE:
nynode2	up		running

Network Parameters:

Checking the status of clusters, nodes, and packages 55

Image 55

HP Serviceguard Metrocluster manual Configured Event Status Duration Last Notification Sent

Contents

HP Part Number Published February Legal Notices Contents Administering Continentalclusters Disaster recovery rehearsal in Continentalclusters Troubleshooting Continentalclusters 114 Sample Continentalclusters Ascii configuration file 115Package attributes Legacy packages 100 Glossary 145 Index 150 Introduction Sample Continentalclusters Configuration Traditional deployment method Building the Continentalclusters configurationCreating the Serviceguard clusters at both the sites Easy deployment method # passwd conclusr Setting up security# cmapplyconf -v -C /etc/cmcluster/cluster.config Csshsetup -r -k rsa -f host-list-file Creating data replication between the clusters Userhost Anyserviceguardnode Userrole MonitorUsername conclusr Using software based logical replication Using any other array based physical replication technology # vgexport -s -p -m mapfilename vgname Creating and Exporting LVM Volume Groups# mkdir /dev/vgxx # mknod /dev/vgxx/group c 64 0xnn0000 # vgchange -a n vgname Creating VxVM Disk Groups Devicegroup # cmmakepkg -m dts/ccxpca temp.configDts/xpca/dtspkgdir Dts/xpca/dtspkgdir pkgdirname Fence # cmcheckconf -P temp.config# cmapplyconf -P temp.config # cmmakepkg -m dts/cccaeva temp.config DC1SMISLIST DtapplicationstartuppolicyDrgroupname DC1STORAGEWORLDWIDENAME # cmhaltpkg packagename # cmmakepkg -m dts/ccsrdf temp.configDts/dts/dtspkgdir Rdfmode # cmrunpkg -n nodename packagename # cmmakepkg -m dts/cc3parrc pkgName.config# cmmodpkg -e packagename # cmcheckconf -P pkgName.config Configuring the monitor package# cmapplyconf -P pkgName.config # cmcheckconf -P ccmonpkg.conf Creating a Continentalclusters configuration# cd /etc/cmcluster # cmqueryconcl -C cmconcl.config Cluster information Recovery groups, the following parameters are available Recovery groups Monitoring definitions has the following parameters Monitoring definitions # cmapplyconcl -v -C cmconcl.config Checking and applying the Continentalclusters configurationVerify the Continentalclusters configuration # cmcheckconcl -v -C cmconcl.config Starting the Continentalclusters monitor package Testing the ContinentalclustersTesting Individual Packages Testing Continentalclusters Operations # cmviewconcl Testing the Continentalclusters Building the Continentalclusters configuration Receiving notification Performing recovery in case of disasterVerifying that recovery is required Preparing the storage manually in the recovery cluster # cmrecovercl Recovering the entire cluster after a cluster alarm# vxdg deport disk group name Viewing the Continentalclusters status Recovering the entire cluster after a cluster alertRecovering a single cluster in an N-1 configuration # cmrecovercl -f Switching the Primary and Recovery Cluster Roles Restoring disaster recovery cluster after a disaster Run this command Check and apply the Continentalclusters configurationRestart the monitor packages on every cluster Creating a new Primary Cluster # cmrunpkg ccmonpkg Creating a new Recovery Cluster# cmapplyconf -p ccmonpkg.config Restart the monitor package on the recovery cluster Configuring maintenance mode in Continentalclusters Disaster recovery rehearsal in ContinentalclustersOverview of Disaster Recovery rehearsal Configuring Continentalclusters Disaster Recovery rehearsal # mkdir directorypath # cmgetconf -p ccmonpkg ccnew.config# vgcreate /dev/vgcc -f /dev/sda1 # lvcreate -L size vgname Mke2fs -j Lvol # cmcheckconf -P packagename Configuring Continentalclusters rehearsal packagesModifying Continentalclusters configuration # cmhaltpkg ccmonpkg # cmapplyconf -P ccnew.config Client access IP address at recovery cluster Precautions to be taken while performing DR RehearsalCluster role switch during rehearsal Start rehearsal Move the recovery group into maintenance modeVerify data replication environment Prepare the replication environment for DR rehearsal Move the recovery group out of maintenance mode Cleanup of secondary mirror copyRecovering the primary cluster disaster during DR Rehearsal Restore replication environment for recovery Run cmrrcovercl command Limitations of DR rehearsal feature Sadta Configuration in Continentalclusters Configuring the primary cluster with a single site Setting up replication # vxdg -g cvmdgname set activation=sw Configuring the recovery cluster with a single siteSetting up the complex workload in the primary cluster # vxdg -s init cvmdgname replicateddisk1 replicateddisk2 # cmapplyconf -P cfspkg1.ascii # newfs -F vxfs /dev/vx/rdsk/cvmdgname/cvmdgvolname# cmmakepkg -m sg/cfsall /etc/cmcluster/cfspkg1.ascii # cmcheckconf -P cfspkg1.ascii # vgimport -s -m mapfilename vgname # cmapplyconf -v -P cvmdgpkgname.conf# mkdir /dev/vgname # mknod /dev/vgname/group c 64 0xnn0000 # lvcreate -L Xxxx /dev/vgname Configuring the complex workload at the primary cluster # cmmakepkg -m dts/sc -m dts/cc3parrc cwsc.config # cmmakepkg -m dts/sc -m dts/ccxpca cwsc.config# cmmakepkg -m dts/sc -m dts/cccaeva cwsc.config # cmmakepkg -m dts/sc -m dts/ccsrdf cwsc.config # cmapplyconf -v -P /etc/cmcluster/cwsc/cwsc.config Suspending the replication to the recovery cluster# cmviewcl -v -p pkgname # vxdg -stfC import cvmdgname Where node3 and node4 are the nodes at the recovery cluster# cmrunpkg siteBcfspkgname # vgchange -c n vgname # vgchange -c y vgname Configuring Site Safety Latch dependencies# vgchange -c n -S n vgname # vgchange -c y -S y vgname Configuring Continentalclusters Resuming the replication to the recovery cluster Checking the status of clusters, nodes, and packages Administering Continentalclusters Configured Event Status Duration Last Notification Sent Package Status State Pkgswitch Node Network Attributes Enabling and disabling maintenance modeStartup and Switching Characteristics # cmrecovercl -d recoverygroup1 # cmrecovercl -g RecoveryGroupName Recovering a cluster when the storage array or disks failStarting a recovery package forcefully # cmrecovercl -e recoverygroup1 # cmforceconcl cmrunpkg -n node3 Pkg1 Adding a Recovery Group to Continentalclusters# cmapplyconf -C cluster.config Adding or Removing a Node from a Cluster Modifying Continentalclusters configuration Modifying a package in a recovery groupRemoving a recovery group from the Continentalclusters Changing monitoring definitions Removing a rehearsal package from a recovery groupModifying a recovery group with a new rehearsal package Behavior of Serviceguard commands in Continentalclusters # ps -ef grep cmclapplyd Verifying the status of Continentalclusters daemonsRenaming the Continentalclusters # ps -ef grep cmclsentryd Maintaining the data replication environment Deleting the Continentalclusters configuration# what /usr/sbin/cmviewconcl Using the pairresync command Continuous Access EVA Link Suspend and Resume Modes Maintaining 3PAR Remote Copy data replication environment Maintaining EMC Srdf data replication environment Maintaining a Node Administering Continentalclusters using Sadta configurationMaintaining the Site Maintaining Site Controller Package Start the Site Controller package by enabling it Deleting the Site Controller PackageStarting a Complex Workload # cfsdgadm deleteems pkg1dg /dts/mcsc/cwsc # cmviewcl -l package Moving a Complex Workload to the Recovery ClusterShutting Down a Complex Workload Restarting a Failed Site Controller Package Reviewing Messages and Log Files of Monitoring Daemon Troubleshooting ContinentalclustersTroubleshooting Continentalclusters Error Messages Reviewing Messages and Log Files Troubleshooting Continentalclusters Error Messages PackageName on cluster ClusterName Migrating to Continentalclusters A.08.00 Verify the configuration and the status of the cluster Continentalclusters Worksheets Recovery Group WorksheetData Center Worksheet Cluster Event Worksheet Recovery Checklist Replication configuration Continentalclusters Site configuration Configuring a CRS sub-cluster using CFS CRS Sub-cluster configuration using CFS RAC database configuration RAC database configuration RAC MNP Site Controller package configuration Site Controller package configuration Configuration file parameters for Continentalclusters Datasenderpackage clustername/packagename Primarypackage Clustername/Packagename Continentalclusters Command and Daemon Reference Cmdeleteconcl -f Cmrunpkg or cmmodpkg commandCmcluster/cmoncl.config Cmrecovercl -f Recovery group name Cmrecovercl -r-gCmviewconcl Autofencedatasplit Package attributes Autopsuepsus Autopsusssws Autosvolpsue Autosvolpfus Clustertype Autosvolpsus Continuous Access link timeout value HorctimeoutHorctakeover command, -t timeout. Pkg Startup Timeout Horctimeout Monnotificationfrequency DTS PKG DIRWaittime Monpollinterval Monnotificationconsole MonnotificationsyslogAutoresync Fails to start with an error Before startup of the package on destination volumeStart with an error If Waittime is 0 default value, and if the state of DR Package Attributes for Continentalcluster with EMC Srdf Querytimeout DefaultEtc/cmcluster/ packagename/FORCEFLAG Consistencygroups is set to Supported mode are sync for synchronous and async for ConfigurationsGroup in an M by N configuration Where the Metrocluster Srdf environment file is located Migrating legacy to modular packages Legacy packagesMigrating legacy monitor package # cmmigratepkg -p packagename -s -o modularsgconf # cmmigratepkg -p packagename -o modularccmonpkg.conf# cmcheckconf -P modularccmonpkg.conf # cmapplyconf -P modularccmonpkg.conf # cmapplyconf -P modularsgecmcc.conf Validate the new modular package configuration filePathtoenvfile modularsgecmcc.conf # cmcheckconf -P modularsgecmcc.conf # cmmigratepkg -p packagename -s -o modularsg.conf # mkdir /etc/cmcluster/ccmonpkg Configuring legacy packagesConfiguring the monitor package in legacy style Validate the new package configuration file # cd /etc/cmcluster/pkgname # cmmakepkg -p pkgname.ascii Use the cmcheckconf command to validate the package# cmcheckconf -P ccmonpkg.config # mkdir /etc/cmcluster/pkgname # cmmakepkg -s pkgname.cntl Create a package control script EVA Legacy packages # mkdir /etc/cmcluster/pkgname Set Pkgdir to /etc/cmcluster/pkgname Example 1 Example # cfsdgadm add cvmdgnamecvmdgpkgname all=sw\ node1node2 Configuring storage devices for complex workloadEtc/cmcluster/pkgname # vxdg -s init cvmdgname replicateddisk1\ replicateddisk2 # mkdir /cfs # chmod 775 /cfs # mkdir /cfs/cvmdgname Where node1 and node2 are the nodes in the Source Disk Site# cfsdgadm activate cvmdgname # newfs -F vxfs \ Dev/vx/rdsk/cvmdgname/cvmdgvolname # cfsmount /cfs/cvmdgname EVA Continentalclusters Ascii configuration file Sample Continentalclusters Ascii configuration fileSample Continentalclusters Ascii configuration file Clustername #### #### Recoverygroupname Datasenderpackage Notification Console #### #### Clusterevent # Cluster Information Clusteralarm Seconds Notification Recoverygroupname RG2 Sample input and output files for cmswitchconcl command Recoverygroupname RG2 Primarypackage Oracle RAC instances in a Continentalclusters environment Supported Continentalclusters and RAC configuration # cfsdgadm activate diskgroup # vxdctl -c mode# cfsdgadm add disk group name all=sw # cfsmntadm add racdgl vol4 /cfs/mntl all=rwPage # cd your own directory # cp /opt/cmconcl/scripts/ccrac.config ccrac.config.mycopyCopy the file Copy the edited file to the final directory # cp ccrac.config.mycopy /etc/cmconcl/ccrac/ccrac.configPage Failover of Oracle RAC instances to the recovery site Oracle RAC instance package can be started in sequence Failback of Oracle RAC instances after a failover # /opt/cmconcl/bin/ccracmgmt.ksh -i indexNumber \ start Rehearsing Oracle RAC databases in Continentalclusters# cfsumount Mount Point Name # cfsdgadm deactivate Disk Group Name Sample Oracle RAC database with ASM in Sadta Configure a primary cluster with a single site Configuring the ASM disk group in the primary cluster Configure a recovery cluster with a single siteInstalling and configuring Oracle Clusterware Installing Oracle Real Application Clusters RAC software Halting the RAC database in the primary cluster Creating the Oracle RAC database in the primary cluster # chown -R oracleoinstall /opt/app/oracle/admin Page # cmviewcl -v -p ASMDGpkgname Run the following command at the remote site# chown -R oracleoinstall /opt/app/oracle/admin/hrdb # srvctl status Site cc2site1 # cmapplyconf -v -P sitecontrollerconfigurationfile # cmrunpkg siteController1 See also quorum server Glossary Span 100s or 1000s of kilometers Or system availability Format /dev/devtype/filename Svol Index Symbols 151