Administering Continentalclusters | HP Serviceguard Metrocluster

7 Administering Continentalclusters

Checking the status of clusters, nodes, and packages

To verify the status of the Continentalclusters and associated packages, use the cmviewconcl command, which lists the status of the clusters, associated package status, and status of the configured events. This command also displays, if configured, the mode of the recovery group.

The following is an example output of the cmviewconcl command in a situation where there is a single recovery group for which the primary cluster is cjc838 and the recovery cluster is cjc1234.

# cmviewconcl

WARNING: Primary cluster cjc838 is in an alarm state (cmrecovercl is enabled on recovery cluster cjc1234)

Continentalclusters cjccc1 RECOVERY CLUSTER cjc1234

PRIMARY CLUSTER STATUS EVENT LEVEL POLLING INTERVAL

cjc838 down	ALARM	20
PACKAGE RECOVERY GROUP prg1
MAINTENANCE MODE	NO
PACKAGE ROLE	STATUS	cjc838/primary primary down
cjc1234/recovery	recovery	up
cjc1234/rehearsal	rehearsal	down

The following is an example of cmviewconcl output from a primary cluster that is down.

cmviewconcl -v

WARNING: Primary cluster cjc838 is in an alarm state (cmrecovercl is enabled on recovery cluster cjc1234)

Primary cluster cjc838 is not configured to monitor recovery cluster cjc1234

Continentalclusters cjccc1

RECOVERY CLUSTER cjc1234

PRIMARY CLUSTER	STATUS	EVENT		LEVEL	POLLING INTERVAL
cjc838	down		ALARM		20
CONFIGURED EVENT	STATUS		DURATION		LAST NOTIFICATION SENT
alert	unreachable		15	sec	--
alarm	unreachable		30	sec	--
alarm	down		0 sec		Fri May 12 12:13:06 PDT 2000
alert	error		0 sec		--
alert	up		20	sec	--
alert	up		40	sec	--
PACKAGE RECOVERY GROUP prg1
MAINTENANCE MODE	NO
PACKAGE	ROLE
STATUS	cjc838/primary
primary	down
cjc1234/recovery	recovery			up
cjc1234/rehearsal	rehearsal			down

The following is the output of the cmviewconcl command that displays data for a mutual recovery configuration in which each cluster has both the primary and the recovery roles—the primary role for one recovery group and the recovery role for the other recovery group:

Continentalclusters ccluster1

RECOVERY CLUSTER PTST_dts1

54 Administering Continentalclusters

Image 54

HP Serviceguard Metrocluster manual Administering Continentalclusters, Checking the status of clusters, nodes, and packages

Contents

HP Part Number Published February Legal Notices Contents Disaster recovery rehearsal in Continentalclusters Administering Continentalclusters Troubleshooting Continentalclusters Legacy packages 100 Sample Continentalclusters Ascii configuration file 115Package attributes 114 Glossary 145 Index 150 Introduction Sample Continentalclusters Configuration Easy deployment method Building the Continentalclusters configurationCreating the Serviceguard clusters at both the sites Traditional deployment method Csshsetup -r -k rsa -f host-list-file Setting up security# cmapplyconf -v -C /etc/cmcluster/cluster.config # passwd conclusr Userhost Anyserviceguardnode Userrole Monitor Creating data replication between the clustersUsername conclusr Using any other array based physical replication technology Using software based logical replication # vgchange -a n vgname Creating and Exporting LVM Volume Groups# mkdir /dev/vgxx # mknod /dev/vgxx/group c 64 0xnn0000 # vgexport -s -p -m mapfilename vgname Creating VxVM Disk Groups Dts/xpca/dtspkgdir pkgdirname # cmmakepkg -m dts/ccxpca temp.configDts/xpca/dtspkgdir Devicegroup # cmmakepkg -m dts/cccaeva temp.config # cmcheckconf -P temp.config# cmapplyconf -P temp.config Fence DC1STORAGEWORLDWIDENAME DtapplicationstartuppolicyDrgroupname DC1SMISLIST Rdfmode # cmmakepkg -m dts/ccsrdf temp.configDts/dts/dtspkgdir # cmhaltpkg packagename # cmmakepkg -m dts/cc3parrc pkgName.config # cmrunpkg -n nodename packagename# cmmodpkg -e packagename Configuring the monitor package # cmcheckconf -P pkgName.config# cmapplyconf -P pkgName.config Cluster information Creating a Continentalclusters configuration# cd /etc/cmcluster # cmqueryconcl -C cmconcl.config # cmcheckconf -P ccmonpkg.conf Recovery groups Recovery groups, the following parameters are available Monitoring definitions Monitoring definitions has the following parameters # cmcheckconcl -v -C cmconcl.config Checking and applying the Continentalclusters configurationVerify the Continentalclusters configuration # cmapplyconcl -v -C cmconcl.config Testing Continentalclusters Operations Testing the ContinentalclustersTesting Individual Packages Starting the Continentalclusters monitor package Testing the Continentalclusters # cmviewconcl Building the Continentalclusters configuration Preparing the storage manually in the recovery cluster Performing recovery in case of disasterVerifying that recovery is required Receiving notification Recovering the entire cluster after a cluster alarm # cmrecovercl# vxdg deport disk group name # cmrecovercl -f Recovering the entire cluster after a cluster alertRecovering a single cluster in an N-1 configuration Viewing the Continentalclusters status Restoring disaster recovery cluster after a disaster Switching the Primary and Recovery Cluster Roles Check and apply the Continentalclusters configuration Run this commandRestart the monitor packages on every cluster Creating a new Primary Cluster Restart the monitor package on the recovery cluster Creating a new Recovery Cluster# cmapplyconf -p ccmonpkg.config # cmrunpkg ccmonpkg Configuring Continentalclusters Disaster Recovery rehearsal Disaster recovery rehearsal in ContinentalclustersOverview of Disaster Recovery rehearsal Configuring maintenance mode in Continentalclusters # lvcreate -L size vgname Mke2fs -j Lvol # cmgetconf -p ccmonpkg ccnew.config# vgcreate /dev/vgcc -f /dev/sda1 # mkdir directorypath # cmhaltpkg ccmonpkg # cmapplyconf -P ccnew.config Configuring Continentalclusters rehearsal packagesModifying Continentalclusters configuration # cmcheckconf -P packagename Precautions to be taken while performing DR Rehearsal Client access IP address at recovery clusterCluster role switch during rehearsal Prepare the replication environment for DR rehearsal Move the recovery group into maintenance modeVerify data replication environment Start rehearsal Restore replication environment for recovery Cleanup of secondary mirror copyRecovering the primary cluster disaster during DR Rehearsal Move the recovery group out of maintenance mode Limitations of DR rehearsal feature Run cmrrcovercl command Sadta Configuration in Continentalclusters Setting up replication Configuring the primary cluster with a single site # vxdg -s init cvmdgname replicateddisk1 replicateddisk2 Configuring the recovery cluster with a single siteSetting up the complex workload in the primary cluster # vxdg -g cvmdgname set activation=sw # cmcheckconf -P cfspkg1.ascii # newfs -F vxfs /dev/vx/rdsk/cvmdgname/cvmdgvolname# cmmakepkg -m sg/cfsall /etc/cmcluster/cfspkg1.ascii # cmapplyconf -P cfspkg1.ascii # lvcreate -L Xxxx /dev/vgname # cmapplyconf -v -P cvmdgpkgname.conf# mkdir /dev/vgname # mknod /dev/vgname/group c 64 0xnn0000 # vgimport -s -m mapfilename vgname Configuring the complex workload at the primary cluster # cmmakepkg -m dts/sc -m dts/ccsrdf cwsc.config # cmmakepkg -m dts/sc -m dts/ccxpca cwsc.config# cmmakepkg -m dts/sc -m dts/cccaeva cwsc.config # cmmakepkg -m dts/sc -m dts/cc3parrc cwsc.config Suspending the replication to the recovery cluster # cmapplyconf -v -P /etc/cmcluster/cwsc/cwsc.config# cmviewcl -v -p pkgname Where node3 and node4 are the nodes at the recovery cluster # vxdg -stfC import cvmdgname# cmrunpkg siteBcfspkgname Configuring Site Safety Latch dependencies # vgchange -c n vgname # vgchange -c y vgname# vgchange -c n -S n vgname # vgchange -c y -S y vgname Resuming the replication to the recovery cluster Configuring Continentalclusters Administering Continentalclusters Checking the status of clusters, nodes, and packages Configured Event Status Duration Last Notification Sent Package Status State Pkgswitch Node # cmrecovercl -d recoverygroup1 Enabling and disabling maintenance modeStartup and Switching Characteristics Network Attributes # cmrecovercl -e recoverygroup1 Recovering a cluster when the storage array or disks failStarting a recovery package forcefully # cmrecovercl -g RecoveryGroupName Adding or Removing a Node from a Cluster Adding a Recovery Group to Continentalclusters# cmapplyconf -C cluster.config # cmforceconcl cmrunpkg -n node3 Pkg1 Modifying a package in a recovery group Modifying Continentalclusters configurationRemoving a recovery group from the Continentalclusters Behavior of Serviceguard commands in Continentalclusters Removing a rehearsal package from a recovery groupModifying a recovery group with a new rehearsal package Changing monitoring definitions # ps -ef grep cmclsentryd Verifying the status of Continentalclusters daemonsRenaming the Continentalclusters # ps -ef grep cmclapplyd Deleting the Continentalclusters configuration Maintaining the data replication environment# what /usr/sbin/cmviewconcl Using the pairresync command Continuous Access EVA Link Suspend and Resume Modes Maintaining EMC Srdf data replication environment Maintaining 3PAR Remote Copy data replication environment Administering Continentalclusters using Sadta configuration Maintaining a NodeMaintaining the Site Maintaining Site Controller Package # cfsdgadm deleteems pkg1dg /dts/mcsc/cwsc Deleting the Site Controller PackageStarting a Complex Workload Start the Site Controller package by enabling it Restarting a Failed Site Controller Package Moving a Complex Workload to the Recovery ClusterShutting Down a Complex Workload # cmviewcl -l package Reviewing Messages and Log Files Troubleshooting ContinentalclustersTroubleshooting Continentalclusters Error Messages Reviewing Messages and Log Files of Monitoring Daemon Troubleshooting Continentalclusters Error Messages PackageName on cluster ClusterName Verify the configuration and the status of the cluster Migrating to Continentalclusters A.08.00 Recovery Group Worksheet Continentalclusters WorksheetsData Center Worksheet Recovery Checklist Cluster Event Worksheet Continentalclusters Site configuration Replication configuration CRS Sub-cluster configuration using CFS Configuring a CRS sub-cluster using CFS RAC database configuration RAC database configuration Site Controller package configuration RAC MNP Site Controller package configuration Configuration file parameters for Continentalclusters Datasenderpackage clustername/packagename Primarypackage Clustername/Packagename Continentalclusters Command and Daemon Reference Cmrecovercl -f Cmrunpkg or cmmodpkg commandCmcluster/cmoncl.config Cmdeleteconcl -f Cmrecovercl -r-g Recovery group nameCmviewconcl Package attributes Autofencedatasplit Autopsuepsus Autopsusssws Autosvolpfus Autosvolpsue Autosvolpsus Clustertype Pkg Startup Timeout Horctimeout HorctimeoutHorctakeover command, -t timeout. Continuous Access link timeout value Monpollinterval DTS PKG DIRWaittime Monnotificationfrequency Monnotificationsyslog MonnotificationconsoleAutoresync If Waittime is 0 default value, and if the state of DR Before startup of the package on destination volumeStart with an error Fails to start with an error Querytimeout Default Package Attributes for Continentalcluster with EMC SrdfEtc/cmcluster/ packagename/FORCEFLAG Consistencygroups is set to Where the Metrocluster Srdf environment file is located ConfigurationsGroup in an M by N configuration Supported mode are sync for synchronous and async for Legacy packages Migrating legacy to modular packagesMigrating legacy monitor package # cmapplyconf -P modularccmonpkg.conf # cmmigratepkg -p packagename -o modularccmonpkg.conf# cmcheckconf -P modularccmonpkg.conf # cmmigratepkg -p packagename -s -o modularsgconf # cmcheckconf -P modularsgecmcc.conf Validate the new modular package configuration filePathtoenvfile modularsgecmcc.conf # cmapplyconf -P modularsgecmcc.conf # cmmigratepkg -p packagename -s -o modularsg.conf Validate the new package configuration file Configuring legacy packagesConfiguring the monitor package in legacy style # mkdir /etc/cmcluster/ccmonpkg # mkdir /etc/cmcluster/pkgname Use the cmcheckconf command to validate the package# cmcheckconf -P ccmonpkg.config # cd /etc/cmcluster/pkgname # cmmakepkg -p pkgname.ascii Create a package control script # cmmakepkg -s pkgname.cntl EVA Legacy packages # mkdir /etc/cmcluster/pkgname Example 1 Example Set Pkgdir to /etc/cmcluster/pkgname # vxdg -s init cvmdgname replicateddisk1\ replicateddisk2 Configuring storage devices for complex workloadEtc/cmcluster/pkgname # cfsdgadm add cvmdgnamecvmdgpkgname all=sw\ node1node2 # newfs -F vxfs \ Dev/vx/rdsk/cvmdgname/cvmdgvolname Where node1 and node2 are the nodes in the Source Disk Site# cfsdgadm activate cvmdgname # mkdir /cfs # chmod 775 /cfs # mkdir /cfs/cvmdgname # cfsmount /cfs/cvmdgname EVA Sample Continentalclusters Ascii configuration file Continentalclusters Ascii configuration fileSample Continentalclusters Ascii configuration file Clustername #### #### Recoverygroupname Datasenderpackage Notification Console #### #### Clusterevent # Cluster Information Clusteralarm Seconds Notification Sample input and output files for cmswitchconcl command Recoverygroupname RG2 Recoverygroupname RG2 Primarypackage Oracle RAC instances in a Continentalclusters environment Supported Continentalclusters and RAC configuration # cfsmntadm add racdgl vol4 /cfs/mntl all=rw # vxdctl -c mode# cfsdgadm add disk group name all=sw # cfsdgadm activate diskgroupPage # cp /opt/cmconcl/scripts/ccrac.config ccrac.config.mycopy # cd your own directoryCopy the file # cp ccrac.config.mycopy /etc/cmconcl/ccrac/ccrac.config Copy the edited file to the final directoryPage Failover of Oracle RAC instances to the recovery site Oracle RAC instance package can be started in sequence Failback of Oracle RAC instances after a failover # cfsdgadm deactivate Disk Group Name Rehearsing Oracle RAC databases in Continentalclusters# cfsumount Mount Point Name # /opt/cmconcl/bin/ccracmgmt.ksh -i indexNumber \ start Sample Oracle RAC database with ASM in Sadta Configure a primary cluster with a single site Installing Oracle Real Application Clusters RAC software Configure a recovery cluster with a single siteInstalling and configuring Oracle Clusterware Configuring the ASM disk group in the primary cluster Creating the Oracle RAC database in the primary cluster Halting the RAC database in the primary cluster # chown -R oracleoinstall /opt/app/oracle/admin Page # srvctl status Run the following command at the remote site# chown -R oracleoinstall /opt/app/oracle/admin/hrdb # cmviewcl -v -p ASMDGpkgname # cmapplyconf -v -P sitecontrollerconfigurationfile Site cc2site1 # cmrunpkg siteController1 Glossary See also quorum server Span 100s or 1000s of kilometers Or system availability Format /dev/devtype/filename Svol Symbols Index 151