Managing Serviceguard Twentieth Edition
 Legal Notices
 Contents
 Contents
 Planning and Documenting an HA Cluster
 Contents
 Building an HA Cluster Configuration 158
 Contents
 Configuring Packages and Their Services 227
 Contents
 Cluster and Package Maintenance 261
 279
 Troubleshooting Your Cluster 321
 Contents
 Software Upgrades 357
Integrating HA Applications with Serviceguard 354
349
 Migrating from LVM to VxVM Data Storage 374
Blank Planning Worksheets 369
377
IPv6 Network Support 378
 396
Using Serviceguard Manager 385
Index 398
 Publishing History
Publishing History
 Preface
 Related Publications
Securing Serviceguard and other Serviceguard white papers
 What is Serviceguard?
Serviceguard at a Glance
 Shows what happens in a failover situation
Failover
 Typical Cluster After Failover
About Veritas CFS and CVM from Symantec
 Using SAM
Using Serviceguard Manager
 Roadmap for Configuring Clusters and Packages
What are the Distributed Systems Administration Utilities?
Usr/sbin/sam -w
 Tasks in Configuring a Serviceguard Cluster
 Redundancy of Cluster Components
Understanding Serviceguard Hardware Configurations
 Rules and Restrictions
Redundant Network Components
 Redundant LANs
Redundant Ethernet Configuration
 Configuration Tasks
Cross-Subnet Configurations
For legacy packages, see Configuring Cross-Subnet Failover
Restrictions
 For More Information
 Replacing Failed Network Cards
Redundant Disk Storage
Supported Disk Interfaces
 Data Protection
Disk Arrays using RAID Levels and Multiple Data Paths
Disk Mirroring
About Multipathing
 Monitoring VxVM and CVM Disks
Monitoring LVM Disks Through Event Monitoring Service
Replacing Failed Disk Mechanisms
Replacing Failed I/O Cards
 Mirrored Disks Connected for High Availability
Sample Scsi Disk Configurations
 Cluster with High Availability Disk Array
Sample Fibre Channel Disk Configuration
 Larger Clusters
Redundant Power Supplies
 Point to Point Connections to Storage Devices
Active/Standby Model
 Eight-Node Cluster with XP or EMC Disk Array
 Serviceguard Architecture
Understanding Serviceguard Software Components
Serviceguard Daemons
 Cluster Daemon cmcld
Configuration Daemon cmclconfd
 File Management Daemon cmfileassistd
Cluster Logical Volume Manager Daemon cmlvmd
Syslog Log Daemon cmlogd
Cluster Object Manager Daemon cmomd
 Quorum Server Daemon qs
Service Assistant Daemon cmserviced
Network Manager Daemon cmnetd
Lock LUN Daemon cmdisklockd
 How the Cluster Manager Works
Configuring the Cluster
Proxy Daemon cmproxyd
CFS Components
 Manual Startup of Entire Cluster
Heartbeat Messages
 Dynamic Cluster Re-formation
Automatic Cluster Startup
Cluster Quorum to Prevent Split-Brain Syndrome
Cluster Lock
 Lock Requirements
Use of a Lock LUN or LVM Lock Disk as the Cluster Lock
 Single Lock Disk or LUN
Use of the Quorum Server as the Cluster Lock
Dual Lock Disk
 Quorum Server Operation
No Cluster Lock
 Package Types
How the Package Manager Works
Non-failover Packages
Failover Packages
 Failover Packages’ Switching Behavior
Deciding When and Where to Run and Halt Failover Packages
 Before Package Switching
 Automatic Rotating Standby
Package Configuration Data
 Rotating Standby Configuration before Failover
 Failback Policy
Configurednode Policy Packages after Failover
 Automatic Failback Configuration After Failover
 Using the Generic Resources Monitoring Service
Using Older Package Configuration Files
 Understanding Serviceguard Software Components
 Using the EMS HA Monitors
Using the Event Monitoring Service
See also Using Generic Resources to Monitor Volume Groups
See also Using EMS to Monitor Volume Groups
 What Makes a Package Run?
How Packages Run
 Legacy Package Time Line Showing Important Events
Before the Control Script Starts
 Package Time Line Legacy Package
During Run Script Execution
 While Services are Running
Service Startup with cmrunserv
Normal and Abnormal Exits from the Run Script
 During Halt Script Execution
When a Package is Halted with a Command
 Legacy Package Time Line for Halt Script Execution
 Error Conditions and Package Movement for Failover Packages
Package Control Script Error and Exit Conditions
Normal and Abnormal Exits from the Halt Script
 Stationary and Relocatable IP Addresses
How the Network Manager Works
 Types of IP Addresses
Adding and Deleting Relocatable IP Addresses
 Load Sharing
Monitoring LAN Interfaces and Detecting Failure Link Level
 Local Switching
 Cluster Before Local Network Switching
 Cmmodnet -e interface
Where interface is the primary interface
 Monitoring LAN Interfaces and Detecting Failure IP Level
Remote Switching
 How the IP Monitor Works
Reasons To Use IP Monitoring
 Failure and Recovery Detection Times
 Reporting Link-Level and IP-Level Failures
Example 1 If Local Switching is Configured
Constraints and Limitations
See also Reporting Link-Level and IP-Level Failures
 Automatic Port Aggregation
Example 2 If There Is No Local Switching
Cmmodnet -e lan2
 What is VLAN?
Vlan Configurations
Support for HP-UX Vlan
 Configuration Restrictions
Volume Managers for Data Storage
Types of Redundant Storage
Additional Heartbeat Requirements
 Examples of Mirrored Storage
About Device File Names Device Special Files
White papers
 Physical Disks Within Shared Storage Units
 Examples of Storage on Disk Arrays
Multiple Devices Configured in Volume Groups
 Multiple Paths to LUNs
 HP-UX Logical Volume Manager LVM
Types of Volume Manager
Veritas Volume Manager VxVM
Propagation of Disk Groups in VxVM
 Cluster Startup Time with CVM
Veritas Cluster Volume Manager CVM
Propagation of Disk Groups with CVM
For heartbeat requirements, see Redundant Heartbeat Subnets
 Redundant Heartbeat Subnets
Comparison of Volume Managers
 Pros and Cons of Volume Managers with Serviceguard
 What Happens when a Node Times Out
System Reset When a Node Fails
Responses to Failures
Example
 Responses to Hardware Failures
 Responses to Package and Generic Resources Failures
Responses to Package and Service Failures
 Network Communication Failure
Service Restarts
 Planning and Documenting an HA Cluster
Serviceguard Memory Requirements
General Planning
Planning for Expansion
 Hardware Planning
Sample Cluster Configuration
 Network Information
SPU Information
LAN Information
Nnn.nnn.nnn.nnn
 Scsi Addressing in Cluster Configuration
Under Cluster Configuration Parameters
 Diskinfo
Disk I/O Information
 Hardware Configuration Worksheet
Power Supply Planning
 Cluster Lock Planning
Power Supply Configuration Worksheet
 Cluster Lock Disk and Re-formation Time
Using a Quorum Server
Quorum Server Worksheet
 LVM Planning
Using Generic Resources to Monitor Volume Groups
 Using EMS to Monitor Volume Groups
 CVM and VxVM Planning
For more information, see Using the EMS HA Monitors
LVM Worksheet
 CVM and VxVM Worksheet
Cluster Configuration Planning
 About Cluster-wide Device Special Files cDSFs
Where cDSFs Reside
Points To Note
 About Easy Deployment
LVM Commands and cDSFs
Limitations of cDSFs
 Advantages of Easy Deployment
Heartbeat Subnet and Cluster Re-formation Time
Limitations of Easy Deployment
 What Is IPv6-Only Mode?
What Is IPv4-only Mode?
Rules and Restrictions for IPv6-Only Mode
Localhost ipv6-localhost ipv6-loopback
 Recommendations for IPv6-Only Mode
What Is Mixed Mode?
IPV6 or ANY
 Rules and Restrictions for Mixed Mode
Cluster Configuration Parameters
Cluster configuration file
Name of the cluster as it will appear in the output
 Planning and Documenting an HA Cluster
 Go/hpux-serviceguard-docs under HP Serviceguard
Happens when You Change the Quorum Configuration
IPv4-Only,IPv6-Only, and Mixed Mode page 106 for
99 and Specifying a Quorum Server
 IPv6-Only, and Mixed Mode page 106 for important
Hpux-serviceguard-docs under HP Serviceguard
See also About Hostname Address Families IPv4-Only
 Their Services page 227 and these in turn must
Configuration file see Configuring Packages
Sitepreferred or
 Cluster Is Running
Configuration Planning page 125 must be specified
 Cluster Configuration Planning
 CVM/CFS on HP Serviceguard A.11.20 April
You cannot change the heartbeat configuration while
Protocols and services. RPC assumes that each network
To that LAN, to risk timeout without being serviced
 Configuration Online page 49 for important information
Also What Happens when You Change the Quorum
See IPv6 Address Types
Lock LUN page 189 for more information
 Cluster is running, see Updating the Cluster Lock Disk
When You Change the Quorum Configuration Online
Failbackpolicy
 See About Package Weights page 144 for more
 Planning and Documenting an HA Cluster
 88, Cluster Daemon cmcld page 41,
See also What Happens when a Node Times Out
69, Monitoring LAN Interfaces and Detecting
IP-Level Failures
 Configuration file specifies one of two ways to decide
Default is
When a network interface card has failed
How Serviceguard will handle the recovery of the primary
 See Monitoring LAN Interfaces and Detecting Failure IP
 Planning and Documenting an HA Cluster
 Cluster Configuration Next Step
Package Configuration Planning
Logical Volume and File System Planning
Access Control Policies also known as Role Based Access
 CVM 4.1 and later without CFS
 CVM 4.1 and later with CFS
 Using the Volume Monitor
About the Volume Monitor
 Or --version
Or --help
Or --log-file
Or --log-level
 Planning for NFS-mounted File Systems
Volumepath
Usr/sbin/cmvolmond /dev/vg01/lvol1 /dev/vg01/lvol2
Usr/sbin/cmvolmond -t 10 /dev/vg00/lvol1
 Package Configuration Planning
 Package Failover Behavior
Choosing Switching and Failover Behavior
 Configuring a Generic Resource
Parameters for Configuring Generic Resources
Extended generic resource
Cmmakepkg -i $SGCONF/pkg1/pkg1.conf -m sg/genericresource
 Cmapplyconf -P $SGCONF/pkg1/pkg1.conf
Cmcheckconf -v -P $SGCONF/pkg1/pkg1.conf
 Cmrunpkg pkg1
Cmviewcl -v -f line -p pkg1 grep genericresource
Cmgetresource -r sfmdisk
Cmsetresource -r sfmdisk -s up
 Online Reconfiguration of Generic Resources
Parameters for Configuring EMS Resources
 Simple Dependencies
About Package Dependencies
Rules for Simple Dependencies
Assume that we want to make pkg1 depend on pkg2
 Planning and Documenting an HA Cluster
 Dragging Rules for Simple Dependencies
 Planning and Documenting an HA Cluster
 Extended Dependencies
 Rules for Exclusionary Dependencies
See Rules for differentnode and anynode Dependencies
 Rules for differentnode and anynode Dependencies
What Happens when a Package Fails
 Package Weights and Node Capacities
About Package Weights
Configuring Weights and Capacities
Cmmakepkg 1m manpage
 Simple Method
Weightname packagelimit weightvalue
Nodename node1 Capacityname packagelimit
For pkg2
 Points to Keep in Mind
Comprehensive Method
Defining Capacities
 Nodename node2
Clustername cluster23 Nodename node1
 Defining Default Weights
Defining Weights
 Weightname B Weightvalue Weightname a
Weightname a Weightvalue
Weightname B Weightvalue
 Cmquerycl 1m manpage
Rules and Guidelines
 About External Scripts
 Pevmonitoringinterval
 Using Serviceguard Commands in an External Script
 About Cross-Subnet Failover
Determining Why a Package Has Shut Down
Lasthaltfailed
Cmviewcl -v -f line displays a lasthaltfailed flag
 Implications for Application Deployment
Configuring a Package to Fail Over across Subnets Example
 Configuring nodename
Configuring a Package Next Steps
Configuring monitoredsubnetaccess
Configuring ipsubnetnode
 Planning for Changes in Cluster Size
 Installing and Updating Serviceguard
Building an HA Cluster Configuration
Where Serviceguard Files Are Kept
Preparing Your Systems Configuring the Cluster
 Before You Start
Creating Cluster-wide Device Special Files cDSFs
Creating cDSFs for a Group of Nodes
Etc/cmcluster.conf
 Csshsetup -r -f /etc/cmcluster/sshhosts
Csshsetup -r node2
Cmpreparecl -n nodename -n nodename
Cmpreparecl -n node1 -n node2 -n node3 -n node4
 Displaying the cDSF Configuration
Using Easy Deployment
Adding a Node to a cDSF Group
Removing a Node from a cDSF Group
 For example
Using Easy Deployment Commands to Configure the Cluster
Cmquerycl -N $SGCONF/mynetwork
 Preparing Your Systems
 Building an HA Cluster Configuration
 PVG bus1 /dev/cdisk/disk14 /dev/cdisk/disk15
 Allowing Root Access to an Unconfigured Node
Configuring Root-Level Access
Format for entries in cmclnodelist is as follows
 About identd
Ensuring that the Root User on Another Node Is Recognized
 Official hostname, as defined by hosts 4, for example
Configuring Name Resolution
Any of the aliases. Examples
 Safeguarding against Loss of Name Resolution Services
 For NIS, enter two lines
Ensuring Consistency of Kernel Configuration
 Enabling the Network Time Protocol
Tuning Network and Kernel Parameters
 Make the new disk a boot disk
Creating Mirrors of Root Logical Volumes
 Backing Up Cluster Lock Disk Information
Choosing Cluster Lock Disks
 Setting Up a Lock LUN
 Usr/sbin/idisk -w -p -f partition.txt /dev/rdsk/c1t4d0
Creating a Disk Partition on an HP Integrity System
Usr/sbin/idisk -w -p -f partition.txt /dev/rdisk/disk12
This will create three device files, for example
 Defining the Lock LUN
Excluding Devices from Probing
 Creating a Storage Infrastructure with LVM
Setting Up and Running the Quorum Server
 Using the EMS Disk Monitor
Using the Generic Resources Disk Monitor
 Creating Volume Groups
Using Mirrored Individual Data Disks
 Setting Logical Volume Timeouts
Creating Logical Volumes
Creating File Systems
Lvchange -t 60 /dev/vg01/lvol1
 Verify the configuration
Distributing Volume Groups to Other Nodes
Deactivating the Volume Group
Distributing the Volume Group
 Still on ftsys9, copy the map file to ftsys10
Deactivate the volume group on ftsys10
Create a directory to mount the disk
 Creating Additional Volume Groups
Making Physical Volume Group Files Consistent
Creating a Storage Infrastructure with VxVM
Converting Disks from LVM to VxVM
 Initializing Disks Previously Used by LVM
Initializing Disks for VxVM
Creating Disk Groups
 Re-Importing Disk Groups
Deporting Disk Groups
 Clearimport at System Reboot Time
Configuring the Cluster
Here is an example of the command enter it all one line
Cmquerycl -v -C $SGCONF/clust1.conf -n ftsys9 -n ftsys10
 Speeding up the Process
Cmquerycl Options
Specifying the Address Family for the Cluster Hostnames
Specifying the Address Family for the Heartbeat
 Specifying the Cluster Lock
Specifying a Lock Disk
Generating a Network Template File
Full Network Probing
 Cmquerycl -v -n ftsys9 -n ftsys10
Specifying a Lock LUN
Vgchange -c y /dev/vglock
See also Choosing Cluster Lock Disks
 Obtaining Cross-Subnet Information
Specifying a Quorum Server
Cmquerycl -q QSHost QSAddr -n ftsys9 -n ftsys10 -C
Will produce the output such as the following
 Configuring the Cluster
 Modifying the Membertimeout Parameter
Specifying Maximum Number of Configured Packages
Controlling Access to the Cluster
Identifying Heartbeat Subnets
 Access Roles
How Access Roles Work
 Setting up Access-Control Policies
Levels of Access
 Userrole must be one of these three values
Monitor Fulladmin Packageadmin
 Role Conflicts
Username john Userhost bit
Username root
 Verifying the Cluster Configuration
Adding Volume Groups
Package versus Cluster Roles
 Distributing the Binary Configuration File
 Modular CFS packages v/s Legacy CFS packages
Storing Volume Group and Cluster Lock Configuration Data
Differences between Legacy CFS and Modular CFS
 Operational commands for Legacy CFS and Modular CFS
 Delete a mount point, check point, or snapshot in a package
 Preparing the Cluster and the System Multi-node Package
Cfscluster config -t 900 -s
Cfscluster status
 Creating the Disk Group Cluster Packages
Creating the Disk Groups
Cfsdgadm add logdata all=sw
Cfsdgadm display
 Use the vxprint command to verify
Creating Volumes
Cfsdgadm showpackage logdata
Vxprint logfiles
 For instructions on creating modular CFS packages, see
Create a package configuration file
Cmmakepkg -m sg/cfsall /etc/cmcluster/cfspkg1.ascii
 Cmcheckconf -P /etc/cmcluster/cfspkg1.ascii
Apply the package configuration file
Cmapplyconf -P /etc/cmcluster/cfspkg1.ascii
 Bdf
Cmviewcl
Cvmconcurrentdgoperations
Cfsconcurrentmountunmountoperations
 Package. For more information, see the manpage
Cmmakepkg -m sg/cfsall /etc/cmcluster/ckpt1.ascii
See the mountvxfs 1m manpage
Current primary, a primary migration is triggered to
 Vxassist -g cvmdg3 make vol1 100m vxvol -g cvmdg3 startall
Create a package configuration file for the snapshot image
Cmmakepkg -m sg/cfsall snap1.ascii
 Snapshotmountoptions
Mount points
Information about the mount options, see
Mountvxfs 1m manpage
 Online reconfiguration of modular CFS package parameters
 Cmcheckconf -P cfspkg1.ascii
Cmviewcl -v -f line -p cfspkg1
 Cmapplyconf -P cfspkg1.ascii
Apply the configuration
Verify the output
 Legacy Style of Packaging
 Modular Style of Packaging
 Managing Disk Groups and Mount Points Using Legacy Packages
 Fsckptadm -n create check2 /tmp/logdata/logfiles
Creating Checkpoint and Snapshot Packages for CFS
Associate it with the cluster and mount it
Cfsmount /tmp/checklogfiles
 Vxassist -g dg1 make vol1 100m vxvol -g dg1 startall
It is persistent
Associate it with the cluster
Cfsmount /local/snap1 cmviewcl
 You need to do the tasks described in the following sections
 Preparing the Cluster for Use with CVM
Initializing the Veritas Volume Manager
Identifying the Master Node
Initializing Disks for CVM
 Usr/lib/vxvm/bin/vxdisksetup -i c4t3d4
Adding Disk Groups to the Package Configuration
Vxdg -s init logdata c0t3d2
Mirror Detachment Policies with CVM
 Checking Cluster Operation with Serviceguard Manager
Using Dsau during Configuration
Checking Cluster Operation with Serviceguard Commands
Managing the Running Cluster
 Setting up Autostart Features
Preventing Automatic Activation of LVM Volume Groups
 Changing the System Message
Here is an example of the /etc/rc.config.d/cmcluster file
Managing a Single-Node Cluster
 Disabling identd
Deleting the Cluster Configuration
Single-Node Operation
Change the cmclconfd entry in /etc/inetd.conf to
 Building an HA Cluster Configuration
 Configuring Packages and Their Services
 Types of Package Failover, Multi-Node, System Multi-Node
Choosing Package Modules
 Failoverpolicy Failbackpolicy Ipsubnet Ipaddress
 Package Modules and Parameters
Differences between Failover and Multi-Node Packages
Base Package Modules
Cmmakepkg -m sg/all $SGCONF/sg-all
 Base Modules
Optional Package Modules
Optional Modules
 Locallanfailoverallowed
 Externalscript
Package Parameter Explanations
 Cmmakepkg $SGCONF/sg-all
 Nodename
Nodefailfastenabled
Autorun
 Haltscripttimeout
Runscripttimeout
 Scriptlogfile
Successorhalttimeout
Operationsequence
Loglevel
 Dependencyname
Priority
 For more information, see About Package Dependencies
Dependencycondition
 Weightname, weightvalue
Specifies where the dependencycondition must be met
Dependencylocation
 Locallanfailoverallowed
Monitoredsubnetaccess
Monitoredsubnet
Clusterinterconnectsubnet
 Ipsubnet
New for A.11.18 for both modular and legacy packages
Ipsubnet Ipaddress
 See the package configuration file for more examples
Servicename
Ipsubnetnode
Ipaddress
 Servicerestart
Servicecmd
Servicefailfastenabled
Servicehalttimeout
 Genericresourceevaluationtype
Defines when the status of a generic resource is evaluated
Genericresourceupcriteria
 Name of a resource to be monitored
Resourcename
Resourcepollinginterval
Resourcestart
 Resourceupvalue
Enablethreadedvgchange
Concurrentvgchangeoperations
Cannot lock /etc/lvmconf//lvmlock still trying
 Cvmactivationcmd
Vgchangecmd
Vxvolcmd
 Cvmdg
Killprocessesaccessingrawdevices
Vxvmdg
Vxvmdgretry
 Concurrentfsckoperations
Fsfsckopt -s Fstype vxfs
Concurrentmountandumountoperations
Fsmountretrycount
 Fsserver
Fsname
Fsdirectory
Fstype
 Fsumountopt
Fsmountopt
Fsfsckopt
Pev
 Userhost
Username
Userrole
Additional Parameters Used Only by Legacy Packages
 Before You Start
Generating the Package Configuration File
Cmmakepkg Examples
Mkdir $SGCONF/pkg1
 Next Step
Editing the Configuration File
 See About Package Dependencies page 137 for more information
Packagetype. Enter failover, multinode, or systemmultinode
 Editing the Configuration File
 Vg vg01 Vg vg02
 Verifying and Applying the Package Configuration
 How Control Scripts Manage VxVM Disk Groups
Adding the Package to the Cluster
# vxdg -tfC import dg01
 Reviewing Cluster and Package Status
Cluster and Package Maintenance
Viewing Dependencies
Cmviewcl -r A.11.16
 Types of Cluster and Package States
Viewing CFS Multi-Node Information
Cluster Status
Node Status and State
 Reviewing Cluster and Package Status
 Down
Unknown
 Failover and Failback Policies
Examples of Cluster and Package States
Normal Running Status
 CFS Package Status
Quorum Server Status
 Status After Halting a Package
 Status After Moving the Package to Another Node
If we use the following command
Then run cmviewcl -v, we’ll see
 Output of the cmviewcl command is now as follows
Status After Auto Run is Enabled
After we halt ftsys10 with the following command
Status After Halting a Node
 Viewing Information about Unowned Packages
This output can be seen on both ftsys9 and ftsys10
Viewing Information about System Multi-Node Packages
 Status of the Packages in a Cluster File System
Checking Status of the Cluster File System CFS
Cmviewcl -v -p SG-CFS-pkg
 Cmviewcl -v -p mpdg1
Status of CFS Modular Disk Group and Mount Point Packages
Status of Legacy CVM Disk Group Packages
Ftsys9 Sw sw
 Ftsys10
Checking the Cluster Configuration and Components
Status of Legacy CFS Mount Point Packages
Cfsmntadm display -v /tmp/logdata/logfiles
 Checking Cluster Components
Etc/nsswitch.conf Etc/services
User-created files if you specify them
 Cmapplyconf 1m
Verifying Cluster Components
 Run cmcheckconf -C
 Managing the Cluster and Nodes
Setting up Periodic Cluster Verification
Limitations
See the cron 1m manpage for more information
 Adding Previously Configured Nodes to a Running Cluster
Starting the Cluster When all Nodes are Down
Using Serviceguard Commands to Start the Cluster
Cmruncl -v -n ftsys9 -n ftsys10
 Halting the Entire Cluster
Removing Nodes from Participation in a Running Cluster
Cmrunnode -v ftsys8
Cmhaltnode -f -v ftsys9
 What You Can Do
Automatically Restarting the Cluster
Rules and Restrictions
 Managing the Cluster and Nodes
 Additional Points To Note
 Halting a Detached Package
Halting a Node and Detaching its Packages
Halting the Cluster and Detaching its Packages
 Starting a Package
Managing Packages and Services
Cmrunnode node1
 Halting a Package
Using Serviceguard Commands to Start a Package
Starting a Package that Has Dependencies
Halting a Package that Has Dependencies
 Using Serviceguard Commands to Halt a Package
Changing Package Switching Behavior
Changing Package Switching with Serviceguard Commands
Moving a Failover Package
 See Performing Maintenance Using Maintenance Mode
Maintaining a Package Maintenance Mode
Cmmodpkg -d -n lptest3 pkg1
 Cluster and Package Maintenance
 Procedure
Performing Maintenance Using Maintenance Mode
 Cmrunpkg -m sg/packageip pkg1
Excluding Modules in Partial-Startup Maintenance Mode
 Cmrunpkg -e sg/service pkg1
Reconfiguring a Cluster
Cmrunpkg -m sg/services -e sg/packageip pkg1
Types of Changes to the Cluster Configuration
 Previewing the Effect of Cluster Changes
Change to the Cluster Configuration
 Mode see Maintaining a Package Maintenance Mode
Using Preview mode for Commands and in Serviceguard Manager
What You Can Preview
Cmmodpkg -e -t pkg1
 Cmeval -v newstate.in
Using cmeval
You would see output something like this
 Reconfiguring a Halted Cluster
Updating the Cluster Lock Configuration
Updating the Cluster Lock Disk Configuration Online
Updating the Cluster Lock LUN Configuration Online
 Cmapplyconf -C clconfig.ascii
Reconfiguring a Running Cluster
Adding Nodes to the Cluster While the Cluster is Running
Cmgetconf -c cluster1 temp.ascii
 What You Can Do
Cmquerycl -C clconfig.ascii -c cluster1 -n ftsys8 -n ftsys9
 What You Must Keep in Mind
 Example Adding a Heartbeat LAN
Cmquerycl -c cluster1 -C clconfig.ascii
 Removing a LAN or Vlan Interface from a Node
Cmgetconf clconfig.ascii
 Changing the VxVM or CVM Storage Configuration
Changing the LVM Configuration while the Cluster is Running
See also Replacing LAN or Fibre Channel Cards
 Creating the Legacy Package Configuration
Configuring a Legacy Package
Cmgetconf -c clustername clconfig.ascii
 Editing the Package Configuration File
Configuring a Package in Stages
Mkdir /etc/cmcluster/pkg1
 Cluster and Package Maintenance
 Cmmakepkg -s /etc/cmcluster/pkg1/pkg1.sh
Creating the Package Control Script
 Customizing the Package Control Script
 Support for Additional Products
Adding Serviceguard Commands in Customer Defined Functions
 Distributing the Configuration
Verifying the Package Configuration
Copying Package Control Scripts with HP-UX commands
Cmcheckconf -v -P /etc/cmcluster/pkg1/pkg1.conf
 Configuring nodename
Configuring Cross-Subnet Failover
 Configuring monitoredsubnetaccess
Reconfiguring a Package
Creating Subnet-Specific Package Control Scripts
IP0 = SUBNET0 IP1 = SUBNET1
 Migrating a Legacy Package to a Modular Package
Reconfiguring a Package on a Running Cluster
Cmgetconf -p pkg1 pkg1.conf
 Adding a Package to a Running Cluster
Reconfiguring a Package on a Halted Cluster
 Cmhaltpkg mypkg Cmdeleteconf -p mypkg
Deleting a Package from a Running Cluster
Cmapplyconf -v -P app1.conf
Unmount the shared file system cfsumount mount point
 Allowable Package States During Reconfiguration
Resetting the Service Restart Counter
Cmmodpkg -R -s myservice pkg1
 Types of Changes to Packages
Change servicerestart modular package
 Locallanfailoverallowed
 Change vxvolcmd
 Cfsmountoptions
 Changes that Will Trigger Warnings
Responding to Cluster Events
 Removing Serviceguard from a System
Disabling Serviceguard
Single-Node Operation
 Testing Cluster Operation
Troubleshooting Your Cluster
Start the Cluster using Serviceguard Manager
Testing the Package Manager
 Testing the Network Manager
Testing the Cluster Manager
Monitoring Hardware
 Using Event Monitoring Service
Using System Fault Management Service
Using EMS Event Monitoring Service Hardware Monitors
Hardware Monitors and Persistence Requests
 Replacing a Faulty Array Mechanism
Using HP Isee HP Instant Support Enterprise Edition
Replacing a Faulty Mechanism in an HA Enclosure
Replacing Disks
 Replacing a Lock LUN
Replacing a Lock Disk
 Cmdisklock reset /dev/dsk/c0t1d1
Online Hardware Maintenance with In-line Scsi Terminator
Replacing I/O Cards
Replacing Scsi Host Bus Adapters
 Online Replacement
Offline Replacement
Replacing LAN or Fibre Channel Cards
 After Replacing the Card
Replacing a Failed Quorum Server System
 Using cmquerycl and cmcheckconf Using cmviewcl
Troubleshooting Approaches
Reviewing Package IP Addresses
Reviewing the System Log File
 Sample System Log Entries
Reviewing Object Manager Log Files
Following is an example of a successful package starting
Cmreadlog /var/opt/cmom/cmomd.log
 Reviewing Configuration Files
Reviewing Serviceguard Manager Log Files
Using the cmcheckconf Command
Reviewing the System Multi-node Package Files
 Using the cmviewconf Command
Solving Problems
Reviewing the LAN Configuration
Serviceguard Command Hangs
 Cluster Re-formations Caused by Temporary Conditions
Networking and Security Configuration Errors
Nslookup ftsys9
 Package Control Script Hangs or Failures
System Administration Errors
 Fuser -kulogical-volume umount logical-volume
 Llt, gab Vxfen W cvm Cfs
Problems with Cluster File System CFS
 Package Movement Errors
Problems with VxVM Disk Groups
Node and Network Failures
Force Import and Deport After Node Failure
 Authorization File Problems
Troubleshooting the Quorum Server
Timeout Problems
Access denied to quorum server
 Messages
 Enterprise Cluster Master Toolkit
 Automating Application Operation
Designing Highly Available Cluster Applications
 Insulate Users from Outages
Controlling the Speed of Application Failover
Define Application Startup and Shutdown
 Replicate Non-Data File Systems
Use Raw Volumes
Evaluate the Use of JFS
Minimize Data Loss
 Use Checkpoints
Use Restartable Transactions
Design for Multiple Servers
Balance Checkpoint Frequency with Performance
 Design for Replicated Data Sites
Designing Applications to Run on Multiple Systems
Avoid Node-Specific Information
 Assign Unique Names to Applications
Avoid Using SPU IDs or MAC Addresses
Obtain Enough IP Addresses
Allow Multiple Instances on Same System
 Bind to a Fixed Port
Use uname2 With Care
Bind to Relocatable IP Addresses
 Use Multiple Destinations for SNA Applications
Give Each Application its Own Volume Group
Avoid File Locking
Call bind before connect
 Usr/sbin/route add net default 128.17.17.1 1 source
Etc/rc.config.d/nddconf as follows
Help menu for ndd -h ipstrongesmodel
 Restoring Client Connections
Usr/sbin/route delete net default 128.17.17.1 1 source
 Create Applications to be Failure Tolerant
Handling Application Failures
Be Able to Monitor Applications
 Provide for Rolling Upgrades
Reducing Time Needed for Application Upgrades and Patches
Minimizing Planned Downtime
Do Not Change the Data Layout Between Releases
 Documenting Maintenance Operations
Providing Online Application Reconfiguration
 Checklist for Integrating HA Applications
Integrating HA Applications with Serviceguard
Defining Baseline Application Behavior on a Single System
 Integrating HA Applications in Multiple Systems
 Move it back
Testing the Cluster
 Special Considerations for Upgrade to Serviceguard A.11.20
Software Upgrades
Special Considerations for Upgrade to Serviceguard A.11.19
 How To Tell when the Cluster Re-formation Is Complete
Types of Upgrade
Rolling Upgrade
Rolling Upgrade Using DRD
 Restrictions for DRD Upgrades
Guidelines for Rolling Upgrade
Non-Rolling Upgrade
Non-Rolling Upgrade Using DRD
 Limitations of Rolling Upgrades
Performing a Rolling Upgrade
 Keeping Kernels Consistent
Running the Rolling Upgrade
Migrating cmclnodelist entries from A.11.15 or earlier
 Running the Rolling Upgrade Using DRD
Performing a Rolling Upgrade Using DRD
 Step
Example of a Rolling Upgrade
Halt the first node, as follows
 Running Cluster with Packages Moved to Node
 Repeat the process on node 2. Halt the node, as follows
Node 1 Rejoining the Cluster
 Performing a Non-Rolling Upgrade
Guidelines for Non-Rolling Upgrade
 Limitations of Non-Rolling Upgrades using DRD
Performing a Non-Rolling Upgrade Using DRD
Steps for a Non-Rolling Upgrade Using DRD
 Checklist for Migration
Guidelines for Migrating a Cluster with Cold Install
 Blank Planning Worksheets
Power Supply Worksheet
Worksheet for Hardware Planning
 Quorum Server Worksheet
LVM Volume Group and Physical Volume Worksheet
 VxVM Disk Group and Disk Worksheet
Cluster Configuration Worksheet
 Package Configuration Worksheet
 Package Configuration Worksheet
 Migrating from LVM to VxVM Data Storage
Migrating Volume Groups
Loading VxVM
 Mntdg0202, respectively
Customizing Packages for VxVM
 Customizing Packages for CVM
Removing LVM Volume Groups
Restart the package
 Migrating from Legacy CFS Packages to Modular CFS Packages
 IPv6 Address Types
IPv6 Network Support
Textual Representation of IPv6 Addresses
 Unicast Addresses
IPv6 Address Prefix
IPv4 and IPv6 Compatibility
IPv4 Compatible IPv6 Addresses
 Link-Local Addresses
Aggregatable Global Unicast Addresses
Site-Local Addresses
Multicast Addresses
 Network Configuration Restrictions
 Local Primary/Standby LAN Patterns
Example Configurations
Ndd -get /dev/ip6 ip6nddadsolicitcount
Ndd -set /dev/ip6 ip6nddadsolicitcountn
 Example Configurations
 384 IPv6 Network Support
 Before Using HP Serviceguard Manager Setting Up
Using Serviceguard Manager
Accessing Serviceguard Manager
About the Online Help System
 Accessing Serviceguard Manager
Launching Serviceguard Manager
Scenario 1 Single cluster management
Opt/hpsmh/bin/hpsmh autostart
 System Management Homepage with Serviceguard Manager
 Sign
Expand HP Serviceguard, and click on a Serviceguard cluster
From the left-hand panel, expand Cluster by Type
 Maximum and Minimum Values for Parameters
Membertimeout
 Sample scripts
Monitoring Script for Generic Resources
Launching Monitoring Scripts
 Launching Monitoring Scripts
 Template of a Monitoring Script
 I L I T Y N C T I O N S
 Monitoring Script for Generic Resources
 Template of a Monitoring Script
 Migrating EMS Resources to Generic Resources
 Identify the equivalent SFM style resource monitor
Start the package
 APA
Index
 399
 Cvmactivationcmd
 Firstclusterlockpv
 LAN
 INONLYORINOUT, 69 Inout
 Pollingtarget defined
 Qsaddr
 Servicename
 Vxvmdg