Managing Serviceguard Twentieth Edition
 Legal Notices
 Contents
 Contents
 Planning and Documenting an HA Cluster
 Contents
 Building an HA Cluster Configuration 158
 Contents
 Configuring Packages and Their Services 227
 Contents
 Cluster and Package Maintenance 261
 279
 Troubleshooting Your Cluster 321
 Contents
 Integrating HA Applications with Serviceguard 354
Software Upgrades 357
349
 377
Blank Planning Worksheets 369
Migrating from LVM to VxVM Data Storage 374
IPv6 Network Support 378
 Using Serviceguard Manager 385
396
Index 398
 Publishing History
Publishing History
 Preface
 Securing Serviceguard and other Serviceguard white papers
Related Publications
 Serviceguard at a Glance
What is Serviceguard?
 Failover
Shows what happens in a failover situation
 About Veritas CFS and CVM from Symantec
Typical Cluster After Failover
 Using Serviceguard Manager
Using SAM
 What are the Distributed Systems Administration Utilities?
Roadmap for Configuring Clusters and Packages
Usr/sbin/sam -w
 Tasks in Configuring a Serviceguard Cluster
 Understanding Serviceguard Hardware Configurations
Redundancy of Cluster Components
 Redundant Network Components
Rules and Restrictions
 Redundant Ethernet Configuration
Redundant LANs
 For legacy packages, see Configuring Cross-Subnet Failover
Cross-Subnet Configurations
Configuration Tasks
Restrictions
 For More Information
 Redundant Disk Storage
Replacing Failed Network Cards
Supported Disk Interfaces
 Disk Mirroring
Disk Arrays using RAID Levels and Multiple Data Paths
Data Protection
About Multipathing
 Replacing Failed Disk Mechanisms
Monitoring LVM Disks Through Event Monitoring Service
Monitoring VxVM and CVM Disks
Replacing Failed I/O Cards
 Sample Scsi Disk Configurations
Mirrored Disks Connected for High Availability
 Sample Fibre Channel Disk Configuration
Cluster with High Availability Disk Array
 Redundant Power Supplies
Larger Clusters
 Active/Standby Model
Point to Point Connections to Storage Devices
 Eight-Node Cluster with XP or EMC Disk Array
 Understanding Serviceguard Software Components
Serviceguard Architecture
Serviceguard Daemons
 Configuration Daemon cmclconfd
Cluster Daemon cmcld
 Syslog Log Daemon cmlogd
Cluster Logical Volume Manager Daemon cmlvmd
File Management Daemon cmfileassistd
Cluster Object Manager Daemon cmomd
 Network Manager Daemon cmnetd
Service Assistant Daemon cmserviced
Quorum Server Daemon qs
Lock LUN Daemon cmdisklockd
 Proxy Daemon cmproxyd
Configuring the Cluster
How the Cluster Manager Works
CFS Components
 Heartbeat Messages
Manual Startup of Entire Cluster
 Cluster Quorum to Prevent Split-Brain Syndrome
Automatic Cluster Startup
Dynamic Cluster Re-formation
Cluster Lock
 Use of a Lock LUN or LVM Lock Disk as the Cluster Lock
Lock Requirements
 Use of the Quorum Server as the Cluster Lock
Single Lock Disk or LUN
Dual Lock Disk
 No Cluster Lock
Quorum Server Operation
 Non-failover Packages
How the Package Manager Works
Package Types
Failover Packages
 Deciding When and Where to Run and Halt Failover Packages
Failover Packages’ Switching Behavior
 Before Package Switching
 Package Configuration Data
Automatic Rotating Standby
 Rotating Standby Configuration before Failover
 Configurednode Policy Packages after Failover
Failback Policy
 Automatic Failback Configuration After Failover
 Using Older Package Configuration Files
Using the Generic Resources Monitoring Service
 Understanding Serviceguard Software Components
 See also Using Generic Resources to Monitor Volume Groups
Using the Event Monitoring Service
Using the EMS HA Monitors
See also Using EMS to Monitor Volume Groups
 How Packages Run
What Makes a Package Run?
 Before the Control Script Starts
Legacy Package Time Line Showing Important Events
 During Run Script Execution
Package Time Line Legacy Package
 Service Startup with cmrunserv
While Services are Running
Normal and Abnormal Exits from the Run Script
 When a Package is Halted with a Command
During Halt Script Execution
 Legacy Package Time Line for Halt Script Execution
 Package Control Script Error and Exit Conditions
Error Conditions and Package Movement for Failover Packages
Normal and Abnormal Exits from the Halt Script
 How the Network Manager Works
Stationary and Relocatable IP Addresses
 Adding and Deleting Relocatable IP Addresses
Types of IP Addresses
 Monitoring LAN Interfaces and Detecting Failure Link Level
Load Sharing
 Local Switching
 Cluster Before Local Network Switching
 Where interface is the primary interface
Cmmodnet -e interface
 Remote Switching
Monitoring LAN Interfaces and Detecting Failure IP Level
 Reasons To Use IP Monitoring
How the IP Monitor Works
 Failure and Recovery Detection Times
 Constraints and Limitations
Example 1 If Local Switching is Configured
Reporting Link-Level and IP-Level Failures
See also Reporting Link-Level and IP-Level Failures
 Example 2 If There Is No Local Switching
Automatic Port Aggregation
Cmmodnet -e lan2
 Vlan Configurations
What is VLAN?
Support for HP-UX Vlan
 Types of Redundant Storage
Volume Managers for Data Storage
Configuration Restrictions
Additional Heartbeat Requirements
 About Device File Names Device Special Files
Examples of Mirrored Storage
White papers
 Physical Disks Within Shared Storage Units
 Multiple Devices Configured in Volume Groups
Examples of Storage on Disk Arrays
 Multiple Paths to LUNs
 Veritas Volume Manager VxVM
Types of Volume Manager
HP-UX Logical Volume Manager LVM
Propagation of Disk Groups in VxVM
 Propagation of Disk Groups with CVM
Veritas Cluster Volume Manager CVM
Cluster Startup Time with CVM
For heartbeat requirements, see Redundant Heartbeat Subnets
 Comparison of Volume Managers
Redundant Heartbeat Subnets
 Pros and Cons of Volume Managers with Serviceguard
 Responses to Failures
System Reset When a Node Fails
What Happens when a Node Times Out
Example
 Responses to Hardware Failures
 Responses to Package and Service Failures
Responses to Package and Generic Resources Failures
 Service Restarts
Network Communication Failure
 General Planning
Serviceguard Memory Requirements
Planning and Documenting an HA Cluster
Planning for Expansion
 Sample Cluster Configuration
Hardware Planning
 LAN Information
SPU Information
Network Information
Nnn.nnn.nnn.nnn
 Under Cluster Configuration Parameters
Scsi Addressing in Cluster Configuration
 Disk I/O Information
Diskinfo
 Power Supply Planning
Hardware Configuration Worksheet
 Power Supply Configuration Worksheet
Cluster Lock Planning
 Using a Quorum Server
Cluster Lock Disk and Re-formation Time
Quorum Server Worksheet
 Using Generic Resources to Monitor Volume Groups
LVM Planning
 Using EMS to Monitor Volume Groups
 For more information, see Using the EMS HA Monitors
CVM and VxVM Planning
LVM Worksheet
 Cluster Configuration Planning
CVM and VxVM Worksheet
 Where cDSFs Reside
About Cluster-wide Device Special Files cDSFs
Points To Note
 LVM Commands and cDSFs
About Easy Deployment
Limitations of cDSFs
 Heartbeat Subnet and Cluster Re-formation Time
Advantages of Easy Deployment
Limitations of Easy Deployment
 Rules and Restrictions for IPv6-Only Mode
What Is IPv4-only Mode?
What Is IPv6-Only Mode?
Localhost ipv6-localhost ipv6-loopback
 What Is Mixed Mode?
Recommendations for IPv6-Only Mode
IPV6 or ANY
 Cluster configuration file
Cluster Configuration Parameters
Rules and Restrictions for Mixed Mode
Name of the cluster as it will appear in the output
 Planning and Documenting an HA Cluster
 IPv4-Only,IPv6-Only, and Mixed Mode page 106 for
Happens when You Change the Quorum Configuration
Go/hpux-serviceguard-docs under HP Serviceguard
99 and Specifying a Quorum Server
 Hpux-serviceguard-docs under HP Serviceguard
IPv6-Only, and Mixed Mode page 106 for important
See also About Hostname Address Families IPv4-Only
 Configuration file see Configuring Packages
Their Services page 227 and these in turn must
Sitepreferred or
 Configuration Planning page 125 must be specified
Cluster Is Running
 Cluster Configuration Planning
 Protocols and services. RPC assumes that each network
You cannot change the heartbeat configuration while
CVM/CFS on HP Serviceguard A.11.20 April
To that LAN, to risk timeout without being serviced
 See IPv6 Address Types
Also What Happens when You Change the Quorum
Configuration Online page 49 for important information
Lock LUN page 189 for more information
 When You Change the Quorum Configuration Online
Cluster is running, see Updating the Cluster Lock Disk
Failbackpolicy
 See About Package Weights page 144 for more
 Planning and Documenting an HA Cluster
 69, Monitoring LAN Interfaces and Detecting
See also What Happens when a Node Times Out
88, Cluster Daemon cmcld page 41,
IP-Level Failures
 When a network interface card has failed
Default is
Configuration file specifies one of two ways to decide
How Serviceguard will handle the recovery of the primary
 See Monitoring LAN Interfaces and Detecting Failure IP
 Planning and Documenting an HA Cluster
 Logical Volume and File System Planning
Package Configuration Planning
Cluster Configuration Next Step
Access Control Policies also known as Role Based Access
 CVM 4.1 and later without CFS
 CVM 4.1 and later with CFS
 About the Volume Monitor
Using the Volume Monitor
 Or --log-file
Or --help
Or --version
Or --log-level
 Usr/sbin/cmvolmond /dev/vg01/lvol1 /dev/vg01/lvol2
Volumepath
Planning for NFS-mounted File Systems
Usr/sbin/cmvolmond -t 10 /dev/vg00/lvol1
 Package Configuration Planning
 Choosing Switching and Failover Behavior
Package Failover Behavior
 Extended generic resource
Parameters for Configuring Generic Resources
Configuring a Generic Resource
Cmmakepkg -i $SGCONF/pkg1/pkg1.conf -m sg/genericresource
 Cmcheckconf -v -P $SGCONF/pkg1/pkg1.conf
Cmapplyconf -P $SGCONF/pkg1/pkg1.conf
 Cmgetresource -r sfmdisk
Cmviewcl -v -f line -p pkg1 grep genericresource
Cmrunpkg pkg1
Cmsetresource -r sfmdisk -s up
 Parameters for Configuring EMS Resources
Online Reconfiguration of Generic Resources
 Rules for Simple Dependencies
About Package Dependencies
Simple Dependencies
Assume that we want to make pkg1 depend on pkg2
 Planning and Documenting an HA Cluster
 Dragging Rules for Simple Dependencies
 Planning and Documenting an HA Cluster
 Extended Dependencies
 See Rules for differentnode and anynode Dependencies
Rules for Exclusionary Dependencies
 What Happens when a Package Fails
Rules for differentnode and anynode Dependencies
 Configuring Weights and Capacities
About Package Weights
Package Weights and Node Capacities
Cmmakepkg 1m manpage
 Nodename node1 Capacityname packagelimit
Weightname packagelimit weightvalue
Simple Method
For pkg2
 Comprehensive Method
Points to Keep in Mind
Defining Capacities
 Clustername cluster23 Nodename node1
Nodename node2
 Defining Weights
Defining Default Weights
 Weightname a Weightvalue
Weightname B Weightvalue Weightname a
Weightname B Weightvalue
 Rules and Guidelines
Cmquerycl 1m manpage
 About External Scripts
 Pevmonitoringinterval
 Using Serviceguard Commands in an External Script
 Lasthaltfailed
Determining Why a Package Has Shut Down
About Cross-Subnet Failover
Cmviewcl -v -f line displays a lasthaltfailed flag
 Configuring a Package to Fail Over across Subnets Example
Implications for Application Deployment
 Configuring monitoredsubnetaccess
Configuring a Package Next Steps
Configuring nodename
Configuring ipsubnetnode
 Planning for Changes in Cluster Size
 Where Serviceguard Files Are Kept
Building an HA Cluster Configuration
Installing and Updating Serviceguard
Preparing Your Systems Configuring the Cluster
 Creating cDSFs for a Group of Nodes
Creating Cluster-wide Device Special Files cDSFs
Before You Start
Etc/cmcluster.conf
 Cmpreparecl -n nodename -n nodename
Csshsetup -r node2
Csshsetup -r -f /etc/cmcluster/sshhosts
Cmpreparecl -n node1 -n node2 -n node3 -n node4
 Adding a Node to a cDSF Group
Using Easy Deployment
Displaying the cDSF Configuration
Removing a Node from a cDSF Group
 Using Easy Deployment Commands to Configure the Cluster
For example
Cmquerycl -N $SGCONF/mynetwork
 Preparing Your Systems
 Building an HA Cluster Configuration
 PVG bus1 /dev/cdisk/disk14 /dev/cdisk/disk15
 Configuring Root-Level Access
Allowing Root Access to an Unconfigured Node
Format for entries in cmclnodelist is as follows
 Ensuring that the Root User on Another Node Is Recognized
About identd
 Configuring Name Resolution
Official hostname, as defined by hosts 4, for example
Any of the aliases. Examples
 Safeguarding against Loss of Name Resolution Services
 Ensuring Consistency of Kernel Configuration
For NIS, enter two lines
 Tuning Network and Kernel Parameters
Enabling the Network Time Protocol
 Creating Mirrors of Root Logical Volumes
Make the new disk a boot disk
 Choosing Cluster Lock Disks
Backing Up Cluster Lock Disk Information
 Setting Up a Lock LUN
 Usr/sbin/idisk -w -p -f partition.txt /dev/rdisk/disk12
Creating a Disk Partition on an HP Integrity System
Usr/sbin/idisk -w -p -f partition.txt /dev/rdsk/c1t4d0
This will create three device files, for example
 Excluding Devices from Probing
Defining the Lock LUN
 Setting Up and Running the Quorum Server
Creating a Storage Infrastructure with LVM
 Using the Generic Resources Disk Monitor
Using the EMS Disk Monitor
 Using Mirrored Individual Data Disks
Creating Volume Groups
 Creating File Systems
Creating Logical Volumes
Setting Logical Volume Timeouts
Lvchange -t 60 /dev/vg01/lvol1
 Deactivating the Volume Group
Distributing Volume Groups to Other Nodes
Verify the configuration
Distributing the Volume Group
 Deactivate the volume group on ftsys10
Still on ftsys9, copy the map file to ftsys10
Create a directory to mount the disk
 Creating a Storage Infrastructure with VxVM
Making Physical Volume Group Files Consistent
Creating Additional Volume Groups
Converting Disks from LVM to VxVM
 Initializing Disks for VxVM
Initializing Disks Previously Used by LVM
Creating Disk Groups
 Deporting Disk Groups
Re-Importing Disk Groups
 Here is an example of the command enter it all one line
Configuring the Cluster
Clearimport at System Reboot Time
Cmquerycl -v -C $SGCONF/clust1.conf -n ftsys9 -n ftsys10
 Specifying the Address Family for the Cluster Hostnames
Cmquerycl Options
Speeding up the Process
Specifying the Address Family for the Heartbeat
 Generating a Network Template File
Specifying a Lock Disk
Specifying the Cluster Lock
Full Network Probing
 Vgchange -c y /dev/vglock
Specifying a Lock LUN
Cmquerycl -v -n ftsys9 -n ftsys10
See also Choosing Cluster Lock Disks
 Cmquerycl -q QSHost QSAddr -n ftsys9 -n ftsys10 -C
Specifying a Quorum Server
Obtaining Cross-Subnet Information
Will produce the output such as the following
 Configuring the Cluster
 Controlling Access to the Cluster
Specifying Maximum Number of Configured Packages
Modifying the Membertimeout Parameter
Identifying Heartbeat Subnets
 How Access Roles Work
Access Roles
 Levels of Access
Setting up Access-Control Policies
 Monitor Fulladmin Packageadmin
Userrole must be one of these three values
 Username john Userhost bit
Role Conflicts
Username root
 Adding Volume Groups
Verifying the Cluster Configuration
Package versus Cluster Roles
 Distributing the Binary Configuration File
 Storing Volume Group and Cluster Lock Configuration Data
Modular CFS packages v/s Legacy CFS packages
Differences between Legacy CFS and Modular CFS
 Operational commands for Legacy CFS and Modular CFS
 Delete a mount point, check point, or snapshot in a package
 Cfscluster config -t 900 -s
Preparing the Cluster and the System Multi-node Package
Cfscluster status
 Cfsdgadm add logdata all=sw
Creating the Disk Groups
Creating the Disk Group Cluster Packages
Cfsdgadm display
 Cfsdgadm showpackage logdata
Creating Volumes
Use the vxprint command to verify
Vxprint logfiles
 Create a package configuration file
For instructions on creating modular CFS packages, see
Cmmakepkg -m sg/cfsall /etc/cmcluster/cfspkg1.ascii
 Apply the package configuration file
Cmcheckconf -P /etc/cmcluster/cfspkg1.ascii
Cmapplyconf -P /etc/cmcluster/cfspkg1.ascii
 Cvmconcurrentdgoperations
Cmviewcl
Bdf
Cfsconcurrentmountunmountoperations
 See the mountvxfs 1m manpage
Cmmakepkg -m sg/cfsall /etc/cmcluster/ckpt1.ascii
Package. For more information, see the manpage
Current primary, a primary migration is triggered to
 Create a package configuration file for the snapshot image
Vxassist -g cvmdg3 make vol1 100m vxvol -g cvmdg3 startall
Cmmakepkg -m sg/cfsall snap1.ascii
 Information about the mount options, see
Mount points
Snapshotmountoptions
Mountvxfs 1m manpage
 Online reconfiguration of modular CFS package parameters
 Cmviewcl -v -f line -p cfspkg1
Cmcheckconf -P cfspkg1.ascii
 Apply the configuration
Cmapplyconf -P cfspkg1.ascii
Verify the output
 Legacy Style of Packaging
 Modular Style of Packaging
 Managing Disk Groups and Mount Points Using Legacy Packages
 Associate it with the cluster and mount it
Creating Checkpoint and Snapshot Packages for CFS
Fsckptadm -n create check2 /tmp/logdata/logfiles
Cfsmount /tmp/checklogfiles
 Associate it with the cluster
It is persistent
Vxassist -g dg1 make vol1 100m vxvol -g dg1 startall
Cfsmount /local/snap1 cmviewcl
 You need to do the tasks described in the following sections
 Identifying the Master Node
Initializing the Veritas Volume Manager
Preparing the Cluster for Use with CVM
Initializing Disks for CVM
 Vxdg -s init logdata c0t3d2
Adding Disk Groups to the Package Configuration
Usr/lib/vxvm/bin/vxdisksetup -i c4t3d4
Mirror Detachment Policies with CVM
 Checking Cluster Operation with Serviceguard Commands
Using Dsau during Configuration
Checking Cluster Operation with Serviceguard Manager
Managing the Running Cluster
 Preventing Automatic Activation of LVM Volume Groups
Setting up Autostart Features
 Here is an example of the /etc/rc.config.d/cmcluster file
Changing the System Message
Managing a Single-Node Cluster
 Single-Node Operation
Deleting the Cluster Configuration
Disabling identd
Change the cmclconfd entry in /etc/inetd.conf to
 Building an HA Cluster Configuration
 Configuring Packages and Their Services
 Choosing Package Modules
Types of Package Failover, Multi-Node, System Multi-Node
 Failoverpolicy Failbackpolicy Ipsubnet Ipaddress
 Base Package Modules
Differences between Failover and Multi-Node Packages
Package Modules and Parameters
Cmmakepkg -m sg/all $SGCONF/sg-all
 Optional Package Modules
Base Modules
Optional Modules
 Locallanfailoverallowed
 Package Parameter Explanations
Externalscript
 Cmmakepkg $SGCONF/sg-all
 Nodefailfastenabled
Nodename
Autorun
 Runscripttimeout
Haltscripttimeout
 Operationsequence
Successorhalttimeout
Scriptlogfile
Loglevel
 Priority
Dependencyname
 Dependencycondition
For more information, see About Package Dependencies
 Specifies where the dependencycondition must be met
Weightname, weightvalue
Dependencylocation
 Monitoredsubnet
Monitoredsubnetaccess
Locallanfailoverallowed
Clusterinterconnectsubnet
 New for A.11.18 for both modular and legacy packages
Ipsubnet
Ipsubnet Ipaddress
 Ipsubnetnode
Servicename
See the package configuration file for more examples
Ipaddress
 Servicefailfastenabled
Servicecmd
Servicerestart
Servicehalttimeout
 Defines when the status of a generic resource is evaluated
Genericresourceevaluationtype
Genericresourceupcriteria
 Resourcepollinginterval
Resourcename
Name of a resource to be monitored
Resourcestart
 Concurrentvgchangeoperations
Enablethreadedvgchange
Resourceupvalue
Cannot lock /etc/lvmconf//lvmlock still trying
 Vgchangecmd
Cvmactivationcmd
Vxvolcmd
 Vxvmdg
Killprocessesaccessingrawdevices
Cvmdg
Vxvmdgretry
 Concurrentmountandumountoperations
Fsfsckopt -s Fstype vxfs
Concurrentfsckoperations
Fsmountretrycount
 Fsdirectory
Fsname
Fsserver
Fstype
 Fsfsckopt
Fsmountopt
Fsumountopt
Pev
 Userrole
Username
Userhost
Additional Parameters Used Only by Legacy Packages
 Cmmakepkg Examples
Generating the Package Configuration File
Before You Start
Mkdir $SGCONF/pkg1
 Editing the Configuration File
Next Step
 Packagetype. Enter failover, multinode, or systemmultinode
See About Package Dependencies page 137 for more information
 Editing the Configuration File
 Vg vg01 Vg vg02
 Verifying and Applying the Package Configuration
 Adding the Package to the Cluster
How Control Scripts Manage VxVM Disk Groups
# vxdg -tfC import dg01
 Viewing Dependencies
Cluster and Package Maintenance
Reviewing Cluster and Package Status
Cmviewcl -r A.11.16
 Cluster Status
Viewing CFS Multi-Node Information
Types of Cluster and Package States
Node Status and State
 Reviewing Cluster and Package Status
 Unknown
Down
 Examples of Cluster and Package States
Failover and Failback Policies
Normal Running Status
 Quorum Server Status
CFS Package Status
 Status After Halting a Package
 If we use the following command
Status After Moving the Package to Another Node
Then run cmviewcl -v, we’ll see
 After we halt ftsys10 with the following command
Status After Auto Run is Enabled
Output of the cmviewcl command is now as follows
Status After Halting a Node
 This output can be seen on both ftsys9 and ftsys10
Viewing Information about Unowned Packages
Viewing Information about System Multi-Node Packages
 Checking Status of the Cluster File System CFS
Status of the Packages in a Cluster File System
Cmviewcl -v -p SG-CFS-pkg
 Status of Legacy CVM Disk Group Packages
Status of CFS Modular Disk Group and Mount Point Packages
Cmviewcl -v -p mpdg1
Ftsys9 Sw sw
 Status of Legacy CFS Mount Point Packages
Checking the Cluster Configuration and Components
Ftsys10
Cfsmntadm display -v /tmp/logdata/logfiles
 Etc/nsswitch.conf Etc/services
Checking Cluster Components
User-created files if you specify them
 Verifying Cluster Components
Cmapplyconf 1m
 Run cmcheckconf -C
 Limitations
Setting up Periodic Cluster Verification
Managing the Cluster and Nodes
See the cron 1m manpage for more information
 Using Serviceguard Commands to Start the Cluster
Starting the Cluster When all Nodes are Down
Adding Previously Configured Nodes to a Running Cluster
Cmruncl -v -n ftsys9 -n ftsys10
 Cmrunnode -v ftsys8
Removing Nodes from Participation in a Running Cluster
Halting the Entire Cluster
Cmhaltnode -f -v ftsys9
 Automatically Restarting the Cluster
What You Can Do
Rules and Restrictions
 Managing the Cluster and Nodes
 Additional Points To Note
 Halting a Node and Detaching its Packages
Halting a Detached Package
Halting the Cluster and Detaching its Packages
 Managing Packages and Services
Starting a Package
Cmrunnode node1
 Starting a Package that Has Dependencies
Using Serviceguard Commands to Start a Package
Halting a Package
Halting a Package that Has Dependencies
 Changing Package Switching with Serviceguard Commands
Changing Package Switching Behavior
Using Serviceguard Commands to Halt a Package
Moving a Failover Package
 Maintaining a Package Maintenance Mode
See Performing Maintenance Using Maintenance Mode
Cmmodpkg -d -n lptest3 pkg1
 Cluster and Package Maintenance
 Performing Maintenance Using Maintenance Mode
Procedure
 Excluding Modules in Partial-Startup Maintenance Mode
Cmrunpkg -m sg/packageip pkg1
 Cmrunpkg -m sg/services -e sg/packageip pkg1
Reconfiguring a Cluster
Cmrunpkg -e sg/service pkg1
Types of Changes to the Cluster Configuration
 Change to the Cluster Configuration
Previewing the Effect of Cluster Changes
 What You Can Preview
Using Preview mode for Commands and in Serviceguard Manager
Mode see Maintaining a Package Maintenance Mode
Cmmodpkg -e -t pkg1
 Using cmeval
Cmeval -v newstate.in
You would see output something like this
 Updating the Cluster Lock Disk Configuration Online
Updating the Cluster Lock Configuration
Reconfiguring a Halted Cluster
Updating the Cluster Lock LUN Configuration Online
 Adding Nodes to the Cluster While the Cluster is Running
Reconfiguring a Running Cluster
Cmapplyconf -C clconfig.ascii
Cmgetconf -c cluster1 temp.ascii
 Cmquerycl -C clconfig.ascii -c cluster1 -n ftsys8 -n ftsys9
What You Can Do
 What You Must Keep in Mind
 Cmquerycl -c cluster1 -C clconfig.ascii
Example Adding a Heartbeat LAN
 Cmgetconf clconfig.ascii
Removing a LAN or Vlan Interface from a Node
 Changing the LVM Configuration while the Cluster is Running
Changing the VxVM or CVM Storage Configuration
See also Replacing LAN or Fibre Channel Cards
 Configuring a Legacy Package
Creating the Legacy Package Configuration
Cmgetconf -c clustername clconfig.ascii
 Configuring a Package in Stages
Editing the Package Configuration File
Mkdir /etc/cmcluster/pkg1
 Cluster and Package Maintenance
 Creating the Package Control Script
Cmmakepkg -s /etc/cmcluster/pkg1/pkg1.sh
 Customizing the Package Control Script
 Adding Serviceguard Commands in Customer Defined Functions
Support for Additional Products
 Copying Package Control Scripts with HP-UX commands
Verifying the Package Configuration
Distributing the Configuration
Cmcheckconf -v -P /etc/cmcluster/pkg1/pkg1.conf
 Configuring Cross-Subnet Failover
Configuring nodename
 Creating Subnet-Specific Package Control Scripts
Reconfiguring a Package
Configuring monitoredsubnetaccess
IP0 = SUBNET0 IP1 = SUBNET1
 Reconfiguring a Package on a Running Cluster
Migrating a Legacy Package to a Modular Package
Cmgetconf -p pkg1 pkg1.conf
 Reconfiguring a Package on a Halted Cluster
Adding a Package to a Running Cluster
 Cmapplyconf -v -P app1.conf
Deleting a Package from a Running Cluster
Cmhaltpkg mypkg Cmdeleteconf -p mypkg
Unmount the shared file system cfsumount mount point
 Resetting the Service Restart Counter
Allowable Package States During Reconfiguration
Cmmodpkg -R -s myservice pkg1
 Change servicerestart modular package
Types of Changes to Packages
 Locallanfailoverallowed
 Change vxvolcmd
 Cfsmountoptions
 Responding to Cluster Events
Changes that Will Trigger Warnings
 Disabling Serviceguard
Removing Serviceguard from a System
Single-Node Operation
 Start the Cluster using Serviceguard Manager
Troubleshooting Your Cluster
Testing Cluster Operation
Testing the Package Manager
 Testing the Cluster Manager
Testing the Network Manager
Monitoring Hardware
 Using EMS Event Monitoring Service Hardware Monitors
Using System Fault Management Service
Using Event Monitoring Service
Hardware Monitors and Persistence Requests
 Replacing a Faulty Mechanism in an HA Enclosure
Using HP Isee HP Instant Support Enterprise Edition
Replacing a Faulty Array Mechanism
Replacing Disks
 Replacing a Lock Disk
Replacing a Lock LUN
 Replacing I/O Cards
Online Hardware Maintenance with In-line Scsi Terminator
Cmdisklock reset /dev/dsk/c0t1d1
Replacing Scsi Host Bus Adapters
 Offline Replacement
Online Replacement
Replacing LAN or Fibre Channel Cards
 Replacing a Failed Quorum Server System
After Replacing the Card
 Reviewing Package IP Addresses
Troubleshooting Approaches
Using cmquerycl and cmcheckconf Using cmviewcl
Reviewing the System Log File
 Following is an example of a successful package starting
Reviewing Object Manager Log Files
Sample System Log Entries
Cmreadlog /var/opt/cmom/cmomd.log
 Using the cmcheckconf Command
Reviewing Serviceguard Manager Log Files
Reviewing Configuration Files
Reviewing the System Multi-node Package Files
 Reviewing the LAN Configuration
Solving Problems
Using the cmviewconf Command
Serviceguard Command Hangs
 Networking and Security Configuration Errors
Cluster Re-formations Caused by Temporary Conditions
Nslookup ftsys9
 System Administration Errors
Package Control Script Hangs or Failures
 Fuser -kulogical-volume umount logical-volume
 Problems with Cluster File System CFS
Llt, gab Vxfen W cvm Cfs
 Node and Network Failures
Problems with VxVM Disk Groups
Package Movement Errors
Force Import and Deport After Node Failure
 Timeout Problems
Troubleshooting the Quorum Server
Authorization File Problems
Access denied to quorum server
 Messages
 Enterprise Cluster Master Toolkit
 Designing Highly Available Cluster Applications
Automating Application Operation
 Controlling the Speed of Application Failover
Insulate Users from Outages
Define Application Startup and Shutdown
 Evaluate the Use of JFS
Use Raw Volumes
Replicate Non-Data File Systems
Minimize Data Loss
 Design for Multiple Servers
Use Restartable Transactions
Use Checkpoints
Balance Checkpoint Frequency with Performance
 Designing Applications to Run on Multiple Systems
Design for Replicated Data Sites
Avoid Node-Specific Information
 Obtain Enough IP Addresses
Avoid Using SPU IDs or MAC Addresses
Assign Unique Names to Applications
Allow Multiple Instances on Same System
 Use uname2 With Care
Bind to a Fixed Port
Bind to Relocatable IP Addresses
 Avoid File Locking
Give Each Application its Own Volume Group
Use Multiple Destinations for SNA Applications
Call bind before connect
 Etc/rc.config.d/nddconf as follows
Usr/sbin/route add net default 128.17.17.1 1 source
Help menu for ndd -h ipstrongesmodel
 Usr/sbin/route delete net default 128.17.17.1 1 source
Restoring Client Connections
 Handling Application Failures
Create Applications to be Failure Tolerant
Be Able to Monitor Applications
 Minimizing Planned Downtime
Reducing Time Needed for Application Upgrades and Patches
Provide for Rolling Upgrades
Do Not Change the Data Layout Between Releases
 Providing Online Application Reconfiguration
Documenting Maintenance Operations
 Integrating HA Applications with Serviceguard
Checklist for Integrating HA Applications
Defining Baseline Application Behavior on a Single System
 Integrating HA Applications in Multiple Systems
 Testing the Cluster
Move it back
 Software Upgrades
Special Considerations for Upgrade to Serviceguard A.11.20
Special Considerations for Upgrade to Serviceguard A.11.19
 Rolling Upgrade
Types of Upgrade
How To Tell when the Cluster Re-formation Is Complete
Rolling Upgrade Using DRD
 Non-Rolling Upgrade
Guidelines for Rolling Upgrade
Restrictions for DRD Upgrades
Non-Rolling Upgrade Using DRD
 Performing a Rolling Upgrade
Limitations of Rolling Upgrades
 Running the Rolling Upgrade
Keeping Kernels Consistent
Migrating cmclnodelist entries from A.11.15 or earlier
 Performing a Rolling Upgrade Using DRD
Running the Rolling Upgrade Using DRD
 Example of a Rolling Upgrade
Step
Halt the first node, as follows
 Running Cluster with Packages Moved to Node
 Node 1 Rejoining the Cluster
Repeat the process on node 2. Halt the node, as follows
 Guidelines for Non-Rolling Upgrade
Performing a Non-Rolling Upgrade
 Performing a Non-Rolling Upgrade Using DRD
Limitations of Non-Rolling Upgrades using DRD
Steps for a Non-Rolling Upgrade Using DRD
 Guidelines for Migrating a Cluster with Cold Install
Checklist for Migration
 Power Supply Worksheet
Blank Planning Worksheets
Worksheet for Hardware Planning
 LVM Volume Group and Physical Volume Worksheet
Quorum Server Worksheet
 Cluster Configuration Worksheet
VxVM Disk Group and Disk Worksheet
 Package Configuration Worksheet
 Package Configuration Worksheet
 Migrating Volume Groups
Migrating from LVM to VxVM Data Storage
Loading VxVM
 Customizing Packages for VxVM
Mntdg0202, respectively
 Removing LVM Volume Groups
Customizing Packages for CVM
Restart the package
 Migrating from Legacy CFS Packages to Modular CFS Packages
 IPv6 Network Support
IPv6 Address Types
Textual Representation of IPv6 Addresses
 IPv4 and IPv6 Compatibility
IPv6 Address Prefix
Unicast Addresses
IPv4 Compatible IPv6 Addresses
 Site-Local Addresses
Aggregatable Global Unicast Addresses
Link-Local Addresses
Multicast Addresses
 Network Configuration Restrictions
 Ndd -get /dev/ip6 ip6nddadsolicitcount
Example Configurations
Local Primary/Standby LAN Patterns
Ndd -set /dev/ip6 ip6nddadsolicitcountn
 Example Configurations
 384 IPv6 Network Support
 Accessing Serviceguard Manager
Using Serviceguard Manager
Before Using HP Serviceguard Manager Setting Up
About the Online Help System
 Scenario 1 Single cluster management
Launching Serviceguard Manager
Accessing Serviceguard Manager
Opt/hpsmh/bin/hpsmh autostart
 System Management Homepage with Serviceguard Manager
 Expand HP Serviceguard, and click on a Serviceguard cluster
Sign
From the left-hand panel, expand Cluster by Type
 Membertimeout
Maximum and Minimum Values for Parameters
 Monitoring Script for Generic Resources
Sample scripts
Launching Monitoring Scripts
 Launching Monitoring Scripts
 Template of a Monitoring Script
 I L I T Y N C T I O N S
 Monitoring Script for Generic Resources
 Template of a Monitoring Script
 Migrating EMS Resources to Generic Resources
 Start the package
Identify the equivalent SFM style resource monitor
 Index
APA
 399
 Cvmactivationcmd
 Firstclusterlockpv
 LAN
 INONLYORINOUT, 69 Inout
 Pollingtarget defined
 Qsaddr
 Servicename
 Vxvmdg