HP UX System Adstration manual Minimizing Message Loss During Failover

Page 53

added and deleted, the DSAU consolidated logging tools will automatically take the appropriate configuration actions. Specifically:

When adding a member to the cluster, the new member is automatically configured to participate in log consolidation according to the cluster’s configuration. The following files are automatically configured on the added member:

/etc/rc.config.d/syslog-ng

/etc/rc.config.d/syslogd

/etc/syslog.conf

/etc/syslog-ng.conf.client, /etc/syslog-ng.conf.server, and the /etc/ syslog-ng.conf symbolic link

/etc/services

When deleting a member from a cluster:

The member is still configured as a log-forwarding client and will continue to forward syslog messages to the cluster if that option had been chosen during the initial run of the clog_wizard. If the system should no longer forward log messages to the cluster, rerun the wizard to configure the system to forward to a different consolidator, or disable log consolidation entirely. Refer to “Disabling Log Consolidation” (page 75) for additional information.

The package logs on the deleted member are still monitored until a reboot. Since this member is no longer part of the cluster, the package logs will not be active.

When adding or deleting a package, the following automated actions occur:

The package is added to or deleted from /etc/syslog-ng.conf.servercluster-wide. There is a reserved section of these files dedicated for use by the DSAU tools. The configuration stanzas added in this section direct syslog-ngto filter package log messages into the appropriate consolidated package logs.

The clog_tail log monitor adds or deletes the package log file from its list of files to monitor.

3.3.1.5Minimizing Message Loss During Failover

When there is a failure on the adoptive node, it takes a finite amount of time for the clog package to fail over to another cluster member. The longer this failover time, the more likely that messages could be lost from the consolidated log. Use the following guidelines to minimize message loss during failover.

Configure clients to use the TCP transport instead of the UDP transport. UDP messages will be lost unconditionally when the package is down. The TCP protocol contains retry mechanisms, congestion control, and so on, that help minimize message loss.

syslog-ng can buffer TCP messages on the client side. The number of messages buffered is controlled by the syslog-ng log_fifo_size setting. This sets an upper limit on the number of messages that can be buffered. The default/etc/syslog-ng.conf file sets log_fifo_size to 10000.

syslog-nghas a time_reopen() option to configure the time to wait before a dead connection is reestablished. The /etc/syslog-ng.conffile has time_reopen() set to 10 seconds.

Serviceguard offers various configuration options to improve failover times such as HEARTBEAT_INTERVAL and NODE_TIMEOUT. Serviceguard Extension for Faster Failover (SGeFF) is also available to optimize failover times for two-node clusters. Since syslog-ngitself starts quickly, SGeFF is an ideal candidate for improving failover times and minimizing message loss.

3.3 Log Consolidation Configuration

53

Image 53
Contents Distributed Systems Administration Utilities Users Guide Copyright 2009 Hewlett-Packard Development Company, L.P Table of Contents HP-Supported Open Source pdsh Options Index List of Figures Syslog-ngLog-Forwarding ConfigurationList of Tables Consolidated Logging CommandsTarget Node Error Messages Intended Audience About this DocumentTypographic Conventions Related InformationProduct Support HP Encourages Your Comments Introduction Configuration Synchronization Command Distributed Systems Administration Utilities CommandsConsolidated Logging Commands Command Fanout CommandsOpen Source cfengine Commands Utility Setup CommandOpen Source pdsh Commands Open Source ComponentsDsau Manual Page Sections Distributed Systems Administration Utilities Manual PagesOpen Source syslog-ng Command Cfengine Overview Configuration SynchronizationConfiguration Synchronization Cfengine Daemons and CommandsCfengine Overview Cfengine Master Server Deployment ModelsUsing the Configuration Synchronization Wizard Configuring cfengineWizard displays the following introductory screen Configuration Data for csyncwizard# /opt/dsau/sbin/csyncwizard Wizard proceeds to configure the system as a master server # /opt/dsau/sbin/csyncwizard Configuration Synchronization Configuring cfengine Would you like to manage clients? N Serviceguard Automation Features Cluster Configuration Notes for cfengineVar/opt/dsau/cfengine/inputs directory Opt/dsau/bin/csyncdispatcher Memberadded newhost Using the Wizard to Configure a Synchronization ClientWhen prompted, enter the name of the client to add Manual Configuration# mkdir -p /var/opt/dsau/cfenginemaster/inputs Manually Configuring a Standalone Synchronization ServerStart by creating the directory # cp localhost.pub root-10.0.0.5.pub # /opt/dsau/sbin/cfkey # /var/opt/dsau/cfengine/ppkeys# cfagent --no-lock --verbose --no-splay # /sbin/init.d/cfservd start# cfrun -- --inform # cfrun -v -- --verbose# mkdir -p /csync/dsau/cfenginemaster/masterfiles Initial Serviceguard Package PreparationList Managed Clients in cfrun.hosts Policyhost = csync.abc.xyz.com# /opt/dsau/sbin/cfkey Edit the cfservd.conf File# cp localhost.pub root-192.10.25.12.pub # ccp /etc/rc.config.d/cfservd /etc/rc.config.d/cfservd# ccp * /var/opt/dsau/cfengine/ppkeys # cexec /sbin/init.d/cfservd start# ccp csync csync.conf /etc/cmcluster/csync Test the configuration by performing the following stepsApply the package and start it # cmapplyconf -P csync.conf # cmmodpkg -e csyncOn a managed client, use the command Configuring a Synchronization Managed ClientChoosing a Synchronization Invocation Method Security NotesCsync Network Port Usage Key ExchangeEncryption Encryption Checksum alertsLogging Options Disabling Use of cfengineChecksum Alerts # /sbin/init.d/cfservd stopSyntax error due to missing or superfluous spaces Cfengine Troubleshooting#cfagent -K Unable to connect to a cfengine client or masterCfagent -d, -d1, -d2, or -d3 cfservd Cfrun Introduction to syslog Consolidated LoggingSyslog Message Format 2describes syslog Facilities MessagesImproved Log Consolidation Log Consolidation OverviewMessage Filtering Syslog Co-existence Etc/cmcluster/package-name/package-name.log Syslog-ng Log Consolidator Configuration Log Consolidation ConfigurationOpt/dsau/sbin/clogwizard Using the Log Consolidation WizardConfiguration Data for clogwizard Answer yes y. The wizard then prompts Where N is the expected number of clientsAnswer yes y or press Enter. The next question is If these choices are correct, continue Next prompt is Log files that reside on this cluster can be consolidated Consolidated package logs would be located here Cluster Configuration Notes for clog Minimizing Message Loss During Failover Or press Enter. The next question is Configuring a Log Forwarding Client Using clogwizardEnter the ssh port to be used for port forwarding Manually Configuring a Standalone Log Consolidation Server Manually Configuring Log ConsolidationFor example, for TCP # /sbin/init.d/syslogd stop # /sbin/init.d/syslogd startReplace the %UDPLOOPBACKLOG% token with Change the Clogconfigured line to Create the following symbolic linkIf using the TCP protocol, add Add the following linesLog Consolidation Configuration SYSLOGDOPTS=-D -N KEEPALIVE% tokens with appropriate values UDPLOOPBACKSOURCE% and %UDPLOOPBACKLOG% tokens Creating the clog Package If consolidating package logs of this cluster, addIf using VxVM, comment out the LVM Volume Group line Then use cmviewcl to make sure it is running Testing and Starting the clog PackageDistribute it cluster-wide Manually Configuring a Standalone Log Forwarding Client Manually Configuring Log Forwarding ClientsUsing VxVM Instead of LVM Ln -sf /etc/syslog-ng.conf.client /etc/syslog-ng.conf # /sbin/init.d/syslog-ng start # cpp /etc/rc.config.d/syslogd /etc/rc.config.d Destination dsyslog%TYPE% %TYPE%%IP%port%PORT% If using ssh port forwarding, add If using the TCP protocol, add the following linesOtherwise, if using the UDP protocol, add Create the following symbolic link on each cluster memberForwarding Ascii Log Data Start syslog-ngon all cluster members usingFor the filter line For the destination lineFor the log line Consolidating Package Logs on the Log Consolidation ServerDisabling Log Consolidation Perform the following steps to disable log consolidationDisabling a Standalone Log Consolidation System #/sbin/init.d/syslogd stopDisabling a Standalone Log Forwarding Client Disabling a Serviceguard Cluster Log Consolidation System#/sbin/init.d/syslogd start # /sbin/init.d/syslog-ng stop#/sbin/init.d/syslogd stop #/sbin/init.d/syslogd start Disabling a Serviceguard Cluster Log Forwarding ClientSsh Port Forwarding Securing Consolidated LogsLog File Protections # cd /opt/ssh/etc # ccp sshhost* /opt/ssh/etc Using Bastille to Harden the SystemClog Network Port Usage Viewing System and Consolidated Logs Using the System and Consolidated Log ViewerStarting System Management Homepage To log in to the System Management Homepage, navigate toViewing System and Consolidated Logs Page Parallel Distributed Shell Command FanoutAll nodes Pdsh Utility WrappersSystems Cwall displays a wall1M broadcast message on multiple hostsRemote Shell Security Setup Security ConfigurationSsh Security Setup # csshsetup -r -f memberslist.txtSsh Command Messages Command Fanout TroubleshootingRsh Command Messages Target Node Error MessagesHP-Supported Open Source pdsh Options Page Cfanouthosts IndexLVM UDP