Problem-Solving Worksheet | HP NonStop NS guide

Introduction to Integrity NonStop NS-Series	A Problem-Solving Worksheet
Operations

Table 1-1. Problem-Solving Worksheet

Problem Facts

Possible Causes

What?

Where?

When?

Magnitude?

Situation Facts

Escalation Decision

Plan to Verify/Fix

Plan to Prevent and Control

Damage

HP Integrity NonStop NS-Series Operations Guide—529869-005

1-5

Page 29

Image 29

HP NonStop NS manual Problem-Solving Worksheet, Problem Facts Possible Causes, Situation Facts Escalation Decision

Contents

Part Number Published Product Version Supported Release Version Updates RVUsAbstract Document History Part Number Product Version Published Introduction to Integrity NonStop NS-Series Operations HP Integrity NonStop NS-Series Operations Guide Determining Your System Configuration Monitoring EMS Event Messages ServerNet Resources Monitoring and Recovery 10-3 10-210-4 10-5 11-2 11-111-5 11-8 Power Failures Preparation and Recovery 15-20 15-1915-21 15-22 Preventive Maintenance Related Reading Converting Numbers Figures Examples Tables Manual Information What’s New in This ManualNew and Changed Information Document History Xiv Who Should Use This Guide About This Guide Converting Numbers What Is in This Guide ServerNet Cluster Manual Where to Get More Information Notation Conventions Support and Service LibraryHypertext Links General Syntax Notation Interrupts MaxattachAllowsu on OFF Inspect OFF on Saveabend Code Received Notation for MessagesEnter RUN Code Event number = number Subject = first-subject-value Backup UpProc-nametrapped in SQL in SQL file system Operator Change Bar Notation Introduction to Integrity NonStop NS-Series Operations Monitoring the System and Performing Recovery Operations When to Use This SectionUnderstanding the Operational Environment What Are the Operator Tasks? Stopping and Powering Off the System Preparing for and Recovering from Power FailuresPowering On and Starting the System Performing Preventive Maintenance Responding to Spooler Problems Determining the Cause of a Problem a Systematic ApproachProblem-Solving Worksheet Updating Firmware Situation Facts Escalation Decision Problem-Solving WorksheetProblem Facts Possible Causes Task 1 Get the Facts Task 1a Determine the Facts About the ProblemTask 1b Determine the Facts About the Situation Category Questions to Ask Task 2a Identify the Most Likely Cause Task 2 Find and Eliminate the Cause of the Problem Task 3 Escalate the Problem If Necessary Task 2b Fix the Most Probable Cause of the ProblemTask 3a Determine Whether You Need to Escalate the Problem Task 3b Provide Documentation System Consoles Task 4 Prevent Future ProblemsLogging On to an Integrity NonStop Server Opening a Tacl Window Directly From OutsideView Opening a Tacl WindowOpening a Tacl Window From the Low-Level Link Select StartProgramsOutsideView32 Launching OSM Applications Overview of OSM Applications Service Procedures Determining Your System Configuration Integrity NonStop NS16000 Systems Modular Hardware Components Integrity NonStop NS1000 Systems Integrity NonStop NS14000 Systems Terms Used to Describe System Hardware Components Recording Your System ConfigurationDevice System Resource or Object SCF System Naming Conventions Using SCF to Determine Your System ConfigurationSCF Configuration Files SCF Assume WS $L1.#TERM1 Using SCF to Display Subsystem Configuration InformationSave Configuration SCF Listdev Example 2-1. SCF Listdev Command OutputSCF Listdev Listing the Devices on Your System Subsystem Name Logical Name Device Type Description Backup processor number and PIN of the specified deviceSpecified device TCP/IP Subsystem Displaying SCF Configuration Information for SubsystemsSCF Assume Process $ZTCO Displaying Information for the TCP/IP Subsystem $ZTCO Storage Subsystem Kernel SubsystemSCF Assume Process $ZZKRN Displaying Information for the Kernel Subsystem $ZZKRN Info Disk $SYSTEM,OBEYFORM Example 2-2. SCF ADD Disk Command OutputMirrorlocation 11,1,12 Cbpoollen ADD Adapter $ZZLAN.E0154 Location Type G4SA Accesslist ServerNet LAN Systems Access Slsa SubsystemInfo Tape $*,OBEYFORM SCF Assume Process $ZZLAN Additional Subsystems Controlled by SCF WAN SubsystemDisplaying Information for the WAN Subsystem $ZZWAN Subsystem Objects Controlled by SCF page 1 X25AM Subsystem Objects Controlled by SCF page 2 Example 2-3. SCF Info Process Command Output Displaying Configuration Information-SCF ExamplesExample 2-4. SCF Info SAC Command Output Info Proc $ZZKRN.# Example 2-6. SCF Info Line Command Output Example 2-5. SCF Info Process $ZZWAN Command OutputInfo Process $ZZWAN Info Line $line-name, Detail Overview of Monitoring and Recovery Monitoring Tasks Working With a Daily ChecklistFunctions of Monitoring Task Operator’s name Date & time Tools for Checking the Status of System Hardware Monitoring System Components Monitored Using These Resource Tools See ServerNet Cluster 6770 Hardware General Tasks Specific Tasks For More Information, See Daily Tasks ChecklistAdditional Monitoring Tasks Using OSM to Monitor the System Monitoring and Resolving Problems-An ApproachUsing the OSM Service Connection Top-Down Approach OSM Management System Icons Indicate Problems Within Expanding the Tree Pane to Locate the Source of Problems Attributes Tab Using System Status Icons to Monitor Multiple Systems Alarm Summary Dialog Box Using Alarm and Problem Summaries Suppressing Problems and Alarms Using SCF to Monitor the SystemRecovery Operations for Problems Detected by OSM Monitoring Problem Incident Reports Determining Device States Example 3-1. SCF Status Tape Command State Substate Explanation SCF Object StatesSCF Object States page 1 Servicing Special SCF Object States page 2 Automating Routine System Monitoring Example 3-2. System Monitoring Command File Example 3-3. System Monitoring Output File page 1 Example 3-3. System Monitoring Output File page 2 Example 3-3. System Monitoring Output File page 3 Location LED Name Color Function Using the Status LEDs to Monitor the SystemStatus LEDs and Their Functions page 1 FC-AL I/O Status LEDs and Their Functions page 2 Status LEDs and Their Functions page 3 Related ReadingRelated Reading for Monitoring Task Tool For information, see Monitoring EMS Event Messages What Is the Event Management Service EMS?Tools for Monitoring EMS Event Messages ViewPoint on ViewPoint OSM Event ViewerWeb ViewPoint Related Reading for Monitoring EMS Event Messages System Processes Processes Monitoring RecoveryTypes of Processes Generic Processes Processes IOPs Monitoring System Processes Monitoring Processes Monitoring Generic Processes Monitoring IOPsMonitoring the Status of $ZZKRN Monitoring the Status of All Generic Processes CLCI-TACL $CLCI Stopped PID Initiates the operation of a generic process Recovery Operations for ProcessesSupported for the subsystem manager processes Communications Subsystems Communications Subsystems Monitoring and Recovery Local Area Networks LANs and Wide Area Networks WANs Object Connectivity By LAN Service Provider Subsystems Supported Monitoring the Slsa Subsystem Monitoring Communications Subsystems and Their ObjectsMonitoring the Status of an Adapter and Its Components To monitor the status of an adapter SCF Status SAC sac-name SCF Status Adapter $ZZLANSCF Status SAC $ZZLAN.G11123 SCF Status PIF pif-name Monitoring Status for a Swan Concentrator Monitoring the WAN SubsystemSCF Status PIF $ZZLAN.G11123 SCF Status LIF $ZZLAN.L11021A , Detail System displays a listing similar to Monitoring Status for a Data Communications DeviceSCF Status Adapter $ZZWAN SCF Status Device $ZZWAN.#device-name SCF Status Process $ZZWAN.#boot-process To monitor a single WANBoot process, typeMonitoring WAN Processes SCF Status Process $ZZWAN Monitoring the NonStop TCP/IP Subsystem Monitoring CLIPsMonitoring the NonStop TCP/IP Process SCF Listdev Tcpip Monitoring NonStop TCP/IP Subnets Monitoring NonStop TCP/IP RoutesMonitoring Line-Handler Process Status SCF Status Route $ZTCO Info Process $NCP, Lineset SCF Status Line $LHCS6S, Detail Tracing a Communications Line For Information About Refer to Recovery Operations for Communications SubsystemsRelated Reading for Communications Lines and Devices page 1 Related Reading for Communications Lines and Devices page 2 WAN Subsystem Configuration and Management Manual ServerNet Communications Network ServerNet Resources Monitoring and Recovery Integrity NonStop NS16000 System Integrity NonStop NS16000 ServerNet Connectivity Integrity NonStop NS14000 System with Ioam Enclosure Integrity NonStop NS14000 ServerNet Connectivity Integrity NonStop NS1000 ServerNet Connectivity System I/O ServerNet ConnectionsMonitoring the Status of the ServerNet Fabrics Monitoring the ServerNet Fabrics Using OSM SCF Status Servernet $ZSNET Monitoring the ServerNet Fabrics Using SCF Normal ServerNet Fabric States Identifying ServerNet Fabric Problems Recovery Operations for a Down Disk Due to a Fabric Failure Recovery Operations for the ServerNet FabricsRecovery Operations for a Down Path Between Processors Recovery Operations for a Down Processor Adapters and Modules Monitoring and Recovery Gigabit Ethernet 4-Port Adapter G4SA Adapters and ModulesFibre Channel ServerNet Adapter Fcsa Port ServerNet Extender 4PSE Monitoring I/O Adapters and Modules State Description Monitoring the FCSAsSCF Status Adapter $ZZSTO.#FCSA*, Detail SCF Status Adapter $ZZLAN.G1123 Monitoring the G4SAs Service, Device, and Enabled States for the G4SA page 1 Monitoring the 4PSEs Recovery Operations for I/O Adapters and Modules ServerNet/DA Manual Related Reading for I/O Adapters and Modules Processors and Components Monitoring and Recovery Overview of the NonStop Blade Complex LSU CPU Monitoring and Maintaining Processors Monitoring Processors Automatically Using TfdsSummary, these terms describe the Nsaa processor Term Description Processor Status Display Monitoring Processor Status Using the OSM Low-Level Link OSM Representation of Processor Complex Monitoring Processor Performance Using ViewSys Identifying Processor ProblemsProcessor or System Hangs Viewsys Processor Halts OSM Alarms and Attribute ValuesHalt code = %nnnnnn Freeze code = %nnnnnn Recovery Operations for a Processor Halt Recovery Operations for Processors Select Processor ActionsHalt Click Perform action Halting One or More ProcessorsReloading a Single Processor on a Running Server Reload / run-option , run-option Using Tacl Reload to Perform ReloadSelect FileStart Terminal Emulator Noswitch Noswitch Primenoprime fabric Omitblade ABCPrimenoprime Fabric Select Reload, click Perform action Using the OSM Service Connection to Perform Reload Recovery Operations for a System Hang Dumping a Processor to Disk Freezing the System and Freeze-Enabled ProcessorsEnabling/Disabling Processor and System Freeze See Using Rcvdump to Dump a Processor to Disk on FUP Purgedata dumpfile Using Rcvdump to Dump a Processor to DiskBefore You Begin FUP Info dumpfile Blade Element ReintegrationCPU n has been dumped to dumpfile Submitting Information to Your Service Provider Troubleshooting and Recovery Operations for Disk DumpsBacking Up a Processor Dump to Tape Replacing Processor Memory Other Files to Submit to Your Service Provider Submitting Tapes of Configuration and Operations FilesBackup $tape, CPU0,$SYSTEM.SYS00.CONFTEXT Submitting Tapes of Processor Dumps Additional Information Required by Your Service Provider For Information About Tool See Disk Drives Monitoring and Recovery For information about Overview of Disk DrivesInternal Scsi Disk Drives For information about See M8xxx Fibre Channel Disk DrivesEnterprise Storage System ESS Disks Task See Monitoring Disk DrivesMonitoring Disk Drives With OSM Status Disk $*, SUB Magnetic Monitoring Disk Drives With SCF To display the status of the disk $DATA01 Status Disk $DATA09, DetailStatus $DATA01 Status $DATA02-M Status Disk $ To display the status of all disks Status $DATA01, Detail To display the detailed status of the disk $DATA01To display status of all paths for $DATA00 Status Disk $DATA00 Primary and Backup Path States for Disk Drives Monitoring the Use of Space on a Disk VolumeMonitoring the State of Disk Drives Monitoring the Size of Database Files Example Monitoring Disk Configuration and PerformanceTo check the size of the file DATA1.MEMOS FUP Info DATA1.MEMOS, Detail Problems Possible Symptoms Identifying Disk Drive ProblemsPossible Causes of Common Disk Drive Problems These SCF commands control Disk objects Recovery Operations for Disk DrivesCommon Recovery Operations for Disk Drives page 1 Command Description Customer Support Center or your service provider Common Recovery Operations for Disk Drives page 2 Reset Disk $volume Recovery Operations for a Down Disk or Down Disk PathReset Disk $WD8 Start Disk $volume Report such as this one is sent to your home terminal Recovery Operations for a Nearly Full Database FileFUP Alter MEMOS, Maxextents Info MEMOS, Detail 10-16 Overview of Tape Drives Tape Drives Monitoring and Recovery Monitoring Tape Drive Status With OSM Monitoring Tape Drives OSM Monitoring Tape Drives Connected to an Fcsa OSM Monitoring Tape Drives Connected to an IOMF2 Listing similar to this one is sent to your home terminal Monitoring Tape Drive Status With SCFSCF Status Tape $TAPE0 Listing such as this one is sent to your home terminal Mediacom Status Tapedrive $TAPE0 Monitoring Tape Drive Status With MediacomMediacom Status Tapedrive Common Tape Drive Problems Identifying Tape Drive ProblemsSymptom Problem Possible Causes Monitoring the Status of Labeled-Tape Operations Recovery Operations Using the OSM Service Connection Recovery Operations for Tape DrivesPerforming an OSM Action on a Tape Drive Performing an OSM Action on a Multiple Tape Drives Related Reading for Tapes and Tape Drives page 1 Recovery Operations Using SCFSCF Command Description Related Reading for Tapes and Tape Drives page 2 Overview of Printers and Terminals Printers and Terminals Monitoring and Recovery Monitoring Printer Status Monitoring Printer and Collector Process StatusMonitoring Collector Process Status Spoolcom DEV $LASER Recovery Operations for a Full Collector Process Recovery Operations for Printers and Terminals 12-4 TMF States on Applications Monitoring and RecoveryMonitoring TMF Monitoring the Status of TMF Monitoring Data VolumesTmfcom ~ Status TMF Tmfcom responds with output similar to TMF StatesTMF subsystem can be in any of the states listed in Table TMF States page 1 TMF States page 2 Monitoring the Status of PathwayStatus *, Prog $*.*.PATHMON = Status Pathway = Status Pathmon Pathmon StatesPathcom responds with output such as Request is waiting for a RUN Program to finish Request is waiting for an object that has been locked byAnother requester ESS Cabinets on Power Failures Preparation and Recovery NonStop NS-Series Cabinets Modular Cabinets System Response to Power FailuresNonStop S-Series I/O Enclosures External Devices Configure OSM Power Fail Support Preparing for Power FailureESS Cabinets Air Conditioning Monitor Power Supplies Power Failure RecoveryMonitor Batteries Maintain Batteries Setting System Time Procedure to Recover From a Power Failure 14-6 Alerts on Starting and Stopping the System Powering On a System Powering On the System From a No Power State Powering On the System From a Low Power StateSelect Power On System Select Hard Reset Click Perform Action 15-4 Loading the System Starting a SystemAlerts Normal System Load System Load Disks System Load to a Specific Processor System Load Paths in Order of Use System Load Paths for a Normal System LoadDisk Drive Enclosure Path Group Module Slot Data Travels Configuration File Performing a System Load Starting Other System Components Click Start system System Load Dialog Box Performing a System Load From a Specific Processor Reloading Processors Using OSM Reloading Processors Using the Reload CommandReloading Processors Reload 01 15, Prime Logical Processor Reload Parameters Anticipating and Planning for Change Minimizing the Frequency of Planned OutagesStopping Application, Devices, and Processes Volume $DSMSCM.ZDSMSCM = SHUTDOWN2, Mode OrderlyStop DSM/SCM RUN Stopscm Stopping the System Halting All Processors Using OSMSpoolcom supervisor-name, SPOOLER, Drain Tmfcom Stop TMF System Power-Off Using OSM Powering Off a SystemSystem Power-Off Using SCF From the Processors Actions menu, select Halt Fans Are Not Turning Troubleshooting and Recovery OperationsEmergency Power-Off Procedure Green LED Is Not Lit After POSTs Finish System Does Not Appear to Be Powered OnComponents Fail When Testing the Power Info Subsys $ZZKRN Recovering From a System Load Failure Backup $TAPE, $SYSTEM.ZSYSCONF.CONFSAVE, Listall Getting a Corrupt System Configuration File AnalyzedRecovering From a Reload Failure Opening Startup Event Stream and Startup Tacl Windows Exiting the OSM Low-Level Link 15-23 Related Reading for Starting and Stopping a System NonStop NS-Series Hardware Installation Guide Startup on Shutdown on Creating Startup and Shutdown Files Startup Managed Configuration Services MCSAutomating System Startup and Shutdown For More Information Processes That Represent the System ConsoleShutdown $ZHOME Alternative Example Command Files Ciin File Modifying a Ciin File Establishing a Ciin File Ciin File Option Results If a Ciin File Is Not Specified or Enabled in OSMConftext Ciin Entry Reload /TERM $ZHOME, OUT $ZHOME Example Ciin Files Command File Syntax Writing Efficient Startup and Shutdown Command Files= Start Term = Start Term TERM1, TERM2, TERM3, TERM4, TERM5, TERM6 Use Parallel Processing Avoid Manual Intervention Investigate Product-Specific Techniques How Process Persistence Affects Configuration and StartupTips for Startup Files Obey $SYSTEM.STARTUP.STRTSYS Startup File ExamplesSystem Startup File 16-13 Spooler Warm-Start File TCP/IP Stack Configuration and Startup FileTMF Warm-Start File Obey $SYSTEM.STARTUP.SPLWARM 16-15 16-16 Lines Startup File CP6100 Lines Startup FileATP6100 Lines Startup File Expand Direct-Connect Line Startup File Printer Line Startup FileExpand-Over-IP Line Startup File Shutdown File Examples Tips for Shutdown Files Obey $SYSTEM.SHUTDOWN.STOPSYS System Shutdown File Lines Shutdown File CP6100 Lines Shutdown FileATP6100 Lines Shutdown File Direct-Connect Line Shutdown File Printer Line Shutdown FileExpand-Over-IP Line Shutdown File TMF Shutdown File Spooler Shutdown FileObey $SYSTEM.SHUTDOWN.SPLDRAIN Tmfcom / in $SYSTEM.SHUTDOWN.TMFSTOP, OUT $ZHOME 16-24 Monitoring Physical Facilities Preventive MaintenanceChecking Air Temperature and Humidity Cleaning System Components Handling and Storing Cartridge Tapes Cleaning Tape Drives 17-4 HP Integrity NonStop NS-Series Operations Guide-529869-005 Page Tools and Utilities for Operations When to Use This Appendix Disk Space Analysis Program Dsap Event Management Service Analyzer EmsaDisk Compression Program Dcom Measure File Utility Program FUPNonStop NET/MASTER Nskcom and the Kernel-Managed Swap Facility Kmsf Subsystem Control Facility SCF Pathcom ViewPoint HP Tandem Advanced Command Language TaclWeb ViewPoint ViewSys Tool Documentation Description Related ReadingTable C-1. Related Reading for Tools and Utilities page 1 NET/MASTER MS Table C-1. Related Reading for Tools and Utilities page 2 Management Manual Table C-1. Related Reading for Tools and Utilities page 3 Table C-1. Related Reading for Tools and Utilities page 4 Recovery Guide Output Table C-1. Related Reading for Tools and Utilities page 5Page Converting Numbers Number System Base Description Overview of Numbering SystemsTable D-1. Descriptions of Number Systems Binary Value Decimal Value Binary to Decimal Octal Value Decimal Value 1375 765 Octal to Decimal Hexadecimal Decimal Hexadecimal to Decimal Hexadecimal Value Decimal Value HBA10 47632 Figure D-3. Hexadecimal to Decimal Conversion Result is Decimal to BinaryStep Division Quotient Remainder Decimal Value Binary Value 354 B101100010 Decimal Value Octal Value Decimal to OctalStep Division Quotient Decimal Value Hexadecimal Value Decimal to HexadecimalDecimal Hexadecimal Page Regulatory Compliance Statements Safety and ComplianceFCC Compliance Canadian Compliance Statements-2 Laser Compliance European Union Notice Safety Caution Waste Electrical and Electronic Equipment Weee Important Safety Information Statements-6 Numbers IndexPort ServerNet Extender 4PSE See Conftext file Dcom 10-15,B-2 FUP Nskcom B-3 Index-5 Spooler 16-14 Startup files SACs SCF B-4 commandsSystem shutdown file 16-20 TMF Lines Tacl 9-22,16-5,B-5 Special Characters Index-8