Cisco Systems 78-16845-01 appendix

J-1

Cisco Transport Manager Release 6.0 User Guide

78-16845-01

APPENDIX

J

Troubleshooting

This chapter offers troubleshooting steps to help solve high-level problems while operating CTM or

CTM GateWay. Follow the troubleshooting procedures described in this chapter before contacting Cisco

technical support.

This chapter includes the following troubleshooting information:

•J.1 Overview, page J-1

•J.2 Server Problems, page J-2

•J.3 Client Connectivity Problems, page J-11

•J.4 Client Operational Problems, page J-13

•J.5 Client Debug Messages, page J-20

•J.6 CTM GateWay/TL1 Problems, page J-21

•J.7 CTM GateWay/CORBA Problems, page J-21

•J.8 Cisco CRS-1 and Catalyst 6509 Problems, page J-22

•J.9 XR 12000 Problems, page J-31

•J.10 Problems with MGX Voice Gateway Devices, page J-34

Note For information about troubleshooting CiscoView problems, see I.9 Troubleshooting, page I-6 in

Appendix I, “Using CiscoView to Configure and Monitor ONS15501, ONS 15530, and ONS 15540

NEs.”

J.1 Overview

Troubleshooting involves:

1. Identifying the source of the problem—Which devices, links, interfaces, hosts, or applications have

the problem?

2. Locating the problem on the network—On what VLAN, subnet, or segment is the problem

occurring?

3. Comparing current network performance against an established baseline—Is the performance better

or worse?

4. Finding out when the problem started—When did you first see the problem? Is it recurring?

5. Determining the extent of the problem—How widespread is the problem? Is it getting worse?

Contents

Main APPENDIX J Troubleshooting J.1 Overview J.2 Server Problems J.2.1 Conditions that Affect CTM Server Performance J.2.2 CTM Server Does Not Respond J.2.3 Cannot Connect to the CTM Server J.2.4 NE Connection State Is Listed as Unavailable J.2.5 Launching Tables Results in Database Errors J.2.6 SNMP Traps Are Not Forwarded from NEs J.2.7 Trap Port Is Unavailable J.2.8 Problems with ONS 15530 and ONS 15540 Configuration J.2.9 Performance Monitoring Data Is Not Displayed J.2.10 CTC-Based NE Is Not Discovered J.2.11 CTC-Based NE Is Not Reachable J.2.12 ONS 15501, ONS 15540, or ONS 15530 Is Not Reachable J.2.13 Circuits Are Not Displayed J.2.14 Circuit State for Monitor Circuits Reads Duplicate ID J.2.15 NE Model Type Appears as Unknown J.2.16 Cannot Copy CTC Binary to the CTM Server J.2.17 Memory Backup, Memory Restore, or Software Download Fails J.2.18 Memory Autobackup, Software Commit, or Software Revert Fails J.2.19 The getinfo.sh Script Fails J.2.20 Using the get nestate Command J.3 Client Connectivity Problems J.3.1 Database Is Not Available J.3.2 Database Timeout Occurs J.3.3 Are the CTM Client and the CTM Server Connected? J.3.4 Cannot Log In as Provisioner or Operator J.3.5 Cannot Authenticate User Message Appears J.4 Client Operational Problems J.4.1 Model Index Is Unknown J.4.2 Cannot Delete an NE J.4.3 CTC Fails to Start J.4.4 Added a New Software Version to the Wrong NE J.4.5 Cannot Delete a Subnetwork J.4.6 Cannot Move an NE Between Subnetworks J.4.7 Cannot Schedule Jobs J.4.8 ONS 15800, ONS 15801, or ONS 15808 NE Generates Too Many Alarms J.4.9 Bandwidth-Intensive Operations Are Blocked J.4.10 NE Displays Incorrect Configuration Management Data J.4.11 Cannot Customize the Network Map J.4.12 How Do I Change Color Settings when the CTM Client Runs on a Sun Ultra 5 Workstation? J.4.13 How Do I Collect Thread Dumps? J.4.14 How Do I Enable or Disable the Automatic Refresh Data Action? J.4.15 How Do I Replace the Alarm Interface Panel? J.5 Client Debug Messages J.6 CTM GateWay/TL1 Problems J.7 CTM GateWay/CORBA Problems J.8 Cisco CRS-1 and Catalyst 6509 Problems J.8.1 CRS-1 Communication State Remains Unavailable After Marking the Device as In Service J.8.2 Catalyst 6509 Communication State Remains Unavailable After Marking the Device as In Service J.8.3 CTM Does Not Receive Configuration Changed Notifications from the CRS-1 J.8.4 CTM Does Not Receive Traps from the Catalyst 6509 J.8.5 CTM Does Not Receive syslog Events from the Catalyst 6509 J-25 J.8.6 CTM Does Not Collect Historical PM Data for the CRS-1 on the CRS-1: J.8.7 CRS-1 PM Tables Collect Data at the Wrong Collection Interval J.8.8 Memory Backup or Restore Fails for the CRS-1 J.8.10 For the CRS-1, CTM Does Not Automatically Discover Neighboring NEs J.8.11 Cannot Delete an Out-of-Service NE from CTM J.8.12 Software Version Is Not Reported in the Domain Explorer and the NE Software Table Is Empty J.8.13 Cannot Use Configuration > Telnet/SSH from the CRS-1 NE Explorer J.8.14 How Do I Configure Telnet on the CRS-1? J.8.15 How Do I Configure SSH on the CRS-1? J.8.16 How Do I Create and Grant Certificates for SSL-Configured CRS-1 NEs? J.8.17 How Do I Trigger a linkDown or linkUp Trap from the Catalyst 6509? J.8.18 How Do I View the SNMP Configuration on the Catalyst 6509? J.8.19 How Do I Load a Software Image on the Catalyst 6509? Page J.9 XR 12000 Problems J.9.1 Reporting XR 12000 Issues J.9.2 CTM Cannot Discover an XR 12000 Node J.9.3 XR 12000 Status Is Not Immediately Reflected in the Domain Explorer J.9.4 XR 12000 Alarm and Event Reporting Issues J.9.5 XR 12000 PM Collection Issues J.9.6 XR 12000 BGP and CDP Neighbor Discovery Issues J.9.7 XR 12000 NE Explorer Applications Are Disabled J.10 Problems with MGX Voice Gateway Devices J.10.1 Discovery Mechanism J.10.2 Discovery Issues at Startup J.10.2.1 No Nodes Are Discovered J.10.2.2 Node Name Is Incorrect in the Database J.10.2.3 Node IP Is Incorrect in the Database J.10.2.4 Node Alarm Is Shown Incorrectly in the Database J.10.2.5 Reachable Node Is Shown as Unreachable J.10.2.6 CTM State Is Incorrect J.10.3 Discovery Issues at Runtime Page J.10.4 Equipment Management Problems J.10.4.1 Equipment Management Declares Successful Node Resync when Node Mode Is 3 J.10.4.2 Network Setup and Configuration Required for OOEMC Sync-Up Process J.10.4.3 Node Mode Remains in 1 J.10.4.4 Node Mode Remains in 2 Page J.10.4.5 Node Resync Mode Is 5 Page J.10.4.7 CTM CM GUI Shows Incomplete Connections After Successful Node Resync and Dbroker Sync-Up J.10.4.8 CTM GUI Shows Mismatched Information Between GUI Views and Database Data Page J.10.4.9 CTM Database and Switch Data Inconsistency Issues After Node Provisioning for MGX J.10.4.9.1 Database Table Population Through Traps and SNMP Upload J.10.4.9.2 Switch Data Does Not Match CTM Database Table After Node Provisioning Page Page Page J.10.5 Configuration Center, Chassis View, Diagnostics Center, and Statistics Report Problems J.10.5.1 CTM Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Basics Page J.10.5.2 Basic Issues J.10.5.3 Topology Discovery Issues Page Page J.10.5.4 Network Element Discovery Issues J.10.5.4.1 All the Cards Under a Node Are Missing Page Page J.10.5.5 Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Alarm Issues J.10.5.5.1 Alarm Processing Basics J.10.5.5.2 XML Schema for Alarm Rules Page J.10.5.5.3 Alarm Severity and Object Severity J.10.5.5.4 Severity Shown on Tree View Does Not Match Severity Shown on Platform J.10.5.5.5 Alarm List Shows Alarm that Does Not Exist on Platform J.10.5.5.6 Transient Event Has Disappeared Unexpectedly J.10.5.5.7 Port in Tree View Displays Aggregate Alarm; However, No Children Exist Under Port J.10.6 Chassis View Problems J.10.6.1 Chassis View Basics J.10.6.2 Lines Not Displayed in the Chassis View J.10.6.3 Card Not Displayed in the Chassis View J.10.6.4 DCA/DCB Status Not Displayed on PXM Cards J.10.6.5 Ethernet Status Not Updated on PXM Cards J.10.6.6 HIST, CBRX, and CBTX Status Not Updated in MGX Nodes J.10.6.7 RPM Card Status Not Updated J.10.6.8 RPM Secondary Card Status Is Blue J.10.7 Configuration Center Management J.10.7.1 Configuration Center Framework J.10.7.1.1 Cannot Launch the Configuration Center J.10.7.1.2 Cannot Launch Other Applications from the Configuration Center J.10.7.1.3 An Exception Is Raised when the Configuration Center Is Launched J.10.7.1.4 An Exception Is Raised when the Configuration Center Launches Another Application J.10.7.1.5 Element TabInternal Frame Does Not Launch J.10.7.1.6 Element TabDrag-and-Drop Functionality Does Not Launch An Internal Frame J.10.7.1.7 Element TabCreate, Details, Modify, and Refresh Button Issues J.10.7.1.8 Element ManagementDrag and Drop Within Configuration Center J.10.7.1.9 Cross ApplicationConfiguration Center as Drag Source J.10.7.1.10 Cross ApplicationConfiguration Centers Element Tab as Drop Target J.10.7.1.11 Element TabInternal Frame Displays Incorrect Object or Object Data J.10.7.1.12 Configuration Center's Element Tab Does Not Respond (GUI Is Grayed Out) J.10.7.2 Configuration CenterElement Management J.10.7.2.1 XML Parsing Error J.10.7.2.2 SNMP No Data Error J.10.7.2.3 Details, Create, Delete, and Refresh Buttons Are Not Enabled J.10.7.2.4 SNMP Timeout Error J.10.7.2.5 SNMP Set Error J.10.7.2.6 Object Not Found In Tree View Error J.10.7.2.7 Element Data Inconsistent with Switch J.10.7.2.8 Config Server Reports Error Messages J.10.8 Connection Management Problems J.10.8.1 Configuration Center GUIFramework Issues J.10.8.1.2 Connections TabDrag and Drop Does Not Launch Connection's Internal Frame J.10.8.1.3 Connection List TabCannot Launch Internal Frame to Modify an Existing Connection J.10.8.1.6 Cross ApplicationCannot Launch Other Applications from the Connection List J.10.8.1.7 Cross ApplicationConnection List as Drag Source J.10.8.1.8 Cross ApplicationConnection Tab's Content Pane as Drop Target J.10.8.1.9 Configuration Center's Connection Tab Does Not Respond J.10.8.2 Configuration Center GUICM Server-Reported Errors J-88 J.10.8.2.1 Connection Creation, Modification, Deletion, and Retrieval Errors J.10.8.2.2 Connection Template Creation, Modification, Deletion, and Retrieval Errors J.10.8.3 Cmsvr Errors J.10.8.3.1 End-to-End Type Cannot Be PVC for Connections with PNNI Segment J.10.8.3.2 Unsupported Service Type for the Foresight Connections for PNNI/SPVC Endpoints J.10.8.3.3 CE-CE Connection Port Speed Should Be the Same on Both Endpoints J.10.8.3.4 Cannot Create One-Segment Hybrid J.10.8.3.6 Unsupported Service TypeGeneral node-slot-port-version J.10.8.3.7 Endpoint Is Not Present in One of the Connection Tables During a Modify Operation J.10.8.3.8 CmsvrConnection Diagnostic Issues J.10.8.3.9 TestConn or TestDelay Failed with Switch Error J.10.8.3.10 Connection Up/Down/Reroute Failed J.10.8.3.11 Connection Trace Failures J.10.8.4 CMGRD J.10.8.4.1 High-Level Connection Management Subsystem Architecture J.10.8.4.2 CmgrdSdbroker Errors Page Page J.10.8.4.9 Cmgrd Errors Page Page J.10.8.4.16 CmgrdSwitch Errors Page Page Page J.10.9 Diagnostics Center Problems J.10.9.1 Diagnostics Center Framework J.10.9.1.1 Cannot Launch Diagnostics Center J.10.9.1.2 Cannot Launch Other Applications from the Diagnostics Center J.10.9.1.3 An Exception Is Raised when the Diagnostics Center Is Launched J.10.9.1.4 Double-Click Operation In Diagnostics Center Tree View Does Not Launch an Internal Frame J.10.9.1.5 Drag-and-Drop Functionality Errors Within the Diagnostics Center J.10.9.1.6 Drag and Drop from/to Diagnostics Center to/from Other Applications J.10.9.1.7 Drop Target Displays Incorrect Object or Object Data J.10.9.1.8 Diagnostics Center Does Not Respond (GUI Is Grayed Out) J.10.9.2 Diagnostics Center Server-Specific Issues J.10.9.2.1 XML Parser Error J.10.9.2.2 SNMPCOMM_TIMEOUT Error While Polling MIB Objects J.10.9.2.3 Node Sync-Up Status Are Not Shown in the Diagnostics Center J.10.9.2.4 Node Resync Fails J.10.9.2.5 Polling of Real-Time Counters Fails J.10.9.2.6 Some Real-Time Counters Are Not Shown J.10.9.2.7 Unable to Start, Stop, or Modify BERT Operations J.10.9.2.8 Unable to Start/Stop IMA Link Test Patterns J.10.9.2.9 Cannot Perform Connection Diagnostics J.10.9.2.10 Miscellaneous Issues or Problems in Diagnostics Center J.10.10 Performance Management Collection and Parsing Problems J.10.10.1 PM Collection Issues J.10.10.1.1 pmcollectorGeneric Troubleshooting J.10.10.1.2 PM Collection Fails for the NodeLog File Shows "Node not discovered" J.10.10.1.3 PM Collection Fails for the NodeLog File Shows "Card not discovered" J.10.10.1.4 PM Collection Fails for the NodeLog File Shows "Time not synchronized" J.10.10.1.5 PM Collection Fails for the NodeLog File Shows "Snmp failed" J.10.10.1.6 PM Collection Fails for the NodeLog File Shows "Ftp failed" J.10.10.2 PM Parsing Issues J.10.10.2.1 StatsParserGeneric Troubleshooting J.10.10.2.2 StatsParserFiles Are Parsed But the Files Are Listed in the BadFileList J.10.11 Statistics Report Problems J.10.11.1 Statistics Report J.10.11.2 You Collected Data but You See No Data Available J.10.11.3 You Generated a Report but Do Not See Data for a Long Time J.10.11.4 You See FDNs for Other Entities when You Generate a Report J.10.11.5 You See the Wrong FDN for a Raw Report J.10.12 Service Agent Problems J.10.12.1 RtmProxy J.10.12.1.1 Registration with RtmProxy Failed J.10.12.1.2 Not Receiving Any Traps J.10.12.1.3 Manager Gets Deregistered J.10.12.1.4 Nodal Community Strings Do Not Work J.10.12.2 Audit Trail Logging Mechanism Problems J.10.12.2.1 AuditLogger.conf Usage J.10.12.2.2 Audit Trail Log File Naming Convention J.10.12.2.3 When You Open a Dialog Box in a CTM GUI, There Is No Record in the Audit Trail Log File J.10.13 Miscellaneous Problems J.10.13.1 NTS J.10.13.1.1 Nodes Stay in Mode 1 After ColdStart J.10.13.1.2 Config Change or Provisioning Activity Not Reflected on CTM J.10.13.1.3 How To Interpret NTS Log J.10.13.2 Data Inconsistency J.10.13.2.1 Connection Inconsistency Between the Switch and GUI Page Page J.10.13.2.2 Inconsistent Connection Status J.10.13.2.3 Inconsistent Connection Secondary Status J.10.13.2.4 Incomplete Connections J.10.13.3 CTM FTP Daemon J.10.13.3.1 FTP Daemon Overview J.10.13.3.2 Generic Troubleshooting J.10.13.3.3 FTP Username and Password J.10.13.3.4 cwmftpdFiles Are Not Transferred Due to Wrong Username/Password J.10.13.3.5 cwmftpdFile Not Available On Switch J.10.13.3.6 FTP Sessions in Switch J.10.13.3.7 421 Session Limit Reached J.10.13.3.8 499 Session Limit Reached J.10.13.3.9 Failed to Acquire a Session with a Switch J.10.13.3.10 Failed to Acquire Session After All Retries J.10.13.3.11 General Log Information