Cisco Systems 78-16845-01 J.2 Server Problems

J-2

Cisco Transport Manager Release 6.0 User Guide

78-16845-01

AppendixJ Troubleshooting

J.2 Server Problems

Note This chapter assumes that the server is installed under the default /opt/CiscoTransportManagerServer

directory and the client is installed under the default /opt/CiscoTransportManagerClient (Solaris) or

C:\Cisco\TransportManagerClient directory (Windows). If a directory other than the default installation

directory is specified, replace the default path with the installed path.

J.2 Server Problems

This section describes troubleshooting procedures for CTM server-related problems.

Note Log in as the root user on the Solaris workstation where the CTM server is installed to perform any

operations on the Solaris workstation.

J.2.1 Conditions that Affect CTM Server Performance

The “System Requirements” chapter in the Cisco Transport Manager Installation Guide lists server,

CPU, CPU speed, disk space, and RAM requirements for different types of CTM installations. These

requirements are derived from guidelines based on scalability simulation testing.

Actual CTM server performance varies for each customer and is affected by:

•The total number of NEs actively managed by CTM

•The number of NE-related services (including northbound services) running on the server

•The rate of circuit provisioning and the methods used for provisioning

•The rate of network growth (for example, adding 50 NEs to CTM at once)

•The rate of configuration change updates received from the NEs

•The number of circuits provisioned with no signals that generate threshold crossing alerts (TCAs)

•The number of unacknowledged alarms in the CTM database

•The rate of alarm bursts received from the NEs

•The actual number of circuits provisioned in the CTM database

The following conditions might indicate that your CTM installation is near or at capacity:

•Your daily CPU utilization is consistently above 80%

•You experience loss of connectivity (LOC) to 1% to 5% of nodes at random

•Your RAM usage is 100% or your system swaps frequently

•You receive many configuration updates from the NEs

•Your system experiences high Java garbage collection time

Note The Java garbage collection time is determined by engineering after analyzing your CPU

utilization data.

Contents

Main APPENDIX J Troubleshooting J.1 Overview J.2 Server Problems J.2.1 Conditions that Affect CTM Server Performance J.2.2 CTM Server Does Not Respond J.2.3 Cannot Connect to the CTM Server J.2.4 NE Connection State Is Listed as Unavailable J.2.5 Launching Tables Results in Database Errors J.2.6 SNMP Traps Are Not Forwarded from NEs J.2.7 Trap Port Is Unavailable J.2.8 Problems with ONS 15530 and ONS 15540 Configuration J.2.9 Performance Monitoring Data Is Not Displayed J.2.10 CTC-Based NE Is Not Discovered J.2.11 CTC-Based NE Is Not Reachable J.2.12 ONS 15501, ONS 15540, or ONS 15530 Is Not Reachable J.2.13 Circuits Are Not Displayed J.2.14 Circuit State for Monitor Circuits Reads Duplicate ID J.2.15 NE Model Type Appears as Unknown J.2.16 Cannot Copy CTC Binary to the CTM Server J.2.17 Memory Backup, Memory Restore, or Software Download Fails J.2.18 Memory Autobackup, Software Commit, or Software Revert Fails J.2.19 The getinfo.sh Script Fails J.2.20 Using the get nestate Command J.3 Client Connectivity Problems J.3.1 Database Is Not Available J.3.2 Database Timeout Occurs J.3.3 Are the CTM Client and the CTM Server Connected? J.3.4 Cannot Log In as Provisioner or Operator J.3.5 Cannot Authenticate User Message Appears J.4 Client Operational Problems J.4.1 Model Index Is Unknown J.4.2 Cannot Delete an NE J.4.3 CTC Fails to Start J.4.4 Added a New Software Version to the Wrong NE J.4.5 Cannot Delete a Subnetwork J.4.6 Cannot Move an NE Between Subnetworks J.4.7 Cannot Schedule Jobs J.4.8 ONS 15800, ONS 15801, or ONS 15808 NE Generates Too Many Alarms J.4.9 Bandwidth-Intensive Operations Are Blocked J.4.10 NE Displays Incorrect Configuration Management Data J.4.11 Cannot Customize the Network Map J.4.12 How Do I Change Color Settings when the CTM Client Runs on a Sun Ultra 5 Workstation? J.4.13 How Do I Collect Thread Dumps? J.4.14 How Do I Enable or Disable the Automatic Refresh Data Action? J.4.15 How Do I Replace the Alarm Interface Panel? J.5 Client Debug Messages J.6 CTM GateWay/TL1 Problems J.7 CTM GateWay/CORBA Problems J.8 Cisco CRS-1 and Catalyst 6509 Problems J.8.1 CRS-1 Communication State Remains Unavailable After Marking the Device as In Service J.8.2 Catalyst 6509 Communication State Remains Unavailable After Marking the Device as In Service J.8.3 CTM Does Not Receive Configuration Changed Notifications from the CRS-1 J.8.4 CTM Does Not Receive Traps from the Catalyst 6509 J.8.5 CTM Does Not Receive syslog Events from the Catalyst 6509 J-25 J.8.6 CTM Does Not Collect Historical PM Data for the CRS-1 on the CRS-1: J.8.7 CRS-1 PM Tables Collect Data at the Wrong Collection Interval J.8.8 Memory Backup or Restore Fails for the CRS-1 J.8.10 For the CRS-1, CTM Does Not Automatically Discover Neighboring NEs J.8.11 Cannot Delete an Out-of-Service NE from CTM J.8.12 Software Version Is Not Reported in the Domain Explorer and the NE Software Table Is Empty J.8.13 Cannot Use Configuration > Telnet/SSH from the CRS-1 NE Explorer J.8.14 How Do I Configure Telnet on the CRS-1? J.8.15 How Do I Configure SSH on the CRS-1? J.8.16 How Do I Create and Grant Certificates for SSL-Configured CRS-1 NEs? J.8.17 How Do I Trigger a linkDown or linkUp Trap from the Catalyst 6509? J.8.18 How Do I View the SNMP Configuration on the Catalyst 6509? J.8.19 How Do I Load a Software Image on the Catalyst 6509? Page J.9 XR 12000 Problems J.9.1 Reporting XR 12000 Issues J.9.2 CTM Cannot Discover an XR 12000 Node J.9.3 XR 12000 Status Is Not Immediately Reflected in the Domain Explorer J.9.4 XR 12000 Alarm and Event Reporting Issues J.9.5 XR 12000 PM Collection Issues J.9.6 XR 12000 BGP and CDP Neighbor Discovery Issues J.9.7 XR 12000 NE Explorer Applications Are Disabled J.10 Problems with MGX Voice Gateway Devices J.10.1 Discovery Mechanism J.10.2 Discovery Issues at Startup J.10.2.1 No Nodes Are Discovered J.10.2.2 Node Name Is Incorrect in the Database J.10.2.3 Node IP Is Incorrect in the Database J.10.2.4 Node Alarm Is Shown Incorrectly in the Database J.10.2.5 Reachable Node Is Shown as Unreachable J.10.2.6 CTM State Is Incorrect J.10.3 Discovery Issues at Runtime Page J.10.4 Equipment Management Problems J.10.4.1 Equipment Management Declares Successful Node Resync when Node Mode Is 3 J.10.4.2 Network Setup and Configuration Required for OOEMC Sync-Up Process J.10.4.3 Node Mode Remains in 1 J.10.4.4 Node Mode Remains in 2 Page J.10.4.5 Node Resync Mode Is 5 Page J.10.4.7 CTM CM GUI Shows Incomplete Connections After Successful Node Resync and Dbroker Sync-Up J.10.4.8 CTM GUI Shows Mismatched Information Between GUI Views and Database Data Page J.10.4.9 CTM Database and Switch Data Inconsistency Issues After Node Provisioning for MGX J.10.4.9.1 Database Table Population Through Traps and SNMP Upload J.10.4.9.2 Switch Data Does Not Match CTM Database Table After Node Provisioning Page Page Page J.10.5 Configuration Center, Chassis View, Diagnostics Center, and Statistics Report Problems J.10.5.1 CTM Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Basics Page J.10.5.2 Basic Issues J.10.5.3 Topology Discovery Issues Page Page J.10.5.4 Network Element Discovery Issues J.10.5.4.1 All the Cards Under a Node Are Missing Page Page J.10.5.5 Configuration Center, Chassis View, Diagnostics Center, or Statistics Report Alarm Issues J.10.5.5.1 Alarm Processing Basics J.10.5.5.2 XML Schema for Alarm Rules Page J.10.5.5.3 Alarm Severity and Object Severity J.10.5.5.4 Severity Shown on Tree View Does Not Match Severity Shown on Platform J.10.5.5.5 Alarm List Shows Alarm that Does Not Exist on Platform J.10.5.5.6 Transient Event Has Disappeared Unexpectedly J.10.5.5.7 Port in Tree View Displays Aggregate Alarm; However, No Children Exist Under Port J.10.6 Chassis View Problems J.10.6.1 Chassis View Basics J.10.6.2 Lines Not Displayed in the Chassis View J.10.6.3 Card Not Displayed in the Chassis View J.10.6.4 DCA/DCB Status Not Displayed on PXM Cards J.10.6.5 Ethernet Status Not Updated on PXM Cards J.10.6.6 HIST, CBRX, and CBTX Status Not Updated in MGX Nodes J.10.6.7 RPM Card Status Not Updated J.10.6.8 RPM Secondary Card Status Is Blue J.10.7 Configuration Center Management J.10.7.1 Configuration Center Framework J.10.7.1.1 Cannot Launch the Configuration Center J.10.7.1.2 Cannot Launch Other Applications from the Configuration Center J.10.7.1.3 An Exception Is Raised when the Configuration Center Is Launched J.10.7.1.4 An Exception Is Raised when the Configuration Center Launches Another Application J.10.7.1.5 Element TabInternal Frame Does Not Launch J.10.7.1.6 Element TabDrag-and-Drop Functionality Does Not Launch An Internal Frame J.10.7.1.7 Element TabCreate, Details, Modify, and Refresh Button Issues J.10.7.1.8 Element ManagementDrag and Drop Within Configuration Center J.10.7.1.9 Cross ApplicationConfiguration Center as Drag Source J.10.7.1.10 Cross ApplicationConfiguration Centers Element Tab as Drop Target J.10.7.1.11 Element TabInternal Frame Displays Incorrect Object or Object Data J.10.7.1.12 Configuration Center's Element Tab Does Not Respond (GUI Is Grayed Out) J.10.7.2 Configuration CenterElement Management J.10.7.2.1 XML Parsing Error J.10.7.2.2 SNMP No Data Error J.10.7.2.3 Details, Create, Delete, and Refresh Buttons Are Not Enabled J.10.7.2.4 SNMP Timeout Error J.10.7.2.5 SNMP Set Error J.10.7.2.6 Object Not Found In Tree View Error J.10.7.2.7 Element Data Inconsistent with Switch J.10.7.2.8 Config Server Reports Error Messages J.10.8 Connection Management Problems J.10.8.1 Configuration Center GUIFramework Issues J.10.8.1.2 Connections TabDrag and Drop Does Not Launch Connection's Internal Frame J.10.8.1.3 Connection List TabCannot Launch Internal Frame to Modify an Existing Connection J.10.8.1.6 Cross ApplicationCannot Launch Other Applications from the Connection List J.10.8.1.7 Cross ApplicationConnection List as Drag Source J.10.8.1.8 Cross ApplicationConnection Tab's Content Pane as Drop Target J.10.8.1.9 Configuration Center's Connection Tab Does Not Respond J.10.8.2 Configuration Center GUICM Server-Reported Errors J-88 J.10.8.2.1 Connection Creation, Modification, Deletion, and Retrieval Errors J.10.8.2.2 Connection Template Creation, Modification, Deletion, and Retrieval Errors J.10.8.3 Cmsvr Errors J.10.8.3.1 End-to-End Type Cannot Be PVC for Connections with PNNI Segment J.10.8.3.2 Unsupported Service Type for the Foresight Connections for PNNI/SPVC Endpoints J.10.8.3.3 CE-CE Connection Port Speed Should Be the Same on Both Endpoints J.10.8.3.4 Cannot Create One-Segment Hybrid J.10.8.3.6 Unsupported Service TypeGeneral node-slot-port-version J.10.8.3.7 Endpoint Is Not Present in One of the Connection Tables During a Modify Operation J.10.8.3.8 CmsvrConnection Diagnostic Issues J.10.8.3.9 TestConn or TestDelay Failed with Switch Error J.10.8.3.10 Connection Up/Down/Reroute Failed J.10.8.3.11 Connection Trace Failures J.10.8.4 CMGRD J.10.8.4.1 High-Level Connection Management Subsystem Architecture J.10.8.4.2 CmgrdSdbroker Errors Page Page J.10.8.4.9 Cmgrd Errors Page Page J.10.8.4.16 CmgrdSwitch Errors Page Page Page J.10.9 Diagnostics Center Problems J.10.9.1 Diagnostics Center Framework J.10.9.1.1 Cannot Launch Diagnostics Center J.10.9.1.2 Cannot Launch Other Applications from the Diagnostics Center J.10.9.1.3 An Exception Is Raised when the Diagnostics Center Is Launched J.10.9.1.4 Double-Click Operation In Diagnostics Center Tree View Does Not Launch an Internal Frame J.10.9.1.5 Drag-and-Drop Functionality Errors Within the Diagnostics Center J.10.9.1.6 Drag and Drop from/to Diagnostics Center to/from Other Applications J.10.9.1.7 Drop Target Displays Incorrect Object or Object Data J.10.9.1.8 Diagnostics Center Does Not Respond (GUI Is Grayed Out) J.10.9.2 Diagnostics Center Server-Specific Issues J.10.9.2.1 XML Parser Error J.10.9.2.2 SNMPCOMM_TIMEOUT Error While Polling MIB Objects J.10.9.2.3 Node Sync-Up Status Are Not Shown in the Diagnostics Center J.10.9.2.4 Node Resync Fails J.10.9.2.5 Polling of Real-Time Counters Fails J.10.9.2.6 Some Real-Time Counters Are Not Shown J.10.9.2.7 Unable to Start, Stop, or Modify BERT Operations J.10.9.2.8 Unable to Start/Stop IMA Link Test Patterns J.10.9.2.9 Cannot Perform Connection Diagnostics J.10.9.2.10 Miscellaneous Issues or Problems in Diagnostics Center J.10.10 Performance Management Collection and Parsing Problems J.10.10.1 PM Collection Issues J.10.10.1.1 pmcollectorGeneric Troubleshooting J.10.10.1.2 PM Collection Fails for the NodeLog File Shows "Node not discovered" J.10.10.1.3 PM Collection Fails for the NodeLog File Shows "Card not discovered" J.10.10.1.4 PM Collection Fails for the NodeLog File Shows "Time not synchronized" J.10.10.1.5 PM Collection Fails for the NodeLog File Shows "Snmp failed" J.10.10.1.6 PM Collection Fails for the NodeLog File Shows "Ftp failed" J.10.10.2 PM Parsing Issues J.10.10.2.1 StatsParserGeneric Troubleshooting J.10.10.2.2 StatsParserFiles Are Parsed But the Files Are Listed in the BadFileList J.10.11 Statistics Report Problems J.10.11.1 Statistics Report J.10.11.2 You Collected Data but You See No Data Available J.10.11.3 You Generated a Report but Do Not See Data for a Long Time J.10.11.4 You See FDNs for Other Entities when You Generate a Report J.10.11.5 You See the Wrong FDN for a Raw Report J.10.12 Service Agent Problems J.10.12.1 RtmProxy J.10.12.1.1 Registration with RtmProxy Failed J.10.12.1.2 Not Receiving Any Traps J.10.12.1.3 Manager Gets Deregistered J.10.12.1.4 Nodal Community Strings Do Not Work J.10.12.2 Audit Trail Logging Mechanism Problems J.10.12.2.1 AuditLogger.conf Usage J.10.12.2.2 Audit Trail Log File Naming Convention J.10.12.2.3 When You Open a Dialog Box in a CTM GUI, There Is No Record in the Audit Trail Log File J.10.13 Miscellaneous Problems J.10.13.1 NTS J.10.13.1.1 Nodes Stay in Mode 1 After ColdStart J.10.13.1.2 Config Change or Provisioning Activity Not Reflected on CTM J.10.13.1.3 How To Interpret NTS Log J.10.13.2 Data Inconsistency J.10.13.2.1 Connection Inconsistency Between the Switch and GUI Page Page J.10.13.2.2 Inconsistent Connection Status J.10.13.2.3 Inconsistent Connection Secondary Status J.10.13.2.4 Incomplete Connections J.10.13.3 CTM FTP Daemon J.10.13.3.1 FTP Daemon Overview J.10.13.3.2 Generic Troubleshooting J.10.13.3.3 FTP Username and Password J.10.13.3.4 cwmftpdFiles Are Not Transferred Due to Wrong Username/Password J.10.13.3.5 cwmftpdFile Not Available On Switch J.10.13.3.6 FTP Sessions in Switch J.10.13.3.7 421 Session Limit Reached J.10.13.3.8 499 Session Limit Reached J.10.13.3.9 Failed to Acquire a Session with a Switch J.10.13.3.10 Failed to Acquire Session After All Retries J.10.13.3.11 General Log Information