Cisco Systems 78-16845-01 appendix SMCARD01SRM.CF

Page 43

Appendix J Troubleshooting

J.10.4 Equipment Management Problems

Note It should not take long to declare node resync failure because of the SNMP request failure. To verify that there is an SNMP failure, check the ooemc and snmpcomm log files.

Note Using ooemc10.6568.log as an example of an ooemc log filename, 10 is the child ID, the 6568 is the ooemc process ID. The child ID is calculated from the following formula: Remainder (NEDBACCSSID / number of ooemc child) + 1. To retrieve the process ID, use the CLI command psg em.

Step 6 Grep RESYNC from the log file. The grep result should include the node ID and node mode. Review the log messages in the log file to determine if an SNMP failure has occurred.

Step 7 Do the following:

a.Check the log file for trap 60903. If you find it, contact the platform team to investigate the bulk file creation failure on the switch.

Very often, the node mode 2 problem is caused by a bulk file creation abort process on the switch. If there are no SNMP request failures, the switch should send traps 60901 and 60902 to ooemc. Trap 60901 indicates that the switch will start bulk file creation, while trap 60902 indicates bulk file creation done. When the ooemc receives a trap 60902, it will start to FTP the bulk file, which is a shelf generic configuration file. This is also the first file to be uploaded in every resync mode (coldstart, warmstart, or periodic resync.) The commonly seen mode 2 problem is that the switch will abort the bulk file creation and send trap 60903 to CTM. CTM will reschedule the next SNMP request unless it exceeds maximum retries.

b.Check the log file for trap 60902.

CARD_01_CC.CF.10. This file is a shelf generic file and is the first file to be uploaded after CTM receives trap 60902 from the switch. The 10 is the node ID. If you find trap 60902 in the log file but no config file has been uploaded after a reasonable amount of time, it is possible that the FTP failed.

To find out if FTP failure has occurred, do the following:

a.Grep “to ftp file" and the config filename, or just "FTP" from the log files.

b.Trace the log messages in the log file.

For a more detailed information on SNMP request failure and FTP failure, check the snmpcomm and cwmftpd log files and the cwmftp.request_log. Search for the filename at the time the error happened in both files. The cwmftp.request_log gives the summary or final result of the FTP operation, and any error is reported. The cwmftp.log shows the FTP operation details.

CTM will upload and parse a set of config files from the switch. The following files are uploaded from the MGX NE, which contains VISM, AXSM, VXSM, SRM, and RPM/RPM-PR cards:

1.CARD_01_CC.CF

2.SM_1_slot#.CF

3.SM_1_slot#.CS

4.SM_CARD_01_slot#.CF

5.SM_CONN_01_slot#.CF

6.SM_ALARM_01_slot#.CF

7.SM_CON_UPDATE_01_slot#.CF

8.SM_CARD_01_SRM.CF

Cisco Transport Manager Release 6.0 User Guide

 

78-16845-01

J-43

 

Image 43
Contents Troubleshooting Conditions that Affect CTM Server Performance Server ProblemsCTM Server Does Not Respond Cannot Connect to the CTM Server NE Connection State Is Listed as Unavailable Snmp Traps Are Not Forwarded from NEs Launching Tables Results in Database ErrorsTrap Port Is Unavailable Performance Monitoring Data Is Not Displayed Problems with ONS 15530 and ONS 15540 ConfigurationCTC-Based NE Is Not Reachable CTC-Based NE Is Not DiscoveredONS 15501, ONS 15540, or ONS 15530 Is Not Reachable Circuit State for Monitor Circuits Reads Duplicate ID Circuits Are Not DisplayedNE Model Type Appears as Unknown Cannot Copy CTC Binary to the CTM Server Memory Backup, Memory Restore, or Software Download FailsGetinfo.sh Script Fails Database Is Not Available Using the get nestate CommandAre the CTM Client and the CTM Server Connected? Database Timeout OccursCannot Log In as Provisioner or Operator Model Index Is Unknown Cannot Authenticate User Message AppearsCTC Fails to Start Cannot Delete an NEAdded a New Software Version to the Wrong NE Cannot Move an NE Between Subnetworks Cannot Delete a SubnetworkCannot Schedule Jobs Bandwidth-Intensive Operations Are Blocked NE Displays Incorrect Configuration Management DataCannot Customize the Network Map Snmptrap How Do I Collect Thread Dumps?Click OK Client Debug Messages How Do I Replace the Alarm Interface Panel?CTM GateWay/TL1 Problems Cisco CRS-1 and Catalyst 6509 Problems CTM Does Not Receive Traps from the Catalyst Cat6509 enable# set snmp trap enable allSyslog CTM Does Not Receive syslog Events from the CatalystCTM Does Not Collect Historical PM Data for the CRS-1 Memory Backup or Restore Fails for the CRS-1 Click NE Service How Do I Configure Telnet on the CRS-1? Cannot Delete an Out-of-Service NE from CTMHow Do I Configure SSH on the CRS-1? How Do I Load a Software Image on the Catalyst 6509? How Do I View the Snmp Configuration on the Catalyst 6509?Output is similar to the following example CTM Cannot Discover an XR 12000 Node Reporting XR 12000 Issues5 XR 12000 PM Collection Issues 4 XR 12000 Alarm and Event Reporting Issues7 XR 12000 NE Explorer Applications Are Disabled 6 XR 12000 BGP and CDP Neighbor Discovery IssuesDiscovery Mechanism Discovery Issues at StartupNo Nodes Are Discovered Node Name Is Incorrect in the Database Output of the switch CLI, selnd, and dbndsNode IP Is Incorrect in the Database Node Alarm Is Shown Incorrectly in the DatabaseCTM State Is Incorrect Reachable Node Is Shown as UnreachableDiscovery Issues at Runtime Collect the output of the switch CLI, selnd, and dbnds Equipment Management Problems Node Mode Remains Ls -ltr *.9 SMCARD01SRM.CF SMCARD01RPM.CF Node Resync Mode IsEMCSnmpFunccProcFuncGenNodeBulkFile1 entering Appendix J Troubleshooting Equipment Management Problems Appendix J Troubleshooting Equipment Management Problems Appendix J Troubleshooting Equipment Management Problems Database Table Population Through Traps and Snmp Upload Appendix J Troubleshooting Equipment Management Problems Appendix J Troubleshooting Equipment Management Problems You should see something like the following Figure J-1shows the end-to-end architecture Figure J-1 CTM End-to-End Architecture Basic Issues Topology Discovery Issues Appendix J Troubleshooting CTM CLI All the Cards Under a Node Are Missing Network Element Discovery IssuesCisco Transport Manager Release 6.0 User Guide 78-16845-01 Appendix J Troubleshooting Alarm Processing Basics Figure J-2 Client/Server Architecture for Alarm Components XML Schema for Alarm RulesFigure J-3 Correlated Alarm Rule Diagram Figure J-5 Transient Alarm Rule Diagram Alarm Severity and Object SeverityCisco Transport Manager Release 6.0 User Guide 78-16845-01 Alarm List Shows Alarm that Does Not Exist on Platform Transient Event Has Disappeared Unexpectedly Chassis View Basics Chassis View ProblemsCard Not Displayed in the Chassis View Lines Not Displayed in the Chassis View10.6.4 DCA/DCB Status Not Displayed on PXM Cards Ethernet Status Not Updated on PXM CardsHIST, CBRX, and Cbtx Status Not Updated in MGX Nodes RPM Card Status Not UpdatedWrong Tooltip Is Displayed Configuration Center ManagementConfiguration Center Framework Lines Not SelectableChoose Tools Configuration Center from any application Cannot Launch the Configuration CenterAppendix J Troubleshooting Configuration Center Management Element Tab-Internal Frame Does Not Launch Element Management-Drag and Drop Within Configuration CenterCross Application-Configuration Center as Drag Source Appendix J Troubleshooting Configuration Center Management XML Parsing Error Configuration Center-Element ManagementDetails, Create, Delete, and Refresh Buttons Are Not Enabled Snmp No Data ErrorSnmp Set Error Snmp Timeout ErrorObject Not Found In Tree View Error Element Data Inconsistent with Switch Configuration Center GUI-Framework Issues Connection Management ProblemsConfig Server Reports Error Messages Appendix J Troubleshooting Connection Management Problems Appendix J Troubleshooting Connection Management Problems Cross Application-Connection List as Drag Source Configuration Centers Connection Tab Does Not Respond Configuration Center GUI-CM Server-Reported ErrorsDATABASEERRORDatabase error %s Appendix J Troubleshooting Connection Management Problems Cmsvr Errors Cannot Create One-Segment Hybrid Unsupported Service Type-General node-slot-port-version TestConn or TestDelay Failed with Switch Error Cmsvr-Connection Diagnostic IssuesConnection Trace Failures Connection Up/Down/Reroute FailedCmgrd Figure J-6 Connection Management Subsystem High-Level Connection Management Subsystem ArchitectureCmgrd-Sdbroker Errors Related key index entries cmgrd, sdbroker, vpi/vci Appendix J Troubleshooting Connection Management Problems 100 Cmgrd Errors101 102 103 Cmgrd-Switch Errors104 105 106 107 Diagnostics Center Problems108 Diagnostics Center Framework109 Cannot Launch Diagnostics Center110 Drop Target Displays Incorrect Object or Object Data Diagnostics Center Does Not Respond GUI Is Grayed Out111 112 Diagnostics Center Server-Specific IssuesSnmpcommtimeout Error While Polling MIB Objects XML Parser Error113 114 Node Sync-Up Status Are Not Shown in the Diagnostics CenterNode Resync Fails Polling of Real-Time Counters FailsUnable to Start, Stop, or Modify Bert Operations Some Real-Time Counters Are Not Shown115 Unable to Start/Stop IMA Link Test Patterns Cannot Perform Connection Diagnostics116 Pmcollector-Generic Troubleshooting Performance Management Collection and Parsing ProblemsPM Collection Issues Miscellaneous Issues or Problems in Diagnostics Center118 119 PM Collection Fails for the Node-Log File Shows Snmp failed120 PM Collection Fails for the Node-Log File Shows Ftp failed121 Statistics Report ProblemsPM Parsing Issues StatsParser-Generic TroubleshootingYou Generated a Report but Do Not See Data for a Long Time You See FDNs for Other Entities when You Generate a ReportStatistics Report You Collected Data but You See No Data Available123 Service Agent ProblemsUtilization Report Value Is Greater than 100% RtmProxyNot Receiving Any Traps Registration with RtmProxy Failed124 125 Audit Trail Logging Mechanism ProblemsManager Gets Deregistered Nodal Community Strings Do Not Work126 Miscellaneous ProblemsAuditLogger.conf Usage Audit Trail Log File Naming Convention127 Nodes Stay in Mode 1 After ColdStartConfig Change or Provisioning Activity Not Reflected on CTM 10.13.1 NTS128 How To Interpret NTS LogData Inconsistency Connection Inconsistency Between the Switch and GUI129 130 Each output line is similar to this131 Process ID of the sdbroker will be returnedInconsistent Connection Secondary Status Inconsistent Connection Status132 133 Incomplete ConnectionsFTP Daemon Overview Generic TroubleshootingFTP Username and Password CTM FTP DaemonFTP Sessions in Switch Cwmftpd-File Not Available On Switch135 10.13.3.8 499 Session Limit Reached 10.13.3.7 421 Session Limit Reached136 137 Failed to Acquire a Session with a SwitchFailed to Acquire Session After All Retries General Log Information138
Related manuals
Manual 68 pages 52.71 Kb