IBM 51 220

220 Tivoli Intelligent Orchestrator Problem Determination and Troubleshooting Guide

Contents

Main T l Problem Determination and Troubleshooting Guide Page T l Problem Determination and Troubleshooting Guide ivo Page Contents Preface . . . . . . . . . . . . . . vii Copyright IBM Corp. 2003, 2006 Chapter 1. Introduction to troubleshooting . . . . . . . . . . .1 Chapter 3. Log file types . . . . . . .35 Chapter 8. Troubleshooting the agent manager . . . . . . . . . . . . . .79 iv Chapter 9. Contacting IBM Tivoli Software Support . . . . . . . . . .91 Chapter 10. Tivoli Intelligent Orchestrator messages . . . . . . .95 Chapter 11. Tivoli Common Agent Services messages . . . . . . . . . 179 Contents v Common Agent messages . . . . . . . . . 179 Agent Manager messages . . . . . . . . . 202 Index . . . . . . . . . . . . . . . 231 Appendix A. Common Agent serviceability tool . . . . . . . . . 221 vi Preface Copyright IBM Corp. 2003, 2006 Who should read this book Publications Tivoli Intelligent Orchestrator library Prerequisite publications viii Related publications and resources Accessing publications online Preface ix Ordering publications Accessibility Tivoli software training x Support information Conventions used in this book Typeface conventions Chapter 1. Introduction to troubleshooting Copyright IBM Corp. 2003, 2006 Product and system overview Product comparison 2 The following table summarizes the complementary roles of the two products: Product architecture Tivoli Intelligent Orchestrator includes the following main components: Chapter 1. Introduction to troubleshooting 3 WebSphere Application Server WebSphere Application Server Prerequisites installed with Tivoli Intelligent Orchestrator 4 Built-in troubleshooting features Chapter 1. Introduction to troubleshooting 5 Problem classification Product maintenance Backing up the system Performing periodic checks and maintenance 6 http://www.ibm.com/software/sysmgmt/products/support/ IBMTivoliIntelligentOrchestrator.html Problem resolution Chapter 2. Problem determination essentials for Tivoli Intelligent Orchestrator Copyright IBM Corp. 2003, 2006 What was done differently this time? Problem determination methodologies Defining the problem Identifying the cause of the problem 8 Fixing the problem Chapter 2. Problem determination essentials for Tivoli Intelligent Orchestrator 9 Troubleshooting methods for Tivoli Intelligent Orchestrator and prerequisites Troubleshooting the prerequisite installation 10 2000Solaris DB2 Universal Database verification netstat -an |grep 50000 IBM Tivoli Directory Server verification v UNIX: ibmdirctl -D cn=root -w <password> status 12 ibmdirctl -D cn=root -w <password> start v Stopping the IBM Tivoli Directory Server: ibmdirctl -D cn=root -w <password> stop WebSphere Application Server verification Chapter 2. Problem determination essentials for Tivoli Intelligent Orchestrator 13 Application Server status: serverStatus -all -user <username> -password <password> 14 tail -f SystemOut.log AdminInitiali A ADMN0015I: AdminService initialized Chapter 2. Problem determination essentials for Tivoli Intelligent Orchestrator 15 Secure Authentication Services (LDAP) Configuration A SECJ0215I: Successfully set JAAS login provider SecurityDM I SECJ0231I: The Security components FFDC Diagnostic Module SecurityCompo I SECJ0240I: Security service initialization completed Map Tivoli Intelligent Orchestrator resource references 16 Simple Object Access Protocol ( SOAP) JMXSoapAdapte A ADMC0013I: SOAP connector available at port 8880 Applications TCEAR Application Tivoli Intelligent Orchestrator installation and subcomponent configuration verification Tivoli Intelligent Orchestrator installation Verifying subcomponent configuration list tables 18 <config> </config> 3. ldapsearch -b cn=schema -s base objectclass=* | grep -i think 20 Chapter 2. Problem determination essentials for Tivoli Intelligent Orchestrator 21 Tivoli Intelligent Orchestrator runtime problem determination Component-based problem determination 22 4. Check the tioldap authentication: 6. Check the tioappadmin user: 8. Search the knowledge bases and, if necessary, contact Support. For more Chapter 2. Problem determination essentials for Tivoli Intelligent Orchestrator 23 Task-based problem determination 24 SQL0532N A parent row cannot be deleted Chapter 2. Problem determination essentials for Tivoli Intelligent Orchestrator 25 restricts the deletion. DB2INST1.SPARE_POOL(POOL_ID) IP = 9.3.127.47 SystemException: COPCOM093E The JDBC driver caused an SQL exception.... Accessing the knowledge bases 26 http://www-306.ibm.com/software/sysmgmt/products/support/ IBMTivoliIntelligentOrchestrator.html http://www-306.ibm.com/software/sysmgmt/products/support/ IBMDirectoryServer.html v For the WebSphere Application Server: Note: The command line tools can be found on TIO_HOME/tools 28 Page 30 Page WebSphere Application Server command line tools 32 IBM Tivoli Directory Server command line tools 34 Chapter 3. Log file types Copyright IBM Corp. 2003, 2006 Message logs Message ID format code Message elements 36 Action title Description Administrator Response Chapter 3. Log file types 37 Describes what response a system programmer might be able to take. Trace logs Common Base Event logs 38 Chapter 4. Locating and configuring log files Copyright IBM Corp. 2003, 2006 Log locations Log directory files Logging levels 40 msg_error#com.thinkdynamics.kanaha. util.logging.MessageLevel Extended for message logs msg_warn#com.thinkdynamics.kanaha. util.logging.MessageLevel Extended for message logs msg_info#com.thinkdynamics.kanaha. util.logging.MessageLevel Page Log level defaults 42 RollingFileAppender defaults msg.log v Maximum number of archived log files: 10 cbe.log v Maximum number of archived log files: 10 Preparing log files for review by IBM Tivoli Support Chapter 4. Locating and configuring log files 43 44 Chapter 5. Tivoli Intelligent Orchestrator log files Copyright IBM Corp. 2003, 2006 /var/ibm/tivoli/common/COP/logs Log files for product setup These log files are for installation and data center model initialization. Log files for starting and stopping the server 46 File name Location Description tio_start.log Windows UNIX or Linux $TIO_LOGS UNIX or Linux $TIO_LOGS Log files for the Web interface Deployment engine log files Workflow logs 48 Logs for data collection and decision making Automation package logs Log files for orchestration packages are stored in the %TIO_LOGS%\tcdrivermanager directory. 50 Chapter 6. Middleware prerequisite log files Installation log files for prerequisites Installation logs for individual prerequisites WebSphere Application Server DB2 Universal Database Runtime logs for prerequisites WebSphere Application Server Chapter 6. Middleware prerequisite log files 53 DB2 Universal Database Directory Server v db2diag.log v ibmds_config.log v Tivoli Agent Manager 54 Copyright IBM Corp. 2003, 2006 Problems with installation In Linux, you cannot create the user tioadmin 56 ibmdirctl -D cn=root -w <password> status start: ibmdirctl -D cn=root -w <password> start Installing the common agent and the agent manager is not supported Chapter 7. Common problems and known limitations in Tivoli Intelligent Orchestrator 57 Manually uninstall the common agent. You should also verify whether the required port is available: netstat -an |grep 50000 The Microsoft Active Directory installation fails with invalid certificate value 58 Environment variables not set for user tioadmin Cause Chapter 7. Common problems and known limitations in Tivoli Intelligent Orchestrator 59 Solution Cause Solution 1 Solution 2 60 WAS_install_dir \bin\stopServer app_server_name - On AIX, Linux, and Solaris ADMU4000I: Server app_server_name stop completed. WAS_install_dir /bin/stopServer.sh app_server_name - On AIX, Linux, and Solaris ./RemoveCASTables.sh database_password Agent_Manager_install_dir\_uninst\uninstall.exe java -jar uninstall.jar Or, to uninstall silently: Problems with logging in You cannot log in Cause The LDAP server might be down. Solution 1 You cannot change your password Cause Solution 6. Change your password again based on the specified password syntax. Problems with workflows 64 DB2 Universal Database errors occur when you deploy resources db2 update db cfg for $DB2_DB_NAME using applheapsz 3072 4. Run the following command to terminate all connections: DB2 Universal Database deadlocks occur when the system runs a logical operation Cause Solution Shell command error: Resource temporarily unavailable Solution 66 Syntax for UNIX: <?xml version="1.0" encoding="UTF-8"?> <workflow-execution-history> Chapter 7. Common problems and known limitations in Tivoli Intelligent Orchestrator 67 Note: Any Unicode character can be used to enter comments in workflow scripts. A workflow does not install Cannot run workflows if the PS1 environment variable is altered 68 $ tioadmin> command$ $ # command# On Windows, Tivoli Intelligent Orchestrator does not start Chapter 7. Common problems and known limitations in Tivoli Intelligent Orchestrator 69 Incorrect SOAP parameters generate detailed Java exception messages 70 XML import does not work properly DB2 Universal Database does not work properly with terminal server Chapter 7. Common problems and known limitations in Tivoli Intelligent Orchestrator 71 Disable the terminal server and reboot before trying to install again. On UNIX, Tivoli Intelligent Orchestrator logs fill up the file system The capacity of the file system is not large enough. 72 1b2ddb28 JMSEmbeddedPr A MSGS0054I: Stopping the Broker When you run ps -ef | grep java it returns only one process: stopServer . A problem occurred in the Linux libstdc++ library. Not enough connection managers are allocated from the database server. Chapter 7. Common problems and known limitations in Tivoli Intelligent Orchestrator 73 74 Chapter 7. Common problems and known limitations in Tivoli Intelligent Orchestrator 75 Common agents unable to communicate with the agent manager $WAS_HOME/installedApps/<nodename>/AgentManager.ear/ AgentManager.war/WEB-INF/classes/resources a. Find the line that looks like: ARS.host=<fully qualified name> ARS.host=9.2.2.2 There is a maximum size for returnResult parameter storage Device Manager limitations The following are known limitations for the Device Manager: Agent receives an HTTP Unauthorized (401) response code Chapter 7. Common problems and known limitations in Tivoli Intelligent Orchestrator 77 Known report limitations Running any report with too many objects selected fails COPJEE389E: The report contains too many objects. Retry with fewer selections. 78 Chapter 8. Troubleshooting the agent manager Copyright IBM Corp. 2003, 2006 Log files for the agent manager Packaging the log files Locating the installation log files 80 Chapter 8. Troubleshooting the agent manager 81 Locating the uninstallation log files Locating the remote registry logs Locating the runtime log files 82 Locating the DB2 Universal Database runtime logs Determining the version of agent manager Chapter 8. Troubleshooting the agent manager 83 Agent_Manager_install_dir/GetAMInfo.sh Agent_Manager_install_dir\GetAMInfo.bat AgentManager Agent_Manager_install_dir/GetAMInfo.sh AgentRecoveryService Verifying the installation of WebSphere Application Server 84 Errors starting the agent manager application server Errors starting the agent manager application Verifying the connection to the registry Enabling and disabling tracing Enabling tracing Disabling tracing Recovering from an incomplete uninstallation of the agent manager 86 WAS_install_dir\bin\WASService.exe -remove "Tivoli Agent Manager" am:2345:once:/opt/IBM/WebSphere/AppServer/bin/rc.am >/dev/console 2>&1 amdb:2345:once:su - db2inst1 -c db2start >/dev/console 2>&1 Common problems Registration and communication failures Problems using the WebSphere Administrative Console 88 Cannot start the WebSphere Administrative Console Configuring the WebSphere Administrative Console to use a different port stopServer server1 startServer server1 5. Access the administrative console using the following Web address: http:// hostName :9091/admin 90 Chapter 9. Contacting IBM Tivoli Software Support Copyright IBM Corp. 2003, 2006 General data to collect for Tivoli Support packageLog ZIP tool 92 Location Windows %TIO_HOME%\tools Where $TIO_HOME is the Tivoli Intelligent Orchestrator home directory. Prerequisites Chapter 9. Contacting IBM Tivoli Software Support 93 set JAVA_HOME=C:\IBM\WebSphere\AppServer\java set WAS_HOME=C:\IBM\WebSphere\AppServer Linux and UNIX export JAVA_HOME=/opt/WebSphere/AppServer/java export WAS_HOME=/opt/WebSphere/AppServer 94 Chapter 10. Tivoli Intelligent Orchestrator messages Copyright IBM Corp. 2003, 2006 Common subsystem messages 96 Chapter 10. Tivoli Intelligent Orchestrator messages 97 98 Chapter 10. Tivoli Intelligent Orchestrator messages 99 100 Chapter 10. Tivoli Intelligent Orchestrator messages 101 102 Chapter 10. Tivoli Intelligent Orchestrator messages 103 104 Chapter 10. Tivoli Intelligent Orchestrator messages 105 106 Chapter 10. Tivoli Intelligent Orchestrator messages 107 108 Chapter 10. Tivoli Intelligent Orchestrator messages 109 110 Chapter 10. Tivoli Intelligent Orchestrator messages 112 Chapter 10. Tivoli Intelligent Orchestrator messages 114 Chapter 10. Tivoli Intelligent Orchestrator messages 116 Chapter 10. Tivoli Intelligent Orchestrator messages 118 Chapter 10. Tivoli Intelligent Orchestrator messages 120 Chapter 10. Tivoli Intelligent Orchestrator messages 121 122 Chapter 10. Tivoli Intelligent Orchestrator messages 123 Deployment engine subsystem messages 124 Chapter 10. Tivoli Intelligent Orchestrator messages 125 126 Chapter 10. Tivoli Intelligent Orchestrator messages 127 128 Chapter 10. Tivoli Intelligent Orchestrator messages 129 130 Chapter 10. Tivoli Intelligent Orchestrator messages 131 132 Chapter 10. Tivoli Intelligent Orchestrator messages 133 134 Chapter 10. Tivoli Intelligent Orchestrator messages 135 Job Distribution Service (JDS) subsystem messages J2ee subsystem messages 136 Chapter 10. Tivoli Intelligent Orchestrator messages 137 138 Chapter 10. Tivoli Intelligent Orchestrator messages 139 140 Chapter 10. Tivoli Intelligent Orchestrator messages 141 142 Chapter 10. Tivoli Intelligent Orchestrator messages 143 144 Chapter 10. Tivoli Intelligent Orchestrator messages 145 146 Chapter 10. Tivoli Intelligent Orchestrator messages 147 148 Chapter 10. Tivoli Intelligent Orchestrator messages 149 Policy engines subsystem messages 150 Chapter 10. Tivoli Intelligent Orchestrator messages 151 152 Chapter 10. Tivoli Intelligent Orchestrator messages 153 Common agent subsystem messages TC driver manager subsystem messages 154 Chapter 10. Tivoli Intelligent Orchestrator messages 155 156 Chapter 10. Tivoli Intelligent Orchestrator messages 157 158 Chapter 10. Tivoli Intelligent Orchestrator messages 159 160 Chapter 10. Tivoli Intelligent Orchestrator messages 161 162 Chapter 10. Tivoli Intelligent Orchestrator messages 163 Utilities subsystem messages 164 Page Compliance Messages 166 Chapter 10. Tivoli Intelligent Orchestrator messages 167 Group Messages 168 Chapter 10. Tivoli Intelligent Orchestrator messages 169 Networking Messages 170 RXA Messages Chapter 10. Tivoli Intelligent Orchestrator messages 171 172 Chapter 10. Tivoli Intelligent Orchestrator messages 173 Software Distribution Messages 174 Chapter 10. Tivoli Intelligent Orchestrator messages 175 176 $BINDIR/../swdis/work/ software_package_name.version .log Chapter 10. Tivoli Intelligent Orchestrator messages 177 password are defined in Tivoli Provisioning Manager WebUI. 178 Chapter 11. Tivoli Common Agent Services messages This chapter lists the available Tivoli Common Agent Services messages. Copyright IBM Corp. 2003, 2006 Common Agent messages 180 Chapter 11. Tivoli Common Agent Services messages 181 182 Chapter 11. Tivoli Common Agent Services messages 183 184 Chapter 11. Tivoli Common Agent Services messages 185 186 Chapter 11. Tivoli Common Agent Services messages 187 188 Chapter 11. Tivoli Common Agent Services messages 189 190 Chapter 11. Tivoli Common Agent Services messages 191 192 Chapter 11. Tivoli Common Agent Services messages 193 194 Chapter 11. Tivoli Common Agent Services messages 195 196 Chapter 11. Tivoli Common Agent Services messages 197 198 Chapter 11. Tivoli Common Agent Services messages 199 Correct the problem, and then start the upgrade again. 200 Chapter 11. Tivoli Common Agent Services messages 201 Agent Manager messages 202 Chapter 11. Tivoli Common Agent Services messages 203 For other errors, contact IBM Customer Support. 204 Chapter 11. Tivoli Common Agent Services messages 205 206 Chapter 11. Tivoli Common Agent Services messages 207 208 Chapter 11. Tivoli Common Agent Services messages 209 210 Chapter 11. Tivoli Common Agent Services messages 211 212 Chapter 11. Tivoli Common Agent Services messages 213 Changes the value of the Registration.Agent.Access.Password parameter in the 214 Chapter 11. Tivoli Common Agent Services messages 215 216 Chapter 11. Tivoli Common Agent Services messages 217 218 installedApps/ node / application_name .ear/ application_name.war/WEB-INF installedApps/ node / application_name .ear/ application_name.war/WEB-INF installedApps/ node / application_name .ear/ application_name.war/WEB-INF Chapter 11. Tivoli Common Agent Services messages 219 220 Appendix A. Common Agent serviceability tool Copyright IBM Corp. 2003, 2006 Components Running the serviceability tool Contents of the CASservice.zip file 222 <basedir>\config\ <basedir>\config\ <basedir>\config\ <basedir>\config\ Page 224 Appendix B. Support information Copyright IBM Corp. 2003, 2006 Searching knowledge bases Search the information center on your local system or network Search the Internet Contacting IBM Software Support 226 Determine the business impact of your problem Appendix B. Support information 227 Describe your problem and gather background information Submit your problem to IBM Software Support 228 Appendix C. Notices Copyright IBM Corp. 2003, 2006 Trademarks 230 WebSphere Tivoli AIX iSeries Index A B C D E M O 232 P S U W X 234