Main
T
l
Problem Determination and Troubleshooting Guide
Page
T
l
Problem Determination and Troubleshooting Guide
ivo
Page
Contents
Preface . . . . . . . . . . . . . . vii
Copyright IBM Corp. 2003, 2006
Chapter 1. Introduction to troubleshooting . . . . . . . . . . .1
Chapter 3. Log file types . . . . . . .35
Chapter 8. Troubleshooting the agent manager . . . . . . . . . . . . . .79
iv
Chapter 9. Contacting IBM Tivoli Software Support . . . . . . . . . .91
Chapter 10. Tivoli Intelligent Orchestrator messages . . . . . . .95
Chapter 11. Tivoli Common Agent Services messages . . . . . . . . . 179
Contents v
Common Agent messages . . . . . . . . . 179 Agent Manager messages . . . . . . . . . 202
Index . . . . . . . . . . . . . . . 231
Appendix A. Common Agent serviceability tool . . . . . . . . . 221
vi
Preface
Copyright IBM Corp. 2003, 2006
Who should read this book
Publications
Tivoli Intelligent Orchestrator library
Prerequisite publications
viii
Related publications and resources
Accessing publications online
Preface ix
Ordering publications
Accessibility
Tivoli software training
x
Support information
Conventions used in this book
Typeface conventions
Chapter 1. Introduction to troubleshooting
Copyright IBM Corp. 2003, 2006
Product and system overview
Product comparison
2
The following table summarizes the complementary roles of the two products:
Product architecture
Tivoli Intelligent Orchestrator includes the following main components:
Chapter 1. Introduction to troubleshooting 3
WebSphere Application Server
WebSphere Application Server
Prerequisites installed with Tivoli Intelligent Orchestrator
4
Built-in troubleshooting features
Chapter 1. Introduction to troubleshooting 5
Problem classification
Product maintenance
Backing up the system
Performing periodic checks and maintenance
6
http://www.ibm.com/software/sysmgmt/products/support/ IBMTivoliIntelligentOrchestrator.html
Problem resolution
Chapter 2. Problem determination essentials for Tivoli Intelligent Orchestrator
Copyright IBM Corp. 2003, 2006
What was done differently this time?
Problem determination methodologies
Defining the problem
Identifying the cause of the problem
8
Fixing the problem
Chapter 2. Problem determination essentials for Tivoli Intelligent Orchestrator 9
Troubleshooting methods for Tivoli Intelligent Orchestrator and prerequisites
Troubleshooting the prerequisite installation
10
2000Solaris
DB2 Universal Database verification
netstat -an |grep 50000
IBM Tivoli Directory Server verification
v UNIX:
ibmdirctl -D cn=root -w <password> status
12
ibmdirctl -D cn=root -w <password> start
v Stopping the IBM Tivoli Directory Server:
ibmdirctl -D cn=root -w <password> stop
WebSphere Application Server verification
Chapter 2. Problem determination essentials for Tivoli Intelligent Orchestrator 13
Application Server status:
serverStatus -all -user <username> -password <password>
14
tail -f SystemOut.log
AdminInitiali A ADMN0015I: AdminService initialized
Chapter 2. Problem determination essentials for Tivoli Intelligent Orchestrator 15
Secure Authentication Services (LDAP)
Configuration A SECJ0215I: Successfully set JAAS login provider
SecurityDM I SECJ0231I: The Security components FFDC Diagnostic Module
SecurityCompo I SECJ0240I: Security service initialization completed
Map Tivoli Intelligent Orchestrator resource references
16
Simple Object Access Protocol ( SOAP)
JMXSoapAdapte A ADMC0013I: SOAP connector available at port 8880
Applications
TCEAR Application
Tivoli Intelligent Orchestrator installation and subcomponent configuration verification
Tivoli Intelligent Orchestrator installation
Verifying subcomponent configuration
list tables
18
<config>
</config>
3.
ldapsearch -b cn=schema -s base objectclass=* | grep -i think
20
Chapter 2. Problem determination essentials for Tivoli Intelligent Orchestrator 21
Tivoli Intelligent Orchestrator runtime problem determination
Component-based problem determination
22
4. Check the tioldap authentication:
6. Check the tioappadmin user:
8. Search the knowledge bases and, if necessary, contact Support. For more
Chapter 2. Problem determination essentials for Tivoli Intelligent Orchestrator 23
Task-based problem determination
24
SQL0532N A parent row cannot be deleted
Chapter 2. Problem determination essentials for Tivoli Intelligent Orchestrator 25
restricts the deletion.
DB2INST1.SPARE_POOL(POOL_ID) IP = 9.3.127.47
SystemException: COPCOM093E The JDBC driver caused an SQL exception....
Accessing the knowledge bases
26
http://www-306.ibm.com/software/sysmgmt/products/support/ IBMTivoliIntelligentOrchestrator.html
http://www-306.ibm.com/software/sysmgmt/products/support/ IBMDirectoryServer.html
v For the WebSphere Application Server:
Note: The command line tools can be found on TIO_HOME/tools
28
Page
30
Page
WebSphere Application Server command line tools
32
IBM Tivoli Directory Server command line tools
34
Chapter 3. Log file types
Copyright IBM Corp. 2003, 2006
Message logs
Message ID format
code
Message elements
36
Action title Description Administrator Response
Chapter 3. Log file types 37
Describes what response a system programmer might be able to take.
Trace logs
Common Base Event logs
38
Chapter 4. Locating and configuring log files
Copyright IBM Corp. 2003, 2006
Log locations
Log directory files
Logging levels
40
msg_error#com.thinkdynamics.kanaha. util.logging.MessageLevel
Extended for message logs msg_warn#com.thinkdynamics.kanaha. util.logging.MessageLevel
Extended for message logs msg_info#com.thinkdynamics.kanaha. util.logging.MessageLevel
Page
Log level defaults
42
RollingFileAppender defaults
msg.log v Maximum number of archived log files: 10
cbe.log v Maximum number of archived log files: 10
Preparing log files for review by IBM Tivoli Support
Chapter 4. Locating and configuring log files 43
44
Chapter 5. Tivoli Intelligent Orchestrator log files
Copyright IBM Corp. 2003, 2006
/var/ibm/tivoli/common/COP/logs
Log files for product setup
These log files are for installation and data center model initialization.
Log files for starting and stopping the server
46
File name Location Description tio_start.log Windows
UNIX or Linux $TIO_LOGS
UNIX or Linux $TIO_LOGS
Log files for the Web interface
Deployment engine log files
Workflow logs
48
Logs for data collection and decision making
Automation package logs
Log files for orchestration packages are stored in the %TIO_LOGS%\tcdrivermanager directory.
50
Chapter 6. Middleware prerequisite log files
Installation log files for prerequisites
Installation logs for individual prerequisites
WebSphere Application Server
DB2 Universal Database
Runtime logs for prerequisites
WebSphere Application Server
Chapter 6. Middleware prerequisite log files 53
DB2 Universal Database
Directory Server
v db2diag.log v ibmds_config.log v
Tivoli Agent Manager
54
Copyright IBM Corp. 2003, 2006
Problems with installation
In Linux, you cannot create the user tioadmin
56
ibmdirctl -D cn=root -w <password> status
start: ibmdirctl -D cn=root -w <password> start
Installing the common agent and the agent manager is not supported
Chapter 7. Common problems and known limitations in Tivoli Intelligent Orchestrator 57
Manually uninstall the common agent.
You should also verify whether the required port is available:
netstat -an |grep 50000
The Microsoft Active Directory installation fails with invalid certificate value
58
Environment variables not set for user tioadmin
Cause
Chapter 7. Common problems and known limitations in Tivoli Intelligent Orchestrator 59
Solution
Cause
Solution 1
Solution 2
60
WAS_install_dir \bin\stopServer app_server_name
- On AIX, Linux, and Solaris
ADMU4000I: Server app_server_name stop completed.
WAS_install_dir /bin/stopServer.sh app_server_name
- On AIX, Linux, and Solaris
./RemoveCASTables.sh database_password
Agent_Manager_install_dir\_uninst\uninstall.exe
java -jar uninstall.jar
Or, to uninstall silently:
Problems with logging in
You cannot log in
Cause
The LDAP server might be down.
Solution 1
You cannot change your password
Cause
Solution
6. Change your password again based on the specified password syntax.
Problems with workflows
64
DB2 Universal Database errors occur when you deploy resources
db2 update db cfg for $DB2_DB_NAME using applheapsz 3072
4. Run the following command to terminate all connections:
DB2 Universal Database deadlocks occur when the system runs a logical operation
Cause
Solution
Shell command error: Resource temporarily unavailable
Solution
66
Syntax for UNIX:
<?xml version="1.0" encoding="UTF-8"?> <workflow-execution-history>
Chapter 7. Common problems and known limitations in Tivoli Intelligent Orchestrator 67
Note: Any Unicode character can be used to enter comments in workflow scripts.
A workflow does not install
Cannot run workflows if the PS1 environment variable is altered
68
$
tioadmin>
command$ $ # command#
On Windows, Tivoli Intelligent Orchestrator does not start
Chapter 7. Common problems and known limitations in Tivoli Intelligent Orchestrator 69
Incorrect SOAP parameters generate detailed Java exception messages
70
XML import does not work properly
DB2 Universal Database does not work properly with terminal server
Chapter 7. Common problems and known limitations in Tivoli Intelligent Orchestrator 71
Disable the terminal server and reboot before trying to install again.
On UNIX, Tivoli Intelligent Orchestrator logs fill up the file system
The capacity of the file system is not large enough.
72
1b2ddb28 JMSEmbeddedPr A MSGS0054I: Stopping the Broker
When you run ps -ef | grep java it returns only one process: stopServer .
A problem occurred in the Linux libstdc++ library.
Not enough connection managers are allocated from the database server.
Chapter 7. Common problems and known limitations in Tivoli Intelligent Orchestrator 73
74
Chapter 7. Common problems and known limitations in Tivoli Intelligent Orchestrator 75
Common agents unable to communicate with the agent manager
$WAS_HOME/installedApps/<nodename>/AgentManager.ear/ AgentManager.war/WEB-INF/classes/resources
a. Find the line that looks like:
ARS.host=<fully qualified name>
ARS.host=9.2.2.2
There is a maximum size for returnResult parameter storage
Device Manager limitations
The following are known limitations for the Device Manager:
Agent receives an HTTP Unauthorized (401) response code
Chapter 7. Common problems and known limitations in Tivoli Intelligent Orchestrator 77
Known report limitations
Running any report with too many objects selected fails
COPJEE389E: The report contains too many objects. Retry with fewer selections.
78
Chapter 8. Troubleshooting the agent manager
Copyright IBM Corp. 2003, 2006
Log files for the agent manager
Packaging the log files
Locating the installation log files
80
Chapter 8. Troubleshooting the agent manager 81
Locating the uninstallation log files
Locating the remote registry logs
Locating the runtime log files
82
Locating the DB2 Universal Database runtime logs
Determining the version of agent manager
Chapter 8. Troubleshooting the agent manager 83
Agent_Manager_install_dir/GetAMInfo.sh
Agent_Manager_install_dir\GetAMInfo.bat AgentManager
Agent_Manager_install_dir/GetAMInfo.sh AgentRecoveryService
Verifying the installation of WebSphere Application Server
84
Errors starting the agent manager application server
Errors starting the agent manager application
Verifying the connection to the registry
Enabling and disabling tracing
Enabling tracing
Disabling tracing
Recovering from an incomplete uninstallation of the agent manager
86
WAS_install_dir\bin\WASService.exe -remove "Tivoli Agent Manager"
am:2345:once:/opt/IBM/WebSphere/AppServer/bin/rc.am >/dev/console 2>&1
amdb:2345:once:su - db2inst1 -c db2start >/dev/console 2>&1
Common problems
Registration and communication failures
Problems using the WebSphere Administrative Console
88
Cannot start the WebSphere Administrative Console
Configuring the WebSphere Administrative Console to use a different port
stopServer server1 startServer server1
5. Access the administrative console using the following Web address:
http:// hostName :9091/admin
90
Chapter 9. Contacting IBM Tivoli Software Support
Copyright IBM Corp. 2003, 2006
General data to collect for Tivoli Support
packageLog ZIP tool
92
Location
Windows %TIO_HOME%\tools
Where $TIO_HOME is the Tivoli Intelligent Orchestrator home directory.
Prerequisites
Chapter 9. Contacting IBM Tivoli Software Support 93
set JAVA_HOME=C:\IBM\WebSphere\AppServer\java set WAS_HOME=C:\IBM\WebSphere\AppServer
Linux and UNIX
export JAVA_HOME=/opt/WebSphere/AppServer/java export WAS_HOME=/opt/WebSphere/AppServer
94
Chapter 10. Tivoli Intelligent Orchestrator messages
Copyright IBM Corp. 2003, 2006
Common subsystem messages
96
Chapter 10. Tivoli Intelligent Orchestrator messages 97
98
Chapter 10. Tivoli Intelligent Orchestrator messages 99
100
Chapter 10. Tivoli Intelligent Orchestrator messages 101
102
Chapter 10. Tivoli Intelligent Orchestrator messages 103
104
Chapter 10. Tivoli Intelligent Orchestrator messages 105
106
Chapter 10. Tivoli Intelligent Orchestrator messages 107
108
Chapter 10. Tivoli Intelligent Orchestrator messages 109
110
Chapter 10. Tivoli Intelligent Orchestrator messages
112
Chapter 10. Tivoli Intelligent Orchestrator messages
114
Chapter 10. Tivoli Intelligent Orchestrator messages
116
Chapter 10. Tivoli Intelligent Orchestrator messages
118
Chapter 10. Tivoli Intelligent Orchestrator messages
120
Chapter 10. Tivoli Intelligent Orchestrator messages 121
122
Chapter 10. Tivoli Intelligent Orchestrator messages 123
Deployment engine subsystem messages
124
Chapter 10. Tivoli Intelligent Orchestrator messages 125
126
Chapter 10. Tivoli Intelligent Orchestrator messages 127
128
Chapter 10. Tivoli Intelligent Orchestrator messages 129
130
Chapter 10. Tivoli Intelligent Orchestrator messages 131
132
Chapter 10. Tivoli Intelligent Orchestrator messages 133
134
Chapter 10. Tivoli Intelligent Orchestrator messages 135
Job Distribution Service (JDS) subsystem messages
J2ee subsystem messages
136
Chapter 10. Tivoli Intelligent Orchestrator messages 137
138
Chapter 10. Tivoli Intelligent Orchestrator messages 139
140
Chapter 10. Tivoli Intelligent Orchestrator messages 141
142
Chapter 10. Tivoli Intelligent Orchestrator messages 143
144
Chapter 10. Tivoli Intelligent Orchestrator messages 145
146
Chapter 10. Tivoli Intelligent Orchestrator messages 147
148
Chapter 10. Tivoli Intelligent Orchestrator messages 149
Policy engines subsystem messages
150
Chapter 10. Tivoli Intelligent Orchestrator messages 151
152
Chapter 10. Tivoli Intelligent Orchestrator messages 153
Common agent subsystem messages
TC driver manager subsystem messages
154
Chapter 10. Tivoli Intelligent Orchestrator messages 155
156
Chapter 10. Tivoli Intelligent Orchestrator messages 157
158
Chapter 10. Tivoli Intelligent Orchestrator messages 159
160
Chapter 10. Tivoli Intelligent Orchestrator messages 161
162
Chapter 10. Tivoli Intelligent Orchestrator messages 163
Utilities subsystem messages
164
Page
Compliance Messages
166
Chapter 10. Tivoli Intelligent Orchestrator messages 167
Group Messages
168
Chapter 10. Tivoli Intelligent Orchestrator messages 169
Networking Messages
170
RXA Messages
Chapter 10. Tivoli Intelligent Orchestrator messages 171
172
Chapter 10. Tivoli Intelligent Orchestrator messages 173
Software Distribution Messages
174
Chapter 10. Tivoli Intelligent Orchestrator messages 175
176
$BINDIR/../swdis/work/ software_package_name.version .log
Chapter 10. Tivoli Intelligent Orchestrator messages 177
password are defined in Tivoli Provisioning Manager WebUI.
178
Chapter 11. Tivoli Common Agent Services messages
This chapter lists the available Tivoli Common Agent Services messages.
Copyright IBM Corp. 2003, 2006
Common Agent messages
180
Chapter 11. Tivoli Common Agent Services messages 181
182
Chapter 11. Tivoli Common Agent Services messages 183
184
Chapter 11. Tivoli Common Agent Services messages 185
186
Chapter 11. Tivoli Common Agent Services messages 187
188
Chapter 11. Tivoli Common Agent Services messages 189
190
Chapter 11. Tivoli Common Agent Services messages 191
192
Chapter 11. Tivoli Common Agent Services messages 193
194
Chapter 11. Tivoli Common Agent Services messages 195
196
Chapter 11. Tivoli Common Agent Services messages 197
198
Chapter 11. Tivoli Common Agent Services messages 199
Correct the problem, and then start the upgrade again.
200
Chapter 11. Tivoli Common Agent Services messages 201
Agent Manager messages
202
Chapter 11. Tivoli Common Agent Services messages 203
For other errors, contact IBM Customer Support.
204
Chapter 11. Tivoli Common Agent Services messages 205
206
Chapter 11. Tivoli Common Agent Services messages 207
208
Chapter 11. Tivoli Common Agent Services messages 209
210
Chapter 11. Tivoli Common Agent Services messages 211
212
Chapter 11. Tivoli Common Agent Services messages 213
Changes the value of the Registration.Agent.Access.Password parameter in the
214
Chapter 11. Tivoli Common Agent Services messages 215
216
Chapter 11. Tivoli Common Agent Services messages 217
218
installedApps/ node / application_name .ear/ application_name.war/WEB-INF
installedApps/ node / application_name .ear/ application_name.war/WEB-INF
installedApps/ node / application_name .ear/ application_name.war/WEB-INF
Chapter 11. Tivoli Common Agent Services messages 219
220
Appendix A. Common Agent serviceability tool
Copyright IBM Corp. 2003, 2006
Components
Running the serviceability tool
Contents of the CASservice.zip file
222
<basedir>\config\ <basedir>\config\ <basedir>\config\ <basedir>\config\
Page
224
Appendix B. Support information
Copyright IBM Corp. 2003, 2006
Searching knowledge bases
Search the information center on your local system or network
Search the Internet
Contacting IBM Software Support
226
Determine the business impact of your problem
Appendix B. Support information 227
Describe your problem and gather background information
Submit your problem to IBM Software Support
228
Appendix C. Notices
Copyright IBM Corp. 2003, 2006
Trademarks
230
WebSphere Tivoli AIX iSeries
Index A
B
C
D
E
M
O
232
P
S
U
W
X
234