Citrix Systems 4.2 22.2. Working with Server Logs

Chapter 22. Troubleshooting

254

22.1.6.1. Permissions

Consider the following:

• The root admin can delete or archive one or multiple alerts or events.

• The domain admin or end user can delete or archive one or multiple events.

22.1.6.2. Procedure

1. Log in as administrator to the CloudPlatform UI.

2. In the left navigation, click Events.

3. Perform either of the following:

• To archive events, click Archive Events, and specify event type and date.

• To archive events, click Delete Events, and specify event type and date.

4. Click OK.

22.2. Working with Server Logs

The CloudPlatform Management Server logs all web site, middle tier, and database activities for

diagnostics purposes in /var/log/cloudstack/management/. The CloudPlatform logs a variety of error

messages. We recommend this command to find the problematic output in the Management Server

log:.

Note

When copying and pasting a command, be sure the command has pasted as a single line before

executing. Some document viewers may introduce unwanted line breaks in copied text.

grep -i -E 'exception|unable|fail|invalid|leak|warn|error' /var/log/cloudstack/

management/management-server.log

The CloudPlatform processes requests with a Job ID. If you find an error in the logs and you are

interested in debugging the issue you can grep for this job ID in the management server log. For

example, suppose that you find the following ERROR message:

2010-10-04 13:49:32,595 ERROR [cloud.vm.UserVmManagerImpl] (Job-Executor-11:job-1076)

Unable to find any host for [User|i-8-42-VM-untagged]

Note that the job ID is 1076. You can track back the events relating to job 1076 with the following grep:

grep "job-1076)" management-server.log

The CloudPlatform Agent Server logs its activities in /var/log/cloudstack/agent/.

Contents

Main Page Page Page Page Page Page Page Page Page Getting More Information and Help 1.1. Additional Documentation Available 1.2. Citrix Knowledge Center 1.3. Contacting Support Page Concepts 2.1. What Is CloudPlatform? 2.2. What Can CloudPlatform Do? 2.3. Deployment Architecture Overview 2.3.1. Management Server Overview 2.3.2. Cloud Infrastructure Overview 2.3.3. Networking Overview Page Page Cloud Infrastructure Concepts 3.1. About Regions 3.2. About Zones Page 3.3. About Pods 3.4. About Clusters 3.5. About Hosts 3.6. About Primary Storage 3.7. About Secondary Storage 3.8. About Physical Networks 3.8.1. Basic Zone Network Traffic Types 3.8.2. Basic Zone Guest IP Addresses 3.8.3. Advanced Zone Network Traffic Types 3.8.4. Advanced Zone Guest IP Addresses 3.8.5. Advanced Zone Public IP Addresses 3.8.6. System Reserved IP Addresses Page Accounts 4.1. Accounts, Users, and Domains Accounts Domains Users 4.1.1. Dedicating Resources to Accounts and Domains 4.1.1.1. How to Dedicate a Zone, Cluster, Pod, or Host to an Account or Domain 4.1.1.2. How to Use Dedicated Hosts 4.1.1.3. Behavior of Dedicated Hosts, Clusters, Pods, and Zones 4.2. Using an LDAP Server for User Authentication 4.2.1. Configuring an LDAP Server 4.2.1.1. Adding an LDAP Server Page 4.2.1.2. Removing an LDAP Configuration 4.2.2. Example LDAP Configuration Commands 4.2.3. Search Base 4.2.4. Query Filter 4.2.4.1. Active Directory 4.2.4.2. ApacheDS 4.2.5. Search User Bind DN 4.2.6. SSL Keystore Path and Password Page User Services Overview 5.1. Service Offerings, Disk Offerings, Network Offerings, and Templates Page User Interface 6.1. Supported Browsers 6.2. Log In to the UI Username Password 6.2.2. Root Administrator's UI Overview 6.2.3. Logging In as the Root Administrator 6.2.4. Changing the Root Password 6.3. Using SSH Keys for Authentication 6.3.1. Creating an Instance from a Template that Supports SSH Keys 6.3.2. Creating the SSH Keypair 6.3.3. Creating an Instance 6.3.4. Logging In Using the SSH Keypair 6.3.5. Resetting SSH Keys Page Using Projects to Organize Users and Resources 7.1. Overview of Projects 7.2. Configuring Projects 7.2.1. Setting Up Invitations 7.2.2. Setting Resource Limits for Projects 7.2.3. Setting Project Creator Permissions 7.3. Creating a New Project 7.4. Adding Members to a Project 7.4.1. Sending Project Membership Invitations 7.4.2. Adding Project Members From the UI 7.5. Accepting a Membership Invitation 7.6. Suspending or Deleting a Project 7.7. Using the Project View Page Steps to Provisioning Your Cloud Infrastructure 8.1. Overview of Provisioning Steps 8.2. Adding Regions (optional) 8.2.1. The First Region: The Default Region 8.2.2. Adding a Region 8.2.3. Adding Third and Subsequent Regions 8.2.4. Deleting a Region 8.3. Adding a Zone 8.3.1. Create a Secondary Storage Mount Point for the New Zone 8.3.2. Prepare the System VM Template 8.3.3. Steps to Add a New Zone 8.3.3.1. Basic Zone Configuration Page Page 8.3.3.2. Advanced Zone Configuration Page Page Page Page 8.4. Adding a Pod 8.5. Adding a Cluster 8.5.1. Add Cluster: KVM or XenServer 8.5.2. Add Cluster: OVM 8.5.3. Add Cluster: vSphere 8.5.3.1. VMware Cluster Size Limit 8.5.3.2. Adding a vSphere Cluster Page Page 8.6. Adding a Host 8.6.1. Adding a Host (XenServer, KVM, or OVM) 8.6.1.1. Requirements for XenServer, KVM, and OVM Hosts 8.6.1.1.1. XenServer Host Additional Requirements 8.6.1.1.2. KVM Host Additional Requirements 8.6.1.2. Adding a XenServer, KVM, or OVM Host 8.6.2. Adding a Host (vSphere) 8.7. Adding Primary Storage 8.8. Adding Secondary Storage 8.8.1. Adding an NFS Secondary Staging Store for Each Zone 8.9. Initialize and Test Page Service Offerings 9.1. Compute and Disk Service Offerings 9.1.1. Creating a New Compute Offering 9.1.2. Creating a New Disk Offering 9.1.3. Modifying or Deleting a Service Offering 9.2. System Service Offerings 9.2.1. Creating a New System Service Offering 9.2.2. Changing the Secondary Storage VM Service Offering on a Guest Network Page Page Setting Up Networking for Users 10.1. Overview of Setting Up Networking for Users 10.2. About Virtual Networks 10.2.1. Isolated Networks 10.2.2. Shared Networks 10.3. Network Service Providers 10.4. Network Service Providers Support Matrix 10.4.1. Individual 10.4.2. Support Matrix for an Isolated Network (Combination) 10.4.3. Support Matrix for Shared Network (Combination) 10.4.4. Support Matrix for Basic Zone 10.5. Network Offerings 10.5.1. Creating a New Network Offering Page Page 10.5.2. Changing the Network Offering on a Guest Network 10.5.3. Creating and Changing a Virtual Router Network Offering Page Page Working With Virtual Machines 11.1. About Working with Virtual Machines 11.2. Best Practices for Virtual Machines 11.2.1. Monitor VMs for Max Capacity 11.2.2. Install Required Tools and Drivers 11.3. VM Lifecycle 11.4. Creating VMs 11.4.1. Creating a VM from a template 11.4.2. Creating a VM from an ISO 11.4.3. Configuring Usage of Linked Clones on VMware 11.5. Accessing VMs 11.6. Appending a Display Name to the Guest VMs Internal Name 11.7. Stopping and Starting VMs 11.8. Assigning VMs to Hosts 11.8.1. Affinity Groups Creating a New Affinity Group Assign a New VM to an Affinity Group Change Affinity Group for an Existing VM View Members of an Affinity Group 11.9. Virtual Machine Snapshots for VMware 11.9.1. Limitations on VM Snapshots 11.9.2. Configuring VM Snapshots 11.9.3. Using VM Snapshots 11.10. Changing the VM Name, OS, or Group 11.11. Changing the Service Offering for a VM 11.11.1. CPU and Memory Scaling for Running VMs 11.11.2. Updating Existing VMs 11.11.3. Configuring Dynamic CPU and RAM Scaling 11.11.4. How to Dynamically Scale CPU and RAM 11.11.5. Limitations 11.12. Resetting the Virtual Machine Root Volume on Reboot 11.13. Moving VMs Between Hosts (Manual Live Migration) 11.14. Deleting VMs 11.15. Recovering a Destroyed VM 11.16. Working with ISOs 11.16.1. Adding an ISO 11.16.2. Attaching an ISO to a VM 11.16.3. Changing a VM's Base Image Page Page Working With Hosts 12.1. Adding Hosts 12.2. Scheduled Maintenance and Maintenance Mode for Hosts 12.2.1. vCenter and Maintenance Mode 12.2.2. XenServer and Maintenance Mode 12.3. Disabling and Enabling Zones, Pods, and Clusters 12.4. Removing Hosts 12.4.1. Removing XenServer and KVM Hosts 12.4.2. Removing vSphere Hosts 12.5. Re-Installing Hosts 12.6. Maintaining Hypervisors on Hosts 12.7. Using Cisco UCS as Bare Metal Host CloudPlatform 12.7.1. Registering a UCS Manager 12.7.2. Associating a Profile with a UCS Blade 12.7.3. Disassociating a Profile from a UCS Blade 12.8. Changing Host Password 12.9. Over-Provisioning and Service Offering Limits 12.9.1. Limitations on Over-Provisioning in XenServer and KVM 12.9.2. Requirements for Over-Provisioning 12.9.2.1. Balloon Driver XenServer VMware 12.10. VLAN Provisioning 12.10.1. VLAN Allocation Example 12.10.2. Adding Non Contiguous VLAN Ranges 12.10.3. Assigning VLANs to Isolated Networks Working with Templates 13.1. Creating Templates: Overview 13.2. Requirements for Templates 13.3. Best Practices for Templates 13.4. The Default Template 13.5. Private and Public Templates 13.6. Creating a Template from an Existing Virtual Machine 13.7. Creating a Template from a Snapshot 13.8. Uploading Templates Page 13.9. Exporting Templates 13.10. Creating a Windows Template 13.10.1. System Preparation for Windows Server 2008 R2 Page Page Page 13.10.2. System Preparation for Windows Server 2003 R2 13.11. Importing Amazon Machine Images Page Page 13.12. Converting a Hyper-V VM to a Template 13.13. Adding Password Management to Your Templates 13.13.1. Linux OS Installation 13.13.2. Windows OS Installation 13.14. Deleting Templates Page Working With Storage 14.1. Storage Overview 14.2. Primary Storage 14.2.1. Best Practices for Primary Storage 14.2.2. Runtime Behavior of Primary Storage 14.2.4. Storage Tags 14.2.5. Maintenance Mode for Primary Storage 14.3. Secondary Storage 14.3.1. Best Practices for Secondary Storage 14.3.2. Changing the Secondary Storage IP Address 14.3.3. Changing Secondary Storage Servers 14.4. Working With Volumes 14.4.1. Creating a New Volume 14.4.1.1. Using Local Storage for Data Volumes 14.4.1.2. To Create a New Volume 14.4.2. Uploading an Existing Volume to a Virtual Machine 14.4.3. Attaching a Volume 14.4.4. Detaching and Moving Volumes 14.4.5. VM Storage Migration 14.4.5.1. Migrating a Data Volume to a New Storage Pool 14.4.5.1.1. Migrating Storage For a Running VM 14.4.5.1.2. Migrating Storage and Attaching to a Different VM 14.4.5.2. Migrating a VM Root Volume to a New Storage Pool 14.4.6. Resizing Volumes 14.4.7. Reset VM to New Root Disk on Reboot 14.4.8. Volume Deletion and Garbage Collection 14.5. Working with Snapshots 14.5.1. Automatic Snapshot Creation and Retention 14.5.2. Incremental Snapshots and Backup 14.5.3. Volume Status 14.5.4. Snapshot Restore 14.5.5. Snapshot Job Throttling 14.5.6. VMware Volume Snapshot Performance Working with Usage 15.1. Configuring the Usage Server Page 15.2. Setting Usage Limits 15.2.1. Globally Configured Limits 15.2.2. Default Account Resource Limits 15.2.3. Per-Domain Limits Managing Networks and Traffic 16.1. Guest Traffic 16.2. Networking in a Pod 16.3. Networking in a Zone 16.4. Basic Zone Physical Network Configuration 16.5. Advanced Zone Physical Network Configuration 16.5.1. Configuring Isolated Guest Network 16.5.2. Configure Public Traffic in an Advanced Zone 16.5.3. Configuring a Shared Guest Network 16.6. Using Security Groups to Control Traffic to VMs 16.6.1. About Security Groups 16.6.2. Security Groups in Advanced Zones (KVM Only) Limitation 16.6.3. Enabling Security Groups 16.6.4. Adding a Security Group 16.6.5. Adding Ingress and Egress Rules to a Security Group 16.7. External Firewalls and Load Balancers 16.7.1. About Using a NetScaler Load Balancer 16.7.2. Configuring SNMPCommunity String on a RHEL Server 16.7.3. Initial Setup of External Firewalls and Load Balancers 16.7.4. Ongoing Configuration of External Firewalls and Load Balancers 16.8. Load Balancer Rules 16.8.1. Adding a Load Balancer Rule 16.8.2. Configuring AutoScale Prerequisites Configuration Page Disabling and Enabling an AutoScale Configuration Updating an AutoScale Configuration Runtime Considerations 16.8.3. Sticky Session Policies for Load Balancer Rules 16.8.4. Health Checks for Load Balancer Rules 16.9. Global Server Load Balancing 16.9.1. About Global Server Load Balancing 16.9.1.1. Components of GSLB 16.9.1.2. How GSLB Works in CloudPlatform 16.9.2. Configuring GSLB Page 16.9.2.1. Prerequisites and Guidelines 16.9.2.2. Enabling GSLB in NetScaler 16.9.2.3. Adding a GSLB Rule 16.9.2.4. Assigning Load Balancing Rules to GSLB 16.10. Using Multiple Guest Networks 16.10.1. Adding an Additional Guest Network 16.10.2. Reconfiguring Networks in VMs 16.10.2.1. Prerequisites 16.10.2.2. Adding a Network 16.10.2.3. Removing a Network 16.10.2.4. Selecting the Default Network 16.11. Guest IP Ranges 16.12. Acquiring a New IP Address 16.13. Releasing an IP Address 16.14. Reserving Public IP Addresses and VLANs for Accounts 16.14.1. Dedicating IP Address Ranges to an Account 16.14.2. Dedicating VLAN Ranges to an Account 16.15. IP Reservation in Isolated Guest Networks 16.15.1. IP Reservation Considerations 16.15.2. Limitations 16.15.3. Best Practices 16.15.4. Reserving an IP Range 16.16. Configuring Multiple IP Addresses on a Single NIC 16.16.1. Use Cases 16.16.2. Guidelines 16.16.3. Assigning Additional IPs to a VM 16.16.4. Port Forwarding and StaticNAT Services Changes 16.17. Multiple Subnets in Shared Network 16.17.1. Prerequisites and Guidelines 16.17.2. Adding Multiple Subnets to a Shared Network 16.18. About Elastic IP Page 16.19. Portable IPs 16.19.1. About Portable IP Guidelines 16.19.2. Configuring Portable IPs 16.19.3. Acquiring a Portable IP 16.19.4. Transferring Portable IP 16.20. Static NAT 16.20.1. Enabling or Disabling Static NAT 16.21. IP Forwarding and Firewalling 16.21.1. Egress Firewall Rules in an Advanced Zone 16.21.1.1. Prerequisites and Guidelines 16.21.1.2. Configuring an Egress Firewall Rule 16.21.1.3. Configuring the Default Egress Policy Allow Deny 16.21.2. Firewall Rules 16.21.3. Port Forwarding 16.22. IP Load Balancing 16.23. DNS and DHCP 16.24. Remote Access VPN 16.24.1. Configuring Remote Access VPN 16.24.2. Using Remote Access VPN with Windows 16.24.3. Using Remote Access VPN with Mac OS X 16.24.4. Setting Up a Site-to-Site VPN Connection 16.24.4.1. Creating and Updating a VPN Customer Gateway Page Page Updating and Removing a VPN Customer Gateway 16.24.4.2. Creating a VPN gateway for the VPC 16.24.4.3. Creating a VPN Connection Page 16.24.4.4. Restarting and Removing a VPN Connection 16.25. Isolation in Advanced Zone Using Private VLAN 16.25.1. About Private VLAN 16.25.2. Prerequisites 16.25.3. Creating a PVLAN-Enabled Guest Network 16.26. About Inter-VLAN Routing Page 16.27. Configuring a Virtual Private Cloud 16.27.1. About Virtual Private Clouds Major Components of a VPC: Network Architecture in a VPC Connectivity Options for a VPC VPC Network Considerations 16.27.2. Adding a Virtual Private Cloud 16.27.3. Adding Tiers Page 16.27.4. Configuring Network Access Control List 16.27.4.1. About Network ACL Lists 16.27.4.2. Creating ACL Lists 16.27.4.3. Creating an ACL Rule 16.27.4.4. Creating a Tier with Custom ACL List 16.27.4.5. Assigning a Custom ACL List to a Tier 16.27.5. Adding a Private Gateway to a VPC 16.27.5.1. Source NAT on Private Gateway 16.27.5.2. ACL on Private Gateway 16.27.5.3. Creating a Static Route 16.27.5.4. Blacklisting Routes 16.27.6. Deploying VMs to the Tier 16.27.7. Deploying VMs to VPC Tier and Shared Networks 16.27.8. Acquiring a New IP Address for a VPC 16.27.9. Releasing an IP Address Alloted to a VPC 16.27.10. Enabling or Disabling Static NAT on a VPC 16.27.11. Adding Load Balancing Rules on a VPC 16.27.11.1. Load Balancing Public Traffic (Public LB) 16.27.11.1.1. Enabling NetScaler as the LB Provider on a VPC Tier 16.27.11.1.2. Creating a Network Offering for Public LB 16.27.11.1.3. Creating a Public LB Rule 16.27.11.2. Load Balancing Tier-to-Tier traffic (Internal LB) 16.27.11.2.1. How Does Internal LB Work in VPC? Page 16.27.11.2.4. Creating an Internal LB Rule 16.27.12. Adding a Port Forwarding Rule on a VPC 16.27.13. Removing Tiers 16.27.14. Editing, Restarting, and Removing a Virtual Private Cloud 16.28. Persistent Networks 16.28.1. Persistent Network Considerations 16.28.2. Creating a Persistent Guest Network Working with System Virtual Machines 17.1. The System VM Template 17.2. Multiple System VM Support for VMware 17.3. Console Proxy 17.3.1. Changing the Console Proxy SSL Certificate and Domain 17.4. Virtual Router 17.4.1. Configuring the Virtual Router 17.4.2. Upgrading a Virtual Router with System Service Offerings 17.4.3. Best Practices for Virtual Routers 17.5. Secondary Storage VM System Reliability and High Availability 18.1. HA for Management Server 18.2. HA-Enabled Virtual Machines 18.3. Dedicated HA Hosts 18.4. Primary Storage Outage and Data Loss 18.5. Secondary Storage Outage and Data Loss 18.6. Limiting the Rate of API Requests 18.6.1. Configuring the API Request Rate 18.6.2. Limitations on API Throttling Page Managing the Cloud 19.1. Using Tags to Organize Resources in the Cloud 19.2. Setting Configuration Parameters 19.2.1. About Configuration Parameters 19.2.2. Setting Global Configuration Parameters 19.2.3. Setting Local Configuration Parameters 19.2.4. Granular Global Configuration Parameters Page 19.3. Changing the Database Configuration 19.4. Administrator Alerts 19.4.1. Customizing Alerts with Global Configuration Settings 19.4.2. Sending Alerts to External SNMP and Syslog Managers 19.4.2.1. SNMP Alert Details 19.4.2.2. Syslog Alert Details 19.4.2.3. Configuring SNMP and Syslog Managers 19.4.2.4. Deleting an SNMP or Syslog Manager 19.5. Customizing the Network Domain Name 19.6. Stopping and Restarting the Management Server CloudPlatform API 20.1. Provisioning and Authentication API 20.2. Allocators 20.3. User Data and Meta Data Page Tuning 21.1. Performance Monitoring 21.2. Increase Management Server Maximum Memory 21.3. Set Database Buffer Pool Size 21.4. Set and Monitor Total VM Limits per Host 21.5. Configure XenServer dom0 Memory Troubleshooting 22.1. Events 22.1.1. Event Logs 22.1.2. Event Notification Use Cases 22.1.3. Standard Events 22.1.4. Long Running Job Events 22.1.5. Event Log Queries 22.1.6. Deleting and Archiving Events and Alerts 22.1.6.1. Permissions 22.1.6.2. Procedure 22.2. Working with Server Logs 22.3. Log Collection Utility cloud-bugtool 22.3.1. Using cloud-bugtool 22.4. Data Loss on Exported Primary Storage Symptom 22.5. Recovering a Lost Virtual Router 22.6. Maintenance mode not working on vCenter 22.7. Unable to deploy VMs from uploaded vSphere template 22.8. Unable to power on virtual machine on VMware 22.9. Load balancer rules fail after changing network offering Appendix A. Event Types Page 261 Appendix B. Alerts