Sun Microsystems 5310 NAS Note

3-12 Sun StorEdge 5310 NAS Troubleshooting Guide • December 2004

1. Stop I/O activity to all modules.

2. Remove the front cover from the command module, if applicable.

3. Determine the status of each module and its components.

a. Note the status of the lights on the front and the back of each mod ule.

A green light indicates a normal status; an amber light indicates a hardware fault.

b. Open the Array Management Window for the storage array.

c. To view the status of its components, select the appropriate component button

for each module in the Physical View of the Array Management Window.

The status for each component will be either Optimal or Needs Attention.

4. Does each module display green lights only and is the status Optimal on each

module component?

■Yes - Go t o ste p 6.

■No - Go to step 5.

Note – If the fault requires you to power off an attached drive module, you may

need to cycle the power on all remaining modules in the storage array. Before you

power off any attached module, contact technical support.

5. Diagnose and correct the fault.

a. To run the Recovery Guru, select the Recovery Guru toolbar button in the Array

Management Window.

b. Complete the recovery procedure.

If the Recovery G uru directs you to replace a failed component, use the individual

lights on the modules to locate the specific failed component. For troubleshooting

procedures, refer to “Troubleshooting and Recovery” on page3-22.

c. When the recovery procedure is completed, select Recheck in the Recovery

Guru to re-run the Recovery Guru and to ensure that the problem has been

corrected.

d. If the problem persists, contact technical support.

Caution – Potential data corruption. An abrupt power loss to any module in a

storage array can cause data corruption, especially if the power loss occurs when

data is being written to a disk or if the power loss occurs when write back caching is

enabled and data is being downloaded to cache memory. Before turning off power to

the modules, always wait until the Cache Active light turns off and all drive Active

lights stop flashing.

Contents

Main Please Recycle Contents Page Page Page Page Page Tab le s Page Figures Page Page Page Preface Who Should Use This Book How This Manual is Organized Typographic Conventions Related Documentation Ordering Sun Documents Accessing Sun Documentation Online Shell Prompts in Command Examples Sun Welcomes Your Comments Troubleshooting Overview 1.1 How to Use This Manual 1.2 Important Notices and Information on the Sun StorEdge 5310 NAS 1.3 Troubleshooting Tools 1.3.0.6 Log Message Severity Levels 1.4 Troubleshooting Procedures 1.4.0.1 High-Level Troubleshooting Tasks Note 1.4.0.2 Initial Troubleshooting Guidelines 1.5 Troubleshooting Flow Charts Page 1.6 Diagnostic Information Sources 1.6.1 StorEdge Diagnostic Email Important 1.6.2 Data Collection for Escalations 1.6.2.1 Collecting Information from the Sun StorEdge 5310 NAS 1.6.2.2 Accurately quantify the problem 1.6.2.3 Collect general data 1.6.2.4 Collect specific data 1.6.2.5 Check remote access capabilities 1.6.2.6 Data Collection for Specific Issues Software compatibility issues 1.6.2.7 Security Issues Cacls User access token Proc filesystem Network trace 1.6.2.8 StorEdge network capture utility Client and Server data Duplication instructions 1.6.2.9 TCP/IP Connectivity problems 1.6.2.10 Performance Issues Network Configuration: Configuration: Other processes / High CPU Utilization Command Line performance utilities 1.6.3 Log Error Messages 1.6.4 SYSLOG 1.6.4.1 Understanding Sun StorEdge 5310 NAS Log Messages Components of an Event Message Time/date Severity Severity Level Definitions (highest to lowest) 1.6.5 Error Codes from the Sun StorEdge 5310 NAS LCD Display and syslog Page UPS Subsystem Errors Refer to TableA-3 for descriptions of UPS error conditions. Page File System Errors PEMS Events Note Page 1.7 Maintenance Precautions 1.7.0.1 Tools Required 1.7.0.2 Electrostatic Discharge Information Note 1.7.0.3 Preparation Procedures 1.8 Static Electricity Precautions 1.8.0.1 Grounding Procedure 1.8.0.2 Static Electricity Caution NAS Head 2.1 Hardware 2.2 Contacting Technical Support 2.2.1 Problems With Initial System Startup 2.2.1.1 Checklist 2.2.2 Resetting the Server 2.2.2.1 Shutdown Commands for Software Menu Note 2.2.2.2 Shutdown Commands for Hardware LCD Display 2.2.3 Preparing the System for Diagnostic Testing Caution 2.2.3.1 Specific Problems and Corrective Actions Problems Starting Up Server Does Not Power On Caution Front Panel is Unresponsive and Video is Disabled Note Server Beeps at Power On or When Booting Note Server Starts Booting Automatically at Power On Power-On Self-Test (POST) Verifying Proper Operation of Key System LEDs 2.2.3.2 Power LED Does Not Light 2.2.3.3 System Cooling Fans Do Not Rotate Properly 2.2.3.4 Cannot Connect to a Server 2.2.3.5 Problems with Network 2.2.3.6 Other Problems 2.3 Troubleshooting the Server Using Built- In Tools 2.4 Diagnosing System Errors 2.4.1 LEDs 2.4.2 Beep Codes 2.4.3 POST Screen Messages 2.5 LEDs and Pushbuttons Note 2.5.1 Front Panel LEDs and Pushbuttons 2.5.1.1 Front Panel LEDs The front panel LEDs are summarized in Table2-4. Page 2.5.1.2 Front Panel Pushbuttons The front panel pushbuttons are summarized in Table2-5. 2.5.2 Rear Panel LEDs The rear panel contains the LEDs shown in Figure2-2. Page 2.5.3 Front-Panel System Status LED System Status LED Critical Condition Non-Critical Condition Degraded Condition 2.5.4 Rear Panel Power Supply Status LED Note 2.5.5 Server Main Board Fault LEDs Page 2.5.6 System ID LEDs 2.6 Power-On Self Test (POST) 2.6.1 POST Screen Messages Note Page Page 2.6.2 POST Error Beep Codes Page 2.6.2.1 BIOS Recovery Beep Codes 2.6.3 POST Progress Code LED Indicators Note Page Page Page Page 2.7 OS Operations 2.7.1 Filesystem Check (fsck) Procedure 2.7.2 StorEdge Network Capture Utility 2.7.3 Upgrades 2.7.4 Cacls - Access Control List 2.7.5 Proc filesystem 2.7.6 FTP Server 2.8 Updating the OS on the Sun StorEdge 5310 NAS 2.9 Sun StorEdge 5310 NAS Firmware 2.9.1 Operating System 2.9.1.1 Upgrading the StorEdge Operating System Important 2.10 Common Problems Encountered on the Sun StorEdge 5310 NAS 2.11 CIFS/SMB/Domain Changes to Windows group membership do not take effect. Changes to user mapping do not take effect. Cannot join Windows Domain. Note Page Page Page Cannot connect or authenticate to Windows 2003 Domain Controller. Lost Connection with Windows Domain. Cannot set share security, all shares inherit the security of the directory object. StorEdge has same files in 2 different shares. User maps are incorrect. User maps are not automatically created. Cant copy greater than 4G file from Windows to StorEdge. Cant map drives via CIFS/SMB. Cant set Windows security at the root of a volume or at the base of a share. Cannot see the security tab from Windows clients. Windows anti-virus, backup or file management software runs endlessly, following symbolic links. CIFS/SMB share created to /cvol is not visible or accessible. Having problems with Outlook .pst files stored on the StorEdge. CIFS/SMB disconnects from MS SQL Server. CIFS/SMB disconnects from MS Access. How can I check/modify which Domain Controller StorEdge is using for authentication? Users from trusted domains cannot access StorEdge Connection to SAMBA domain controller fails. Dial-up CIFS/SMB clients cannot connect to StorEdge. Windows local groups cannot be added to Access Control List. Note Log Message: Share database corrupt. Cant create new DTQ Files moved from DTQ to another directory get new time stamps and copy more slowly. 2.12 NFS Issues NFS root user doesn't have appropriate access. NFS root user can't change ownership. NFS root user can't change security. Trusted host does not have root access. Local NIS files are no longer updating. Windows created files are root owned when viewed via NFS. (In Windows Domain mode) Windows created files are root owned when viewed via NFS. (In Workgroup mode) International NFS filenames are garbled or cannot be read from Windows. (Or vice versa.) GID for new NFS objects is incorrect. StorEdge doesnt recognize the set GID bit. Chown by root fails if it would put target user over hard quota. NFS User cant access his own files created with CIFS/SMB account. 2.13 Network Issues When is it necessary to add a TCP/IP route? How do I manually add a TCP/IP route? Note 2.13.1 NIC speed and duplex negotiation issues. StorEdge is reporting Ethernet transmit and receive errors on a switched network. Note Default gateway changes without user input. How can I disable RIP routing? StorEdge is only reachable from systems on the local subnet. Multiple NICs are installed, but all outbound traffic is being sent through a single NIC. 2.14 File System Issues Note File system inaccessible (mount failure) Cant write to file system File system related error messages Reoccurrence of filesystem related error messages / mount problems after repair Checkpoint database problems reported in system log Cant delete checkpoints 2.15 Drive Failure Messages Note The light on one of the hard drives is red. Drive failure in log or an email received stating drive has failed or is about to fail. A drive has failed how do I replace it? Log message: LUN critical. Email alert: LUN critical. 2.16 File and Volume Operations StorEdge doesnt allow creation of volumes larger than 256 GB. Free space not immediately available after delete. After deleting files, volume free space remains the same. Cant delete a file. Cant delete files from /etc directory. Log message mbtowc[0xXX]: invalid first byte, or : invalid sequence. 2.17 Administration Interfaces Can't run GUI, Java certificate expired message. Can't run StorEdge Web Admin or some screens incorrect. StorEdge Web Admin missing ICONS or GUI does not respond Web GUI login failure. Web GUI session aborted while performing administration. How do I reset the StorEdge administrator password? StorEdge Web Admin does not work properly through a firewall. The keyboard arrows keys do not respond properly when using telnet to the StorEdge Having problems after software upgrade, how do I return to the previous version? After software upgrade and reboot, system appears to hang, LCD displays booting. 2.18 StorEdge Features and Utilities StorEdge fails to create scheduled checkpoints Previously created checkpoints are missing. StorEdge fails to send diagnostic email 2.19 Hardware Warning Messages Log message: Controller write-back cache is disabled. Log message: System on battery backup. Log message: Low Battery. Log message: Controller write-back cache is enabled. Log Message: Blower Fan has failed. UPS Messages Front LCD panel message: P21 Power 1 Failed. The light on one of the power supplies is amber. What do the Status LED Indicators on front panel indicate? LED Status indicators at the front panel signal current activities taking place in the system.. 2.20 Backup Issues Tape library not recognized. Network backup fails due to .attic$ directory. NDMP backup fails: access denied message. NDMP: Cant browse backup history. NDMP incremental/differential backups back up all files. 2.21 Direct Attached Tape Libraries Tape library not recognized. Network backup fails due to .attic$ directory. NDMP backup fails: access denied message. Local backup or restore fails with ?PNReduce error? in log. NDMP: Can?t browse backup history. NDMP incremental/differential backups back up all files. 2.22 Frequently Asked Questions 2.23 CIFS/SMB/Domain Issues How do I configure local SMB groups? Page Note How do I share files with SMB users? How do I create SMB shares? Note How do I create hidden shares? What are the default SMB shares? How do I use SMB administrative shares? What does the umask setting do? How do I set up user and group credential mapping? How do I share files between NFS and SMB users? Page How do I modify existing user and group credential mappings? Page How do I set up the SMB Autohome directory feature? How do I configure StorEdge to authenticate to a Windows Domain? How do I set up Active Directory? How do I set up dynamic DNS? Page What are the limitations of Workgroup mode? Does StorEdge support Domain Local Groups? Can StorEdge serve as a domain controller in a Windows Domain? Does StorEdge support DFS? What support is there for Windows Server Manager? 2.24 NIS/NIS+ Issues How do I set up NIS or NIS+? How do I configure the NS lookup order? 2.25 TCP/IP and Network Configuration What is port aggregation? How do I set up port aggregation? How do I disable port aggregation? What is IP aliasing? Important How do I set up IP aliasing? 2.26 Quota Configuration Page How do I configure user and group quotas at the CLI? How do I set up Directory Tree Quotas (DTQs)? Note How do I create a DTQ for an existing directory? How do I disable quotas? Important Note Whats the difference between a hard quota and a soft quota? 2.27 Checkpoint Configuration What are checkpoints? How do I set up checkpoints? Page How do I manage checkpoints from the command line? How do I disable checkpoints? How do I change the name of automatically created checkpoints? How do I recover deleted files from a checkpoint? 2.28 Volume Creation and Expansion How do I create a volume? How do I extend the size of an existing volume? How do I rename an existing volume? How do I delete an existing volume? 2.29 Reserved Filesystems and Directories What is the function of the .attic$ directory? What is the function of the /etc directory? What is the function of the /proc volume? What is the function of the /cvol volume? 2.30 NFS Issues How do I share files with NFS users? Note How do I authorize an entire subnet as trusted hosts? How do I manage NFS exports via the StorEdge Web Admin? Important How do I manage NFS exports via the configuration files? Important 2.31 Administration Interfaces and Utilities How do I access the StorEdge administration interfaces? What is the default administrator password? How do I configure an administrator password? How do I collect diagnostics from StorEdge? How do I set up SMTP (email)? How do I set up local or remote logging? Page How do I set up NTP or RDATE? Note How can I capture network traffic to and from the StorEdge? How do I access command history at the CLI? Page How do I delete files from the StorEdge administration utilities? Note How do I recursively delete directories from the StorEdge administration utilities? How do I set up an FTP server on StorEdge? How do I configure the FTP service to load automatically? How do I enable server-to-server FTP copying on the StorEdge? How do I configure and secure rsh access? How do I access a shell prompt from the StorEdge CLI? How do I enable or disable ftp, tftp, rlogin, rsh, telnet, ssh, smb or the Web Admin? 2.32 Backup and Migration Issues What steps do I need to take before migrating data to StorEdge? Page What the best way to migrate data to or from StorEdge? How do I set up NDMP to backup the StorEdge? How do I find the names of NDMP devices connected to the StorEdge? How do I view the NDMP backup log? 2.33 Macintosh Connectivity 2.34 Miscellaneous Log Messages 2.35 Direct Attached Tape Libraries 2.35.1 SCSI ID Settings 2.36 StorEdge File Replicator How Does File Replicator work? What are the applications for File Replicator? Disaster Recovery Backup Data Distribution How do I set up File Replicator? Page 2.37 StorEdge File Replicator Issues Configuration Page Page Storage Arrays 3.1 Fibre Channel FC 3.1.1 Array Overview 3.1.1.1 Module Description 3.1.1.2 Controllers 3.1.1.3 Controller Battery 3.1.1.4 Controller Memory Note 3.1.1.5 Drives 3.1.1.6 Fans 3.1.1.7 Power Supplies 3.1.1.8 SFP Transceivers and Cables 3.1.1.9 Tray ID Switch Note 3.1.2 Using the Array 3.1.2.1 Removing and Replacing the Back Cover Caution 3.1.2.2 Turning On the Power Caution Page Note 3.1.2.3 Turning Off Power for a Planned Shutdown Note Page Page Caution 3.1.2.4 Turning Off Power for an Unplanned Shutdown Caution Note Caution 3.1.2.5 Restoring Power After an Unplanned Shutdown Caution Note 3.1.2.6 Responding to the Optional Audible Alarm Page 3.2 Troubleshooting and Recovery 3.2.1 Troubleshooting the Module Page Page Page 3.2.2 Recovering from an Overheated Power Supply Caution Page 3.2.3 Setting the Tray ID Switch Caution Note 3.2.4 Verifying the Link Rate Setting 3.3 Relocating a Command Module 3.3.1 Upgrade Requirements Caution 3.3.1.1 Upgrade Methods Adding New Drives Replace Existing Drives with Greater Capacity Drives Caution Replace All Drives at the Same Time 3.3.2 Adding New Drives to Empty Slots Caution Note Caution Page 3.3.3 Replacing All Drives at the Same Time Caution Note 3.3.4 Replacing One Drive at a Time Caution Note Caution Note 3.3.5 Relocation Considerations Caution 3.3.5.1 Relocate Drives or Modules Note 3.3.5.2 Convert an Command Module to a Drive Module 3.3.6 Raid Storage Manager (RSM) Page 3.3.6.2 The Enterprise Management Window Device Tree Device Table Enterprise Management Window Menus 3.3.6.3 The Array Management Window Enterprise Management Window Toolbar The Enterprise Management Window toolbar buttons are described in Table3-5. Note Array Management Window Tabs Page Array Management Window Menus Page Array Management Window Toolbar 3.3.6.4 Protecting Your Data 3.3.6.5 Software Redundancy Volume Copy Remote Volume Mirroring Snapshot Volumes 3.3.6.6 RAID Levels and Data Redundancy Page Page Page 3.3.6.7 Hardware Redundancy Controller Cache Memory Caution Hot Spare Drives Background Media Scan Channel Protection 3.3.6.8 I/O Data Path Protection Note Multi-Path Driver with AVT Enabled Multi-Path Driver with AVT Disabled 3.3.6.9 Password Protection Note Password Failure Reporting and Lockout Note 3.3.6.10 Configuring Storage Arrays 3.3.6.11 Volumes and Volume Groups Volume Groups 3.3.6.12 Standard Volumes Note Specifying Volume Parameters from Free Capacity Note Specifying Volume Parameters from Unconfigured Capacity Note 3.3.6.13 Mappings View Page Page Register the Volume with the Operating System Note 3.3.6.14 SANshare Storage Partitioning Note SANshare Storage Partitioning Example Note 3.3.6.15 Heterogeneous Hosts Heterogeneous Hosts Example Note Snapshot Volumes Page Dynamic Volume Expansion (DVE) Caution Remote Volume Mirroring Page Data Replication Volume Copy Note Caution Note Caution 3.3.6.16 Managing Persistent Reservations Caution 3.3.6.17 Maintaining and Monitoring Storage Arrays Storage Array Health Note Storage Array Status Icons Event Monitor Page Page Note Problem Notification Page Page Page Page Note 3.4 Updating Firmware and NVSRAM on the Array Page Page Page 3.5 Updating ESM Firmware Page StorEdge File Replicator 4.1 Overview Page 4.1.1 Real-time Mirroring 4.1.2 Pseudo Real-time Mirroring 4.1.3 StorEdge File Replicator Page Page Page 4.1.4 Mirroring Variations 4.1.4.1 Many-to-One Mirroring 4.1.4.2 One-to-Many Mirroring 4.1.4.3 Piggyback or Cascading Mirroring 4.1.4.4 Bi-directional Mirroring 4.1.4.5 Replication 4.1.4.6 Push Technology 4.2 Operational State 4.2.1 Mirror Creation 4.2.2 Mirror Replication 4.2.3 Mirror Sequencing 4.2.4 Link Down and Idle Conditions 4.2.5 Cracked and Broken Mirrors Page Page Page Page Checkpoints/Snapshots 6.1 Overview 6.1.1 Volumes Page 6.1.2 Checkpoint Lifecycle 6.1.2.1 Checkpoint Creation 6.1.2.2 Active Checkpoint Page Page 6.1.2.3 Translation of File System Objects in Checkpoints checkpoint block 6.1.2.4 Checkpoint Deletion Page 6.1.2.5 Checkpoint Scheduling 6.1.2.6 Checkpoint management commands 6.1.2.7 Local Directory Checkpoint Access Accessing Checkpointed Data Page Compatibility Issues Page 6.1.3 Object Checkpoint Restore 6.1.4 StorEdge cp Command Page Page Page FRU/CRU Replacement Procedures Note 7.1 Tools and Supplies Needed 7.2 Determining a Faulty Component 7.3 Safety: Before You Remove the Cover 7.4 Removing and Replacing the Cover Page 7.5 Field Replaceable Unit (FRU) Procedures 7.6 NAS Head FRU Replacement Procedures 7.6.1 Opening the Front Bezel 7.6.2 Memory Caution Note Caution 7.6.3 Power Supply Unit Caution Warning 7.6.3.1 Sun StorEdge 5310 NAS Power Supply Page 7.6.4 Fan Module 7.6.4.1 Sun StorEdge 5310 NAS Fan Module Removal Page 7.6.4.2 Sun StorEdge 5310 NAS Fan Module Replacement 7.6.5 High Profile Riser PCI Cards 7.6.6 Gigabit Ethernet Card Page 7.6.7 Low Profile Riser PCI Cards 7.6.8 Qlogic HBA Removal and Replacement 7.6.9 LCD Display Module Note Caution 7.6.10 Flash Disk Module 7.6.10.1 Backup of /dvol/etc 7.6.10.2 Replacing the Flash Disk Note Caution Note 7.6.10.3 Upgrade and Configuration Recovery Note 7.6.11 System FRU (Super FRU) Caution 7.6.11.1 Super FRU Installation 7.7 Array FRU replacement Procedures 7.7.1 Replacing a Controller Note 7.7.1.1 Tools and Equipment 7.7.1.2 Procedure Page Page Page Caution Note 7.7.2 Replacing a Controller Battery 7.7.2.1 Tools and Equipment 7.7.2.2 Procedure Page Page Page Caution Note Caution 7.7.3 Replacing a Drive 7.7.3.1 Tools and Equipment 7.7.3.2 Procedure Page Caution Note 7.7.4 Replacing a Fan 7.7.4.1 Tools and Equipment 7.7.4.2 Procedure Page 7.7.5 Replacing a Power Supply 7.7.5.1 Tools and Equipment 7.7.5.2 Procedure Caution Page 7.7.6 Replacing an SFP Transceiver 7.7.6.1 Tools and Equipment 7.7.6.2 Procedure Caution