IBM REDP-4285-00 iostat command

4285ch03.fm Draft Document for Review May 4, 2007 11:35 am

86 Linux Performance and Tuning Guidelines

Example 3-2 vmstat output

[root@x232 root]# vmstat 2

r b swpd free buff cache si so bi bo in cs us sy id wa

2 1 0 9004 47196 1141672 0 0 0 950 149 74 87 13 0 0

0 2 0 9672 47224 1140924 0 0 12 42392 189 65 88 10 0 1

0 2 0 9276 47224 1141308 0 0 448 0 144 28 0 0 0 100

0 2 0 9160 47224 1141424 0 0 448 1764 149 66 0 1 0 99

0 2 0 9272 47224 1141280 0 0 448 60 155 46 0 1 0 99

0 2 0 9180 47228 1141360 0 0 6208 10730 425 413 0 3 0 97

1 0 0 9200 47228 1141340 0 0 11200 6 631 737 0 6 0 94

1 0 0 9756 47228 1140784 0 0 12224 3632 684 763 0 11 0 89

0 2 0 9448 47228 1141092 0 0 5824 25328 403 373 0 3 0 97

0 2 0 9740 47228 1140832 0 0 640 0 159 31 0 0 0 100

iostat command

Performance problems can be encountered when too many files are opened, being read and

written to, then closed repeatedly. This could become apparent as seek times (the time it

takes to move to the exact track where the data is stored) start to increase. Using the iostat

tool, you can monitor the I/O device loading in real time. Different options enable you to drill

down even farther to gather the necessary data.

Example 3-3 shows a potential I/O bottleneck on the device /dev/sdb1. This output shows

average wait times (await) of about 2.7 seconds and service times (svctm) of 270 ms.

Example 3-3 Sample of an I/O bottleneck as shown with iostat 2 -x /dev/sdb1

[root@x232 root]# iostat 2 -x /dev/sdb1

avg-cpu: %user %nice %sys %idle

11.50 0.00 2.00 86.50

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz

avgqu-sz await svctm %util

/dev/sdb1 441.00 3030.00 7.00 30.50 3584.00 24480.00 1792.00 12240.00 748.37

101.70 2717.33 266.67 100.00

avg-cpu: %user %nice %sys %idle

10.50 0.00 1.00 88.50

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz

avgqu-sz await svctm %util

/dev/sdb1 441.00 3030.00 7.00 30.00 3584.00 24480.00 1792.00 12240.00 758.49

101.65 2739.19 270.27 100.00

avg-cpu: %user %nice %sys %idle

10.95 0.00 1.00 88.06

Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz

avgqu-sz await svctm %util

/dev/sdb1 438.81 3165.67 6.97 30.35 3566.17 25576.12 1783.08 12788.06 781.01

101.69 2728.00 268.00 100.00

For a more detailed explanation of the fields, see the man page for iostat(1).

Contents

Main Page Page Page Contents Page Page Page Notices Trademarks Preface How this Redpaper is structured The team that wrote this Redpaper Become a published author Comments welcome Page operating system Page 1.1 Linux process management 1.1.1 What is a process? 1.1.2 Lifecycle of a process 1.1.3 Thread 1.1.4 Process priority and nice level Process priority 1.1.5 Context switching CPU 1.1.6 Interrupt handling Suspend Resume 1.1.7 Process state Processor Zombie processes 1.1.8 Process memory segments Text Data Stack Process address space 1.1.9 Linux CPU scheduler 4285ch01.fm 10 Figure 1-8 Linux kernel 2.6 O(1) scheduler via scheduler_tick() P active expired array[0] array[1] PP P active expired array[0] array[1] PP Parent Child 1.2 Linux memory architecture 1.2.1 Physical and virtual memory Virtual memory addressing layout 32-bit Architecture 64-bit Architecture User space Kernel space 32-bit Architecture 64-bit Architecture User space Kernel space 1.2.2 Virtual memory manager Page frame allocation Buddy system Page frame reclaiming cache and process address space. The page cache is pages mapped to a file on disk. The swap 1.3 Linux file systems 1.3.1 Virtual file system 1.3.2 Journaling 1. write journal logs write File system Journal area 1.3.3 Ext2 Ext2 1.3.4 Ext3 Mode of journaling 1.3.5 ReiserFS 1.3.6 Journal File System 1.3.7 XFS 1.4 Disk I/O subsystem 1.4.1 I/O subsystem architecture 1.4.2 Cache Memory hierarchy Locality of reference Flushing dirty buffer 1.4.3 Block layer Disk Cache Disk Cache Block sizes 1.4.4 I/O device driver SCSI ips qla2300mptscsih 1.4.5 RAID and Storage system st sr_modsd_modsg scsi_mod Upper level driver Mid level driver Device Process Low level driver 1.5 Network subsystem 1.5.1 Networking implementation Socket buffer TCP/IP IPX Appletalk DEVICE NAPI way NAPI way DEVICE Netfilter Netfilter Connection tracking 1.5.2 TCP/IP Connection establishment Chapter 1. Understanding the Linux operating system 31 Draft Document for Review May 4, 2007 11:35 am 4285ch01.fm Figure 1-27 TCP 3-way handshake Client Server 4285ch01.fm 32 Traffic control TCP/IP transfer window CLOSED LISTEN SYN SENT LAST-ACK ESTAB SYN RCVD 1.5.3 Offload 1.5.4 Bonding module 1.6 Understanding Linux performance metrics 1.6.1 Processor metrics 1.6.2 Memory metrics 1.6.3 Network interface metrics 1.6.4 Block device metrics transfers per second metric in conjunction with the kBytes per second value helps you to Page Page Page 2.1 Introduction 2.2 Overview of tool function 2.3 Monitoring tools 2.3.1 top 2.3.2 vmstat 2.3.3 uptime 2.3.4 ps and pstree Page Thread information 2.3.5 free Memory used in a zone 2.3.6 iostat Page 2.3.7 sar 2.3.8 mpstat 2.3.9 numastat 2.3.10 pmap 2.3.11 netstat 2.3.12 iptraf 2.3.13 tcpdump / ethereal tcpdump ethereal 2.3.14 nmon 2.3.15 strace 2.3.16 Proc file system Page 2.3.17 KDE System Guard Work space System Load Process Table Configuring a work sheet Page 2.3.18 Gnome System Monitor 2.3.19 Capacity Manager Page Page 2.4 Benchmark tools 2.4.1 LMbench 2.4.2 IOzone 2.4.3 netperf Page Page 2.4.4 Other useful tools bottlenecks 3.1 Identifying bottlenecks 3.1.1 Gathering information Page 3.1.2 Analyzing the servers performance 3.2 CPU bottlenecks 3.2.1 Finding CPU bottlenecks 3.2.2 SMP 3.2.3 Performance tuning options 3.3 Memory bottlenecks 3.3.1 Finding memory bottlenecks Paging and swapping indicators 3.3.2 Performance tuning options 3.4 Disk bottlenecks 3.4.1 Finding disk bottlenecks vmstat command iostat command 3.4.2 Performance tuning options 3.5 Network bottlenecks 3.5.1 Finding network bottlenecks Page 3.5.2 Performance tuning options Page Page 4.1 Tuning principals 4.1.1 Change management 4.2 Installation considerations 4.2.1 Installation Page 4.2.2 Check the current configuration dmesg Page ulimit 4.2.3 Minimize resource use Daemons Page Page Page Changing runlevels Limiting local terminals 4.2.4 SELinux SELinux Kernel Process User Request Access Grant Access Grant/Deny Access Based on Policy 4.3 Changing kernel parameters Page Page 4.3.1 Where the parameters are stored 4.3.2 Using the sysctl command 4.4 Tuning the processor subsystem 4.4.1 Tuning process priority 4.4.2 CPU affinity for interrupt handling 4.4.3 Considerations for NUMA systems numactl 4.5 Tuning the vm subsystem 4.5.1 Setting kernel swap and pdflush behavior 4.5.2 Swap partition 4.5.3 HugeTLBfs 4.6 Tuning the disk subsystem 4.6.1 Hardware considerations before installing Linux Number of drives Guidelines for setting up partitions 4.6.2 I/O elevator tuning and selection Selecting the right I/O elevator in kernel 2.6 Page Impact of nr_requests Page Impact of read_ahead_kb 4.6.3 File system selection and tuning Using ionice to assign I/O priority Access time updates Select the journaling mode of the file system Block sizes stripe size of the array (or segment in the case of Fibre Channel). The stripe-unit size is the 4.7 Tuning the network subsystem 4.7.1 Considerations of traffic characteristics 4.7.2 Speed and duplexing 4.7.3 MTU size 4.7.4 Increasing network buffers Tuning window sizes Page Impact of socket buffer size 4.7.5 Additional TCP/IP tuning Tuning IP and ICMP behavior Page Tuning TCP behavior Tuning TCP options Chapter 4. Tuning the operating system 133 Draft Document for Review May 4, 2007 11:35 am 4285ch04.fm Figure 4-19 Netfilter rules applied 4.7.6 Performance impact of Netfilter Figure 4-18 No Netfilter rule applied TCP_CRR benchmark TCP_CRR benckmark TCP_CRR benchmark 4.7.7 Offload configuration Impact of offloading Chapter 4. Tuning the operating system 135 Draft Document for Review May 4, 2007 11:35 am 4285ch04.fm Figure 4-21 Throughput degradation by offloading Figure 4-20 CPU usage improvement by offloading cpu usage improvement - default vs offload off 4.7.8 Increasing the packet queues 4.7.9 Increasing the transmit queue length 4.7.10 Decreasing interrupts Page Page Page 4285ax01.fm 140 Hardware and software configurations Linux installed on guest IBM z/VM systems Linux installed on IBM System x servers Three IBM System x x236 servers were configured as shown in TableA-2. Page Page Abbreviations and acronyms 4285abrv.fm 144 Related publications IBM Redbooks Other publications Online resources Page How to get IBM Redbooks Help from IBM Page Index Symbols Numerics A B E F G H I N O P 4285IX.fm 152 Q R S T U V W X Z Page Page Redpaper Back cover INTERNATIONAL TECHNICAL SUPPORT ORGANIZATION Linux Performance and Tuning Guidelines