The iostat(1M) utility provides several I/O statistics, which can be useful for analyzing I/O workloads and troubleshooting performance problems. When reviewing I/O problems, I usually start by reviewing the number of reads and writes to a device, which are available in iostat’s “r/s” and “w/s” columns:
$ iostat -zxnM 5
extended device statistics
r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device
85.2 22.3 10.6 2.6 7.2 1.4 67.0 13.5 18 89 c0t0d0
Once I know how many reads and writes are being issued, I like to find the number of Megabytes read and written to each device. This information is available in iostat’s “Mr/s” and “Mw/s” columns:
$ iostat -zxnM 5
extended device statistics r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device 85.2 22.3 10.6 2.6 7.2 1.4 67.0 13.5 18 89 c0t0d0
After reviewing these items, I like to check iostat’s “wait” value to see the I/O queue depth for each device:
$ iostat -zxnM 5
extended device statistics
r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device
85.2 22.3 10.6 2.6 7.2 1.4 67.0 13.5 18 89 c0t0d0
To see how these can be applied to a real problem, I captured the following data from device c0t0d0 a week or two back:
$ iostat -zxnM 5
extended device statistics
r/s w/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device
0.2 71.2 0.0 7.7 787.3 2.0 11026.8 28.0 100 100 c0t0d0
Device c0t0d0 was overloaded, had 787 I/O operations waiting to be serviced, and was causing application latecy (since the application in question performed lots of reads/writes, and the files were open O_SYNC). Once iostat returned the above statistics, I used ps(1) to find the processes that were causing the excessive disk activity, and used kill(1) to terminate them!
* The Solaris iostat utility was used to produce this output.
** The first iostat line contains averages since the system was booted, and should be ignored.