echo «::memstat» | mdb -k
Page Summary Pages Bytes %Tot
—————————- —————- —————- —-
Kernel 2356318 17.9G 8%
Defdump prealloc 375179 2.8G 1%
ZFS 602721 4.5G 2%
Anon 5160416 39.3G 18%
Exec and libs 7942 62.0M 0%
Page cache 19943103 152.1G 69%
Free (cachelist) 82585 645.1M 0%
Free (freelist) 569720 4.3G 2%
Total 29097984 222G
How to Check and Analyze Memory Usage in Solaris
Troubleshooting performance issues is an important skill every system admin must have. This post is intended to give hints, where to look for in checking and troubleshooting memory usage. In principle, investigation of memory usage is split in checking usage of kernel memory and user memory.
Please be aware that in case of a memory-usage problem on a system, corrective actions usually requires deep knowledge and must be performed with great care. First, you need to check how much Memory is used in Kernel and how much is used in User Memory. This is important to decide, which further troubleshooting steps are required.
A very useful mdb dcmd is ‘::memstat‘ (this command can take several minutes to complete). Below example is from Solaris 11.2.
# echo "::memstat" | mdb -k Page Summary Pages Bytes %Tot ----------------- ---------------- ---------------- ---- Kernel 585584 4.4G 14% Defdump prealloc 204802 1.5G 5% Guest 0 0 0% ZFS Metadata 21436 167.4M 0% ZFS File Data 342833 2.6G 8% Anon 56636 442.4M 1% Exec and libs 1131 8.8M 0% Page cache 4339 33.8M 0% Free (cachelist) 8011 62.5M 0% Free (freelist) 2969532 22.6G 71% Total 4194304 32G
User memory usage : print out processes using most USER – memory
% prstat -s size – sorted by userland virtual memory consumption.
% prstat -s rss – sorted by userland physical memory consumption.
% prstat -s rss PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP 4051 user1 297M 258M sleep 59 0 1:35:05 0.0% mysqld/10 26286 user2 229M 180M sleep 59 0 0:05:07 0.0% java/53 27101user2 237M 150M sleep 59 0 0:02:21 0.0% soffice.bin/5 23335user2 193M 135M sleep 59 0 0:12:33 0.0% firefox-bin/10 3727 noaccess 192M 131M sleep 59 0 0:36:22 0.0% java/18 22751 root 165M 131M sleep 59 0 1:13:12 0.0% java/46 1448 noaccess 192M 108M sleep 59 0 0:34:47 0.0% java/18 10115 root 129M 82M sleep 59 0 0:31:29 0.0% java/41 20274 root 136M 77M stop 59 0 0:04:08 0.0% java/25 3397 root 138M 76M sleep 59 0 0:12:42 0.0% java/37 12949 pgsql 81M 70M sleep 59 0 0:09:36 0.0% postgres/1 12945 pgsql 80M 70M sleep 59 0 0:00:05 0.0% postgres/1
User Memory Usage : shows Shared Memory and Semaphores
% ipcs -a IPC status from T ID KEY MODE OWNER GROUP CREATOR CGROUP CBYTES QNUM QBYTES LSPID LRPID STIME RTIME CTIME Message Queues: q 0 0x55460272 -Rrw-rw---- root root root root 0 0 4194304 1390 18941 14:12:20 14:12:21 10:23:32 q 1 0x41460272 --rw-rw---- root root root root 0 0 4194304 5914 1390 8:03:34 8:03:34 10:23:39 q 2 0x4b460272 --rw-rw---- root root root root 0 0 4194304 0 0 no-entry no-entry 10:23:39 T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID LPID ATIME DTIME CTIME Shared Memory: m 0 0x50000b3f --rw-r--r-- root root root root 1 4 738 738 18:50:36 18:50:36 18:50:36 m 1 0x52574801 --rw-rw---- root oracle root oracle 35 1693450240 2049 26495 10:30:00 10:30:00 18:51:13 m 2 0x52574802 --rw-rw---- root oracle root oracle 35 1258291200 2049 26495 10:30:00 10:30:00 18:51:16 m 3 0x52594801 --rw-rw---- root oracle root oracle 12 241172480 2098 14328 7:58:33 7:58:33 18:51:27 m 4 0x52594802 --rw-rw---- root oracle root oracle 12 78643200 2098 14329 7:58:32 7:58:33 18:51:27 m 5 0x52584801 --rw-rw---- root oracle root oracle 13 125829120 2125 27492 1:36:12 1:36:12 18:51:34 m 6 0x52584802 --rw-rw---- root oracle root oracle 13 268435456 2125 27487 1:36:10 1:36:11 18:51:34 m 7 0x525a4801 --rw-rw---- root oracle root oracle 15 912261120 2160 27472 1:36:09 1:36:09 18:51:40 m 8 0x525a4802 --rw-rw---- root oracle root oracle 15 268435456 2160 27467 1:36:08 1:36:09 18:51:42 m 8201 0x4d2 --rw-rw-rw- root root root root 0 32008 1528 1543 10:26:03 10:26:04 10:25:53 T ID KEY MODE OWNER GROUP CREATOR CGROUP NSEMS OTIME CTIME Semaphores: s 0 0x1 --ra-ra-ra- root root root root 1 16:17:35 18:50:33 s 1 0 --ra-ra---- root oracle root oracle 36 10:33:28 18:51:17 s 2 0 --ra-ra---- root oracle root oracle 13 10:33:28 18:51:27 s 3 0 --ra-ra---- root oracle root oracle 14 10:33:28 18:51:34 s 4 0 --ra-ra---- root oracle root oracle 16 10:33:27 18:51:42 s 5 0x4d2 --ra-ra-ra- root root root root 1 no-entry 10:25:53 s 6 0x4d3 --ra-ra-ra- root root root root 1 no-entry 10:25:53
User Memory Usage : lists User Memory usage of all processes (except PID 0,2,3)
# pmap -x /proc/* > /var/tmp/pmap-x
short list of total usage of these processes
% egrep "[0-9]:|^total" /var/tmp/pmap-x
1: /sbin/init
total Kb 2336 2080 128 -
1006: rlogin cores4
total Kb 2216 1696 80 -
1007: rlogin cores4
total Kb 2216 1696 104 -
115: /usr/sbin/nscd
total Kb 4208 3784 1704 -
-- snip --
User Memory Usage : check the usage of /tmp
% df -kl /tmp Filesystem kbytes used avail capacity Mounted on swap 1355552 2072 1353480 1% /tmp
Print the biggest 10 files and dirs in /tmp:
% du -akd /tmp/ | sort -n | tail -10 288 /tmp/SUNWut 328 /tmp/log 576 /tmp/ips2 584 /tmp/explo 608 /tmp/ipso 3408 /tmp/sshd-truss.out 17992 /tmp/truss.p 22624 /tmp/js 49208 /tmp
User Memory Usage : Overall Memory usage on system
% vmstat -p 3
memory page executable anonymous filesystem
swap free re mf fr de sr epi epo epf api apo apf fpi fpo fpf
19680912 27487976 21 94 0 0 0 0 0 0 0 0 0 14 0 0
3577608 11959480 0 20 0 0 0 0 0 0 0 0 0 0 0 0
3577328 11959240 0 5 0 0 0 0 0 0 0 0 0 0 0 0
3577328 11959112 38 207 0 0 0 0 0 0 0 0 0 0 0 0
3577280 11958944 0 1 0 0 0 0 0 0 0 0 0 0 0 0
scanrate ‘sr’ should be 0 or near zero
User Memory Usage : Swap usage
% swap -l swapfile dev swaplo blocks free /dev/dsk/c0t0d0s1 32,25 16 1946032 1946032
% swap -s total: 399400k bytes allocated + 18152k reserved = 417552k used, 1355480k available
common kernel statistics
Print out all kernel statistics in a parse’able format.
% kstat -p > /var/tmp/kstat-p
kernel memory statistics
% kstat -p -c kmem_cache % kstat -p -m vmem % kstat -p -c vmem % kstat -p | egrep zfs_file_data_buf | egrep mem_total
alternatively to kstat you can get kernel memory usage with kmastat prints kmastat buffers.
# echo "::kmastat" | mdb -k > /var/tmp/kmastat
% more /var/tmp/kmastat
cache buf buf buf memory alloc lloc
name size in use total in use succeed fail
------------------------- ------ ------ ------ --------- --------- -----
kmem_magazine_1 16 470 508 8192 470 0
kmem_magazine_3 32 970 1016 32768 1164 0
kmem_magazine_7 64 1690 1778 114688 1715 0
Look for the highest numbers in column “memory in use” and for any numbers higher than ‘0’ in column “alloc fail”.