Americas

  • United States
sandra_henrystocker
Unix Dweeb

Half a dozen ways to look at Unix processes

How-To
Jan 31, 20167 mins
Data CenterLinux

On Unix/Linux systems, there are quite a few commands that can provide insights into the processes that are running, the resources those processes use, and the users responsible for them. Some of these are Unix 101 type commands. Others require the use of more exotic command options that you might not have tried.

Some of the things that you might want to know about the processes running on your servers are:

  • what processes are running
  • who’s running them
  • how much CPU time have they used
  • how long have they been running
  • what files they have open
  • what resources they are consuming now

So, let’s run through some handy ways to bring this kind of data into view.

What processes are running?

One of the first commands that Unix users learn is ps with and without arguments to show them what processes are running. Here are very simple examples showing just the current user’s processes. Notice that the ps command with the -u option displays one additional process — the sshd login process.

$ ps
  PID TTY          TIME CMD
10542 pts/0    00:00:00 bash
10573 pts/0    00:00:00 ps
$ ps -u `whoami`
  PID TTY          TIME CMD
10541 ?        00:00:00 sshd
10542 pts/0    00:00:00 bash
10575 pts/0    00:00:00 ps

Depending on what version of Unix you’re using, you’ll have various forms of ps and the information listed will be a little different. In the two command below, the ps axu (i.e., Berkely form of the command) shows some additional data fields.

$ ps -ef | head -6
UID    PID  PPID  C STIME TTY       TIME CMD
root     1     0  0  2015 ?     00:00:10 /sbin/init
root     2     0  0  2015 ?     00:00:00 [kthreadd]
root     3     2  0  2015 ?     00:00:04 [ksoftirqd/0]
root     5     2  0  2015 ?     00:00:00 [kworker/0:0H]
root     6     2  0  2015 ?     00:00:12 [kworker/u30:0]
$ ps axu | head -6
USER PID %CPU %MEM    VSZ   RSS TTY  STAT START  TIME COMMAND
root   1  0.0  0.1  19596  1616 ?    Ss    2015  0:10 /sbin/init
root   2  0.0  0.0      0     0 ?    S     2015  0:00 [kthreadd]
root   3  0.0  0.0      0     0 ?    S     2015  0:04 [ksoftirqd/0]
root   5  0.0  0.0      0     0 ?    S     2015  0:00 [kworker/0:0H]
root   6  0.0  0.0      0     0 ?    S     2015  0:12 [kworker/u30:0]

Note that the STIME and START fields in the output above only tell us that the processes were started last year. Other commands can provide details on the start time.

With certain options, the ps command can also show you some details of process heredity — in other words, details on what process started other processes. For example, in the truncated output below, you can see some of the processes that kreadd, init, and sshd started. The ps command that generated this output is near the bottom and five levels deep (note the indentation).

$ ps -ejH
  PID  PGID   SID TTY          TIME CMD
    2     0     0 ?        00:00:00 kthreadd
    3     0     0 ?        00:00:04   ksoftirqd/0
    5     0     0 ?        00:00:00   kworker/0:0H
...
    1     1     1 ?        00:00:10 init
 1380  1380  1380 ?        00:00:00   udevd
 1498  1380  1380 ?        00:00:00     udevd
 1499  1380  1380 ?        00:00:00     udevd
 1848  1848  1848 ?        00:00:03   dhclient
 1889  1889  1889 ?        00:01:12   auditd
 1907  1904  1678 ?        00:00:25   rsyslogd
...
22551 22551 22551 ?        00:00:55   sshd
10539 10539 10539 ?        00:00:00     sshd
10541 10539 10539 ?        00:00:00       sshd
10542 10542 10542 pts/0    00:00:00         bash
10670 10670 10542 pts/0    00:00:00           ps
 7449  7361  7054 ?        00:00:00   mysqld_safe
 7644  7361  7054 ?        01:06:43     mysqld

Another command for displaying the parentage of processes is pstree which shows the processes were spawned by other processes in a diagram like that shown below.

$ pstree
init─┬─agetty
     ├─atd
     ├─auditd───{auditd}
     ├─crond
     ├─dbus-daemon
     ├─dhclient
     ├─6*[mingetty]
     ├─ntpd
     ├─rngd
     ├─rsyslogd───3*[{rsyslogd}]
     ├─2*[sendmail]
     ├─sshd───sshd───sshd───bash───pstree
     └─udevd───2*[udevd]

What processes are really running?

The ps r command shows you tasks that are currently using the CPU. This information is only interesting if you have multiple CPUs. The r refers to running processes and, when you’re working on a system with a single CPU, the running process when the command is used is that process itself. Not much to excite anyone here:

$ ps r
  PID TTY      STAT   TIME COMMAND
11377 pts/0    R+     0:00 ps r

How much CPU time have processes consumed?

The TIME field is the amount of CPU time the process has accumulated, not how long it has been running. On a system that has been running a long time, you can get some interestingly large TIME values.

USER   PID %CPU %MEM    VSZ   RSS TTY  STAT START   TIME COMMAND
root     1  0.0  0.0  10344   688 ?    Ss    2013  39:50 init [5]
root     2  0.0  0.0      0     0 ?    S     2013   0:34 [migration/0]
root     3  0.0  0.0      0     0 ?    SN    2013   0:00 [ksoftirqd/0]
root 28406  0.7  0.9 2036884 347712 ?  Sl    2014 5056:03 java -Dmule.home=
root 28650  1.2  1.5 1754852 560012 ?  Sl    2014 8102:08 /usr/bin/jdk1.6.0

Who’s running the processes?

It’s easy to see from the ps output who is running each task as that’s the first field in the ps output, but if you want to count how many processes each user is running, you might want to use a command like the ps -eo that list just the usernames and then count processes by user.

$ ps -eo euser | sort | uniq -c
      1 dbus
      7 jdoe    
      1 EUSER
      1 ntp
     60 root
      1 smmsp

The appearance of “EUSER” in the output above would be the column header if we weren’t sorting the output. You can either ignore this or remove it from the ps output before you start counting up the fields.

$ ps -eo euser | grep -v EUSER | sort | uniq -c
      1 dbus
      6 jdoe    
      1 ntp
     54 root
      1 smmsp

To list in order of usage, we can just sort that output on the counts.

$ ps -eo euser | grep -v EUSER | sort | uniq -c | sort -nr
     54 root
      7 jdoe    
      1 smmsp
      1 ntp
      1 dbus

The euser argument in the ps command represents the effective username. These options would do much the same thing:

euser      effective user name
fuser      filesystem access user ID
ruser      real user ID
suser      saved user name

How long have processes been running?

The ps command on Linux has some nice options for looking at elapsed time. Even though the stime and time options may not tell you what you want to know, the etime option (see the third command below) provides that data.

$ ps -o stime,time 22551
STIME     TIME
 2015 00:00:55
$ ps -p "22551" -o etime=
175-05:02:22
$ ps -o etime,stime,time 22551
    ELAPSED STIME     TIME
175-05:04:23 2015 00:00:55

While it may not be obvious, that 175-05:04:23 value conforms to the overall [[dd-]hh:]mm:ss format. In other words, the first number is the number of days. Then we have hours, minutes, and seconds. Remember that the TIME field that we see in the more standard ps output represents only time on the CPU — a small portion of the time any particular task has been running.

What files do processes have open?

To view the files that any process has open, use the lsof (list open files) command. Without any arguments, the command will show you the files that every process has open — if you run it with superuser privilege.

$ sudo lsof | head -6
COMMAND  PID  USER   FD  TYPE  DEVICE SIZE/OFF    NODE NAME
init       1  root  cwd   DIR   202,1     4096       2 /
init       1  root  rtd   DIR   202,1     4096       2 /
init       1  root  txt   REG   202,1   150360  397362 /sbin/init
init       1  root  DEL   REG   202,1            96485 /lib64/libnss_files-2.17.so
init       1  root  DEL   REG   202,1           396493 /lib64/libpthread-2.17.so

You can also look at a single process but, depending on the process, you might still see a huge number of open files.

$ sudo lsof -p $$
COMMAND  PID  USER   FD  TYPE DEVICE  SIZE/OFF   NODE NAME
bash   11630  jdoe  cwd   DIR  202,1      4096 410779 /home/ec2-user
bash   11630  jdoe  rtd   DIR  202,1      4096      2 /
bash   11630  jdoe  txt   REG  202,1    898032 396531 /bin/bash
bash   11630  jdoe  mem   REG  202,1 106065056 410803 /usr/lib/locale/locale-archive
bash   11630  jdoe  mem   REG  202,1     58288 396484 /lib64/libnss_files-2.17.so
bash   11630  jdoe  mem   REG  202,1   2107600 396466 /lib64/libc-2.17.so
bash   11630  jdoe  mem   REG  202,1     19512 396472 /lib64/libdl-2.17.so
bash   11630  jdoe  mem   REG  202,1    135616 396516 /lib64/libtinfo.so.5.7
bash   11630  jdoe  mem   REG  202,1    160240 396459 /lib64/ld-2.17.so
bash   11630  jdoe  mem   REG  202,1     26254   1596 /usr/lib64/gconv/gconv-modules.cache
bash   11630  jdoe    0u  CHR  136,0       0t0      3 /dev/pts/0
bash   11630  jdoe    1u  CHR  136,0       0t0      3 /dev/pts/0
bash   11630  jdoe    2u  CHR  136,0       0t0      3 /dev/pts/0
bash   11630  jdoe  255u  CHR  136,0       0t0      3 /dev/pts/0

The number of shared object files that system processes use will generally be much longer than what you see above — a command that is looking only at the current process.

What resources are processes consuming now?

While some forms of the ps command display the percentage of memory and CPU time processes are taking, I still prefer looking at these statistics with top if looking to see which processes are using the most of these resources.

$ top -n 1
top - 00:38:43 up 175 days, 11:09,  1 user,  load average: 0.00, 0.01, 0.05
Tasks:  60 total,   1 running,  59 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1020188k total,   864620k used,   155568k free,   159528k buffers
Swap:        0k total,        0k used,        0k free,   568356k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
    1 root      20   0 19596 1616 1292 S  0.0  0.2   0:10.15 init
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthreadd
    3 root      20   0     0    0    0 S  0.0  0.0   0:04.25 ksoftirqd/0
    5 root       0 -20     0    0    0 S  0.0  0.0   0:00.00 kworker/0:0H
    6 root      20   0     0    0    0 S  0.0  0.0   0:12.87 kworker/u30:0

Wrap up

That’s a nice handful of commands for looking at processes and we haven’t even started looking at all there is to see in the /proc file system. Maybe next time!

sandra_henrystocker
Unix Dweeb

Sandra Henry-Stocker has been administering Unix systems for more than 30 years. She describes herself as "USL" (Unix as a second language) but remembers enough English to write books and buy groceries. She lives in the mountains in Virginia where, when not working with or writing about Unix, she's chasing the bears away from her bird feeders.

The opinions expressed in this blog are those of Sandra Henry-Stocker and do not necessarily represent those of IDG Communications, Inc., its parent, subsidiary or affiliated companies.