A couple of weeks ago I promised some answers to the exercises I proposed at the end of my last post. What we have here is a case of, "Better late than never!"

1. If you go back and look at the example where I counted the number of processes per user, you'll notice that the "UID" header from the ps command ends up being counted. How would you suppress this?

There's a couple of different ways you could attack this using the material I showed you in the previous post. One way would be to do string comparison on field $1:

$ ps -ef | awk '$1 != "UID" {print $1}' | sort | uniq -c | sort -nr
    178 root
     58 hal
      2 www-data
    ...

An alternative approach would be to use pattern matching to print lines that don't match the string "UID". The "!" operator means "not", so the expression "!/UID/" does what we want:

$ ps -ef | awk '!/UID/ {print $1}' | sort | uniq -c | sort -nr
    178 root
     57 hal
      2 www-data
    ...

You'll notice that the "!/UID/" version counts one less process for user "hal" than the string comparison version. That's because the pattern match is matching the "UID" in the awk code and not showing you that process. So the string comparison version is slightly more accurate.

2. Print out the usernames of all accounts with superuser privileges (UID is 0 in /etc/passwd).

Remember that /etc/passwd file is colon-delimited, so we'll use awk's "-F" operator to split on colons. UID is field #3 and the username is field #1:

$ awk -F: '$3 == 0 {print $1}' /etc/passwd
root

Normally, a Unix-like OS will only have a single UID 0 account named "root". If you find other UID 0 accounts in your password file, they could be a sign that somebody's doing something naughty.

3. Print out the usernames of all accounts with null password fields in /etc/shadow.

You'll need to be root to do this one, since /etc/shadow is only readable by the superuser:

# awk -F: '$2 == "" {print $1}' /etc/shadow

Again, we use "-F:" to split the fields in /etc/shadow. We look for lines where the second field (containing the password hash) is the empty string and print the first field (the username) when this condition is true. It's really not much different from the previous /etc/passwd example.

You should get no output. There shouldn't be any entries in /etc/shadow with null password hashes!

4. Print out process data for all commands being run as root by interactive users on the system (HINT: If the command is interactive, then the "TTY" column will have something other than a "?" in it)

The "TTY" column in the "ps" output is field #6 and the username field is #1:

# ps -ef | awk '$1 == "root" && $6 != "?" {print}'
root      1422     1  0 Jan05 tty4     00:00:00 /sbin/getty -8 38400 tty4
root      1427     1  0 Jan05 tty5     00:00:00 /sbin/getty -8 38400 tty5
root      1434     1  0 Jan05 tty2     00:00:00 /sbin/getty -8 38400 tty2
root      1435     1  0 Jan05 tty3     00:00:00 /sbin/getty -8 38400 tty3
root      1438     1  0 Jan05 tty6     00:00:00 /sbin/getty -8 38400 tty6
root      1614  1523  0 Jan05 tty7     00:09:00 /usr/bin/X :0 -nr -verbose -auth ... 
root      2082     1  0 Jan05 tty1     00:00:00 /sbin/getty -8 38400 tty1
root      5909  5864  0 13:42 pts/3    00:00:00 su -
root      5938  5909  0 13:42 pts/3    00:00:00 -su
root      5968  5938  0 13:47 pts/3    00:00:00 ps -ef
root      5969  5938  0 13:47 pts/3    00:00:00 awk $1 == "root" && $6 != "?" {print}

We look for the keyword "root" in the first field, and anything that's not "?" in the sixth field. If both conditions are true, then we just print out the entire line with "{print}".

Actually, "{print}" is the default action for awk. So we could shorten our code just a bit:

# ps -ef | awk '$1 == "root" && $6 != "?"'
root      1422     1  0 Jan05 tty4     00:00:00 /sbin/getty -8 38400 tty4
root      1427     1  0 Jan05 tty5     00:00:00 /sbin/getty -8 38400 tty5
root      1434     1  0 Jan05 tty2     00:00:00 /sbin/getty -8 38400 tty2
...

5. I mentioned that if you kill all the sshd processes while logged in via SSH, you'll be kicked out of the box (you killed your own sshd process) and unable to log back in (you've killed the master SSH daemon). Fix the awk so that it only prints out the PIDs of SSH daemon processes that (a) don't belong to you, and (b) aren't the master SSH daemon (HINT: The master SSH daemon is the one who's parent process ID is 1).

This one's a little tricky. Take a look at the sshd processes on my system:

# ps -ef | grep sshd
root      3394     1  0  2012 ?        00:00:00 /usr/sbin/sshd
root     13248  3394  0 Jan05 ?        00:00:00 sshd: hal [priv] 
hal      13250 13248  0 Jan05 ?        00:00:02 sshd: hal@pts/0  
root     25189  3394  0 08:27 ?        00:00:00 sshd: hal [priv] 
hal      25191 25189  0 08:27 ?        00:00:00 sshd: hal@pts/1  
root     25835 25807  0 15:33 pts/1    00:00:00 grep sshd

For modern SSH daemons with "Privilege Separation" enabled, there are actually two sshd processes per login. There's a root-owned process marked as "sshd: <user> [priv]" and a process owned by the user marked as "sshd: <user>@<tty>". Life would be a whole lot easier if both processes were identified with the associated pty, but alas things didn't work out that way. So here's what I came up with:

# ps -ef | awk '/sshd/ && !($3 == 1 || /sshd: hal[@ ]/) {print $2}'

First we eliminate all processes except for the sshd processes with "/sshd/". We only want to print out the process IDs if it's not the master SSH daemon ("$3 == 1" to make sure the PPID isn't 1) or if it's not one of my SSH daemons ("/sshd: hal[@ ]/" means the string "sshd: hal" followed by either "@" or space). If everything looks good, then print the process ID of the process ("{print $2}").

Frankly, that's some pretty nasty awk. I'm not sure it's something I'd come up with easily on the spur of the moment.

6. Use awk to parse the output of the ifconfig command and print out the IP address of the local system.

Here's the output from ifconfig on my system:

$ ifconfig eth0
eth0      Link encap:Ethernet  HWaddr f0:de:f1:29:c7:18  
          inet addr:192.168.0.14  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::f2de:f1ff:fe29:c718/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:7724312 errors:0 dropped:0 overruns:0 frame:0
          TX packets:13553720 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100 
          RX bytes:711630936 (711.6 MB)  TX bytes:17529013051 (17.5 GB)
          Memory:f2500000-f2520000 

So this is a reasonable first approximation:

$ ifconfig eth0 | awk '/inet addr:/ {print $2}'
addr:192.168.0.14

The only problem is the "addr:" bit that's still hanging on. awk has a number of built-in functions, including substr() which can help us in this case:

$ ifconfig eth0 | awk '/inet addr:/ {print substr($2, 6)}'
192.168.0.14

substr() takes as arguments the string we're working on (field $2 in this case) and the place in the string where you want to start (for us, that's the sixth character so we skip over the "addr:"). There's an optional third argument which is the number of characters to grab. If you leave that off, then you just get the rest of the string, which is what we want here.

There are lots of other useful built-in functions in awk. Consult the manual page for further info.

7. Parse the output of "lsof -nPi" and output the unique process name, PID, user ID, and port combinations for all processes that are in "LISTEN" mode on ports on the system.

Let's take a look at the "lsof -nPi" output using awk to match only the lines for "LISTEN" mode:

# lsof -nPi | awk '/LISTEN/'
sshd      1216     root    3u  IPv4   5264      0t0  TCP *:22 (LISTEN)
sshd      1216     root    4u  IPv6   5266      0t0  TCP *:22 (LISTEN)
mysqld    1610    mysql   10u  IPv4   6146      0t0  TCP 127.0.0.1:3306 (LISTEN)
vmware-au 1804     root    8u  IPv4   6440      0t0  TCP *:902 (LISTEN)
cupsd     1879     root    6u  IPv6  73057      0t0  TCP [::1]:631 (LISTEN)
cupsd     1879     root    8u  IPv4  73058      0t0  TCP 127.0.0.1:631 (LISTEN)
apache2   1964     root    4u  IPv4   7412      0t0  TCP *:80 (LISTEN)
apache2   1964     root    5u  IPv4   7414      0t0  TCP *:443 (LISTEN)
apache2   4112 www-data    4u  IPv4   7412      0t0  TCP *:80 (LISTEN)
apache2   4112 www-data    5u  IPv4   7414      0t0  TCP *:443 (LISTEN)
apache2   4113 www-data    4u  IPv4   7412      0t0  TCP *:80 (LISTEN)
apache2   4113 www-data    5u  IPv4   7414      0t0  TCP *:443 (LISTEN)
skype     5133      hal   41u  IPv4 104783      0t0  TCP *:6553 (LISTEN)

Process name, PID, and process owner are fields 1-3 and the protocol and port are in fields 8-9. So that suggests the following awk:

# lsof -nPi | awk '/LISTEN/ {print $1, $2, $3, $8, $9}'
sshd 1216 root TCP *:22
sshd 1216 root TCP *:22
mysqld 1610 mysql TCP 127.0.0.1:3306
vmware-au 1804 root TCP *:902
cupsd 1879 root TCP [::1]:631
cupsd 1879 root TCP 127.0.0.1:631
apache2 1964 root TCP *:80
apache2 1964 root TCP *:443
apache2 4112 www-data TCP *:80
apache2 4112 www-data TCP *:443
apache2 4113 www-data TCP *:80
apache2 4113 www-data TCP *:443
skype 5133 hal TCP *:6553

And if we want the unique entries, then just use "sort -u":

# lsof -nPi | awk '/LISTEN/ {print $1, $2, $3, $8, $9}' | sort -u
apache2 1964 root TCP *:443
apache2 1964 root TCP *:80
apache2 4112 www-data TCP *:443
apache2 4112 www-data TCP *:80
apache2 4113 www-data TCP *:443
apache2 4113 www-data TCP *:80
cupsd 1879 root TCP 127.0.0.1:631
cupsd 1879 root TCP [::1]:631
mysqld 1610 mysql TCP 127.0.0.1:3306
skype 5133 hal TCP *:6553
sshd 1216 root TCP *:22
vmware-au 1804 root TCP *:902

Looking at the output, I'm not sure I care about all of the different apache2 instances. All I really want to know is which program is using port 80/tcp and 443/tcp. So perhaps we should just drop the PID and process owner:

# lsof -nPi | awk '/LISTEN/ {print $1, $8, $9}' | sort -u
apache2 TCP *:443
apache2 TCP *:80
cupsd TCP 127.0.0.1:631
cupsd TCP [::1]:631
mysqld TCP 127.0.0.1:3306
skype TCP *:6553
sshd TCP *:22
vmware-au TCP *:902

In the above output you see cupsd bound to both the IPv4 and IPv6 loopback address. If you just care about the port numbers, we can flash a little sed to clean things up:

# lsof -nPi | awk '/LISTEN/ {print $1, $8, $9}' | \
    sed 's/[^ ]*:\([0-9]*\)/\1/' | sort -u -n -k3
sshd TCP 22
apache2 TCP 80
apache2 TCP 443
cupsd TCP 631
vmware-au TCP 902
mysqld TCP 3306
skype TCP 6553

In the sed expression I'm matching "some non-space characters followed by a colon" ("[^ ]*:") with some digits afterwards ("[0-9]*"). The digits are the port number, so we replace the matching expression with just the port number. Notice I used "\(...\)" around the "[0-9]*" to create a sub-expression that I can substitute on the right-hand side as "\1".

I've also modified the final "sort" command so that we get a numeric ("-n") sort on the port number ("-k3" for the third column). That makes the output look more natural to me.

I guess the moral of the story here is that awk is good for many things, but not necessarily for everything. Don't forget that there are other standard commands like sed and sort that can help produce the output that you're looking for.

Happy awk-ing everyone!