Dissecting and removing the SHV5 rootkit

Yesterday I noticed some odd behavior on one of the Linux servers that I maintain. For one thing, every time I would run ‘top’ or ‘ps’, I would see the following message

Unknown Hz value! (75) Assume 100

I decided to run rkhunter to check for rootkits. Needless to say, I was disappointed to discover that I had the SHV5 Rootkit on my machine. Let me tell you, it’s not a good feeling to know you’ve been hacked. Now I know common wisdom dictates that rootkit-infected machines be formatted and reloaded from scratch but that is not something you do without a lot of planning and preparation. The first thing I wanted to do was get this bad guy off my machine and try to minimize the damage.

How it works

I did some quick googling and I discovered this page which was very helpful. However, I wanted to add a few more details that might help somebody out in the future. Basically what I discovered is that the rootkit installs an SSH server as a backdoor for the hacker to obtain access to your machine. In my case it was running on port 6522 but this is configurable by the hacker so your mileage my vary.

The hacker does not need to know any passwords to login to your system because the rootkit installs a public key file. The hacker simply needs to have the corresponding private key to authenticate to your server. Unfortunately all of this was invisible to me because the rootkit modifies several core utilities such as ‘ls’ and ‘netstat’ to hide it’s presence. The modified versions of these utilities work just like the real ones, except they don’t display anything that might be incriminating such as certain files and folders, or tell-tale processes.

Cleaning it up

The first thing you have to do before you can start cleaning up a rootkit infection is to get your original system files back. Luckily, the rootkit makes this rather easy by setting the immutable bit on the modified files. The purpose is to prevent anyone from deleting or overwriting the hacked files but it also gives us an easy way to tell what’s been modified. After you run rkhunter, there should be a log file in /var/log called rkhunter.log. Search through this log file to see what files have the immutable bit set. These are the ones we need to replace. I’ll use /bin/ls in my examples.

First, unset the immutable flag by using this command:

chattr -sia /bin/ls

Then, you can optionally make a copy of the hacked file for future use. This is helpful to compare the output from the hacked version to the output of the real version to see what’s different.

mv /bin/ls /bin/ls.hacked

Next, figure out which RPM your file belongs to by running this:

rpm -qif /bin/ls

This will tell you the source RPM. To get the binary RPM just replace ‘.src.rpm’ with ‘.i386.rpm’. For example, on my machine, the source RPM for /bin/ls is:

coreutils-5.97-19.el5.src.rpm

So the file I needed to download was:

coreutils-5.97-19.el5.i386.rpm

You can Google the name of this file or look on popular RPM sites such as rpmfind.net or rpm.pbone.net. Once you are sure you have the correct file, install it using the –force command.

rpm -i –force coreutils-5.97-19.el5.i386.rpm

This tells the installer to try to reinstall the package even though it’s already installed and to replace files as necessary. If you get any errors about files that it can’t  replace, it’s probably because those files have the immutable bit set as well. Clear the immutable bit on those files and install the package again until there are no errors.

Repeat this whole process for every file in rkhunter.log that has the immutable bit set. You will probably have to download several different packages.

Now that you have reliable system commands you can begin ripping out the rootkit. Use netstat to see what ports are open and look for anything suspicious.

netstat -ln –programs

This is where it’s handy to have the output from the hacked version of netstat so you can compare. In my case, I found a program called ttyload that was listening on port 6522. I killed the process and deleted the file. After that I confirmed that my server was no longer accessible on that port. I also used the information in rkhunter.log to locate and delete a number of other files and folders as well. Here’s a list of some of the more interesting ones.

Make sure you check your /etc/inittab and all the files in /etc/rc.d to see if the rootkit is loading anything at startup. In my case I had to remove the line that calls /usr/sbin/ttyload from /etc/inittab.

Final thoughts

Here are some final thoughts on what to do next.

1. Change the root password and the passwords for any other privileged accounts you may have.
2. Reboot the server and make sure things are still normal after the system comes back up.
3. Use iptables to block incoming connections on all ports other than the ones you need.
4. Strongly consider a full format and reload of your system.
5. Run rkhunter on a regular basis