On Thu, 2007-02-01 at 17:58 -0600, Chris Mohler wrote: > Well, through no one's fault but my own our file server has been compromised. > > It looks like the SHV5 kit. I plan a reformat/reinstall tomorrow and > I was wondering if anyone had advice. I discovered that some of the > coreutils had been replaced with compromised versions, so I (stupidly) > downloaded the coreutils RPM, then did 'rpm -ev coreutils' and tried > 'rpm --Uvh coreutils'. Should have researched that a bit, because (as > root) I don't have permission to remove/rename the hacked binaries! The standard rootkit thing is to "chattr +us" the files so you can't delete them in a standard way. To fix it, boot off the first CD in rescue mode and let the system mount your / volume. Then: cd /mnt/sysimage chattr -R -us sbin/* chattr -R -us bin/* mount (path to your /usr partition) usr cd usr chattr -R -us sbin/* chattr -R -us bin/* That should allow you to delete the hacked versions of the various files. Next, take a REAL CAREFUL look at the /mnt/sysimage/etc/rc.d directory tree and make SURE there's no items in there that start up the hidden binaries. Often, those binaries are buried in the /dev directory and are given non-displayable names (non-graphic characters, names that start with ".", etc.) > Oops. For the time being, I've (physically) removed the server's > network connection. Good call! > So - the plan: > 1. telinit 1 > 2. try to reinstall coreutils > 3. telinit 3 I suggest doing what I mentioned above. Then reboot the machine in single user mode (append "single" to the "kernel" line in grub), reinstall the coreutils and THEN "telinit 3" if you must. There's really no need to telinit 3--you can get the network up with a simple "service network start" from single user mode. > 4. rsync the last week's worth of data to another machine > 5. reformat/reinstall > 6. create new home dirs > 7. rsync the data back - do a recursive chown/chmod > 8. run rkhunter > > Any thoughts on this plan of attack are welcome. You have mine. > And of course the moral of all of this is UPDATE and DON'T RUN > UNNEEDED WEB SERVICES. This happened on a FC2 server (I know ;) ), > and possibly via the SWAT or phpMyAdmin web interfaces. Evil! I always iptable the hell out a box with any form of outside management stuff. Also, once you've got the machine up and BEFORE you plug in the net cable: 1. Take a good hard look at what's running with "ps ax". Turn off ANYTHING you don't need (sendmail, nfs, snmpd, samba, etc.). 2. Do a "netstat -pan" and look at what network ports are active and make sure you know what they are and that they should be there. 3. Run an nmap against the primary IP address of the machine (not 127.0.0.1, but the one that ties it to the network) and make sure there aren't any open ports you don't know about. 4. Set up iptables and firewall the hell out of the machine. Only allow webmin access to the machines you use to manage it. Block ssh except from your management machines. Set up tripwire. You get the idea. As someone once observed, "The Internet is rough. Wear a cup!" ---------------------------------------------------------------------- - Rick Stevens, Senior Systems Engineer rstevens@xxxxxxxxxxxxxxx - - VitalStream, Inc. http://www.vitalstream.com - - - - Hard work has a future payoff. Laziness pays off now. - ----------------------------------------------------------------------
