On Tue, 2006-11-21 at 12:55 -0600, olga@xxxxxxxxxxxxxx wrote: > > El Jueves, 16 de Noviembre de 2006 22:56, olga@xxxxxxxxxxxxxx escribió: > >> > On Thu, 2006-11-16 at 10:26 -0600, olga@xxxxxxxxxxxxxx wrote: > >> >> Hi, > >> >> > >> >> I wrote about kernel errors which somebody pointed out was because > >> the > >> >> server was running out of memory. > >> >> > >> >> Now I found the following which makes me think that that server may > >> have > >> >> been compromized. > >> >> > >> >> Here's what I get when I issued: netstat -nap > >> >> > >> >> tcp 0 0 131.x.x.x:38423 72.x.x.x:80 ESTABLISHED > >> >> 5226/ps x > >> >> tcp 0 0 131.x.x.x:38420 72.x.x.x:80 ESTABLISHED > >> >> 5365/ps x > >> >> > >> >> About a hundred instances of that program 'ps x' running. > >> >> > >> >> Also here's what ps -ef produced: > >> >> > >> >> apache 6323 1 0 10:30 ? 00:00:00 ps x > >> >> apache 6324 1 0 10:30 ? 00:00:00 ps x > >> >> apache 6326 1 0 10:30 ? 00:00:00 ps x > >> >> apache 6328 1 0 10:30 ? 00:00:00 ps x > >> >> apache 6330 1 0 10:30 ? 00:00:00 ps x > >> > > >> > What does ls -l /proc/6323/exe say? That would be a symlink to the > >> > executable for that process. Normal ps lives in /bin so the link > >> should > >> > point at /bin/ps. If it is connecting out to a remote host, it's > >> likely > >> > not the normal ps, just something that's masking itself to make it > >> less > >> > likely to get picked up. > >> > > >> > -- > >> > David Hollis <dhollis@xxxxxxxxxxxxxx> > >> > >> apache 3102 1 0 15:53 ? 00:00:00 httpd > >> apache 3104 1 0 15:53 ? 00:00:00 httpd > >> apache 3106 1 0 15:53 ? 00:00:00 httpd > >> apache 3108 1 0 15:53 ? 00:00:00 httpd > >> apache 3110 1 0 15:53 ? 00:00:00 httpd > >> apache 3112 1 0 15:53 ? 00:00:00 httpd > >> apache 3114 1 0 15:53 ? 00:00:00 httpd > >> apache 3116 1 0 15:53 ? 00:00:00 httpd > >> apache 3118 1 0 15:53 ? 00:00:00 httpd > >> apache 3120 1 0 15:53 ? 00:00:00 httpd > >> apache 3122 1 0 15:53 ? 00:00:00 httpd > >> apache 3125 1 0 15:54 ? 00:00:00 httpd > >> apache 3127 1 0 15:54 ? 00:00:00 httpd > >> apache 3129 1 0 15:54 ? 00:00:00 httpd > >> apache 3131 1 0 15:54 ? 00:00:00 httpd > >> apache 3133 1 0 15:54 ? 00:00:00 httpd > >> apache 3135 1 0 15:54 ? 00:00:00 httpd > >> apache 3137 1 0 15:54 ? 00:00:00 httpd > >> apache 3139 1 0 15:54 ? 00:00:00 httpd > >> apache 3141 1 0 15:54 ? 00:00:00 httpd > >> apache 3143 1 0 15:54 ? 00:00:00 httpd > >> apache 3145 1 0 15:54 ? 00:00:00 httpd > >> apache 3639 1 0 15:57 ? 00:00:00 ps x > >> apache 3642 1 0 15:57 ? 00:00:00 ps x > >> apache 3645 1 0 15:58 ? 00:00:00 ps x > >> apache 3647 1 0 15:58 ? 00:00:00 ps x > >> > >> > >> I am getting a ton of these... > >> Here's what ls -l /proc/3147/exe says > >> lrwxrwxrwx 1 apache apache 0 Nov 16 15:56 /proc/3147/exe > >> -> > >> /usr/bin/perl > >> > >> When I do netstat -nap I get: > >> tcp 0 0 131.x.x.x:44160 72.14.x.x:80 ESTABLISHED - > >> tcp 0 0 131.x.x.x:44161 72.14.x.x:80 ESTABLISHED - > >> tcp 0 0 131.x.x.x:44162 72.14.x.x:80 ESTABLISHED - > >> > >> The ip points to google... > >> > >> And these appeared in the /tmp folder: > >> > >> drwxrwxrwt 8 root root 4096 Nov 16 16:00 . > >> drwxr-xr-x 23 root root 4096 Nov 16 14:35 .. > >> srwx------ 1 root nobody 0 Nov 16 14:36 .fam_socket > >> drwxrwxrwt 2 xfs xfs 4096 Nov 16 14:35 .font-unix > >> srw-rw-rw- 1 root root 0 Nov 16 14:36 .gdm_socket > >> -rw-r--r-- 1 apache apache 0 Nov 15 15:20 .httpd > >> drwxrwxrwt 2 root root 4096 Nov 16 14:36 .ICE-unix > >> drwx------ 2 root root 4096 Nov 16 14:59 mc-root > >> drwx------ 2 root root 12288 Nov 16 15:16 orbit-root > >> -rw-r--r-- 1 apache apache 0 Nov 16 15:58 > >> sess_azx3a4wq3x1f2aad4a34sxx1w2o52a45 > >> -rw-r--r-- 1 apache apache 11669 Nov 16 15:43 > >> sess_rdav631df3a1ddfaa34s1x1wwo521459 > >> -r--r--r-- 1 root root 11 Nov 16 14:36 .X0-lock > >> drwxrwxrwt 2 root root 4096 Nov 16 14:36 .X11-unix > >> > >> What is going on? > >> > > > > Finally...did they break into your system? Did you find something strange > > on > > the logs? I wonder what happened, give us some information this thread is > > quite interesting and will help other folks in a near future ;-) > > One way or another, if they got shell access (even remote text shell, you > > know...) you should think about reinstalling your system, as far as i > > know, > > if the left a rootkit you must not trust your system anymore. > > > > By the way, let me give you and advice, installing Babel Enterprise could > > be a > > nice idea, ( http://babel.sourceforge.net/en/ ), yeah yeah, it's GPL ;-) > > > > Babel is an enterprise-grade auditing system to manage a consistency on > > security policy between different systems in a non-homogeneus > > architecture. > > Babel allows to manage very different operating systems, like AIX, > > Solaris, > > Windows 2000, Windows XP, Linux, *BSD or HPUX. > > > > Babel allows administrator team to monitor the hardening level of their > > systems and keep constantly monitored, using periodic policy polling, and > > of > > course, a WEB Based, graphical reporting, and of course, a centralized > > management for all systems > > > > There's a demo online, try it. > > > > Hope this helps. > > It does appear that there has been a break-in. Some kind of script was > running that was consuming all system resourses. At the time it was > running, it was also deleting log entries, so if I looked at the log and > searched for the time we brough the server up on the network, logs would > show no activity at that time. And that 72.x.x.x IP was probably bogus as > well. > > Here's what I found in the httpd error log: > > --06:31:56-- http://autocoutureinc.com/borek.txt > => `borek.txt' > Resolving autocoutureinc.com... 208.67.181.244 > Connecting to autocoutureinc.com|208.67.181.244|:80... connected. > HTTP request sent, awaiting response... 200 OK > Length: 11,666 (11K) [text/plain] > > 0K .......... . 100% 169.99 > KB/s > > 06:31:56 (169.99 KB/s) - `borek.txt' saved [11666/11666] > > Died at sess_rdav631df3a1ddfaa34s1x1w2o521459 line 24. > Died at sess_rdav631df3a1ddfaa34s1x1w2o521459 line 24. > rm: cannot remove `borek.txt*': No such file or directory > % Total % Received % Xferd Average Speed Time > Curr. > Dload Upload Total Current Left > Speed > 100 11666 100 11666 0 0 23100 0 0:00:00 0:00:00 0:00:00 > 156k > Died at sess_dda2631df3a1ddfaa34s1x1wwo521459 line 24. > Died at sess_dda2631df3a1ddfaa34s1x1wwo521459 line 24. > rm: cannot remove `borek.txt*': No such file or directory > Died at sess_edav631df3a15dfaa34s1x1wwo521459 line 24. > Died at sess_edav631df3a15dfaa34s1x1wwo521459 line 24. > sh: line 1: lynx: command not found > sh: line 1: fetch: command not found > Died at sess_tdx4d3td33a1ddfaa34s1x11x2521459 line 24. > Died at sess_tdx4d3td33a1ddfaa34s1x11x2521459 line 24. > --06:32:02-- http://autocoutureinc.com/borek.txt > => `borek.txt' > Resolving autocoutureinc.com... 208.67.181.244 > Connecting to autocoutureinc.com|208.67.181.244|:80... connected. > HTTP request sent, awaiting response... 200 OK > Length: 11,666 (11K) [text/plain] > > 0K .......... . 100% 166.39 > KB/s > > A bunch of these with other file names instead of borek.txt and other ips > as well. > Someone else has already suggested it, but I second the suggestion. Wipe the disk clean and reformat with a new install. You have no idea what garbage may be laying around to bite you later if you just try to clean it up. A new install with a formatted disk will at least make sure no surprises are waiting for you. > > >
