The 1st warning was a message from my hosting company which stated -



I was a bit worried and installed f-prot on my webserver. Running this found a Unix/blitz virus which no one seems to have heard of. The only other viruses were w32 viruses in peoples emails. I removed all of the infections listed.

Worse news was to follow... on the 25th of september by webserver managed to generate 36,523.00MB of traffic on 1 day. THis cost me £150 for the one days activities. So the questions is.... how do i track down what i going on? where do i begin to investigate this traffic. the PLESK system provided by my isp to manage the box reports that there was no unusual traffic on the system. so i guess it was not normal web traffic.

I was installing trip wire and a firewall when i got another bandwidth warning so I lost my bottle and shut the box down.

oh and its running redhat 9.0

any ideas where to begin.

andy

have installed the firewall as noted here http://www.linuxhelp.net/guides/iptables/ - do you think this will help? also someone tell me how to accuratly monitor traffic going over eth0

cheers

andy

i have used "ps aux" and this showed a large number of processes of qmail attempting to send info to adfadsfadsfadsfadf@yahoo.com ewraewrweraewrwr@yahoo.com adfadsfasfasd@yahoo.com etc

it looks like this smap was the cuase of the 40 gig of traffic in a single day. does this sound likly. i have the webserver running with the firewall script on and qmail turned off. hoiwever i really need my email back up and running but a bit scared to do so. suggestions anyone!!! please.

also i cant find the qmail logs to check them i think they should be at /bin/log/qmail but there is nothing there

help meeeeee

It sounds to me like someone has broken in and installed a rootkit on your box and is using it for SPAM (or your e-mail server is misconfigured and is an open relay).

I rant here all the time concerning the need to keep all updates installed to prevent just this issue.

Download ckrootkit from here and install it via rpm, then run the command:

chkrootkit

Wheter you find one or not, some one has somehow compromised your box (or it is an open relay)
-----------------
First, RedHat 9 is no longer supported, and gets no updates from RedHat. There is a group providing security updates for RH9(Fedora Legacy Project), but I would instead use a free Enterprise solution like CentOS (CentOS 3.3 ISOs) (you don't need the source ISOs, just the first 3).
-----------------
At this point, I would take down the box, buy another hard drive, install CentOS, a LAMP server and postfix (or qmail) and then copy over all the files from the original drive that are required.

I would then install a good firewall script ... the one you posted is good, but it is setup for a box that has a network behind it (it does IP Masquerading). The best firewall scripts I have ever seen (in my opinion) come from here ... get the book and the scripts. There is a newer version of the book HERE that costs $20 ... but version 2 (the free version) works great as well.

right i have run 2 rootkit detectors and the only things that look abnormal is ....

chkroot

Checking `lkm'... You have 6 process hidden for ps command
Warning: Possible LKM Trojan installed

root check

== Check process/ps: ==

PID 1077 in use but "ps" do not show!

rkhunter
says everthing is clean

anything to worry about here?

other than that all seems to be back up to running - i just dont reallly have the capabilities or the time to rebuild this server from scratch. so am really hoping to clean out the existing setup! fingers crossed

oh and this appears when you do a f-prot check

/var/tmp/httpd Infection: Unix/Osf.A

what the hell is this?

please tell me i can avoid a rebuild

cheers

andy

The 6 items hiding from ps and the OSF.A issue are indicative of trojans installed on your system. I would absolutely not use it with out rebuilding it.

bugger

i think you are right - i was getting a load of port scans from IRC networks - the whois explained that this was natural if you were running an irc client... however, i was not. i blocked all ports used to connect to irc networks. this morning i found the following after running f-prot..

[quote]
/var/tmp/.bash_history/logs/eggdrop-1.6.10  Infection: Unix/Osf.A
Unable to remove the virus.
/var/tmp/.bash_history/logs/kik  Infection: Unix/Osf.A
Unable to remove the virus.
/var/tmp/.bash_history/logs/kik.4  Infection: Unix/Osf.A
Unable to remove the virus.
/var/tmp/httpd  Infection: Unix/Osf.A
Unable to remove the virus.
[/quote]


i just hope i have learned enough about security now to have a safe and secure webserver the next time around.



oh and for anyone who is interested - the following was found in my .bash_history folder - note the owner is listed as apache

[quote]drwxr-xr-x    9 apache   apache       4096 Oct  8 13:44 .
drwxr-xr-x    4 apache   apache       4096 Oct  4 15:59 ..
-rw-r--r--    1 apache   apache       3665 Oct  4 16:18 238
-rw-r--r--    1 apache   apache       1880 Oct  8 13:44 AlreadyAsked.txt
lrwxrwxrwx    1 apache   apache         14 Oct  4 15:58 bin -> eggdrop-1.6.10
-rw-r--r--    1 apache   apache        451 Oct  8 09:03 BotScore.html
-rw-r--r--    1 apache   apache         49 Oct  8 09:03 BotScores.txt
drwxr-xr-x    5 apache   apache       4096 Oct  4 15:58 doc
-rwxr-xr-x    1 apache   apache    2523568 May 11  2002 eggdrop-1.6.10
-rw-r--r--    1 apache   apache      45658 May 11  2002 eggdrop.advanced.conf
-rw-r--r--    1 apache   apache      49936 May 11  2002 eggdrop.complete.conf
-rw-r--r--    1 apache   apache       4823 May 11  2002 eggdrop.simple.conf
drwxr-xr-x    3 apache   apache       4096 May 11  2002 filesys
-rw-r--r--    1 apache   apache      41895 Oct  4 16:40 file.txt
drwxr-xr-x    4 apache   apache       4096 Oct  4 15:58 help
-rwxr-xr-x    1 apache   apache      21149 Mar 24  2003 kik
-rw-r--r--    1 apache   apache      21149 Mar 24  2003 kik.4
drwxr-xr-x    2 apache   apache       4096 Oct  8 13:36 language
drwxr-xr-x    2 apache   apache       4096 Oct  4 15:58 logs
-rw-r--r--    1 apache   apache        283 Oct  7 22:27 MonthlyScores.html
-rw-r--r--    1 apache   apache         10 Oct  7 22:27 MonthlyScores.txt
-rw-r--r--    1 apache   apache          6 Oct  7 22:24 pid.bangku-
-rw-r--r--    1 apache   apache      28591 May 11  2002 README
-rw-r--r--    1 apache   apache        275 Oct  7 22:27 ScoresOct.html
-rw-r--r--    1 apache   apache         10 Oct  7 22:27 ScoresOct.txt
drwxr-xr-x    2 apache   apache       4096 Oct  7 22:39 scripts
drwxr-xr-x    2 apache   apache       4096 Oct  4 15:58 text
[/quote]

and there appears to be a back door script (see aboce 238) installed as well - looking into that now - looks nasty - nice of them to leave if for me to look at though

egg drop is a irc bot! is that all they were after? but what caused the 40 gig of traffic that made me realise they were there?

very strange

lol - analysing the stuff in the above files and found Linux.Jac.8759! see here interstesting that the copy of egg-drop has the same virus inside it! did they know this? it was almost certainly infected after they installed the bot as when unpacked from bete.tar.gz then it contains no virus!

that script begins
[quote]
set my-hostname "localhost"
set my-ip "***.***.***.***"
set nick "bangku-"
set owner "anak_baik"
set basechan "#sobatmu"
set username "games"

[/quote]
so i guess i now know where that game user came from!

great - as i type a dos folder just appeared - inside is smurf6-linux+LPG.c and a list of ip addresess and a few other DOS tools - that explains where my bandwidth went to!

this is just getting silly now

I'm no rocket scientist, but wouldn't you be able to type in "Who" to find out who is logged into the box and kill the terminal? Or is it being executed via remote commands through like a irc channel? Also since it is running under username apache, couldn't you create another user like apache-backup, and kill your current version, and just start a new apache under a different user?

the problem is ... you have no idea which sytem binaries are good, which ones contain a trojan allowing login.

There are trojans that open up a remote windows (like ssh), but the commands like ps, who, etc. don't show those users and processes. Not all the processes are like that, but some are.

The only safe way to fix these issues is to reinstall the OS and immediately install all security patches ... then use a good iptables firewall that only allows connections that you want. Use SSH (and sftp) only to connect and transfer files from the outside line.

The OS that is installed should be one where security updates are routinely done (including the kernel).

Again, I recommend either CentOS or WBEL as the OS of choice ... and only compile things that you have to ... use the default versions of Apache, PHP, MySQL, etc. Those will be updated by RedHat whan there are problems.

yeh - thanks for all the help chaps. I have requested that my hosting company re-image my setup. it will be red hat 9 - no choice in this. so will get it patched up as soon as the reimage takes place

thanks for all the help

andy

Then make sure you use the updates produced by Fedora Legacy ... there are security updates since RedHat stopped support for RedHat 9.0 in April.

Your hosting company should not require an OS that is no longer supported by it's developer and that doesn't recieve security updates (RedHat 9) ... that is a major problem. I'd make them pay for downtime if they are going to use a non-supported OS.

(there is an new apache and a new php that were both produced on Oct, 3, 2004 {httpd-2.0.40-21.16.legacy.i386.rpm, php-4.2.2-17.7.legacy.i386.rpm} that fix security issues for RedHat 9 at Fedora Legacy)

thanks hughesjr for all your help

the web server is now zapped and web interface shut down till i get it all sorted

it now looks like they managed to use misconfigured coppermine permissions (my fault) to install their own webserver which allowed them run stuff - which i knew what the gaps were but either way they got in and thats that

cheers for all your suggestions and help

andy