Linux Help
guides forums blogs
Home Desktops Distributions ISO Images Logos Newbies Reviews Software Support & Resources Linuxhelp Wiki

Welcome Guest ( Log In | Register )



Advanced DNS Management
New ZoneEdit. New Managment.

FREE DNS Is Back

Sign Up Now
 
Reply to this topicStart new topic
> Webserver Traffic Gone Nutz Help!
ajbird
post Oct 2 2004, 07:32 AM
Post #1


Whats this Lie-nix Thing?
*

Group: Members
Posts: 9
Joined: 2-October 04
Member No.: 3,879



The 1st warning was a message from my hosting company which stated -

QUOTE
Dear Mr. Bird,

we have recently received several complaints regarding illegal access
attempts (port scans / hack attempts) originating from your 1&1 RootServer
(contract 4721466). Please check your server for viruses / internet
worms etc. immediatly

Should further complaints reach us concerning this matter we'll feel
impelled to take the server offline in order to prevent further abuse of our
infrastructure. Thank you for your understanding.

Furthermore we would ask you to contact us (abuse@kundenserver.de) within
three days in order to receive your comment on this concern. Thank you.


I was a bit worried and installed f-prot on my webserver. Running this found a Unix/blitz virus which no one seems to have heard of. The only other viruses were w32 viruses in peoples emails. I removed all of the infections listed.

Worse news was to follow... on the 25th of september by webserver managed to generate 36,523.00MB of traffic on 1 day. THis cost me £150 for the one days activities. So the questions is.... how do i track down what i going on? where do i begin to investigate this traffic. the PLESK system provided by my isp to manage the box reports that there was no unusual traffic on the system. so i guess it was not normal web traffic.

I was installing trip wire and a firewall when i got another bandwidth warning so I lost my bottle and shut the box down.

oh and its running redhat 9.0

any ideas where to begin.

andy
Go to the top of the page
 
+Quote Post
ajbird
post Oct 2 2004, 09:58 AM
Post #2


Whats this Lie-nix Thing?
*

Group: Members
Posts: 9
Joined: 2-October 04
Member No.: 3,879



have installed the firewall as noted here http://www.linuxhelp.net/guides/iptables/ - do you think this will help? also someone tell me how to accuratly monitor traffic going over eth0

cheers

andy
Go to the top of the page
 
+Quote Post
ajbird
post Oct 3 2004, 05:01 AM
Post #3


Whats this Lie-nix Thing?
*

Group: Members
Posts: 9
Joined: 2-October 04
Member No.: 3,879



i have used "ps aux" and this showed a large number of processes of qmail attempting to send info to adfadsfadsfadsfadf@yahoo.com ewraewrweraewrwr@yahoo.com adfadsfasfasd@yahoo.com etc

it looks like this smap was the cuase of the 40 gig of traffic in a single day. does this sound likly. i have the webserver running with the firewall script on and qmail turned off. hoiwever i really need my email back up and running but a bit scared to do so. suggestions anyone!!! please.

also i cant find the qmail logs to check them i think they should be at /bin/log/qmail but there is nothing there

help meeeeee
Go to the top of the page
 
+Quote Post
hughesjr
post Oct 4 2004, 06:33 AM
Post #4


Its GNU/Linuxhelp.net
*******

Group: Admin
Posts: 3,433
Joined: 25-July 03
From: Corpus Chrsiti, TX, USA
Member No.: 1,151



It sounds to me like someone has broken in and installed a rootkit on your box and is using it for SPAM (or your e-mail server is misconfigured and is an open relay).

I rant here all the time concerning the need to keep all updates installed to prevent just this issue.

Download ckrootkit from here and install it via rpm, then run the command:

chkrootkit

Wheter you find one or not, some one has somehow compromised your box (or it is an open relay)
-----------------
First, RedHat 9 is no longer supported, and gets no updates from RedHat. There is a group providing security updates for RH9(Fedora Legacy Project), but I would instead use a free Enterprise solution like CentOS (CentOS 3.3 ISOs) (you don't need the source ISOs, just the first 3).
-----------------
At this point, I would take down the box, buy another hard drive, install CentOS, a LAMP server and postfix (or qmail) and then copy over all the files from the original drive that are required.

I would then install a good firewall script ... the one you posted is good, but it is setup for a box that has a network behind it (it does IP Masquerading). The best firewall scripts I have ever seen (in my opinion) come from here ... get the book and the scripts. There is a newer version of the book HERE that costs $20 ... but version 2 (the free version) works great as well.


--------------------
Johnny Hughes
hughesjr@linuxhelp.net
Enterprise Alternatives: CentOS, WhiteBoxEL
Favorite Workstation Distros (in order): CentOS, Gentoo, Debian Sarge, Ubuntu, Mandrake, FedoraCore, Slackware, SUSE
Favorite Server Distros (in order): CentOS, WhiteBoxEL, Debian Sarge, Slackware, Mandrake, FedoraCore, Gentoo, SUSE
Go to the top of the page
 
+Quote Post
ajbird
post Oct 6 2004, 09:00 AM
Post #5


Whats this Lie-nix Thing?
*

Group: Members
Posts: 9
Joined: 2-October 04
Member No.: 3,879



right i have run 2 rootkit detectors and the only things that look abnormal is ....

chkroot

Checking `lkm'... You have 6 process hidden for ps command
Warning: Possible LKM Trojan installed

root check

== Check process/ps: ==

PID 1077 in use but "ps" do not show!

rkhunter
says everthing is clean

anything to worry about here?

other than that all seems to be back up to running - i just dont reallly have the capabilities or the time to rebuild this server from scratch. so am really hoping to clean out the existing setup! fingers crossed

oh and this appears when you do a f-prot check

/var/tmp/httpd Infection: Unix/Osf.A

what the hell is this?

please tell me i can avoid a rebuild

cheers

andy
Go to the top of the page
 
+Quote Post
hughesjr
post Oct 6 2004, 05:35 PM
Post #6


Its GNU/Linuxhelp.net
*******

Group: Admin
Posts: 3,433
Joined: 25-July 03
From: Corpus Chrsiti, TX, USA
Member No.: 1,151



QUOTE
From: Sophos
Linux/OSF-A will attempt to infect 200 ELF executables in the current working directory and the directory /bin. The virus will avoid the file ps or any files ending in ps.

If the virus is executed by a privileged user then it will attempt to create a backdoor server on the system. This is achieved by opening a socket on port 3049 or above and waiting for specially configured packets containing instructions for the backdoor program. The server may be asked to create a TCP connection with the attacker and to then attempt to supply them with a shell to use remotely.


The 6 items hiding from ps and the OSF.A issue are indicative of trojans installed on your system. I would absolutely not use it with out rebuilding it.


--------------------
Johnny Hughes
hughesjr@linuxhelp.net
Enterprise Alternatives: CentOS, WhiteBoxEL
Favorite Workstation Distros (in order): CentOS, Gentoo, Debian Sarge, Ubuntu, Mandrake, FedoraCore, Slackware, SUSE
Favorite Server Distros (in order): CentOS, WhiteBoxEL, Debian Sarge, Slackware, Mandrake, FedoraCore, Gentoo, SUSE
Go to the top of the page
 
+Quote Post
ajbird
post Oct 8 2004, 07:08 AM
Post #7


Whats this Lie-nix Thing?
*

Group: Members
Posts: 9
Joined: 2-October 04
Member No.: 3,879



bugger

i think you are right - i was getting a load of port scans from IRC networks - the whois explained that this was natural if you were running an irc client... however, i was not. i blocked all ports used to connect to irc networks. this morning i found the following after running f-prot..

[quote]
/var/tmp/.bash_history/logs/eggdrop-1.6.10  Infection: Unix/Osf.A
Unable to remove the virus.
/var/tmp/.bash_history/logs/kik  Infection: Unix/Osf.A
Unable to remove the virus.
/var/tmp/.bash_history/logs/kik.4  Infection: Unix/Osf.A
Unable to remove the virus.
/var/tmp/httpd  Infection: Unix/Osf.A
Unable to remove the virus.
[/quote]


i just hope i have learned enough about security now to have a safe and secure webserver the next time around.

sad.gif

oh and for anyone who is interested - the following was found in my .bash_history folder - note the owner is listed as apache

[quote]drwxr-xr-x    9 apache   apache       4096 Oct  8 13:44 .
drwxr-xr-x    4 apache   apache       4096 Oct  4 15:59 ..
-rw-r--r--    1 apache   apache       3665 Oct  4 16:18 238
-rw-r--r--    1 apache   apache       1880 Oct  8 13:44 AlreadyAsked.txt
lrwxrwxrwx    1 apache   apache         14 Oct  4 15:58 bin -> eggdrop-1.6.10
-rw-r--r--    1 apache   apache        451 Oct  8 09:03 BotScore.html
-rw-r--r--    1 apache   apache         49 Oct  8 09:03 BotScores.txt
drwxr-xr-x    5 apache   apache       4096 Oct  4 15:58 doc
-rwxr-xr-x    1 apache   apache    2523568 May 11  2002 eggdrop-1.6.10
-rw-r--r--    1 apache   apache      45658 May 11  2002 eggdrop.advanced.conf
-rw-r--r--    1 apache   apache      49936 May 11  2002 eggdrop.complete.conf
-rw-r--r--    1 apache   apache       4823 May 11  2002 eggdrop.simple.conf
drwxr-xr-x    3 apache   apache       4096 May 11  2002 filesys
-rw-r--r--    1 apache   apache      41895 Oct  4 16:40 file.txt
drwxr-xr-x    4 apache   apache       4096 Oct  4 15:58 help
-rwxr-xr-x    1 apache   apache      21149 Mar 24  2003 kik
-rw-r--r--    1 apache   apache      21149 Mar 24  2003 kik.4
drwxr-xr-x    2 apache   apache       4096 Oct  8 13:36 language
drwxr-xr-x    2 apache   apache       4096 Oct  4 15:58 logs
-rw-r--r--    1 apache   apache        283 Oct  7 22:27 MonthlyScores.html
-rw-r--r--    1 apache   apache         10 Oct  7 22:27 MonthlyScores.txt
-rw-r--r--    1 apache   apache          6 Oct  7 22:24 pid.bangku-
-rw-r--r--    1 apache   apache      28591 May 11  2002 README
-rw-r--r--    1 apache   apache        275 Oct  7 22:27 ScoresOct.html
-rw-r--r--    1 apache   apache         10 Oct  7 22:27 ScoresOct.txt
drwxr-xr-x    2 apache   apache       4096 Oct  7 22:39 scripts
drwxr-xr-x    2 apache   apache       4096 Oct  4 15:58 text
[/quote]

and there appears to be a back door script (see aboce 238) installed as well - looking into that now - looks nasty - nice of them to leave if for me to look at though

egg drop is a irc bot! is that all they were after? but what caused the 40 gig of traffic that made me realise they were there?

very strange

lol - analysing the stuff in the above files and found Linux.Jac.8759! see here interstesting that the copy of egg-drop has the same virus inside it! did they know this? it was almost certainly infected after they installed the bot as when unpacked from bete.tar.gz then it contains no virus!

that script begins
[quote]
set my-hostname "localhost"
set my-ip "***.***.***.***"
set nick "bangku-"
set owner "anak_baik"
set basechan "#sobatmu"
set username "games"

[/quote]
so i guess i now know where that game user came from!
Go to the top of the page
 
+Quote Post
ajbird
post Oct 8 2004, 08:34 AM
Post #8


Whats this Lie-nix Thing?
*

Group: Members
Posts: 9
Joined: 2-October 04
Member No.: 3,879



great - as i type a dos folder just appeared - inside is smurf6-linux+LPG.c and a list of ip addresess and a few other DOS tools - that explains where my bandwidth went to!

this is just getting silly now
Go to the top of the page
 
+Quote Post
technick
post Oct 8 2004, 09:42 AM
Post #9


Whats this Lie-nix Thing?
*

Group: Members
Posts: 2
Joined: 8-October 04
Member No.: 3,917



I'm no rocket scientist, but wouldn't you be able to type in "Who" to find out who is logged into the box and kill the terminal? Or is it being executed via remote commands through like a irc channel? Also since it is running under username apache, couldn't you create another user like apache-backup, and kill your current version, and just start a new apache under a different user?
Go to the top of the page
 
+Quote Post
hughesjr
post Oct 8 2004, 10:33 AM
Post #10


Its GNU/Linuxhelp.net
*******

Group: Admin
Posts: 3,433
Joined: 25-July 03
From: Corpus Chrsiti, TX, USA
Member No.: 1,151



the problem is ... you have no idea which sytem binaries are good, which ones contain a trojan allowing login.

There are trojans that open up a remote windows (like ssh), but the commands like ps, who, etc. don't show those users and processes. Not all the processes are like that, but some are.

The only safe way to fix these issues is to reinstall the OS and immediately install all security patches ... then use a good iptables firewall that only allows connections that you want. Use SSH (and sftp) only to connect and transfer files from the outside line.

The OS that is installed should be one where security updates are routinely done (including the kernel).

Again, I recommend either CentOS or WBEL as the OS of choice ... and only compile things that you have to ... use the default versions of Apache, PHP, MySQL, etc. Those will be updated by RedHat whan there are problems.


--------------------
Johnny Hughes
hughesjr@linuxhelp.net
Enterprise Alternatives: CentOS, WhiteBoxEL
Favorite Workstation Distros (in order): CentOS, Gentoo, Debian Sarge, Ubuntu, Mandrake, FedoraCore, Slackware, SUSE
Favorite Server Distros (in order): CentOS, WhiteBoxEL, Debian Sarge, Slackware, Mandrake, FedoraCore, Gentoo, SUSE
Go to the top of the page
 
+Quote Post
ajbird
post Oct 9 2004, 03:02 AM
Post #11


Whats this Lie-nix Thing?
*

Group: Members
Posts: 9
Joined: 2-October 04
Member No.: 3,879



yeh - thanks for all the help chaps. I have requested that my hosting company re-image my setup. it will be red hat 9 - no choice in this. so will get it patched up as soon as the reimage takes place

thanks for all the help

andy
Go to the top of the page
 
+Quote Post
hughesjr
post Oct 10 2004, 06:34 AM
Post #12


Its GNU/Linuxhelp.net
*******

Group: Admin
Posts: 3,433
Joined: 25-July 03
From: Corpus Chrsiti, TX, USA
Member No.: 1,151



Then make sure you use the updates produced by Fedora Legacy ... there are security updates since RedHat stopped support for RedHat 9.0 in April.

Your hosting company should not require an OS that is no longer supported by it's developer and that doesn't recieve security updates (RedHat 9) ... that is a major problem. I'd make them pay for downtime if they are going to use a non-supported OS.

(there is an new apache and a new php that were both produced on Oct, 3, 2004 {httpd-2.0.40-21.16.legacy.i386.rpm, php-4.2.2-17.7.legacy.i386.rpm} that fix security issues for RedHat 9 at Fedora Legacy)


--------------------
Johnny Hughes
hughesjr@linuxhelp.net
Enterprise Alternatives: CentOS, WhiteBoxEL
Favorite Workstation Distros (in order): CentOS, Gentoo, Debian Sarge, Ubuntu, Mandrake, FedoraCore, Slackware, SUSE
Favorite Server Distros (in order): CentOS, WhiteBoxEL, Debian Sarge, Slackware, Mandrake, FedoraCore, Gentoo, SUSE
Go to the top of the page
 
+Quote Post
ajbird
post Oct 13 2004, 08:01 AM
Post #13


Whats this Lie-nix Thing?
*

Group: Members
Posts: 9
Joined: 2-October 04
Member No.: 3,879



thanks hughesjr for all your help

the web server is now zapped and web interface shut down till i get it all sorted

it now looks like they managed to use misconfigured coppermine permissions (my fault) to install their own webserver which allowed them run stuff - which i knew what the gaps were but either way they got in and thats that

cheers for all your suggestions and help

andy
Go to the top of the page
 
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 22nd October 2017 - 10:42 AM