Linux Help
guides forums blogs
Home Desktops Distributions ISO Images Logos Newbies Reviews Software Support & Resources Linuxhelp Wiki

Welcome Guest ( Log In | Register )



Advanced DNS Management
New ZoneEdit. New Managment.

FREE DNS Is Back

Sign Up Now
 
Reply to this topicStart new topic
> Nothing but too-fast scrolling “garbage” on console, How to troubleshoot when messages are unreadable?
Charlweed Hymerf...
post Aug 18 2006, 04:33 PM
Post #1


Whats this Lie-nix Thing?
*

Group: Members
Posts: 4
Joined: 18-August 06
Member No.: 7,007



My Fedora 3 system randomly fails every few days. I apologize for the lack of specific information in this post but that is exactly my problem, lack of diagnostic information.

Fedora 3
Kernel 2.6.12-1.1381_FC3 i686

I use this system as a server without X running, so I have the console on. I don’t “see” it fail, I only know it is down when ssh, samba, or some other service fails to respond to a client. When I look at the console, all I see is a screen full of text whipping by too fast to read. The first chars are “[“ and there are hex numbers mixed in, but that’s all I can see. I’ve tried the pause key, with various combos of crtl, alt and shift, but that does not stop the scrolling.

The system does not respond to the keyboard, or a remote login. So I have to do a hard reset. Of course this clears the screen, so I can’t see what the messages are.

After a reboot, and ( sometimes ) a forced fsck, I go to /var/log, and look around, but I don’t see anything informative. I’ve recursively greped for “panic” and “fault” but nothing shows up.

I strongly suspect this is a disk drive hardware failure, but that’s only an assumption, with no solid proof. When I tried a warm reboot, “something” flashed a message about /dev/hdc1 before it stoped at the “maintenance” login. I don’t want to buy a new drive just to try and troubleshoot a fault that shows up every few days.

My mtab is:
/dev/hda2 / ext3 rw 0 0
/dev/hda1 /boot ext3 rw 0 0
/dev/hdc1 /usr/export/fitness ext3 rw 0 0
/dev/hdd1 /usr/export/garage ext3 rw 0 0


How can I troubleshoot this?

Thanks!

Charlweed
Go to the top of the page
 
+Quote Post
michaelk
post Aug 19 2006, 11:22 AM
Post #2


Its GNU/Linuxhelp.net
*******

Group: Support Specialist
Posts: 1,797
Joined: 23-January 03
Member No.: 360



So nothing is obvious in the logs.
Can you determine what was running via the logs right before it crashed?
To check out the disks go get the drives manufactures diagnotsics tools to see you have problems.
Check out the system to make sure the CPU, graphics card and hard drives are not running hot.

What applications do you run on the server. How much swap is being used? Is it anywhere near the max?
Go to the top of the page
 
+Quote Post
DS2K3
post Aug 20 2006, 04:58 AM
Post #3


Its GNU/Linuxhelp.net
*******

Group: Support Specialist
Posts: 1,284
Joined: 14-November 04
From: Southampton, Hampshire
Member No.: 4,165



Take out any RAM that you don't need and replace it one bit at a time - faulty RAM used to that to me...


--------------------
Fixed your problem? Let us know!
richard@linuxhelp.net

www.Gathr.co.uk Online Event Management
Go to the top of the page
 
+Quote Post
Charlweed Hymerf...
post Aug 22 2006, 11:27 AM
Post #4


Whats this Lie-nix Thing?
*

Group: Members
Posts: 4
Joined: 18-August 06
Member No.: 7,007



Thanks folks!
At the moment, the logs have too much noise. I'm going to kill some services, and wait for the next failure.
Meanwhile, I'm gonna try the DOS drive diags ( are there Linux based diags ?) .

DS2K3, so you think that the POST "full" ram test would miss bad ram? That's scary. I only have one 1gb DIMM, so if thats the problem... well I'd hate to spend the bucks on ram for this old system.


Charlweed
Go to the top of the page
 
+Quote Post
DS2K3
post Aug 22 2006, 02:47 PM
Post #5


Its GNU/Linuxhelp.net
*******

Group: Support Specialist
Posts: 1,284
Joined: 14-November 04
From: Southampton, Hampshire
Member No.: 4,165



POST tests do seem to miss things sometimes. If possible try and borrow some RAM that fits. Failing that, your Fedora CD might have a "memtest86" utility on it (which cyou can run when you boot from it) which should give your memory a good going over.

If you have the resources, try upgrading to FC5 which has a newer kernel and may have fixed a bug that is cuasing this.

D


--------------------
Fixed your problem? Let us know!
richard@linuxhelp.net

www.Gathr.co.uk Online Event Management
Go to the top of the page
 
+Quote Post
dishawjp
post Aug 23 2006, 09:46 AM
Post #6


./configure
***

Group: Members
Posts: 56
Joined: 8-April 04
Member No.: 2,734



QUOTE
[Meanwhile, I'm gonna try the DOS drive diags ( are there Linux based diags ?)

These are pretty good for Linux/BSD systems and I think are included in the Fedora distributions

http://smartmontools.sourceforge.net/

Also, to check RAM, if you can leave the server down for 12-24 hours, memtest86 is a really good utility.

http://freshmeat.net/projects/memtest86/

HTH,

Jim Dishaw


--------------------
Registered Linux User 294493
Go to the top of the page
 
+Quote Post
Charlweed Hymerf...
post Aug 23 2006, 11:53 AM
Post #7


Whats this Lie-nix Thing?
*

Group: Members
Posts: 4
Joined: 18-August 06
Member No.: 7,007



Thanks for the URL's folks, I'll install them, and triy them out.

The server failed again, The scrolling test looks like
[<c0XXXXXXX>] XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

If that looks familiar to anyone.

I did a tail on recent logs, and my httpd log looks suspisious. Thoses "Search" requests are several kb of the same character. I snipped it for this post.
Also, the IP 24.23.252.30 is a dynamic comcast address. it's not mine.

********* ./httpd/access_log *********
24.23.229.58 - - [22/Aug/2006:08:55:36 -0700] "GET / HTTP/1.0" 200 20903 "-" "-"
24.23.229.58 - - [22/Aug/2006:11:40:01 -0700] "GET / HTTP/1.0" 200 20903 "-" "-"
24.23.229.58 - - [22/Aug/2006:13:05:04 -0700] "GET / HTTP/1.0" 200 20903 "-" "-"
24.23.246.163 - - [22/Aug/2006:15:15:47 -0700] "GET / HTTP/1.0" 200 20903 "-" "-"
24.23.246.163 - - [22/Aug/2006:15:15:47 -0700] "SEARCH /\x90\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90" 414 324 "-" "-"
24.23.246.163 - - [22/Aug/2006:15:16:18 -0700] "POST /_vti_bin/_vti_aut/fp30reg.dll HTTP/1.1" 404 301 "-" "-"
24.23.246.163 - - [22/Aug/2006:16:06:51 -0700] "GET / HTTP/1.0" 200 20903 "-" "-"
24.23.246.163 - - [22/Aug/2006:16:06:52 -0700] "SEARCH /\x90\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\xc9\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90\x90" 414 324 "-" "-"
24.23.246.163 - - [22/Aug/2006:16:07:18 -0700] "POST /_vti_bin/_vti_aut/fp30reg.dll HTTP/1.1" 404 301 "-" "-"
24.23.229.58 - - [23/Aug/2006:08:49:51 -0700] "GET / HTTP/1.0" 200 20903 "-" "-"

********* ./httpd/error_log *********
[Tue Aug 22 07:14:34 2006] [error] [client 24.23.91.92] File does not exist: /var/www/html/_vti_bin
[Tue Aug 22 15:15:47 2006] [error] [client 24.23.246.163] request failed: URI too long (longer than 8190)
[Tue Aug 22 15:16:18 2006] [error] [client 24.23.246.163] File does not exist: /var/www/html/_vti_bin
[Tue Aug 22 16:06:52 2006] [error] [client 24.23.246.163] request failed: URI too long (longer than 8190)
[Tue Aug 22 16:07:18 2006] [error] [client 24.23.246.163] File does not exist: /var/www/html/_vti_bin
[Wed Aug 23 08:31:13 2006] [notice] core dump file size limit raised to 4294967295 bytes
[Wed Aug 23 08:31:14 2006] [notice] Digest: generating secret for digest authentication ...
[Wed Aug 23 08:31:14 2006] [notice] Digest: done
[Wed Aug 23 08:31:16 2006] [notice] mod_python: Creating 4 session mutexes based on 150 max processes and 0 max threads.
[Wed Aug 23 08:31:16 2006] [notice] Apache/2.0.53 (Fedora) configured -- resuming normal operations



Thanks again folks, you are terriffic!

Charlweed
Go to the top of the page
 
+Quote Post
Charlweed Hymerf...
post Aug 23 2006, 01:02 PM
Post #8


Whats this Lie-nix Thing?
*

Group: Members
Posts: 4
Joined: 18-August 06
Member No.: 7,007



Sorry, wrong Ips,
the 24.23.229.58 and 24.23.246.163 are not mine,
and as Imy server was not public, the requests are not "legitimate".

Also, I'm tring to find a way to "track" swap usage. If my system is crashing under load, before it can write an out-of-memory error to /var/log/messages, I don't see how I can tell.

After boot, basically at idle, free -m shows me

total used free shared buffers cached
Mem: 979 122 856 0 13 70
-/+ buffers/cache: 38 941
Swap: 1788 0 1788

To me, this looks like I have plenty, but only at idle, and assuming my swap drive is ok.

I now run samba,swat, and practically nothing else.

I also HAD several idle services loaded, including teamspeak,ventrillo,http, and a (non functional) "MS Exchange replacement suite" which included cyrus, openldap,mysql,postgres and maybe other junk.
I don't think it was any of these, because the crashes are recent, and those have been installed for months. I've disabled them all anyway.

Charlweed
Go to the top of the page
 
+Quote Post

Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 22nd October 2017 - 04:11 PM