Thursday 8 July 2010

Kernel crash

Recently,a few times during boot-up,my kernel crashed printing a trace to console. After the second time, I decided to find out more about it. So, first I needed logs to find clues. However, as I found out [from talking on #archlinux and ##kernel on freenode], those messages are not logged anywhere. So, I was advised to just jot down the console logs.

The following was what I jotted down the next time it happened. I have replaced memory address values by [<mem>].

[<mem>]?__find_get_block
[<mem>]submit_bio
[<mem>]submit_bh
[<mem>]__ext3_get_inode_loc
[<mem>]ext3_iget
[<mem>]ext3_lookup
[<mem>]do_lookup
[<mem>]do_last
[<mem>]do_flip_open
[<mem>]?anon_vma_chain_free
[<mem>]?unmap_region
[<mem>]?alloc_fd
[<mem>]?strncpy_from_user
[<mem>]do_sys_open
[<mem>]sys_open
[<mem>]system_call_fastpath
Code: ff ff ...
RIP [<mem>]__rb_rotate_left
RSP <mem>
--[end of trace]
note:modprobe exited with preempt_count 1

On irc, this time I was advised to do a memcheck. I took my Ubuntu disk [I keep it as a rescue disk.] and ran the memtest. My system passed that. So, the next step was to check the hard drive. I used to have smartmontools on my earlier linux installations; but I had not installed it on my Arch system. Getting it was a matter of time. It was already enabled on my hard disk; so I just had to check the logs.

smartctl --all /dev/sda
smartctl 5.39.1 2010-01-28 r3054 [x86_64-unknown-linux-gnu] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family: Seagate Momentus 5400.2 series
Device Model: ST9120821AS
Serial Number: 5PL4ZYQ9
Firmware Version: 7.24
User Capacity: 120,034,123,776 bytes
Device is: In smartctl database [for details use: -P show]
ATA Version is: 7
ATA Standard is: Exact ATA specification draft version not indicated
Local Time is: Mon Jul 5 11:48:01 2010 IST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.

General SMART Values:
<snip>

SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
No General Purpose Logging support.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 68) minutes.
SCT capabilities: (0x0001) SCT Status supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 100 253 006 Pre-fail Always - 0
3 Spin_Up_Time 0x0002 096 095 000 Old_age Always - 0
4 Start_Stop_Count 0x0033 096 096 020 Pre-fail Always - 4348
5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 075 060 030 Pre-fail Always - 21656905082
9 Power_On_Hours 0x0032 094 094 000 Old_age Always - 5840
10 Spin_Retry_Count 0x0013 100 100 034 Pre-fail Always - 0
12 Power_Cycle_Count 0x0033 096 096 020 Pre-fail Always - 4450
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
189 High_Fly_Writes 0x003a 001 001 000 Old_age Always - 1096
190 Airflow_Temperature_Cel 0x0022 045 034 045 Old_age Always FAILING_NOW 55 (255 255 58 46)
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 1276
193 Load_Cycle_Count 0x0032 002 002 000 Old_age Always - 197088
194 Temperature_Celsius 0x0022 055 066 000 Old_age Always - 55 (0 19 0 0)
195 Hardware_ECC_Recovered 0x001a 048 044 000 Old_age Always - 131646341
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0000 100 253 000 Old_age Offline - 0
202 Data_Address_Mark_Errs 0x0032 100 253 000 Old_age Always - 0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
<snip>

SMART Selective self-test log data structure revision number 1
<snip>
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

As is apparent from the logs, it was a hard disk temperature problem. It was probably caused by blocked airflow coupled with high ambient temperature.

3 comments:

DarkShield said...

hahaha Ubuuntu FTW. miss karuchi be. ubuntu mac ku speed re katti pariba?

DarkShield said...

hahaha Ubuuntu FTW. miss karuchi be. ubuntu mac ku speed re katti pariba?

Phoenix said...

mo current distro arch linux ubuntu tharu faster; tested by phoronix.