Kernel crash

Recently,a few times during boot-up,my kernel crashed printing a trace to console. After the second time, I decided to find out more about it. So, first I needed logs to find clues. However, as I found out [from talking on #archlinux and ##kernel on freenode], those messages are not logged anywhere. So, I was advised to just jot down the console logs.

The following was what I jotted down the next time it happened. I have replaced memory address values by [<mem>].

[<mem>]?__find_get_block
[<mem>]submit_bio
[<mem>]submit_bh
[<mem>]__ext3_get_inode_loc
[<mem>]ext3_iget
[<mem>]ext3_lookup
[<mem>]do_lookup
[<mem>]do_last
[<mem>]do_flip_open
[<mem>]?anon_vma_chain_free
[<mem>]?unmap_region
[<mem>]?alloc_fd
[<mem>]?strncpy_from_user
[<mem>]do_sys_open
[<mem>]sys_open
[<mem>]system_call_fastpath
Code: ff ff ...
RIP [<mem>]__rb_rotate_left
RSP <mem>
--[end of trace]
note:modprobe exited with preempt_count 1

On irc, this time I was advised to do a memcheck. I took my Ubuntu disk [I keep it as a rescue disk.] and ran the memtest. My system passed that. So, the next step was to check the hard drive. I used to have smartmontools on my earlier linux installations; but I had not installed it on my Arch system. Getting it was a matter of time. It was already enabled on my hard disk; so I just had to check the logs.

smartctl --all /dev/sda
smartctl 5.39.1 2010-01-28 r3054 [x86_64-unknown-linux-gnu] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family: Seagate Momentus 5400.2 series
Device Model: ST9120821AS
Serial Number: 5PL4ZYQ9
Firmware Version: 7.24
User Capacity: 120,034,123,776 bytes
Device is: In smartctl database [for details use: -P show]
ATA Version is: 7
ATA Standard is: Exact ATA specification draft version not indicated
Local Time is: Mon Jul 5 11:48:01 2010 IST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
See vendor-specific Attribute list for marginal Attributes.

General SMART Values:
<snip>

SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
No General Purpose Logging support.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 68) minutes.
SCT capabilities: (0x0001) SCT Status supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 100 253 006 Pre-fail Always - 0
3 Spin_Up_Time 0x0002 096 095 000 Old_age Always - 0
4 Start_Stop_Count 0x0033 096 096 020 Pre-fail Always - 4348
5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 075 060 030 Pre-fail Always - 21656905082
9 Power_On_Hours 0x0032 094 094 000 Old_age Always - 5840
10 Spin_Retry_Count 0x0013 100 100 034 Pre-fail Always - 0
12 Power_Cycle_Count 0x0033 096 096 020 Pre-fail Always - 4450
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
189 High_Fly_Writes 0x003a 001 001 000 Old_age Always - 1096
190 Airflow_Temperature_Cel 0x0022 045 034 045 Old_age Always FAILING_NOW 55 (255 255 58 46)
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 1276
193 Load_Cycle_Count 0x0032 002 002 000 Old_age Always - 197088
194 Temperature_Celsius 0x0022 055 066 000 Old_age Always - 55 (0 19 0 0)
195 Hardware_ECC_Recovered 0x001a 048 044 000 Old_age Always - 131646341
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0000 100 253 000 Old_age Offline - 0
202 Data_Address_Mark_Errs 0x0032 100 253 000 Old_age Always - 0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
<snip>

SMART Selective self-test log data structure revision number 1
<snip>
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

As is apparent from the logs, it was a hard disk temperature problem. It was probably caused by blocked airflow coupled with high ambient temperature.