How I saved my MDADM RAID

From Notepedia
Jump to: navigation, search

RAID not assembling on boot

I'm running a RAID 5 with 4x1Tb disks allowing one "spate" disk for failure, inside the MDADM RAID there's partitions in LVM that I have adjusted in size as required by the system.

Before I start, here a list of Tools and Technologies used in this problem

  • MDADM for software RAID
  • LVM for Logic Volume Management
  • S.M.A.R.T. or SmartMonTools for disk checking
  • badblocks to locate bad blocks on a corrupt disk

First thing, I need to look at the RAID and see what may be going on, the quickest way to inspect is through mdstat. Here's a guide on how to read mdstat

 # cat /proc/mdstat
 Personalities : [raid6] [raid5] [raid4]
 md0 : inactive raid5 sdd1[1] sdc1[4] sda1[2]
 2929283712 blocks super 1.2 level 5, 128k chunk, algorithm 2 [4/3] [_UUU]

That looks like 3 out of the 4 disks in my array are working, and the RAID is inactive - it's not been able to start because there may be problems with the fourth drive.

Let's try assembling it:

 # mdadm --assemble --scan
 mdadm: failed to add /dev/sdb to /dev/md127: Invalid argument
 mdadm: /dev/md127 assembled from -1 drives and 1 spare - not enough to start the array.

That doesn't work, it's looking like sdb is the problem drive from the above. Not sure why the RAID won't start because it should be able to start with 3 disks instead of 4.

Next, I want to see what's wrong with these disks.

Smartmontools Analysis

Check if device can run SMART tests:

 # smartctl -i /dev/<device>

See types of tests that can be run on the device:

 # smartctl -c /dev/<device>
 ...
 Short self-test routine
 recommended polling time: 	 (   2) minutes.
 Extended self-test routine
 recommended polling time: 	 ( 244) minutes.
 Conveyance self-test routine
 recommended polling time: 	 (   5) minutes.
 ...

To run the tests do the following, in order of length:

 # smartctl -t short /dev/<device>
 # smartctl -t conveyance /dev/<device>
 # smartctl -t long /dev/<device>

To view the test's overall health status compiled from all tests:

 # smartctl -H /dev/<device>

To view the test's result errors:

 # smartctl -l selftest /dev/<device>

To view the test's detailed results:

 # smartctl -a /dev/<device>


Test Results

For all the drives in the array, I want to do a short test then a conveyance test and record the results. I ran tests for sda sdb sdc and sdc - following are the results in brief:

SDA From these results, sda is OK

 # smartctl -a /dev/sda
 Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
 # 1  Conveyance offline  Completed without error       00%      7650         -
 # 2  Short offline       Completed without error       00%      7647         -

SDB From these results, sdb is BAD - I've included more details on this one:

 # smartctl -a /dev/sdb
 Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
 # 1  Conveyance offline  Completed: read failure       90%     36056         11447653
 # 2  Short offline       Completed: read failure       90%     36056         11447653

SDC From these results, sdc is OK

 # smartctl -a /dev/sdc
 Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
 # 1  Conveyance offline  Completed without error       00%      7650         -
 # 2  Short offline       Completed without error       00%      7647         -

SDD From these results, sdd is OK

 [20:55 [email protected] ~] > smartctl -a /dev/sdd
 Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
 # 1  Conveyance offline  Completed without error       00%     28959         -
 # 2  Short offline       Completed without error       00%     28957         -

Badblocks Analysis

Exactly how stuffed is the sdb drive? I ran badblocks to find out:

 # badblocks /dev/sdb >> sdb.badblocks
 # wc -l sdb.badblocks
 90 sdb.badblocks

Ok so that means sdb has 90 bad blocks, I'm not going to manipulate these blocks directly for example with dd because of the complexity of the drives being both in MDADM and LVM, so I'm going to see if I can get this drive back into the array and hope the array will recreate itself from there.

Getting the RAID running again

This will involve stopping the RAID, force reassembling it, and trying to add the busted drive back in order to get the RAID to rebuild.

First step, try to get the RAID running even if there is a broken drive within it:

 # mdadm --stop /dev/md0
 # mdadm --assemble --run /dev/md0 /dev/sd[abcd]1

OK now we've got a running array, now to get the LVM up:

 $ lvm
 lvm> lvdisplay

After doing the lvdisplay comment I'm getting shown my logic volumes are not available, eg:

 ...
 LV Status              NOT available

This is not actually a problem, all I need to do is run a change command to get them available again:

 lvm> vgchange -a y
 4 logical volume(s) in volume group "RAID_LVM" now active

Remount Logic Volumes as I've had them before. The data for this is already in my fstab file, I just confirm they're ok then do an auto-remount:

 # vim /etc/fstab
 # mount -a
 /dev/mapper/RAID_LVM-video     ext4      1.8T  1.6T  123G  93% /home/michael/Video
 /dev/mapper/RAID_LVM-music     ext4      148G  135G  5.4G  97% /home/michael/Music
 /dev/mapper/RAID_LVM-pictures  ext4       30G   19G  9.4G  67% /home/michael/Pictures
 /dev/mapper/RAID_LVM-downloads ext4       50G  230M   47G   1% /home/michael/Downloads

Re-adding the missing RAID drive

Here I add the busted drive /dev/sdb back into the MD0 array

Inspecting the RAID's current state, note the "removed" device

 $  mdadm -D /dev/md0
 /dev/md0:
 Raid Level : raid5
 Raid Devices : 4
 Total Devices : 3
 State : active, degraded
 Active Devices : 3
 Working Devices : 3
 Failed Devices : 0
 Spare Devices : 0
   Number   Major   Minor   RaidDevice State
      0       0        0        0      removed
      1       8       49        1      active sync   /dev/sdd1
      2       8        1        2      active sync   /dev/sda1
      4       8       33        3      active sync   /dev/sdc1

I want to Bitmap the array if it is not already been bitmapped, this should allow easy add/removal of drives.

First - check if I've already got it running with an mdstat

 $ cat /proc/mdstat
 Personalities : [raid6] [raid5] [raid4]
 md0 : active raid5 sdd1[1] sdc1[4] sda1[2]
 2929283712 blocks super 1.2 level 5, 128k chunk, algorithm 2 [4/3] [_UUU]

Ok, that doesn't look like it's bitmapped, if it's not bitmapped then:

 mdadm -G /dev/md0 -b internal

Then check that the changes have been applied

 $ cat /proc/mdstat
 Personalities : [raid6] [raid5] [raid4]
 md0 : active raid5 sdd1[1] sdc1[4] sda1[2]
 2929283712 blocks super 1.2 level 5, 128k chunk, algorithm 2 [4/3] [_UUU]
 bitmap: 8/8 pages [32KB], 65536KB chunk

Finally, add the broken disk back to the RAID array:

 # mdadm /dev/md0 --add /dev/sdb
 mdadm: added /dev/sdb

Then have a look at its status of the RAID:

 # mdadm -D /dev/md0
 /dev/md0:
 Raid Level : raid5
 Raid Devices : 4
 Total Devices : 4
 State : active, degraded, recovering
 Active Devices : 3
 Working Devices : 4
 Failed Devices : 0
 Spare Devices : 1
 Rebuild Status : 0% complete
   Number   Major   Minor   RaidDevice State
      5       8       16        0      spare rebuilding   /dev/sdb
      1       8       49        1      active sync   /dev/sdd1
      2       8        1        2      active sync   /dev/sda1
      4       8       33        3      active sync   /dev/sdc1

Now the busted disk is added back and is currently rebuilding, now how long will it take to resync? Back to mdstat:

 # cat /proc/mdstat
 Personalities : [raid6] [raid5] [raid4]
 md0 : active raid5 sdb[5] sdd1[1] sdc1[4] sda1[2]
     2929283712 blocks super 1.2 level 5, 128k chunk, algorithm 2 [4/3] [_UUU]
     [>....................]  recovery =  0.2% (2066856/976427904) finish=717.0min speed=22648K/sec
     bitmap: 0/8 pages [0KB], 65536KB chunk

Cool, so that's only ~12 hours until my raid is fully synced... that's quick isn't it? ;)

Note: If things were totally busted I would have used: https://www.gnu.org/software/ddrescue/manual/ddrescue_manual.html