new 500GB SATA RAID1 mirror

For fault tolerance, I finally got around to upgrading my drives to a RAID1 mirror this weekend. I bought a pair of Seagate ST3500641AS-RK Barracuda from Outpost.com for $99 each. Sale is over now, of course.

Here's what I had to do:

1. partition the drives using fdisk
2. create the mirror
3. wait for the mirror to sync
   
4. format the mirror
5. test a mount
6. add to /etc/fstab
7. reboot to make sure everything comes up
   

Partition the drives with good ol' fdisk
I had to partition each drive and give each drive a partition type of "Linux raid autodetect". This is type "fd". Here is a snip of my fdisk session for one of the drives:

[mule@ogre ~]# fdisk /dev/sdf
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.


The number of cylinders for this disk is set to 60801.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): p

Disk /dev/sdf: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4):
Value out of range.
Partition number (1-4): 1
First cylinder (1-60801, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-60801, default 60801):
Using default value 60801

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): fd

Command (m for help): p

Disk /dev/sdf: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdf1 1 60801 488384001 fd Linux raid autodetect

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Create the mirror
I created the mirror using mdadm. Note the "--level=1" for the RAID1 mirror:
[mule@ogre ~]# mdadm --create /dev/md2 --level=1 --raid-devices=2 /dev/sde1 /dev/sdf1

Let's check out the details:
[mule@ogre ~]# mdadm --detail /dev/md2
/dev/md2:
Version : 00.90.03
Creation Time : Thu Jan 17 19:58:30 2008
Raid Level : raid1
Array Size : 488383936 (465.76 GiB 500.11 GB)
Used Dev Size : 488383936 (465.76 GiB 500.11 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 2
Persistence : Superblock is persistent

Update Time : Sun Jan 20 16:19:10 2008
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0

UUID : 1705b387:1c71d83e:364b60b4:fb0cce92
Events : 0.6

Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sde1
1 8 49 1 active sync /dev/sdf1

Wait for the mirror to build
You can also look in /proc/mdstat for status information:
[mule@ogre ~]# cat /proc/mdstat
Personalities : [raid0] [raid6] [raid5] [raid4] [raid1]
md2 : active raid1 sdb1[0] sdd1[1]
488383936 blocks [2/2] [UU]

md0 : active raid0 sda3[0] sdc2[1]
483411456 blocks 256k chunks

unused devices:

Format the partition
The -j parameter to mke2fs creates the filesystem with an ext3 journal, for another level of safety. File access on an ext3 filesystem is a bit slower than on an ext2 filesystem. However, the benefit is that you have a journaling filesystem that recovers from errors quickly and safely. Here's some documentation on the ext3 filesystem:
http://www.ibm.com/developerworks/linux/library/l-fs7.html

Here is the command I ran to create the ext3 filesystem:
[mule@ogre ~]# mke2fs -j /dev/md2

Test the mount
You should always test mounting the new filesystem. Since this filesystem is going to store all my videos, I am going to create my mount point and mount the new RAID mirror to /mnt/videos:
[mule@ogre ~]# mkdir /mnt/videos
[mule@ogre ~]# mount -t ext3 /dev/md2 /mnt/videos
[mule@ogre ~]#

You see no errors after the mount command. This means that the mount was successful. Yes!

Add to /etc/fstab
Of course, we want the new filesystem to be mounted when we reboot, so I add the following line to my /etc/fstab:
/dev/md2 /mnt/videos ext3 defaults 1 1

Reboot!
The final test is to reboot. The following lines in dmesg output make me happy:
md: md2 stopped.
md: bind
md: bind
md: raid1 personality registered for level 1
raid1: raid set md2 active with 2 out of 2 mirrors

Rock and roll! We're good with the new RAID mirror and safe from drive failures!
The Mule

rebuilding the workstation: lost my damn RAID set!

I *almost* had a minor tragedy last night while rebuilding my Fedora Core 4 box. The *almost* tragedy occurred while reinstalling Core 4. While going through the process, I told the installer to use my existing RAID set, but do not reformat it. I figured what harm could come to the drive if it doesn't get formatted? Well, a lot, apparently. Since I was rushing through the install, I neglected to take proper care of the 40GB of MPEG2 files on the RAID set. Here's what happened.

After the FC installer finished, it asks to reboot the box, so I rebooted. On bootup, the system gave me errors regarding an unrecognized filesystem and dropped me to a filesystem shell to fix what was wrong. I didn't know what was wrong, so I rebooted into a linux rescue disk and simply took the RAID filesystem out of /etc/fstab and rebooted.

On the second bootup, I started fdisk to look at the drives. To my dismay, fdisk did not recognize the RAID set partitions. Agh! I did some preliminary research online. I came to the determination that I figured I had lost my data, though I did have it backed up. But it would be a pain to retrieve the 40GB or so of videos from my backup system. Damn it. So I bit the bullet and created new partitions of type "fd", a Linux RAID autodetect partition as I had done when I setup the RAID set initially. When I wrote the partition table, fdisk gave me some error saying that the drives will resync after the box reboots. I rebooted and looked at the output of "fdisk -l". Things seemed alright, as the drives were recognized as Linux RAID autodetect. So I then reenabled /dev/md0 in /etc/fstab and made sure that /etc/mdadm.conf was correct. Dejectedly I rebooted.

When the system started, it dropped me into a prompt complaining of filesystem problems again. Now what? This time, I looked at the man pages for fsck and figured out the commands I needed to repair the disk. They ran something like this:
fsck -t ext2 /dev/md0 -V -r
-t filesystem type
-V verbose
-r prompt for each repair

A couple of inodes were missing or corrupt. OK. Fsck seemed to continue on with the different stages of the five stage check procedure. About five or ten minutes of this, I was getting worried. Happily, it finished the checks and dropped me to a prompt in order to exit and reboot. OK..that's progress! Also, the box came up clean with no errors. Sweet. I now went to view the filesystem, and to my shock and surprise, my original files were there! Awesome!! But now the real test is reading from and writing to files. I first viewed one of the videos in mplayer. This worked! I then performed an extensive write test. Cinelerra needs table of contents files for each video. These are index file, essentially. So at a prompt, I generated a bunch of toc files for the 30GB or so of video I had by using this one command:
for i in `ls -1 *.m2t` ; do echo $i ; mpeg3toc $i $i.toc ; done

The file creation took about twenty minutes but worked! I then loaded the files into Cinelerra, started editing and wouldn't you know they are good to go! Hooray! But I'm still an idiot.

Moral of the story is make sure you do your research on RAID before you decide to implement it.

system reconfig, #2

Woops with Ghost
OK. A minor foul up in the steps from the last post. Seems that Ghosting the utility partition is not working as expected. Norton Ghost created a total new partition at the end of the XP partition on the 80 gigger. So, I went ahead and deleted it, as this 2GB partition eats into my Linux root partition. I never used the utility partition anyway, but I thought that Ghost would have correctly selected the 100MB partition I had chosen. Oh well. I'm not that concerned.

FDISK is for F*@K!
Another foul up was trying to use a Windows 98 boot disk and the classic fdisk program to both create the XP/Linux partitions and mark one as active. Word to the wise: DON'T USE FDISK! It has an annoying habit of "Verifying Drive Integrity" that takes FOREVER. So, I just decided to reboot and use my old 40GB XP system to carve up the drive and mark the 30GB partition as active. Saved a good deal of time there with that brainstorm. Phffflt!

Going SLEOOWLY.
Applying the XP image to the partition designated for it went super smooth, but the Dell has this ugly habit of taking forever to start up when I have the power to some disks plugged in and some not, but still connected to my Sil680 IDE stripe set RAID card. Frustrating, as it adds two minutes at the front end of every boot cycle. Argh. But now I have my XP system back up and running on my new 80GB drive! Hooray!

Core 4 from Scratch..with a bad CD!
I've decided that installing Fedora Core 4 from disks will make life cleaner. I could apply a partimage of my FC3 build or take the time to reimage the FC4 system, but I just want this to be over, so I am now in the middle of reinstalling FC4. The install has been crapping out on the Open Office RPMs. Seems there is a bad spot on the CD for them, so I deselected those packages for install. Only took me three FC4 installations and an hour to realize that I should do this. Oh, I'm just not very smart. :(

OK! What's left? Finishing the FC4 install and the stripe set.