Asho's Blog

raid5 setup and manager with mdadm on Debian Stable above

I am planning to set up a personal NAS for storaging.

I choose the raid5 as prevention and performance.....

Goal:
1: set up the raid5
2: LVM over raid5 in order to make partitions

Hardware:
3 disks for raid5
1 disk for spare
1 disk for additional

OS:
Debian Stable + kernel-2.4.18
LVM
mdadm

1:Set up raid5
vmware:~# mdadm -C /dev/md0 -l5 -n3 /dev/sdb1 /dev/sdc1 /dev/sdd1

vmware:~# cat /proc/mdstat
Personalities : [raid5]
read_ahead 1024 sectors
md0 : active raid5 scsi/host0/bus0/target3/lun0/part1[3] scsi/host0/bus0/target2/lun0/part1[1] scsi/host0/bus0/target1/lun0/part1[0]
8385664 blocks level 5, 64k chunk, algorithm 2 [3/2] [UU_]
[>....................] recovery = 2.2% (96844/4192832) finish=2.8min speed=24211K/sec
unused devices:

you may find out raid5 is trying to sync all the 3 disks

2: LVM over raid
vgscan # create /etc/lvmtab" and "/etc/lvmtab.d

vmware:~# pvcreate /dev/md0
pvcreate -- physical volume "/dev/md0" successfully created

vmware:~# vgcreate -s 32M ideraid5 /dev/md0
vgcreate -- INFO: maximum logical volume size is 2 Terabyte
vgcreate -- doing automatic backup of volume group "ideraid5"
vgcreate -- volume group "ideraid5" successfully created and activated

vmware:~# lvcreate -L 4000 ideraid5 (for home)
lvcreate -- doing automatic backup of "ideraid5"
lvcreate -- logical volume "/dev/ideraid5/lvol1" successfully created

vmware:~# lvcreate -L 4000 ideraid5 (for /usr/src my flavor)
lvcreate -- doing automatic backup of "ideraid5"
lvcreate -- logical volume "/dev/ideraid5/lvol2" successfully created

3: mount the LVM partitions
mkfs.xfs /dev/ideraid5/lvol1
mkfs.xfs /dev/ideraid5/lvol2
mount -t auto /dev/ideraid5/lvol1 /home
mount -t auto /dev/ideraid5/lvol2 /usr/src

4:Create spare disk
mdadm -a /dev/md0 /dev/sde1 # become spare disk

5:Simulating a drive failure
vmware:~# mdadm /dev/md0 -f /dev/sdd1
mdadm: set /dev/sdd1 faulty in /dev/md0

vmware:~# cat /proc/mdstat # because we set the spare disk so rebuild the raid5
Personalities : [raid5]
read_ahead 1024 sectors
md0 : active raid5 scsi/host0/bus0/target4/lun0/part1[3] scsi/host0/bus0/target3/lun0/part1[2](F) scsi/host0/bus0/target2/lun0/part1[1] scsi/host0/bus0/target1/lun0/part1[0]
8385664 blocks level 5, 64k chunk, algorithm 2 [3/2] [UU_]
[>....................] recovery = 3.1% (132804/4192832) finish=2.0min speed=33201K/sec
unused devices:

vmware:~# mdadm -D /dev/md0 # rebuild array
/dev/md0:
Version : 00.90.00
Creation Time : Mon Oct 4 18:00:23 2004
Raid Level : raid5
Array Size : 8385664 (7.99 GiB 8.58 GB)
Device Size : 4192832 (3.99 GiB 4.29 GB)
Raid Disks : 3
Total Disks : 4
Preferred Minor : 0
Persistance : Superblock is persistant

Update Time : Mon Oct 4 19:06:25 2004
State : dirty, no-errors
Active Drives : 2
Working Drives : 3
Failed Drives : 1
Spare Drives : 1

Layout : left-symmetric
Chunk Size : 64K

Number Major Minor RaidDisk State
0 8 17 0 active sync /dev/sdb1
1 8 33 1 active sync /dev/sdc1
2 8 49 2 faulty /dev/sdd1
3 8 65 3 /dev/sde1
UUID : 277448fd:35686cf4:068d4813:5447c9f4

vmware:~# mdadm -D /dev/md0 # after rebuild
/dev/md0:
Version : 00.90.00
Creation Time : Mon Oct 4 18:00:23 2004
Raid Level : raid5
Array Size : 8385664 (7.99 GiB 8.58 GB)
Device Size : 4192832 (3.99 GiB 4.29 GB)
Raid Disks : 3
Total Disks : 4
Preferred Minor : 0
Persistance : Superblock is persistant

Update Time : Mon Oct 4 19:08:43 2004
State : dirty, no-errors
Active Drives : 3
Working Drives : 3
Failed Drives : 0
Spare Drives : 1

Layout : left-symmetric
Chunk Size : 64K

Number Major Minor RaidDisk State
0 8 17 0 active sync /dev/sdb1
1 8 33 1 active sync /dev/sdc1
2 8 65 2 active sync /dev/sde1
UUID : 277448fd:35686cf4:068d4813:5447c9f4

vmware:~# mdadm /dev/md0 -r /dev/sdd1 # remove the failed disk

6: edit /etc/mdadm/mdadm.conf # can be detect while reboot
DEVICE /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1
ARRAY /dev/md0 UUID=1185fff6:e19598ba:ee9c5aa0:7bd7af5e

7: grow your raid5 disks
The kernel-2.6 in Debian doesn't support growing more disks to raid5.Try raidreconf in raidtools2 instead.

vmware:~# vim /etc/raidtab
# Sample raid-5 configuration
raiddev /dev/md0
raid-level 5
nr-raid-disks 3
chunk-size 64

# Parity placement algorithm

#parity-algorithm left-asymmetric

#
# the best one for maximum performance:
#
parity-algorithm left-symmetric

#parity-algorithm right-asymmetric
#parity-algorithm right-symmetric

# Spare disks for hot reconstruction
nr-spare-disks 1

device /dev/sdb1
raid-disk 0

device /dev/sdc1
raid-disk 1

device /dev/sdd1
raid-disk 2

device /dev/sde1
spare-disk 0

vmware:~# vim /etc/raidtab.new
# Sample raid-5 configuration
raiddev /dev/md0
raid-level 5
nr-raid-disks 4
chunk-size 64

# Parity placement algorithm

#parity-algorithm left-asymmetric

#
# the best one for maximum performance:
#
parity-algorithm left-symmetric

#parity-algorithm right-asymmetric
#parity-algorithm right-symmetric

# Spare disks for hot reconstruction
nr-spare-disks 1

device /dev/sdb1
raid-disk 0

device /dev/sdc1
raid-disk 1

device /dev/sdd1
raid-disk 2

device /dev/sde1
raid-disk 3

device /dev/sdf1
spare-disk 0

vmware:~# raidreconf -o /etc/raidtab -n /etc/raidtab.new -m /dev/md0
Working with device /dev/md0
Parsing /etc/raidtab
Parsing /etc/raidtab.new
Size of old array: 33543468 blocks, Size of new array: 41929335 blocks
Old raid-disk 0 has 65513 chunks, 4192832 blocks
Old raid-disk 1 has 65513 chunks, 4192832 blocks
Old raid-disk 2 has 65513 chunks, 4192832 blocks
Old raid-disk 3 has 65513 chunks, 4192832 blocks
New raid-disk 0 has 65513 chunks, 4192832 blocks
New raid-disk 1 has 65513 chunks, 4192832 blocks
New raid-disk 2 has 65513 chunks, 4192832 blocks
New raid-disk 3 has 65513 chunks, 4192832 blocks
New raid-disk 4 has 65513 chunks, 4192832 blocks
Using 64 Kbyte blocks to move from 64 Kbyte chunks to 64 Kbyte chunks.
Detected 256248 KB of physical memory in system
A maximum of 854 outstanding requests is allowed
---------------------------------------------------
I will grow your old device /dev/md0 of 196539 blocks
to a new device /dev/md0 of 262052 blocks
using a block-size of 64 KB
Is this what you want? (yes/no): yes
Converting 196539 block device to 262052 block device
Allocated free block map for 4 disks
5 unique disks detected.
Working (|) [00124932/00196539] [###########################
Source drained, flushing sink.
Reconfiguration succeeded, will update superblocks...
Updating superblocks...
handling MD device /dev/md0
analyzing super-block
disk 0: /dev/sdb1, 4192933kB, raid superblock at 4192832kB
disk 1: /dev/sdc1, 4192933kB, raid superblock at 4192832kB
disk 2: /dev/sdd1, 4192933kB, raid superblock at 4192832kB
disk 3: /dev/sde1, 4192933kB, raid superblock at 4192832kB
disk 4: /dev/sdf1, 4192933kB, raid superblock at 4192832kB
Array is updated with kernel.
Disks re-inserted in array... Hold on while starting the array...
Maximum friend-freeing depth: 6
Total wishes hooked: 196539
Maximum wishes hooked: 854
Total gifts hooked: 196539
Maximum gifts hooked: 807
Congratulations, your array has been reconfigured,
and no errors seem to have occured.

8:resize your md
Ext2/Ext3:resize2fs
Reiserfs:resize_reiserfs
XFS:please mount all your partitions first and try xfs_growfs

PS:
1:please mv the /etc/rcS/S25lvm to SXXlvm which XX must bigger than 25 if you use LVM over raid
2:want to grow your raid5 fs with mdamd ?!
please install Debian Sarge and kernel-2.6. Or...use raidtools2 with raidreconf

Appendix:reference
mdadm A New Tool For Linux Software RAID Management
SoftRaid HOWTO

Posted by asho at October 4, 2004 12:24 PM