How to replace Faulty Linux RAID disk
Suppose that a disk in the RAID array failed. In that case, you need replace Faulty Linux RAID disk. You must have seen my post about creating RAID 1 array same way I have created Raid 5 array with below command, so that I can demonstrate how we can replace Faulty Linux RAID disk.
[root@rhel1 ~]# mdadm -Cv /dev/md1 --level=5 -n3 /dev/sdc1 /dev/sdd1 /dev/sde1 mdadm: layout defaults to left-symmetric mdadm: layout defaults to left-symmetric mdadm: layout defaults to left-symmetric mdadm: chunk size defaults to 512K mdadm: layout defaults to left-symmetric mdadm: /dev/sdd1 appears to be part of a raid array: level=raid0 devices=0 ctime=Thu Jan 1 06:00:00 1970 mdadm: partition table exists on /dev/sdd1 but will be lost or meaningless after creating array mdadm: layout defaults to left-symmetric mdadm: size set to 1043968K Continue creating array? y mdadm: Defaulting to version 1.2 metadata mdadm: array /dev/md1 started.
Recheck now:
[root@rhel1 ~]# mdadm -D /dev/md1 /dev/md1: Version : 1.2 Creation Time : Wed Dec 7 19:05:53 2016 Raid Level : raid5 Array Size : 2087936 (2039.34 MiB 2138.05 MB) Used Dev Size : 1043968 (1019.67 MiB 1069.02 MB) Raid Devices : 3 Total Devices : 3 Persistence : Superblock is persistent Update Time : Wed Dec 7 19:05:58 2016 State : clean Active Devices : 3 Working Devices : 3 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 512K Name : rhel1.lab.com:1 (local to host rhel1.lab.com) UUID : fcef1f48:223e87c7:53e9eecc:d5f55e79 Events : 18 Number Major Minor RaidDevice State 0 8 33 0 active sync /dev/sdc1 1 8 49 1 active sync /dev/sdd1 3 8 65 2 active sync /dev/sde1
For demo purpose , You can manually fail a disk to see what it would be like if a disk failed in the array, so that I can demonstrate replace Faulty Linux RAID disk.
[root@rhel1 ~]# mdadm /dev/md1 -f /dev/sdd1 mdadm: set /dev/sdd1 faulty in /dev/md1
Check the status of md1
[root@rhel1 ~]# mdadm -D /dev/md1 /dev/md1: Version : 1.2 Creation Time : Wed Dec 7 19:05:53 2016 Raid Level : raid5 Array Size : 2087936 (2039.34 MiB 2138.05 MB) Used Dev Size : 1043968 (1019.67 MiB 1069.02 MB) Raid Devices : 3 Total Devices : 3 Persistence : Superblock is persistent Update Time : Wed Dec 7 19:07:26 2016 State : clean, degraded Active Devices : 2 Working Devices : 2 Failed Devices : 1 Spare Devices : 0 Layout : left-symmetric Chunk Size : 512K Name : rhel1.lab.com:1 (local to host rhel1.lab.com) UUID : fcef1f48:223e87c7:53e9eecc:d5f55e79 Events : 19 Number Major Minor RaidDevice State 0 8 33 0 active sync /dev/sdc1 1 0 0 1 removed 3 8 65 2 active sync /dev/sde1 1 8 49 - faulty spare /dev/sdd1
you can see that /dev/sdd1 is marked as faulty now remove the same from configuration using below command:
[root@rhel1 ~]# mdadm /dev/md1 -r /dev/sdd1 mdadm: hot removed /dev/sdd1 from /dev/md1
When you have replace Faulty Linux RAID disk, you need to make it active in the array. First, you need to partition the disk like you did originally when setting up the RAID array.
When the disk is partitioned, you can add it back to the array as follows:
[root@rhel1 ~]# mdadm /dev/md1 -a /dev/sdd1 mdadm: added /dev/sdd1
Verify that it has been added properly, you can see that its in rebuilding phase:
[root@rhel1 ~]# mdadm -D /dev/md1 /dev/md1: Version : 1.2 Creation Time : Wed Dec 7 19:05:53 2016 Raid Level : raid5 Array Size : 2087936 (2039.34 MiB 2138.05 MB) Used Dev Size : 1043968 (1019.67 MiB 1069.02 MB) Raid Devices : 3 Total Devices : 3 Persistence : Superblock is persistent Update Time : Wed Dec 7 19:09:35 2016 State : clean, degraded, recovering Active Devices : 2 Working Devices : 3 Failed Devices : 0 Spare Devices : 1 Layout : left-symmetric Chunk Size : 512K Rebuild Status : 10% complete Name : rhel1.lab.com:1 (local to host rhel1.lab.com) UUID : fcef1f48:223e87c7:53e9eecc:d5f55e79 Events : 50 Number Major Minor RaidDevice State 0 8 33 0 active sync /dev/sdc1 4 8 49 1 spare rebuilding /dev/sdd1 3 8 65 2 active sync /dev/sde1
After some time all will become in active:sync state as below:
[root@rhel1 ~]# mdadm -D /dev/md1 /dev/md1: Version : 1.2 Creation Time : Wed Dec 7 19:05:53 2016 Raid Level : raid5 Array Size : 2087936 (2039.34 MiB 2138.05 MB) Used Dev Size : 1043968 (1019.67 MiB 1069.02 MB) Raid Devices : 3 Total Devices : 3 Persistence : Superblock is persistent Update Time : Wed Dec 7 19:09:39 2016 State : clean Active Devices : 3 Working Devices : 3 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 512K Name : rhel1.lab.com:1 (local to host rhel1.lab.com) UUID : fcef1f48:223e87c7:53e9eecc:d5f55e79 Events : 68 Number Major Minor RaidDevice State 0 8 33 0 active sync /dev/sdc1 4 8 49 1 active sync /dev/sdd1 3 8 65 2 active sync /dev/sde1 [root@rhel1 ~]#