I am trying to make mdadm call into a simple bash script which writes a message in the kernel log in case of a state change. The script is as follows,

# cat /tmp/test.sh
#!/bin/bash

echo "raid array status change" > /dev/kmsg

I have added the following to the mdadm config file

PROGRAM /tmp/test.sh
ARRAY /dev/md0 UUID=

But when I do the test, it says the md0 is being picked up, and the correct program option is being used, but nothing gets printed in dmesg

mdadm --monitor --test /dev/md0 -1

Note that when I manually run the script, it prints the message in kernel log. My questions 1. Does the above process of calling the program depend on the mail setting also? Because my mail config is not set. 2. Any idea what could be wrong or missing?

A little bit of background first. I store a bunch of data on a Thecus N4200Pro NAS array. I had gotten a report that one of the 4 drives in the array was showing smart errors. - So I swapped out the offending drive (#4) and it got to work rebuilding. About 60% into the rebuild one of the other drives in the array drops out, #1 in this case. - Great.. I shut down and try swapping back in the original #4 to see if it will come back up. No dice. - So I shut down and swap #1 & #2 to see if they can recover with the bad drive swapped around and replace the #4 with the half-rebuilt #4. In hindsight this was bad. I should have shut down after the first one and cloned all the original discs from there. - The device boots back up and of course the raid fails to assemble, showing only discs 3 and 4, 4 being marked as a spare. - At this point I shut everything down and pull all the discs and clone them, making sure to keep track of the number order. - I put all 4 cloned discs into my unbutu 16.04 LTS box in the correct drive order and booted up. - All 4 discs show up, and show the partitions in Disks. It shows a raid5 array and a raid1 array as well. The raid1 array is the system info for the NAS, not really concerned with that. The raid5 array is the one I'm interested in with all my data on it, but I can't access anything on it. So time to start digging. First I ran cat /proc/mdstat to see the arrays- jake@ubuntu-box:~$ cat /proc/mdstat Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] md0 : active raid1 sdd1 1959884 blocks super 1.0 [4/1] [___U] md1 : inactive sdd2(S) sdc2(S) sdb2(S) sda2(S) 3899202560 blocks unused devices: Ok, sees two arrays. So we get the details on md1 from: mdadm --detail /dev/md1 jake@ubuntu-box:~$ sudo mdadm --detail /dev/md1 /dev/md1: Version : 0.90 Raid Level : raid0 Total Devices : 4 Preferred Minor : 0 Persistence : Superblock is persistent State : inactive UUID : e7ab07c3:b9ffa9ae:377e3cd3:a8ece374 Events : 0.14344 Number Major Minor RaidDevice - 8 50 - /dev/sdd2 - 8 34 - /dev/sdc2 - 8 18 - /dev/sdb2 - 8 2 - /dev/sda2[/CODE] Hrmm.. that's odd. showing the raid as raid0, which is not the case. Ok, lets check out each individual partition with: mdadm --examine /dev/sdXX Disc 1 jake@ubuntu-box:~$ sudo mdadm --examine /dev/sda2/ dev/sda2: Magic : a92b4efc Version : 0.90.00 UUID : e7ab07c3:b9ffa9ae:377e3cd3:a8ece374 Creation Time : Thu Aug 18 14:30:36 2011 Raid Level : raid5 Used Dev Size : 974800000 (929.64 GiB 998.20 GB) Array Size : 2924400000 (2788.93 GiB 2994.59 GB) Raid Devices : 4 Total Devices : 4 Preferred Minor : 1 Update Time : Tue Mar 13 14:00:33 2018 State : clean Active Devices : 3 Working Devices : 4 Failed Devices : 1 Spare Devices : 1 Checksum : e52c5f8 - correct Events : 20364 Layout : left-symmetric Chunk Size : 64K Number Major Minor RaidDevice State this 0 8 2 0 active sync /dev/sda2 0 0 8 2 0 active sync /dev/sda2 1 1 8 18 1 active sync /dev/sdb2 2 2 8 34 2 active sync /dev/sdc2 3 3 0 0 3 faulty removed 4 4 8 50 4 spare /dev/sdd2 Disc 2 jake@ubuntu-box:~$ sudo mdadm --examine /dev/sdb2/ dev/sdb2: Magic : a92b4efc Version : 0.90.00 UUID : e7ab07c3:b9ffa9ae:377e3cd3:a8ece374 Creation Time : Thu Aug 18 14:30:36 2011 Raid Level : raid5 Used Dev Size : 974800000 (929.64 GiB 998.20 GB) Array Size : 2924400000 (2788.93 GiB 2994.59 GB) Raid Devices : 4 Total Devices : 4 Preferred Minor : 1 Update Time : Tue Mar 13 14:56:30 2018 State : clean Active Devices : 2 Working Devices : 3 Failed Devices : 1 Spare Devices : 1 Checksum : e597e42 - correct Events : 238868 Layout : left-symmetric Chunk Size : 64K Number Major Minor RaidDevice State this 1 8 18 1 active sync /dev/sdb2 0 0 0 0 0 removed 1 1 8 18 1 active sync /dev/sdb2 2 2 8 34 2 active sync /dev/sdc2 3 3 0 0 3 faulty removed 4 4 8 50 4 spare /dev/sdd2 Disc 3 jake@ubuntu-box:~$ sudo mdadm --examine /dev/sdc2/ dev/sdc2: Magic : a92b4efc Version : 0.90.00 UUID : e7ab07c3:b9ffa9ae:377e3cd3:a8ece374 Creation Time : Thu Aug 18 14:30:36 2011 Raid Level : raid5 Used Dev Size : 974800000 (929.64 GiB 998.20 GB) Array Size : 2924400000 (2788.93 GiB 2994.59 GB) Raid Devices : 4 Total Devices : 3 Preferred Minor : 1 Update Time : Tue Mar 13 15:10:07 2018 State : clean Active Devices : 1 Working Devices : 2 Failed Devices : 2 Spare Devices : 1 Checksum : e598570 - correct Events : 239374 Layout : left-symmetric Chunk Size : 64K Number Major Minor RaidDevice State this 2 8 34 2 active sync /dev/sdc2 0 0 0 0 0 removed 1 1 0 0 1 faulty removed 2 2 8 34 2 active sync /dev/sdc2 3 3 0 0 3 faulty removed 4 4 8 50 4 spare /dev/sdd2 and Disc 4 jake@ubuntu-box:~$ sudo mdadm --examine /dev/sdd2/ dev/sdd2: Magic : a92b4efc Version : 0.90.00 UUID : e7ab07c3:b9ffa9ae:377e3cd3:a8ece374 Creation Time : Thu Aug 18 14:30:36 2011 Raid Level : raid5 Used Dev Size : 974800000 (929.64 GiB 998.20 GB) Array Size : 2924400000 (2788.93 GiB 2994.59 GB) Raid Devices : 4 Total Devices : 4 Preferred Minor : 1 Update Time : Tue Mar 13 11:03:10 2018 State : clean Active Devices : 4 Working Devices : 4 Failed Devices : 0 Spare Devices : 0 Checksum : e526d87 - correct Events : 14344 Layout : left-symmetric Chunk Size : 64K Number Major Minor RaidDevice State this 3 8 50 3 active sync /dev/sdd2 0 0 8 2 0 active sync /dev/sda2 1 1 8 18 1 active sync /dev/sdb2 2 2 8 34 2 active sync /dev/sdc2 3 3 8 50 3 active sync /dev/sdd2 So - Magic numbers and UUID are all good between the set. Events are all out of whack because it had tried to rebuild the replaced #4 as a spare instead of just rebuilding #4 Disc 4 has the correct info for the raid, and the sequencing as it was the drive I pulled originally and didn't get anything re-written. Discs 1-3 are showing in various states of chaos from swapping things around. So two questions - 1. Why is it showing up as raid0 in the mdadm --detail 2. Is it possible to update the info for the first three discs that I got from the mdadm --examine /dev/sdd2 so that it sees everything as it should be, instead of the cluster that I inadvertently made of it. I *think* if I can find a way to update the info for those partitions or discs the raid should reassemble correctly and rebuild itself so I can access my data Any ideas would be helpful, as I've gotten about as far as I can get trying to figure this out on my own and doing a ton of searching.

Background ===== I'm running Centos 7. Originally, it was running on a single disk that looked something like this: 1 200M EFI System (/boot/efi) 2 500M Microsoft basic (/boot) 3 465.1G Linux LVM LVM VG centos - LVM LV ext4 centos-root (/) - LVM LV swap centos-swap (swap) This was just a temporary solution as it was originally supposed to be installed on a Linux software RAID1 array. I got around to migrating it today. This is what it currently looks like: Both new disks have this partition layout: 1 200M EFI System (/boot/efi) 2 457.6G Linux RAID /dev/md0 RAID1 (for boot and LVM) 3 8G Linux RAID /dev/md1 RAID0 (so 16GB total, for swap) /dev/md0 looks like this: 1 500M Linux filesystem (/boot) 2 457G Linux LVM (centos-root is migrated to this) LVM now has only one LV, centos-root /etc/mdadm.conf looks like this: ARRAY /dev/md0 level=raid1 num-devices=2 metadata=1.2 name=main.centos.local:0 UUID=5b5057b4:4235ba4b:5342dfda:acf63302 devices=/dev/sda2,/dev/sdb2 ARRAY /dev/md1 level=raid0 num-devices=2 metadata=1.2 name=main.centos.local:1 UUID=f82a8c99:9b391d83:4efc9456:9e9bad98 devices=/dev/sda3,/dev/sdb3 /etc/fstab looks like this: /dev/mapper/centos-root / xfs defaults 0 0 UUID=fcb5f82f-ce6b-460b-800f-329e010bc403 /boot xfs defaults 0 0 UUID=C532-14AE /boot/efi vfat umask=0077,shortname=winnt 0 0 /dev/md1 swap swap defaults 0 0 blkid outputs this (relevant entries only): /dev/sdb1: SEC_TYPE="msdos" UUID="C532-14AE" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="ed301bbd-c15c-40af-ae75-bf238d0e6270" /dev/sda1: SEC_TYPE="msdos" UUID="C532-14AE" TYPE="vfat" PARTLABEL="EFI System Partition" PARTUUID="f3a76412-41a0-4e04-9b04-ad1c159133cf" /dev/md0p1: LABEL="boot" UUID="fcb5f82f-ce6b-460b-800f-329e010bc403" TYPE="xfs" PARTLABEL="primary" PARTUUID="df8d6481-c6ce-423a-b5d5-205d355e5653" /dev/md0p2: UUID="7LfywM-oPHy-MTEt-swlI-EVbZ-opTo-m82E6R" TYPE="LVM2_member" PARTLABEL="primary" PARTUUID="19e7f9d5-a955-4036-8338-03a748faa1f6" /dev/mapper/centos-root: UUID="deaa9788-b487-4991-adf7-2945788fb6cd" TYPE="xfs" I have a script which automatically mounts the other EFI partition to /boot/efi_[device], and when the kernel is updated, the grub.cfg gets copied to this partition to keep everything in sync. /dev/sda1 and /dev/sdb1 are kept in sync by the script (I've verified this), so it shouldn't be an issue that fstab mounts either one to /boot/efi (this also means that if one drive was removed due to failure, the system is still guaranteed to boot). I could have put swap in a LV to simplify things, but the RAID0 gets better performance (for what it's worth) and I get an extra 16GB of space. I migrated the LV from the old drive to the new PV using the following commands: pvcreate /dev/md0p2 vgextend centos /dev/md0p2 pvmove /dev/sdg3 vgreduce centos /dev/sdg3 Then I regenerated the initramfs with dracut (after backing up the original), and finally regenerated grub.cfg. Afterwards, I mounted the new /boot and /boot/efi partitions and copied everything over. Problem ===== After disconnecting the old drive and booting, dracut fails to find my RAID arrays, and of course the /boot partition and my LVG as well. It appears that it's simply not calling mdadm --assemble on /dev/md0 and /dev/md. I'm able to do just that from the dracut prompt, after which lvm_scan finds my LVG, I can link /dev/centos/root to /dev/root, and the system continues booting without any problems once exiting the prompt. Everything seems to be exactly where it should be. There was a kernel update available, so I tried installing it (assuming I messed something up the first time around when regenerating the initramfs and grub.cfg files), but no dice. System still fails in the exact same way. This is true when I boot from either EFI partition manually (as it should be since the two are identical). Link to rdsosreport.txt on pastebin What am I missing here? How do I get dracut to assemble my arrays?

LVM and mdadm/dmraid both offer software RAID functionality on Linux. This is essentially a follow-up to a question from 2014 . Back then, @derobert recommended using mdadm over LVM RAID due to its maturity — but that was more than four years ago. I imagine things may have changed since then. However, I’ve never used LVM RAID before, and I couldn't find many recent experiences or discussions about it. So, what’s the current state of LVM RAID? Has it become more mature? Have the flaws mentioned in @derobert’s post been resolved, or do they still exist? Specifically, how does it compare to mdadm in terms of: - Stability - Features (grow, shrink, convert) - Repair and recovery - Community support - Performance I’d like to know if people actually use LVM RAID now, or if most still stick with mdadm. Is it more advisable to use LVM on top of mdadm for logical volume management, or is it now acceptable to let LVM manage the RAID as well? Would it even make sense to use LVM RAID instead of mdadm, even if you don’t plan to take advantage of logical volume management? I considered commenting on the original answer and asking @derobert for an update, but I decided to post a new question to reach more community members and gather fresh perspectives — not just update the original post to the present tense.

I have created a raid5 array with 4 disks. Initially i had 3x 3tb and 1x 4tb (because 3tb was unavailable at the time). After some years i have replaced most of these disks and have come to the point where all array disks are now 4tb in size. Still my mdadm array is 3TB. Is there any way to change the mdadm array size to match the 4tb disk size w/o loosing my data? Thanks for your help!

today I run in a disaster... I have a RAID 6 with 7 HDDs and yesterday one disk failed. After replacing the disk and did a rebuild over night I found out that a 2nd HDD was out of the RAID... So today I 've started to backup my Files on external Drives but then the copying stopped and as I've checked why and saw in Webmins RAID that sdg was "down". I shut down the server and checked the hardware and found out, that the backplate, where the HDDs are connected got lose... After repairing it all drives are now back but my RAID 6 don't start anymore :-/ dmesg shows me: md: kicking non-fresh sdg from array! md: kicking non-fresh sdf from array! md: kicking non-fresh sde from array! md/raid:md0: not enough operational devices (3/7 failed) ... and after many md0: ADD_NEW_DISK not supported I can read this: EXT4-fs (md0): unable to read superblock With sudo mdadm --examine I checked the sdg, sdf and sde and e and f shows "State clean" where the sdg, which was "down" before repairing shows "Active". So 6 of 7 Devices shows "Clean" except the sdg. Here is the list of the output of all devices:

Disk sdb
/dev/sdb:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : e866cf54:90d5c74e:fe00b6e7:d25c82f4
           Name : N5550:0  (local to host N5550)
  Creation Time : Fri Oct 29 14:43:58 2021
     Raid Level : raid6
   Raid Devices : 7

 Avail Dev Size : 3906770096 (1862.89 GiB 2000.27 GB)
     Array Size : 9766906880 (9314.45 GiB 10001.31 GB)
  Used Dev Size : 3906762752 (1862.89 GiB 2000.26 GB)
    Data Offset : 259072 sectors
   Super Offset : 8 sectors
   Unused Space : before=258992 sectors, after=7344 sectors
          State : clean
    Device UUID : 9180f101:1dacdd9e:4adae9c4:fbeb2552

Internal Bitmap : 8 sectors from superblock
    Update Time : Sat Mar 26 18:13:45 2022
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : 38019182 - correct
         Events : 256508

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAA.A.. ('A' == active, '.' == missing, 'R' == replacing)
Disk sdc
/dev/sdc:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : e866cf54:90d5c74e:fe00b6e7:d25c82f4
           Name : N5550:0  (local to host N5550)
  Creation Time : Fri Oct 29 14:43:58 2021
     Raid Level : raid6
   Raid Devices : 7

 Avail Dev Size : 3906770096 (1862.89 GiB 2000.27 GB)
     Array Size : 9766906880 (9314.45 GiB 10001.31 GB)
  Used Dev Size : 3906762752 (1862.89 GiB 2000.26 GB)
    Data Offset : 259072 sectors
   Super Offset : 8 sectors
   Unused Space : before=258992 sectors, after=7344 sectors
          State : clean
    Device UUID : 889c6877:5ee5c647:eebd209c:d9c6abcb

Internal Bitmap : 8 sectors from superblock
    Update Time : Sat Mar 26 18:13:45 2022
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : a71ea53d - correct
         Events : 256508

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AAA.A.. ('A' == active, '.' == missing, 'R' == replacing)
Disk sdd
/dev/sdd:
   MBR Magic : aa55
Partition :   3907026944 sectors at         2048 (type fd)
Disk sde
/dev/sde:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : e866cf54:90d5c74e:fe00b6e7:d25c82f4
           Name : N5550:0  (local to host N5550)
  Creation Time : Fri Oct 29 14:43:58 2021
     Raid Level : raid6
   Raid Devices : 7

 Avail Dev Size : 3906770096 (1862.89 GiB 2000.27 GB)
     Array Size : 9766906880 (9314.45 GiB 10001.31 GB)
  Used Dev Size : 3906762752 (1862.89 GiB 2000.26 GB)
    Data Offset : 259072 sectors
   Super Offset : 8 sectors
   Unused Space : before=258992 sectors, after=7344 sectors
          State : clean
    Device UUID : 34198042:3d4c802b:36727b02:fdf65808

Internal Bitmap : 8 sectors from superblock
    Update Time : Sat Mar 26 18:05:00 2022
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : f8fb6b18 - correct
         Events : 256494

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AAAAA.. ('A' == active, '.' == missing, 'R' == replacing)
Disk sdf
/dev/sdf:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : e866cf54:90d5c74e:fe00b6e7:d25c82f4
           Name : N5550:0  (local to host N5550)
  Creation Time : Fri Oct 29 14:43:58 2021
     Raid Level : raid6
   Raid Devices : 7

 Avail Dev Size : 3906770096 (1862.89 GiB 2000.27 GB)
     Array Size : 9766906880 (9314.45 GiB 10001.31 GB)
  Used Dev Size : 3906762752 (1862.89 GiB 2000.26 GB)
    Data Offset : 259072 sectors
   Super Offset : 8 sectors
   Unused Space : before=258992 sectors, after=7344 sectors
          State : clean
    Device UUID : b2e8d640:1f21336f:88d823fe:66ef7be7

Internal Bitmap : 8 sectors from superblock
    Update Time : Wed Mar 23 14:46:56 2022
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : 15cd05bb - correct
         Events : 238681

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 4
   Array State : AAAAAA. ('A' == active, '.' == missing, 'R' == replacing)
Disk sdg
/dev/sdg:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : e866cf54:90d5c74e:fe00b6e7:d25c82f4
           Name : N5550:0  (local to host N5550)
  Creation Time : Fri Oct 29 14:43:58 2021
     Raid Level : raid6
   Raid Devices : 7

 Avail Dev Size : 3906770096 (1862.89 GiB 2000.27 GB)
     Array Size : 9766906880 (9314.45 GiB 10001.31 GB)
  Used Dev Size : 3906762752 (1862.89 GiB 2000.26 GB)
    Data Offset : 259072 sectors
   Super Offset : 8 sectors
   Unused Space : before=258992 sectors, after=7344 sectors
          State : active
    Device UUID : 2bc06e22:49aa73e2:3cf7eb79:55df1180

Internal Bitmap : 8 sectors from superblock
    Update Time : Sat Mar 26 17:57:06 2022
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : 7f0ddb2a - correct
         Events : 256372

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 5
   Array State : AAAAAA. ('A' == active, '.' == missing, 'R' == replacing)
Disk sdh
/dev/sdh:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : e866cf54:90d5c74e:fe00b6e7:d25c82f4
           Name : N5550:0  (local to host N5550)
  Creation Time : Fri Oct 29 14:43:58 2021
     Raid Level : raid6
   Raid Devices : 7

 Avail Dev Size : 3906770096 (1862.89 GiB 2000.27 GB)
     Array Size : 9766906880 (9314.45 GiB 10001.31 GB)
  Used Dev Size : 3906762752 (1862.89 GiB 2000.26 GB)
    Data Offset : 259072 sectors
   Super Offset : 8 sectors
   Unused Space : before=258992 sectors, after=7344 sectors
          State : clean
    Device UUID : 7af89a18:52ef08ae:dec5ad7b:75626355

Internal Bitmap : 8 sectors from superblock
    Update Time : Sat Mar 26 18:13:45 2022
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : 17d7b107 - correct
         Events : 256508

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 4
   Array State : AAA.A.. ('A' == active, '.' == missing, 'R' == replacing)

I've tried to start the RAID with mdadm --run /dev/md0 and get: mdadm: failed to start array /dev/md0: Input/output error But after I started it with this Webmin shows me then: /dev/md0 active, FAILED, Not Started RAID6 (Dual Distributed Parity) 7.27 TiB Its 7.27 from 9TB. Any ideas how to get my RAID back to work again without data loss? I've read about that I could add devices back again to the RAID but I'm unsure and wanted to ask before. Any help would be appreciated! **UPDATE**: I forgot that one of the device is /dev/sdd1 and not /sdd! Here the examine of it: ~~~ /dev/sdd1: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : e866cf54:90d5c74e:fe00b6e7:d25c82f4 Name : N5550:0 (local to host N5550) Creation Time : Fri Oct 29 14:43:58 2021 Raid Level : raid6 Raid Devices : 7 Avail Dev Size : 3906767872 (1862.89 GiB 2000.27 GB) Array Size : 9766906880 (9314.45 GiB 10001.31 GB) Used Dev Size : 3906762752 (1862.89 GiB 2000.26 GB) Data Offset : 259072 sectors Super Offset : 8 sectors Unused Space : before=258992 sectors, after=5120 sectors State : clean Device UUID : d8df004e:44ee4060:ba4d2c22:e7e6bdcb Internal Bitmap : 8 sectors from superblock Update Time : Sat Mar 26 18:13:45 2022 Bad Block Log : 512 entries available at offset 16 sectors Checksum : 1c4e98a4 - correct Events : 256508 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 2 Array State : AAA.A.. ('A' == active, '.' == missing, 'R' == replacing) ~~~ And here an mdadm -D /dev/md0: ~~~ /dev/md0: Version : 1.2 Raid Level : raid0 Total Devices : 7 Persistence : Superblock is persistent State : inactive Working Devices : 7 Name : N5550:0 (local to host N5550) UUID : e866cf54:90d5c74e:fe00b6e7:d25c82f4 Events : 256494 Number Major Minor RaidDevice - 8 64 - /dev/sde - 8 32 - /dev/sdc - 8 112 - /dev/sdh - 8 80 - /dev/sdf - 8 16 - /dev/sdb - 8 49 - /dev/sdd1 - 8 96 - /dev/sdg ~~~

I am facing an issue with my RAID 1 (mdadm softraid) on an AlmaLinux/CloudLinux OS server, which is a production server with live data. Here's the chronology of events: 1. Initially, I created a RAID 1 array with two 1TB NVMe disks (2 x 1TB). 2. At some point, the second NVMe disk failed. I replaced it with a new 2TB NVMe disk. I then added this new 2TB NVMe disk to the RAID array, but it was partitioned/configured to match the 1TB capacity of the remaining active disk. 3. Currently, the first 1TB disk has failed and was automatically kicked out by the RAID system when I rebooted the server. So, only the 2TB NVMe disk (which is currently acting as a 1TB member of the degraded RAID) remains. **Replacement and Setup Plan** I have already replaced the failed 1TB disk with a new 2TB NVMe disk. I want to utilize the full 2TB capacity since both disks are now 2 x 2TB.

[root@id1 ~]# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md124 : active raid5 sdd2 sdc2 sda2
      62945280 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
      bitmap: 1/1 pages [4KB], 65536KB chunk

md125 : active raid5 sdd1 sdc1 sda1
      1888176128 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
      bitmap: 7/8 pages [28KB], 65536KB chunk

md126 : active raid5 sda3 sdc3 sdd3
      2097152 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md127 : active raid1 nvme1n1p1
      976628736 blocks super 1.2 [2/1] [_U]
      bitmap: 8/8 pages [32KB], 65536KB chunk

unused devices:

--- mdadm --detail /dev/md127

[root@id1 ~]# mdadm --detail /dev/md127
/dev/md127:
           Version : 1.2
     Creation Time : Tue Aug 29 05:57:10 2023
        Raid Level : raid1
        Array Size : 976628736 (931.39 GiB 1000.07 GB)
     Used Dev Size : 976628736 (931.39 GiB 1000.07 GB)
      Raid Devices : 2
     Total Devices : 1
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Thu May 29 01:33:09 2025
             State : active, degraded
    Active Devices : 1
   Working Devices : 1
    Failed Devices : 0
     Spare Devices : 0

Consistency Policy : bitmap

              Name : idweb.webserver.com:root  (local to host idweb.webserver.com)
              UUID : 3fb9f52f:45f39d12:e7bb3392:8eb1481f
            Events : 33132451

    Number   Major   Minor   RaidDevice State
       -       0        0        0      removed
       2     259        2        1      active sync   /dev/nvme1n1p1

--- lsblk -o NAME,FSTYPE,SIZE,MOUNTPOINT,LABEL

[root@id1 ~]# lsblk -o NAME,FSTYPE,SIZE,MOUNTPOINT,LABEL
NAME        FSTYPE              SIZE MOUNTPOINT LABEL
sda                           931.5G
├─sda1      linux_raid_member 900.5G            web1srv.serverhostweb.com:home2
│ └─md125   ext4                1.8T /home2
├─sda2      linux_raid_member    30G            web1srv.serverhostweb.com:tmp
│ └─md124   ext4                 60G /var/tmp
└─sda3      linux_raid_member     1G            web1srv.serverhostweb.com:boot
  └─md126   xfs                   2G /boot
sdb         ext4                5.5T
sdc                           931.5G
├─sdc1      linux_raid_member 900.5G            web1srv.serverhostweb.com:home2
│ └─md125   ext4                1.8T /home2
├─sdc2      linux_raid_member    30G            web1srv.serverhostweb.com:tmp
│ └─md124   ext4                 60G /var/tmp
└─sdc3      linux_raid_member     1G            web1srv.serverhostweb.com:boot
  └─md126   xfs                   2G /boot
sdd                           931.5G
├─sdd1      linux_raid_member 900.5G            web1srv.serverhostweb.com:home2
│ └─md125   ext4                1.8T /home2
├─sdd2      linux_raid_member    30G            web1srv.serverhostweb.com:tmp
│ └─md124   ext4                 60G /var/tmp
└─sdd3      linux_raid_member     1G            web1srv.serverhostweb.com:boot
  └─md126   xfs                   2G /boot
nvme0n1                         1.8T
nvme1n1                         1.8T
└─nvme1n1p1 linux_raid_member 931.5G            web1srv.serverhostweb.com:root
  └─md127   ext4              931.4G /

What are the steps to repair my soft RAID 1, maximize the storage to 2TB, and ensure the data remains safe? I have some example step but not really sure, does the below step right?:

# Create a partition on the new disk with a full size of 2TB
fdisk /dev/nvme0n1

mdadm --manage /dev/md127 --add /dev/nvme0n1p1

# Wait for First Sync

# Fail and remove the old disk
mdadm --manage /dev/md127 --fail /dev/nvme1n1p1
mdadm --manage /dev/md127 --remove /dev/nvme1n1p1

# Repartition the old disk for full 2TB
gdisk /dev/nvme1n1

# Add back to RAID
mdadm --manage /dev/md127 --add /dev/nvme1n1p1

# Wait for Second Sync

# Expand RAID array to maximum
mdadm --grow /dev/md127 --size=max

# Verify new size
mdadm --detail /dev/md127

# Resize ext4 filesystem
resize2fs /dev/md127

# Update mdadm.conf
mdadm --detail --scan > /etc/mdadm.conf

# Update initramfs
dracut -f

Server Spec: - Os Almalinux/Cloudlinux 8

I'm running debian stable with a 2 x nvme Raid 1. Here is the hardware/hoster it's running on https://www.hetzner.com/dedicated-rootserver/ex62-nvme?country=us Almost every second day mdadm monitoring reports a fail event and leaves the array degraded. It only disables 1 partition as you can see here: This is an automatically generated mail message from mdadm running on xxx A Fail event had been detected on md device /dev/md/2. It could be related to component device /dev/nvme1n1p3. Faithfully yours, etc. P.S. The /proc/mdstat file currently contains the following: Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] md2 : active raid1 nvme1n1p3(F) nvme0n1p3 465895744 blocks super 1.2 [2/1] [U_] bitmap: 4/4 pages [16KB], 65536KB chunk md0 : active (auto-read-only) raid1 nvme1n1p1 nvme0n1p1 33521664 blocks super 1.2 [2/2] [UU] md1 : active raid1 nvme0n1p2 nvme1n1p2 523712 blocks super 1.2 [2/2] [UU] unused devices: This happens on both disks. One time it's nvme0n1p3 and next time it's nvme1n1p3. I then just re-add the failed partition with mdadm --re-add /dev/md2 /dev/nvme0n1p3 or mdadm --re-add /dev/md2 /dev/nvme1n1p3 and after the resync it works for a day or two. In dmesg I found this: [94879.144892] nvme nvme1: I/O 311 QID 1 timeout, reset controller [94879.252851] nvme nvme1: completing aborted command with status: 0007 [94879.252970] blk_update_request: I/O error, dev nvme1n1, sector 452352001 [94879.253091] nvme nvme1: completing aborted command with status: fffffffc [94879.253223] blk_update_request: I/O error, dev nvme1n1, sector 68159504 [94879.253418] md: super_written gets error=-5 I tried to check the health of the devices with these commands, but they don't give me stats like "Reallocated_Sector_Ct" or "Reported_Uncorrect". smartctl -x /dev/nvme1 smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.9.0-8-amd64] (local build) Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Number: KXG50ZNV512G TOSHIBA Serial Number: 28SS10F6TYST Firmware Version: AAGA4102 PCI Vendor/Subsystem ID: 0x1179 IEEE OUI Identifier: 0x00080d Total NVM Capacity: 512,110,190,592 [512 GB] Unallocated NVM Capacity: 0 Controller ID: 0 Number of Namespaces: 1 Namespace 1 Size/Capacity: 512,110,190,592 [512 GB] Namespace 1 Formatted LBA Size: 512 Local Time is: Mon May 13 10:34:11 2019 CEST Firmware Updates (0x14): 2 Slots, no Reset required Optional Admin Commands (0x0017): Security Format Frmw_DL *Other* Optional NVM Commands (0x005f): Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat *Other* Maximum Data Transfer Size: 512 Pages Warning Comp. Temp. Threshold: 78 Celsius Critical Comp. Temp. Threshold: 82 Celsius Namespace 1 Features (0x02): NA_Fields Supported Power States St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat 0 + 6.00W - - 0 0 0 0 0 0 1 + 2.40W - - 1 1 1 1 0 0 2 + 1.90W - - 2 2 2 2 0 0 3 - 0.0500W - - 3 3 3 3 1500 1500 4 - 0.0050W - - 4 4 4 4 6000 14000 5 - 0.0030W - - 5 5 5 5 50000 80000 Supported LBA Sizes (NSID 0x1) Id Fmt Data Metadt Rel_Perf 0 + 512 0 2 1 - 4096 0 1 === START OF SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED SMART/Health Information (NVMe Log 0x02, NSID 0xffffffff) Critical Warning: 0x00 Temperature: 47 Celsius Available Spare: 100% Available Spare Threshold: 10% Percentage Used: 57% Data Units Read: 31,858,921 [16.3 TB] Data Units Written: 293,589,002 [150 TB] Host Read Commands: 4,130,502,428 Host Write Commands: 889,121,505 Controller Busy Time: 13,552 Power Cycles: 7 Power On Hours: 6,720 Unsafe Shutdowns: 0 Media and Data Integrity Errors: 0 Error Information Log Entries: 0 Warning Comp. Temperature Time: 0 Critical Comp. Temperature Time: 0 Temperature Sensor 1: 47 Celsius Error Information (NVMe Log 0x01, max 128 entries) No Errors Logged nvme smart-log /dev/nvme1 Smart Log for NVME device:nvme1 namespace-id:ffffffff critical_warning : 0 temperature : 47 C available_spare : 100% available_spare_threshold : 10% percentage_used : 57% data_units_read : 31,858,921 data_units_written : 293,589,023 host_read_commands : 4,130,502,429 host_write_commands : 889,122,059 controller_busy_time : 13,552 power_cycles : 7 power_on_hours : 6,720 unsafe_shutdowns : 0 media_errors : 0 num_err_log_entries : 0 Warning Temperature Time : 0 Critical Composite Temperature Time : 0 Temperature Sensor 1 : 47 C Temperature Sensor 2 : 0 C Temperature Sensor 3 : 0 C Temperature Sensor 4 : 0 C Temperature Sensor 5 : 0 C Temperature Sensor 6 : 0 C Temperature Sensor 7 : 0 C Temperature Sensor 8 : 0 C nvme smart-log-add /dev/nvme1 NVMe Status:INVALID_LOG_PAGE(4109) smartctl -A /dev/nvme1 smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.9.0-8-amd64] (local build) Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org === START OF SMART DATA SECTION === SMART/Health Information (NVMe Log 0x02, NSID 0xffffffff) Critical Warning: 0x00 Temperature: 46 Celsius Available Spare: 100% Available Spare Threshold: 10% Percentage Used: 57% Data Units Read: 31,858,924 [16.3 TB] Data Units Written: 293,591,327 [150 TB] Host Read Commands: 4,130,502,490 Host Write Commands: 889,172,096 Controller Busy Time: 13,552 Power Cycles: 7 Power On Hours: 6,721 Unsafe Shutdowns: 0 Media and Data Integrity Errors: 0 Error Information Log Entries: 0 Warning Comp. Temperature Time: 0 Critical Comp. Temperature Time: 0 Temperature Sensor 1: 46 Celsius I only noticed the issue after apache failed to start and I repaired the filesystem with fsck.ext4 -f. Before I didn't have setup root mail correctly. So looks to me like a hardware error and I should get rid of both nvmes. Is there anything I can try to fix these issues and save the nvmes? Or at least to get all the smart values like "Reported_Uncorrect" or "Offline_Uncorrectable".

I'm getting weird behaviour while setting up an mdadm RAID1 array on debian 8.2. After I set-up the array, lsblk shows: simon@debian-server:~$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 931.5G 0 disk `-sda1 8:1 0 931.5G 0 part `-md0 9:0 0 931.4G 0 raid1 sdb 8:16 0 931.5G 0 disk `-sdb1 8:17 0 931.5G 0 part `-md0 9:0 0 931.4G 0 raid1 sdc 8:32 0 232.9G 0 disk |-sdc1 8:33 0 512M 0 part /boot/efi |-sdc2 8:34 0 244M 0 part /boot `-sdc3 8:35 0 232.2G 0 part |-debian--server--vg-root 254:0 0 228.3G 0 lvm / `-debian--server--vg-swap_1 254:1 0 3.9G 0 lvm [SWAP] After a reboot, lsblk shows: simon@debian-server:~$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 931.5G 0 disk `-sda1 8:1 0 931.5G 0 part `-md0 9:0 0 931.4G 0 raid1 |-md0p1 259:0 0 811.6G 0 md `-md0p2 259:1 0 346.1G 0 md sdb 8:16 0 931.5G 0 disk `-sdb1 8:17 0 931.5G 0 part `-md0 9:0 0 931.4G 0 raid1 |-md0p1 259:0 0 811.6G 0 md `-md0p2 259:1 0 346.1G 0 md sdc 8:32 0 232.9G 0 disk |-sdc1 8:33 0 512M 0 part /boot/efi |-sdc2 8:34 0 244M 0 part /boot `-sdc3 8:35 0 232.2G 0 part |-debian--server--vg-root 254:0 0 228.3G 0 lvm / `-debian--server--vg-swap_1 254:1 0 3.9G 0 lvm [SWAP] I don't know where the md0p1 and md0p2 partitions are coming from. My /etc/fstab and /etc/mdadm/mdadm.conf both have nothing about this in them. parted shows one partition on md0: simon@debian-server:~$ sudo parted /dev/md0 print Model: Linux Software RAID Array (md) Disk /dev/md0: 1000GB Sector size (logical/physical): 512B/4096B Partition Table: loop Disk Flags: Number Start End Size File system Flags 1 0.00B 1000GB 1000GB ntfs Any ideas where the md0p1 and md0p2 partitions are coming from? I'm setting up the array by doing as follows: - Delete existing device (I've done this a few times): sudo mdadm --stop /dev/md0 sudo mdadm --remove /dev/md0 - Zero drives: sudo dd if=/dev/zero of=/dev/sda bs=1M count=1024 sudo dd if=/dev/zero of=/dev/sdb bs=1M count=1024 - Create partition tables: sudo parted /dev/sda mklabel gpt sudo parted /dev/sdb mklabel gpt - Create full-disk partitions: sudo parted -a optimal /dev/sda mkpart primary '0%' '100%' sudo parted -a optimal /dev/sdb mkpart primary '0%' '100%' - Set raid flag on partitions: sudo parted /dev/sda set 1 raid on sudo parted /dev/sdb set 1 raid on - Create RAID array: sudo mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sd[ab]1 - Add filesystem (I'm using NTFS, but the problem also happens with ext4) sudo mkfs.ntfs -f /dev/md0

I've been running a RAID1 setup using two Samsung 970 EVO 500GB NVMEs on an Asus Z390 Maximus XI Formula motherboard for the last two years. This is my home server running Ubuntu 22.04 and I chose RAID1 for this specific reason in case one of the drives would fail. I wanted to upgrade the machine from 32GB to 128GB RAM for more ZFS Cache, but as the system didn't POST by itself I opted for a CMOS reset using the button on the motherboards I/O. After the reset I was anticipating having to change SATA Mode from AHCI to RAID (Intel Rapid Storage Technology) in the BIOS before being able to boot from my storage again as normal. Unfortunately this was not the case. I tried CSM (Compatibility Support Module) Settings recommended online, enable it and set Boot Device Control to UEFI and Legacy OPROM and UEFI priority for booting from storage devices. No change. I also tried manually rebuilding the RAID Volume without initializing/formatting the disks by creating a new RAID1 volume, but no change. So I moved on to data recovery by moving one of the disks to a separate PC I have with a minimal Ubuntu install.

and

fdisk -l

both show the disk being recognized, while the type reported by fdisk is cleary Linux RAID:

/dev/nvme0n1: 465.76 GiB, 500107862016 bytes, 976773168 sectors
Disk model: Samsung SSD 970 EVO 500GB               
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 23ABA6D9-5923-4891-B67E-2208F422E40C

Device         Start       End   Sectors   Size Type
/dev/nvme0n1p1  4096 976506887 976502792 465.6G Linux RAID

## Steps taken in Ubuntu: ## 1. Copied the entire disk content to my local device:

dd if=/dev/nvme0n1 of=~/nvme_backup.img bs=4M status=progress

2. Installed and ran testdisk

testdisk /dev/nvme0n1

3. Chose

[EFI GPT]

partition table type, analyzed and found

Raid        4096 976506887 976502792 [ubuntu-server:0]

*The name of my raid volume!* So I accepted and WROTE. 4. After reboot, I attempted to mount the volume

: /mnt/recovery: unknown filesystem type 'linux_raid_member'

and realized I need to assemble the degraded RAID using

. 5. I followed this blogpost to examine the metadata, finding the following:

sudo mdadm --examine /dev/nvme0n1p1 
/dev/nvme0n1p1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : f4c5d883:1a67c158:7d34d079:c5e09397
           Name : ubuntu-server:0
  Creation Time : Mon Jul 10 16:26:43 2023
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 976502784 sectors (465.63 GiB 499.97 GB)
     Array Size : 488251392 KiB (465.63 GiB 499.97 GB)
    Data Offset : 264192 sectors
   Super Offset : 8 sectors
   Unused Space : before=264112 sectors, after=-264183 sectors DEVICE TOO SMALL
          State : clean TRUNCATED DEVICE
    Device UUID : de5af620:88784a78:693bf52a:bab766a4

Internal Bitmap : 8 sectors from superblock
    Update Time : Sun Apr 27 23:57:35 2025
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : 88404970 - correct
         Events : 9909


   Device Role : Active device 1
   Array State : AA ('A' == active, '.' == missing, 'R' == replacing)

6. Attempted to start the raid again using mdadm (my knowledge is very limited here so I tried both):

~$ sudo mdadm --assemble --run /dev/md0 /dev/nvme0n1p1 --force
mdadm: Device /dev/nvme0n1p1 is not large enough for data described in superblock
mdadm: no RAID superblock on /dev/nvme0n1p1
mdadm: /dev/nvme0n1p1 has no superblock - assembly aborted
~$ sudo mdadm --assemble --run /dev/md0 /dev/nvme0n1 --force
mdadm: Cannot assemble mbr metadata on /dev/nvme0n1
mdadm: /dev/nvme0n1 has no superblock - assembly aborted

My hopes was that this last step would allow me to mount the disk directly and find my now inaccessible data (OS files and configs, all important data is still on the ZFS pool luckily). I initially thought it would be as simple as moving a single disk to another system, enabling IRST and booting from the drive as the disks are "mirrored". Worth noting is that data is clearly still readable if I use:

photorec /dev/nvme0n1

But I would of course prefer if there were some way for me to get access to the proper file system again. Any help or guidance is greatly appreciated.

My current idea is to create one software array, class RAID-6, with 4 member drives, using mdadm. Specifically, the drives would be 1 TB HDDs on SATA in a small server Dell T20. Operating System is GNU/Linux Debian 8.6 (later upgraded: Jessie ⟶ Stretch ⟶ Buster ) That would make 2 TB of disk space with 2 TB of parity in my case. *** I would also like to have it with GPT partition table, for that to work, I am unsure how to proceed specifically supposing I would prefer to do this purely over the terminal. As I never created a RAID array, could you guide me on how I should proceed? *** Notes: - This array will serve for the sole data only. No boot or OS on it. - I opted for RAID-6 due to the purpose of this array. Two drive failures the array must be able to survive. Since I am limited by hardware to 4 drives, there is no alternative to RAID-6 that I know of. (However ugly the RAID-6 slowdown may seem, it does not matter in this array.)

I have a problem with a RAID 1 on my Raspberry Pi 2. as system is use jessie-lite. If I reboot or boot the system, RAID won't start. root@raspberrypi:~# cat /proc/mdstat Personalities : unused devices: How can I get RAID to start at boot? After mdadm --assemble --scan, the RAID will start. root@raspberrypi:~# mdadm --assemble --scan mdadm: /dev/md/0 has been started with 2 drives. mdadm config mdadm.conf # # Please refer to mdadm.conf(5) for information about this file. # # by default (built-in), scan all partitions (/proc/partitions) and all # containers for MD superblocks. alternatively, specify devices to scan, using # wildcards if desired. #DEVICE partitions containers # auto-create devices with Debian standard permissions CREATE owner=root group=disk mode=0660 auto=yes # automatically tag new arrays as belonging to the local system HOMEHOST # instruct the monitoring daemon where to send mail alerts MAILADDR ****@******.*** # definitions of existing MD arrays ARRAY /dev/md/0 metadata=1.2 UUID=287dd2d2:88e06dc7:25dd3aaa:196d016f name=raspberrypi:0 # This configuration was auto-generated on Wed, 30 Nov 2016 12:08:42 +0000 by mkconf the fstab file proc /proc proc defaults 0 0 /dev/mmcblk0p1 /boot vfat defaults 0 2 /dev/mmcblk0p2 / ext4 defaults,noatime 0 1 # a swapfile is not a swap partition, no line here # use dphys-swapfile swap[on|off] for that /dev/md0 /mnt/md0 ext4 defaults,nofail 0 The mounted devices: /dev/mmcblk0: PTUUID="752385f3" PTTYPE="dos" /dev/mmcblk0p1: SEC_TYPE="msdos" LABEL="boot" UUID="70F7-FA1D" TYPE="vfat" PARTUUID="752385f3-01" /dev/mmcblk0p2: UUID="3598ef8e-09be-47ef-9d01-f24cf61dff1d" TYPE="ext4" PARTUUID="752385f3-02" /dev/sda: UUID="287dd2d2-88e0-6dc7-25dd-3aaa196d016f" UUID_SUB="6bfd7be2-7a7b-d01e-7813-66cc1d91966e" LABEL="raspberrypi:0" TYPE="linux_raid_member" /dev/md0: UUID="5831b12b-8bdc-49ef-bdd4-4578d703491b" TYPE="ext4" /dev/sdb: UUID="287dd2d2-88e0-6dc7-25dd-3aaa196d016f" UUID_SUB="60adaf57-72f9-ec6e-8653-331682fdd90d" LABEL="raspberrypi:0" TYPE="linux_raid_member" If I know reboot the same. RAID is not active root@raspberrypi:~# cat /proc/mdstat Personalities : unused devices:

I want to remove raid from my system as I am low on storage and I want to recover the second disk. How can I recover the second disk, I tried but to no avail, here is my current state : root@miirabox ~ # cat /proc/mdstat Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] md2 : active raid1 nvme1n1p3 nvme0n1p3 965467456 blocks super 1.2 [2/2] [UU] bitmap: 8/8 pages [32KB], 65536KB chunk md0 : active raid1 nvme1n1p1 nvme0n1p1 33520640 blocks super 1.2 [2/2] [UU] md1 : active raid1 nvme0n1p2(F) nvme1n1p2 1046528 blocks super 1.2 [2/1] [_U] unused devices: root@miirabox ~ # sudo mdadm --detail --scan ARRAY /dev/md/1 metadata=1.2 name=rescue:1 UUID=36e3a554:de955adc:98504c1a:836763fb ARRAY /dev/md/0 metadata=1.2 name=rescue:0 UUID=b7eddc10:a40cc141:c349f876:39fa07d2 ARRAY /dev/md/2 metadata=1.2 name=rescue:2 UUID=2eafee34:c51da1e0:860a4552:580258eb root@miirabox ~ # mdadm -E /dev/nvme0n1p1 /dev/nvme0n1p1: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : b7eddc10:a40cc141:c349f876:39fa07d2 Name : rescue:0 Creation Time : Sun Sep 10 16:52:20 2023 Raid Level : raid1 Raid Devices : 2 Avail Dev Size : 67041280 sectors (31.97 GiB 34.33 GB) Array Size : 33520640 KiB (31.97 GiB 34.33 GB) Data Offset : 67584 sectors Super Offset : 8 sectors Unused Space : before=67432 sectors, after=0 sectors State : clean Device UUID : 5f8a86c6:80e71724:98ee2d01:8a295f5a Update Time : Thu Sep 19 19:31:55 2024 Bad Block Log : 512 entries available at offset 136 sectors Checksum : f2954bfe - correct Events : 60 Device Role : Active device 0 Array State : AA ('A' == active, '.' == missing, 'R' == replacing) root@miirabox ~ # mdadm -E /dev/nvme0n1p2 /dev/nvme0n1p2: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : 36e3a554:de955adc:98504c1a:836763fb Name : rescue:1 Creation Time : Sun Sep 10 16:52:20 2023 Raid Level : raid1 Raid Devices : 2 Avail Dev Size : 2093056 sectors (1022.00 MiB 1071.64 MB) Array Size : 1046528 KiB (1022.00 MiB 1071.64 MB) Data Offset : 4096 sectors Super Offset : 8 sectors Unused Space : before=4016 sectors, after=0 sectors State : clean Device UUID : 8d8e044d:543e1869:9cd0c1ee:2b644e57 Update Time : Thu Sep 19 19:07:25 2024 Bad Block Log : 512 entries available at offset 16 sectors Checksum : 4ce9a898 - correct Events : 139 Device Role : Active device 0 Array State : AA ('A' == active, '.' == missing, 'R' == replacing) root@miirabox ~ # mdadm -E /dev/nvme0n1p3 /dev/nvme0n1p3: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : 2eafee34:c51da1e0:860a4552:580258eb Name : rescue:2 Creation Time : Sun Sep 10 16:52:20 2023 Raid Level : raid1 Raid Devices : 2 Avail Dev Size : 1930934960 sectors (920.74 GiB 988.64 GB) Array Size : 965467456 KiB (920.74 GiB 988.64 GB) Used Dev Size : 1930934912 sectors (920.74 GiB 988.64 GB) Data Offset : 264192 sectors Super Offset : 8 sectors Unused Space : before=264112 sectors, after=48 sectors State : clean Device UUID : 68758969:5218958f:9c991c6b:12bfdca1 Internal Bitmap : 8 sectors from superblock Update Time : Thu Sep 19 19:32:42 2024 Bad Block Log : 512 entries available at offset 16 sectors Checksum : 4a44ff36 - correct Events : 13984 Device Role : Active device 0 Array State : AA ('A' == active, '.' == missing, 'R' == replacing) root@miirabox ~ # mdadm -E /dev/nvme1n1p1 /dev/nvme1n1p1: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : b7eddc10:a40cc141:c349f876:39fa07d2 Name : rescue:0 Creation Time : Sun Sep 10 16:52:20 2023 Raid Level : raid1 Raid Devices : 2 Avail Dev Size : 67041280 sectors (31.97 GiB 34.33 GB) Array Size : 33520640 KiB (31.97 GiB 34.33 GB) Data Offset : 67584 sectors Super Offset : 8 sectors Unused Space : before=67432 sectors, after=0 sectors State : clean Device UUID : 0dfdf4af:d88b2bf1:0764dcbd:1179639e Update Time : Thu Sep 19 19:33:07 2024 Bad Block Log : 512 entries available at offset 136 sectors Checksum : a9ca2845 - correct Events : 60 Device Role : Active device 1 Array State : AA ('A' == active, '.' == missing, 'R' == replacing) root@miirabox ~ # mdadm -E /dev/nvme1n1p2 /dev/nvme1n1p2: Magic : a92b4efc Version : 1.2 Feature Map : 0x0 Array UUID : 36e3a554:de955adc:98504c1a:836763fb Name : rescue:1 Creation Time : Sun Sep 10 16:52:20 2023 Raid Level : raid1 Raid Devices : 2 Avail Dev Size : 2093056 sectors (1022.00 MiB 1071.64 MB) Array Size : 1046528 KiB (1022.00 MiB 1071.64 MB) Data Offset : 4096 sectors Super Offset : 8 sectors Unused Space : before=4016 sectors, after=0 sectors State : clean Device UUID : 228202fa:0491e478:b0a0213b:0484d5e3 Update Time : Thu Sep 19 19:24:14 2024 Bad Block Log : 512 entries available at offset 16 sectors Checksum : e29be2bc - correct Events : 141 Device Role : Active device 1 Array State : .A ('A' == active, '.' == missing, 'R' == replacing) root@miirabox ~ # mdadm -E /dev/nvme1n1p3 /dev/nvme1n1p3: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : 2eafee34:c51da1e0:860a4552:580258eb Name : rescue:2 Creation Time : Sun Sep 10 16:52:20 2023 Raid Level : raid1 Raid Devices : 2 Avail Dev Size : 1930934960 sectors (920.74 GiB 988.64 GB) Array Size : 965467456 KiB (920.74 GiB 988.64 GB) Used Dev Size : 1930934912 sectors (920.74 GiB 988.64 GB) Data Offset : 264192 sectors Super Offset : 8 sectors Unused Space : before=264112 sectors, after=48 sectors State : clean Device UUID : 431be888:cb298461:ba2a0000:4b5294fb Internal Bitmap : 8 sectors from superblock Update Time : Thu Sep 19 19:33:21 2024 Bad Block Log : 512 entries available at offset 16 sectors Checksum : 2a2ddb09 - correct Events : 13984 Device Role : Active device 1 Array State : AA ('A' == active, '.' == missing, 'R' == replacing) root@miirabox ~ # mdadm -D /dev/md0 /dev/md0: Version : 1.2 Creation Time : Sun Sep 10 16:52:20 2023 Raid Level : raid1 Array Size : 33520640 (31.97 GiB 34.33 GB) Used Dev Size : 33520640 (31.97 GiB 34.33 GB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent Update Time : Thu Sep 19 19:34:08 2024 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 Consistency Policy : resync Name : rescue:0 UUID : b7eddc10:a40cc141:c349f876:39fa07d2 Events : 60 Number Major Minor RaidDevice State 0 259 1 0 active sync /dev/nvme0n1p1 1 259 5 1 active sync /dev/nvme1n1p1 root@miirabox ~ # mdadm -D /dev/md1 /dev/md1: Version : 1.2 Creation Time : Sun Sep 10 16:52:20 2023 Raid Level : raid1 Array Size : 1046528 (1022.00 MiB 1071.64 MB) Used Dev Size : 1046528 (1022.00 MiB 1071.64 MB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent Update Time : Thu Sep 19 19:24:14 2024 State : clean, degraded Active Devices : 1 Working Devices : 1 Failed Devices : 1 Spare Devices : 0 Consistency Policy : resync Name : rescue:1 UUID : 36e3a554:de955adc:98504c1a:836763fb Events : 141 Number Major Minor RaidDevice State - 0 0 0 removed 1 259 6 1 active sync /dev/nvme1n1p2 0 259 2 - faulty /dev/nvme0n1p2 root@miirabox ~ # mdadm -D /dev/md2 /dev/md2: Version : 1.2 Creation Time : Sun Sep 10 16:52:20 2023 Raid Level : raid1 Array Size : 965467456 (920.74 GiB 988.64 GB) Used Dev Size : 965467456 (920.74 GiB 988.64 GB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent Intent Bitmap : Internal Update Time : Thu Sep 19 19:34:46 2024 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 Consistency Policy : bitmap Name : rescue:2 UUID : 2eafee34:c51da1e0:860a4552:580258eb Events : 13984 Number Major Minor RaidDevice State 0 259 3 0 active sync /dev/nvme0n1p3 1 259 7 1 active sync /dev/nvme1n1p3 root@miirabox ~ # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS loop0 7:0 0 4K 1 loop /snap/bare/5 loop2 7:2 0 74.3M 1 loop /snap/core22/1586 loop3 7:3 0 40.4M 1 loop loop4 7:4 0 269.8M 1 loop /snap/firefox/4793 loop5 7:5 0 74.3M 1 loop /snap/core22/1612 loop6 7:6 0 91.7M 1 loop /snap/gtk-common-themes/1535 loop8 7:8 0 38.8M 1 loop /snap/snapd/21759 loop9 7:9 0 271.2M 1 loop /snap/firefox/4848 loop10 7:10 0 504.2M 1 loop /snap/gnome-42-2204/172 loop12 7:12 0 505.1M 1 loop /snap/gnome-42-2204/176 loop13 7:13 0 38.7M 1 loop /snap/snapd/21465 nvme0n1 259:0 0 953.9G 0 disk ├─nvme0n1p1 259:1 0 32G 0 part │ └─md0 9:0 0 32G 0 raid1 [SWAP] ├─nvme0n1p2 259:2 0 1G 0 part │ └─md1 9:1 0 1022M 0 raid1 └─nvme0n1p3 259:3 0 920.9G 0 part └─md2 9:2 0 920.7G 0 raid1 / nvme1n1 259:4 0 953.9G 0 disk ├─nvme1n1p1 259:5 0 32G 0 part │ └─md0 9:0 0 32G 0 raid1 [SWAP] ├─nvme1n1p2 259:6 0 1G 0 part │ └─md1 9:1 0 1022M 0 raid1 └─nvme1n1p3 259:7 0 920.9G 0 part └─md2 9:2 0 920.7G 0 raid1 / root@miirabox ~ # cat /etc/fstab proc /proc proc defaults 0 0 # /dev/md/0 UUID=e9dddf2b-f061-403e-a12f-d98915569492 none swap sw 0 0 # /dev/md/1 UUID=d32210de-6eb0-4459-85a7-6665294131ee /boot ext3 defaults 0 0 # /dev/md/2 UUID=7abe3389-fe7d-4024-a57e-e490f5e04880 / ext4 defaults 0 0 This is what I managed to do : root@miirabox ~ # df -h df: /run/user/1000/gvfs: Transport endpoint is not connected Filesystem Size Used Avail Use% Mounted on tmpfs 6.3G 5.7M 6.3G 1% /run /dev/md2 906G 860G 0 100% / tmpfs 32G 0 32G 0% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock /dev/md1 989M 271M 667M 29% /boot tmpfs 6.3G 132K 6.3G 1% /run/user/134 tmpfs 32G 648K 32G 1% /run/qemu tmpfs 6.3G 244K 6.3G 1% /run/user/1000 tmpfs 6.3G 116K 6.3G 1% /run/user/140 root@miirabox ~ # cat cat /proc/mdstat cat: cat: No such file or directory Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] md2 : active raid1 nvme1n1p3 nvme0n1p3 965467456 blocks super 1.2 [2/2] [UU] bitmap: 8/8 pages [32KB], 65536KB chunk md0 : active raid1 nvme1n1p1 nvme0n1p1 33520640 blocks super 1.2 [2/2] [UU] md1 : active raid1 nvme0n1p2 nvme1n1p2 1046528 blocks super 1.2 [2/2] [UU] root@miirabox ~ # umount /dev/md1 root@miirabox ~ # umount /dev/md2 root@miirabox ~ # umount /dev/md0 umount: /dev/md0: not mounted. root@miirabox ~ # mdadm --fail /dev/md1 /dev/nvme0n1p2 mdadm: set /dev/nvme0n1p2 faulty in /dev/md1 root@miirabox ~ # mdadm --remove /dev/md1 root@miirabox ~ # mdadm --fail /dev/md1 /dev/nvme1n1p2 mdadm: set device faulty failed for /dev/nvme1n1p2: Device or resource busy root@miirabox ~ # sudo mdadm --stop /dev/md1 mdadm: Cannot get exclusive access to /dev/md1:Perhaps a running process, mounted filesystem or active volume group? root@miirabox ~ # sudo vgdisplay root@miirabox ~ # lvdisplay I was following a guide and could not proceed. Please do not hesitate if you want more details. Thanks in advance. EDIT : I apologise I was not very clear, it's that I have 2TB and my system is only using 1TB (os+data) and the rest is used by raid. I just want to remove raid and recover the second 1TB so I can get the 2TB.

I am attempting to recover data on a MDADM RAID1 array with one HDD that suffered a mechanical failure and another that is stuck in a rebuilding state. I have two 2TB HDDs that were installed on a machine running under a mdadm RAID1 array named /dev/md1. Sda1 and Sdb1 were both part of the array. Sda had a mechanical failure and it's been replaced with a new HDD with the same capacity, on which a new partition of the same size was created. Upon attempting to add sda1 to the array I've received the following error:

sudo mdadm /dev/md1 --manage --add /dev/sda1
mdadm: cannot load array metadata from /dev/md1

What I've tried --------------- I've been following this guide (https://ahelpme.com/linux/recovering-md-array-and-mdadm-cannot-get-array-info-for-dev-md0/) to attempt to 'activate' the array. These are the steps of the guide: 1. Remove ALL current configuration by issuing multiple stop commands with mdadm, no inactive raids or any raids should be reported in “/proc/mdstat”. 2. Rename mdadm configuration files in /etc/mdadm/mdadm.conf 3. Rescan for MD devices with mdadm. The mdadm will load the configuration from your disks. 4. Add the missing partitions to your software raid devices. I've executed the first two steps. On running step 3, the raid was discovered but, unlike the guide, it seems sdb1 is in rebuilding state and the array cannot be started because of it. I'm reluctant to try anything involving --force since I'm unsure as to the exact state of the data and the fact that, unfortunately, the entirety of the data is very precious. My questions ------------ * How can I recover the data? * How can I know what files on the disk that is 'Rebuilding' are 'incomplete' or corrupted? * Why, when running

mdadm --misc --detail /dev/md1

it says raid0 but when running

mdadm -E /dev/sdb1

it says raid1? Any and all help is highly appreciated. System info -----------

:/$ sudo mdadm --assemble --scan --verbose

mdadm: looking for devices for further assembly
mdadm: no recogniseable superblock on /dev/loop10
mdadm: no recogniseable superblock on /dev/loop9
mdadm: no recogniseable superblock on /dev/loop8
mdadm: no recogniseable superblock on /dev/sdc2
mdadm: Cannot assemble mbr metadata on /dev/sdc1
mdadm: Cannot assemble mbr metadata on /dev/sdc
mdadm: no recogniseable superblock on /dev/sda1
mdadm: Cannot assemble mbr metadata on /dev/sda
mdadm: No super block found on /dev/sdb (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdb
mdadm: No super block found on /dev/loop7 (Expected magic a92b4efc, got 118a6b61)
mdadm: no RAID superblock on /dev/loop7
mdadm: No super block found on /dev/loop6 (Expected magic a92b4efc, got e7e108a6)
mdadm: no RAID superblock on /dev/loop6
mdadm: No super block found on /dev/loop5 (Expected magic a92b4efc, got 3a23b8f9)
mdadm: no RAID superblock on /dev/loop5
mdadm: No super block found on /dev/loop4 (Expected magic a92b4efc, got 3a23b8f9)
mdadm: no RAID superblock on /dev/loop4
mdadm: No super block found on /dev/loop3 (Expected magic a92b4efc, got e7e108a6)
mdadm: no RAID superblock on /dev/loop3
mdadm: No super block found on /dev/loop2 (Expected magic a92b4efc, got a6eff301)
mdadm: no RAID superblock on /dev/loop2
mdadm: No super block found on /dev/loop1 (Expected magic a92b4efc, got e06997af)
mdadm: no RAID superblock on /dev/loop1
mdadm: /dev/sdb1 is identified as a member of /dev/md/1, slot 1.
mdadm: no uptodate device for slot 0 of /dev/md/1
mdadm: added /dev/sdb1 to /dev/md/1 as 1

This is more informational than a question though someone may have a better answer than the solution I came up with. This was a fresh install of Ubuntu server, literally nothing done other than install ssh keys and lock out password authentication. It boots to a RAID1 and I created another RAID1 to mount to /var/backup. In the process of creating that array my dog started bothering me to go out and I totally lost track of where I was, and I ran this command twice. Smart eh?

--detail --scan | sudo tee -a /etc/mdadm/mdadm.conf

Obviously that inserted the UUID of both drives into mdadm.conf twice which caused a conflict. After I ran an update-initramfs I mounted the drive, edited fstab to mount at boot, and booted the server. When I came back I was surprised to see the initramfs prompt instead of a normal login. Using the mdadm commands to assemble the boot array worked, but when trying to mount it all it said was:

mount: mounting /dev/md0 on /mnt failed invalid argument

The mnt folder is there, so this should mount and it was driving me nuts. I could have easily just reloaded this system, and that proably would have been faster but I really wanted to figure this out. Either I am blind or the Ubuntu server installer doesn't have a normal Rescue Mode, and I wanted to get into something where I could run a shell in /dev/md0. I KNEW for a fact that I had probably screwed up mdadm.conf and therefore also initramfs so I figured this should be an easy fix if I could just get it to mount. So I downloaded Ubuntu 16.04 and booted to Rescue Mode and when it came time to select a file system to mount in, it showed me 3. md125, md126, and md127. There should have only been 2, but whatever. I knew it was screwed up anyways. None of them would mount. And manually assembling them failed as well. So I ran the shell without a root file system and checked mdadm --detail and dev/md127 was the file system I wanted, so again I tried to mount it and got:

mount /dev/md127 /mnt
initramfs mount: mounting /dev/md127 on /mnt failed invalid argument

Checking fdkisk got me this

fdisk -l /dev/sda
Device     Start        End    Sectors   Size Type
/dev/sda1   2048       4095       2048     1M BIOS boot
/dev/sda2   4096 1172119551 1172115456 558.9G Linux filesystem

root:/# fdisk -l /dev/md127
Disk /dev/md0: 558.78 GiB, 599987847168 bytes, 1171851264 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 9507C8B6-1F1F-48E1-B2C2-53EDC1570690

Device     Start        End    Sectors   Size Type
/dev/md127p1  2048 1171849215 1171847168 558.8G Linux filesystem

The p1 tagged onto md127 threw me. I assume that's partition 1 of /dev/md127. But mounting it up worked just fine.

mount /dev/md127p1 /mnt

I went in and edited mdadm.conf to remove the extra lines. Actually just commented them out for now.

ARRAY /dev/md0 metadata=1.2 UUID=42ec458b:8083cf4d:ae8b79c7:a364f02e
ARRAY /dev/md1 metadata=1.2 UUID=98581732:4d93dad7:ae175d16:e292b2c9
MAILADDR root
#ARRAY /dev/md0 metadata=1.2 UUID=42ec458b:8083cf4d:ae8b79c7:a364f02e
#ARRAY /dev/md1 metadata=1.2 UUID=98581732:4d93dad7:ae175d16:e292b2c9

And when I booted I went right back to initramfs, because when I generated initramfs earlier those lines were still in mdadm.conf. So I booted back into rescue mode and ran

mount -t proc proc /tmp/proc
update-initramfs -u

Booted the system and it came back up just fine. It did throw an error after it generated the kernel image. I don't remember the exact line but something along the lines of /proc/cpu not existing and a couple of others. I updated initramfs again once I was booted normally and it updated no problem. Anyone else have a similar experience. Anything I should have done differently? This took me a while to figure out. As I said I could have easily reloaded this, but this could have been on a live machine which is why I wanted to solve it without a reload.

I'm starting to get a collection of computers at home and to support them I have my "server" linux box running a RAID array. Its currently mdadm RAID-1, going to RAID-5 once I have more drives (and then RAID-6 I'm hoping for). However I've heard various stories about data getting corrupted on one drive and you never noticing due to the other drive being used, up until the point when the first drive fails, and you find your second drive is also screwed (and 3rd, 4th, 5th drive). Obviously backups are important and I'm taking care of that also, however I know I've previously seen scripts which claim to help against this problem and allow you to check your RAID while its running. However looking for these scripts again now I'm finding it hard to find anything which seems similar to what I ran before and I feel I'm out of date and not understanding whatever has changed. How would you check a running RAID to make sure all disks are still preforming normally? I monitor SMART on all the drives and also have mdadm set to email me in case of failure but I'd like to know my drives occasionally "check" themselves too.

I rather randomly checked the status of my RAID arrays with cat/proc/mdstat and realized, that one of my arrays seems to be resyncing: md1 : active raid1 sdb7 sdc7 238340224 blocks [2/2] [UU] [==========>..........] resync = 52.2% (124602368/238340224) finish=75.0min speed=25258K/sec Why is this and what does it mean? I seemingly can access the mount point just fine with r/w permissions. **EDIT 1** (in response to **SLM's ANSWER**) I can't really see anything if I grep through dmesg and the --detail switch doesn't tell me much either, i.e. it displays that the resync is in progress... but no hint for the reason or why it might have gotten out of sync... - I guess I might just need to keep an eye on it before I start swapping out my hardware.

My RAID1 array is currently running with 3 disks (2x6TB and 1x3TB). I want to remove the smaller disk and wonder if it makes sense to put it in the safe as cold backup. The idea is to be able to mount it as degraded RAID1 array, independent of the running RAID. Question 1: Should I set the disk to faulty before removing it? Question 2: Will I be able to mount the backup disk on the same device where the RAID is running or will mdadm detect the disk to belong to the array already there? Linux kernel is 6.8.0, mdadm is v4.3.

I had raid5 8TB * 4 then ,now I added one more 8TB drive and try to make raid5 grow $mdadm --grow /dev/md128 --raid-disks=5 It comes to 6.1% but it stuck for a few days, and number 482870548 doesn't move any more. less /proc/mdstat Personalities : [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] md128 : active raid5 sdk sdh3 sdi sdj sdg3 23427528384 blocks super 1.2 level 5, 64k chunk, algorithm 2 [5/5] [UUUUU] [=>...................] reshape = 6.1% (482870548/7809176128) finish=63150741.3min speed=1K/sec However it still consumes 100% CPU $top 10070 root 20 0 0 0 0 R 100.0 0.0 3639:00 md128_raid5 7 root 20 0 0 0 0 S 0.3 0.0 8:27.71 rcu_sched 1400 root 20 0 0 0 0 S 0.3 0.0 21:12.08 kswapd0 2507 root 20 0 0 0 0 S 0.3 0.0 12:18.42 usb-storage 2561 root 0 -20 0 0 0 S 0.3 0.0 2:30.18 kworker/1:1H 28016 root 20 0 28904 3428 2780 R 0.3 0.1 0:00.02 top 1 root 20 0 138992 5400 3372 S 0.0 0.1 0:08.39 systemd 2 root 20 0 0 0 0 S 0.0 0.0 0:00.04 kthreadd 3 root 20 0 0 0 0 S 0.0 0.0 14:01.07 ksoftirqd/0 I checked the situation with this $mdadm -D /dev/md128 /dev/md128: Version : 1.2 Creation Time : Tue Sep 22 08:59:59 2020 Raid Level : raid5 Array Size : 23427528384 (22342.23 GiB 23989.79 GB) Used Dev Size : 7809176128 (7447.41 GiB 7996.60 GB) Raid Devices : 5 Total Devices : 5 Persistence : Superblock is persistent Update Time : Wed Jul 24 09:24:26 2024 State : clean, reshaping Active Devices : 5 Working Devices : 5 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 64K Consistency Policy : unknown Reshape Status : 6% complete Delta Devices : 1, (4->5) Name : 0a438048:RaidSecond-0 (local to host 0a438048) UUID : a9bcec2a:5772e6a5:045f28ac:62ae0002 Events : 53642 Number Major Minor RaidDevice State 0 8 115 0 active sync /dev/sdh3 1 8 99 1 active sync /dev/sdg3 5 8 144 2 active sync /dev/sdj 4 8 128 3 active sync /dev/sdi 6 8 160 4 active sync /dev/sdk Also , I can't find the reason why it stuck and I am not what is the safe way to escape this situation. So, How can I stop the safely with this process? Or is there any other place to check ? --------------- For now my raid is stuck like this , So, I made my mind to give up growing. mdadm -D /dev/md128 /dev/md128: Version : 1.2 Creation Time : Tue Sep 22 08:59:59 2020 Raid Level : raid5 Array Size : 23427528384 (22342.23 GiB 23989.79 GB) Used Dev Size : 7809176128 (7447.41 GiB 7996.60 GB) Raid Devices : 5 Total Devices : 5 Persistence : Superblock is persistent Update Time : Sat Jul 27 01:18:05 2024 State : clean, reshaping Active Devices : 5 Working Devices : 5 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 64K Consistency Policy : unknown Reshape Status : 6% complete Delta Devices : 1, (4->5) Name : 0a438048:RaidSecond-0 (local to host 0a438048) UUID : a9bcec2a:5772e6a5:045f28ac:62ae0002 Events : 53644 Number Major Minor RaidDevice State 0 8 115 0 active sync /dev/sdh3 1 8 99 1 active sync /dev/sdg3 5 8 144 2 active sync /dev/sdj 4 8 128 3 active sync /dev/sdi 6 8 160 4 active sync /dev/sdk then I would like to do this, and remove /dev/sdk $mdadm /dev/md128 --grow --raid-devices=4 However with this command, how mdadm understood the target drive is /dev/sdk? Or is there any good method to shrink the Raid5? ------------------ I found out there is bad blocks in one device /dev/sdj I guess this is the reason to stuck in reshape root@nas316:~# mdadm --examine /dev/sdh3 /dev/sdh3: Magic : a92b4efc Version : 1.2 Feature Map : 0x44 Array UUID : a9bcec2a:5772e6a5:045f28ac:62ae0002 Name : 0a438048:RaidSecond-0 (local to host 0a438048) Creation Time : Tue Sep 22 08:59:59 2020 Raid Level : raid5 Raid Devices : 5 Avail Dev Size : 15618352320 (7447.41 GiB 7996.60 GB) Array Size : 31236704512 (29789.64 GiB 31986.39 GB) Used Dev Size : 15618352256 (7447.41 GiB 7996.60 GB) Data Offset : 263552 sectors New Offset : 263040 sectors Super Offset : 8 sectors State : clean Device UUID : 1d396d49:23a9dd4f:648792aa:3acd8cc3 Reshape pos'n : 1931009792 (1841.55 GiB 1977.35 GB) Delta Devices : 1 (4->5) Update Time : Sun Feb 2 16:39:55 2025 Bad Block Log : 512 entries available at offset 264 sectors Checksum : 8808a712 - correct Events : 53647 Layout : left-symmetric Chunk Size : 64K Device Role : Active device 0 Array State : A.AAA ('A' == active, '.' == missing, 'R' == replacing) root@nas316:~# mdadm --examine /dev/sdi /dev/sdi: Magic : a92b4efc Version : 1.2 Feature Map : 0x4c Array UUID : a9bcec2a:5772e6a5:045f28ac:62ae0002 Name : 0a438048:RaidSecond-0 (local to host 0a438048) Creation Time : Tue Sep 22 08:59:59 2020 Raid Level : raid5 Raid Devices : 5 Avail Dev Size : 15627789616 (7451.91 GiB 8001.43 GB) Array Size : 31236704512 (29789.64 GiB 31986.39 GB) Used Dev Size : 15618352256 (7447.41 GiB 7996.60 GB) Data Offset : 263552 sectors New Offset : 263040 sectors Super Offset : 8 sectors State : clean Device UUID : 05652fdf:8edc650e:9cea98d8:36aee81d Reshape pos'n : 1931009792 (1841.55 GiB 1977.35 GB) Delta Devices : 1 (4->5) Update Time : Sun Feb 2 16:39:55 2025 Bad Block Log : 512 entries available at offset 264 sectors - bad blocks present. Checksum : 65454b15 - correct Events : 53647 Layout : left-symmetric Chunk Size : 64K Device Role : Active device 3 Array State : A.AAA ('A' == active, '.' == missing, 'R' == replacing) root@nas316:~# mdadm --examine /dev/sdj /dev/sdj: Magic : a92b4efc Version : 1.2 Feature Map : 0x4c Array UUID : a9bcec2a:5772e6a5:045f28ac:62ae0002 Name : 0a438048:RaidSecond-0 (local to host 0a438048) Creation Time : Tue Sep 22 08:59:59 2020 Raid Level : raid5 Raid Devices : 5 Avail Dev Size : 15627789616 (7451.91 GiB 8001.43 GB) Array Size : 31236704512 (29789.64 GiB 31986.39 GB) Used Dev Size : 15618352256 (7447.41 GiB 7996.60 GB) Data Offset : 263552 sectors New Offset : 263040 sectors Super Offset : 8 sectors State : clean Device UUID : 4a0e778b:1b6d8f53:e77d3caa:daeff61d Reshape pos'n : 1931009792 (1841.55 GiB 1977.35 GB) Delta Devices : 1 (4->5) Update Time : Sun Feb 2 16:39:55 2025 Bad Block Log : 512 entries available at offset 264 sectors - bad blocks present. Checksum : 286859e7 - correct Events : 53647 Layout : left-symmetric Chunk Size : 64K Device Role : Active device 2 Array State : A.AAA ('A' == active, '.' == missing, 'R' == replacing) root@nas316:~# root@nas316:~# mdadm --examine /dev/sdk /dev/sdk: Magic : a92b4efc Version : 1.2 Feature Map : 0x44 Array UUID : a9bcec2a:5772e6a5:045f28ac:62ae0002 Name : 0a438048:RaidSecond-0 (local to host 0a438048) Creation Time : Tue Sep 22 08:59:59 2020 Raid Level : raid5 Raid Devices : 5 Avail Dev Size : 15627789616 (7451.91 GiB 8001.43 GB) Array Size : 31236704512 (29789.64 GiB 31986.39 GB) Used Dev Size : 15618352256 (7447.41 GiB 7996.60 GB) Data Offset : 263552 sectors New Offset : 263040 sectors Super Offset : 8 sectors State : clean Device UUID : 9ba494c3:2c33742e:ba372d11:dbcc6d97 Reshape pos'n : 1931009792 (1841.55 GiB 1977.35 GB) Delta Devices : 1 (4->5) Update Time : Sun Feb 2 16:39:55 2025 Bad Block Log : 512 entries available at offset 264 sectors Checksum : 1bd24cb6 - correct Events : 53647 Layout : left-symmetric Chunk Size : 64K Device Role : Active device 4 Array State : A.AAA ('A' == active, '.' == missing, 'R' == replacing) root@nas316:~# mdadm --examine /dev/sdg3 /dev/sdg3: Magic : a92b4efc Version : 1.2 Feature Map : 0x44 Array UUID : a9bcec2a:5772e6a5:045f28ac:62ae0002 Name : 0a438048:RaidSecond-0 (local to host 0a438048) Creation Time : Tue Sep 22 08:59:59 2020 Raid Level : raid5 Raid Devices : 5 Avail Dev Size : 15618352320 (7447.41 GiB 7996.60 GB) Array Size : 31236704512 (29789.64 GiB 31986.39 GB) Used Dev Size : 15618352256 (7447.41 GiB 7996.60 GB) Data Offset : 263552 sectors New Offset : 263040 sectors Super Offset : 8 sectors State : clean Device UUID : 09b3c7dc:78455438:3ff2291a:70531266 Reshape pos'n : 1931009792 (1841.55 GiB 1977.35 GB) Delta Devices : 1 (4->5) Update Time : Sun Feb 2 16:39:55 2025 Bad Block Log : 512 entries available at offset 264 sectors Checksum : 15f6ae6b - correct Events : 53647 Layout : left-symmetric Chunk Size : 64K Device Role : spare Array State : A.AAA ('A' == active, '.' == missing, 'R' == replacing)

I have a RAID5 array configured with MDADM in Ubuntu 24.04 LTS. The array consists of five 8TB drives, and there have been no changes to it recently. Today I noticed that the array is not accessible. * mdadm --detail /dev/md0 shows that the array is Inactive:

-shellsession
    $ mdadm --detail /dev/md0
    /dev/md0:
               Version : 1.2
            Raid Level : raid5
         Total Devices : 5
           Persistence : Superblock is persistent
                 State : inactive
       Working Devices : 5
    
                  Name : Europa:0  (local to host Europa)
                  UUID : 62595935:e04505fc:3e79426a:40326185
                Events : 76498
    
        Number   Major   Minor   RaidDevice
    
           -       8        1        -        /dev/sda1
           -       8       81        -        /dev/sdf1
           -       8       65        -        /dev/sde1
           -       8       49        -        /dev/sdd1
           -       8       33        -        /dev/sdc1

* Using mdadm --examine of each drive, I find that they all say the state is Clean, and 4 of them show the array state is AAAAA (all drives are active), but one of them (sda1) shows the array state is ....A * Using cat /proc/mdstat I find that ONLY device sda1 appears!

-shellsession
    $ sudo cat /proc/mdstat
    Personalities : [raid1] [raid0] [raid6] [raid5] [raid4] [raid10] 
    md1 : active raid1 sdg1 sdh1
          2930132992 blocks super 1.2 [2/2] [UU]
          bitmap: 0/22 pages [0KB], 65536KB chunk
    
    md0 : inactive sda1
          7813893632 blocks super 1.2

* By looking at mdadm --examine for the event count and update time, I can see that sda1 has slightly more events and a more recent update time than the other drives:

-shellsession
    $ mdadm --examine /dev/sda1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 | egrep 'Event|/dev/sd'
    /dev/sda1:
             Events : 76498
    /dev/sdc1:
             Events : 76490
    /dev/sdd1:
             Events : 76490
    /dev/sde1:
             Events : 76490
    /dev/sdf1:
             Events : 76490

and

-shellsession
    $ mdadm --examine /dev/sda1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 | egrep 'Update Time|/dev/sd'
    /dev/sda1:
        Update Time : Mon Jan 13 14:51:59 2025
    /dev/sdc1:
        Update Time : Mon Jan 13 05:03:20 2025
    /dev/sdd1:
        Update Time : Mon Jan 13 05:03:20 2025
    /dev/sde1:
        Update Time : Mon Jan 13 05:03:20 2025
    /dev/sdf1:
        Update Time : Mon Jan 13 05:03:20 2025

So my question is how to interpret this? When searching online I found most people reporting the inverse: one drive has fewer events than the others, and one drive has an earlier Update Time than the others. I can't imagine that 4 drives went bad at exactly the same time, especially since there was no power failure or anything else that might explain a widespread hardware issue. So what does this mean, and how do I recover the array?