md: kicking non-fresh sdg from array! md/raid:md0: and then not enough operational devices (3/7 failed)

today I run in a disaster... I have a RAID 6 with 7 HDDs and yesterday one disk failed. After replacing the disk and did a rebuild over night I found out that a 2nd HDD was out of the RAID... So today I 've started to backup my Files on external Drives but then the copying stopped and as I've checked why and saw in Webmins RAID that sdg was "down". I shut down the server and checked the hardware and found out, that the backplate, where the HDDs are connected got lose... After repairing it all drives are now back but my RAID 6 don't start anymore :-/ dmesg shows me: md: kicking non-fresh sdg from array! md: kicking non-fresh sdf from array! md: kicking non-fresh sde from array! md/raid:md0: not enough operational devices (3/7 failed) ... and after many md0: ADD_NEW_DISK not supported I can read this: EXT4-fs (md0): unable to read superblock With sudo mdadm --examine I checked the sdg, sdf and sde and e and f shows "State clean" where the sdg, which was "down" before repairing shows "Active". So 6 of 7 Devices shows "Clean" except the sdg. Here is the list of the output of all devices:

Disk sdb
/dev/sdb:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : e866cf54:90d5c74e:fe00b6e7:d25c82f4
           Name : N5550:0  (local to host N5550)
  Creation Time : Fri Oct 29 14:43:58 2021
     Raid Level : raid6
   Raid Devices : 7

 Avail Dev Size : 3906770096 (1862.89 GiB 2000.27 GB)
     Array Size : 9766906880 (9314.45 GiB 10001.31 GB)
  Used Dev Size : 3906762752 (1862.89 GiB 2000.26 GB)
    Data Offset : 259072 sectors
   Super Offset : 8 sectors
   Unused Space : before=258992 sectors, after=7344 sectors
          State : clean
    Device UUID : 9180f101:1dacdd9e:4adae9c4:fbeb2552

Internal Bitmap : 8 sectors from superblock
    Update Time : Sat Mar 26 18:13:45 2022
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : 38019182 - correct
         Events : 256508

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAA.A.. ('A' == active, '.' == missing, 'R' == replacing)
Disk sdc
/dev/sdc:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : e866cf54:90d5c74e:fe00b6e7:d25c82f4
           Name : N5550:0  (local to host N5550)
  Creation Time : Fri Oct 29 14:43:58 2021
     Raid Level : raid6
   Raid Devices : 7

 Avail Dev Size : 3906770096 (1862.89 GiB 2000.27 GB)
     Array Size : 9766906880 (9314.45 GiB 10001.31 GB)
  Used Dev Size : 3906762752 (1862.89 GiB 2000.26 GB)
    Data Offset : 259072 sectors
   Super Offset : 8 sectors
   Unused Space : before=258992 sectors, after=7344 sectors
          State : clean
    Device UUID : 889c6877:5ee5c647:eebd209c:d9c6abcb

Internal Bitmap : 8 sectors from superblock
    Update Time : Sat Mar 26 18:13:45 2022
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : a71ea53d - correct
         Events : 256508

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AAA.A.. ('A' == active, '.' == missing, 'R' == replacing)
Disk sdd
/dev/sdd:
   MBR Magic : aa55
Partition :   3907026944 sectors at         2048 (type fd)
Disk sde
/dev/sde:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : e866cf54:90d5c74e:fe00b6e7:d25c82f4
           Name : N5550:0  (local to host N5550)
  Creation Time : Fri Oct 29 14:43:58 2021
     Raid Level : raid6
   Raid Devices : 7

 Avail Dev Size : 3906770096 (1862.89 GiB 2000.27 GB)
     Array Size : 9766906880 (9314.45 GiB 10001.31 GB)
  Used Dev Size : 3906762752 (1862.89 GiB 2000.26 GB)
    Data Offset : 259072 sectors
   Super Offset : 8 sectors
   Unused Space : before=258992 sectors, after=7344 sectors
          State : clean
    Device UUID : 34198042:3d4c802b:36727b02:fdf65808

Internal Bitmap : 8 sectors from superblock
    Update Time : Sat Mar 26 18:05:00 2022
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : f8fb6b18 - correct
         Events : 256494

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AAAAA.. ('A' == active, '.' == missing, 'R' == replacing)
Disk sdf
/dev/sdf:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : e866cf54:90d5c74e:fe00b6e7:d25c82f4
           Name : N5550:0  (local to host N5550)
  Creation Time : Fri Oct 29 14:43:58 2021
     Raid Level : raid6
   Raid Devices : 7

 Avail Dev Size : 3906770096 (1862.89 GiB 2000.27 GB)
     Array Size : 9766906880 (9314.45 GiB 10001.31 GB)
  Used Dev Size : 3906762752 (1862.89 GiB 2000.26 GB)
    Data Offset : 259072 sectors
   Super Offset : 8 sectors
   Unused Space : before=258992 sectors, after=7344 sectors
          State : clean
    Device UUID : b2e8d640:1f21336f:88d823fe:66ef7be7

Internal Bitmap : 8 sectors from superblock
    Update Time : Wed Mar 23 14:46:56 2022
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : 15cd05bb - correct
         Events : 238681

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 4
   Array State : AAAAAA. ('A' == active, '.' == missing, 'R' == replacing)
Disk sdg
/dev/sdg:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : e866cf54:90d5c74e:fe00b6e7:d25c82f4
           Name : N5550:0  (local to host N5550)
  Creation Time : Fri Oct 29 14:43:58 2021
     Raid Level : raid6
   Raid Devices : 7

 Avail Dev Size : 3906770096 (1862.89 GiB 2000.27 GB)
     Array Size : 9766906880 (9314.45 GiB 10001.31 GB)
  Used Dev Size : 3906762752 (1862.89 GiB 2000.26 GB)
    Data Offset : 259072 sectors
   Super Offset : 8 sectors
   Unused Space : before=258992 sectors, after=7344 sectors
          State : active
    Device UUID : 2bc06e22:49aa73e2:3cf7eb79:55df1180

Internal Bitmap : 8 sectors from superblock
    Update Time : Sat Mar 26 17:57:06 2022
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : 7f0ddb2a - correct
         Events : 256372

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 5
   Array State : AAAAAA. ('A' == active, '.' == missing, 'R' == replacing)
Disk sdh
/dev/sdh:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : e866cf54:90d5c74e:fe00b6e7:d25c82f4
           Name : N5550:0  (local to host N5550)
  Creation Time : Fri Oct 29 14:43:58 2021
     Raid Level : raid6
   Raid Devices : 7

 Avail Dev Size : 3906770096 (1862.89 GiB 2000.27 GB)
     Array Size : 9766906880 (9314.45 GiB 10001.31 GB)
  Used Dev Size : 3906762752 (1862.89 GiB 2000.26 GB)
    Data Offset : 259072 sectors
   Super Offset : 8 sectors
   Unused Space : before=258992 sectors, after=7344 sectors
          State : clean
    Device UUID : 7af89a18:52ef08ae:dec5ad7b:75626355

Internal Bitmap : 8 sectors from superblock
    Update Time : Sat Mar 26 18:13:45 2022
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : 17d7b107 - correct
         Events : 256508

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 4
   Array State : AAA.A.. ('A' == active, '.' == missing, 'R' == replacing)

I've tried to start the RAID with mdadm --run /dev/md0 and get: mdadm: failed to start array /dev/md0: Input/output error But after I started it with this Webmin shows me then: /dev/md0 active, FAILED, Not Started RAID6 (Dual Distributed Parity) 7.27 TiB Its 7.27 from 9TB. Any ideas how to get my RAID back to work again without data loss? I've read about that I could add devices back again to the RAID but I'm unsure and wanted to ask before. Any help would be appreciated! **UPDATE**: I forgot that one of the device is /dev/sdd1 and not /sdd! Here the examine of it: ~~~ /dev/sdd1: Magic : a92b4efc Version : 1.2 Feature Map : 0x1 Array UUID : e866cf54:90d5c74e:fe00b6e7:d25c82f4 Name : N5550:0 (local to host N5550) Creation Time : Fri Oct 29 14:43:58 2021 Raid Level : raid6 Raid Devices : 7 Avail Dev Size : 3906767872 (1862.89 GiB 2000.27 GB) Array Size : 9766906880 (9314.45 GiB 10001.31 GB) Used Dev Size : 3906762752 (1862.89 GiB 2000.26 GB) Data Offset : 259072 sectors Super Offset : 8 sectors Unused Space : before=258992 sectors, after=5120 sectors State : clean Device UUID : d8df004e:44ee4060:ba4d2c22:e7e6bdcb Internal Bitmap : 8 sectors from superblock Update Time : Sat Mar 26 18:13:45 2022 Bad Block Log : 512 entries available at offset 16 sectors Checksum : 1c4e98a4 - correct Events : 256508 Layout : left-symmetric Chunk Size : 512K Device Role : Active device 2 Array State : AAA.A.. ('A' == active, '.' == missing, 'R' == replacing) ~~~ And here an mdadm -D /dev/md0: ~~~ /dev/md0: Version : 1.2 Raid Level : raid0 Total Devices : 7 Persistence : Superblock is persistent State : inactive Working Devices : 7 Name : N5550:0 (local to host N5550) UUID : e866cf54:90d5c74e:fe00b6e7:d25c82f4 Events : 256494 Number Major Minor RaidDevice - 8 64 - /dev/sde - 8 32 - /dev/sdc - 8 112 - /dev/sdh - 8 80 - /dev/sdf - 8 16 - /dev/sdb - 8 49 - /dev/sdd1 - 8 96 - /dev/sdg ~~~