Problems accessing files on a RAID1+LVM from failed QNAP

# Brief context I have a QNAP TS-451. The NAS has failed, but I'm pretty sure the drives are still fine. I have a new QNAP TS-462 to try to replace it and I'd like to ensure my data is intact in the new system. # Question How can I access the contents of the LVM when I cannot assemble the RAID? (more details to the question below) ### Environment The **TS-451** system is dead and no longer boots. It was configured like so: - Drive A - 20TB RAID1 mirror - Drive B - 20TB RAID1 mirror - Drive C - 8TB no RAID - Drive D - 14TB no RAID The **TS-462** is a new system that I want to migrate Drives A/B/C/D to. Separate **Linux system** to (ideally) non-destructively read data from drives A/B/C/D: * Fedora 38 * Linux kernel 6.2.14-300 * Old BIOS (i.e. I do not think it can boot off GPT partitions -- not a problem in this case) * Pentium Dual Core E6500 (just to give you an idea how old this system is) ## Experiment 1 I tried moving one of the drives that is not important (Drive C) to the new TS-462, but it appears that the TS-462 is not able to read it. From what I can tell, it seems confused about the partition table (fdisk reports one thing and blkid/lsblk reports differently. On a separate Linux computer (Fedora 38), I'm able to read the LVM and mount the partition off Drives C & D and confirm the files are intact. ## Experiment 2 So I tried to do the same thing to read off Drive B. Drives A/B are part of a RAID1 mirror, so I figured it should be fine to just experiment (cautiously) on one of the drives and leave the other as backup. Unfortunately, I don't seem to be able to activate the LVM. Here are some of the things I've tried on the Linux system:

# fdisk -l /dev/sdc
Disk /dev/sdc: 18.19 TiB, 20000588955648 bytes, 39063650304 sectors
Disk model: WDC WD201KFGX-68
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: DFC9F3CE-9CFA-4251-8DD1-48BF99476C04

Device           Start         End     Sectors   Size Type
/dev/sdc1           40     1060289     1060250 517.7M Microsoft basic data
/dev/sdc2      1060296     2120579     1060284 517.7M Microsoft basic data
/dev/sdc3      2120584 39045853979 39043733396  18.2T Microsoft basic data
/dev/sdc4  39045853984 39046914269     1060286 517.7M Microsoft basic data
/dev/sdc5  39046914272 39063621869    16707598     8G Microsoft basic data

From what I gather from http://www.rodsbooks.com/linux-fs-code/ , the fact the these partitions show up as Microsoft basic data is not a problem, but just evidence that this was setup on a very old version of Linux/fdisk. (I've had the TS-451 since about 2015).

# gdisk /dev/sdc
GPT fdisk (gdisk) version 1.0.9

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.

Command (? for help): p
Disk /dev/sdc: 39063650304 sectors, 18.2 TiB
Model: WDC WD201KFGX-68
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): DFC9F3CE-9CFA-4251-8DD1-48BF99476C04
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 39063650270
Partitions will be aligned on 8-sector boundaries
Total free space is 28423 sectors (13.9 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1              40         1060289   517.7 MiB   0700  primary
   2         1060296         2120579   517.7 MiB   0700  primary
   3         2120584     39045853979   18.2 TiB    0700  primary
   4     39045853984     39046914269   517.7 MiB   0700  primary
   5     39046914272     39063621869   8.0 GiB     0700  primary

# cat /proc/mdstat 
Personalities : 
md123 : inactive sdc5(S)
      8353780 blocks super 1.0
       
md124 : inactive sdc4(S)
      530124 blocks super 1.0
       
md125 : inactive sdc1(S)
      530108 blocks super 1.0
       
md126 : inactive sdc2(S)
      530124 blocks super 1.0
       
md127 : inactive sdc3(S)
      19521865728 blocks super 1.0
       
unused devices:

# mdadm --assemble --readonly /dev/sdc3
mdadm: device /dev/sdc3 exists but is not an md array.

Why??? :-( pvdisplay and the various LVM tools did not discover the LVMs initially. And when specifying the partition explicitly:

# pvdisplay /dev/sdc3
  Cannot use /dev/sdc3: device is an md component

What? But mdadm said it wasn't. :-(

# mdadm --examine /dev/sdc3

/dev/sdc3:
          Magic : a92b4efc
        Version : 1.0
    Feature Map : 0x0
     Array UUID : 5a6e38ee:eb6bf302:79856803:58887046
           Name : 1
  Creation Time : Wed Nov 25 03:18:54 2015
     Raid Level : raid1
   Raid Devices : 2

 Avail Dev Size : 39043731456 sectors (18.18 TiB 19.99 TB)
     Array Size : 19521865728 KiB (18.18 TiB 19.99 TB)
   Super Offset : 39043733376 sectors
   Unused Space : before=0 sectors, after=1920 sectors
          State : active
    Device UUID : f2a96ebc:996e4231:1576a39a:8606a71c

    Update Time : Sun Mar 26 00:56:13 2023
       Checksum : 893841dd - correct
         Events : 436627


   Device Role : Active device 1
   Array State : AA ('A' == active, '.' == missing, 'R' == replacing)

Checksum is good. So shouldn't that mean this is a valid RAID partition? (Although the other partitions (even the swap partitions) show valid raid info and checksums, so I don't know what that means.) Let's try a different route... However, by setting md_component_detection=0 in /etc/lvm/lvm.conf, and doing pvscan --cache --devices /dev/sdc, I was able to get at the LVM data...

# pvdisplay /dev/sdc3
  --- Physical volume ---
  PV Name               /dev/sdc3
  VG Name               vg1
  PV Size               18.18 TiB / not usable 0   
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              4766080
  Free PE               0
  Allocated PE          4766080
  PV UUID               Xt2uVv-PMCy-oHuK-0UAc-43ZI-z7TH-hHAK7A

# vgdisplay vg1
  --- Volume group ---
  VG Name               vg1
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  131
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               18.18 TiB
  PE Size               4.00 MiB
  Total PE              4766080
  Alloc PE / Size       4766080 / 18.18 TiB
  Free  PE / Size       0 / 0   
  VG UUID               oNdjeV-lPuv-PgPR-J11o-zbHQ-X7az-93sr9J

# lvdisplay vg1
  --- Logical volume ---
  LV Path                /dev/vg1/lv544
  LV Name                lv544
  VG Name                vg1
  LV UUID                z1gyiX-FhGG-cTaM-shPx-wZg4-G1xN-b9fS37
  LV Write Access        read/write
  LV Creation host, time NASEFF3AE, 2015-11-25 03:18:56 -0800
  LV Status              NOT available
  LV Size                144.00 GiB
  Current LE             36864
  Segments               2
  Allocation             inherit
  Read ahead sectors     8192
   
  --- Segments ---
  Logical extents 0 to 5119:
    Type                linear
    Physical volume     /dev/sdc3
    Physical extents    0 to 5119
   
  Logical extents 5120 to 36863:
    Type                linear
    Physical volume     /dev/sdc3
    Physical extents    1428360 to 1460103
   
   
  --- Logical volume ---
  LV Path                /dev/vg1/lv1
  LV Name                lv1
  VG Name                vg1
  LV UUID                4lsaNW-E4Bg-g93F-ko0a-Ep1m-wHD0-3Hc3Cb
  LV Write Access        read/write
  LV Creation host, time NASEFF3AE, 2015-11-25 03:19:03 -0800
  LV Status              NOT available
  LV Size                18.04 TiB
  Current LE             4729216
  Segments               2
  Allocation             inherit
  Read ahead sectors     8192
   
  --- Segments ---
  Logical extents 0 to 1423239:
    Type                linear
    Physical volume     /dev/sdc3
    Physical extents    5120 to 1428359
   
  Logical extents 1423240 to 4729215:
    Type                linear
    Physical volume     /dev/sdc3
    Physical extents    1460104 to 4766079

Yay, so I should be able to mount the drives, right?

# vgchange -ay vg1
  WARNING: PV /dev/sdc3 in VG vg1 is using an old PV header, modify the VG to update.
  device-mapper: reload ioctl on  (253:3) failed: Device or resource busy
  device-mapper: reload ioctl on  (253:3) failed: Device or resource busy
  0 logical volume(s) in volume group "vg1" now active

Hrm... so maybe md_component_detection=0 is not the right approach? ## Experiment 3 Just for thoroughness, I tested foremost, scalpel, and testdisk. These are fine tools, but I feel a bit cumbersome for the current situation. The disks should be perfectly fine and readable... partitions and filesystems intact. I'm assuming there's just some sort of version incompatibility somewhere? # Goal & Question (revisited) My goal is primarily to access my old data (from Drive A/B) one way or another. I don't mind reformatting one of the drives and migrating from another system. Or putting it in the TS-462 if I had any reasonable expectation of being able to access the data. In my current path (along Experiment 2), where I am stuck is figuring out how to get Linux to recognize the RAID? I think I'm going to try [frostschutz](https://unix.stackexchange.com/users/30851/frostschutz) 's excellent suggestion of [copy-on-write overlays](https://raid.wiki.kernel.org/index.php/Recovering_a_failed_software_RAID#Making_the_harddisks_read-only_using_an_overlay_file) shared in https://unix.stackexchange.com/questions/728449/recover-files-from-linux-raid1-member-disk-as-bad-as-it-gets . But I'm open to suggestions!