Sample Header Ad - 728x90

How to guarantee temporary immutability of LVM2 LV at raw block level?

6 votes
0 answers
272 views
I inherited an Azure VM (Ubuntu 20.04) which has a 7 disk VG fully occupied by a RAID5 LV formatted as ext4. I need to take backups and was hoping to use Azure Backup to snapshot the Azure Disks comprising the VG. Azure Disk snapshots are not point-in-time consistent so I need to freeze the storage whilst the backup runs, both for filesystem integrity and LVM metadata reasons. My workload will tolerate this; I am trying to figure out the best method of making the raw disk blocks temporarily immutable. fsfreeze - I tested freezing the filesystem, taking snapshots, unfreezing, then switching to the snapshots. In my limited testing this is working ok and I don't see anything scary from LVM when the 'restored' disks are swapped back in, but I can only perform so many tests and *if* there is a 1% edge case where my disk metadata will be inconsistent I may not find it. I'm apprehensive that I'm locking activity at such a high layer: no filesystem ops will occur whilst the FIFREEZE ioctl is active, but does this stop LVM from doing any kind of lower-level operation e.g. metadata updates, RAID-related activity? I then tried dmsetup suspend /dev/mapper/my-lvol and this *feels* like a better solution. Test setup: **fsfreeze** 1. echo 3 > /proc/sys/vm/drop_caches 2. sync ; sync (old habits die hard :) 3. fsfreeze -f /export 4. dd if=/dev/mapper/my-lvol of=/dev/null status=progress The dd runs to completion. I accept this is valid because I'm not accessing via the frozen filesystem, but it makes me wonder whether LVM could still be doing things at a low level whilst I'm assuming my Azure Disks are unchanging. **dmsetup suspend** 1. echo 3 > /proc/sys/vm/drop_caches 2. sync ; sync 3. dmsetup suspend /dev/mapper/my-lvol 4. dd if=/dev/mapper/my-lvol of=/dev/null status=progress The dd blocks as long as the suspend is in place. I can still dd the rmeta and rimage devices, but I sort of expected that. With the dmsetup option I get a hung task syslog warning for jbd2. The stacktrace shows it's trying to commit journal transaction (jbd2_journal_commit_transaction()) which both reassures me that the LV is **really** locked, but also concerns me that I'm snapshotting the filesystem in an inconsistent state and it might need to replay the journal should we ever roll back to the snapshots. Our RPO will permit some rollback but ideally I'd like to design a solution which removes this risk. **Options I've discarded** 1. File-based backups: possible, but setup & management seemed more complicated than freezing for snapshots did - to begin with! 2. Temporarily snapshotting the LV and backing up from that. The VG is full and I'd really prefer not to add more disk/resize VG/etc. **Questions** I would really appreciate any input here. As you can tell I'm at the edge (and possibly beyond) my understanding of Linux filesystems/block IO. 1. Overall, does freezing/suspending seem like a workable solution to get point-in-time consistent snapshots? 2. Am I still not looking deeply enough - just because jdb2 is unable to write a transaction could lvm or dm still be doing metadata updates at a lower level? Thanks, tim
Asked by Tim Matthews (61 rep)
Mar 21, 2023, 03:15 PM