How to replace faulty disk in software raid using lvm

one of the SuSE 12 LinuxServers has reported disk failure. Fortunately the Database Server has Software Raid hence the system is still up and running. But as recommended, we would like to replace the failed disk with a new one and rebuild the software raid on it. System Information is : Total 4 Internal Disks. sda, sdb , sdc and sdd The fdisk partitions are : fdisk -l Disk /dev/sdb: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disklabel type: dos Disk identifier: 0x0007d757 Device Boot Start End Sectors Size Id Type /dev/sdb1 * 2048 2105343 2103296 1G fd Linux raid autodetect /dev/sdb2 2105344 39858175 37752832 18G fd Linux raid autodetect /dev/sdb3 39858176 1953523711 1913665536 912.5G fd Linux raid autodetect Disk /dev/sdc: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disklabel type: dos Disk identifier: 0x000a0e8a Device Boot Start End Sectors Size Id Type /dev/sdc1 2048 1953523711 1953521664 931.5G fd Linux raid autodetect Disk /dev/sdd: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disklabel type: dos Disk identifier: 0x000caaad Device Boot Start End Sectors Size Id Type /dev/sdd1 2048 1953523711 1953521664 931.5G fd Linux raid autodetect Software RAID --> sda + sdb (sda is failed disk) Software RAID --> sdb + sdc DBServer# cat /proc/mdstat Personalities : [raid1] md3 : active raid1 sdc1 sdd1 976760640 blocks super 1.0 [2/2] [UU] bitmap: 2/8 pages [8KB], 65536KB chunk md0 : active raid1 sdb1 sda10 1051584 blocks super 1.0 [2/1] [_U] bitmap: 1/1 pages [4KB], 65536KB chunk md2 : active raid1 sdb3 sda30 956832576 blocks super 1.0 [2/1] [_U] bitmap: 2/8 pages [8KB], 65536KB chunk md1 : active raid1 sdb2 sda20 18876288 blocks super 1.0 [2/1] [_U] bitmap: 0/1 pages [0KB], 65536KB chunk unused devices: So md0,md1 and md2 have failed devices namely sda1,sda2 and sda3 Please note that it also has 2 VGs defined as shown below, 1 VG - system (/dev/md2) 2 VG - ora_db (/dev/md3) pvdisplay --- Physical volume --- PV Name /dev/md3 VG Name ora_db PV Size 931.51 GiB / not usable 3.81 MiB Allocatable yes PE Size 4.00 MiB Total PE 238466 Free PE 84866 Allocated PE 153600 PV UUID vgPdWQ-x6CW-vvdF-moxh-FKyb-wpSU-NdJqSm --- Physical volume --- PV Name /dev/md2 VG Name system PV Size 912.51 GiB / not usable 2.81 MiB Allocatable yes PE Size 4.00 MiB Total PE 233601 Free PE 182401 Allocated PE 51200 PV UUID rdff2n-ztxd-lcBY-nAqk-8O9u-fnFG-BVI91v The grub.conf shows : (Relevant part) if [ x$feature_default_font_path = xy ] ; then font=unicode else insmod part_msdos msdos insmod diskfilter mdraid1x lvm insmod ext2 set root='lvmid/m7AEp0-79EG-D2Vi-ELzE-BTzh-C8mN-CLxrpz/S0eZEl-PlBX-E1ZL-oCwL-SmUx-4Qe4-Mz9NHX' if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hint='lvmid/m7AEp0-79EG-D2Vi-ELzE-BTzh-C8mN-CLxrpz/S0eZEl-PlBX-E1ZL-oCwL-SmUx-4Qe4-Mz9NHX' 7c2e3a9c-5f5b-47e3-8a0a-d1e66f12747c else search --no-floppy --fs-uuid --set=root 7c2e3a9c-5f5b-47e3-8a0a-d1e66f12747c fi font="/share/grub2/unicode.pf2" fi if loadfont $font ; then set gfxmode=auto load_video insmod gfxterm set locale_dir=$prefix/locale set lang=POSIX insmod gettext fi terminal_output gfxterm insmod part_msdos msdos insmod diskfilter mdraid1x insmod ext2 set root='mduuid/531cd341e2c7d5a71c542ad04d9ea589' if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hint='mduuid/531cd341e2c7d5a71c542ad04d9ea589' 96c11697-c3b7-4f11-90fc-3aef207db526 else search --no-floppy --fs-uuid --set=root 96c11697-c3b7-4f11-90fc-3aef207db526 fi Quote: The procedure to follow should go like this, > First we mark /dev/sda1 as failed: mdadm --manage /dev/md0 --fail /dev/sda1 > Then we remove /dev/sda1 from /dev/md0: mdadm --manage /dev/md0 --remove /dev/sda1 > Now we do the same steps again for /dev/sda2 and sda3 (which is part > of /dev/md1 and /dev/md2) > > Then power down the system: shutdown -h now > and replace the old /dev/sdb hard drive with a new one > > After inserting new SATA disk /dev/sda, boot the system. > > Then we create the exact same partitioning as on /dev/sda. We can do > this with one simple command: > sfdisk -d /dev/sdb | sfdisk /dev/sda > Check if both the disks have same partitions (fdisk -l) > 8.Next we add /dev/sda1 to /dev/md0 and /dev/sda2 to /dev/md1 and /dev/sda3 to /dev/md3: mdadm --manage /dev/md0 --add /dev/sda1 mdadm --manage /dev/md1 --add /dev/sda2 mdadm --manage /dev/md2 --add /dev/sda3 > Confirm the synchronisation in progress cat /proc/mdstat Please let me know if I have missed something. 2 important points I guess would be, how should I take care of lvm and grub in this case. Do I have to do something extra to take care of it or the command sfdisk -d /dev/sdb | sfdisk /dev/sda , should take care of LVM as well. How should I take care of grub in this case? As grun.conf shows entries pertaining to LVM as well as MDADM. Do I have to change anything here before I shutdown the system? I understand the system has 2 pointers to take care of mdadm+lvm. Which have complicated things. Else would it be easier to setup completely new system? Not yet tried anything. Only collecting information. grub.cfg ### BEGIN /etc/grub.d/00_header ### if [ -f ${config_directory}/grubenv ]; then load_env -f ${config_directory}/grubenv elif [ -s $prefix/grubenv ]; then load_env fi if [ "${env_block}" ] ; then load_env -f "${env_block}" fi if [ "${next_entry}" ] ; then set default="${next_entry}" set next_entry= save_env next_entry if [ "${env_block}" ] ; then save_env -f "${env_block}" next_entry fi set boot_once=true else set default="${saved_entry}" fi if [ x"${feature_menuentry_id}" = xy ]; then menuentry_id_option="--id" else menuentry_id_option="" fi export menuentry_id_option if [ "${prev_saved_entry}" ]; then set saved_entry="${prev_saved_entry}" save_env saved_entry set prev_saved_entry= save_env prev_saved_entry set boot_once=true fi function savedefault { if [ -z "${boot_once}" ]; then saved_entry="${chosen}" save_env saved_entry fi } function load_video { if [ x$feature_all_video_module = xy ]; then insmod all_video else insmod efi_gop insmod efi_uga insmod ieee1275_fb insmod vbe insmod vga insmod video_bochs insmod video_cirrus fi } if [ x$feature_default_font_path = xy ] ; then font=unicode else insmod part_msdos msdos insmod diskfilter mdraid1x lvm insmod ext2 set root='lvmid/m7AEp0-79EG-D2Vi-ELzE-BTzh-C8mN-CLxrpz/S0eZEl-PlBX-E1ZL-oCwL-SmUx-4Qe4-Mz9NHX' if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hint='lvmid/m7AEp0-79EG-D2Vi-ELzE-BTzh-C8mN-CLxrpz/S0eZEl-PlBX-E1ZL-oCwL-SmUx-4Qe4-Mz9NHX' 7c2e3a9c-5f5b-47e3-8a0a-d1e66f12747c else search --no-floppy --fs-uuid --set=root 7c2e3a9c-5f5b-47e3-8a0a-d1e66f12747c fi font="/share/grub2/unicode.pf2" fi if loadfont $font ; then set gfxmode=auto load_video insmod gfxterm set locale_dir=$prefix/locale set lang=POSIX insmod gettext fi terminal_output gfxterm insmod part_msdos msdos insmod diskfilter mdraid1x insmod ext2 set root='mduuid/531cd341e2c7d5a71c542ad04d9ea589' if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hint='mduuid/531cd341e2c7d5a71c542ad04d9ea589' 96c11697-c3b7-4f11-90fc-3aef207db526 else search --no-floppy --fs-uuid --set=root 96c11697-c3b7-4f11-90fc-3aef207db526 fi insmod gfxmenu loadfont ($root)/grub2/themes/SLE/DejaVuSans-Bold14.pf2 loadfont ($root)/grub2/themes/SLE/DejaVuSans10.pf2 loadfont ($root)/grub2/themes/SLE/DejaVuSans12.pf2 loadfont ($root)/grub2/themes/SLE/ascii.pf2 insmod png set theme=($root)/grub2/themes/SLE/theme.txt export theme if [ x${boot_once} = xtrue ]; then set timeout=0 elif [ x$feature_timeout_style = xy ] ; then set timeout_style=menu set timeout=8 # Fallback normal timeout code in case the timeout_style feature is # unavailable. else set timeout=8 fi ### END /etc/grub.d/00_header ### ### BEGIN /etc/grub.d/10_linux ### menuentry 'SLES12' --class sles12 --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-690785da-f0f0-4250-b693-5a008acbba10' { load_video set gfxpayload=keep insmod gzio insmod part_msdos msdos insmod diskfilter mdraid1x insmod ext2 set root='mduuid/531cd341e2c7d5a71c542ad04d9ea589' if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hint='mduuid/531cd341e2c7d5a71c542ad04d9ea589' 96c11697-c3b7-4f11-90fc-3aef207db526 else search --no-floppy --fs-uuid --set=root 96c11697-c3b7-4f11-90fc-3aef207db526 fi echo 'Loading Linux 3.12.28-4-default ...' linux /vmlinuz-3.12.28-4-default root=UUID=690785da-f0f0-4250-b693-5a008acbba10 resume=/dev/md1 splash=silent quiet crashkernel=232M-:116M showopts echo 'Loading initial ramdisk ...' initrd /initrd-3.12.28-4-default } submenu 'Advanced options for SLES12' --hotkey=1 $menuentry_id_option 'gnulinux-advanced-690785da-f0f0-4250-b693-5a008acbba10' { menuentry 'SLES12, with Linux 3.12.28-4-default' --hotkey=2 --class sles12 --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-3.12.28-4-default-advanced-690785da-f0f0-4250-b693- 5a008acbba10' { load_video set gfxpayload=keep insmod gzio insmod part_msdos msdos insmod diskfilter mdraid1x insmod ext2 set root='mduuid/531cd341e2c7d5a71c542ad04d9ea589' if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hint='mduuid/531cd341e2c7d5a71c542ad04d9ea589' 96c11697-c3b7-4f11-90fc-3aef207db526 else search --no-floppy --fs-uuid --set=root 96c11697-c3b7-4f11-90fc-3aef207db526 fi echo 'Loading Linux 3.12.28-4-default ...' linux /vmlinuz-3.12.28-4-default root=UUID=690785da-f0f0-4250-b693-5a008acbba10 resume=/dev/md1 splash=silent quiet crashkernel=232M-:116M showopts echo 'Loading initial ramdisk ...' initrd /initrd-3.12.28-4-default } menuentry 'SLES12, with Linux 3.12.28-4-default (recovery mode)' --hotkey=3 --class sles12 --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-3.12.28-4-default-recovery- 690785da-f0f0-4250-b693-5a008acbba10' { load_video set gfxpayload=keep insmod gzio insmod part_msdos msdos insmod diskfilter mdraid1x insmod ext2 set root='mduuid/531cd341e2c7d5a71c542ad04d9ea589' if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hint='mduuid/531cd341e2c7d5a71c542ad04d9ea589' 96c11697-c3b7-4f11-90fc-3aef207db526 else search --no-floppy --fs-uuid --set=root 96c11697-c3b7-4f11-90fc-3aef207db526 fi echo 'Loading Linux 3.12.28-4-default ...' linux /vmlinuz-3.12.28-4-default root=UUID=690785da-f0f0-4250-b693-5a008acbba10 showopts apm=off noresume edd=off powersaved=off nohz=off highres=off processor.max_cstate=1 nomodeset x11failsafe crashkernel=232M-:116M echo 'Loading initial ramdisk ...' initrd /initrd-3.12.28-4-default } } ### END /etc/grub.d/10_linux ### ### BEGIN /etc/grub.d/20_linux_xen ### ### END /etc/grub.d/20_linux_xen ### ### BEGIN /etc/grub.d/20_memtest86+ ### ### END /etc/grub.d/20_memtest86+ ### ### BEGIN /etc/grub.d/30_os-prober ### ### END /etc/grub.d/30_os-prober ### ### BEGIN /etc/grub.d/40_custom ### # This file provides an easy way to add custom menu entries. Simply type the # menu entries you want to add after this comment. Be careful not to change # the 'exec tail' line above. ### END /etc/grub.d/40_custom ### ### BEGIN /etc/grub.d/41_custom ### if [ -f ${config_directory}/custom.cfg ]; then source ${config_directory}/custom.cfg elif [ -z "${config_directory}" -a -f $prefix/custom.cfg ]; then source $prefix/custom.cfg; fi ### END /etc/grub.d/41_custom ### ### BEGIN /etc/grub.d/80_suse_btrfs_snapshot ### ### END /etc/grub.d/80_suse_btrfs_snapshot ### ### BEGIN /etc/grub.d/90_persistent ### ### END /etc/grub.d/90_persistent ### I did install grub on /dev/sdb. Please find the output. # grub2-install /dev/sdb Installing for i386-pc platform. error: cannot read `/dev/sda': Input/output error. error: cannot read `/dev/sda': Input/output error. error: cannot read `/dev/sda': Input/output error. error: cannot read `/dev/sda': Input/output error. error: cannot read `/dev/sda': Input/output error. grub2-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image.. error: cannot read `/dev/sda': Input/output error. error: cannot read `/dev/sda': Input/output error. error: cannot read `/dev/sda': Input/output error. error: cannot read `/dev/sda': Input/output error. error: cannot read `/dev/sda': Input/output error. grub2-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image.. Installation finished. No error reported. Not sure why it gave sda errors though. And then cross checked with following command again. # dd bs=512 count=1 if=/dev/sdb 2>/dev/null| strings ZRr= `|f \|f1 GRUB Geom Hard Disk Read Error