How to Rescue Data from Inconsistent LVM rootvg in RHEL 9.5

When dealing with inconsistent LVM metadata on an old rootvg in RHEL 9.5, you can recover your data using either metadata restoration or the built-in metadata correction tools. Based on your situation with old disks mounted as additional storage and the specific warnings you’re seeing, here are the recommended steps to resolve the inconsistency and access your data.

---

Contents

• Understanding the Metadata Inconsistency
• Method 1: Using vgck --updatemetadata
• Method 2: Restoring from Metadata Backups
• Mounting the Logical Volumes
• Preventive Measures for Future Recovery
• Troubleshooting Common Issues

---

Understanding the Metadata Inconsistency

The warnings you’re seeing indicate that your LVM (Logical Volume Manager) metadata is inconsistent across the physical volumes in your rootvg volume group. This commonly occurs when:

• Physical volumes are removed and re-added to the system
• The system experiences power failures or crashes during LVM operations
• Metadata gets corrupted due to disk errors

The WARNING: ignoring metadata seqno 11 on /dev/sda3 for seqno 13 on /dev/sdc4 for VG rootvg error shows that /dev/sda3 has older metadata (seqno 11) while /dev/sdc4 has newer metadata (seqno 13), creating an inconsistency.

According to the Red Hat documentation, this inconsistency typically prevents LVM from properly accessing the volume group until resolved.

---

Method 1: Using vgck --updatemetadata

The vgck --updatemetadata command is specifically designed to correct metadata inconsistencies as suggested by your warning message.

Step-by-Step Process:

1. Check current LVM status:

   bash

   pvs
   vgs
   lvs
   

2. Use updatemetadata to correct inconsistency:

   bash

   vgck --updatemetadata rootvg
   

This command will rewrite the Volume Group metadata to correct the problems. According to the Linux manual pages, vgck --updatemetadata VG rewrites VG metadata to correct problems.

3. Verify the correction:
   bash

   pvs
   vgs
   

If successful, the inconsistent metadata warnings should disappear.

---

Method 2: Restoring from Metadata Backups

If the vgck --updatemetadata approach doesn’t work, you can restore the volume group metadata from backups. LVM automatically maintains backup files in /etc/lvm/backup/.

Step-by-Step Process:

1. List available backup files:

   bash

   ls -la /etc/lvm/backup/
   

2. List archived versions (if available):

   bash

   ls -la /etc/lvm/archive/
   

3. Restore from the most recent backup:

   bash

   vgcfgrestore --test rootvg  # Test mode first
   vgcfgrestore rootvg         # Actual restore
   

As mentioned in The Geek Diary, restoring older versions of VG metadata with vgcfgrestore works in most situations for inconsistent metadata errors.

4. If you need to restore from a specific archive file:
   bash

   cd /etc/lvm/archive/
   ls -lrt  # List by modification time
   vgcfgrestore --test rootvg -f /etc/lvm/archive/rootvg_00000-1234567890.vg
   vgcfgrestore rootvg -f /etc/lvm/archive/rootvg_00000-1234567890.vg
   

The Red Hat documentation shows that you can use the vgcfgrestore command to restore the volume group’s metadata from backup files.

---

Mounting the Logical Volumes

Once the metadata is consistent, you can mount the logical volumes to access your data.

Step-by-Step Process:

1. List the logical volumes in rootvg:

   bash

   lvs rootvg
   

2. Create mount points:

   bash

   mkdir /mnt/rootvg_recovery
   

3. Mount the root logical volume (typically named rootvg/root):

   bash

   mount /dev/mapper/rootvg-root /mnt/rootvg_recovery
   

4. Mount other important volumes:

   bash

   # Common logical volumes
   mount /dev/mapper/rootvg-home /mnt/rootvg_recovery/home
   mount /dev/mapper/rootvg-var /mnt/rootvg_recovery/var
   mount /dev/mapper/rootvg-opt /mnt/rootvg_recovery/opt
   

5. Verify the mount:

   bash

   df -h /mnt/rootvg_recovery
   

---

Preventive Measures for Future Recovery

To make future recovery easier, consider these preventive measures:

1. Regular LVM metadata backups:

   bash

   vgcfgbackup
   

2. Keep archive copies:

   bash

   # Archive current state
   vgcfgbackup --archive rootvg
   

3. Document LVM configuration:

   bash

   vgs --options=all
   pvs --options=all
   lvs --options=all
   

4. Create filesystem-level backups of critical data.

As noted in The Geek Diary, always take a backup of the /etc/lvm directory before performing recovery operations.

---

Troubleshooting Common Issues

If vgck --updatemetadata fails:

1. Check for active volumes:

   bash

   vgs -o +lv_count,lv_count rootvg
   

2. Force update with caution:

   bash

   vgck --updatemetadata --force rootvg
   

If vgcfgrestore reports active volumes:

1. Check for mounted volumes:

   bash

   mount | grep mapper
   

2. Unmount conflicting volumes:

   bash

   umount /dev/mapper/rootvg-*
   

3. Retry restore:

   bash

   vgcfgrestore rootvg
   

If checksum errors persist:

According to The Geek Diary, checksum errors indicate more serious metadata corruption. In such cases, you may need to:

1. Use older archive files
2. Consider recreating the volume group from scratch if critical data is already backed up

---

Sources

1. Stack Overflow - Linux LVM Rescue data from old rootvg
2. The Geek Diary - LVM error “WARNING: Inconsistent metadata found” – How to resolve in CentOS / RHEL
3. Linux man pages - vgck(8)
4. Arch Linux manual pages - vgck(8)
5. The Geek Diary - LVM VG Metadata Corruption with ‘Checksum error’
6. Red Hat Customer Portal - How do I resolve inconsistent LVM Volume Group metadata errors in RHEL?
7. The Geek Diary - How to rebuild LVM from Archive (metadata backups) in RHEL/CentOS
8. Red Hat Documentation - Recovering Physical Volume Metadata
9. The Geek Diary - How to recover deleted Logical volume (LV) in LVM using vgcfgrestore
10. GoLinuxCloud - 5 easy steps to recover LVM2 partition, PV, VG, LVM metdata in Linux

Conclusion

Rescuing data from an old rootvg with inconsistent LVM metadata in RHEL 9.5 is achievable through systematic metadata correction and restoration. The two primary approaches are:

1. Use vgck --updatemetadata rootvg for straightforward metadata inconsistencies
2. Restore from backup using vgcfgrestore when metadata corruption is more severe

Always start with test runs and dry modes before making actual changes to your system. The /etc/lvm/backup/ and /etc/lvm/archive/ directories contain your safety net for recovery operations.

For future prevention, maintain regular LVM metadata backups and document your volume group configurations. This will make any future recovery operations much smoother and less risky.

Remember that LVM recovery operations can be destructive if performed incorrectly, so always work from a rescue environment when possible and have your data backed up before attempting major metadata changes.