Ytdyklly

I am struggling getting a customer's server back up.Apparently the machine just slowed down and when they rebooted it, it booted into dracut claiming it can't find the UUID of the disks.

The original O/S that ran on the now missing LVM is RHEL 7.
I added a 10Gb drive and installed CentOS 7 onto the new drive in order to have a live system to troubleshoot from.

'Naturally' there's no backups...

#pvs -a

PV         VG Fmt Attr PSize PFree

/dev/sda1         ---     0     0

/dev/sda2         ---     0     0

/dev/sdb1         ---     0     0

/dev/sdc1         ---     0     0

/dev/sdc2         ---     0     0

/dev/sdc3         ---     0     0

/dev/sdc is the new drive, I am trying to recover sda2 and sdb1.

# lsblk

NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT

fd0      2:0    1     4K  0 disk

sda      8:0    0   500G  0 disk

├─sda1   8:1    0   500M  0 part

└─sda2   8:2    0 499.5G  0 part

sdb      8:16   0     2T  0 disk

└─sdb1   8:17   0     2T  0 part

sdc      8:32   0    10G  0 disk

├─sdc1   8:33   0     1G  0 part /boot

├─sdc2   8:34   0     1G  0 part [SWAP]

└─sdc3   8:35   0     8G  0 part /

sr0     11:0    1  1024M  0 rom

PVScan reveals nothing

 # pvscan

  No matching physical volumes found

Fdisk still reports LVM partitions

# fdisk -l



Disk /dev/sda: 536.9 GB, 536870912000 bytes, 1048576000 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk label type: dos

Disk identifier: 0x000a2e76



Device Boot      Start         End      Blocks   Id  System

/dev/sda1   *        2048     1026047      512000   83  Linux

/dev/sda2         1026048  1048575999   523774976   8e  Linux LVM



Disk /dev/sdc: 10.7 GB, 10737418240 bytes, 20971520 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk label type: dos

Disk identifier: 0x000ddc56



Device Boot      Start         End      Blocks   Id  System

/dev/sdc1   *        2048     2099199     1048576   83  Linux

/dev/sdc2         2099200     4196351     1048576   82  Linux swap / Solaris

/dev/sdc3         4196352    20971519     8387584   83  Linux



Disk /dev/sdb: 2199.0 GB, 2199023255552 bytes, 4294967296 sectors

Units = sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk label type: dos

Disk identifier: 0xde7fc33e



Device Boot      Start         End      Blocks   Id  System

/dev/sdb1            2048  4294967294  2147482623+  8e  Linux LVM

What really worries me, is that it seems the beginning of the disk is empty:

# dd if=/dev/sdb1 bs=512 count=255 skip=1

255+0 records in

255+0 records out

130560 bytes (131 kB) copied, 0.00671742 s, 19.4 MB/s

** EDIT: Update without cache **
VGScan output:

#lvmcmdline.c:2829          Parsing: vgscan -vvvv

#lvmcmdline.c:1881          Recognised command vgscan_general (id 141 / enum 127).

#config/config.c:1480        devices/global_filter not found in config: defaulting to global_filter = [ "a|.*/|" ]

#libdm-config.c:1002        global/lvmetad_update_wait_time not found in config: defaulting to 10

#daemon-client.c:33            /run/lvm/lvmetad.socket: Opening daemon socket to lvmetad for protocol lvmetad version 1.

#daemon-client.c:52            Sending daemon lvmetad: hello

#cache/lvmetad.c:143           Successfully connected to lvmetad on fd 3.

#filters/filter-sysfs.c:327           Sysfs filter initialised.

#filters/filter-internal.c:77            Internal filter initialised.

#filters/filter-type.c:56            LVM type filter initialised.

#filters/filter-usable.c:183           Usable device filter initialised.

#filters/filter-mpath.c:291           mpath filter initialised.

#filters/filter-partitioned.c:69            Partitioned filter initialised.

#filters/filter-md.c:169           MD filter initialised.

#filters/filter-composite.c:109           Composite filter initialised.

#config/config.c:1480        devices/filter not found in config: defaulting to filter = [ "a|.*/|" ]

#filters/filter-regex.c:216           Regex filter initialised.

#filters/filter-usable.c:183           Usable device filter initialised.

#filters/filter-composite.c:109           Composite filter initialised.

#libdm-config.c:975         devices/cache not found in config: defaulting to /etc/lvm/cache/.cache

#filters/filter-persistent.c:415           Persistent filter initialised.

#filters/filter-composite.c:109           Composite filter initialised.

#libdm-config.c:1074        metadata/record_lvs_history not found in config: defaulting to 0

#lvmcmdline.c:2897          DEGRADED MODE. Incomplete RAID LVs will be processed.

#lvmcmdline.c:2903          Processing command: vgscan -vvvv

#lvmcmdline.c:2904          Command pid: 10060

#lvmcmdline.c:2905          System ID:

#lvmcmdline.c:2908          O_DIRECT will be used

#locking/locking.c:129         File-based locking selected.

#libdm-common.c:984           Preparing SELinux context for /run/lock/lvm to system_u:object_r:lvm_lock_t:s0.

#libdm-common.c:987           Resetting SELinux context to default value.

#misc/lvm-flock.c:202         Locking /run/lock/lvm/P_global WB

#libdm-common.c:984           Preparing SELinux context for /run/lock/lvm/P_global to system_u:object_r:lvm_lock_t:s0.

#misc/lvm-flock.c:100           _do_flock /run/lock/lvm/P_global:aux WB

#misc/lvm-flock.c:100           _do_flock /run/lock/lvm/P_global WB

#misc/lvm-flock.c:47            _undo_flock /run/lock/lvm/P_global:aux

#libdm-common.c:987           Resetting SELinux context to default value.

#cache/lvmcache.c:2539          Dropping VG info

#cache/lvmcache.c:755           lvmcache has no info for vgname "#orphans_lvm1" with VGID #orphans_lvm1.

#cache/lvmcache.c:755           lvmcache has no info for vgname "#orphans_lvm1".

#cache/lvmcache.c:2086          lvmcache: Initialised VG #orphans_lvm1.

#cache/lvmcache.c:755           lvmcache has no info for vgname "#orphans_pool" with VGID #orphans_pool.

#cache/lvmcache.c:755           lvmcache has no info for vgname "#orphans_pool".

#cache/lvmcache.c:2086          lvmcache: Initialised VG #orphans_pool.

#cache/lvmcache.c:755           lvmcache has no info for vgname "#orphans_lvm2" with VGID #orphans_lvm2.

#cache/lvmcache.c:755           lvmcache has no info for vgname "#orphans_lvm2".

#cache/lvmcache.c:2086          lvmcache: Initialised VG #orphans_lvm2.

#cache/lvmetad.c:256           Sending lvmetad get_global_info

#label/label.c:833           Finding devices to scan

#device/dev-cache.c:1212          Creating list of system devices.

#device/dev-cache.c:723           Found dev 11:0 /dev/sr0 - new.

#device/dev-cache.c:763           Found dev 11:0 /dev/cdrom - new alias.

#device/dev-cache.c:763           Found dev 11:0 /dev/disk/by-id/ata-VMware_Virtual_SATA_CDRW_Drive_00000000000000000001 - new alias.

#device/dev-cache.c:763           Found dev 11:0 /dev/disk/by-path/pci-0000:02:01.0-ata-1.0 - new alias.

#device/dev-cache.c:723           Found dev 8:0 /dev/sda - new.

#device/dev-cache.c:763           Found dev 8:0 /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:0 - new alias.

#device/dev-cache.c:723           Found dev 8:1 /dev/sda1 - new.

#device/dev-cache.c:763           Found dev 8:1 /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:0-part1 - new alias.

#device/dev-cache.c:763           Found dev 8:1 /dev/disk/by-uuid/530982ac-2bdd-4310-857b-ff132f4a4d41 - new alias.

#device/dev-cache.c:723           Found dev 8:2 /dev/sda2 - new.

#device/dev-cache.c:763           Found dev 8:2 /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:0-part2 - new alias.

#device/dev-cache.c:763           Found dev 8:2 /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:0:0-part2 - new alias.

#device/dev-cache.c:723           Found dev 8:16 /dev/sdb - new.

#device/dev-cache.c:763           Found dev 8:16 /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:1:0 - new alias.

#device/dev-cache.c:723           Found dev 8:17 /dev/sdb1 - new.

#device/dev-cache.c:763           Found dev 8:17 /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:1:0-part1 - new alias.

#device/dev-cache.c:723           Found dev 8:32 /dev/sdc - new.

#device/dev-cache.c:763           Found dev 8:32 /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:2:0 - new alias.

#device/dev-cache.c:723           Found dev 8:33 /dev/sdc1 - new.

#device/dev-cache.c:763           Found dev 8:33 /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:2:0-part1 - new alias.

#device/dev-cache.c:763           Found dev 8:33 /dev/disk/by-uuid/0a81d8a5-6edd-40fb-ae16-14684f045ef7 - new alias.

#device/dev-cache.c:723           Found dev 8:34 /dev/sdc2 - new.

#device/dev-cache.c:763           Found dev 8:34 /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:2:0-part2 - new alias.

#device/dev-cache.c:763           Found dev 8:34 /dev/disk/by-uuid/52eb5eff-aa0c-435e-809c-5fbda45c6403 - new alias.

#device/dev-cache.c:723           Found dev 8:35 /dev/sdc3 - new.

#device/dev-cache.c:763           Found dev 8:35 /dev/disk/by-path/pci-0000:03:00.0-scsi-0:0:2:0-part3 - new alias.

#device/dev-cache.c:763           Found dev 8:35 /dev/disk/by-uuid/0dba35bd-de0b-4e59-b579-f9d3b18024d4 - new alias.

#device/dev-cache.c:723           Found dev 2:0 /dev/fd0 - new.

#filters/filter-type.c:27            /dev/fd0: Skipping: Unrecognised LVM device type 2

#device/dev-io.c:609           Opened /dev/sda RO O_DIRECT

#device/dev-io.c:359         /dev/sda: size is 1048576000 sectors

#device/dev-io.c:658           Closed /dev/sda

#filters/filter-partitioned.c:30            filter partitioned deferred /dev/sda

#filters/filter-md.c:99            filter md deferred /dev/sda

#filters/filter-persistent.c:346           filter caching good /dev/sda

#filters/filter-type.c:27            /dev/cdrom: Skipping: Unrecognised LVM device type 11

#device/dev-io.c:609           Opened /dev/sda1 RO O_DIRECT

#device/dev-io.c:359         /dev/sda1: size is 1024000 sectors

#device/dev-io.c:658           Closed /dev/sda1

#filters/filter-mpath.c:196           /dev/sda1: Device is a partition, using primary device sda for mpath component detection

#filters/filter-partitioned.c:30            filter partitioned deferred /dev/sda1

#filters/filter-md.c:99            filter md deferred /dev/sda1

#filters/filter-persistent.c:346           filter caching good /dev/sda1

#device/dev-io.c:609           Opened /dev/sda2 RO O_DIRECT

#device/dev-io.c:359         /dev/sda2: size is 1047549952 sectors

#device/dev-io.c:658           Closed /dev/sda2

#filters/filter-mpath.c:196           /dev/sda2: Device is a partition, using primary device sda for mpath component detection

#filters/filter-partitioned.c:30            filter partitioned deferred /dev/sda2

#filters/filter-md.c:99            filter md deferred /dev/sda2

#filters/filter-persistent.c:346           filter caching good /dev/sda2

#device/dev-io.c:609           Opened /dev/sdb RO O_DIRECT

#device/dev-io.c:359         /dev/sdb: size is 4294967296 sectors

#device/dev-io.c:658           Closed /dev/sdb

#filters/filter-partitioned.c:30            filter partitioned deferred /dev/sdb

#filters/filter-md.c:99            filter md deferred /dev/sdb

#filters/filter-persistent.c:346           filter caching good /dev/sdb

#device/dev-io.c:609           Opened /dev/sdb1 RO O_DIRECT

#device/dev-io.c:359         /dev/sdb1: size is 4294965247 sectors

#device/dev-io.c:658           Closed /dev/sdb1

#filters/filter-mpath.c:196           /dev/sdb1: Device is a partition, using primary device sdb for mpath component detection

#filters/filter-partitioned.c:30            filter partitioned deferred /dev/sdb1

#filters/filter-md.c:99            filter md deferred /dev/sdb1

#filters/filter-persistent.c:346           filter caching good /dev/sdb1

#device/dev-io.c:609           Opened /dev/sdc RO O_DIRECT

#device/dev-io.c:359         /dev/sdc: size is 20971520 sectors

#device/dev-io.c:658           Closed /dev/sdc

#filters/filter-partitioned.c:30            filter partitioned deferred /dev/sdc

#filters/filter-md.c:99            filter md deferred /dev/sdc

#filters/filter-persistent.c:346           filter caching good /dev/sdc

#device/dev-io.c:609           Opened /dev/sdc1 RO O_DIRECT

#device/dev-io.c:359         /dev/sdc1: size is 2097152 sectors

#device/dev-io.c:658           Closed /dev/sdc1

#filters/filter-mpath.c:196           /dev/sdc1: Device is a partition, using primary device sdc for mpath component detection

#filters/filter-partitioned.c:30            filter partitioned deferred /dev/sdc1

#filters/filter-md.c:99            filter md deferred /dev/sdc1

#filters/filter-persistent.c:346           filter caching good /dev/sdc1

#device/dev-io.c:609           Opened /dev/sdc2 RO O_DIRECT

#device/dev-io.c:359         /dev/sdc2: size is 2097152 sectors

#device/dev-io.c:658           Closed /dev/sdc2

#filters/filter-mpath.c:196           /dev/sdc2: Device is a partition, using primary device sdc for mpath component detection

#filters/filter-partitioned.c:30            filter partitioned deferred /dev/sdc2

#filters/filter-md.c:99            filter md deferred /dev/sdc2

#filters/filter-persistent.c:346           filter caching good /dev/sdc2

#device/dev-io.c:609           Opened /dev/sdc3 RO O_DIRECT

#device/dev-io.c:359         /dev/sdc3: size is 16775168 sectors

#device/dev-io.c:658           Closed /dev/sdc3

#filters/filter-mpath.c:196           /dev/sdc3: Device is a partition, using primary device sdc for mpath component detection

#filters/filter-partitioned.c:30            filter partitioned deferred /dev/sdc3

#filters/filter-md.c:99            filter md deferred /dev/sdc3

#filters/filter-persistent.c:346           filter caching good /dev/sdc3

#label/label.c:869           Found 9 devices to scan

#label/label.c:628           Scanning 9 devices for VG info

#label/label.c:664           Scanning submitted 9 reads

#label/label.c:683           Processing data from device /dev/sda 8:0 fd 4 block 0x56207e2a8870

#label/label.c:407         /dev/sda: No lvm label detected

#label/label.c:412           <backtrace>

#label/label.c:683           Processing data from device /dev/sda1 8:1 fd 6 block 0x56207e2a88c0

#label/label.c:407         /dev/sda1: No lvm label detected

#label/label.c:412           <backtrace>

#label/label.c:683           Processing data from device /dev/sda2 8:2 fd 7 block 0x56207e2a8910

#label/label.c:407         /dev/sda2: No lvm label detected

#label/label.c:412           <backtrace>

#label/label.c:683           Processing data from device /dev/sdb 8:16 fd 8 block 0x56207e2a8960

#label/label.c:407         /dev/sdb: No lvm label detected

#label/label.c:412           <backtrace>

#label/label.c:683           Processing data from device /dev/sdb1 8:17 fd 9 block 0x56207e2a89b0

#label/label.c:407         /dev/sdb1: No lvm label detected

#label/label.c:412           <backtrace>

#label/label.c:683           Processing data from device /dev/sdc 8:32 fd 10 block 0x56207e2a8a00

#label/label.c:407         /dev/sdc: No lvm label detected

#label/label.c:412           <backtrace>

#label/label.c:683           Processing data from device /dev/sdc1 8:33 fd 11 block 0x56207e2a8a50

#label/label.c:407         /dev/sdc1: No lvm label detected

#label/label.c:412           <backtrace>

#label/label.c:683           Processing data from device /dev/sdc2 8:34 fd 12 block 0x56207e2a8aa0

#label/label.c:407         /dev/sdc2: No lvm label detected

#label/label.c:412           <backtrace>

#label/label.c:683           Processing data from device /dev/sdc3 8:35 fd 13 block 0x56207e2a8af0

#label/label.c:407         /dev/sdc3: No lvm label detected

#label/label.c:412           <backtrace>

#label/label.c:763           Scanned devices: read errors 0 process errors 0 failed 0

#cache/lvmetad.c:2374      Scanning all devices to update lvmetad.

#cache/lvmetad.c:558           Sending lvmetad token_update update in progress

#cache/lvmetad.c:2412          Telling lvmetad to clear its cache

#filters/filter-type.c:27            /dev/fd0: Skipping: Unrecognised LVM device type 2

#device/dev-io.c:336         /dev/sda: using cached size 1048576000 sectors

#filters/filter-partitioned.c:37            /dev/sda: Skipping: Partition table signature found

#filters/filter-type.c:27            /dev/cdrom: Skipping: Unrecognised LVM device type 11

#device/dev-io.c:336         /dev/sda1: using cached size 1024000 sectors

#filters/filter-mpath.c:196           /dev/sda1: Device is a partition, using primary device sda for mpath component detection

#device/dev-io.c:336         /dev/sda1: using cached size 1024000 sectors

#cache/lvmetad.c:2247          Scan metadata from dev /dev/sda1

#cache/lvmetad.c:2260      No PV info found on /dev/sda1 for PVID .

#cache/lvmetad.c:1746          Telling lvmetad to forget any PV on /dev/sda1

#cache/lvmetad.c:784         Request to drop PV /dev/sda1 in lvmetad did not find any matching object.

#device/dev-io.c:336         /dev/sda2: using cached size 1047549952 sectors

#filters/filter-mpath.c:196           /dev/sda2: Device is a partition, using primary device sda for mpath component detection

#device/dev-io.c:336         /dev/sda2: using cached size 1047549952 sectors

#cache/lvmetad.c:2247          Scan metadata from dev /dev/sda2

#cache/lvmetad.c:2260      No PV info found on /dev/sda2 for PVID .

#cache/lvmetad.c:1746          Telling lvmetad to forget any PV on /dev/sda2

#cache/lvmetad.c:784         Request to drop PV /dev/sda2 in lvmetad did not find any matching object.

#device/dev-io.c:336         /dev/sdb: using cached size 4294967296 sectors

#filters/filter-partitioned.c:37            /dev/sdb: Skipping: Partition table signature found

#device/dev-io.c:336         /dev/sdb1: using cached size 4294965247 sectors

#filters/filter-mpath.c:196           /dev/sdb1: Device is a partition, using primary device sdb for mpath component detection

#device/dev-io.c:336         /dev/sdb1: using cached size 4294965247 sectors

#cache/lvmetad.c:2247          Scan metadata from dev /dev/sdb1

#cache/lvmetad.c:2260      No PV info found on /dev/sdb1 for PVID .

#cache/lvmetad.c:1746          Telling lvmetad to forget any PV on /dev/sdb1

#cache/lvmetad.c:784         Request to drop PV /dev/sdb1 in lvmetad did not find any matching object.

#device/dev-io.c:336         /dev/sdc: using cached size 20971520 sectors

#filters/filter-partitioned.c:37            /dev/sdc: Skipping: Partition table signature found

#device/dev-io.c:336         /dev/sdc1: using cached size 2097152 sectors

#filters/filter-mpath.c:196           /dev/sdc1: Device is a partition, using primary device sdc for mpath component detection

#device/dev-io.c:336         /dev/sdc1: using cached size 2097152 sectors

#cache/lvmetad.c:2247          Scan metadata from dev /dev/sdc1

#cache/lvmetad.c:2260      No PV info found on /dev/sdc1 for PVID .

#cache/lvmetad.c:1746          Telling lvmetad to forget any PV on /dev/sdc1

#cache/lvmetad.c:784         Request to drop PV /dev/sdc1 in lvmetad did not find any matching object.

#device/dev-io.c:336         /dev/sdc2: using cached size 2097152 sectors

#filters/filter-mpath.c:196           /dev/sdc2: Device is a partition, using primary device sdc for mpath component detection

#device/dev-io.c:336         /dev/sdc2: using cached size 2097152 sectors

#cache/lvmetad.c:2247          Scan metadata from dev /dev/sdc2

#cache/lvmetad.c:2260      No PV info found on /dev/sdc2 for PVID .

#cache/lvmetad.c:1746          Telling lvmetad to forget any PV on /dev/sdc2

#cache/lvmetad.c:784         Request to drop PV /dev/sdc2 in lvmetad did not find any matching object.

#device/dev-io.c:336         /dev/sdc3: using cached size 16775168 sectors

#filters/filter-mpath.c:196           /dev/sdc3: Device is a partition, using primary device sdc for mpath component detection

#device/dev-io.c:336         /dev/sdc3: using cached size 16775168 sectors

#cache/lvmetad.c:2247          Scan metadata from dev /dev/sdc3

#cache/lvmetad.c:2260      No PV info found on /dev/sdc3 for PVID .

#cache/lvmetad.c:1746          Telling lvmetad to forget any PV on /dev/sdc3

#cache/lvmetad.c:784         Request to drop PV /dev/sdc3 in lvmetad did not find any matching object.

#cache/lvmetad.c:558           Sending lvmetad token_update filter:3239235440

#vgscan.c:118     Reading volume groups from cache.

#toollib.c:2246          Processing each VG

#toollib.c:2294        Obtaining the complete list of VGs to process

#cache/lvmetad.c:1447          Asking lvmetad for complete list of known VG ids/names

#toollib.c:2314      No volume groups found.

#misc/lvm-flock.c:70          Unlocking /run/lock/lvm/P_global

#misc/lvm-flock.c:47            _undo_flock /run/lock/lvm/P_global

#cache/lvmcache.c:2539          Dropping VG info

#cache/lvmcache.c:755           lvmcache has no info for vgname "#orphans_lvm1" with VGID #orphans_lvm1.

#cache/lvmcache.c:755           lvmcache has no info for vgname "#orphans_lvm1".

#cache/lvmcache.c:2086          lvmcache: Initialised VG #orphans_lvm1.

#cache/lvmcache.c:755           lvmcache has no info for vgname "#orphans_pool" with VGID #orphans_pool.

#cache/lvmcache.c:755           lvmcache has no info for vgname "#orphans_pool".

#cache/lvmcache.c:2086          lvmcache: Initialised VG #orphans_pool.

#cache/lvmcache.c:755           lvmcache has no info for vgname "#orphans_lvm2" with VGID #orphans_lvm2.

#cache/lvmcache.c:755           lvmcache has no info for vgname "#orphans_lvm2".

#cache/lvmcache.c:2086          lvmcache: Initialised VG #orphans_lvm2.

#lvmcmdline.c:3050          Completed: vgscan -vvvv

At this stage we suspect it was a side effect of a process ran by the storage teams.
They ran a process that would ‘zero unlinked blocks‘. So our guess is that process caused some destruction.