Ytdyklly

This morning a user came up to me to let me know that email being sent from the outside world was not showing up in his inbox for his work email. We host the email server internally (not my idea). I tested by sending an email from my gmail account to my work email, and as expected the email never arrives.

I go to reboot the mail server (which is running Ubuntu 10.10) and right before the point where the GRUB boot loader is started, the server would reboot back to a POST/BIOS "Checking memory" message as if I had hit the reset button.

I went and burned a copy of 10.10 and tried to run the system rescue to see if I was able to fix the boot issue, but I was not able to fix it. I then burned a copy of Ubuntu 18.04 to see if there were any additional tools that might allow me to dig a little deeper to find out what was going on.

I checked /etc/fstab on the drive where Ubuntu is currently installed, and here is what I found

# /etc/fstab: static file system information.

#

# Use 'blkid -o value -s UUID' to print the universally unique identifier

# for a device; this may be used with UUID= as a more robust way to name

# devices that works even if disks are added and removed. See fstab(5).

#

# <file system> <mount point>   <type>  <options>       <dump>  <pass>

proc            /proc           proc    nodev,noexec,nosuid 0       0

/dev/mapper/mailadmin-root /               ext4    errors=remount-ro 0       1

# /boot was on /dev/sda2 during installation

UUID=e8c63c19-db49-4013-bec8-ef8479fb52f1 /boot           ext2    defaults        0       2

/dev/mapper/mailadmin-swap_1 none            swap    sw              0       0

10.6.9.5:/volume1/Backups/Zimbra/TarBackups /zmbackup nfs,size=8192,wsize=8192,timeo=14,intr

Notice the comment above the UUID for the /boot partition? I did not enter that comment, it was left by the previous IT manager a number of years ago. I modified the /etc/fstab file to point at /dev/sda2 (which I have confirmed is the actual location where the grub config is located) instead of the UUID, rebooted - but still the server goes into a reboot loop right as it is about to load grub.

After doing some googling, I found a script called bootinfoscript.sh that outputs a text file with a bunch of information related to the hard disks that are detected in my system. Here is the output of bootinfoscript.sh

                  Boot Info Script 0.61      [1 April 2012]





============================= Boot Info Summary: ===============================



 => Grub2 (v1.97-1.98) is installed in the MBR of /dev/sda and looks at sector 

    2048 of the same hard drive for core.img. core.img is at this location and 

    looks in partition 2 for /grub.

 => Syslinux MBR (4.04 and higher) is installed in the MBR of /dev/sdb.



sda1: __________________________________________________________________________



    File system:       BIOS Boot partition

    Boot sector type:  Grub2's core.img

    Boot sector info: 



sda2: __________________________________________________________________________



    File system:       ext2

    Boot sector type:  -

    Boot sector info: 

    Operating System:  

    Boot files:        /grub/grub.cfg /grub/core.img



sda3: __________________________________________________________________________



    File system:       LVM2_member

    Boot sector type:  -

    Boot sector info: 



sdb1: __________________________________________________________________________



    File system:       vfat

    Boot sector type:  SYSLINUX 6.03

    Boot sector info:  Syslinux looks at sector 32784 of /dev/sdb1 for its 

                       second stage. The integrity check of Syslinux failed. 

                       No errors found in the Boot Parameter Block.

    Operating System:  

    Boot files:        /boot/grub/grub.cfg /syslinux.cfg 

                       /efi/BOOT/grubx64.efi /ldlinux.sys



mailadmin-root': _______________________________________________________________



    File system:       

    Boot sector type:  Unknown

    Boot sector info: 

    Mounting failed:   mount: /tmp/BootInfo-TMXYqs5T/LVM/mailadmin-root': unknown filesystem type ''.



mailadmin-swap_1': _____________________________________________________________



    File system:       

    Boot sector type:  Unknown

    Boot sector info: 

    Mounting failed:   mount: /tmp/BootInfo-TMXYqs5T/LVM/mailadmin-root': unknown filesystem type ''.

mount: /tmp/BootInfo-TMXYqs5T/LVM/mailadmin-swap_1': unknown filesystem type ''.



============================ Drive/Partition Info: =============================



Drive: sda _____________________________________________________________________

Disk /dev/sda: 10.9 TiB, 11999064883200 bytes, 23435673600 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disklabel type: gpt



Partition  Boot  Start Sector    End Sector  # of Sectors  Id System



/dev/sda1                   1 4,294,967,295 4,294,967,295  ee GPT





GUID Partition Table detected.



Partition    Start Sector    End Sector  # of Sectors System

/dev/sda1           2,048         4,095         2,048 BIOS Boot partition

/dev/sda2           4,096       503,807       499,712 Data partition (Windows/Linux)

/dev/sda3         503,808 7,329,544,191 7,329,040,384 Logical Volume Manager (LVM) partition (Linux)



Drive: sdb _____________________________________________________________________

Disk /dev/sdb: 14.4 GiB, 15500574720 bytes, 30274560 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disklabel type: dos



Partition  Boot  Start Sector    End Sector  # of Sectors  Id System



/dev/sdb1    *          2,048    30,274,559    30,272,512   c W95 FAT32 (LBA)





"blkid" output: ________________________________________________________________



Device           UUID                                   TYPE       LABEL



/dev/loop0                                              squashfs   

/dev/loop1                                              squashfs   

/dev/loop2                                              squashfs   

/dev/loop3                                              squashfs   

/dev/loop4                                              squashfs   

/dev/loop5                                              squashfs   

/dev/loop6                                              squashfs   

/dev/loop7                                              squashfs   

/dev/mapper/mailadmin-root 3e2b9e3a-7190-4240-804e-f1761fa96ade   ext4       

/dev/mapper/mailadmin-swap_1 2b7f558a-2abe-4981-879e-d79f7ed3fb01   swap       

/dev/sda1                                                          

/dev/sda2        e8c63c19-db49-4013-bec8-ef8479fb52f1   ext2       

/dev/sda3        WFe0Xa-LXJa-BzOr-Oyo9-kC5E-3W7p-qWA5Uy LVM2_member 

/dev/sdb1        6045-3252                              vfat       UBUNTU 18_0



========================= "ls -R /dev/mapper/" output: =========================



/dev/mapper:

control

mailadmin-root

mailadmin-swap_1



================================ Mount points: =================================



Device           Mount_Point              Type       Options



/dev/loop0       /rofs                    squashfs   (ro,noatime)

/dev/sdb1        /cdrom                   vfat       (ro,noatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)





============================= sda2/grub/grub.cfg: ==============================



--------------------------------------------------------------------------------

#

# DO NOT EDIT THIS FILE

#

# It is automatically generated by /usr/sbin/grub-mkconfig using templates

# from /etc/grub.d and settings from /etc/default/grub

#



### BEGIN /etc/grub.d/00_header ###

if [ -s $prefix/grubenv ]; then

  load_env

fi

set default="0"

if [ ${prev_saved_entry} ]; then

  set saved_entry=${prev_saved_entry}

  save_env saved_entry

  set prev_saved_entry=

  save_env prev_saved_entry

  set boot_once=true

fi



function savedefault {

  if [ -z ${boot_once} ]; then

    saved_entry=${chosen}

    save_env saved_entry

  fi

}



function recordfail {

  set recordfail=1

  if [ -n ${have_grubenv} ]; then if [ -z ${boot_once} ]; then save_env recordfail; fi; fi

}

insmod ext2

set root='(hd0,2)'

search --no-floppy --fs-uuid --set e8c63c19-db49-4013-bec8-ef8479fb52f1

set locale_dir=($root)/grub/locale

set lang=en

insmod gettext

if [ ${recordfail} = 1 ]; then

  set timeout=-1

else

  set timeout=10

fi

### END /etc/grub.d/00_header ###



### BEGIN /etc/grub.d/05_debian_theme ###

set menu_color_normal=white/black

set menu_color_highlight=black/light-gray

### END /etc/grub.d/05_debian_theme ###



### BEGIN /etc/grub.d/10_linux ###

### END /etc/grub.d/10_linux ###



### BEGIN /etc/grub.d/20_memtest86+ ###

menuentry "Memory test (memtest86+)" {

    insmod ext2

    set root='(hd0,2)'

    search --no-floppy --fs-uuid --set e8c63c19-db49-4013-bec8-ef8479fb52f1

    linux16 /memtest86+.bin

}

menuentry "Memory test (memtest86+, serial console 115200)" {

    insmod ext2

    set root='(hd0,2)'

    search --no-floppy --fs-uuid --set e8c63c19-db49-4013-bec8-ef8479fb52f1

    linux16 /memtest86+.bin console=ttyS0,115200n8

}

### END /etc/grub.d/20_memtest86+ ###



### BEGIN /etc/grub.d/30_os-prober ###

if [ ${timeout} != -1 ]; then

  if keystatus; then

    if keystatus --shift; then

      set timeout=-1

    else

      set timeout=0

    fi

  else

    if sleep --interruptible 3 ; then

      set timeout=0

    fi

  fi

fi

### END /etc/grub.d/30_os-prober ###



### BEGIN /etc/grub.d/40_custom ###

# This file provides an easy way to add custom menu entries.  Simply type the

# menu entries you want to add after this comment.  Be careful not to change

# the 'exec tail' line above.

### END /etc/grub.d/40_custom ###

--------------------------------------------------------------------------------



=================== sda2: Location of files loaded by Grub: ====================



           GiB - GB             File                                 Fragment(s)





=========================== sdb1/boot/grub/grub.cfg: ===========================



--------------------------------------------------------------------------------



if loadfont /boot/grub/font.pf2 ; then

    set gfxmode=auto

    insmod efi_gop

    insmod efi_uga

    insmod gfxterm

    terminal_output gfxterm

fi



set menu_color_normal=white/black

set menu_color_highlight=black/light-gray



set timeout=5

menuentry "Try Ubuntu without installing" {

    set gfxpayload=keep

    linux   /casper/vmlinuz  file=/cdrom/preseed/ubuntu.seed boot=casper quiet splash ---

    initrd  /casper/initrd.lz

}

menuentry "Install Ubuntu" {

    set gfxpayload=keep

    linux   /casper/vmlinuz  file=/cdrom/preseed/ubuntu.seed boot=casper only-ubiquity quiet splash ---

    initrd  /casper/initrd.lz

}

menuentry "OEM install (for manufacturers)" {

    set gfxpayload=keep

    linux   /casper/vmlinuz  file=/cdrom/preseed/ubuntu.seed boot=casper only-ubiquity quiet splash oem-config/enable=true ---

    initrd  /casper/initrd.lz

}

menuentry "Check disc for defects" {

    set gfxpayload=keep

    linux   /casper/vmlinuz  boot=casper integrity-check quiet splash ---

    initrd  /casper/initrd.lz

}

--------------------------------------------------------------------------------



============================== sdb1/syslinux.cfg: ==============================



--------------------------------------------------------------------------------

DEFAULT loadconfig



LABEL loadconfig

  CONFIG /isolinux/isolinux.cfg

  APPEND /isolinux/

--------------------------------------------------------------------------------



=================== sdb1: Location of files loaded by Grub: ====================



           GiB - GB             File                                 Fragment(s)





================= sdb1: Location of files loaded by Syslinux: ==================



           GiB - GB             File                                 Fragment(s)





======================== Unknown MBRs/Boot Sectors/etc: ========================



Unknown BootLoader on mailadmin-root'





Unknown BootLoader on mailadmin-swap_1'







=============================== StdErr Messages: ===============================



cat: /tmp/BootInfo-TMXYqs5T/Tmp_Log: No such file or directory

cat: /tmp/BootInfo-TMXYqs5T/Tmp_Log: No such file or directory

cat: /tmp/BootInfo-TMXYqs5T/Tmp_Log: No such file or directory

  Failed to find logical volume "mailadmin/root'"

  Failed to find logical volume "mailadmin/root'"

  Failed to find logical volume "mailadmin/root'"

hexdump: /dev/mapper/mailadmin-root': No such file or directory

hexdump: /dev/mapper/mailadmin-root': No such file or directory

  Failed to find logical volume "mailadmin/swap_1'"

  Failed to find logical volume "mailadmin/swap_1'"

  Failed to find logical volume "mailadmin/swap_1'"

hexdump: /dev/mapper/mailadmin-swap_1': No such file or directory

hexdump: /dev/mapper/mailadmin-swap_1': No such file or directory

Just for some background info, /dev/sda is a 12TB partition where the mail server software is installed. /dev/sdb is a 16GB USB thumbdrive that I installed Ubuntu onto.

I mounted the 12TB (which is actually a RAID array) /dev/sda1 drive and when I browse to the /boot folder on the mounted drive, it appears as if the /boot directory is completely empty!

Anyone have any idea what is going on here? Is there some way to fix the grub boot loader on this old server?

Please let me know if you need any additional details.

We were able to resolve this issue by installing the same version of Ubuntu on another server and copying over the contents of /boot from the newly installed server and modifying the /etc/fstab to point to the correct partitions in /dev.