paulgorman.org/technical

Linux Booting

(Updated 2019)

  1. Machine firmware (UEFI) holds a menu of next-stage boot loaders (like GRUB) and can read them into memory from a boot partition (a FAT32 ESP partition).
  2. The second-stage boot loader (GRUB2) knows how to read its configuration and system files (GRUB can read ext4 partitions).
  3. GRUB loads the kernel.
  4. The kernel starts init.

BIOS booting

BIOS booting relies on a boot sector known as a master boot record (MBR) that the BIOS reads into RAM. BIOS expects to find this boot record at the start of the disk, wedged in before the start of the first partition. The MBR may also contain information about the disk’s partitions.

The BIOS boot record runs a second-stage boot loader (e.g. GRUB), and that second boot loader starts the OS. This chain boot loading is necessary because a modern boot loader, like GRUB, is too large for BIOS to load directly.

BIOS only knows about the MBR; it doesn’t know about the OS. The OS, once loaded, knows little about the BIOS.

BIOS is a de-facto standard of IBM PC-compatible lineage. BIOS is going away. The PC world is moving to the type of more advanced firmware long used on Mac’s and *nix workstations.

On linux, grub-install can write a master boot record.

BIOS boot failure

If BIOS fails to find a bootable device, on most devices it throws an error message like “No bootable device found” and/or automatically enters the BIOS config utility.

UEFI

UEFI (unified extensible firmware interface) is a new, formal standard for PC firmware.

UEFI doesn’t need to cram a boot record into the gap between the start of the disk and the first partition.

UEFI boot devices start with an EFI system partition (ESP). This partition is relatively large, and can contain a much more robust boot loader than the BIOS first-stage boot loader. The EFI partition should be formatted as FAT32.

ESP partitions may be as small as 100–200 MB, but 550 MB is a safer value.

Each boot loader resides in a subdirectory of EFI/ on the ESP. Convention names these subdirectories after the OS (e.g., EFI/debian or EFI/redhat).

GUID partition table (GPT) arrived along with UEFI. GPT improves on the MBR partition scheme with an unrestricted number of primary and extended partitions, the ability to provide a unique GUID (UUID) for physical disks and partitions, much larger maximum partition sizes (upt to 2 ZiB), and checksums for partition tables.

Linux manipulates the UEFI boot menu with efibootmgr.

$  efibootmgr -v
BootCurrent: 0000
Timeout: 1 seconds
BootOrder: 0000,0005,0003,0004
Boot0000* debian        HD(1,GPT,e0ab63df-2c5c-48f3-b48c-f5806f147619,0x800,0xee000)/File(\EFI\debian\grubx64.efi)
Boot0003  UEFI: Built-in EFI Shell      VenMedia(5023b95c-db26-429b-a648-bd47664c8012)..BO
Boot0004  Network Card  BBS(Network,,0x0)..GO..NO........q.I.B.A. .G.E. .S.l.o.t. .0.7.0.0. .v.1.4.0.4........BO
Boot0005* SanDisk SDSSDH31000G  BBS(HD,,0x0)..BO
Boot0007  Network Card  BBS(Network,,0x0)..GO..NO........q.I.B.A. .G.E. .S.l.o.t. .0.7.0.0. .v.1.4.0.4........BO

$  sudo blkid | grep e0ab63df
/dev/sdc1: UUID="FDEE-B7EF" TYPE="vfat" PARTUUID="e0ab63df-2c5c-48f3-b48c-f5806f147619"

$  df -h | grep efi
/dev/sdc1                   476M  160K  475M   1% /boot/efi

$  sudo ls -l /boot/efi/EFI/debian
total 148
-rwx------ 1 root root 148992 Nov 13 12:19 grubx64.efi

$  sudo file /boot/efi/EFI/debian/grubx64.efi
/boot/efi/EFI/debian/grubx64.efi: PE32+ executable (EFI application) x86-64 (stripped to external PDB), for MS Windows

UEFI boot failure

If UEFI fails to find a bootable device, it should drop the user into a UEFI shell. Available options in UEFI shells vary, but generally include manually selecting a boot device.

GRUB

GRUB is the linux boot loader.

Everything uses GRUB 2 now, rather than the old version of GRUB (i.e. GRUB Legacy). The two differ significantly in configuration. The following applies only to GRUB 2.

GRUB config

The GRUB config file is /etc/default/grub. Any GRUB changes made by hand will almost certainly be made in this file.

Running update-grub reads /etc/default/grub and a number of other files (notably the shell scripts in ‘/etc/grub.d/’) to generate the file /boot/grub/grub.cfg. Never edit ‘grub.cfg’ manually, because future runs of update-grub overwrite such changes.

(/usr/sbin/update-grub is a little shell script that wraps grub-mkconfig. Unfortunately, much of the grub documentation is in ‘info’ rather than man pages. See info grub.)

Then, run grub-install --recheck --bootloader-id foo /dev/sdX. This generates the grub boot image, and copies files to /boot/grub/. It also, on a BIOS system, writes the MRB; on a UEFI system, it adds an entry to the firmware boot menu.

GRUB boot failure

GRUB has a command shell, and it will dump us in its shell if boot fails. The prompt may give us a clue about the problem.

grub> set pager=1
grub> ls
(hd0) (hd0,gpt2) (hd0,gpt1) (vg0-lv0)
grub> ls (hd0,1)/
lost+found/ bin/ boot/ cdrom/ dev/ etc/ home/  lib/
lib64/ media/ mnt/ opt/ proc/ root/ run/ sbin/
[...snip...]
grub> set root=(hd0,1)
grub> linux /boot/vmlinuz-3.16.0-4-amd root=/dev/sda1
grub> initrd /boot/initrd.img-3.16.0-4-amd
grub> boot

…and we should boot.

If we’re at the rescue prompt, we’ll need to (minimally) load GRUB’s normal.mod and linux.mod from the boot partition. GRUB2 includes hundreds of modules, including: iso9660, ext2, part_gpt, part_msdos, mdraid1x, and lvm. If GRUB can’t access the partition holding its modules, boot from rescue media.

grub rescue> insmod part_gpt
grub rescue> insmod diskfilter
grub rescue> insmod mdraid1x
grub rescue> insmod lvm
grub rescue> insmod ext2
grub rescue> ls
(hd0) (hd0,gpt2) (hd0,gpt1) (hd1) (hd1,gpt1)
grub rescue> ls (hd0,1)/
lost+found/ bin/ boot/ cdrom/ dev/ etc/ home/  lib/
lib64/ media/ mnt/ opt/ proc/ root/ run/ sbin/
[...snip...]
grub rescue> set
grub rescue> set prefix=(hd0,gpt1)/boot/grub
grub rescue> set root=(hd0,gpt1)
grub rescue> ls /
grub rescue> insmod normal
grub rescue> normal
grub rescue> insmod linux
grub rescue> linux /boot/vmlinuz-4.9.0-7-amd64 root=/dev/sda1
grub rescue> initrd /boot/initrd.img-4.9.0-7-amd64
grub rescue> boot

Once booted, run /usr/sbin/update-grub to make permanent repairs.

And (or first?) /usr/sbin/grub-install?

On UEFI systems, grub-install tries to do three things: generate a grub image (e.g. grubx64.efi), install the grub image to “/boot/efi/EFI/debian/”, and it attempts to add an appropriate entry to the UEFI boot menu.

If the system boots with BIOS, something like grub-install /dev/sda will fix the MBR.

What does grub-install do?

grub-install does several things (what exactly may vary by platform).

First, grub-install adds an EFI boot menu entry to the firware pointing a label like “debian” or “fedora” (customizable with the --bootloader-id flag) to PARTUUID of the ESP partition. (We must mount the ESP partition for this, and GRUB assumes the default mount point /boot/efi.)

$  efibootmgr -v
BootCurrent: 0001
Timeout: 1 seconds
BootOrder: 0001,0000,0002,0003
Boot0000* linux HD(1,MBR,0x72,0x800,0xec800)/File(\EFI\LINUX\GRUBX64.EFI)
Boot0001* debian        HD(1,GPT,abed03c7-fb75-44f2-969d-8e281f3b1937,0x800,0x100000)/File(\EFI\DEBIAN\GRUBX64.EFI)

Second, grub-install adds the boot loader where the firmware can read it. It finds the ESP partition on the device specified, creates a directory there named after the EFI boot menu entry, and puts a grub executable in that directory. Grub doesn’t write any configuration there — it just a standard executable that doesn’t change (except for updated versions of GRUB or whatever).

--- mizzen /boot %  ls -l /boot/efi/EFI/debian/grubx64.efi 
-rwxr-xr-x 1 root root 146432 Mar 26 10:30 /boot/efi/EFI/debian/grubx64.efi
--- mizzen /boot %  file /boot/efi/EFI/debian/grubx64.efi
/boot/efi/EFI/debian/grubx64.efi: PE32+ executable (EFI application) x86-64 (stripped to external PDB), for MS Windows

Third, grub-install copies images and modules into /boot (mostly from /usr/lib/grub/cpu-platform, which GRUB calls the “image directory”).

Other stuff?

What does update-grub do?

On Debian 9, update-grub is just this simple shell script to call grub-mkconfig:

#!/bin/sh
set -e
exec grub-mkconfig -o /boot/grub/grub.cfg "$@"

This generates the configuration file /boot/grub/grub.cfg using templates from /etc/grub.d and settings from /etc/default/grub.

How do we remove an unwanted entry from the EFI boot menu?

#  efibootmgr -B -b 0001

Software RAID

With software RAID, where you might have a two-disk mirror for example, you may want to install GRUB to both disks like grub-install --recheck --bootloader-id linux-sdb /dev/sdb and grub-install --recheck --bootloader-id linux-sda /dev/sda.

N.B. — if we make multiple devices bootable (sda and sdb), the device against which we run grub-install last/finally will become the first/default boot device.