Bootloader/Firmware Testing and Recovery¶

What is recovery mode?¶

Recovery mode is a hardware feature which allows boards to be unbricked after a serious system error or a failed update of the bootloader, kernel or operating system. Recovery mode is not necessarily the same as the methods used to deploy firmware to a bare metal device during manufacture.

• Recovery mode needs to implement a way to write a new bootloader build which must be fully usable even if the bootloader is absent or invalid, i.e. without any attempt to execute the current bootloader.
• Recovery mode must be able to write a new bootloader to the board directly after applying power to the device and then allow the board to execute the new bootloader to continue operation upon leaving recovery mode.

Bootloader recovery criteria¶

Juno, HiKey 6220 and X15 GPEVM are the only boards capable of supporting bootloader testing and recovery out of the hundreds of boards which have been integrated into LAVA over the years.

The difficulties cover a number of areas:

Comparison of barriers to bootloader testing and recovery¶

LAVA has a low tolerance for failure so that issues can be detected early, avoiding a false positive where a test passes because an error was not detected. Developers will need reliable result data from the automated bootloader testing and recovery to avoid wasting time on spurious false negative reports which mis-report a failure as being caused by the code when it was due to the automation infrastructure. The hardest problems to solve are intermittent faults in a complex stack Every requirement which is not fully supported adds complexity to the final automation and risks increasing the failure rate.

Problems with bootloader execution¶

Some boards can support replacing the bootloader within a test job but still rely on the current bootloader to execute to allow the board to boot to a stage where that functionality is available. For example, it is possible to write new U-Boot files to many U-Boot devices once booted into a ramdisk from within a test job. Alternatively, fastboot devices support writing a new fastboot binary at the same stage as writing a new kernel or system image. This does not qualify as recovery mode because those same files must be executed before the board can boot to the stage where new files can be written.

Two serious failures are possible:

• the new files fail to execute in the ways required by the automation, requiring manual intervention
• the new files contain a failure not executed by the automation, potentially requiring a product recall

Therefore, the bootloader files must undergo manual testing before being made available and this dramatically limits the opportunity for CI on those bootloader files.

For example, fastboot mode is not recovery mode for two reasons:

• the fastboot binary can be overwritten (by fastboot) within normal operation of a test job and therefore rendered corrupt or missing,
• every fastboot binary needs to be executed to write a replacement of itself.

Similarly, ADB cannot provide recovery mode support because it relies on the board being booted into userspace and userspace must be configured to allow access.

Problems with bootloader storage¶

It remains possible for any test job to corrupt any filesystem to which the test job kernel has access. Boards which host the bootloader files on a partition of the primary storage media are particularly vulnerable to bricking through this method. Many U-Boot devices suffer from this problem and LAVA uses TFTP on most of these devices to limit the number of failures.

Problems with full system images¶

Some boards host the bootloader files on the same storage medium as the kernel and/or operating system and then expect to write a full system image to that storage. The best way to manage such boards is to use a BMC. If that is not available, ensure that the bootloader build used in the system images is never changed until it has been tested and the update has been approved by the lab admins. i.e. treat the bootloader part of the system image build in exactly the same way as a firmware or bootloader update on any other board which lacks bootloader testing and recovery support.

It is not recommended to allow full system images to change more than one component of the image in a single test job.

HiKey 6220¶

This board provides jumpers which can be bridged and connected to relays to change the boot order when power is applied. Custom software is needed to select recovery mode boot by setting the correct jumpers. Once in recovery mode, the board offers a USB TTY device which can be used to deploy the firmware using custom software from the manufacturer. Once the firmware is transferred, the board comes up in fastboot mode and the rest of the boot files can be transferred over the USB OTG port.

The HiKey 6220 uses a combined deployment of UEFI firmware and a fastboot client which is then executed to deploy the boot and system images.

If the firmware fails, the board can be switched back to recovery mode without needing to interact with the device and a known working build can be deployed instead (using a health check or a second deploy stage of the test job itself). By using the control of the jumpers, the board can be forced into recovery mode without any need to interact with the bootloader.

An example test job for the hikey 6220 using recovery mode includes three deploy actions: recovery, fastboot, and operating system (OpenEmbedded or AOSP). This is to ensure that the firmware is tested in a wide range of modes after being deployed. (Specifically, the test ensures that the firmware continues to allow changes to the partition tables without affecting serial numbers or other device behavior.) This example is used as a health check, deploying a known good build of the firmware, so can be used to recover from a broken build.

1
2
3
4
5
6

- deploy:
    timeout:
      minutes: 10
    to: recovery
    namespace: recovery
    connection: lxc

1
2
3
4
5

- deploy:
    timeout:
      minutes: 15
    to: fastboot
    namespace: hikey

1
2
3
4
5
6
7

- deploy:
    timeout:
      minutes: 45
    to: fastboot
    # OE deployment
    namespace: hikey
    connection: lxc

Juno¶

The Juno board provides a BMC which allows LAVA to deploy a new build of bootloader without interacting with the current bootloader. Juno also supports testing two different deployments, U-Boot bootloader and UEFI firmware.

The BMC is accessible before the bootloader is executed, so if the test job determines that the bootloader has failed, a known working build can be deployed to recover the board.

1
2
3
4
5

- deploy:
    namespace: recovery
    timeout:
      minutes: 10
    to: vemsd

1
2
3
4
5

- boot:
    namespace: debian
    connection-namespace: recovery
    method: u-boot
    commands: norflash

X15 GPEVM¶

Most of the TI boards provide means to load bootloader using serial interface. This is the case for X15 GPEVM. As compared to other X15 family boards, GPEVM is wired to provide jumpers to switch boot mode. The only HW modification required is populating J5 in order to avoid board powering down before bootloader is loaded to memory and started.

In order to start board in recovery mode, the following HW settings should be set:

1. Jumpers J3, J4 and J6 should have 1-2 link closed
2. Jumper J5 should be populated and closed

Example LAVA recovery test job loads ‘known good’ u-boot to memory and follows with flashing the same u-boot to eMMC. This means that host has to have the board connected as USB slave. Example LAVA settings assume that all jumpers are controlled with relay.

1
2
3
4
5
6

- deploy:
    timeout:
      minutes: 10
    to: recovery
    namespace: recovery
    connection: lxc

1
2
3
4
5
6

- deploy:
    timeout:
      minutes: 25
    to: fastboot
    namespace: droid
    connection: lxc