Simple Administration¶

Requirements¶

You need to be familiar with these sections:

1. First steps installing LAVA
2. creating a pipeline worker.
3. Adding devices to a worker
4. Superuser
5. Logging In (as superuser)
6. Device types and Database elements for a device type
7. Adding your first devices
8. Creating Backups

Outline¶

LAVA is complex and administering a LAVA instance can be an open-ended task covering a wide range of skills.

• Debian system administration
• Device integration
• Network configuration
• Writing tests
• Triage
• Python/Django knowledge - for debugging

Debian system administration¶

At a simple level, LAVA requires a variety of Debian system administration tasks, including:

• installing, upgrading and maintaining the installed packages and apt sources

• configuring services outside LAVA, including:

  • apache - LAVA provides an example apache configuration but many instances will need to adapt this for their own hosting requirements.

  • DHCP - Most devices will need networking support using DHCP.

  • configuration management - LAVA has a variety of configuration files and a number of other services and tools will also need to be configured, for example serial console services, TFTP services and authentication services.

    See also

    Creating Backups

  • email - LAVA can use email for notifications, if test writers include appropriate requests in the test job submissions. To send email, LAVA relies on the basic Django email support using a standard sendmail interface. Only the master needs to be configured to send email, notifications from workers are handled via the master.

service lava-server-gunicorn restart
service lava-publisher restart
service lava-scheduler restart
service lava-worker restart

Infrastructure¶

LAVA instances will need some level of infrastructure, including:

• UPS
• network switches
• remote power control hardware
• master and worker hardware

Many instances will also require specialized hardware to assist with the automation of specific devices, including switchable USB hubs or specialized relay boards.

Start small¶

These rules may seem harsh or obvious or tedious. However, multiple people have skipped one or more of these requirements and have learned that these steps provide valuable advice and assistance that can dramatically improve your experience of LAVA. Everyone setting up LAVA, is strongly advised to follow all of these rules.

1. Start with a minimal LAVA install with at most one or two devices - at this stage only QEMU devices should be considered. This provides the best platform for learning LAVA, before learning how to administer a LAVA instance.

2. Use the worked examples in the documentation which refer back to standard builds and proven test jobs. There will be enough to do in becoming familiar with how to fix problems and issues local to your own instance without adding the complexity of devices or kernel builds to which only you have access.

3. Avoid rushing to your custom device - device integration into any automated system is hard. It does not become any easier if you are trying to learn how to use the automation as well.

4. Plan how to test

   • use the examples and device types which are known to work.
   • Read through all the worked examples before starting your planning, there are likely to be useful ways to do what you want to do and advice on why it is a bad idea to do some of the things you may have considered at the start.
   • plan out how to do the testing of other custom devices by looking for similar device support already available in other LAVA instances.
   • Avoid shortcuts - it may seem that you only want to connect & test but there are known problems with overly simplistic approaches and you are likely to need to use deploy actions and boot actions to be able to produce reliable results.

5. Have at least one test instance. A single instance of LAVA is never sufficient for any important testing. Everyone needs at least one test instance in a VM or on another machine to have confidence that administrative changes will not interfere with test jobs.

6. Control your changes - configuration, test job definitions, test shell definitions, device dictionaries, template changes and any code changes - all need to be in version control.

7. Control access to the dispatcher and devices - device configuration details like the connection command and remote power commands can be viewed by all users who are able to submit to that device. In many cases, these details are sufficient to allow anyone with the necessary access to administer those devices, including modifying bootloader configuration. Only administrators should have access to any machine which itself has access to the serial console server and/or remote power control services. Typically, this will be controlled using SSH keys.

   See also

   Power Commands and Running lava-run directly

8. Subscribe to the Mailing lists where you will find others who have setup their own LAVA instances. IRC is fine for quick queries but it is trivial to lose track of previous comments, examples and links when the channel gets busy. Mailing lists have public archives which are fully indexed by search engines. The archives will help you solve your current issue and help many others find answers for their own issues later.

Problems with simplistic testing¶

There are a number of common fallacies relating to automation. Check your test ideas against these before starting to make your plans:

Roles of LAVA administrators¶

The ongoing roles of administrators include:

• monitor the number of devices which are online

• identify the reasons for health check failures

• communicate with users when a test job has made the device unbootable (i.e. bricked)

• recover devices which have gone offline

• restrict command line access to the dispatcher(s) and device(s) to only other administrators. This includes access to the serial console server and the remote power control service. Ideally, users must not have any access to the same subnet as the dispatchers and devices, except for the purposes of accessing devices during LAVA Hacking Sessions. This may involve port forwarding or firewall configuration and is not part of the LAVA software support.

• to keep the instance at a sufficiently high level of reliability that Continuous Integration produces results which are themselves reliable and useful to the developers. To deliver this reliability, administrators do need to sometimes prevent users from making mistakes which are likely to take devices offline.

• prepare and routinely test backups and disaster recovery support. Many lab admin teams use salt or ansible or other configuration management software. Always ensure you have a fast way of deploying a replacement worker or master in case of hardware failure.

  See also

  Creating Backups for details of what to backup and test.

Best practice¶

• Before you upgrade the server or dispatcher, run the standard test jobs and a few carefully chosen stable jobs of your own as a set of functional tests - just as the LAVA team do upstream.
• Keep all the servers and dispatchers regularly updated with regard to security updates and bug fixes. The more often you run the upgrades, the fewer packages will be involved in each upgrade and so the easier it will be to spot that one particular upgrade may be misbehaving.
• Repeat your functional tests after all upgrades.
• Use health checks and tweak the frequency so that busy devices run health checks often enough to catch problems early.
• Add standard investigative tools. You may choose to use nagios and / or munin or other similar tools.
• Use configuration management. Various LAVA instances use salt or puppet or ansible. Test out various tools and make your own choice.

Triage¶

When you come across problems with your LAVA instance, there are some basic information sources, methods and tools which will help you identify the problem(s).

Dispatcher unable to meet job compatibility requirement.

lava-server/lava_scheduler_app/tests/device-types/

/etc/lava-server/dispatcher-config/device-types/

Overriding device configuration¶

Some device configuration can be overridden without making changes to the Jinja2 Templates. This does require some understanding of how template engines like jinja2 operate.

• Values hard-coded into the jinja2 template cannot be overridden. The template would need to be modified and re-tested.
• Variables in the jinja2 template typically have a default value.
• Variables in the jinja2 template can be override the default in the following sequence:
  1. by the next template
  2. by the device dictionary or, if neither of those set the variable
  3. by the job context.

To identify which variables can be overridden, check the template for placeholders. A commonly set value for QEMU device types is the amount of memory (on the dispatcher) which QEMU will be allowed to use for each test job:

- -m {{ memory|default(512) }}

Most administrators will need to set the memory constraint in the device dictionary so that test jobs cannot allocate all the available memory and cause the dispatcher to struggle to provide services to other test jobs. An example device dictionary to override the default (and also prevent test jobs from setting a different value) would be:

{% extends 'qemu.jinja2' %}
{% set memory = 1024 %}

Admins need to balance the memory constraint against the number of other devices on the same dispatcher. There are occasions when multiple test jobs can start at the same time, so admins may also want to limit the number of emulated devices on any one dispatcher to the number of cores on that dispatcher and set the amount of memory so that with all devices in use there remains some memory available for the system itself.

Most administrators will not set the arch variable of a QEMU device so that test writers can use the one device to run test jobs using a variety of architectures by setting the architecture in the job context. The QEMU template has conditional logic for this support:

{% if arch == 'arm64' or arch == 'aarch64' %}
           qemu-system-aarch64
{% elif arch == 'arm' %}
           qemu-system-arm
{% elif arch == 'amd64' %}
           qemu-system-x86_64
{% elif arch == 'i386' %}
           qemu-system-x86
{% endif %}

{% extends 'my-device.jinja2' %}
{% set shutdown_message = "reboot: Restarting system" %}
{% set boot_message = "Booting Linux" %}

- boot:
   method: u-boot
   commands: ramdisk
   parameters:
     shutdown-message: "reboot: Restarting system"
   prompts:
   - 'linaro-test'
   timeout:
     minutes: 2

Adding more devices¶

Once you have a couple of QEMU devices running and you are happy with how to maintain, debug and test using those devices, start adding known working devices. These are devices which already have templates in:

/etc/lava-server/dispatcher-config/device-types/

The majority of the known device types are low-cost ARM developer boards which are readily available. Even if you are not going to use these boards for your main testing, you are recommended to obtain a couple of these devices as these will make it substantially easier to learn how to administer LAVA for any devices other than emulators.

Physical hardware like these dev-boards have hardware requirements like:

• serial console servers
• remote power control
• network infrastructure
• uninterruptible power supplies
• shelving
• cables
• removable media

Understanding how all of those bits fit together to make a functioning LAVA instance is much easier when you use devices which are known to work in LAVA.

Early admin stuff:

• recommendations on how to do admin:
  • start simple using our examples
  • build complexity slowly
  • only once you’re confident, start adding novel devices
• where to find logs and debug information
• device configuration and templates
• getting a number of cheap ARMv7 development boards

Adding users and groups¶

Users and groups can be added and modified in the Django administration interface or from the command line.

Newly created users will need permission to submit test jobs. This can be done by adding the user to a group which already has the Can cancel or resubmit test jobs permission or by adding this permission for each individual user.

$ sudo lava-server manage users add <username>  --passwd <password>

$ sudo lava-server manage addldapuser --username {username}

$ sudo lava-server manage groups add <name>

$ sudo lava-server manage devices update {hostname} --owner {username}
$ sudo lava-server manage devices update {hostname} --physical-user {username}

$ sudo lava-server manage devices update {hostname} --group {group_name}
$ sudo lava-server manage devices update {hostname} --physical-group {group_name}