Lava Dispatcher Design¶

Pipeline Architecture¶

Objectives¶

The new dispatcher design is intended to make it easier to adapt the dispatcher flow to new boards, new mechanisms and new deployments. It also shifts support to do less work on the dispatcher, make fewer assumptions about the test in the dispatcher configuration and put more flexibility into the hands of the test writer.

From 2015.8 onwards the sample jobs supporting the unit tests conform to the LAVA schema.

Design¶

Start with a Job which is broken up into a Deployment, a Boot and a Test class. Results are transmitted live during any part of the job.

The Job manages the Actions using a Pipeline structure. Actions can specialize actions by using internal pipelines and an Action can include support for retries and other logical functions:

If a Job includes one or more Test definitions, the Deployment can then extend the Deployment to overlay the LAVA test scripts without needing to mount the image twice:

The TestDefinitionAction has a similar structure with specialist tasks being handed off to cope with particular tools:

Following the code flow¶

The deployment is determined from the device_type specified in the Job (or the device_type of the specified target) by reading the list of support methods from the device_types YAML configuration.

Each Action can define an internal pipeline and add sub-actions in the Action.populate function.

Particular Logic Actions (like RetryAction) require an internal pipeline so that all actions added to that pipeline can be retried in the same order. (Remember that actions must be idempotent.) Actions which fail with a JobError or InfrastructureError can trigger Diagnostic actions. See Logical actions.

actions:
  deploy:
    allow:
      - image
  boot:
    allow:
      - image

This then matches the python class structure:

actions/
   deploy/
       image.py

The class defines the list of Action classes needed to implement this deployment. See also Dispatcher Action Reference.

Pipeline construction and flow¶

The pipeline is a FIFO and has branches which are handled as a tree walk. The top level object is the job, based on the YAML definition supplied by the lava-master. The definition is processed by the scheduler and the submission interface with information specific to the actual device. The processed definition is parsed to generate the top level pipeline and strategy classes. Each strategy class adds a top level action to the top level pipeline. The top level action then populates branches containing more actions.

Actions are populated, validated and executed in strict order. The next action in any branch waits until all branches of the preceding action have completed. Populating an action in a pipeline creates a level string, e.g. all actions in level 1.2.1, including all actions in sublevel 1.2.1.2 are executed before the pipeline moves on to processing level 1.3 or 2:

Deploy (1)
   |
   \___ 1.1
   |
   \ __ 1.2
   |     |
   |     \_ 1.2.1
   |     |   |
   |     |   \_ 1.2.1.1
   |     |   |
   |     |   \_ 1.2.1.2
   |     |         |
   |     |         \__ 1.2.1.2.1
   |     |
   |     \__1.2.2
   |
   \____1.3
   |
  Boot (2)
   |
   \_ 2.1
   |
   \_ 2.2

1. One device per job. One top level pipeline per job
   • loads only the configuration required for this one job.
2. A NewDevice is built from the target specified (commands.py)
3. A Job is generated from the YAML by the parser.
4. The top level Pipeline is constructed by the parser.
5. Strategy classes are initialized by the parser
   1. Strategy classes add the top level Action for that strategy to the top level pipeline.
   2. Top level pipeline calls populate() on each top level Action added.
      1. Each Action.populate() function may construct one internal pipeline, based on parameters.
      2. internal pipelines call populate() on each Action added.
      3. A sublevel is set for each action in the internal pipeline. Level 1 creates 1.1 and level 2.3.2 creates 2.3.2.1.
6. Parser waits while each Strategy completes branch population.
7. Parser adds the FinalizeAction to the top-level pipeline
8. Loghandlers are set up
9. Job validates the completed pipeline
   1. Dynamic data can be added to the context
10. If --validate not specified, the job runs.
    1. Each run() function can add dynamic data to the context and/or results to the pipeline.
    2. Pipeline walks along the branches, executing actions.
11. Job ends, check for errors
12. Completed pipeline is available.

Pipeline error handling¶

Result bundle identifiers¶

Old style result bundles are assigned a text based UUID during submission. This has several issues:

• The UUID is not sequential or predictable, so finding this one, the next one or the previous one requires a database lookup for each. The new dispatcher model will not have a persistent database connection.
• The UUID is not available to the dispatcher while running the job, so cannot be cross-referenced to logs inside the job.
• The UUID makes the final URL of individual test results overly long, unmemorable and complex, especially as the test run is also given a separate UUID in the old dispatcher model.

The new dispatcher creates a pipeline where every action within the pipeline is guaranteed to have a unique level string which is strictly sequential, related directly to the type of action and shorter than a UUID. To make a pipeline result unique on a per instance basis, the only requirement is that the result includes the JobID which is a sequential number, passed to the job in the submission YAML. This could also have been a UUID but the JobID is already a unique ID for this instance.

When bundles are downloaded, the database query will need to assign a UUID to that downloaded file but the file will also include the job number and the query can also insert the source of the bundle in a comment in the YAML.

Secondary media¶

With the migration from master images on an SD card to dynamic master images over NFS, other possibilities arise from the refactoring.

• Deploy a ramdisk, boot and deploy an entire image to a USB key, boot and direct bootloader at USB filesystem, including kernel and initrd.
• Deploy an NFS system, boot and bootstrap an image to SATA, boot and direct bootloader at SATA filesystem, including kernel and initrd.
• Deploy using a script written by the test author (e.g. debootstrap) which is installed in the initial deployment. Parameters for the script need to be contained within the test image.

Device configuration design¶

Device configuration, as received by lava_dispatch has moved to YAML and the database device configuration has moved to Jinja2 templates. This method has a much larger scope of possible methods, related to the pipeline strategies as well as allowing simple overrides and reuse of common device configuration stanzas.

There is no need for the device configuration to include the hostname in the YAML as there is nothing on the dispatcher to check against - the dispatcher uses the command line arguments and the supplied device configuration. The configuration includes all the data the dispatcher needs to be able to run the job on the device attached to the specified ports.

The device type configuration on the dispatcher is replaced by a device type template on the server which is used to generate the YAML device configuration sent to the dispatcher.

$ sudo lava-server manage device-dictionary --hostname mypanda --export
{% extends 'panda.jinja2' %}
{% set power_off_command = '/usr/bin/pduclient --daemon tweetypie --hostname pdu --command off --port 08' %}
{% set hard_reset_command = '/usr/bin/pduclient --daemon tweetypie --hostname pdu --command reboot --port 08' %}
{% set connection_list = [‘uart0’] %}
{% set connection_commands = {‘uart0’: ‘telnet dispatcher01 7001’} %}
{% set connection_tags = {‘uart0’: [‘primary’, 'telnet']} %}
{% set power_on_command = '/usr/bin/pduclient --daemon tweetypie --hostname pdu --command on --port 08' %}

$ sudo lava-server manage device-dictionary --hostname mypanda --review

device_type: beaglebone-black
commands:
  connect: telnet localhost 6000
  hard_reset: /usr/bin/pduclient --daemon localhost --hostname pdu --command reboot --port 08
  power_off: /usr/bin/pduclient --daemon localhost --hostname pdu --command off --port 08
  power_on: /usr/bin/pduclient --daemon localhost --hostname pdu --command on --port 08

parameters:
 bootm:
  kernel: '0x80200000'
  ramdisk: '0x81600000'
  dtb: '0x815f0000'
 bootz:
  kernel: '0x81000000'
  ramdisk: '0x82000000'
  dtb: '0x81f00000'

actions:
 deploy:
   # list of deployment methods which this device supports
   methods:
     # - image # not ready yet
     - tftp

 boot:
   # list of boot methods which this device supports.
   methods:
     - u-boot:
         parameters:
           bootloader_prompt: U-Boot
           boot_message: Booting Linux
           # interrupt: # character needed to interrupt u-boot, single whitespace by default
         # method specific stanza
         oe:
           commands:
           - setenv initrd_high '0xffffffff'
           - setenv fdt_high '0xffffffff'
           - setenv bootcmd 'fatload mmc 0:3 0x80200000 uImage; fatload mmc 0:3 0x815f0000 board.dtb;
             bootm 0x80200000 - 0x815f0000'
           - setenv bootargs 'console=ttyO0,115200n8 root=/dev/mmcblk0p5 rootwait ro'
           - boot
         nfs:
           commands:
           - setenv autoload no
           - setenv initrd_high '0xffffffff'
           - setenv fdt_high '0xffffffff'
           - setenv kernel_addr_r '{KERNEL_ADDR}'
           - setenv initrd_addr_r '{RAMDISK_ADDR}'
           - setenv fdt_addr_r '{DTB_ADDR}'
           - setenv loadkernel 'tftp ${kernel_addr_r} {KERNEL}'
           - setenv loadinitrd 'tftp ${initrd_addr_r} {RAMDISK}; setenv initrd_size ${filesize}'
           - setenv loadfdt 'tftp ${fdt_addr_r} {DTB}'
           # this could be a pycharm bug or a YAML problem with colons. Use : for now.
           # alternatively, construct the nfsroot argument from values.
           - setenv nfsargs 'setenv bootargs console=ttyO0,115200n8 root=/dev/nfs rw nfsroot={SERVER_IP}:{NFSROOTFS},tcp,hard,intr ip=dhcp'
           - setenv bootcmd 'dhcp; setenv serverip {SERVER_IP}; run loadkernel; run loadinitrd; run loadfdt; run nfsargs; {BOOTX}'
           - boot
         ramdisk:
           commands:
           - setenv autoload no
           - setenv initrd_high '0xffffffff'
           - setenv fdt_high '0xffffffff'
           - setenv kernel_addr_r '{KERNEL_ADDR}'
           - setenv initrd_addr_r '{RAMDISK_ADDR}'
           - setenv fdt_addr_r '{DTB_ADDR}'
           - setenv loadkernel 'tftp ${kernel_addr_r} {KERNEL}'
           - setenv loadinitrd 'tftp ${initrd_addr_r} {RAMDISK}; setenv initrd_size ${filesize}'
           - setenv loadfdt 'tftp ${fdt_addr_r} {DTB}'
           - setenv bootargs 'console=ttyO0,115200n8 root=/dev/ram0 ip=dhcp'
           - setenv bootcmd 'dhcp; setenv serverip {SERVER_IP}; run loadkernel; run loadinitrd; run loadfdt; {BOOTX}'
           - boot

$ sudo lava-server manage device-dictionary --hostname mypanda --import mypanda.yaml

$ sudo lava-run --device devices/fred.yaml panda-ramdisk.yaml --output-dir=/tmp/test