Writing an interactive test action

See also

Interactive test action reference

Parsing test output will always involve some amount of pattern matching. The test action creates a stream of output, (typically ASCII), and this needs to be turned into test results either using scripts on the DUT or after transmission over a serial connection.

Interactive test actions move all of the parsing to the test writer commands by setting a sequence of strings with expected output and support for determining failures.

Note

All interactive test actions need to specify all possible failure strings for each command, as well as the type of exception to raise when a string is matched. Interactive test actions must always have a start and end to the parsing or the entire job will simply time out because none of the expected matches exist.

Interactive test actions cannot provide support for monitoring kernel boot messages or LAVA Test Helpers, all pattern matching is done solely by the strings set by the test writer. This rules out support like MultiNode and Lava-Test Test Definition 1.0.

See also

Write portable test definitions

Advantages of interactive pattern matching

  1. Keeps pattern matches tightly wrapped around limited sections of predictable test action output, not across the entire test job or even the complete test action output.
  2. Provides methods to execute complex instructions (including regular expressions) without mangling the syntax due to the constraints of the submission format (in the case of LAVA, YAML strings).
  3. Avoids issues with needing to deploy a set of POSIX scripts to the DUT, e.g. when the filesystem is inaccessible or read-only or has access controls like SELinux.
  4. Allowing test writers to use any arbitrary test output style.
  5. Includes ability for test writers to raise exceptions based on specific matches, controlled by the test writer.
  6. Encourages test writers to create portable custom scripts which are idempotent. this can make it easier to reproduce failures when reporting bugs to upstream developers who do not have access to the CI system.

Limits of interactive pattern matching

  1. Lacks support for optimizations when specific sections of the test output comply with a known and strictly enforced format.
  2. Can be difficult to convert into test action methods which aid portability. CI becomes difficult for the developers receiving the bug reports if the developers cannot reproduce the bug without using the same CI.
  3. Supporting test jobs with multiple, different, test action behaviors for each stage of the test operation. For example, dependency resolution and setup commands could be done with an overlay and Lava Test Shell Definition whilst running the test to output a known result format could use a parser specific to that format.
  4. Lacks support for versioning of test action components. Test writers need to implement their own versioning system and ensure that versions are output into the LAVA test job log file and/or as a test result.

Example interactive test job

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
device_type: beaglebone-black

job_name: run tests in U-Boot
timeouts:
  job:
    minutes: 10
  action:
    minutes: 5
  connection:
    minutes: 2
priority: medium
visibility: public

metadata:
  source: https://git.lavasoftware.org/lava/lava.git
  path: doc/v2/examples/test-jobs/bbb-uboot-interactive.yaml

actions:
- boot:
    method: bootloader
    bootloader: u-boot
    commands: []
    prompts:
    - '=>'
- test:
    timeout:
      minutes: 4
    interactive:
    - name: network
      prompts: ["=> ", "/ # "]
      script:
      - command: dhcp
        name: dhcp
        successes:
        - message: "DHCP client bound to address"
        failures:
        - message: "TIMEOUT"
          exception: InfrastructureError
          error: "dhcp failed"

Download or view example interactive test job: examples/test-jobs/bbb-uboot-interactive.yaml

Combining different test actions

In some situations, there can be a need to combine two or more different types of test action in a single test job. For example, a POSIX overlay test action to do initial setup, then a monitors or interactive test action to parse the output of a specific task.

It is very important to understand that Lava-Test Test Definition 1.0 cannot occur more than once in a single test job once any other test action type is used. If further test actions are required via the POSIX shell, an interactive test action must be used.

A single Lava-Test Test Definition 1.0 test action can already include multiple different tests, potentially from different repositories:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# TEST_BLOCK
- test:
    timeout:
      minutes: 5
    definitions:
    - repository: http://git.linaro.org/lava-team/lava-functional-tests.git
      from: git
      path: lava-test-shell/smoke-tests-basic.yaml
      name: smoke-tests
    - repository: https://git.linaro.org/lava-team/lava-functional-tests.git
      from: git

It is fully supported to then add an interactive test action:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
- test:
    timeout:
      minutes: 4
    interactive:
    - name: network
      prompts: ["=> ", "/ # "]
      script:
      - command: dhcp
        name: dhcp
        successes:
        - message: "DHCP client bound to address"
        failures:
        - message: "TIMEOUT"
          exception: InfrastructureError
          error: "dhcp failed"

However, until Test Definition 2.0 is fully scoped and delivered, it is not possible to add another Lava-Test Test Definition 1.0 action.

Note

This applies within any one namespace, it does not apply between different namespaces.