Writing a monitors test action

See also

Monitors and Writing Tests

Writing a Lava-Test Test Definition 1.0 involves adding an overlay of POSIX scripts to the DUT but some devices cannot support an overlay. The monitors test action supports IoT devices using pattern matching against non-interactive output streams. As with pattern matching in the Lava-Test Test Definition 1.0, this can be difficult at times.

The IoT application needs to be specially written to create output which is compatible with automation. Test actions for IoT devices tend to be short and specific to one dedicated application executing on the DUT. Applications need to be written to emit a START and STOP message which have to be unique across all test job output, not just the test action. Test writers then need to correlate specific pattern match expressions in the test job with the behavior of the application deployed in that test job. This puts the work of creating a parser into the hands of the writers of the test application and can thus interfere with development of the test application.

Restrictions

Test jobs using Monitors must:

  • Use carefully designed applications which are designed to automatically execute after boot.
  • Emit a unique start string:
    • Only once per boot operation.
    • Before any test operation starts.
  • Emit a unique end string:
    • Only once per boot operation.
    • After all test operations have completed.
  • Provide a regular expression which matches all expected test output and maps the output to results without leading to excessively long test case names.
  • start and end strings will match part of a line but make sure that each string is long enough that it can only match once per boot.

If start does not match, the job will timeout with no results.

If end does not match, the job will timeout but the results (of the current boot) will already have been reported.

Example test monitors job

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# Zephyr JOB definition for IoT
device_type: frdm-kw41z
job_name: frdm-kw41z zephyr IoT example

timeouts:
  job:
    minutes: 10
  action:
    minutes: 3
  actions:
    wait-usb-device:
      seconds: 40

priority: medium
visibility: public

actions:
- deploy:
    timeout:
      minutes: 3
    to: tmpfs
    images:
        zephyr:
            url:  http://images.validation.linaro.org/snapshots.linaro.org/components/kernel/zephyr/master/zephyr/frdm_kw41z/737/tests/kernel/libs/test/zephyr.bin

- boot:
    method: pyocd
    timeout:
      minutes: 10

- test:
    monitors:
    - name: tests
      start: BOOTING ZEPHYR
      end: PROJECT EXECUTION SUCCESSFUL
      pattern: '(?P<test_case_id>.*) (?P<measurement>.*) tcs = [0-9]* nsec'
      fixupdict:
        PASS: pass
        FAIL: fail

Download or view example test monitors job: examples/test-jobs/frdm-kw41z-zephyr.yaml