Using Jinja2 to generate LAVA test job submissions¶

Starting with templates¶

The format you start with, is the format you will output.

Any YAML file can be a starting template. Jinja2 will only operate directly on text which is written in Jinja2 syntax. Anything else will be passed through unaltered. This has two valuable properties for our YAML job submissions: comments are preserved, and the ordering of items (dictionaries, etc.) will be kept.

The same cannot be said if you load YAML data into a Python program directly, e.g:

# example dictionary
first_key: 1
second_key: 2
third_key: 3
fourth_key: 4

Loading this snippet (a YAML dictionary) into Python2 will generate:

{'second_key': 2, 'third_key': 3, 'fourth_key': 4, 'first_key': 1}

Loading the same snippet into Python3 will generate:

{'first_key': 1, 'second_key': 2, 'third_key': 3, 'fourth_key': 4}

Loading this as a template would generate:

# example dictionary
first_key: 1
second_key: 2
third_key: 3
fourth_key: 4

To make a change in that output, we save the template to a file and add some Jinja2 syntax to the template:

1
2
3
4
5

# example dictionary
first_key: {{ first_key }}
second_key: 2
third_key: 3
fourth_key: 4

Download or view the complete example: examples/templates/content.jinja2

Data is then presented as a details.jinja dictionary:

1
2

{% extends 'content.jinja2' %}
{% set first_key = 15 %}

Download or view the complete example: examples/templates/details.jinja2

The python script to process the template is based on the LAVA source code and only requires Jinja2:

https://git.lavasoftware.org/lava/lava/blob/master/lava_scheduler_app/tests/test_base_templates.py#L22

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

import os
import jinja2


CONFIG_PATH = os.getcwd()  # set this to your local needs.

job_ctx = {"first_key": 9}


with open("details.jinja2", "r") as details:
    data = details.read()
string_loader = jinja2.DictLoader({"details.jinja2": data})
type_loader = jinja2.FileSystemLoader([CONFIG_PATH])
env = jinja2.Environment(  # nosec - YAML, not HTML, no XSS scope.
    loader=jinja2.ChoiceLoader([string_loader, type_loader]),
    trim_blocks=True,
    autoescape=False,
)

Download or view the complete example: examples/templates/templating.py

The templating.py script loads the details data as a dictionary (line 12) and then looks up the content.jinja2 as the extended template, based on the CONFIG_PATH. (The template can easily be in a different directory to the details dictionary.)

# example dictionary
first_key: 15
second_key: 2
third_key: 3
fourth_key: 4

Extending templates with contexts¶

templating.py above includes a job_ctx dictionary which sets a different value but this value is currently ignored. This is because the details dictionary sets first_key without allowing for an override.

This is a key part of how templating works with Jinja2. Each element controls the ability of the next element to provide overrides or changes.

1. hard coded template values - nothing can override the raw YAML of the template itself. In the example for this section, content.jinja2 contains a YAML comment and a basic YAML dictionary. Anything (including whitespace) in the template which is not marked up with Jinja2 syntax will be preserved in the final output unchanged and without an ability to support changes via the details dictionary or the job context.

2. template defaults - templates can include default values:

   first_key: {{ first_key | default(4) }}
   

   If neither the details dictionary nor the job context set a value for first_key, the default from the template will be used. In the original content.jinja2, there is no default. If the details dictionary and job context do not override first_key, the value will be left blank:

   first_key:
   

   If this output is then loaded into Python as a YAML file, the value of the first_key key in the Python dictionary would be None.

3. details dictionary values - in the example above, the details dictionary sets a value of 15 and the job context cannot change that value.

   It is also possible to add new blocks of YAML output using the same set function of the details dictionary. Take care with the whitespace in such dictionaries. Always test the output and verify not only that the output is valid YAML but that when loaded into Python, that the dictionary contains the correct values for the correct keys. Whitespace problems are a common cause of errors. It’s easy to cause invalid syntax in the output, or for a value to be assigned in the wrong place (as a value for a sub key instead of as a new key, or vice versa).

   {key: value}
   # not the same as:
   {key: None, value: None}
   

   Details dictionary values can be other Python types. It is common to find strings, floats, lists and dictionaries. Syntax can be either Python or YAML.

4. details dictionary defaults - the details dictionary can use defaults in the same way as the template:

   {% set first_key = first_key | default(15) %}
   

   This allows a value in the job context to override the value. Download the example files and edit details.jinja2 to set the value 15 but only as a default. Re-run templating.py and the job context value of 9 will be used in the output instead:

   # example dictionary
   first_key: 9
   second_key: 2
   third_key: 3
   fourth_key: 4
   

5. job context - To set a value in the job context, there must be a placeholder in the template and usually a default value as well.

Templates can extend templates¶

Once you have more than a couple of related templates, it becomes obvious that some content will be duplicated across many of those templates. Create a new template (possibly using the conventions of base-foo or foo-common) containing a single copy of the base content and then each template simply {% extends 'base.jinja2' %}.

Logic control within templates¶

Jinja2 supports a range of logical elements, for example:

{% for a in b %}
{% endfor %}

{% if a is b %}
{% endif %}

{% block label %}
{% endblock %}

See the Jinja documentation and the LAVA device-type templates for more examples.

Template best practice¶

Here are some recommendations on how to use templates effectively with LAVA job submissions.

• Avoid repetition - of content or logic.

• Avoid complexity - keep the number of possible overrides, defaults and templates to a workable level.

• Keep the YAML in the templates - the structure of the final output should be visible directly from the template, subject to templates extending other templates. The details dictionary should ideally contain only Jinja2 syntax content. This makes it much easier to verify that the templates will always produce valid YAML.

• Build tests alongside the templates - check that every template outputs valid YAML on it’s own, with example details dictionaries and with example job contexts.

• Use version control - templates can quickly become complex. Make sure errors can be traced and triaged against the history of changes.

• Keep the output human readable - use Comments liberally to describe the content, particularly when using logic in templates. Structure the YAML in a way that makes sense to humans. For example, for a LAVA Test Job submission, job_name and device_type are commonly at the top and the actions list follows at the end. A big advantage of templating is that comments and structure are preserved; use that to your advantage.

• Use automated submission bots - some habits are hard to break and many test writers will simply copy and paste a test job submission without changing either the job_name or the metadata. To be sure that you can retrieve only the data that was submitted via the templating, ensure that the output of the templates is submitted by an automated user and create sufficient automated users for each Test Plan or “Project” if multiple projects share a LAVA instance.

  This will allow you to easily filter results later.

• Use checksums - when referring to build artifacts like kernel binaries and filesystem images, include checksums for those artifacts in your job submission too. Most build systems can be configured to generate checksums at build time; use the job dictionary and templates to insert checksums. This will ensure that LAVA is using exactly what was specified for your test. This can be particularly important when network caches and proxies might affect what your system downloads.

• Avoid using latest directory names - again, specify exactly what artifacts your test will use, with stable URLs. It is a common and useful feature of build systems to include a human-friendly latest link in a download directory. However, what happens when the next build happens? The latest link now moves. You might not be able to determine later exactly which files were used in your test, and you will not be able to resubmit the same test in the future (for example if you want to debug some of your test definitions).

  Templating provides a simple way to refer to the exact file or directory that you need, to match the metadata you specify.

Using templates for test jobs¶

The specifics of how to build templates for test job submission will depend on the data being provided by the rest of your CI system.

Common elements which would need to be modified by the details dictionary (or job dictionary for test jobs) might be:

1. job_name: Avoid making this cryptic or excessively long. The job name is meant to be human readable and is not well suited to searching or data analysis. Use metadata for that. Let the job name be a descriptive sentence which is sufficiently identifiable without having to include a mangled version of every identifier used in the CI process to this point.
2. metadata: This is a free form dictionary which test writers can use to provide searchable key:value pairs which preserve all of the variables and generated dynamic data from the CI loop up to this point. Commit hashes, build numbers, branch names and URLs, series labels, name and/or URL of the configuration, build logs …
3. urls: when testing software using hardware, the software will be constantly changing. Templating is the best way to ensure that the URLs and the metadata for those URLs change in lockstep.