State machine

The state machine describes and controls the state and health of Workers, Devices and Test Jobs.

Workers

For each worker, two variables describe the current status:

  • state:
    • Online
    • Offline
  • health:
    • Active
    • Maintenance
    • Retired

state is an internal variable, set by lava-master when receiving (or not) pings from each worker.

Caution

When a worker is in Offline, none of the attached devices will be used to schedule new jobs.

health can be used by admins to control the health of all attached devices. For instance, when set to Maintenance, all attached devices will be automatically put into maintenance mode so that no jobs will be scheduled on those devices.

Devices

For each device, two variables describe the current status:

  • state:
    • Idle: not in use by any test job
    • Reserved: has been reserved for a test job but the test job is not running yet
    • Running: currently running a test job
  • health:
    • Good
    • Unknown
    • Looping: should run health-checks in a loop
    • Bad
    • Maintenance
    • Retired

state is an internal variable, set by lava-server-gunicorn and lava-scheduler when scheduling, starting, canceling and ending test jobs.

health can be used by admins to indicate if a device should be used by the scheduler or not. Moreover, when ending an health-check, the device health will be set according to the test job health.

TestJobs

For each test job, two variables are describing the current status:

  • state:
    • Submitted: waiting in the queue
    • Scheduling: part of a multinode test job where some sub-jobs are still in *Submitted
    • Scheduled: has been scheduled. For multinode, it means that all sub-jobs are also scheduled
    • Running: currently running on a device
    • Canceling: has been canceled but not ended yet
    • Finished

Note

Only multinode test jobs use Scheduling. When all sub-jobs are in Scheduling, lava-master will transition all test jobs to Scheduled.

  • health:
    • Unknown: default value that will be overridden when the job is finished.
    • Complete
    • Incomplete
    • Canceled: the test job was canceled.

Scheduler

The scheduler is called by lava-master approximately every 20 seconds. The scheduler starts by scheduling health-checks. The remaining devices are then considered for test jobs.

Health-checks

To ensure that health-checks are always scheduled when needed, they will be considered first by the scheduler before regular test jobs.

The scheduler will only consider devices where:

  • state is Idle
  • health is Good, Unknown or Looping
  • worker’s state is Online

Note

A device whose health is Bad, Maintenance or Retired is never considered by the scheduler when it is looking for devices to run test jobs

Test jobs

The scheduler will only consider devices where:

  • state is Idle
  • health is Good or Unknown
  • worker’s state is Online