Templates in OpenStack's Zuul

This is a followup to my post  "Creating new test jobs in OpenStack CI". Last time I covered the basic setup of jobs by Jenkins and Zuul. Since many OpenStack projects run the same jobs, the Zuul developers have introduced templates to easily group and reuse these jobs.

Let's look at one common example, it's the python-jobs template. Like all examples in this article, it is defined in file zuul/layout.yaml in the openstack-infra/project-config repository and to use it, you need to edit the same file.

Defining your own templates

Since you now know how to use a template, let's explain how they really look like. A template consists of a name, definitions for which jobs should run in which queue and allows to substitute the name of the repository in the job, so {name} gets replaced by your repository, in the example by amazing-repo.

Let's look at the python-jobs template:



  - name: python-jobs
    check:
      - 'gate-{name}-pep8'
      - 'gate-{name}-docs'
      - 'gate-{name}-python27'
    gate:
      - 'gate-{name}-docs'
      - 'gate-{name}-pep8'
      - 'gate-{name}-python27'
    post:
      - '{name}-branch-tarball'

The template has the name python-jobs, adds three jobs to the check queue and the same jobs also to the gate queue. An additional job is added to the post queue. Jobs in the check get queue get triggered when a change gets submitted, jobs in the gate queue get triggered when a change gets approved by a core reviewer and jobs in the post queue get triggered after a change has merged.
If you are adding the same class of jobs to several repositories, create a template for it. A template can contain of a single job that is associated with one queue, or contain several jobs in several queues like the example above.

Using a template

So, if your project amazing-project wants to reuse the python-jobs template as is, just add it as template:

  - name: openstack/amazing-repo
    template:
      - name: merge-check
      - name: python-jobs
You can also limit, on which branches those are jobs are triggered. For example, to run the docs job only on stable/liberty and newer branches, you can add a condition:

  - name: gate-amazing-project-docs
    branch: ^(?!stable/kilo).*$


So, instead of saying run on liberty and newer, we block it on older supported branches, in this case kilo is the only older supported branch.

If you're introducing jobs, best practice is to add them first to the experimental queue, and then add them as non-voting, and only finally as voting. In this case, the templates do not help you at all for the first two steps, you have to look at their definition and add them manually.

First step, using the jobs in the experimental queue:

  - name: openstack/amazing-repo
    template:
      - name: merge-check
      - name: noop-jobs
    experimental:
      - gate-amazing-repo-pep8
      - gate-amazing-repo-docs
      - gate-amazing-repo-python27

Note that we use noop-jobs as a template, so that both check and gate queue have at least one job. The noop jobs do nothing but are important since Zuul requires at least one job to run with success, otherwise you will not be able to merge anything.

With this definition, you can now submit a change and add as review comment "check experimental" and the jobs are run and the results are reported.

Later, the manually triggered jobs run fine, so it's time to run them on each change but keep them non-voting to not block any merges:

  - name: gate-amazing-repo-docs
    voting: false

  - name: gate-amazing-repo-pep8
    voting: false

  - name: gate-amazing-repo-python27
    voting: false
....

  - name: openstack/amazing-repo
    template:
      - name: merge-check
    check:
      - gate-amazing-repo-pep8
      - gate-amazing-repo-docs
      - gate-amazing-repo-python27
    gate:
      - noop
Here we added the noop job to the gate since otherwise no job would run in the gate and Zuul requires at least one job to run.

Once the jobs all run fine, you can add them to the gate as well - and for that case, let's finally use the template:

  - name: openstack/amazing-repo
    template:
      - name: merge-check
      - name: python-jobs

Defining your own templates

Since you now know how to use a template, let's explain how they really look like. A template consists of a name, definitions for which jobs should run in which queue and allows to substitute the name of the repository in the job, so {name} gets replaced by your repository, in the example by amazing-repo.
Let's review the python-jobs template again:

 
  - name: python-jobs
    check:
      - 'gate-{name}-pep8'
      - 'gate-{name}-docs'
      - 'gate-{name}-python27'
    gate:
      - 'gate-{name}-docs'
      - 'gate-{name}-pep8'
      - 'gate-{name}-python27'
    post:
      - '{name}-branch-tarball'

The template has the name python-jobs, adds three jobs to the check queue and the same jobs also to the gate queue. An additional job is added to the post queue. Jobs in the check get queue get triggered when a change gets submitted, jobs in the gate queue get triggered when a change gets approved by a core reviewer and jobs in the post queue get triggered after a change has merged.
If you are adding the same class of jobs to several repositories, create a template for it. A template can contain of a single job that is associated with one queue, or contain several jobs in several queues like the example above.

References

For more information about templates, you can look at the file zuul/layout.yaml on definitions and usage.  Zuul has been written for OpenStack CI and has its own documentation. For information about Zuul's OpenStack instance, read the Project Config Infrastructure page about Zuul. The best starting place learn about using the OpenStack CI infrastructure is the Infra Manual.

Followup?

If you liked this post and like to learn more about OpenStack CI, please leave a comment with details.

Creating new test jobs in OpenStack CI

Reviewing patches for the OpenStack CI infrastructure, there's one piece that often confuse contributors: The question how Zuul and Jenkins configuration are working together.

While we have the Infra Manual with a whole page on how to create a project - and I advise everyone to read it - , let me try to tackle the specific topic of adding new jobs from a different angle.

What we're discussing here are job, or tests, that are run. Jenkins actually runs these jobs. Zuul watches for changes in gerrit (URL for OpenStack is review.openstack.org) to trigger the appropriate jobs so that Jenkins runs them.


To understand the relationship between these two systems, let's try as an analogy programming languages: As a developer, you create a library of functions that do a variety of actions. You also write a script that uses this library to execute them. Jenkins can be considered the library of test functions. But just defining these is not enough, you have to call them. Zuul takes care of calling them, so in the analogy is your script.

So, to actually get a job running for a repository, you first need to define it in the Jenkins "library", and then you trigger its invocation in Zuul. You can also add certain conditions to limit when the job runs or whether it is voting.

If you dig deeper into Jenkins and Zuul, keep in mind that these are two different programming languages, even if both use YAML as format. Jenkins runs jobs and these are defined as text files using the Jenkins job builder. To define them, you can write a job, or use a job-template and instantiate it, or group several job-template in a job-group and instantiate that job-group to create with a few lines many jobs. Zuul uses these jobs and has as syntactic sugar templates to reuse jobs and the queues they run in.

Let's look at a simple examples, adding a new docs job to your repository called amazing-repo:

1. Check out the project-config repository and make it ready for patch submission like creating a branch where you work on.
2. Since for the docs job already a template exists, you can reuse it. It is called'gate-{name}-docs', so add it to your repository in file jenkins/jobs/projects.yaml:
   - project:
     name: amazing-repo
     node: bare-trusty
     jobs:
         - gate-{name}-docs
3. Now define how to trigger the job. Edit file zuul/layout.yaml and update your repository entry to add the job:
   
   - name: openstack/amazing-repo
     template:
       - name: merge-check
     check:
       - gate-amazing-repo-docs
     gate:
       - gate-amazing-repo-docs
   This adds the job to both the check and gate queue. So, it will notonly be run when a patch is initially submitted for review in the check queue but also after a patch gets approved in the gate queue. Since your tree might be different when you submitted a change and when it merges, we run jobs in both situations so that the tree istested exactly as it merges.
4. Let's go one step back: Your repository is not ready yet to havethe docs job voting, so you only want to run it as non-voting.
   In that case add a condition in the jobs section of zuul/layout.yaml:
   
   - name: gate-amazing-repo-docs
       voting: false
   
   And in your repository, only add it to the check queue. Non-voting jobs should not be in the gate, they get ignored completely and just waste resources:
   
     - name: openstack/amazing-repo
       template:
         - name: merge-check
       check:
         - gate-amazing-repo-docs
       gate:
         - ...

So, these are simple jobs. Stay tuned for a followup article that will cover how to use templates in Zuul - and how to modify your repository in the context of templates.

Thanks to Doug Fish for reviewing the text and giving suggestions on how to improve it - and urging me to write a follow-up.

P.S. Follow-up article is called "Templates in OpenStack's Zuul".