MQTT

The MQTT driver by default operates as a reporter only. It can optionally be configured with a subscription topic that will listen for events that trigger pipelines.

Connection Configuration

<mqtt connection>

<mqtt connection>.driver (required)

mqtt: The connection must set driver=mqtt for MQTT connections.

<mqtt connection>.server Default: localhost: MQTT server hostname or address to use.

<mqtt connection>.port Default: 1883: MQTT server port.

<mqtt connection>.keepalive Default: 60: Maximum period in seconds allowed between communications with the broker.

<mqtt connection>.user: Set a username for optional broker authentication.

<mqtt connection>.password: Set a password for optional broker authentication.

<mqtt connection>.ca_certs: A string path to the Certificate Authority certificate files to enable TLS connection.

<mqtt connection>.certfile: A strings pointing to the PEM encoded client certificate to enable client TLS based authentication. This option requires keyfile to be set too.

<mqtt connection>.keyfile: A strings pointing to the PEM encoded client private keys to enable client TLS based authentication. This option requires certfile to be set too.

<mqtt connection>.ciphers: A string specifying which encryption ciphers are allowable for this connection. More information in this openssl doc.

Reporter Configuration

A connection that uses the mqtt driver must be supplied to the reporter. Each pipeline must provide a topic name. For example:

- pipeline:
    name: check
    success:
      mqtt:
        topic: "{tenant}/zuul/{pipeline}/{project}/{branch}/{change}"
        qos: 2

pipeline.<reporter>.<mqtt>

To report via MQTT message, the dictionaries passed to any of the pipeline reporter support the following attributes:

pipeline.<reporter>.<mqtt>.topic: The MQTT topic to publish messages. The topic can be a format string that can use the following parameters: tenant, pipeline, project, branch, change, patchset and ref. MQTT topic can have hierarchy separated by /, more details in this doc

pipeline.<reporter>.<mqtt>.qos Default: 0: The quality of service level to use, it can be 0, 1 or 2. Read more in this guide

pipeline.<reporter>.<mqtt>.include-returned-data Default: false: If set to true, Zuul will include any data returned from the job via Return Values.

Reporter Message Schema

An MQTT report uses this schema:

<mqtt schema>

<mqtt schema>.uuid: The item UUID. Each item enqueued into a Zuul pipeline is assigned a UUID which remains the same even as Zuul’s speculative execution algorithm re-orders pipeline contents.

<mqtt schema>.action: The reporter action name, e.g.: ‘start’, ‘success’, ‘failure’, ‘merge-conflict’, …

<mqtt schema>.tenant: The tenant name.

<mqtt schema>.pipeline: The pipeline name.

<mqtt schema>.project: The project name.

<mqtt schema>.branch: The branch name.

<mqtt schema>.change_url: The change url.

<mqtt schema>.message: The report message.

<mqtt schema>.change: The change number.

<mqtt schema>.patchset: The patchset number.

<mqtt schema>.commit_id: The commit id number.

<mqtt schema>.owner: The owner username of the change.

<mqtt schema>.ref: The change reference.

<mqtt schema>.zuul_ref: The internal zuul change reference.

<mqtt schema>.trigger_time: The timestamp when the event was added to the scheduler.

<mqtt schema>.enqueue_time: The timestamp when the event was added to the pipeline.

<mqtt schema>.buildset

The buildset information.

uuid

The buildset global uuid.

<mqtt schema>.buildset.result

The buildset result

<mqtt schema>.buildset.builds

The list of builds.

<mqtt schema>.buildset.builds.job_name: The job name.

<mqtt schema>.buildset.builds.voting: The job voting status.

<mqtt schema>.buildset.builds.uuid: The build uuid (not present in start report).

<mqtt schema>.buildset.builds.execute_time: The build execute time.

<mqtt schema>.buildset.builds.start_time: The build start time (not present in start report).

<mqtt schema>.buildset.builds.end_time: The build end time (not present in start report).

<mqtt schema>.buildset.builds.log_url: The build log url (not present in start report).

<mqtt schema>.buildset.builds.web_url: The url to the build result page. Not present in start report.

<mqtt schema>.buildset.builds.result: The build results (not present in start report).

<mqtt schema>.buildset.builds.artifacts Type: list

The build artifacts (not present in start report).

This is a list of dictionaries corresponding to the returned artifacts.

<mqtt schema>.buildset.builds.artifacts.name: The name of the artifact.

<mqtt schema>.buildset.builds.artifacts.url: The url of the artifact.

<mqtt schema>.buildset.builds.artifacts.metadata Type: dict: The metadata of the artifact. This is a dictionary of arbitrary key values determined by the job.

Here is an example of a start message:

{
  'action': 'start',
  'tenant': 'openstack.org',
  'pipeline': 'check',
  'project': 'sf-jobs',
  'branch': 'master',
  'change_url': 'https://gerrit.example.com/r/3',
  'message': 'Starting check jobs.',
  'trigger_time': '1524801056.2545864',
  'enqueue_time': '1524801093.5689457',
  'change': '3',
  'patchset': '1',
  'commit_id': '2db20c7fb26adf9ac9936a9e750ced9b4854a964',
  'owner': 'username',
  'ref': 'refs/changes/03/3/1',
  'zuul_ref': 'Zf8b3d7cd34f54cb396b488226589db8f',
  'buildset': {
    'uuid': 'f8b3d7cd34f54cb396b488226589db8f',
    'builds': [{
      'job_name': 'linters',
      'voting': True
    }],
  },
}

Here is an example of a success message:

{
  'action': 'success',
  'tenant': 'openstack.org',
  'pipeline': 'check',
  'project': 'sf-jobs',
  'branch': 'master',
  'change_url': 'https://gerrit.example.com/r/3',
  'message': 'Build succeeded.',
  'trigger_time': '1524801056.2545864',
  'enqueue_time': '1524801093.5689457',
  'change': '3',
  'patchset': '1',
  'commit_id': '2db20c7fb26adf9ac9936a9e750ced9b4854a964',
  'owner': 'username',
  'ref': 'refs/changes/03/3/1',
  'zuul_ref': 'Zf8b3d7cd34f54cb396b488226589db8f',
  'buildset': {
    'uuid': 'f8b3d7cd34f54cb396b488226589db8f',
    'builds': [{
      'job_name': 'linters',
      'voting': True
      'uuid': '16e3e55aca984c6c9a50cc3c5b21bb83',
      'execute_time': 1524801120.75632954,
      'start_time': 1524801179.8557224,
      'end_time': 1524801208.928095,
      'log_url': 'https://logs.example.com/logs/3/3/1/check/linters/16e3e55/',
      'web_url': 'https://tenant.example.com/t/tenant-one/build/16e3e55aca984c6c9a50cc3c5b21bb83/',
      'result': 'SUCCESS',
      'dependencies': [],
      'artifacts': [],
    }],
  },
}

Trigger Configuration

pipeline.trigger.<mqtt trigger>

The dictionary passed to the MQTT pipeline trigger attribute supports the following attributes:

pipeline.trigger.<mqtt trigger>.topic (required) Type: str

An MQTT topic subscription definition for which any matching mqtt event topics will trigger evaluation for Zuul triggering.

The MQTT driver will not subscribe to any events until at least one MQTT trigger is configured with a topic subscription. This means that users of the driver only for reporting purposes will not subscribe to any events.

pipeline.trigger.<mqtt trigger>.required-payload Type: dict: A dictionary of jsonpath expression keys to json primitive values (int, float, bool, string, null) or regular expressions values. All jsonpath keys must be found in the MQTT event payload and have matching values in the MQTT payload. If these keys are not found or the values do not match then the MQTT event is filtered out and does not trigger the pipeline.

pipeline.trigger.<mqtt trigger>.data-mapping (required) Type: dict

Because MQTT is arbitrary and may differ from source to source on the same message bus we need to map the data in the MQTT event payload to data that Zuul knows how to handle. In particular Zuul needs two major pieces of information.

First is the source of the “change” event. To lookup source info we need to map either project-canonical-name or project-hostname and project from the MQTT payload to Zuul. To do that add key value entries where the keys are project-canonical-name or project-hostname and project and the values are jsonpath expressions. These jsonpath expressions will be looked up in the payload and the resulting value is assigned to project-canonical-name etc.

Next we need to know what “change” should be enqueued into the pipeline. For this information we need to map either a ref or change (with optional patchset) from MQTT event payload into Zuul. To do this add key value entries where the keys are ref or change and the values are jsonpath expressions. These jsonpath expressions will be looked up in the payload and the resulting value is assigned to ref etc.

If any of these jsonpath expressions fail the entire MQTT event is ignored. If a minimal subset of these required attributes is not properly mapped then the entire MQTT event is ignored.

pipeline.trigger.<mqtt trigger>.debug Default: False Type: bool: Because MQTT payload data and how jsonpath may apply against that payload is somewhat opqaue triggers can be annotated with the debug flag to add additional debug logging around these events. Note that this logging occurs in the Zuul scheduler and won’t be exposed to end users.

For example:

- topic: some/topic/path
  required-payload:
    "$.action": enqueue
  data-mapping:
    project: "$.project_name"
    project-hostname: "$.project_hostname"
    change: "$.change"
    patchset: "$.patchset"