Zuul Admin Client
Zuul includes a simple command line client that may be used to affect Zuul’s behavior while running.
Note
For operations related to normal workflow like enqueue, dequeue, autohold and promote, the zuul-client CLI should be used instead.
Configuration
The client uses the same zuul.conf file as the server, and will look for it in the same locations if not specified on the command line.
Usage
The general options that apply to all subcommands are:
usage: zuul-admin [-h] [-c CONFIG] [--version] [-v] [--auth-token AUTH_TOKEN]
[--zuul-url ZUUL_URL] [--insecure]
{autohold,autohold-delete,autohold-info,autohold-list,enqueue,enqueue-ref,dequeue,promote,show,tenant-conf-check,create-auth-token,import-keys,export-keys,copy-keys,delete-keys,delete-state,delete-pipeline-state,prune-database}
...
Zuul CLI client.
options:
-h, --help show this help message and exit
-c CONFIG specify the config file
--version show zuul version
-v verbose output
--auth-token AUTH_TOKEN
Authentication Token, needed if using theREST API
--zuul-url ZUUL_URL Zuul API URL, needed if using the REST API without a
configuration file
--insecure Do not verify SSL connection to Zuul, when using the
REST API (Defaults to False)
commands:
valid commands
{autohold,autohold-delete,autohold-info,autohold-list,enqueue,enqueue-ref,dequeue,promote,show,tenant-conf-check,create-auth-token,import-keys,export-keys,copy-keys,delete-keys,delete-state,delete-pipeline-state,prune-database}
additional help
autohold [DEPRECATED - use zuul-client] hold nodes for failed
job
autohold-delete [DEPRECATED - use zuul-client] delete autohold request
autohold-info [DEPRECATED - use zuul-client] retrieve autohold
request detailed info
autohold-list [DEPRECATED - use zuul-client] list autohold requests
enqueue [DEPRECATED - use zuul-client] enqueue a change
enqueue-ref [DEPRECATED - use zuul-client] enqueue a ref
dequeue [DEPRECATED - use zuul-client] dequeue a buildset by
its change or ref
promote [DEPRECATED - use zuul-client] promote one or more
changes
show [DEPRECATED - use zuul-client] show current statuses
tenant-conf-check validate the tenant configuration
create-auth-token create an Authentication Token for the web API
import-keys import project keys to ZooKeeper
export-keys export project keys from ZooKeeper
copy-keys copy keys from one project to another
delete-keys delete project keys
delete-state delete ephemeral ZooKeeper state
delete-pipeline-state
delete single pipeline ZooKeeper state
prune-database prune old database entries
The following subcommands are supported:
tenant-conf-check
usage: zuul-admin tenant-conf-check [-h]
options:
-h, --help show this help message and exit
Example:
zuul-admin tenant-conf-check
This command validates the tenant configuration schema. It exits ‘-1’ in case of errors detected.
create-auth-token
Note
This command is only available if an authenticator is configured in
zuul.conf. Furthermore the authenticator’s configuration must
include a signing secret.
usage: zuul-admin create-auth-token [-h] --auth-config AUTH_CONFIG --tenant
TENANT --user USER
[--expires-in EXPIRES_IN]
Create an Authentication Token for the administration web API
Create a bearer token that can be used to access Zuul's
administration web API. This is typically used to delegate
privileged actions such as enqueueing and autoholding to
third parties, scoped to a single tenant.
At least one authenticator must be configured with a secret
that can be used to sign the token.
options:
-h, --help show this help message and exit
--auth-config AUTH_CONFIG
The authenticator to use. Must match an authenticator
defined in zuul's configuration file.
--tenant TENANT tenant name
--user USER The user's name. Used for traceability in logs.
--expires-in EXPIRES_IN
Token validity duration in seconds (default: 600)
Example:
zuul-admin create-auth-token --auth-config zuul-operator --user alice --tenant tenantA --expires-in 1800
The return value is the value of the Authorization header the user must set
when querying a protected endpoint on Zuul’s REST API.
Example:
bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJodHRwOi8vbWFuYWdlc2Yuc2ZyZG90ZXN0aW5zdGFuY2Uub3JnIiwienV1bC50ZW5hbnRzIjp7ImxvY2FsIjoiKiJ9LCJleHAiOjE1Mzc0MTcxOTguMzc3NTQ0fQ.DLbKx1J84wV4Vm7sv3zw9Bw9-WuIka7WkPQxGDAHz7s
export-keys
usage: zuul-admin export-keys [-h] path
Export project secret keys from ZooKeeper
This command exports project secret keys from ZooKeeper
and writes them to a file which is suitable for backing
up and later use with the import-keys command.
The key contents are still protected by the keystore
password and can not be used or decrypted without it.
positional arguments:
path key export file path
options:
-h, --help show this help message and exit
Example:
zuul-admin export-keys /var/backup/zuul-keys.json
import-keys
usage: zuul-admin import-keys [-h] [--force] path
Import previously exported project secret keys to ZooKeeper
Given a file with previously exported project keys, this
command will import them into ZooKeeper. Existing keys
will not be overwritten; to overwrite keys, add the
--force flag.
positional arguments:
path key export file path
options:
-h, --help show this help message and exit
--force overwrite existing keys
Example:
zuul-admin import-keys /var/backup/zuul-keys.json
copy-keys
usage: zuul-admin copy-keys [-h]
src_connection src_project dest_connection
dest_project
Copy secret keys from one project to another
When projects are renamed, this command may be used to
copy the secret keys from the current name to the new name
in order to avoid service interruption.
positional arguments:
src_connection original connection name
src_project original project name
dest_connection new connection name
dest_project new project name
options:
-h, --help show this help message and exit
Example:
zuul-admin copy-keys gerrit old_project gerrit new_project
delete-keys
usage: zuul-admin delete-keys [-h] connection project
Delete the ssh and secrets keys for a project
positional arguments:
connection connection name
project project name
options:
-h, --help show this help message and exit
Example:
zuul-admin delete-keys gerrit old_project
delete-state
usage: zuul-admin delete-state [-h]
Delete all ephemeral state stored in ZooKeeper
Zuul stores a considerable amount of ephemeral state
information in ZooKeeper. Generally it should be able to
detect and correct any errors, but if the state becomes
corrupted and it is unable to recover, this command may be
used to delete all ephemeral data from ZooKeeper and start
anew.
Do not run this command while any Zuul component is
running (perform a complete shutdown first).
This command will only remove ephemeral Zuul data from
ZooKeeper; it will not remove private keys or Nodepool
data.
options:
-h, --help show this help message and exit
Example:
zuul-admin delete-state
delete-pipeline-state
usage: zuul-admin delete-pipeline-state [-h] tenant pipeline
Delete a single pipeline state stored in ZooKeeper
In the unlikely event that a bug in Zuul or ZooKeeper data
corruption occurs in such a way that it only affects a
single pipeline, this command might be useful in
attempting to recover.
The circumstances under which this command will be able to
effect a recovery are very rare and even so it may not be
sufficient. In general, if an error occurs it is better
to shut Zuul down and run "zuul delete-state".
This command will lock the specified tenant and
then completely delete the pipeline state.
positional arguments:
tenant tenant name
pipeline pipeline name
options:
-h, --help show this help message and exit
Example:
zuul-admin delete-pipeline-state tenant pipeline
prune-database
usage: zuul-admin prune-database [-h] [--before BEFORE]
[--older-than OLDER_THAN]
Prune old database entries
This command will delete database entries older than the
specified cutoff (which can be specified as either an
absolute or relative time).
options:
-h, --help show this help message and exit
--before BEFORE absolute timestamp (e.g., "2022-01-31 12:00:00")
--older-than OLDER_THAN
relative time (e.g., "24h" or "180d")
Example:
zuul-admin prune-database --older-than 180d
Deprecated commands
The following commands are deprecated in the zuul-admin CLI, and thus may not be entirely supported in Zuul’s current version. They will be removed in a future release of Zuul. They can still be performed via the zuul-client CLI. Please refer to zuul-client’s documentation for more details.
In order to run these commands, the webclient section is required in the configuration file.
It is also possible to run the client without a configuration file, by using the
--zuul-url option to specify the base URL of the Zuul web server.
Autohold
usage: zuul-admin autohold [-h] --tenant TENANT --project PROJECT --job JOB
[--change CHANGE] [--ref REF] --reason REASON
[--count COUNT]
[--node-hold-expiration NODE_HOLD_EXPIRATION]
options:
-h, --help show this help message and exit
--tenant TENANT tenant name
--project PROJECT project name
--job JOB job name
--change CHANGE specific change to hold nodes for
--ref REF git ref to hold nodes for
--reason REASON reason for the hold
--count COUNT number of job runs (default: 1)
--node-hold-expiration NODE_HOLD_EXPIRATION
how long in seconds should the node set be in HOLD
status (default: scheduler's default_hold_expiration
value)
Example:
zuul-admin autohold --tenant openstack --project example_project --job example_job --reason "reason text" --count 1
Autohold Delete
usage: zuul-admin autohold-delete [-h] REQUEST_ID
positional arguments:
REQUEST_ID the hold request ID
options:
-h, --help show this help message and exit
Example:
zuul-admin autohold-delete --id 0000000123
Autohold Info
usage: zuul-admin autohold-info [-h] REQUEST_ID
positional arguments:
REQUEST_ID the hold request ID
options:
-h, --help show this help message and exit
Example:
zuul-admin autohold-info --id 0000000123
Autohold List
usage: zuul-admin autohold-list [-h] --tenant TENANT
options:
-h, --help show this help message and exit
--tenant TENANT tenant name
Example:
zuul-admin autohold-list --tenant openstack
Dequeue
usage: zuul-admin dequeue [-h] --tenant TENANT --pipeline PIPELINE --project
PROJECT [--change CHANGE] [--ref REF]
options:
-h, --help show this help message and exit
--tenant TENANT tenant name
--pipeline PIPELINE pipeline name
--project PROJECT project name
--change CHANGE change id
--ref REF ref name
Examples:
zuul-admin dequeue --tenant openstack --pipeline check --project example_project --change 5,1
zuul-admin dequeue --tenant openstack --pipeline periodic --project example_project --ref refs/heads/master
Enqueue
usage: zuul-admin enqueue [-h] --tenant TENANT [--trigger TRIGGER] --pipeline
PIPELINE --project PROJECT --change CHANGE
options:
-h, --help show this help message and exit
--tenant TENANT tenant name
--trigger TRIGGER trigger name (deprecated and ignored. Kept only for
backward compatibility)
--pipeline PIPELINE pipeline name
--project PROJECT project name
--change CHANGE change id
Example:
zuul-admin enqueue --tenant openstack --trigger gerrit --pipeline check --project example_project --change 12345,1
Note that the format of change id is <number>,<patchset>.
Enqueue-ref
usage: zuul-admin enqueue-ref [-h] --tenant TENANT [--trigger TRIGGER]
--pipeline PIPELINE --project PROJECT --ref REF
[--oldrev OLDREV] [--newrev NEWREV]
Submit a trigger event
Directly enqueue a trigger event. This is usually used
to manually "replay" a trigger received from an external
source such as gerrit.
options:
-h, --help show this help message and exit
--tenant TENANT tenant name
--trigger TRIGGER trigger name
--pipeline PIPELINE pipeline name
--project PROJECT project name
--ref REF ref name
--oldrev OLDREV old revision
--newrev NEWREV new revision
This command is provided to manually simulate a trigger from an
external source. It can be useful for testing or replaying a trigger
that is difficult or impossible to recreate at the source. The
arguments to enqueue-ref will vary depending on the source and
type of trigger. Some familiarity with the arguments emitted by
gerrit update hooks
such as patchset-created and ref-updated is recommended. Some
examples of common operations are provided below.
Manual enqueue examples
It is common to have a release pipeline that listens for new tags
coming from gerrit and performs a range of code packaging jobs.
If there is an unexpected issue in the release jobs, the same tag can
not be recreated in gerrit and the user must either tag a new
release or request a manual re-triggering of the jobs. To re-trigger
the jobs, pass the failed tag as the ref argument and set
newrev to the change associated with the tag in the project
repository (i.e. what you see from git show X.Y.Z):
zuul-admin enqueue-ref --tenant openstack --trigger gerrit --pipeline release --project openstack/example_project --ref refs/tags/X.Y.Z --newrev abc123..
The command can also be used asynchronosly trigger a job in a
periodic pipeline that would usually be run at a specific time by
the timer driver. For example, the following command would
trigger the periodic jobs against the current master branch
top-of-tree for a project:
zuul-admin enqueue-ref --tenant openstack --trigger timer --pipeline periodic --project openstack/example_project --ref refs/heads/master
Another common pipeline is a post queue listening for gerrit
merge results. Triggering here is slightly more complicated as you
wish to recreate the full ref-updated event from gerrit. For
a new commit on master, the gerrit ref-updated trigger
expresses “reset refs/heads/master for the project from oldrev
to newrev” (newrev being the committed change). Thus to
replay the event, you could git log in the project and take the
current HEAD and the prior change, then enqueue the event:
NEW_REF=$(git rev-parse HEAD)
OLD_REF=$(git rev-parse HEAD~1)
zuul-admin enqueue-ref --tenant openstack --trigger gerrit --pipeline post --project openstack/example_project --ref refs/heads/master --newrev $NEW_REF --oldrev $OLD_REF
Note that zero values for oldrev and newrev can indicate
branch creation and deletion; the source code is the best reference
for these more advanced operations.
Promote
usage: zuul-admin promote [-h] --tenant TENANT --pipeline PIPELINE --changes
CHANGES [CHANGES ...]
options:
-h, --help show this help message and exit
--tenant TENANT tenant name
--pipeline PIPELINE pipeline name
--changes CHANGES [CHANGES ...]
change ids
Example:
zuul-admin promote --tenant openstack --pipeline gate --changes 12345,1 13336,3
Note that the format of changes id is <number>,<patchset>.
The promote action is used to reorder the changes in a pipeline, by putting the provided changes at the top of the queue.
The most common use case for the promote action is the need to merge an urgent fix when the gate pipeline has several patches queued ahead. This is especially needed if there is concern that one or more changes ahead in the queue may fail, thus increasing the time to land for the fix; or concern that the fix may not pass validation if applied on top of the current patch queue in the gate.
Any items in a dependent pipeline which have had items ahead of them changed will have their jobs canceled and restarted based on the new ordering.
If items in independent pipelines are promoted, no jobs will be restarted, but their change queues within the pipeline will be re-ordered so that they will be processed first and their node request priorities will increase.