Remove Jenkins documentation; replace with zuul-launcher

Change-Id: Ie6d077906a4d6e204d4cd1847486dc098de2f8f6
This commit is contained in:
James E. Blair 2016-06-15 13:42:23 -07:00
parent 5690f9971b
commit 6af492e20d
5 changed files with 42 additions and 267 deletions

View File

@ -1,217 +0,0 @@
:title: Jenkins
.. _jenkins:
Jenkins
#######
Jenkins is a Continuous Integration system that runs tests and
automates some parts of project operations. It is controlled for the
most part by :ref:`zuul` which determines what jobs are run when.
At a Glance
===========
:Hosts:
* http://jenkins.openstack.org
* http://jenkins-dev.openstack.org
:Puppet:
* https://git.openstack.org/cgit/openstack-infra/puppet-jenkins/tree/
* :file:`modules/openstack_project/manifests/jenkins.pp`
* :file:`modules/openstack_project/manifests/jenkins_dev.pp`
:Configuration:
* :config:`jenkins/jobs`
:Projects:
* http://jenkins-ci.org/
* :ref:`zuul`
* :ref:`jjb`
:Bugs:
* https://storyboard.openstack.org/#!/project/748
* https://wiki.jenkins-ci.org/display/JENKINS/Issue+Tracking
:Resources:
* :ref:`zuul`
* :ref:`jjb`
Overview
========
A large number and variety of jobs are defined in Jenkins. The
configuration of all of those jobs is stored in git in the
openstack-infra/project-config repository. They are defined in YAML
files that are read by :ref:`jjb` which configures the actual jobs in
Jenkins.
Anyone may submit a change to the openstack-infra/project-config
repository that defines a new job or alters an existing job by editing
the appropriate YAML files. See :ref:`jjb` for more information.
Because of the large number of builds that Jenkins executes, the
OpenStack project favors the following approach in configuring Jenkins
jobs:
* Minimal use of plugins: the more post-processing work that Jenkins
needs to perform on a job, the more likely we are to run into
compatibility problems among plugins, and contention for shared
resources on the Jenkins master. A number of popular plugins
will cause all builds of a job to be serialized even if the jobs
otherwise run in parallel.
* Minimal build history: Jenkins stores build history in individual
XML files on disk, and accessing a large build history can cause
the Jenkins master to be unresponsive for a significant time while
loading them. It also increases memory usage. Instead, we
generally keep no more than a day's worth of builds.
* Move artifacts off of Jenkins: Jenkins is not efficient at serving
static information such as build artifacts (e.g., tarballs) or
logs. Instead, we copy them to a static webserver which is far
more efficient.
Authorization
=============
Jenkins is set up to use OpenID in a Single Sign On mode with Launchpad.
This means that all of the user and group information is managed via
Launchpad users and teams. In the Jenkins Security Matrix, a Launchpad team
name can be specified and any members of that team will be granted those
permissions. However, because of the way the information is processed, a
user will need to re-log in upon changing either team membership on
Launchpad, or changing that team's authorization in Jenkins for the new
privileges to take effect.
Devstack Gate
=============
OpenStack integration testing is performed by the devstack gate test
framework. This framework runs the devstack exercises and Tempest
smoketests against a devstack install on single use cloud servers. The
devstack gate source can be found on `git.openstack.org
<https://git.openstack.org/cgit/openstack-infra/devstack-gate>`_ and the `Readme
<https://git.openstack.org/cgit/openstack-infra/devstack-gate/tree/README.rst>`_
describes the process of using devstack gate to run your own devstack
based tests.
For management of the devstack and other instances, a tool called
:ref:`nodepool` creates and deletes Jenkins slaves as needed in order to
maintain the pool.
Sysadmin
========
Jenkins is largely hidden, and has no sensitive data exposed
publicly, so we use self-signed certs for Jenkins masters.
After bringing up a jenkins node (16G memory instance if you use the
stock jenkins.default) with puppet, log in and configure Jenkins by
hand:
#. Configure the site so it knows it's correct url.
(Jenkins URL in global config). This is needed to complete an SSO
sign-in.
#. Configure the OpenID plugin for your SSO site (e.g. Launchpad)
#. Do not set CSRF protection - that causes problems with various components
such as nodepool and swift log uploader.
#. Login.
#. Setup matrix security: add the 'authenticated' pseudo user and
grant Admin access to your own user.
#. Setup one account per `http://docs.openstack.org/infra/jenkins-job-builder/installation.html#configuration-file`
and grab the API token for it.
#. Configure the number of executors you want on the Jenkins Master
(e.g. 1)
#. Configure a maven environment (if you have Maven projects to test).
#. Enable the gearman plugin globally. Your gearman server is
zuul.$project. If Test Connection fails, do a puppet run (puppet
agent --test) on the zuul machine, as gearman wouldn't have started
with no workers configured.
#. Configure the timestamper plugin. E.g. to
'<b>'yyyy-MM-dd HH:mm:ss'</b> '
#. Enable the zmq plugin globally if it is visible. No settings were
visible when writing this doc.
#. You will configure global scp and ftp credentials for static and
docs sites respectively later, but as we haven't setup those sites
yet, that's not possible :).
Puppet takes care of the rest.
Quirks
------
Note that jenkins talks to its slaves via ssh, the
modules/openstack_project/manifests/init.pp file contains the ssh
public key that puppet installs on the slaves.
Slaves
------
Statically provisioned slaves have labels assigned by hand. E.g.
centos6, and are added to a chosen Jenkins master by hand. Adding a
slave is then:
#. Launch a slave
#. Add it to Jenkins
Add your jenkins master key for the credentials (make it global,
one-time operation).
Set the jenkins home to /home/jenkins
#. Set appropriate labels on it
#. Profit!
Safe Master Restarts
====================
Jenkins masters periodically leak threads reducing their job
throughput and eventually leading to crashes. We work around this
by performing weekly rolling restarts of the Jenkins masters with
an ansible playbook.
If you need to perform a safe restart against a single master you
can do this by running the same playbook and limiting it to a
specific jenkins master
To do this::
root@puppetmaster# ansible-playbook -f1 --limit $server_fqdn \
/opt/system-config/production/playbooks/restart_jenkins_masters.yaml \
--extra-vars "user=hudson-openstack \
password=$(/opt/system-config/production/tools/hieraedit.py \
--yaml /etc/puppet/hieradata/production/fqdn/nodepool.openstack.org.yaml jenkins_api_key)"
Consider running this in screen as the worst case run time is as
long as our longest running job.
How to manually run jenkins job builder
=======================================
Jenkins job builder may need to be run manually under certain situations. If the expected
jobs are not being created in jenkins masters, running jjb manually on the masters where
it failed is suggested. To do this::
user@jenkins01# sudo -H jenkins-jobs --conf /etc/jenkins_jobs/jenkins_jobs.ini \
update --delete-old /etc/jenkins_jobs/config
Consider running this in screen as the worst case run time can be of several hours.
In the case of incorrect jobs configuration caused by some jjb malfunction, all jobs
will need to be regenerated. As jjb uses a local cache, to force the regeneration
of all jobs, the cache needs to be ignored. To do this::
user@jenkins01# sudo -H jenkins-jobs --ignore-cache --conf \
/etc/jenkins_jobs/jenkins_jobs.ini update --delete-old /etc/jenkins_jobs/config
In order to speed up the massive job reconfiguration, it may be desired to set jenkins
on shutdown mode, visiting this link::
`https://jenkins[xx].openstack.org/quietDown`
And make Jenkins alive again after job reconfiguration finishes.

View File

@ -12,10 +12,7 @@ At a Glance
=========== ===========
:Hosts: :Hosts:
* http://jenkins.openstack.org * http://zm*.openstack.org
* http://jenkins-dev.openstack.org
:Puppet:
* https://git.openstack.org/cgit/openstack-infra/puppet-jenkins/tree/manifests/job_builder.pp
:Configuration: :Configuration:
* :config:`jenkins/jobs/` * :config:`jenkins/jobs/`
:Projects: :Projects:
@ -228,44 +225,11 @@ values.
Notice that example1 makes use of the job group and example2 makes use of the Notice that example1 makes use of the job group and example2 makes use of the
job template. job template.
Job Caching Zuul
----------- ====
The Jenkins Jobs builder maintains a special `cache`_ that In our environment, we no longer use Jenkins to execute jobs. Zuul
contains an MD5 of every generated XML that it builds. If itself, via Ansible, runs the actual workload. Zuul reads JJB config
it finds the XML is different then it will proceed to send this files in order to define its jobs, so, aside from the detail of not
to Jenkins, otherwise it is skipped. If a job is accidentally deleted actually using Jenkins or creating any jobs in it, the use of JJB to
then this file should be modified or removed. configure jobs in Zuul is the same.
.. _cache: http://docs.openstack.org/infra/jenkins-job-builder/installation.html#running
Sending a Job to Jenkins
------------------------
The Jenkins Jobs builder talks to Jenkins using the Jenkins API. This
means that it can create and modify jobs directly without the need to
restart or reload the Jenkins server. It also means that Jenkins will
verify the XML and cause the Jenkins Jobs builder to fail if there is
a problem.
For this to work a configuration file is needed. There is an erb
template for this configuration file at
:file:`modules/jenkins/templates/jenkins_jobs.ini.erb`. The contents
of this template are:
.. code-block:: ini
[jenkins]
user=<%= username %>
password=<%= password %>
url=<%= url %>
The values for user and url are hardcoded in the Puppet repo in
:file:`modules/openstack_project/manifests/jenkins.pp`, but the
password is stored in hiera. Make sure you have it defined as
``jenkins_jobs_password`` in the hiera DB.
The password can be obtained by logging into the Jenkins user,
clicking on your username in the top-right, clicking on `Configure`
and then `Show API Token`. This API Token is your password for the
API.

View File

@ -309,7 +309,7 @@ Stage 5 - Jenkins Master(s)
~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~
For Zuul to schedule work, it needs one or more Gearman connected Jenkins For Zuul to schedule work, it needs one or more Gearman connected Jenkins
masters. See :ref:`jenkins` for details. masters.
The minimum setup is one master, but if you will be permitting any code The minimum setup is one master, but if you will be permitting any code
submitter to trigger test runs, we recommend having two: one untrusted and one submitter to trigger test runs, we recommend having two: one untrusted and one
@ -360,7 +360,7 @@ which get the most load (as they run jobs from anyone).
#. Launch the node with a size larger than the jenkins size you specified. #. Launch the node with a size larger than the jenkins size you specified.
#. Setup Jenkins per :ref:`jenkins`. #. Setup Jenkins.
At this stage doing a 'recheck' should still report LOST on a change. At this stage doing a 'recheck' should still report LOST on a change.
But in the zuul debug.log in /var/log/zuul you should see a 'build xxx not But in the zuul debug.log in /var/log/zuul you should see a 'build xxx not

View File

@ -10,7 +10,6 @@ Major Systems
gerrit gerrit
grafana grafana
grafyaml grafyaml
jenkins
zuul zuul
jjb jjb
logstash logstash

View File

@ -15,6 +15,8 @@ At a Glance
* http://status.openstack.org/zuul * http://status.openstack.org/zuul
* http://zuul.openstack.org * http://zuul.openstack.org
* http://zuul-dev.openstack.org * http://zuul-dev.openstack.org
* zm*.openstack.org
* zl*.openstack.org
:Puppet: :Puppet:
* https://git.openstack.org/cgit/openstack-infra/puppet-zuul/tree/ * https://git.openstack.org/cgit/openstack-infra/puppet-zuul/tree/
* :file:`modules/openstack_project/manifests/zuul_prod.pp` * :file:`modules/openstack_project/manifests/zuul_prod.pp`
@ -35,12 +37,11 @@ The OpenStack project uses a number of pipelines in Zuul:
**check** **check**
Newly uploaded patchsets enter this pipeline to receive an initial Newly uploaded patchsets enter this pipeline to receive an initial
+/-1 Verified vote from Jenkins. +/-1 Verified vote.
**gate** **gate**
Changes that have been approved by core developers are enqueued in Changes that have been approved by core developers are enqueued in
order in this pipeline, and if they pass tests in Jenkins, will be order in this pipeline, and if they pass tests, will be merged.
merged.
**post** **post**
This pipeline runs jobs that operate after each change is merged. This pipeline runs jobs that operate after each change is merged.
@ -146,3 +147,31 @@ before the restart::
You may watch the `Zuul Status Page You may watch the `Zuul Status Page
<http://status.openstack.org/zuul/>`_ to confirm that changes are <http://status.openstack.org/zuul/>`_ to confirm that changes are
returning to the queues. returning to the queues.
Mergers
-------
Servers with names matching the pattern zm*.openstack.org are Zuul
Mergers. These are horizontally scalable components of Zuul which
perform git operations for the benefit of jobs. They serve git
repositories via Apache over http, and jobs fetch changes to test from
them. They can be started and stopped at will, and new ones added as
necessary to accommodate load. If you remove a merger, be sure to
leave Apache running for several hours until the last job that may
have been launched with instructions to fetch from that merger has
completed.
Launchers
---------
We use an Ansible based launcher in Zuul to actually run jobs. This
component runs on a horizontally scalable set of servers named
zl*.openstack.org. It reads job configuration from Jenkins Job
Builder files in the project-config repository and translates that
into Ansible playbooks which it runs on our workers. Our jobs are
configured to upload as much information as possible along with their
logs, but if there is an error which can not be diagnosed in that
manner, Ansible logs are available in the launcher-debug log file on
the launcher host. You may use the Zuul build UUID to track
assignment of a given job from the Zuul scheduler to the Zuul launcher
used by that job.