From 6af492e20d1de7f40d66bbe437bb85898859607d Mon Sep 17 00:00:00 2001 From: "James E. Blair" Date: Wed, 15 Jun 2016 13:42:23 -0700 Subject: [PATCH] Remove Jenkins documentation; replace with zuul-launcher Change-Id: Ie6d077906a4d6e204d4cd1847486dc098de2f8f6 --- doc/source/jenkins.rst | 217 -------------------------------- doc/source/jjb.rst | 52 ++------ doc/source/running-your-own.rst | 4 +- doc/source/systems.rst | 1 - doc/source/zuul.rst | 35 +++++- 5 files changed, 42 insertions(+), 267 deletions(-) delete mode 100644 doc/source/jenkins.rst diff --git a/doc/source/jenkins.rst b/doc/source/jenkins.rst deleted file mode 100644 index 24f70c8165..0000000000 --- a/doc/source/jenkins.rst +++ /dev/null @@ -1,217 +0,0 @@ -:title: Jenkins - -.. _jenkins: - -Jenkins -####### - -Jenkins is a Continuous Integration system that runs tests and -automates some parts of project operations. It is controlled for the -most part by :ref:`zuul` which determines what jobs are run when. - -At a Glance -=========== - -:Hosts: - * http://jenkins.openstack.org - * http://jenkins-dev.openstack.org -:Puppet: - * https://git.openstack.org/cgit/openstack-infra/puppet-jenkins/tree/ - * :file:`modules/openstack_project/manifests/jenkins.pp` - * :file:`modules/openstack_project/manifests/jenkins_dev.pp` -:Configuration: - * :config:`jenkins/jobs` -:Projects: - * http://jenkins-ci.org/ - * :ref:`zuul` - * :ref:`jjb` -:Bugs: - * https://storyboard.openstack.org/#!/project/748 - * https://wiki.jenkins-ci.org/display/JENKINS/Issue+Tracking -:Resources: - * :ref:`zuul` - * :ref:`jjb` - -Overview -======== - -A large number and variety of jobs are defined in Jenkins. The -configuration of all of those jobs is stored in git in the -openstack-infra/project-config repository. They are defined in YAML -files that are read by :ref:`jjb` which configures the actual jobs in -Jenkins. - -Anyone may submit a change to the openstack-infra/project-config -repository that defines a new job or alters an existing job by editing -the appropriate YAML files. See :ref:`jjb` for more information. - -Because of the large number of builds that Jenkins executes, the -OpenStack project favors the following approach in configuring Jenkins -jobs: - - * Minimal use of plugins: the more post-processing work that Jenkins - needs to perform on a job, the more likely we are to run into - compatibility problems among plugins, and contention for shared - resources on the Jenkins master. A number of popular plugins - will cause all builds of a job to be serialized even if the jobs - otherwise run in parallel. - * Minimal build history: Jenkins stores build history in individual - XML files on disk, and accessing a large build history can cause - the Jenkins master to be unresponsive for a significant time while - loading them. It also increases memory usage. Instead, we - generally keep no more than a day's worth of builds. - * Move artifacts off of Jenkins: Jenkins is not efficient at serving - static information such as build artifacts (e.g., tarballs) or - logs. Instead, we copy them to a static webserver which is far - more efficient. - -Authorization -============= - -Jenkins is set up to use OpenID in a Single Sign On mode with Launchpad. -This means that all of the user and group information is managed via -Launchpad users and teams. In the Jenkins Security Matrix, a Launchpad team -name can be specified and any members of that team will be granted those -permissions. However, because of the way the information is processed, a -user will need to re-log in upon changing either team membership on -Launchpad, or changing that team's authorization in Jenkins for the new -privileges to take effect. - -Devstack Gate -============= - -OpenStack integration testing is performed by the devstack gate test -framework. This framework runs the devstack exercises and Tempest -smoketests against a devstack install on single use cloud servers. The -devstack gate source can be found on `git.openstack.org -`_ and the `Readme -`_ -describes the process of using devstack gate to run your own devstack -based tests. - -For management of the devstack and other instances, a tool called -:ref:`nodepool` creates and deletes Jenkins slaves as needed in order to -maintain the pool. - -Sysadmin -======== - -Jenkins is largely hidden, and has no sensitive data exposed -publicly, so we use self-signed certs for Jenkins masters. - -After bringing up a jenkins node (16G memory instance if you use the -stock jenkins.default) with puppet, log in and configure Jenkins by -hand: - -#. Configure the site so it knows it's correct url. - (Jenkins URL in global config). This is needed to complete an SSO - sign-in. - -#. Configure the OpenID plugin for your SSO site (e.g. Launchpad) - -#. Do not set CSRF protection - that causes problems with various components - such as nodepool and swift log uploader. - -#. Login. - -#. Setup matrix security: add the 'authenticated' pseudo user and - grant Admin access to your own user. - -#. Setup one account per `http://docs.openstack.org/infra/jenkins-job-builder/installation.html#configuration-file` - and grab the API token for it. - -#. Configure the number of executors you want on the Jenkins Master - (e.g. 1) - -#. Configure a maven environment (if you have Maven projects to test). - -#. Enable the gearman plugin globally. Your gearman server is - zuul.$project. If Test Connection fails, do a puppet run (puppet - agent --test) on the zuul machine, as gearman wouldn't have started - with no workers configured. - -#. Configure the timestamper plugin. E.g. to - ''yyyy-MM-dd HH:mm:ss' ' - -#. Enable the zmq plugin globally if it is visible. No settings were - visible when writing this doc. - -#. You will configure global scp and ftp credentials for static and - docs sites respectively later, but as we haven't setup those sites - yet, that's not possible :). - -Puppet takes care of the rest. - -Quirks ------- - -Note that jenkins talks to its slaves via ssh, the -modules/openstack_project/manifests/init.pp file contains the ssh -public key that puppet installs on the slaves. - -Slaves ------- - -Statically provisioned slaves have labels assigned by hand. E.g. -centos6, and are added to a chosen Jenkins master by hand. Adding a -slave is then: - -#. Launch a slave - -#. Add it to Jenkins - Add your jenkins master key for the credentials (make it global, - one-time operation). - Set the jenkins home to /home/jenkins - -#. Set appropriate labels on it - -#. Profit! - -Safe Master Restarts -==================== - -Jenkins masters periodically leak threads reducing their job -throughput and eventually leading to crashes. We work around this -by performing weekly rolling restarts of the Jenkins masters with -an ansible playbook. - -If you need to perform a safe restart against a single master you -can do this by running the same playbook and limiting it to a -specific jenkins master - -To do this:: - - root@puppetmaster# ansible-playbook -f1 --limit $server_fqdn \ - /opt/system-config/production/playbooks/restart_jenkins_masters.yaml \ - --extra-vars "user=hudson-openstack \ - password=$(/opt/system-config/production/tools/hieraedit.py \ - --yaml /etc/puppet/hieradata/production/fqdn/nodepool.openstack.org.yaml jenkins_api_key)" - -Consider running this in screen as the worst case run time is as -long as our longest running job. - -How to manually run jenkins job builder -======================================= - -Jenkins job builder may need to be run manually under certain situations. If the expected -jobs are not being created in jenkins masters, running jjb manually on the masters where -it failed is suggested. To do this:: - - user@jenkins01# sudo -H jenkins-jobs --conf /etc/jenkins_jobs/jenkins_jobs.ini \ - update --delete-old /etc/jenkins_jobs/config - -Consider running this in screen as the worst case run time can be of several hours. - -In the case of incorrect jobs configuration caused by some jjb malfunction, all jobs -will need to be regenerated. As jjb uses a local cache, to force the regeneration -of all jobs, the cache needs to be ignored. To do this:: - - user@jenkins01# sudo -H jenkins-jobs --ignore-cache --conf \ - /etc/jenkins_jobs/jenkins_jobs.ini update --delete-old /etc/jenkins_jobs/config - -In order to speed up the massive job reconfiguration, it may be desired to set jenkins -on shutdown mode, visiting this link:: - -`https://jenkins[xx].openstack.org/quietDown` - -And make Jenkins alive again after job reconfiguration finishes. diff --git a/doc/source/jjb.rst b/doc/source/jjb.rst index a00c73bdfb..6ecada814b 100644 --- a/doc/source/jjb.rst +++ b/doc/source/jjb.rst @@ -12,10 +12,7 @@ At a Glance =========== :Hosts: - * http://jenkins.openstack.org - * http://jenkins-dev.openstack.org -:Puppet: - * https://git.openstack.org/cgit/openstack-infra/puppet-jenkins/tree/manifests/job_builder.pp + * http://zm*.openstack.org :Configuration: * :config:`jenkins/jobs/` :Projects: @@ -228,44 +225,11 @@ values. Notice that example1 makes use of the job group and example2 makes use of the job template. -Job Caching ------------ +Zuul +==== -The Jenkins Jobs builder maintains a special `cache`_ that -contains an MD5 of every generated XML that it builds. If -it finds the XML is different then it will proceed to send this -to Jenkins, otherwise it is skipped. If a job is accidentally deleted -then this file should be modified or removed. - -.. _cache: http://docs.openstack.org/infra/jenkins-job-builder/installation.html#running - -Sending a Job to Jenkins ------------------------- - -The Jenkins Jobs builder talks to Jenkins using the Jenkins API. This -means that it can create and modify jobs directly without the need to -restart or reload the Jenkins server. It also means that Jenkins will -verify the XML and cause the Jenkins Jobs builder to fail if there is -a problem. - -For this to work a configuration file is needed. There is an erb -template for this configuration file at -:file:`modules/jenkins/templates/jenkins_jobs.ini.erb`. The contents -of this template are: - -.. code-block:: ini - - [jenkins] - user=<%= username %> - password=<%= password %> - url=<%= url %> - -The values for user and url are hardcoded in the Puppet repo in -:file:`modules/openstack_project/manifests/jenkins.pp`, but the -password is stored in hiera. Make sure you have it defined as -``jenkins_jobs_password`` in the hiera DB. - -The password can be obtained by logging into the Jenkins user, -clicking on your username in the top-right, clicking on `Configure` -and then `Show API Token`. This API Token is your password for the -API. +In our environment, we no longer use Jenkins to execute jobs. Zuul +itself, via Ansible, runs the actual workload. Zuul reads JJB config +files in order to define its jobs, so, aside from the detail of not +actually using Jenkins or creating any jobs in it, the use of JJB to +configure jobs in Zuul is the same. diff --git a/doc/source/running-your-own.rst b/doc/source/running-your-own.rst index 12cb15e22b..2962b4c6c5 100644 --- a/doc/source/running-your-own.rst +++ b/doc/source/running-your-own.rst @@ -309,7 +309,7 @@ Stage 5 - Jenkins Master(s) ~~~~~~~~~~~~~~~~~~~~~~~~~~~ For Zuul to schedule work, it needs one or more Gearman connected Jenkins -masters. See :ref:`jenkins` for details. +masters. The minimum setup is one master, but if you will be permitting any code submitter to trigger test runs, we recommend having two: one untrusted and one @@ -360,7 +360,7 @@ which get the most load (as they run jobs from anyone). #. Launch the node with a size larger than the jenkins size you specified. -#. Setup Jenkins per :ref:`jenkins`. +#. Setup Jenkins. At this stage doing a 'recheck' should still report LOST on a change. But in the zuul debug.log in /var/log/zuul you should see a 'build xxx not diff --git a/doc/source/systems.rst b/doc/source/systems.rst index 26c3142151..185e22e607 100644 --- a/doc/source/systems.rst +++ b/doc/source/systems.rst @@ -10,7 +10,6 @@ Major Systems gerrit grafana grafyaml - jenkins zuul jjb logstash diff --git a/doc/source/zuul.rst b/doc/source/zuul.rst index 9ae010046e..dce7321309 100644 --- a/doc/source/zuul.rst +++ b/doc/source/zuul.rst @@ -15,6 +15,8 @@ At a Glance * http://status.openstack.org/zuul * http://zuul.openstack.org * http://zuul-dev.openstack.org + * zm*.openstack.org + * zl*.openstack.org :Puppet: * https://git.openstack.org/cgit/openstack-infra/puppet-zuul/tree/ * :file:`modules/openstack_project/manifests/zuul_prod.pp` @@ -35,12 +37,11 @@ The OpenStack project uses a number of pipelines in Zuul: **check** Newly uploaded patchsets enter this pipeline to receive an initial - +/-1 Verified vote from Jenkins. + +/-1 Verified vote. **gate** Changes that have been approved by core developers are enqueued in - order in this pipeline, and if they pass tests in Jenkins, will be - merged. + order in this pipeline, and if they pass tests, will be merged. **post** This pipeline runs jobs that operate after each change is merged. @@ -146,3 +147,31 @@ before the restart:: You may watch the `Zuul Status Page `_ to confirm that changes are returning to the queues. + +Mergers +------- + +Servers with names matching the pattern zm*.openstack.org are Zuul +Mergers. These are horizontally scalable components of Zuul which +perform git operations for the benefit of jobs. They serve git +repositories via Apache over http, and jobs fetch changes to test from +them. They can be started and stopped at will, and new ones added as +necessary to accommodate load. If you remove a merger, be sure to +leave Apache running for several hours until the last job that may +have been launched with instructions to fetch from that merger has +completed. + +Launchers +--------- + +We use an Ansible based launcher in Zuul to actually run jobs. This +component runs on a horizontally scalable set of servers named +zl*.openstack.org. It reads job configuration from Jenkins Job +Builder files in the project-config repository and translates that +into Ansible playbooks which it runs on our workers. Our jobs are +configured to upload as much information as possible along with their +logs, but if there is an error which can not be diagnosed in that +manner, Ansible logs are available in the launcher-debug log file on +the launcher host. You may use the Zuul build UUID to track +assignment of a given job from the Zuul scheduler to the Zuul launcher +used by that job.