Remove contrib/README.md
This file has now been converted to a rst file located in doc/source/third_party_ci.rst file. Remove it to avoid divergence. Change-Id: I77e1accbc1c7f8aa05b232d11f7170caa1f48979 Depends-on: Idf1502828da3fbb7a9423f8712e504723a36df52
This commit is contained in:
parent
bdf985bea1
commit
f667f01961
|
@ -1,407 +1,4 @@
|
|||
# OpenStack Third-Party CI
|
||||
|
||||
These instructions provide a **Third Party Testing** solution using the same
|
||||
tools and scripts used by the OpenStack Infrastructure 'Jenkins' CI
|
||||
system.
|
||||
|
||||
If you are setting up a similar system for use outside of OpenStack, many of these
|
||||
steps are still valid, while others can be skipped. These will be mentioned within
|
||||
each step.
|
||||
|
||||
If you are creating a third-party CI system for use within OpenStack,
|
||||
you'll need to familiarize yourself with the contents of the
|
||||
[third party manual](http://docs.openstack.org/infra/system-config/third_party.html),
|
||||
and in particular you'll need to [create a service account]
|
||||
(http://docs.openstack.org/infra/system-config/third_party.html#creating-a-service-account).
|
||||
|
||||
## Overview
|
||||
|
||||
This CI solution uses a few open-source tools:
|
||||
|
||||
* [Jenkins](http://docs.openstack.org/infra/system-config/jenkins.html) - an open-source continuous integration server.
|
||||
|
||||
* [Zuul](http://docs.openstack.org/infra/system-config/zuul.html) - a project gating system
|
||||
|
||||
* [Nodepool](http://docs.openstack.org/infra/system-config/nodepool.html)- a node management system for testing
|
||||
|
||||
* [Jenkins Job Builder](http://docs.openstack.org/infra/system-config/jjb.html) - a tool to manage jenkins job definitions
|
||||
|
||||
* [os-loganalyze](http://git.openstack.org/cgit/openstack-infra/os-loganalyze/) - a tool to facilitate
|
||||
browsing, sharing, and filtering log files by log level.
|
||||
|
||||
|
||||
The following steps will help you integrate and deploy the first 4 tools on a single
|
||||
node. An initial system with 8GB RAM, 4CPUs, 80GB HD should be sufficient,
|
||||
running Ubuntu 14.04.
|
||||
|
||||
A second node will be used to store the log files and create a public log server to
|
||||
host the static log files generated by jenkins jobs. This log server node is an Apache
|
||||
server serving log files stored on disk or on a Swift service.
|
||||
It is hosted on a separate node because it usually
|
||||
needs to be publicly accessible to share job results whereas the rest of the CI
|
||||
system can be located behind a firewall or within a VPN. At the end of a Jenkins Job,
|
||||
`publishers` will scp log files from the jenkins slave to the log server node or upload
|
||||
to the Swift Service.
|
||||
|
||||
The system requires two external resources:
|
||||
|
||||
* A source for Nodepool nodes. This is a service that implements the OpenStack Nova API
|
||||
to provide virtual machines or bare metal nodes. Nodepool will use this service to
|
||||
manage a pool of Jenkins slaves that will run the actual CI jobs.
|
||||
You can use a public or private OpenStack cloud, or even run your own
|
||||
[devstack](https://git.openstack.org/cgit/openstack-dev/devstack/) to get started.
|
||||
|
||||
|
||||
* A Gerrit server (for OpenStack users, this is provided to you at review.openstack.org)
|
||||
Zuul will listen to the Gerrit event stream to decide which jobs to run when it receives
|
||||
a desired event. Zuul will also post a comment with results to this Gerrit server with the
|
||||
job results along with a link to the related log files.
|
||||
|
||||
These instructions are for a 'masterless' puppet setup, which is the simplest
|
||||
version to set up for those not familiar with puppet.
|
||||
|
||||
|
||||
## Install and Configure Puppet
|
||||
|
||||
On each node, you will need to install and configure puppet. These scripts assume
|
||||
a dedicated 'clean' node built with a base [ubuntu 14.04 server image](http://www.ubuntu.com/download/server).
|
||||
|
||||
### Install Puppet
|
||||
|
||||
Puppet is a tool to automate the installation of servers by defining the desired
|
||||
end state. You can quickly install puppet along with basic tools (such as pip and git)
|
||||
using this script:
|
||||
|
||||
sudo su -
|
||||
wget https://git.openstack.org/cgit/openstack-infra/system-config/plain/install_puppet.sh
|
||||
bash install_puppet.sh
|
||||
exit
|
||||
|
||||
### Install Puppet Modules
|
||||
|
||||
You can get the latest version of the puppet modules needed using this script.
|
||||
|
||||
sudo su -
|
||||
git clone https://git.openstack.org/openstack-infra/system-config
|
||||
cd system-config
|
||||
./install_modules.sh
|
||||
exit
|
||||
|
||||
This script will install all the puppet modules used by upstream to
|
||||
`/etc/puppet/modules`. In many cases, these are git cloned, and running
|
||||
the `install_modules.sh` script again will update them to the latest version.
|
||||
This script uses `modules.env` as its configuration input.
|
||||
|
||||
### Configure Masterless Puppet
|
||||
|
||||
The instructions in this section apply to both the single-node CI server node as
|
||||
well as the log server node.
|
||||
|
||||
It is useful to save the history, so set up a
|
||||
git repo as root user:
|
||||
|
||||
sudo su -
|
||||
cd /etc/puppet
|
||||
git init
|
||||
echo "modules/" >> .gitignore
|
||||
git add .
|
||||
git commit -m "initial files"
|
||||
exit
|
||||
|
||||
You will be configuring 3 puppet files. The first is `site.pp` which is the top
|
||||
level entry point for puppet to start managing the node. The second is a `hiera.yaml`
|
||||
which configures Puppet Hiera to store local configurations and secrets
|
||||
such as passwords and private keys, and finally some `yaml` files which store the
|
||||
actual configurations and secrets.
|
||||
|
||||
Set up these 3 files by starting with the samples provided. For each node,
|
||||
select the corresponding `single_node_ci*` or `log_server*` files.
|
||||
|
||||
sudo su -
|
||||
cp /etc/puppet/modules/openstackci/contrib/hiera.yaml /etc/puppet
|
||||
|
||||
cp /etc/puppet/modules/openstackci/contrib/single_node_ci_site.pp /etc/puppet/manifests/site.pp
|
||||
cp /etc/puppet/modules/openstackci/contrib/single_node_ci_data.yaml /etc/puppet/environments/common.yaml
|
||||
|
||||
OR
|
||||
|
||||
cp /etc/puppet/modules/openstackci/contrib/log_server_site.pp /etc/puppet/manifests/site.pp
|
||||
cp /etc/puppet/modules/openstackci/contrib/log_server_data.yaml /etc/puppet/environments/common.yaml
|
||||
exit
|
||||
|
||||
At this point, you should not need to modify either of the first two files.
|
||||
Modify `/etc/puppet/environments/common.yaml` as you need using the parameter
|
||||
documentation described in [single_node_ci.pp](http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/manifests/single_node_ci.pp)
|
||||
or [logserver.pp](http://git.openstack.org/cgit/openstack-infra/puppet-openstackci/tree/manifests/logserver.pp)
|
||||
(which are the top level puppet class that is used in `site.pp`).
|
||||
|
||||
Once completed, you should commit these 3 files to the `/etc/puppet` git repo.
|
||||
Your git workflow may vary a bit, but here is an example:
|
||||
|
||||
sudo su -
|
||||
cd /etc/puppet
|
||||
git checkout -b setup
|
||||
git add environments/common.yaml
|
||||
# repeat for other modified files
|
||||
git commit -a -m 'initial setup'
|
||||
exit
|
||||
|
||||
# Set up the log server
|
||||
|
||||
Set up the log server node first as it is simpler to configure. Besides, its FQDN
|
||||
(or IP address) is needed to set up the CI server node.
|
||||
|
||||
While setting up jenkins_ssh_public_key in `common.yaml` it is important that
|
||||
the same ssh key pair is used when setting up the CI server node in the next step.
|
||||
This is the ssh key that Jenkins will use to scp files.
|
||||
|
||||
At this point you are ready to invoke Puppet for the first time. Puppet needs to
|
||||
be run as root.
|
||||
|
||||
sudo puppet apply --verbose /etc/puppet/manifests/site.pp
|
||||
|
||||
You can simulate a jenkins file upload using:
|
||||
|
||||
scp -i $JENKINS_SSH_PRIVATE_KEY_FILE -o StrictHostKeyChecking=no $your-log-file jenkins@<fqdn_or_ip>:/srv/static/logs/
|
||||
|
||||
You should now be able to see the file you uploaded at `http://<fqnd_or_ip>/$your-log-file`
|
||||
|
||||
# Set up the CI server
|
||||
Follow the steps above to install and configure puppet on the CI server node.
|
||||
|
||||
## Create an Initial 'project-config' Repository
|
||||
|
||||
Setting up a CI system consists of two major operational aspects. The first is
|
||||
system configuration, which focuses on the installation and deployment of the
|
||||
services, including any ssh keys, credentials, databases, etc., and ensure all
|
||||
system components are able to interact together. This portion is
|
||||
performed by a System Administrator.
|
||||
|
||||
The second is project configuration, which includes the configuration files
|
||||
that the services use to perform the desired project-specific operations.
|
||||
|
||||
The instructions provided here are mainly focused on the system configuration aspect.
|
||||
However, system configuration requires an initial set of project configurations in order
|
||||
to work. These project configurations are provided via a git URL to a `project-config` repository.
|
||||
Before moving on, create an initial `project-config` repository. You can start with this
|
||||
[project-config-example](https://git.openstack.org/cgit/openstack-infra/project-config-example/)
|
||||
following the instructions provided in its README.md. While tailored for OpenStack users,
|
||||
the instructions provided will help non-OpenStack users also start with this repository.
|
||||
After your system is deployed, you can make further changes to the `project-config`
|
||||
repository to continuously tailor it to your needs.
|
||||
|
||||
## Add 'jenkins' to your host name
|
||||
|
||||
Add 'jenkins' to your /etc/hosts file so that Apache (which will be installed by
|
||||
the puppet scripts) is happy. This is needed because the scripts will install
|
||||
multiple services on a single node. For example:
|
||||
|
||||
head -n 1 /etc/hosts
|
||||
127.0.0.1 localhost jenkins
|
||||
|
||||
## Run masterless Puppet
|
||||
|
||||
At this point you are ready to invoke Puppet for the first time. Puppet needs to
|
||||
be run as root.
|
||||
|
||||
sudo puppet apply --verbose /etc/puppet/manifests/site.pp
|
||||
|
||||
Puppet will install nodepool, jenkins, zuul, jenkins jobs builder, etc.
|
||||
|
||||
Your `project-config` repository will be cloned to /etc/project-config, and the puppet scripts
|
||||
will use these configuration files located in this folder. Do not update these files directly.
|
||||
Instead, you should update them from a clone on a dev host, merge the changes to master, and push
|
||||
them to the same git remote location. Puppet will always pull down the latest version of master
|
||||
from the git remote and use that to update services.
|
||||
|
||||
If you get the following error, manually run the failed `jenkins-jobs update` command with
|
||||
the arguments specified in the error message as root. This is caused by a bug in the puppet
|
||||
scripts where Jenkins is not yet running when Jenkins Job Builder attempts to load the
|
||||
Jenkins jobs.
|
||||
|
||||
Notice: /Stage[main]/Jenkins::Job_builder/Exec[jenkins_jobs_update]/returns: jenkins.JenkinsException: Error in request: [Errno 111] Connection refused
|
||||
Notice: /Stage[main]/Jenkins::Job_builder/Exec[jenkins_jobs_update]/returns: INFO:jenkins_jobs.builder:Cache saved
|
||||
Error: /Stage[main]/Jenkins::Job_builder/Exec[jenkins_jobs_update]: Failed to call refresh: jenkins-jobs update --delete-old /etc/jenkins_jobs/config returned 1 instead of one of [0]
|
||||
Error: /Stage[main]/Jenkins::Job_builder/Exec[jenkins_jobs_update]: jenkins-jobs update --delete-old /etc/jenkins_jobs/config returned 1 instead of one of [0]
|
||||
|
||||
|
||||
## Restart apache if necessary
|
||||
|
||||
There are some known issues with Puppet automation. If you get the following error:
|
||||
|
||||
AH00526: Syntax error on line 21 of /etc/apache2/sites-enabled/50-<fqdn/ip>.conf:
|
||||
Invalid command 'RewriteEngine', perhaps misspelled or defined by a module not included in the server configuration
|
||||
|
||||
A simple restart works around the issue:
|
||||
|
||||
sudo service apache2 restart
|
||||
|
||||
## Start zuul
|
||||
|
||||
We'll start zuul first:
|
||||
|
||||
sudo service zuul start
|
||||
sudo service zuul-merger start
|
||||
|
||||
You should see 2 zuul-server processes and 1 zuul-merger process
|
||||
|
||||
ps -ef | grep zuul
|
||||
zuul 5722 1 2 18:13 ? 00:00:00 /usr/bin/python /usr/local/bin/zuul-server
|
||||
zuul 5725 5722 0 18:13 ? 00:00:00 /usr/bin/python /usr/local/bin/zuul-server
|
||||
zuul 5741 1 2 18:13 ? 00:00:00 /usr/bin/python /usr/local/bin/zuul-merger
|
||||
|
||||
You can view the log files for any errors:
|
||||
|
||||
view /var/log/zuul/zuul.log
|
||||
|
||||
Most zuul files are located in either of the following directories. They should not need
|
||||
to be modified directly, but are useful to help identify root causes:
|
||||
|
||||
/var/lib/zuul
|
||||
/etc/zuul
|
||||
|
||||
## Start nodepool
|
||||
|
||||
The first time starting nodepool, it's recommended to manually build the image
|
||||
to aid in debugging any issues:
|
||||
|
||||
|
||||
sudo su - nodepool
|
||||
|
||||
# Ensure the NODEPOOL_SSH_KEY variable is in the environment
|
||||
# Otherwise nodepool won't be able to ssh into nodes based
|
||||
# on the image built manually using these instructions
|
||||
source /etc/default/nodepool
|
||||
|
||||
# In the command below <image-name> references one of the
|
||||
# images defined in your project-config/nodepool/nodepool.yaml
|
||||
# file as the 'name' field in the section 'diskimages'.
|
||||
nodepool image-build <image-name>
|
||||
|
||||
If you run into issues building the image, the [documentation provided
|
||||
here can help you debug](https://git.openstack.org/cgit/openstack-infra/project-config/tree/nodepool/elements/README.rst)
|
||||
|
||||
|
||||
After you have successfully built an image, manually upload it to the provider to ensure
|
||||
provider authentication and image uploading work:
|
||||
|
||||
nodepool image-upload all <image-name>
|
||||
|
||||
Once successful, you can start nodepool.
|
||||
(Note that if you don't yet have an image, this is one of the first actions nodepool
|
||||
will do when it starts, before creating any nodes):
|
||||
|
||||
sudo service nodepool start
|
||||
|
||||
You should see at least one process running. In particular:
|
||||
|
||||
ps -ef | grep nodepool
|
||||
nodepool 5786 1 28 18:14 ? 00:00:01 /usr/bin/python /usr/local/bin/nodepoold -c /etc/nodepool/nodepool.yaml -l /etc/nodepool/logging.conf
|
||||
|
||||
After building and uploading the images to the providers, nodepool will
|
||||
start to build nodes on those providers based on the image
|
||||
and will register those nodes as jenkins slaves.
|
||||
|
||||
If that does not happen, the nodepool log files will help identify the causes.
|
||||
|
||||
view /var/log/nodepool/nodepool.log
|
||||
view /var/log/nodepool/debug.log
|
||||
|
||||
Most nodepool configuration files are located in either of the following directories. They should never
|
||||
to be modified directly as puppet will overwrite any changes, but are useful to help identify root causes:
|
||||
|
||||
/etc/nodepool
|
||||
/home/nodepool/.config/openstack/clouds.yaml
|
||||
|
||||
|
||||
## Setup Jenkins
|
||||
|
||||
First Restart Jenkins so that plugins will be fully installed:
|
||||
|
||||
sudo service jenkins restart
|
||||
|
||||
Then open the Jenkins UI to finish manual configuration steps.
|
||||
|
||||
Enable Gearman, which is the Jenkins plugin zuul uses to queue jobs:
|
||||
|
||||
http://<host fqdn/ip>:8080/
|
||||
Manage Jenkins --> Configure System
|
||||
Under "Gearman Plugin Config" Check the box "Enable Gearman"
|
||||
Click "Test Connection" It should return success if zuul is running.
|
||||
|
||||
Enable ZMQ Event Publisher, which is how nodepool is notified of Jenkin
|
||||
slaves status events:
|
||||
|
||||
http://<host fqdn/ip>:8080/
|
||||
Manage Jenkins --> Configure System
|
||||
Under "ZMQ Event Publisher"
|
||||
Check the box "Enable on all Jobs"
|
||||
|
||||
## Securing Jenkins (optional)
|
||||
|
||||
By default, Jenkins is installed with security disabled. While this is fine
|
||||
for development environments where external access to Jenkins UI is restricted,
|
||||
you are strongly encouraged to enable it. You can skip this step and do it
|
||||
at a later time if you wish:
|
||||
|
||||
Create a jenkins 'credentials':
|
||||
|
||||
http://<host fqdn/ip>:8080/
|
||||
Manage Jenkins --> Add Credentials --> SSH Username with private key
|
||||
Username 'jenkins'
|
||||
Private key --> From a file on Jenkins master
|
||||
"/var/lib/jenkins/.ssh/id_rsa"
|
||||
--> Save
|
||||
|
||||
Save the credential uuid in your hiera data:
|
||||
|
||||
sudo su jenkins
|
||||
cat /var/lib/jenkins/credentials.xml | grep "<id>"
|
||||
Copy the id to the 'jenkins_credentials_id' value in /etc/puppet/environments/common.yaml
|
||||
|
||||
Enable basic Jenkins security:
|
||||
|
||||
http://<host fqdn/ip>:8080/
|
||||
Manage Jenkins --> Configure Global Security
|
||||
Check "Enable Security"
|
||||
Under "Security Realm"
|
||||
Select Jenkin's own user database
|
||||
Uncheck allow users to sign up
|
||||
Under "Authorization" select "logged-in users can do anything"
|
||||
|
||||
Create a user 'jenkins'
|
||||
Choose a password.
|
||||
check 'Sign up'
|
||||
Save the password to the 'jenkins_password' value in /etc/puppet/environments/common.yaml
|
||||
|
||||
Get the new 'jenkins' user API token:
|
||||
|
||||
http://<host fqdn/ip>:8080/
|
||||
Manage Jenkins --> People --> Select user 'jenkins' --> configure --> Show API Token
|
||||
Save this token to the 'jenkins_api_key' value in /etc/puppet/environments/common.yaml
|
||||
|
||||
Reconfigure your system to use Jenkins security settings stored in
|
||||
`/etc/puppet/environments/common.yaml`
|
||||
|
||||
sudo puppet apply --verbose /etc/puppet/manifests/site.pp
|
||||
|
||||
# Updating your masterless puppet hosts
|
||||
|
||||
Any time you check-in changes to your `project-config` repo, make changes to the
|
||||
hiera data (`/etc/puppet/environments/common.yaml`), or update
|
||||
the puppet files (in /etc/puppet/modules, either manually or via the `install_modules.sh` script),
|
||||
run the same puppet command to update the host.
|
||||
|
||||
sudo puppet apply --verbose /etc/puppet/manifests/site.pp
|
||||
|
||||
If you need to change the git url in your `project-config` or any other git urls
|
||||
in your `common.yaml`, delete the respective repository, e.g.
|
||||
`/etc/project-config`, and puppet will reclone it from the new location when the above `puppet apply`
|
||||
command is reinvoked.
|
||||
|
||||
Note that it is safe, and expected, to rerun the above `puppet apply` command. Puppet will
|
||||
update the configuration of the host as described in the puppet classes. This means that if
|
||||
you delete or modify any files managed by puppet, rerunning the `puppet apply` command will
|
||||
restore those settings back to the specified state (and remove your local changes for better or worse).
|
||||
You could even run the `puppet apply` command as a cron job to enable continuous deployment
|
||||
in your CI system.
|
||||
These instructions have been moved to doc/source/third_party_ci.rst
|
||||
You can also view the [published version here](http://docs.openstack.org/infra/openstackci/)
|
Loading…
Reference in New Issue