diff --git a/README.rst b/README.rst index a43e0850..675bf6e9 100644 --- a/README.rst +++ b/README.rst @@ -200,6 +200,143 @@ a specific devstack gate node provider, we'd love it if you could help identify what the variable might be (whether in the devstack-gate scripts, devstack itself, OpenStack, or even the provider's service). +Simulating Devstack Gate Tests +============================== + +Developers often have a need to recreate gating integration tests +manually, and this provides a walkthrough of making a DG-slave-like +throwaway server without the overhead of building other CI +infrastructure to manage a pool of them. This can be useful to reproduce +and troubleshoot failures or tease out nondeterministic bugs. + +First, it helps if you have access to a virtual machine from one of the +providers the OpenStack project is using for gating, since their +performance characteristics and necessary build parameters are already +known. The same thing can of course be done locally or on another +provider, but you'll want to make sure you have a basic Ubuntu 12.04 LTS +(Precise Pangolin) image with sufficient memory and processor count. +These days Tempest testing is requiring in excess of 2GiB RAM (4 should +be enough but we typically use 8) and completes within an hour on a +4-CPU virtual machine. + +If you're using a nova provider, it's usually helpful to set up an +environment variable list you can include into your shell so you don't +have to feed a bunch of additional options on the nova client command +line. A provider settings file for Rackspace would look something like:: + + export OS_USERNAME= + export OS_PASSWORD='' + export OS_TENANT_NAME= + export OS_AUTH_URL=https://identity.api.rackspacecloud.com/v2.0/ + export OS_REGION_NAME=DFW + export NOVA_RAX_AUTH=1 + export FLAVOR='8GB Standard Instance' + export IMAGE='Ubuntu 12.04 LTS (Precise Pangolin)' + +By comparison, a provider settings file for HPCloud:: + + export OS_USERNAME= + export OS_PASSWORD='' + export OS_TENANT_NAME= + export OS_AUTH_URL=https://region-a.geo-1.identity.hpcloudsvc.com:35357/v2.0 + export OS_REGION_NAME=az-3.region-a.geo-1 + export FLAVOR='standard.large' + export IMAGE='Ubuntu Precise 12.04 LTS Server 64-bit 20121026 (b)' + +Source the provider settings, boot a server named "testserver" (chosen +arbitrarily for this example) with your SSH key allowed, and log into +it:: + + . provider_settings.sh + nova boot --poll --flavor "$FLAVOR" --image "$IMAGE" \ + --file /root/.ssh/authorized_keys=$HOME/.ssh/id_rsa.pub testserver + nova ssh testserver + +If you get a cryptic error like ``ERROR: 'public'`` then you may need to +manually look up the IP address with ``nova list --name testserver`` and +connect by running ``ssh root@`` instead. + +Upgrade the server, install git and pip packages, add tox via pip +(because the packaged version is too old), set up a "jenkins" account +and reboot to make sure you're running a current kernel:: + + apt-get install -y git \ + && git clone https://review.openstack.org/p/openstack-infra/config \ + && config/install_puppet.sh && config/install_modules.sh \ + && puppet apply --modulepath=/root/config/modules:/etc/puppet/modules \ + -e "class { openstack_project::slave_template: install_users => false, + ssh_key => \"$( cat .ssh/authorized_keys )\" }" \ + && echo HostKey /etc/ssh/ssh_host_ecdsa_key >> /etc/ssh/sshd_config \ + && reboot + +Wait a few moments for the reboot to complete, then log back in with +``nova ssh --login jenkins testserver`` or ``ssh jenkins@`` +and set up parts of the environment expected by devstack-gate testing +(the "devstack-vm-gate-dev.sh" script mentioned below in the +`Developer Setup`_ section implements a similar workflow for testing +changes to devstack-gate itself, but could be modified to automate much +of this for ease of repetition):: + + export REPO_URL=https://review.openstack.org/p + export ZUUL_URL=/home/jenkins/workspace-cache + export ZUUL_REF=HEAD + export WORKSPACE=/home/jenkins/workspace/testing + mkdir -p $WORKSPACE + +Specify the project and branch you want to test for integration:: + + export ZUUL_PROJECT=openstack/nova + export ZUUL_BRANCH=master + +Get a copy of the tested project. After these steps, apply relevant +patches on the target branch (via cherry-pick, rebase, et cetera) and +make sure ``HEAD`` is at the ref you want tested:: + + git clone $REPO_URL/$ZUUL_PROJECT $ZUUL_URL/$ZUUL_PROJECT \ + && cd $ZUUL_URL/$ZUUL_PROJECT \ + && git checkout remotes/origin/$ZUUL_BRANCH + +Switch to the workspace and get a copy of devstack-gate:: + + cd $WORKSPACE \ + && git clone --depth 1 $REPO_URL/openstack-infra/devstack-gate + +At this point you're ready to set the same environment variables and run +the same commands/scripts as used in the desired job. The definitions +for these are found in the openstack-infra/config project under the +modules/openstack_project/files/jenkins_job_builder/config directory in +a file named devstack-gate.yaml. It will probably look something like:: + + export PYTHONUNBUFFERED=true + export DEVSTACK_GATE_TEMPEST=1 + export DEVSTACK_GATE_TEMPEST_FULL=1 + cp devstack-gate/devstack-vm-gate-wrap.sh ./safe-devstack-vm-gate-wrap.sh + ./safe-devstack-vm-gate-wrap.sh + +If you're trying to figure out which devstack gate jobs run for a given +project+branch combination, this is encoded in the +openstack-infra/config project under the +modules/openstack_project/files/zuul directory in a file named +layout.yaml. You'll want to look in the "projects" section for a list of +jobs run on a given project in the "gate" pipeline, and then consult the +"jobs" section of the file to see if there are any overrides indicating +which branches qualify for the job and whether or not its voting is +disabled. + +After the script completes, investigate any failures. Then log out and +``nova delete testserver`` or similar to get rid of it once no longer +needed. It's possible to re-run certain jobs or specific tests on a used +VM (sometimes with a bit of manual clean-up in between runs), but for +proper testing you'll want to validate your fixes on a completely fresh +one. + +Refer to the `Jenkins Job Builder`_ and Zuul_ documentation for more +information on their configuration file formats. + +.. _`Jenkins Job Builder`: http://ci.openstack.org/jjb.html + +.. _Zuul: http://ci.openstack.org/zuul.html + Contributions Welcome =====================