Document how to manually simulate infrastructure.
Fixes bug 1172428. * README.rst: Developers often have a need to recreate gating integration tests manually, and this provides a walkthrough of making a DG-slave-like throwaway server without the overhead of building other CI infrastructure to manage a pool of them. This can be useful to reproduce and troubleshoot failures or tease out nondeterministic bugs. The driving use case is for pre-testing embargoed vulnerability fixes, increasing the chance they can get through gating on the first try. Change-Id: I0f78011c183f64b894b336eb8e0d25dc5ba00bde
This commit is contained in:
Jeremy Stanley
parent
b6ab7536b2
commit
59953b4851
137
README.rst
137
README.rst
|
|
@ -200,6 +200,143 @@ a specific devstack gate node provider, we'd love it if you could help
|
|||
identify what the variable might be (whether in the devstack-gate
|
||||
scripts, devstack itself, OpenStack, or even the provider's service).
|
||||
|
||||
Simulating Devstack Gate Tests
|
||||
==============================
|
||||
|
||||
Developers often have a need to recreate gating integration tests
|
||||
manually, and this provides a walkthrough of making a DG-slave-like
|
||||
throwaway server without the overhead of building other CI
|
||||
infrastructure to manage a pool of them. This can be useful to reproduce
|
||||
and troubleshoot failures or tease out nondeterministic bugs.
|
||||
|
||||
First, it helps if you have access to a virtual machine from one of the
|
||||
providers the OpenStack project is using for gating, since their
|
||||
performance characteristics and necessary build parameters are already
|
||||
known. The same thing can of course be done locally or on another
|
||||
provider, but you'll want to make sure you have a basic Ubuntu 12.04 LTS
|
||||
(Precise Pangolin) image with sufficient memory and processor count.
|
||||
These days Tempest testing is requiring in excess of 2GiB RAM (4 should
|
||||
be enough but we typically use 8) and completes within an hour on a
|
||||
4-CPU virtual machine.
|
||||
|
||||
If you're using a nova provider, it's usually helpful to set up an
|
||||
environment variable list you can include into your shell so you don't
|
||||
have to feed a bunch of additional options on the nova client command
|
||||
line. A provider settings file for Rackspace would look something like::
|
||||
|
||||
export OS_USERNAME=<provider_username>
|
||||
export OS_PASSWORD='<provider_password>'
|
||||
export OS_TENANT_NAME=<provider_tenant>
|
||||
export OS_AUTH_URL=https://identity.api.rackspacecloud.com/v2.0/
|
||||
export OS_REGION_NAME=DFW
|
||||
export NOVA_RAX_AUTH=1
|
||||
export FLAVOR='8GB Standard Instance'
|
||||
export IMAGE='Ubuntu 12.04 LTS (Precise Pangolin)'
|
||||
|
||||
By comparison, a provider settings file for HPCloud::
|
||||
|
||||
export OS_USERNAME=<provider_username>
|
||||
export OS_PASSWORD='<provider_password>'
|
||||
export OS_TENANT_NAME=<provider_tenant>
|
||||
export OS_AUTH_URL=https://region-a.geo-1.identity.hpcloudsvc.com:35357/v2.0
|
||||
export OS_REGION_NAME=az-3.region-a.geo-1
|
||||
export FLAVOR='standard.large'
|
||||
export IMAGE='Ubuntu Precise 12.04 LTS Server 64-bit 20121026 (b)'
|
||||
|
||||
Source the provider settings, boot a server named "testserver" (chosen
|
||||
arbitrarily for this example) with your SSH key allowed, and log into
|
||||
it::
|
||||
|
||||
. provider_settings.sh
|
||||
nova boot --poll --flavor "$FLAVOR" --image "$IMAGE" \
|
||||
--file /root/.ssh/authorized_keys=$HOME/.ssh/id_rsa.pub testserver
|
||||
nova ssh testserver
|
||||
|
||||
If you get a cryptic error like ``ERROR: 'public'`` then you may need to
|
||||
manually look up the IP address with ``nova list --name testserver`` and
|
||||
connect by running ``ssh root@<ip_address>`` instead.
|
||||
|
||||
Upgrade the server, install git and pip packages, add tox via pip
|
||||
(because the packaged version is too old), set up a "jenkins" account
|
||||
and reboot to make sure you're running a current kernel::
|
||||
|
||||
apt-get install -y git \
|
||||
&& git clone https://review.openstack.org/p/openstack-infra/config \
|
||||
&& config/install_puppet.sh && config/install_modules.sh \
|
||||
&& puppet apply --modulepath=/root/config/modules:/etc/puppet/modules \
|
||||
-e "class { openstack_project::slave_template: install_users => false,
|
||||
ssh_key => \"$( cat .ssh/authorized_keys )\" }" \
|
||||
&& echo HostKey /etc/ssh/ssh_host_ecdsa_key >> /etc/ssh/sshd_config \
|
||||
&& reboot
|
||||
|
||||
Wait a few moments for the reboot to complete, then log back in with
|
||||
``nova ssh --login jenkins testserver`` or ``ssh jenkins@<ip_address>``
|
||||
and set up parts of the environment expected by devstack-gate testing
|
||||
(the "devstack-vm-gate-dev.sh" script mentioned below in the
|
||||
`Developer Setup`_ section implements a similar workflow for testing
|
||||
changes to devstack-gate itself, but could be modified to automate much
|
||||
of this for ease of repetition)::
|
||||
|
||||
export REPO_URL=https://review.openstack.org/p
|
||||
export ZUUL_URL=/home/jenkins/workspace-cache
|
||||
export ZUUL_REF=HEAD
|
||||
export WORKSPACE=/home/jenkins/workspace/testing
|
||||
mkdir -p $WORKSPACE
|
||||
|
||||
Specify the project and branch you want to test for integration::
|
||||
|
||||
export ZUUL_PROJECT=openstack/nova
|
||||
export ZUUL_BRANCH=master
|
||||
|
||||
Get a copy of the tested project. After these steps, apply relevant
|
||||
patches on the target branch (via cherry-pick, rebase, et cetera) and
|
||||
make sure ``HEAD`` is at the ref you want tested::
|
||||
|
||||
git clone $REPO_URL/$ZUUL_PROJECT $ZUUL_URL/$ZUUL_PROJECT \
|
||||
&& cd $ZUUL_URL/$ZUUL_PROJECT \
|
||||
&& git checkout remotes/origin/$ZUUL_BRANCH
|
||||
|
||||
Switch to the workspace and get a copy of devstack-gate::
|
||||
|
||||
cd $WORKSPACE \
|
||||
&& git clone --depth 1 $REPO_URL/openstack-infra/devstack-gate
|
||||
|
||||
At this point you're ready to set the same environment variables and run
|
||||
the same commands/scripts as used in the desired job. The definitions
|
||||
for these are found in the openstack-infra/config project under the
|
||||
modules/openstack_project/files/jenkins_job_builder/config directory in
|
||||
a file named devstack-gate.yaml. It will probably look something like::
|
||||
|
||||
export PYTHONUNBUFFERED=true
|
||||
export DEVSTACK_GATE_TEMPEST=1
|
||||
export DEVSTACK_GATE_TEMPEST_FULL=1
|
||||
cp devstack-gate/devstack-vm-gate-wrap.sh ./safe-devstack-vm-gate-wrap.sh
|
||||
./safe-devstack-vm-gate-wrap.sh
|
||||
|
||||
If you're trying to figure out which devstack gate jobs run for a given
|
||||
project+branch combination, this is encoded in the
|
||||
openstack-infra/config project under the
|
||||
modules/openstack_project/files/zuul directory in a file named
|
||||
layout.yaml. You'll want to look in the "projects" section for a list of
|
||||
jobs run on a given project in the "gate" pipeline, and then consult the
|
||||
"jobs" section of the file to see if there are any overrides indicating
|
||||
which branches qualify for the job and whether or not its voting is
|
||||
disabled.
|
||||
|
||||
After the script completes, investigate any failures. Then log out and
|
||||
``nova delete testserver`` or similar to get rid of it once no longer
|
||||
needed. It's possible to re-run certain jobs or specific tests on a used
|
||||
VM (sometimes with a bit of manual clean-up in between runs), but for
|
||||
proper testing you'll want to validate your fixes on a completely fresh
|
||||
one.
|
||||
|
||||
Refer to the `Jenkins Job Builder`_ and Zuul_ documentation for more
|
||||
information on their configuration file formats.
|
||||
|
||||
.. _`Jenkins Job Builder`: http://ci.openstack.org/jjb.html
|
||||
|
||||
.. _Zuul: http://ci.openstack.org/zuul.html
|
||||
|
||||
Contributions Welcome
|
||||
=====================
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue