tripleo-quickstart-extras/roles/create-zuul-based-reproducer/templates/README-reproducer-zuul-base...

<!DOCTYPE HTML>
<html lang="en-US">
<head>
<title>WARNING - WIP - NOT FOR GENERAL USE - README for Quickstart Job Zuul-based Reproducer Script</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<style>
.collapsible {
  background-color: #777;
  color: white;
  cursor: pointer;
  padding: 18px;
  width: 100%;
  border: none;
  text-align: left;
  outline: none;
  font-size: 15px;
}

.active, .collapsible:hover {
  background-color: #555;
}

.content {
  padding: 0 18px;
  max-height: 0;
  overflow: hidden;
  transition: max-height 0.2s ease-out;
  background-color: #f1f1f1;
}
</style>
</head>
</head>
<body>
<h1> WARNING - WIP - NOT FOR GENERAL USE - How to reproduce a job result using the Zuul-based reproducer playbooks and script</h1>
<p>Check the top-level logs directory for a tar file <b>reproducer-zuul-based-quickstart.tar</b>.
This tar archive contains three files:
<ul>
<li><code>launcher-env-setup-playbook.yaml</code> - a playbook to clone repos and set up
an Ansible environment to launch jobs</li>
<li><code>launcher-playbook.yaml - a playbook that installs and starts up Zuul, Gerrit and
other required services, and also creates and launches a reproducer job using
the same dlrn hashes as in CI</code></li>
<li><code>reproducer-zuul-based-quickstart.sh - a Bash script to provide an interface
to run the above playbooks with various options</code></li>
</ul>
Running these playbooks will set up Zuul, Gerrit, and related services in containers
on the local system, and will then create a reproducer job which can be run on a
cloud personal tenant or via libvirt on the local system.
If no <b>reproducer-zuul-based-quickstart.tar</b> file is found, that usually means an
infra failure before Quickstart could start or a problem in collecting logs.
Check on IRC Freenode channel <i>#tripleo</i> to see if there's an ongoing infra
issue.</p>

<button class="collapsible">Setting up to run the Zuul-based reproducer</button>
<div class="content">
<p>
To setup and run the reproducer you must first decide <b>where</b> it will run
- on a <b>pet</b> (your laptop) or on <b>cattle</b> (rdo-cloud vm or other
temporary environment)? For each case see notes below for required setup.

<p>
<b>If this is the first time you are running the reproducer</b> you must
ensure you have an externally routable network in your RDO-cloud tenant called
'private'. That is you will need a router attached to the external network that
is also attached to your network called 'private' (which you may need to create).
Assuming you have a clouds.yaml that let's you talk to RDO cloud you can use
the following commands to create the required setup:
<pre>
<code>
    openstack --os-cloud rdo-cloud network create private
    openstack --os-cloud rdo-cloud subnet create private --network private --gateway 192.168.0.1 --subnet-range 192.168.0.0/24
    openstack --os-cloud rdo-cloud router create privaterouter
    openstack --os-cloud rdo-cloud router set --external-gateway 38.145.32.0/22  privaterouter
    openstack --os-cloud rdo-cloud router add subnet privaterouter private
</code>
</pre>
</p>

Alternatively log in to <a href="https://phx2.cloud.rdoproject.org/dashboard/auth/login/?next=/dashboard/project/networks/">RDO cloud horizon</a> and create the network and router
that way.

<hr></hr>
<h1>Cattle</h1>
If you're using a temporary environment like RDO-cloud vm to run the
reproducer you can use the script inside the reproducer-quickstart/ directory
from the logs of the job you are reproducing. The script is acalled
<code>reproducer-fresh-box-setup.sh</code>:

<pre><code><b>
    curl -o reproducer-fresh-box-setup.sh http://logs.openstack.org/79/657779/2/check/tripleo-ci-centos-7-containers-multinode/67d8eb6/logs/reproducer-quickstart/reproducer-fresh-box-setup.sh
    chmod 775 reproducer-fresh-box-setup.sh
    ./reproducer-fresh-box-setup.sh -u marios -p password -c reprozuul
</b></code></pre>

<p><b>note:</b> If you're using an RDO-cloud vm be sure to use at least m1.large2
flavour as we've hit hard disk limits with m1.large</p>
<p> There are two required parameters <code>-u USER</code/> and <code>-p PASS</code>
which correspond to the RDO cloud username and password. These are written to
the generated <code>$HOME/.config/openstack/clouds.yaml</code> file expected by the
reproducer.
</p>
<p>This script creates the default user 'reprozuul' with passwordless sudo
privileges, creates $HOME/.config/openstack/clouds.yaml with the -u USER and
-p PASS parameters and creates the required ssh keypair. It must run as root,
or as a user with passwordless sudo.
</p>
<p> The script finally switches to the newly created user and outputs the newly
created public key. You must upload this key to gerrit (<b>both</b>
<a href="https://review.opendev.org/#/settings/ssh-keys">opendev.org</a> as well
as <a href="https://review.rdoproject.org/r/#/settings/ssh-keys">rdoproject.org</a>).
</p>
<hr></hr>
<h1>Pet</h1>

If you are running the reproducer on a non-transient machine, that is your
laptop or any other box which isn't temporary (a pet), then you need:

<ul>
<li> Ensure user running reproducer has passwordless sudo </li>
<li> That user has $HOME/.config/openstack/clouds.yaml with "rdo-cloud" entry </li>
<li> A ssh keypair generated with <b><code>-m PEM -t rsa</code></b> and uploaded
     to gerrit (<b>both </b><a href="https://review.opendev.org/#/settings/ssh-keys">opendev.org</a>
     as well as <a href="https://review.rdoproject.org/r/#/settings/ssh-keys">rdoproject.org</a>).
</li>
<li> A externally routable network in your RDO-cloud tenant named 'private'.
     See example commands above if you need to create this or log in to
     <a href="https://phx2.cloud.rdoproject.org/dashboard/auth/login/?next=/dashboard/project/networks/"> RDO cloud horizon</a> and verify your setup or create as needed.
</ul>

You only need to do these things once on your pet and then rerun reproducer or
update as needed.

<ul>
<li>Example clouds.yaml:</li>
<pre>
<code>CLOUDS_YAML_PATH=$HOME/.config/openstack/
mkdir -p $CLOUDS_YAML_PATH
/bin/bash -c "cat <<EOF>>$CLOUDS_YAML_PATH/clouds.yaml
clouds:
  rdo-cloud:
    identity_api_version: 3
    region_name: regionOne
    auth:
      auth_url: https://phx2.cloud.rdoproject.org:13000/v3
      password: $RDO_PASS
      project_name: $RDO_USER
      username: $RDO_USER
      user_domain_name: Default
      project_domain_name: Default
    regions:
    - name: regionOne
      values:
        networks:
        - name: 38.145.32.0/22
          routes_externally: true
        - name: private
EOF"
</code>
</pre>
</ul>
<ul>
<br/>
<li> Create an ssh key pair without a pass phrase:</li>
  <ul>
  <li>The ssh key pair is configurable - see variables <code>ssh_path</code>
  <code>user_pri_key</code>. The default key is:<code>~/.ssh/id_rsa[.pub]</code></li>
  <li> <b>Note:</b> Use the following to create keys: <b><code> ssh-keygen -m PEM -t rsa </code></b>
  See FAQ below for more information about ssh keys requirements.
  </ul>
</ul>
</br>
</div>

<button class="collapsible">Running the Zuul-based reproducer - using the Bash script</button>
<div class="content">
<p>
wget or curl the <b>reproducer-quickstart/reproducer-zuul-based-quickstart.tar</b> file.
<pre><code><b>
    curl -Lo reproducer-zuul-based-quickstart.tar http://logs.openstack.org/79/657779/2/check/tripleo-ci-centos-7-standalone-upgrade/92fd476/logs/reproducer-quickstart/reproducer-zuul-based-quickstart.tar
    tar -xvf reproducer-zuul-based-quickstart.tar
</b></code></pre>
One of the extracted files is the script called <code>./reproducer-zuul-based-quickstart.sh</code>
and you can run it with <code>--help</code> to see available options.
</br>
</br>
<b>Note:</b> If the local $USERNAME does not match the gerrit user IDs for
<a href="https://review.opendev.org/#/settings/ssh-keys">opendev.org</a> or
 <a href="https://review.rdoproject.org/r/#/settings/ssh-keys">rdoproject.org</a>)
then you will need to supply the -ug (upstream gerrit) and -rg (rdo gerrit)
parameters:

<ul>
</ul>
<pre><code><b>
    mkdir WORKSPACE
    ./reproducer-zuul-based-quickstart.sh --workspace WORKSPACE -ug slim_shady -rg slim_shadier
</b></code></pre>
</p>
Monitor the output and follow any instructions - for example the script will
exit and require re-run once the user is added to the docker group.
</br>
</br>
</div>

<button class="collapsible">Running the Zuul-based reproducer - using the playbooks directly</button>
<div class="content">
<p>
The Bash script is not required although it is the supported path.
The playbooks can be run directly using <code>ansible-playbook</code>.
The Bash script contains some package installation and Docker user/group setup
that you will need to cover prior to running the playbooks directly.
Please see the Bash scripts for details, especially if you are running the
reproducer for the first time on the local environment.
The playbooks can be run as follows:
<ul>
<li> Note that the <code>launcher-env-setup-playbook.yaml</code> will clone
repos and set up 'roles', 'playbooks', and 'library' directories
from which the tasks will be executed. </li>
<li><code>ansible-playbook ./launcher-env-setup-playbook.yaml</code></li>
<li><code>ansible-playbook ./launcher-playbook.yaml $OPTIONS</code></li>
</ul>
The <code>launcher-playbook.yaml</code> includes comments on how to run the playbook
with the same options listed in the Bash script section above.
</p>
</div>

<button class="collapsible">Running the Zuul-based reproducer - autohold nodes for debugging</button>
<div class="content">
<p>
The reproducer can be launched such that if there is an error the environment will
remain up and running available to the developer to debug.

<ul>
<li>For example:</li>

</ul>
<li><code> cd ~/tripleo-ci-reproducer </code></li>
<li><code>docker-compose exec scheduler zuul autohold --tenant tripleo-ci-reproducer --job tripleo-ci-centos-7-standalone --reason debug --project test1</code></li>
</p>
</div>

<button class="collapsible">Ansible tags and running repeatedly</button>
<div class="content">
<p>

<ul>
<li> The Bash scripts runs with <code>--tags all</code> passed to the
<code>launcher-playbook.yaml</code>.
</li>
<li>
After the initial docker setup has run, it is not necessary to run through
those setup steps for every reproducer. The <code>launcher-playbook.yaml</code>
can be run with <code>--skip-tags start</code> after the initial run.
</li>
</p>
</div>

<button class="collapsible">Monitoring the launched job</button>
<div class="content">
<p>
The <code>launcher-playbook.yaml</code> will create and push a job
to a test1 testproject repo in a temp directory. This job will start and the
related Zuul infrastructure logs can be viewed in:</br>
<pre>
<code>cd ~/tripleo-ci-reproducer
docker-compose logs --tail=10 -f
# Just view the scheduler and executor logs:
docker-compose logs --tail=10 -f scheduler executor</code>
</pre>
</br>
To see the job progress, you can access the web consoles of:
<ul>
<li>Gerrit: http://localhost:8080</li>
<li>Zuul: http://localhost:9000</li>
<li>Logs: http://localhost:8000</li>
</ul>
If you are accessing a remote machine, replace <code>localhost</code> with the ip address.
</p>
</div>

<button class="collapsible">Known Issues and FAQ</button>
<div class="content">
<p>
<ul>
<li>Gerrit keys<br/>
If your ssh keys have being created with a new version of OpenSSH, Zuul will
fail when running with these keys as there is a Paramiko bug [1].
It is recommended  that keys be generated as follows:
<code>ssh-keygen -m PEM -t rsa</code> - accepting the passwordless option.
Then update the public keys used with Gerrit upstream and RDO Gerrit.</br>
[1] https://storyboard.openstack.org/#!/story/2004842</li>
</ul>
<ul>
<li>If you currently do not have access to upstream-cloudinit-[centos-7, fedora-28] : </li>
<code>
<pre>
source the openrc from the openstack-nodepool tenant
openstack image list | grep upstream-cloudinit
glance member-create ${IMAGE_ID} ${YOUR_TENANT_ID}
glance member-create ${IMAGE_ID} ${YOUR_TENANT_ID}
</code>
</pre>
</ul>
</p>
</p>
</div>

<script>
  var coll = document.getElementsByClassName("collapsible");
  var i;

  for (i = 0; i < coll.length; i++) {
    coll[i].addEventListener("click", function() {
      this.classList.toggle("active");
      var content = this.nextElementSibling;
      if (content.style.maxHeight){
        content.style.maxHeight = null;
      } else {
        content.style.maxHeight = content.scrollHeight + "px";
      }
    });
  }
  </script>

</body>
</html>