Modifications for 2014 Gerrit User conference

This slide deck was presented at the 2014 Gerrit user conference.  It essentially contains
the same info as the overview slides with the following exceptions:
1. Presented to Gerrit developers and integrators so removed the Gerrit overview info
2. Slides concentrated on openstack integration with gerrit so added a few slides
emphasizing stuff like automated Gerrit installation, automated project creation,
and zuul reporting.
3. The talk was only 30 mins so abbreviated the slides by removing some of the openstack
intro slides, removing launchpad slides, and removing detailed zuul configuration slides.
4. The talk also wanted to present an overview of the entire openstack CI toolbox so
added a few slides about multi-master jenkins and nodepool.

Note: This change should only be tagged, not merged with master

Change-Id: I041acf626fdd0825dead57e8bc595c69bc540d1d
This commit is contained in:
Khai Do 2014-03-24 09:56:29 -07:00
parent 7aa69a5847
commit f1d0dfcfae
24 changed files with 198 additions and 284 deletions

BIN
images/WIP-state.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 60 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 67 KiB

BIN
images/elastic-recheck.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 88 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 89 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 121 KiB

BIN
images/gerrit-config1.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 116 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 200 KiB

BIN
images/gerrit-events.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 62 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 120 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 98 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 418 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 60 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 20 KiB

BIN
images/jobs-launched2.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 46 KiB

BIN
images/kibana.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 146 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 124 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 33 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 55 KiB

BIN
images/rechecks.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 64 KiB

BIN
images/status1.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 132 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 238 KiB

BIN
images/zuul_reports1.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 14 KiB

BIN
images/zuul_reports2.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 114 KiB

View File

@ -3,7 +3,7 @@
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en-US">
<head>
<title>Scaling OpenStack Development: Continuous Integration Overview</title>
<title>Collaboration at Scale: The Openstack CI toolbox</title>
<meta name="generator" content=
"HTML Tidy for Linux/x86 (vers 1st November 2003), see www.w3.org" />
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
@ -36,8 +36,8 @@ src="graphics/openstack-cloud-software-horizontal-small.png" /></div>
<img class="hidden" src="graphics/bullet-nofold-dim.gif" alt="" />
<img class="hidden" src="graphics/bullet-unfold-dim.gif" alt="" />
<img src="graphics/openstack-cloud-software-vertical-large.png" alt="OpenStack logo" class="cover" />
<h1>Scaling OpenStack Development<br />
<span class="smaller">Continuous Integration Overview</span></h1>
<h1>Collaboration at Scale<br />
<span class="smaller">The Openstack CI toolbox</span></h1>
<hr />
<div class="smaller">Monty Taylor
&lt;<a href="mailto:mordred@inaugust.com">mordred@inaugust.com</a>&gt;</div>
@ -47,6 +47,8 @@ src="graphics/openstack-cloud-software-horizontal-small.png" /></div>
&lt;<a href="mailto:fungi@yuggoth.org">fungi@yuggoth.org</a>&gt;</div>
<div class="smaller">Clark Boylan
&lt;<a href="mailto:cboylan@sapwetik.org">cboylan@sapwetik.org</a>&gt;</div>
<div class="smaller">Khai Do
&lt;<a href="mailto:zaro0508@gmail.com">zaro0508@gmail.com</a>&gt;</div>
</div>
<div class="slide">
@ -60,7 +62,7 @@ src="graphics/openstack-cloud-software-horizontal-small.png" /></div>
</div>
<div class="slide">
<h1>Projects</h1>
<h1>Openstack Projects</h1>
<div style="float:left;width:50%"><ul>
<li>Servers<ul>
<li>nova (compute)</li>
@ -93,7 +95,7 @@ src="graphics/openstack-cloud-software-horizontal-small.png" /></div>
<li class="gray">python-marconiclient</li>
</ul></li>
</ul></div>
<div class="indent"><a href="https://wiki.openstack.org/wiki/Projects">https://wiki.openstack.org/wiki/Projects</a></div>
<div class="indent" style="position: absolute; bottom: 65px"><a href="https://wiki.openstack.org/wiki/Projects">https://wiki.openstack.org/wiki/Projects</a></div>
</div>
<div class="slide">
@ -123,22 +125,7 @@ src="graphics/openstack-cloud-software-horizontal-small.png" /></div>
<li>Translation/Internationalization</li>
<li>Vulnerability Management</li>
</ul></div>
</div>
<div class="slide">
<h1>Release Management</h1>
<ul>
<li>Time Based Releases</li>
<li>Six Month Cadence<ul>
<li>Similar to Ubuntu and Fedora Release Schedules</li>
</ul>
<li>Design summits each cycle</li>
<li>Continuously Open Trunk<ul>
<li>Develop directly on master</li>
</ul>
<li>Intra-Cycle Milestone Releases</li>
<li>Post-Release Stable Branches</li>
</ul>
<div class="indent" style="position: absolute; bottom: 65px"><a href="http://git.openstack.org/cgit">~ 250 Projects</a></div>
</div>
<div class="slide">
@ -150,17 +137,7 @@ src="graphics/openstack-cloud-software-horizontal-small.png" /></div>
<li>Nonprofit organizations</li>
<li>National and local governments</li>
<li>Number, quality, and area of contributions can change daily</li>
</ul>
</div>
<div class="slide">
<h1>Consistent Tooling</h1>
<ul>
<li>Minimize meta-development</li>
<li>Process divergence == wasted developer time</li>
<li>Lowers onboarding time</li>
<li>Consolidate tool development</li>
<li>Minimize project-specific weird build crud</li>
<li>1000+ active contributors</li>
</ul>
</div>
@ -232,106 +209,63 @@ src="graphics/openstack-cloud-software-horizontal-small.png" /></div>
</div>
<div class="slide">
<h1>System Administration is in Git</h1>
<h1>Openstack CI Preface</h1>
<ul>
<li>Environment<ul>
<li>Open Source</li>
<li>Python</li>
<li>High volume of changes</li>
<li>Not resource contrained</li>
</ul>
<li>Goals<ul>
<li>Keep mainline development clean</li>
<li>Validate correctness of changes</li>
<li>Reduce regressions</li>
<li>Reduce feedback loop</li>
<li>Automate everything</li>
</ul>
</ul>
</div>
<div class="slide">
<h1>System Administration is in Gerrit</h1>
<ul>
<li>All system administration performed via code review</li>
<li>Anyone on the internet can look at our changes</li>
<li>Anyone can sign up for a code review account</li>
<li>Anyone can write and propose changes</li>
<li>http://git.openstack.org/cgit/openstack-infra/config/tree/</li>
</ul>
</div>
<div class="slide">
<h1>Development Environment</h1>
<ul>
<li>Python<ul>
<li>CentOS (2.6), Ubuntu LTS (2.7, 3.3, pypy)</li>
<li>PEP-8 standards</li>
<li>Oslo (common libraries)</li>
<li>virtualenv/pip/tox</li>
</ul></li>
<li>Freenode IRC (#openstack-dev, #openstack-meeting)</li>
<li>DevStack</li>
<li>Tests run on all newly submitted changes</li>
<li>Code merges are gated on tests</li>
</ul>
</div>
<div class="slide">
<h1>Project Gating</h1>
<img style="float:right; margin-right:24pt" src="images/jobs-launched.png"/>
<!--
http://graphite.openstack.org/render/?from=00%3A00_20130601&fgcolor=000000&title=Jobs%20Launched%20%28per%20Day%29&_t=0.7862690862083328&height=300&bgcolor=ffffff&width=400&showTarget=alias%28summarize%28sumSeries%28stats_counts.zuul.job.*%29%2C%271h%27%29%2C%27All%20Jobs%27%29&_salt=1373041766.855&target=alias%28summarize%28sumSeries%28stats_counts.zuul.job.*%29%2C%271d%27%29%2C%27All%20Jobs%27%29&until=23%3A59_20130630
-->
<ul>
<li>Ensures Code Quality</li>
<li>Protects developers<ul>
<li>Devs always start from working code</li>
</ul><li>Protects tree<ul>
<li>Bad code doesn't land</li>
</ul><li>Egalitarian<ul>
<li>Process is the same for everyone</li>
<li>Process is transparent</li>
<li>Process is automated</li>
</ul></ul>
<li>Any changes to Gerrit are reviewable</li>
<li><a href="http://git.openstack.org/cgit/openstack-infra/config/tree/">source</a></li>
<li><a href="https://review.openstack.org/#/q/status:open+project:openstack-infra/config,n,z">reviews</a></li>
</ul>
</div>
<div class="slide">
<h1>Everything Is Automated</h1>
<h1>CI Workflow</h1>
<center>
<img src="images/status.png"/>
</center>
</div>
</div>
<div class="slide">
<h1>Process Flow</h1>
<center>
<img src="images/contribution-path.png"/>
<img src="images/openstack-ci-workflow.png"/>
</center>
</div>
<div class="slide">
<h1>Gerrit</h1>
<ul>
<li>Developed by Google for Android</li>
<li>Stand-alone patch review system</li>
<li>Integration points: hooks, JSON queries, event-stream</li>
<li>Extensible review categories, default: Verified, Code-Review</li>
<li>Currently on version 2.4.x transitioning to 2.8.x</li>
<li>Votes: verified, code-review, approved</li>
<li>Hooks: integration with issue tracker</li>
<li>Event-stream: integration with Zuul and Chat</li>
<li>Command Line: <a href="https://review.openstack.org/#/c/79900/6">auto create projects</a></li>
<li>*Custom Work In Progress state</li>
<img src="images/WIP-state.png"/>
</ul>
</div>
<div class="slide">
<h1>Gerrit</h1>
<h1>Process Flow</h1>
<center>
<img style="margin-top:2em" src="images/gerrit.png"/>
</center>
</div>
<div class="slide">
<h1>Gerrit Diff View</h1>
<center>
<img style="margin-top:2em" src="images/gerrit-diff.png"/>
</center>
</div>
<div class="slide">
<h1>Bug Integration - Launchpad</h1>
<center>
<img style="margin-top:2em" src="images/lp-bug.png"/>
</center>
</div>
<div class="slide">
<h1>Approved Reviews</h1>
<center>
<img style="margin-top:2em"src="images/gerrit-approved.png"/>
<img src="images/contribution-path.png"/>
</center>
</div>
@ -351,95 +285,54 @@ corvus@shiprock:~/git/neutron$ git commit -a
corvus@shiprock:~/git/neutron$ git review
remote: Resolving deltas: 0% (0/3)
remote:
remote:
remote: New Changes:
remote: https://review.openstack.org/3072
remote:
remote:
To ssh://corvus@review.openstack.org:29418/openstack/neutron.git
* [new branch] HEAD -> refs/for/master/bug/916018
</pre>
</div>
<div class="slide">
<h1>Types of Tests</h1>
<h1>Gerrit Events</h1>
<center>
<img src="images/gerrit-events.png"/>
</center>
</div>
<div class="slide">
<h1>Feature Request</h1>
<ul>
<li>Unit tests<ul>
<li>Should be run in virtual environments, may run in operating systems</li>
<li>Fairly quick and easy for a developer to run</li>
</ul></li>
<li>Integration tests<ul>
<li>May be able to run on virtual servers, should run on real servers</li>
<li>Difficult and time-consuming for a developer to run</li>
</ul></li>
</li>
<li>Cross project dependencies</li>
<li>Single page diffs</li>
<li>Reviewable tagging</li>
<li>Configurable columns</li>
<li>Sortable columns</li>
<li>Filtering reviews</li>
<li>Grouping reviews</li>
<li>Reduce the noise</li>
<li>Multi-master</li>
</ul>
</div>
<div class="slide">
<h1>Specific Challenges/Solutions</h1>
<h1>Challenges</h1>
<ul>
<li>Testing effect of merging change</li>
<li>Using cloud builders</li>
<li>Large numbers of similar projects</li>
<li>Testing effect of pre-merge change</li>
<li>High volume of changes</li>
<li>Keep mainline clean</li>
<li>Keep feedback loop short</li>
<li>Disparate hardware configurations</li>
</ul>
</div>
<div class="slide">
<h1>Gerrit Git Prep</h1>
<ul>
<li>Test the result of the change, not the change</li>
<li>openstack-infra/config:modules/jenkins/files/slave_scripts/gerrit-git-prep.sh</li>
<li>For each gerrit change:
<ul>
<li>grabs target branch</li>
<li>cleans tree</li>
<li>merges change to be tested</li>
</ul>
</li>
</ul>
</div>
<div class="slide">
<h1>Interrelated Integration Testing</h1>
<ul>
<li>devstack-gate<ul>
<li>boot fresh servers</li>
<li>run DevStack</li>
<li>run integration tests</li>
</ul></li>
<li>Gate proposed change against current state of other projects</li>
</ul>
</div>
<div class="slide">
<h1>Devstack-Gate Problems</h1>
<ul>
<li>Tests are slow</li>
<li>Cloud API calls can fail</li>
<li>External services are unreliable</li>
</ul>
</div>
<div class="slide">
<h1>Devstack-Gate Solutions</h1>
<ul>
<li>Create a new node</li>
<li>Pre-fetch all needed packages, repos</li>
<li>Snapshot to cloud image</li>
<li>Maintain a pool of cloud nodes</li>
<li>Slave can only be used for one test run</li>
<li>Python and shell scripts run as jobs</li>
<li>Partition tests and run them in parallel</li>
</ul>
</div>
<div class="slide">
<h1>Zuul</h1>
<ul>
<li>A general purpose trunk gating system</li>
<li>Interfaces with Gerrit and Jenkins</li>
<li>Interfaces with Gerrit and Gearman/Jenkins</li>
<li>Flexible configuration allows for many kinds of project automation</li>
<li>Allows parallel testing of serialized changes</li>
</ul>
@ -560,7 +453,7 @@ function Dependency(paper, source, target, delay)
this.set = paper.set();
this.source = source;
this.target = target;
startx = this.source.right + 8;
starty = this.source.y;
tipx = this.target.left - 8;
@ -611,11 +504,11 @@ function TestSeries(paper, change, tests)
height = len;
}
}
this.width = (1+10*tests.length)
this.height = offset + height + 10;
this.set = paper.set();
startpath = "M" + this.x + "," + this.y;
line = paper.path(startpath+"l0," + offset);
this.set.push(line);
@ -639,7 +532,7 @@ function TestSeries(paper, change, tests)
curx = curx + 10;
line = paper.path("M" + curx + "," + cury + "l0," + len);
this.set.push(line);
line = paper.path("M" + curx + "," + cury + "m0," + len +
line = paper.path("M" + curx + "," + cury + "m0," + len +
"l0," + (this.height-offset-len));
this.set.push(line);
line.attr("stroke-dasharray", "-");
@ -721,7 +614,7 @@ var zuul_simulation = {
nova_label.attr({x:this.paper.width-(nova_label.getBBox().width/2)-10});
nova_end = [(this.paper.width-nova_label.getBBox().width-16), 300];
nova_line = this.paper.path(linePath(520, 300, nova_end[0], nova_end[1]))
nova_line.attr({"stroke-width": 2, fill: "#000", stroke: "#000"});
nova_head = this.paper.circle(520, 300, 9);
nova_head.attr({fill: "#5069c9", stroke:"#5069c9"});
@ -779,7 +672,7 @@ var zuul_simulation = {
if (this.state == 9) {
change1.merge(480, 290);
dep21.hide();
setTimeout(function() {
setTimeout(function() {
nova_line.attr({path: linePath(490, 300, nova_end[0], nova_end[1])})
} , 2000);
}
@ -791,7 +684,7 @@ var zuul_simulation = {
change2.merge(450, 290);
dep32.hide();
dep42.hide();
setTimeout(function() {
setTimeout(function() {
nova_line.attr({path: linePath(460, 300, nova_end[0], nova_end[1])})
} , 2000);
}
@ -808,7 +701,7 @@ var zuul_simulation = {
}
if (this.state == 15) {
change4.merge(420, 290);
setTimeout(function() {
setTimeout(function() {
nova_line.attr({path: linePath(430, 300, nova_end[0], nova_end[1])})
} , 2000);
}
@ -867,113 +760,25 @@ $(document).ready(function() {
</div>
<div class="slide">
<h1>Zuul Gate Pipeline</h1>
<pre>
pipelines:
- name: gate
manager: DependentPipelineManager
precedence: high
trigger:
gerrit:
- event: comment-added
approval:
- approved: 1
start:
gerrit:
verified: 0
success:
gerrit:
verified: 2
submit: true
failure:
gerrit:
verified: -2
</pre>
<h1>Zuul Status</h1>
<center>
<img style="margin-top:2em;" src="images/status1.png"/>
</center>
</div>
<div class="slide">
<h1>Zuul Project Configuration</h1>
<pre>
projects:
- name: openstack/nova
check:
- gate-nova-pep8
- gate-nova-python26
- gate-nova-python27
- gate-tempest-devstack-vm
gate:
- gate-nova-pep8
- gate-nova-python26
- gate-nova-python27
- gate-tempest-devstack-vm
post:
- nova-branch-tarball
- nova-docs
- nova-upstream-translation-update
release:
- nova-tarball
- nova-docs
</pre>
<h1>Zuul Reports</h1>
<center>
<img style="margin-top:2em;" src="images/zuul_reports1.png"/>
<img style="margin-top:2em;" src="images/zuul_reports2.png"/>
</center>
</div>
<div class="slide">
<h1>Templated Jobs</h1>
<ul>
<li>TONS of consistent projects</li>
<li>Manage everything through Git and code review</li>
<li><a href="http://ci.openstack.org/jenkins-job-builder/">Jenkins Job Builder</a></li>
<li>Manage jobs as YAML files in Git</li>
<li>Jobs updated via Jenkins API</li>
<li>groupings of jobs supported</li>
</ul>
</div>
<div class="slide">
<h1>Example Job Template</h1>
<pre>
- job-template:
name: 'gate-{name}-pep8'
builders:
- gerrit-git-prep
- pep8:
github-org: '{github-org}'
project: '{name}'
publishers:
- console-log
</pre>
</div>
<div class="slide">
<h1>Apply Template</h1>
<pre>
- project:
name: nova
github-org: openstack
tarball-site: tarballs.openstack.org
doc-publisher-site: docs.openstack.org
jobs:
- python-jobs
- gate-{name}-pep8
- translation-jobs
</pre>
</div>
<div class="slide">
<h1>Logs</h1>
<img style="float:right; margin-right:24pt" src="images/logstash-diagram-small.png"/>
<ul style="width: 20em">
<li>Massive amount of logs generated<ul>
<li>Terabytes of compressed logs per development cycle</li>
</ul></li>
<li>Filtered and normalized by LogStash</li>
<li>Recent logs indexed by ElasticSearch<ul>
<li>Searchable through Kibana and ElasticSearch API</li>
</ul></li>
<li>Driving automatic failure classification</li>
</ul>
<h1>Zuul Jobs</h1>
<center>
<img style="margin-top:2em;" src="images/jobs-launched2.png"/>
</center>
</div>
<div class="slide">
@ -990,6 +795,115 @@ projects:
</ul>
</div>
<div class="slide">
<h1>Third Party Reports</h1>
<center>
<img style="margin-top:2em;" src="images/third-party-reports.png"/>
</center>
</div>
<div class="slide">
<h1>Multi-Master Jenkins</h1>
<ul>
<li>Horizontal Scaling</li>
<li>Redundancy</li>
<li>Uses the <a href="https://wiki.jenkins-ci.org/display/JENKINS/Gearman+Plugin">Jenkins Gearman Plugin</a></li>
<li>Masters: jenkins01.openstack.org, jenkins02.o.o, jenkins03.o.o, etc..</li>
<li><a href="http://graphite.openstack.org/render/?from=-24hours&height=680&until=now&width=1034&bgcolor=ffffff&fgcolor=000000&target=color(alias(sumSeries(stats.gauges.nodepool.target.*.*.*.used),%20%27In%20Use%27),%20%276464ff%27)&title=Test%20Nodes&_t=0.605142645072192#1395182490817">600+ test nodes</a></div></li>
</ul>
</div>
<div class="slide">
<h1>Jenkins Slaves</h1>
<center>
<img style="margin-top:2em;" src="images/jenkins_in_use_slaves.png"/>
</center>
</div>
<div class="slide">
<h1>Templated Jobs</h1>
<ul>
<li>TONS of consistent projects</li>
<li>Manage everything through Git and code review</li>
<li><a href="http://ci.openstack.org/jenkins-job-builder/">Jenkins Job Builder</a></li>
<li>Manage jobs as YAML files in Git</li>
<li>Jobs updated via Jenkins API</li>
<li>groupings of jobs supported</li>
</ul>
<div class="indent" style="position: absolute; bottom: 65px"><a href="http://jenkins01.openstack.org">2000+ jobs</a></div>
</div>
<div class="slide">
<h1>Elastic test nodes</h1>
<ul>
<li><a href="http://git.openstack.org/cgit/openstack-infra/nodepool/tree/">Nodepool</a></li>
<li>Automate management of test nodes</li>
<li>Registers nodes with Jenkins masters</li>
<li>Uses Jenkins ZeroMQ events for build status</li>
<li>Manages multiple openstack clouds</li>
</ul>
</div>
<div class="slide">
<h1>Elastic test nodes</h1>o
<center>
<img style="margin-top:2em;" src="images/elastic_test_nodes.png"/>
</center>
</div>
<div class="slide">
<h1>Logs</h1>
<img style="float:right; margin-right:24pt" src="images/logstash-diagram-small.png"/>
<ul style="width: 20em">
<li>Massive amount of logs generated<ul>
<li>Terabytes of compressed logs per development cycle</li>
</ul></li>
<li>Filtered and normalized by LogStash</li>
<li>Recent logs indexed by ElasticSearch<ul>
<li>Searchable through <a href="http://logstash.openstack.org">Kibana</a> and ElasticSearch API</li>
</ul></li>
<li>Driving automatic <a href="http://status.openstack.org/rechecks">failure classification</a> </li>
</ul>
</div>
<div class="slide">
<h1>Query Logs</h1>
<center>
<img style="margin-top:2em;" src="images/kibana.png"/>
</center>
</div>
<div class="slide">
<h1>Elastic Recheck</h1>
<center>
<img style="margin-top:2em;" src="images/rechecks.png"/>
</center>
</div>
<div class="slide">
<h1>Elastic Recheck</h1>
<center>
<img style="margin-top:2em;" src="images/elastic-recheck.png"/>
</center>
</div>
<div class="slide">
<h1>Elastic Recheck</h1>
<center>
<img style="margin-top:2em;" src="images/elastic-recheck-comment.png"/>
</center>
</div>
<div class="slide">
<h1>Everything Is Automated</h1>
<center>
<img src="images/status.png"/>
</center>
</div>
</div>
<div class="slide">
<h1>Thanks!</h1>