opendev
/
publications
Code Issues Proposed changes

Initial commit for publication template branch. 

* .gitrevew: Set defaultbranch to template.

* README.rst: Document how new publications are created using this
template.

* index.html: Create skeleton slide deck with enough content to get
rolling.

* images/images/489px-MySQL.svg.png
* images/Drizzle-med.png
* images/contribution-path.png
* images/contributor-pie-chart.png
* images/gerrit-approved.png
* images/gerrit-diff.png
* images/gerrit-jenkins.png
* images/gerrit-verify.png
* images/gerrit.png
* images/jobs-launched.png
* images/lp-bug.png
* images/lp-os-projects.png
* images/status.png:
Remove images that were not used in the existing overview talk.

* notes: These notes correspond to a different talk, remove them.

* scripts/jquery-1.7.2.min.js
* scripts/raphael-min.js:
Remove js that isn't needed to make the slides go.

Change-Id: I42af21a79d9001ec68582a6b7fba3e3d9f3fbb42
This commit is contained in:
Clark Boylan 2013-09-12 14:44:17 -07:00
parent 2b84bf5c42
commit 154ca6e937
19 changed files with 43 additions and 1085 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.gitreview
View File

@ -2,4 +2,4 @@
host=review.openstack.org
port=29418
project=openstack-infra/publications.git
defaultbranch=overview
defaultbranch=template

139
README.rst
View File

@ -1,116 +1,31 @@
Scaling OpenStack Development: Continuous Integration Overview
==============================================================
Template Branch for new OpenStack Infra Publications
====================================================
Abstract
--------
Creating new OpenStack Infra Publications is easy and you have started
in the right place.
OpenStack is not only the fastest-growing open-source cloud project but
is also a large-scale, complex system with a rapidly expanding code base
and more than 1,000 contributors to date. Handling the quantity and
pace of contributions is a huge challenge on its own. We've been able
to handle the dramatic scale of development by having automation systems
that allow us to treat all developers equally from a process perspective
and keep our trunk always clean by performing testing pre-merge. The
beautiful thing about this approach has been that it doesn't just keep
up with demands, it facilities and encourages more development.
1. Clone publications, create and checkout template tracking branch::
This talk will cover the design and implementation of the current
system, based around a combination of Gerrit and Jenkins, as well as the
workflow that we support and require, how we implemented it and what the
challenges were. At the end of this talk you should have a good
understanding of how OpenStack handles up to 200 contribution
activities an hour.
git clone https://git.openstack.org/openstack-infra/publications
cd publications
git review -s
git checkout -b template origin/template
Talking points
--------------
The OpenStack Grizzly release brought roughly 400,000 new lines of
source code in 10,000 changes from 500 developers over a 6-month period.
Every change passed a battery of style, unit, functional and integration
tests over the course if its development and review, and again before
being merged into the official codebase. This presentation will provide
a high-level examination of the techniques used to coordinate and
automate software development efforts at such a scale.
* Projects:
- scaling challenge faced as a community
- number of individual software projects being integrated
- server projects are part of an integrated release
- client library projects are released on separate schedules
- incubated projects shown in gray
* Programs/Horizontal Efforts:
- scaling challenge faced as a community
- many important contributions are not focused on a project
- might span multiple software projects
- could be something other than traditional software development
* Release Management:
- scaling challenge faced as a community
- integrated release and roadmap follow a rigid schedule
- limitations chosen to improve output and quality
* Contributors:
- scaling challenge faced as a community
- contributors come from varied places and backgrounds
- different levels of involvement
- have a variety of goals and motives (usually not a bad thing)
- there are many, many, many of them
* Consistent Tooling:
- meta-development happens independently without consistency
- consistency increases throughput with fewer people
* Developer Infrastructure:
- both a scaling challenge and solution
- vast array of systems used for development efforts
* Developer Environment
- operating systems and python versions supported (and how that
impacts testing)
- developers on IRC aids distributed development
- openstack is complicated to install; devstack makes it simple
- lots of testing helps stabilize rapid development and
facilitates contributions
* Project Gating
* Everything Is Automated
* Process Flow
* Gerrit
* Gerrit (screenshot)
- change information
- commit message
- bug and blueprint linkage
- review status, categories
- patchsets
- comments
- pre merge testing
- gating
* Gerrit Diff View
* Bug Integration - Launchpad
* Approved Reviews
* Git Review
* Types of Tests
* Specific Challenges/Solutions
* Gerrit Git Prep
* Interrelated Integration Testing
* Devstack-Gate Problems
* Devstack-Gate Solutions
* Zuul
* Bottlenecking
* Zuul Simulation:
1. branch tip of repositories for nova and keystone
2. changes enter the gate pipeline, merged to a test queue
3. zuul initiates tests on each in parallel, keystone fails
4. first change passes all tests
5. merged to nova and becomes the branch tip
6. second change passes all tests
7. merged to nova and becomes the branch tip
8. third change is ejected and keystone branch tip is unchanged
9. fourth change is merged to a new test queue, now in first
10. zuul restarts tests on the new queue of one change
11. first change passes all tests
12. merged to nova and becomes the branch tip
* Zuul Check Pipeline
* Zuul Gate Pipeline
* Zuul Post-Merge Pipeline
* Zuul Release Pipeline
* Zuul Project Configuration
* Templated Jobs
* Example Job
* Example Template
* Scaling Hardware Needs
* Thanks!
2. Create a new branch based on this template branch.
``git checkout -b $BRANCH_NAME``.
3. Edit ``.gitreview`` and change the defaultbranch value to
``$BRANCH_NAME``.
4. Create ``$BRANCH_NAME`` in Gerrit. It should be based on the
template branch as well to avoid any potential merge conflicts in
your first commit. Not everyone has the ability to do this in
Gerrit. If you don't, ask an openstack infra core to do it for you.
5. Now we get to do the fun stuff. Edit index.html editing lines with
``CHANGEME`` or ``changeme`` in them. You can also add new slides
if you like but that isn't necessary in this bootstrapping change.
You can follow up with new slides in subsequent changes.
6. Edit this file, ``README.rst``. The title of this document will
be the name used on the root publications index so be sure to
update the title. You should also add an Abstract section and
general talk info.
7. At this point you are ready to push your new changes back up to
Gerrit. ``git commit -a && git review``

BIN
images/489px-MySQL.svg.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 19 KiB

BIN
images/Drizzle-med.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 26 KiB

BIN
images/contribution-path.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 92 KiB

BIN
images/contributor-pie-chart.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 24 KiB

BIN
images/gerrit-approved.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 121 KiB

BIN
images/gerrit-diff.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 200 KiB

BIN
images/gerrit-jenkins.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 120 KiB

BIN
images/gerrit-verify.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 98 KiB

BIN
images/gerrit.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 418 KiB

BIN
images/jobs-launched.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 21 KiB

BIN
images/lp-bug.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 124 KiB

BIN
images/lp-os-projects.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 33 KiB

BIN
images/status.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 80 KiB

904
index.html
View File

@ -3,7 +3,7 @@
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en-US">
<head>
<title>Scaling OpenStack Development: Continuous Integration Overview</title>
<title>Publication Title CHANGE ME</title>
<meta name="generator" content=
"HTML Tidy for Linux/x86 (vers 1st November 2003), see www.w3.org" />
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
@ -14,11 +14,6 @@
<link rel="stylesheet" href="styles/slidy.css" type="text/css" />
<link rel="stylesheet" href="styles/openstack.css" type="text/css" />
<script src="scripts/slidy.js" charset="utf-8" type="text/javascript"></script>
<script src="scripts/jquery-1.7.2.min.js" charset="utf-8" type="text/javascript">
</script>
<script src="scripts/jquery-watch.js" charset="utf-8" type="text/javascript">
</script>
<script src="scripts/raphael-min.js" type="text/javascript" charset="utf-8"></script>
</head>
<body>
<div class="background"><img alt="" id="head-icon"
@ -38,900 +33,31 @@ src="graphics/openstack-cloud-software-horizontal-small.png" /></div>
<img class="hidden" src="graphics/bullet-nofold-dim.gif" alt="" />
<img class="hidden" src="graphics/bullet-unfold-dim.gif" alt="" />
<img src="graphics/openstack-cloud-software-vertical-large.png" alt="OpenStack logo" class="cover" />
<h1>Scaling OpenStack Development<br />
<span class="smaller">Continuous Integration Overview</span></h1>
<h1>Publication Title CHANGE ME<br />
<span class="smaller">Publication SubTitle CHANGE ME</span></h1>
<hr />
<div class="smaller">Monty Taylor
&lt;<a href="mailto:mordred@inaugust.com">mordred@inaugust.com</a>&gt;</div>
<div class="smaller">James E. Blair
&lt;<a href="mailto:jeblair@openstack.org">jeblair@openstack.org</a>&gt;</div>
<div class="smaller">Jeremy Stanley
&lt;<a href="mailto:fungi@yuggoth.org">fungi@yuggoth.org</a>&gt;</div>
<div class="smaller">Your Name CHANGE ME
&lt;<a href="mailto:changeme@example.com">changeme@example.com</a>&gt;</div>
</div>
<div class="slide">
<h1>OpenStack</h1>
<h1>OpenStack</h1>
<p>Is open source software for building private and public clouds.</p>
<p>This is the first slide. CHANGE ME and add many more.</p>
<center>
<img src="images/openstack-software-diagram.png"/>
</center>
<center>
<img src="images/openstack-software-diagram.png"/>
</center>
</div>
<div class="slide">
<h1>Projects</h1>
<div style="float:left;width:50%"><ul>
<li>Servers<ul>
<li>nova (compute)</li>
<li>swift (object storage)</li>
<li>glance (image service)</li>
<li>keystone (identity service)</li>
<li>neutron (network service)</li>
<li>cinder (volume service)</li>
<li>heat (orchestration)</li>
<li>ceilometer (measurement)</li>
<li>horizon (dashboard)</li>
<li class="gray">trove (databases)</li>
<li class="gray">ironic (bare metal)</li>
</ul></li>
</ul></div>
<div style="float:left;width:50%"><ul>
<li>Client libraries<ul>
<li>python-novaclient</li>
<li>python-swiftclient</li>
<li>python-glanceclient</li>
<li>python-keystoneclient</li>
<li>python-neutronclient</li>
<li>python-cinderclient</li>
<li>python-heatclient</li>
<li>python-ceilometerclient</li>
<li>python-openstackclient</li>
<li class="gray">python-troveclient</li>
<li class="gray">python-ironicclient</li>
</ul></li>
</ul></div>
<div class="indent"><a href="https://wiki.openstack.org/wiki/Projects">https://wiki.openstack.org/wiki/Projects</a></div>
</div>
<h1>Thanks!</h1>
<div class="slide">
<h1>Programs/Horizontal Efforts</h1>
<div style="float:left;width:50%"><ul>
<li>Documentation<ul>
<li>installation guides</li>
<li>operations manuals</li>
<li>API references</li>
<li>howtos and tutorials</li>
</ul></li>
<li>Infrastructure<ul>
<li>continuous integration systems</li>
<li>developer community resources</li>
</ul></li>
</ul></div>
<div style="float:left;width:50%"><ul>
<li>Oslo<ul>
<li>common libraries</li>
<li>copied code incubation</li>
</ul></li>
<li>Quality Assurance<ul>
<li>integration testing</li>
<li>upgrade testing</li>
</ul></li>
<li>Release Management</li>
<li>Translation/Internationalization</li>
<li>Vulnerability Management</li>
</ul></div>
</div>
<p><img src="images/stack-o-pancakes-150x150.png"/>
<div class="slide">
<h1>Release Management</h1>
<ul>
<li>Time Based Releases</li>
<li>Six Month Cadence<ul>
<li>Similar to Ubuntu and Fedora Release Schedules</li>
</ul>
<li>Design summits each cycle</li>
<li>Continuously Open Trunk<ul>
<li>Develop directly on master</li>
</ul>
<li>Intra-Cycle Milestone Releases</li>
<li>Post-Release Stable Branches</li>
</ul>
</div>
<div class="slide">
<h1>Contributors</h1>
<img style="float:right; margin-right:24pt" src="images/contributor-pie-chart.png"/>
<ul>
<li>Unaffiliated individuals</li>
<li>Commercial entities</li>
<li>Nonprofit organizations</li>
<li>National and local governments</li>
<li>Number, quality, and area of contributions can change daily</li>
</ul>
</div>
<div class="slide">
<h1>Consistent Tooling</h1>
<ul>
<li>Minimize meta-development</li>
<li>Process divergence == wasted developer time</li>
<li>Lowers onboarding time</li>
<li>Consolidate tool development</li>
<li>Minimize project-specific weird build crud</li>
</ul>
</div>
<div class="slide">
<h1>Developer Infrastructure</h1>
<small>
<div style="float:left;width:33%"><ul>
<li>Code review and VCS<ul>
<li>Gerrit</li>
<li>Git</li>
<li>git-review</li>
<li>reviewday</li>
</ul></li>
<li>Test/build automation<ul>
<li>devstack-gate</li>
<li>Jenkins</li>
<li>JJB</li>
<li>Zuul</li>
</ul></li>
<li>Repository mirrors<ul>
<li>Apache</li>
<li>Cgit</li>
<li>GitHub</li>
<li>PyPI packages</li>
</ul></li>
</ul></div>
<div style="float:left;width:33%"><ul>
<li>Job logs, artifacts<ul>
<li>logs site</li>
<li>Logstash</li>
</ul></li>
<li>Documentation/publication<ul>
<li>Sphinx</li>
<li>Wiki</li>
</ul></li>
<li>Releases/pre-releases<ul>
<li>PyPI uploads</li>
<li>tarballs</li>
</ul></li>
<li>IRC<ul>
<li>eavesdrop</li>
<li>gerritbot</li>
<li>meetbot</li>
<li>statusbot</li>
</ul></li>
</ul></div>
<div style="float:left;width:33%"><ul>
<li>Project blogs<ul>
<li>Foundation</li>
<li>Planet</li>
</ul></li>
<li>Collaboration<ul>
<li>Mailing lists</li>
<li>Lodgeit paste</li>
<li>Etherpad</li>
</ul></li>
<li>Authentication<ul>
<li>CLA</li>
<li>membership</li>
<li>Launchpad SSO</li>
</ul></li>
<li>Feature/bug tracking<ul>
<li>LP blueprints</li>
<li>LP bugs</li>
</ul></li>
<!-- TODO: add transifex? -->
</ul></div>
</small>
</div>
<div class="slide">
<h1>Development Environment</h1>
<ul>
<li>Python<ul>
<li>CentOS (2.6), Ubuntu LTS (2.7), Fedora (3.3)</li>
<li>PEP-8 standards</li>
<li>Oslo (common libraries)</li>
<li>virtualenv/pip/tox</li>
</ul></li>
<li>Freenode IRC (#openstack-dev, #openstack-meeting)</li>
<li>DevStack</li>
<li>Tests run on all newly submitted changes</li>
<li>Code merges are gated on tests</li>
</ul>
</div>
<div class="slide">
<h1>Project Gating</h1>
<img style="float:right; margin-right:24pt" src="images/jobs-launched.png"/>
<!--
http://graphite.openstack.org/render/?from=00%3A00_20130601&fgcolor=000000&title=Jobs%20Launched%20%28per%20Day%29&_t=0.7862690862083328&height=300&bgcolor=ffffff&width=400&showTarget=alias%28summarize%28sumSeries%28stats_counts.zuul.job.*%29%2C%271h%27%29%2C%27All%20Jobs%27%29&_salt=1373041766.855&target=alias%28summarize%28sumSeries%28stats_counts.zuul.job.*%29%2C%271d%27%29%2C%27All%20Jobs%27%29&until=23%3A59_20130630
-->
<ul>
<li>Ensures Code Quality</li>
<li>Protects developers<ul>
<li>Devs always start from working code</li>
</ul><li>Protects tree<ul>
<li>Bad code doesn't land</li>
</ul><li>Egalitarian<ul>
<li>Process is the same for everyone</li>
<li>Process is transparent</li>
<li>Process is automated</li>
</ul></ul>
</ul>
</div>
<div class="slide">
<h1>Everything Is Automated</h1>
<center>
<img src="images/status.png"/>
</center>
</div>
</div>
<div class="slide">
<h1>Process Flow</h1>
<center>
<img src="images/contribution-path.png"/>
</center>
</div>
<div class="slide">
<h1>Gerrit</h1>
<ul>
<li>Developed by Google for Android</li>
<li>Stand-alone patch review system</li>
<li>Integration points: hooks, JSON queries, event-stream</li>
<li>Extensible review categories, default: Verified, Code-Review</li>
</ul>
</div>
<div class="slide">
<h1>Gerrit</h1>
<center>
<img style="margin-top:2em" src="images/gerrit.png"/>
</center>
</div>
<div class="slide">
<h1>Gerrit Diff View</h1>
<center>
<img style="margin-top:2em" src="images/gerrit-diff.png"/>
</center>
</div>
<div class="slide">
<h1>Bug Integration - Launchpad</h1>
<center>
<img style="margin-top:2em" src="images/lp-bug.png"/>
</center>
</div>
<div class="slide">
<h1>Approved Reviews</h1>
<center>
<img style="margin-top:2em"src="images/gerrit-approved.png"/>
</center>
</div>
<div class="slide">
<h1>Git Review</h1>
<ul>
<li>External Git subcommand</li>
<li>Developers can easily incorporate code review into git workflow</li>
<li>Zero-configuration</li>
<li>Can be used for any project, being adopted by other projects</li>
</ul>
<pre>
corvus@shiprock:~/git/neutron$ git commit -a
[new-versionpy ddf1dce] Base version.py on glance.
3 files changed, 28 insertions(+), 107 deletions(-)
delete mode 100644 version.py
corvus@shiprock:~/git/neutron$ git review
remote: Resolving deltas: 0% (0/3)
remote:
remote: New Changes:
remote: https://review.openstack.org/3072
remote:
To ssh://corvus@review.openstack.org:29418/openstack/neutron.git
* [new branch] HEAD -> refs/for/master/bug/916018
</pre>
</div>
<div class="slide">
<h1>Types of Tests</h1>
<ul>
<li>Unit tests<ul>
<li>Should be run in virtual environments, may run in operating systems</li>
<li>Fairly quick and easy for a developer to run</li>
</ul></li>
<li>Integration tests<ul>
<li>May be able to run on virtual servers, should run on real servers</li>
<li>Difficult and time-consuming for a developer to run</li>
</ul></li>
</li>
</ul>
</div>
<div class="slide">
<h1>Specific Challenges/Solutions</h1>
<ul>
<li>Testing effect of merging change</li>
<li>Using cloud builders</li>
<li>Large numbers of similar projects</li>
<li>Disparate hardware configurations</li>
</ul>
</div>
<div class="slide">
<h1>Gerrit Git Prep</h1>
<ul>
<li>Test the result of the change, not the change</li>
<li>openstack-infra/config:modules/jenkins/files/slave_scripts/gerrit-git-prep.sh</li>
<li>For each gerrit change:
<ul>
<li>grabs target branch</li>
<li>cleans tree</li>
<li>merges change to be tested</li>
</ul>
</li>
</ul>
</div>
<div class="slide">
<h1>Interrelated Integration Testing</h1>
<ul>
<li>devstack-gate<ul>
<li>boot fresh servers</li>
<li>run DevStack</li>
<li>run integration tests</li>
</ul></li>
<li>Gate proposed change against current state of other projects</li>
</ul>
</div>
<div class="slide">
<h1>Devstack-Gate Problems</h1>
<ul>
<li>Tests are slow</li>
<li>Cloud API calls can fail</li>
<li>External services are unreliable</li>
</ul>
</div>
<div class="slide">
<h1>Devstack-Gate Solutions</h1>
<ul>
<li>Create a new node</li>
<li>Pre-fetch all needed packages, repos</li>
<li>Snapshot to cloud image</li>
<li>Maintain a pool of cloud nodes</li>
<li>Slave can only be used for one test run</li>
<li>Python and shell scripts run as jobs</li>
</ul>
</div>
<div class="slide">
<h1>Zuul</h1>
<ul>
<li>A general purpose trunk gating system</li>
<li>Interfaces with Gerrit and Jenkins</li>
<li>Flexible configuration allows for many kinds of project automation</li>
<li>Allows parallel testing of serialized changes</li>
</ul>
</div>
<div class="slide">
<h1>Bottlenecking</h1>
<ul>
<li>Serialize changes across all projects</li>
<li>Speculative execution of tests</li>
<li>Run in parallel in order triggered</li>
<li>Assume success</li>
<li>Start over on failure</li>
</ul>
</div>
<div class="slide" id="zuul">
<h1>Zuul Simulation</h1>
<script>
var green = "#00bb44";
var red = "#bb0000";
var light_green = "#8dbb9e";
var light_red = "#bb7777";
function Change(paper, name, number, x, y, delay)
{
this.set = paper.set();
this.x = x;
this.y = y;
this.text = paper.text(x, y, name+'\n#'+number);
this.set.push(this.text);
this.text.attr({"font-size": 16});
width = this.text.getBBox().width+16;
height = this.text.getBBox().height+12;
this.left = x-width/2;
this.right = x+width/2;
this.top = y-height/2;
this.bottom = y+height/2;
this.rect = paper.rect(this.left, this.top, width, height, 2);
this.set.push(this.rect);
this.rect.attr({fill: "#ccc", stroke: "#bbb"});
this.rect.toBack();
this.set.attr({opacity: 0});
this.set.animate(Raphael.animation({opacity: 1}, 500, 'linear').delay(delay));
this.hide = function ()
{
var set = this.set;
var remove = function () { set.remove(); };
this.set.animate({opacity: 0}, 1000, 'linear', remove);
}
this.merge = function(x, y)
{
this.set.toFront();
this.text.animate(Raphael.animation({opacity: 0}, 500, 'linear'));
this.rect.animate(Raphael.animation(
{width: 18, height: 18, r: 9, x: x, y: y, stroke: green},
1000).delay(500));
}
this.provisionalResult = function(success)
{
if (success)
{
color = light_green;
} else {
color = light_red;
}
this.rect.animate({fill: color}, 500);
}
this.result = function(success)
{
if (success)
{
color = green;
} else {
color = red;
}
this.rect.animate({fill: color}, 500);
}
this.reset = function(success)
{
this.rect.animate({fill: "#ccc"}, 500);
}
}
function linePath(x1, y1, x2, y2) {
return "M"+x1+","+y1+"L"+x2+","+y2;
}
function Dependency(paper, source, target, delay)
{
this.set = paper.set();
this.source = source;
this.target = target;
startx = this.source.right + 8;
starty = this.source.y;
tipx = this.target.left - 8;
tipy = this.target.y;
path = "M" + (startx) + "," + starty;
path = path + "L" + (tipx-15) + "," + tipy;
this.line = paper.path(path);
this.set.push(this.line);
this.line.attr("stroke-width", "2");
this.line.attr("fill", "#000");
this.line.attr("stroke", "#000");
path = "M" + (tipx - 20) + "," + (tipy - 10);
path = path + "L" + (tipx) + "," + (tipy);
path = path + "L" + (tipx - 20) + "," + (tipy + 10);
path = path + "Q" + (tipx - 12) + "," + (tipy);
path = path + "," + (tipx - 20) + "," + (tipy - 10);
path = path + "Z";
this.arrow = paper.path(path);
this.set.push(this.arrow);
this.arrow.attr("fill", "#000");
this.arrow.attr("stroke", "#000");
this.set.attr({opacity: 0});
this.set.animate(Raphael.animation({opacity: 1}, 500, 'linear').delay(delay));
this.hide = function ()
{
var set = this.set;
var remove = function () { set.remove(); };
this.set.animate({opacity: 0}, 1000, 'linear', remove);
}
}
function TestSeries(paper, change, tests)
{
this.x = change.x - 20;
this.y = change.bottom + 6;
offset = 40; // the "merge" test length
height = 0;
for (test in tests)
{
len = tests[test][0];
if (len > height)
{
height = len;
}
}
this.width = (1+10*tests.length)
this.height = offset + height + 10;
this.set = paper.set();
startpath = "M" + this.x + "," + this.y;
line = paper.path(startpath+"l0," + offset);
this.set.push(line);
startpath = "M" + this.x + "," + (this.y + offset);
line = paper.path(startpath+"l0," + (this.height - offset));
this.set.push(line);
line.attr("stroke-dasharray", "-");
line.attr("stroke", green);
line = paper.path(startpath + "l" + (1+10*tests.length) + ",0");
this.set.push(line);
var global_success = true;
curx = this.x;
cury = this.y + offset;
for (var test in tests)
{
len = tests[test][0];
success = tests[test][1];
curx = curx + 10;
line = paper.path("M" + curx + "," + cury + "l0," + len);
this.set.push(line);
line = paper.path("M" + curx + "," + cury + "m0," + len +
"l0," + (this.height-offset-len));
this.set.push(line);
line.attr("stroke-dasharray", "-");
if (success) {
line.attr("stroke", green);
} else {
line.attr("stroke", red);
global_success = false;
}
}
var complete = function ()
{
change.provisionalResult(global_success);
}
this.rect = paper.rect(this.x-2, this.y-2, this.width+4, this.height+2);
this.rect.attr({fill: "#fff", stroke: "#fff"})
this.rect.animate({y: this.y+this.height+2, height: 0}, 36*this.height, //36
'linear', complete);
this.hide = function ()
{
var set = this.set;
var remove = function () { set.remove(); };
this.set.animate({opacity: 0}, 2000, 'linear', remove);
}
}
var zuul_simulation = {
paper: null,
state: 0,
finished: false,
setup: function()
{
this.paper = Raphael(200, 200, 760, 400);
this.finished = false;
this.state = 0;
},
teardown: function()
{
if (this.paper != null) {
this.paper.remove();
this.paper = null;
}
this.finished = true;
},
next: function()
{
this.state = this.state + 1;
if (this.state == 1)
{
head_label = this.paper.text(520, 385, "HEAD");
head_label.attr({"font-size": 16});
nova_label = this.paper.text(520, 300, "nova");
nova_label.attr({"font-size": 16});
nova_label.attr({x:this.paper.width-(nova_label.getBBox().width/2)-10});
nova_end = [(this.paper.width-nova_label.getBBox().width-16), 300];
nova_line = this.paper.path(linePath(520, 300, nova_end[0], nova_end[1]))
nova_line.attr({"stroke-width": 2, fill: "#000", stroke: "#000"});
nova_head = this.paper.circle(520, 300, 9);
nova_head.attr({fill: "#5069c9", stroke:"#5069c9"});
keystone_label = this.paper.text(520, 360, "keystone");
keystone_label.attr({"font-size": 16});
keystone_label.attr({x:this.paper.width-(keystone_label.getBBox().width/2)-10});
keystone_end = [(this.paper.width-keystone_label.getBBox().width-16), 360];
keystone_line = this.paper.path(linePath(520, 360, keystone_end[0], keystone_end[1]));
keystone_line.attr({"stroke-width": 2, fill: "#000", stroke: "#000"});
keystone_head = this.paper.circle(520, 360, 9);
keystone_head.attr({fill: "#eead5f", stroke:"#eead5f"});
}
if (this.state == 2)
{
change1 = new Change(this.paper, 'nova', '1', 480, 40, 0);
change2 = new Change(this.paper, 'nova', '2', 340, 40, 1000);
dep21 = new Dependency(this.paper, change2, change1, 1800);
change3 = new Change(this.paper, 'keystone', '3', 180, 40, 2000);
dep32 = new Dependency(this.paper, change3, change2, 2800);
change4 = new Change(this.paper, 'nova', '4', 40, 40, 3000);
dep43 = new Dependency(this.paper, change4, change3, 3800);
}
if (this.state == 3) {
change1_test = new TestSeries(this.paper, change1, [[80, 1], [60, 1], [95, 1]]);
change2_test = new TestSeries(this.paper, change2, [[80, 1], [60, 1], [90, 1]]);
change3_test = new TestSeries(this.paper, change3, [[50, 1], [30, 0]]);
change4_test = new TestSeries(this.paper, change4, [[80, 1], [60, 1], [85, 0]]);
}
if (this.state == 4) {
change1_test.hide();
change1.result(1);
}
if (this.state == 5) {
change1.merge(480, 290);
dep21.hide();
setTimeout(function() {
nova_line.attr({path: linePath(490, 300, nova_end[0], nova_end[1])})
} , 2000);
}
if (this.state == 6) {
change2_test.hide();
change2.result(1);
}
if (this.state == 7) {
change2.merge(450, 290);
dep32.hide();
setTimeout(function() {
nova_line.attr({path: linePath(460, 300, nova_end[0], nova_end[1])})
} , 2000);
}
if (this.state == 8) {
change3_test.hide();
change3.result(0);
}
if (this.state == 9) {
change4.reset();
change4_test.hide();
change3.hide();
dep43.hide();
}
if (this.state == 10) {
change4_test = new TestSeries(this.paper, change4, [[80, 1], [60, 1], [85, 1]]);
}
if (this.state == 11) {
change4_test.hide();
change4.result(1);
}
if (this.state == 12) {
change4.merge(420, 290);
setTimeout(function() {
nova_line.attr({path: linePath(430, 300, nova_end[0], nova_end[1])})
} , 2000);
}
if (this.state == 13) {
this.finished = true;
}
},
}; /* zuul_simulation */
$(document).ready(function() {
$("#zuul").bind("DOMSubtreeModified", function() {
if ($(this).hasClass("hidden")) {
$("#zuul").unbind('click');
w3c_slidy.mouse_click_enabled = true;
zuul_simulation.teardown();
} else {
zuul_simulation.setup();
zuul_simulation.next();
w3c_slidy.mouse_click_enabled = false;
$("#zuul").bind('click', function() {
zuul_simulation.next();
if (zuul_simulation.finished) {
$("#zuul").unbind('click');
w3c_slidy.mouse_click_enabled = true;
}
});
}
}); /* dom modified */
}); /* document ready */
</script>
</div>
<div class="slide">
<h1>Zuul Check Pipeline</h1>
<pre>
pipelines:
- name: check
manager: IndependentQueueManager
trigger:
- event: patchset-created
success:
verified: 1
failure:
verified: -1
</pre>
</div>
<div class="slide">
<h1>Zuul Gate Pipeline</h1>
<pre>
pipelines:
- name: gate
manager: DependentPipelineManager
trigger:
- event: comment-added
approval:
- approved: 1
start:
verified: 0
success:
verified: 2
submit: true
failure:
verified: -2
</pre>
</div>
<div class="slide">
<h1>Zuul Post-Merge Pipeline</h1>
<pre>
pipelines:
- name: post
manager: IndependentQueueManager
trigger:
- event: ref-updated
ref: ^(?!refs/.*).*$
</pre>
</div>
<div class="slide">
<h1>Zuul Release Pipeline</h1>
<pre>
pipelines:
- name: release
manager: IndependentPipelineManager
trigger:
- event: ref-updated
ref: ^refs/tags/([0-9]+\.)+[0-9]+$
</pre>
</div>
<div class="slide">
<h1>Zuul Project Configuration</h1>
<pre>
projects:
- name: openstack/nova
check:
- gate-nova-pep8
- gate-nova-python26
- gate-nova-python27
- gate-tempest-devstack-vm
gate:
- gate-nova-pep8
- gate-nova-python26
- gate-nova-python27
- gate-tempest-devstack-vm
post:
- nova-branch-tarball
- nova-docs
- nova-upstream-translation-update
release:
- nova-tarball
- nova-docs
</pre>
</div>
<div class="slide">
<h1>Templated Jobs</h1>
<ul>
<li>TONS of consistent projects</li>
<li>Manage everything through Git and code review</li>
<!-- TODO: link to jjb documentation instead --><li><a href="https://github.com/openstack/openstack-ci-puppet/tree/master/modules/jenkins_jobs">https://github.com/openstack/openstack-ci-puppet/tree/master/modules/jenkins_jobs</a></li>
<li>Manage jobs as YAML files in Git</li>
<li>Jobs updated via Jenkins API</li>
<li>groupings of jobs supported</li>
</ul>
</div>
<div class="slide">
<h1>Example Job</h1>
<pre>
- job:
name: gate-nova-pep8
builders:
- gerrit_git_prep
- pep8
publishers:
- console-log
</pre>
</div>
<div class="slide">
<h1>Example Template</h1>
<pre>
- project:
name: nova
github-org: openstack
tarball-site: tarballs.openstack.org
doc-publisher-site: docs.openstack.org
jobs:
- python-jobs
- python-folsom-bitrot-jobs
- python-grizzly-bitrot-jobs
- openstack-publish-jobs
- gate-{name}-pylint
- translation-jobs
</pre>
</div>
<div class="slide">
<h1>Scaling Hardware Needs</h1>
<ul>
<li>Different vendors have different combinations of configurations</li>
<li><a
href="http://ci.openstack.org/third_party.html#the-jenkins-gerrit-trigger-plugin-way">http://ci.openstack.org/third_party.html#the-jenkins-gerrit-trigger-plugin-way</a></li><!-- TODO: update this URL -->
<li>Run their own Jenkins</li>
<li>Consume Gerrit event stream</li>
<li>Their Jenkins votes VRFY +1/-1</li>
<li>Our Zuul votes VRFY +2/-2</li>
<li>They can run their lab - we don't have to know anything about it</li>
</ul>
</div>
<div class="slide">
<h1>Thanks!</h1>
<p><img src="images/stack-o-pancakes-150x150.png"/>
<p>
These slides available at: <a href="http://docs.openstack.org/infra/publications/">http://docs.openstack.org/infra/publications/</a>
</p>
<p>
These slides available at: <a href="http://docs.openstack.org/infra/publications/">http://docs.openstack.org/infra/publications/</a>
</p>
</div>

69
notes
View File

@ -1,69 +0,0 @@
[title]
Hi. I'm Monty This is Jim.
[Lineage]
Our build infrasturcture and many of its philosophies come from:
MySQL - Launchpad/bzr
Drizzle - gated trunk, using all of the launchpad features
OpenStack - automated gated trunk via tarmac
[gated trunk]
So with openStack, we did automated checks of every proposed commit through
jenkins and everything was happy
[Git Revolt]
Then we had a revolt, because the developers wanted to use git.
Launchpad has no git support (or in-line code reviews)
Github's pull-request have no approval state
[Enter Gerrit]
Gerrit, from google used by android, has all of the things we needed
Except for Launchpad integration
So we added it
[Bug Integration]
Gerrit review launchpad links on bug metions
Gerrit topics set from bugs
[Launchpad side]
Gerrit sends bug control emails with links to the review and the commit
State changes- in progress when review is submitted. Fix committed when
patch is accepted
[Blueprint Integration]
Blueprint links in commit messages
Topics set from blueprint mentions
[Launchpad side]
Inject information on commits to blueprints just like bugs
Have to use the whiteboard
Link to review - but also to the general gerrit topic, since a blueprint
might have more than one branch
[SSO]
Gerrit and Jenkins both use Launchpad OpenID for auth
[Jenkins Integration]
Gerrit has deep jenkins integration. Listens on an ssh stream and takes
active action. Posts back to the review with jenkins links on success or
failure
[Gerrit Verification]
Jenkins fills the gerrit role of verifier.
In addition to being approved or denied, a given change can be verified as
working or not.
[Feature REquest time]
Subscribably event triggers
Verification status
launchpadlib api bindings in non-python
[While we're at it]
Foreign merge props - bugs were cool, lots of people do code review
elsewhere
mirrored branches
Branch/merge prop integration with blueprints
Structured info in blueprints

4
scripts/jquery-1.7.2.min.js vendored
View File

File diff suppressed because one or more lines are too long

10
scripts/raphael-min.js vendored
View File

File diff suppressed because one or more lines are too long