Added in my notes so far.
This commit is contained in:
Monty Taylor
parent
6d74587a66
commit
870b8d400f
|
|
@ -36,9 +36,10 @@ src="graphics/open-stack-cloud-computing-logo-2.png" /></div>
|
|||
|
||||
<img src="graphics/openstack-cloud-software-vertical-large.png" alt="OpenStack logo"
|
||||
class="cover" /><br clear="all" />
|
||||
<h1>OpenStack Integration with Launchpad</h1>
|
||||
<h1>How OpenStack integrates Launchpad and Continuous Integration</h1>
|
||||
|
||||
<p>Monty Taylor
|
||||
<p>
|
||||
Monty Taylor
|
||||
<<a href="mailto:mordred@inaugust.com">mordred@inaugust.com</a>><br />
|
||||
James E. Blair
|
||||
<<a href="mailto:corvus@inaugust.com">corvus@inaugust.com</a>><br />
|
||||
|
|
@ -48,11 +49,269 @@ James E. Blair
|
|||
<br /><em>Hit the space bar or swipe right for next slide</em></p>
|
||||
</div>
|
||||
|
||||
<div class="slide">
|
||||
<h1></h1>
|
||||
<!--
|
||||
I'm Monty Taylor and this is Jim Blair. We're with OpenStack and run the
|
||||
core developer infrastructure for the project, including the Code Review
|
||||
and Continuous Integration systems, which is what we're here to talk to you
|
||||
about today.
|
||||
|
||||
<img src="images/"/>
|
||||
OpenStack's developer infrastructure has a lineage that is mostly based on
|
||||
what we used in Drizzle, with some things we learned from the bzr/launchpad
|
||||
teams, and then a bunch of taking the next step. The key parts are that we
|
||||
manage the project on Launchpad using just about all of the features that
|
||||
are there- bugs, complete with use of release series and milestones,
|
||||
blueprints, ppas, translations. Drizzle was one of the first people who
|
||||
weren't Ubuntu to use blueprints extensively - OpenStack continues the
|
||||
trend.
|
||||
|
||||
One of the biggest philosophical points we took from Drizzle was having a
|
||||
consistent and automated developer infrastructure. Trunk should never break-
|
||||
code should be clean, and this should be enforced by computers. In OpenStack
|
||||
we started using tarmac to further automate the processing of approved merge
|
||||
requests and only letting code be merged if it passes tests (we call this
|
||||
trunk gating) and it served us well for our first two releases.
|
||||
|
||||
Then we had a revolt in OpenStack because a indeterminate amount of people
|
||||
hated bzr, and a possibly intersecting set of people who found launchpad's
|
||||
code reviews insufficient. (speed and lack of in-line review comments were
|
||||
the two biggest consistent complaints) So we were tasked with replacing code
|
||||
hosting and code reviews with something else (the main suggestion being
|
||||
github, which we did investigate)
|
||||
|
||||
As we looked at the situation, we knew several things:
|
||||
- launchpad had no git support, so staying with Launchpad and just moving
|
||||
code to git was a non-starter
|
||||
- we wanted to stay on Launchpad for bugs and blueprints, as the features
|
||||
contained within them do not have an analogue on other systems
|
||||
- we wanted to keep a system where we gated trunk.
|
||||
|
||||
Long story short - github was an unusable choice for our workflow for two
|
||||
reasons:
|
||||
1) It had no workflow concept of a pull request being "approved" or
|
||||
"rejected" - only textual votes in the comments.
|
||||
2) The folks at github were unwilling to add that state concept, and github
|
||||
is not open source, so we couldn't do it ourselves
|
||||
|
||||
Gerrit, from google and used for Android development, on the other hand, did
|
||||
have the workflow elements that we needed (and a few new ones we're going to
|
||||
point out) So we moved forward with gerrit, and then started doing work to
|
||||
integrate it with Launchpad ... and we're pretty happy with the results
|
||||
(honestly, happier than I expected to be)
|
||||
|
||||
-----
|
||||
|
||||
First of all, we wanted bug integration just like we had with Launchpad
|
||||
before. So we wrote a gerrit-side hook which examines the commit message for
|
||||
references to bugs (similar to closing bugs in debian/changelog files) and
|
||||
we have it send a gpg-signed email to the bug with information about the
|
||||
code review instance in gerrit, and commands to operate the appropriate
|
||||
state-changes needed on the bug. We've currently decided that a bug will
|
||||
transition to In Progress when a new review mentioning the bug is pushed,
|
||||
and when that review is accepted and merged, will transition to Fix
|
||||
Committed.
|
||||
|
||||
Next, something I've always wanted with launchpad was the same kind of
|
||||
integration with blueprints - so we added that too - this time using
|
||||
launchpad API. When a commit message mentions a blueprint, we add a set of
|
||||
text to the whiteboard listing links to the work in progress.
|
||||
|
||||
We configured our gerrit to use launchpad openid for SSO, and we have a
|
||||
script which pulls launchpad user and team information for all teams that
|
||||
are members of ~openstack - so we're able to keep a single point of ACL
|
||||
information.
|
||||
|
||||
Gerrit has a workflow state that launchad doesn't, which is a verified
|
||||
state. So a review can be voted on, it can be approved or rejected, and it
|
||||
can be verified or not. Only a review that is both approved and verified
|
||||
gets merged.
|
||||
|
||||
Possibly most important is the Jenkins trigger plugin, which actually
|
||||
listens actively to the even stream from gerrit, so instead of polling an
|
||||
API to see if there are any interesting things to do - the Jenkins gerrit
|
||||
plugin takes action immediately. It's also intelligent enough to know that
|
||||
if there are three jobs that are all tied to the same gerrit event, that
|
||||
they all need to pass in order for the calling event to be considered
|
||||
successful. This makes for excellent jenkins-level parallelism.
|
||||
|
||||
----------------
|
||||
|
||||
So what does a patch lifecycle look like for OpenStack?
|
||||
|
||||
A developer writes a patch, and then commits it.
|
||||
|
||||
In the commit message, he mentions any bug or blueprint he's working on.
|
||||
|
||||
He submits it for review. With gerrit, this is done via git push - but we
|
||||
have a helper program we wrote which does some fancy things.
|
||||
|
||||
If the commit message mentioned a bug, we update the bug. If it mentioned a
|
||||
blueprint, we update the blueprint.
|
||||
|
||||
Peer review of the patch happens.
|
||||
|
||||
If the patch is approved, jenkins is trigger and runs tests on the patch.
|
||||
|
||||
Jenkins votes on the patch with verified status or with a negative code
|
||||
review comment.
|
||||
|
||||
Gerrit completes the merge. Updates bug and blueprint status.
|
||||
|
||||
SO - all the dev needs to do is write patches, push them to a location and
|
||||
respond to code reviews. All of the rest of the meta-information tracking
|
||||
that's necessary is taken care of.
|
||||
|
||||
-----
|
||||
|
||||
Why are we telling you about this?
|
||||
|
||||
Launchpad is awesome, but there are some integration points that would make
|
||||
our lives either easier, or the overall meta system more complete.
|
||||
|
||||
Also, people are using git a lot these days - and I think we've got a good
|
||||
model for how those people can use launchpad.
|
||||
|
||||
In order for us to have been able to build an equivilantly strong and
|
||||
resilient system using pure bzr/launchpad, we would have needed three
|
||||
things:
|
||||
- subscribably event triggers. We do a lot of things- and we have seen java
|
||||
thread starvation just from having a bazillion things polling
|
||||
launchpad/bzr all the time
|
||||
- verified status in merge requests. I didn't know I needed this until I
|
||||
had it, but it makes perfect sense. The state of the code being verified
|
||||
as working is different information from the patch being accepted as
|
||||
approved. It also allows an automated system to track where the state of
|
||||
a thing is right now
|
||||
- non-python language bindings for lp api (and I don't mean go - I
|
||||
specifically mean java) Jenkins is deeply pluggable, but that's all
|
||||
meaningless if you can't call the API from it. Hell - you should see the
|
||||
github plugin for jenkins - it's pretty baller. (oh, and the launchpad
|
||||
wadl blows up all of the java wadl reading/generation tools)
|
||||
|
||||
In order to build our current system without gerrit:
|
||||
- first class git support in launchpad
|
||||
|
||||
Other nice things:
|
||||
- Foreign merge props. Launchpad was ground-breaking in its thoughts that
|
||||
it would integrate with whatever upstream was doing dev in, and this can
|
||||
be seen in the support for upstream bug trackers. Understanding that
|
||||
upstream might be using a system for code review and being able to follow
|
||||
that might be nice (start with gerrit and github here)
|
||||
- mirrored branches - currently there is a delay in launchpad git branch
|
||||
mirroring, and you can also only mirror master. Additionally, an upstream
|
||||
branch is associated with a person/team, so if jim here decides to create
|
||||
a mirrored branch of openstack/nova from github, I become unable to
|
||||
create one of those in the ~openstack project. oops.
|
||||
- codee branch/review integration with blueprints (without having to click
|
||||
a bunch)
|
||||
- blueprint external metadata links (we're having to hack whiteboard
|
||||
messages right now)
|
||||
|
||||
-->
|
||||
|
||||
|
||||
|
||||
|
||||
<div class="slide">
|
||||
<h1>Where We Came From</h1>
|
||||
|
||||
<img src="images/jenkins-gate.png"/>
|
||||
|
||||
</div>
|
||||
|
||||
<div class="slide">
|
||||
<h1>Types of Tests</h1>
|
||||
|
||||
|
||||
<ul class="incremental">
|
||||
<li> Unit tests </li>
|
||||
<li> Functional tests
|
||||
<ul>
|
||||
<li> Can be run on real or virtual servers </li>
|
||||
<li> Easy for developers to run </li>
|
||||
</ul>
|
||||
</li>
|
||||
<li> Integration tests
|
||||
<ul>
|
||||
<li> May be able to run on virtual servers, should run on real servers </li>
|
||||
<li> Difficult or impossible for a developer to run </li>
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
|
||||
<div class="slide">
|
||||
<h1>OpenStack Projects</h1>
|
||||
|
||||
<center>
|
||||
<img src="images/lp-os-projects.png"/>
|
||||
</center>
|
||||
|
||||
</div>
|
||||
|
||||
<div class="slide">
|
||||
<h1>Test Process</h1>
|
||||
|
||||
<ul>
|
||||
<li> PXE boot </li>
|
||||
<li> Install Ubuntu </li>
|
||||
<li> Install OpenStack </li>
|
||||
<li> Run test suite </li>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
|
||||
<div class="slide">
|
||||
<h1>Test Process Improved</h1>
|
||||
|
||||
<p>One time:</p>
|
||||
<ul>
|
||||
<li> PXE boot </li>
|
||||
<li> Install Ubuntu </li>
|
||||
<li> Install OpenStack dependencies </li>
|
||||
<li> Snapshot LVM volume </li>
|
||||
</ul>
|
||||
<p>Each test:</p>
|
||||
<ul>
|
||||
<li> Kexec boot into LVM snapshot
|
||||
<li> Install OpenStack </li>
|
||||
<li> Run test suite </li>
|
||||
</ul>
|
||||
|
||||
</div>
|
||||
|
||||
<!--
|
||||
I'm going to give you a 30 second history for those of you who don't know me
|
||||
or how the project got to where it is today. In 2008, Chad Miller and I
|
||||
convinced MySQL to move from Bitkeeper to bzr/launchpad. It was a move that
|
||||
was (ironically) just about code hosting, as bugs were done in the MySQL bug
|
||||
tracker and MySQL used a system called WorkLog for what blueprints are used
|
||||
for. Shortly after Brian Aker forked MySQL to start Drizzle, and as we were
|
||||
setting that up, we decided to stay with bzr, which made the decision to
|
||||
locate all of our infrastructure on Launchpad. We became one of the first
|
||||
set of people who weren't Ubuntu itself to actually use blueprint
|
||||
extensively, and used just about every feature of launchpad.
|
||||
|
||||
Last year, when we started OpenStack, the team at Rackspace who were in
|
||||
charge of putting the project together were three of us from Drizzle, plus
|
||||
Rick Clark and Soren Hansen who you all know have strong Ubuntu backgrounds.
|
||||
So we decided to stick with a Launchpad centric model, and again made strong
|
||||
use of all of the Launchpad features - including blueprints, release series,
|
||||
milestones, PPAs and. translations. I imagine someone even uses answers from
|
||||
time to time.
|
||||
|
||||
Along side of all of this, with OpenStack we automated a process that we had
|
||||
been doing by hand in Drizzle, which is a concept we refer to as trunk
|
||||
gating. Essentially, this means that we test all code _before_ merge, with
|
||||
the theory that every commit to trunk should be golden. I could babble about
|
||||
why that's important for QA or release management, but the real gold has to
|
||||
do with streamlining the workflow of an individual developer. If all of the
|
||||
tests always run on trunk, then a developer can be certain that any test
|
||||
failures are caused by their code and are something that they should feel
|
||||
responsibility to fix. In general, this is a Good Thing.
|
||||
|
||||
To gate trunk, we had jenkins run tarmac, which would request approved merge
|
||||
requests, test them, and then complete the merge on successful test run.
|
||||
-->
|
||||
</body>
|
||||
</html>
|
||||
|
|
|
|||
Loading…
Reference in New Issue