Add Queens goal split out tempest plugins
This commit adds a new goal for splitting out bundled tempest plugins into separate python projects. Change-Id: I0f10c5380ef25059ceeac56af4eda734084bf48e
This commit is contained in:
parent
fbed10f195
commit
f80aa4202e
|
@ -0,0 +1,629 @@
|
|||
.. -*- mode: rst -*-
|
||||
|
||||
==================================================
|
||||
Split Tempest Plugins into Seperate Repos/Projects
|
||||
==================================================
|
||||
|
||||
Tempest plugins rely on setuptools entrypoints and therefore can be included
|
||||
in any python project as long as it properly sets up the entrypoint. A common
|
||||
pattern has emerged of bundling these plugins inside a project repo. This is
|
||||
likely because it makes contributing tests at the same time as an API feature
|
||||
must simpler. However, this is really an antipattern and makes consuming the
|
||||
test plugins much more difficult, especially in all-in-one deployments. In
|
||||
order to make using plugins more consistent and easier for end users,
|
||||
packagers, and deployers this goal is to make sure we always use a separate
|
||||
python project for each plugin.
|
||||
|
||||
There are several issues with the current situation including:
|
||||
|
||||
1. When you deploy an all in one system with multiple services that have
|
||||
bundled tempest plugins and then run tempest outside of a venv tempest will see
|
||||
the tests for all the plugins whether you intended to use them or not.
|
||||
It is also a fairly common occurrence for a project to ship a broken plugin
|
||||
that breaks at import time. This prevents unittest discovery from working which
|
||||
will block any tests from running.
|
||||
|
||||
2. A packager tries to use a stable version of your project with a newer
|
||||
tempest. This results in conflicting requirements because tempest is branchless
|
||||
and follows master requirements and makes using stable release plugins with
|
||||
master tempest impossible.
|
||||
|
||||
3. Most projects don't actually use the same tests across release boundaries
|
||||
(or just don't use plugins on stable branches), which means breaking api
|
||||
changes can land. There are 2 exceptions to this trove and ironic which we
|
||||
hacked together a mechanism to install the project repo from master in the
|
||||
tempest venv.
|
||||
|
||||
4. Plugin requirements aren't exposed at install time. Some projects use
|
||||
test-requirements to set tempest plugin requirements (since it is for tests)
|
||||
which do not get installed when you pip install a project. However the
|
||||
entrypoint always gets exposed and tempest will pick it up regardless of
|
||||
whether all the requirements are installed.
|
||||
|
||||
Having tempest plugins in a separate repo/project solves all of these issues
|
||||
and makes using a plugin opt-in in all cases. (ie you have to ``pip install
|
||||
plugin-foo`` to use it and won't show up as an unexpected side effect of
|
||||
installing a service) For community development and testing it means we are
|
||||
actually ensuring API compatibility across releases. It also means for packagers
|
||||
that you don't have to install all of a project repo to ship a plugin package.
|
||||
|
||||
|
||||
Gerrit Topic
|
||||
============
|
||||
|
||||
To facilitate tracking, commits related to this goal should use the
|
||||
gerrit topic::
|
||||
|
||||
goal-split-tempest-plugins
|
||||
|
||||
Completion Criteria
|
||||
===================
|
||||
|
||||
For all projects with a bundled tempest plugin:
|
||||
|
||||
#. Create a new seperate repo for the tempest plugin
|
||||
#. Migrate all the functionality from the bundled plugin to the new repo
|
||||
#. Switch gating jobs to use the new plugin project instead of the bundled one
|
||||
#. Delete the bundled tempest plugin from the project repo
|
||||
|
||||
Note, there is no change in governance or ownership of a repository involved
|
||||
with this change. The new plugin repo/project will still be owned by the
|
||||
project it was spun out from.
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
The tempest documentation elaborates on why seperate plugins are a better
|
||||
pattern:
|
||||
|
||||
http://docs.openstack.org/developer/tempest/plugin.html#standalone-plugin-vs-in-repo-plugin
|
||||
|
||||
The following links may be useful to developers attempting to create their
|
||||
tempest plugin project:
|
||||
|
||||
* https://www.openstack.org/videos/boston-2017/tempest-plugin-crash-course
|
||||
* https://etherpad.openstack.org/p/tempest-separate-plugin
|
||||
* https://etherpad.openstack.org/p/tempest-plugin-testing
|
||||
* https://etherpad.openstack.org/p/pike-tempest-plugin-dev
|
||||
|
||||
Current State / Anticipated Impact
|
||||
==================================
|
||||
|
||||
The following script was used to find all the openstack projects with tempest
|
||||
plugins.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
|
||||
import requests
|
||||
import yaml
|
||||
|
||||
url = 'https://review.openstack.org/projects/'
|
||||
with open(sys.argv[1], 'r') as fd:
|
||||
projects_list = yaml.load(fd.read())
|
||||
|
||||
repo_list = []
|
||||
for i in projects_list:
|
||||
deliverables = projects_list[i]['deliverables']
|
||||
repo_paths = [x['repos'] for x in deliverables.values()]
|
||||
for repos in repo_paths:
|
||||
for name in repos:
|
||||
repo_list.append(name)
|
||||
|
||||
# This is what a project looks like
|
||||
'''
|
||||
"openstack-attic/akanda": {
|
||||
"id": "openstack-attic%2Fakanda",
|
||||
"state": "READ_ONLY"
|
||||
},
|
||||
'''
|
||||
|
||||
def is_openstack_proj(proj):
|
||||
res = False
|
||||
if proj in repo_list:
|
||||
res = True
|
||||
return res
|
||||
|
||||
# Rather than returning a 404 for a nonexistent file, cgit delivers a
|
||||
# 0-byte response to a GET request. It also does not provide a
|
||||
# Content-Length in a HEAD response, so the way we tell if a file exists
|
||||
# is to check the length of the entire GET response body.
|
||||
|
||||
|
||||
def has_tempest_plugin(proj):
|
||||
r = requests.get(
|
||||
"https://git.openstack.org/cgit/%s/plain/setup.cfg" % proj)
|
||||
p = re.compile('^tempest\.test_plugins', re.M)
|
||||
if p.findall(r.text):
|
||||
return True
|
||||
else:
|
||||
False
|
||||
|
||||
r = requests.get(url)
|
||||
# Gerrit prepends 4 garbage octets to the JSON, in order to counter
|
||||
# cross-site scripting attacks. Therefore we must discard it so the
|
||||
# json library won't choke.
|
||||
projects = sorted(filter(is_openstack_proj, json.loads(r.text[4:])))
|
||||
|
||||
found_plugins = filter(has_tempest_plugin, projects)
|
||||
|
||||
# Every element of the found_plugins list begins with "openstack/".
|
||||
# We drop those initial 10 octets when printing the list.
|
||||
for project in found_plugins:
|
||||
print(project)
|
||||
|
||||
On 03 Jan 2017 this was run and found all the projects with bundled tempest
|
||||
plugins. 4 matches were found that are standalone plugins:
|
||||
barbican-tempest-plugin, tempest-horizon, designate-tempest-plugin, and
|
||||
sahara-tests (this is a multitest repo but it's not bundled with the project so
|
||||
it's not applicable here either) Also all the deb packaging repos were removed
|
||||
as they are duplicates. This list is:
|
||||
|
||||
::
|
||||
|
||||
openstack/aodh
|
||||
openstack/ceilometer
|
||||
openstack/cinder
|
||||
openstack/congress
|
||||
openstack/ec2-api
|
||||
openstack/freezer
|
||||
openstack/freezer-api
|
||||
openstack/gnocchi
|
||||
openstack/heat
|
||||
openstack/ironic
|
||||
openstack/ironic-inspector
|
||||
openstack/keystone
|
||||
openstack/magnum
|
||||
openstack/manila
|
||||
openstack/mistral
|
||||
openstack/monasca-api
|
||||
openstack/monasca-log-api
|
||||
openstack/murano
|
||||
openstack/networking-bgpvpn
|
||||
openstack/networking-midonet
|
||||
openstack/networking-sfc
|
||||
openstack/neutron
|
||||
openstack/neutron-dynamic-routing
|
||||
openstack/neutron-fwaas
|
||||
openstack/neutron-lbaas
|
||||
openstack/octavia
|
||||
openstack/senlin
|
||||
openstack/tricircle
|
||||
openstack/trove
|
||||
openstack/vitrage
|
||||
openstack/watcher
|
||||
openstack/zaqar
|
||||
openstack/zun
|
||||
|
||||
Project Teams
|
||||
=============
|
||||
|
||||
barbican
|
||||
--------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
Chef OpenStack
|
||||
--------------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
cinder
|
||||
------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
cloudkitty
|
||||
----------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
Community App Catalog
|
||||
---------------------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
congress
|
||||
--------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
designate
|
||||
---------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
Documentation
|
||||
-------------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
dragonflow
|
||||
----------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
ec2-api
|
||||
-------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
freezer
|
||||
-------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
fuel
|
||||
----
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
glance
|
||||
------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
heat
|
||||
----
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
horizon
|
||||
-------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
I18n
|
||||
----
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
Infrastructure
|
||||
--------------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
ironic
|
||||
------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
karbor
|
||||
------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
keystone
|
||||
--------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
kolla
|
||||
-----
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
kuryr
|
||||
-----
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
magnum
|
||||
------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
manila
|
||||
------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
mistral
|
||||
-------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
monasca
|
||||
-------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
murano
|
||||
------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
neutron
|
||||
-------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
nova
|
||||
----
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
octavia
|
||||
-------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
OpenStack Charms
|
||||
----------------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
OpenStack UX
|
||||
------------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
OpenStackAnsible
|
||||
----------------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
OpenStackClient
|
||||
---------------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
oslo
|
||||
----
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
Packaging-deb
|
||||
-------------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
Packaging-rpm
|
||||
-------------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
Puppet OpenStack
|
||||
----------------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
Quality Assurance
|
||||
-----------------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
rally
|
||||
-----
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
RefStack
|
||||
--------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
Release Management
|
||||
------------------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
requirements
|
||||
------------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
sahara
|
||||
------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
searchlight
|
||||
-----------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
Security
|
||||
--------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
senlin
|
||||
------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
solum
|
||||
-----
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
Stable branch maintenance
|
||||
-------------------------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
storlets
|
||||
--------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
swift
|
||||
-----
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
tacker
|
||||
------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
Telemetry
|
||||
---------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
tricircle
|
||||
---------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
tripleo
|
||||
-------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
trove
|
||||
-----
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
vitrage
|
||||
-------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
watcher
|
||||
-------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
winstackers
|
||||
-----------
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
zaqar
|
||||
-----
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
||||
|
||||
zun
|
||||
---
|
||||
|
||||
Planning Artifacts:
|
||||
|
||||
Completion Artifacts:
|
Loading…
Reference in New Issue