Merge "Add Queens goal split out tempest plugins"

This commit is contained in:
Jenkins 2017-05-26 08:21:09 +00:00 committed by Gerrit Code Review
commit fcc6b0dc90
1 changed files with 629 additions and 0 deletions

View File

@ -0,0 +1,629 @@
.. -*- mode: rst -*-
==================================================
Split Tempest Plugins into Seperate Repos/Projects
==================================================
Tempest plugins rely on setuptools entrypoints and therefore can be included
in any python project as long as it properly sets up the entrypoint. A common
pattern has emerged of bundling these plugins inside a project repo. This is
likely because it makes contributing tests at the same time as an API feature
must simpler. However, this is really an antipattern and makes consuming the
test plugins much more difficult, especially in all-in-one deployments. In
order to make using plugins more consistent and easier for end users,
packagers, and deployers this goal is to make sure we always use a separate
python project for each plugin.
There are several issues with the current situation including:
1. When you deploy an all in one system with multiple services that have
bundled tempest plugins and then run tempest outside of a venv tempest will see
the tests for all the plugins whether you intended to use them or not.
It is also a fairly common occurrence for a project to ship a broken plugin
that breaks at import time. This prevents unittest discovery from working which
will block any tests from running.
2. A packager tries to use a stable version of your project with a newer
tempest. This results in conflicting requirements because tempest is branchless
and follows master requirements and makes using stable release plugins with
master tempest impossible.
3. Most projects don't actually use the same tests across release boundaries
(or just don't use plugins on stable branches), which means breaking api
changes can land. There are 2 exceptions to this trove and ironic which we
hacked together a mechanism to install the project repo from master in the
tempest venv.
4. Plugin requirements aren't exposed at install time. Some projects use
test-requirements to set tempest plugin requirements (since it is for tests)
which do not get installed when you pip install a project. However the
entrypoint always gets exposed and tempest will pick it up regardless of
whether all the requirements are installed.
Having tempest plugins in a separate repo/project solves all of these issues
and makes using a plugin opt-in in all cases. (ie you have to ``pip install
plugin-foo`` to use it and won't show up as an unexpected side effect of
installing a service) For community development and testing it means we are
actually ensuring API compatibility across releases. It also means for packagers
that you don't have to install all of a project repo to ship a plugin package.
Gerrit Topic
============
To facilitate tracking, commits related to this goal should use the
gerrit topic::
goal-split-tempest-plugins
Completion Criteria
===================
For all projects with a bundled tempest plugin:
#. Create a new seperate repo for the tempest plugin
#. Migrate all the functionality from the bundled plugin to the new repo
#. Switch gating jobs to use the new plugin project instead of the bundled one
#. Delete the bundled tempest plugin from the project repo
Note, there is no change in governance or ownership of a repository involved
with this change. The new plugin repo/project will still be owned by the
project it was spun out from.
References
==========
The tempest documentation elaborates on why seperate plugins are a better
pattern:
http://docs.openstack.org/developer/tempest/plugin.html#standalone-plugin-vs-in-repo-plugin
The following links may be useful to developers attempting to create their
tempest plugin project:
* https://www.openstack.org/videos/boston-2017/tempest-plugin-crash-course
* https://etherpad.openstack.org/p/tempest-separate-plugin
* https://etherpad.openstack.org/p/tempest-plugin-testing
* https://etherpad.openstack.org/p/pike-tempest-plugin-dev
Current State / Anticipated Impact
==================================
The following script was used to find all the openstack projects with tempest
plugins.
.. code-block:: python
import json
import re
import sys
import requests
import yaml
url = 'https://review.openstack.org/projects/'
with open(sys.argv[1], 'r') as fd:
projects_list = yaml.load(fd.read())
repo_list = []
for i in projects_list:
deliverables = projects_list[i]['deliverables']
repo_paths = [x['repos'] for x in deliverables.values()]
for repos in repo_paths:
for name in repos:
repo_list.append(name)
# This is what a project looks like
'''
"openstack-attic/akanda": {
"id": "openstack-attic%2Fakanda",
"state": "READ_ONLY"
},
'''
def is_openstack_proj(proj):
res = False
if proj in repo_list:
res = True
return res
# Rather than returning a 404 for a nonexistent file, cgit delivers a
# 0-byte response to a GET request. It also does not provide a
# Content-Length in a HEAD response, so the way we tell if a file exists
# is to check the length of the entire GET response body.
def has_tempest_plugin(proj):
r = requests.get(
"https://git.openstack.org/cgit/%s/plain/setup.cfg" % proj)
p = re.compile('^tempest\.test_plugins', re.M)
if p.findall(r.text):
return True
else:
False
r = requests.get(url)
# Gerrit prepends 4 garbage octets to the JSON, in order to counter
# cross-site scripting attacks. Therefore we must discard it so the
# json library won't choke.
projects = sorted(filter(is_openstack_proj, json.loads(r.text[4:])))
found_plugins = filter(has_tempest_plugin, projects)
# Every element of the found_plugins list begins with "openstack/".
# We drop those initial 10 octets when printing the list.
for project in found_plugins:
print(project)
On 03 Jan 2017 this was run and found all the projects with bundled tempest
plugins. 4 matches were found that are standalone plugins:
barbican-tempest-plugin, tempest-horizon, designate-tempest-plugin, and
sahara-tests (this is a multitest repo but it's not bundled with the project so
it's not applicable here either) Also all the deb packaging repos were removed
as they are duplicates. This list is:
::
openstack/aodh
openstack/ceilometer
openstack/cinder
openstack/congress
openstack/ec2-api
openstack/freezer
openstack/freezer-api
openstack/gnocchi
openstack/heat
openstack/ironic
openstack/ironic-inspector
openstack/keystone
openstack/magnum
openstack/manila
openstack/mistral
openstack/monasca-api
openstack/monasca-log-api
openstack/murano
openstack/networking-bgpvpn
openstack/networking-midonet
openstack/networking-sfc
openstack/neutron
openstack/neutron-dynamic-routing
openstack/neutron-fwaas
openstack/neutron-lbaas
openstack/octavia
openstack/senlin
openstack/tricircle
openstack/trove
openstack/vitrage
openstack/watcher
openstack/zaqar
openstack/zun
Project Teams
=============
barbican
--------
Planning Artifacts:
Completion Artifacts:
Chef OpenStack
--------------
Planning Artifacts:
Completion Artifacts:
cinder
------
Planning Artifacts:
Completion Artifacts:
cloudkitty
----------
Planning Artifacts:
Completion Artifacts:
Community App Catalog
---------------------
Planning Artifacts:
Completion Artifacts:
congress
--------
Planning Artifacts:
Completion Artifacts:
designate
---------
Planning Artifacts:
Completion Artifacts:
Documentation
-------------
Planning Artifacts:
Completion Artifacts:
dragonflow
----------
Planning Artifacts:
Completion Artifacts:
ec2-api
-------
Planning Artifacts:
Completion Artifacts:
freezer
-------
Planning Artifacts:
Completion Artifacts:
fuel
----
Planning Artifacts:
Completion Artifacts:
glance
------
Planning Artifacts:
Completion Artifacts:
heat
----
Planning Artifacts:
Completion Artifacts:
horizon
-------
Planning Artifacts:
Completion Artifacts:
I18n
----
Planning Artifacts:
Completion Artifacts:
Infrastructure
--------------
Planning Artifacts:
Completion Artifacts:
ironic
------
Planning Artifacts:
Completion Artifacts:
karbor
------
Planning Artifacts:
Completion Artifacts:
keystone
--------
Planning Artifacts:
Completion Artifacts:
kolla
-----
Planning Artifacts:
Completion Artifacts:
kuryr
-----
Planning Artifacts:
Completion Artifacts:
magnum
------
Planning Artifacts:
Completion Artifacts:
manila
------
Planning Artifacts:
Completion Artifacts:
mistral
-------
Planning Artifacts:
Completion Artifacts:
monasca
-------
Planning Artifacts:
Completion Artifacts:
murano
------
Planning Artifacts:
Completion Artifacts:
neutron
-------
Planning Artifacts:
Completion Artifacts:
nova
----
Planning Artifacts:
Completion Artifacts:
octavia
-------
Planning Artifacts:
Completion Artifacts:
OpenStack Charms
----------------
Planning Artifacts:
Completion Artifacts:
OpenStack UX
------------
Planning Artifacts:
Completion Artifacts:
OpenStackAnsible
----------------
Planning Artifacts:
Completion Artifacts:
OpenStackClient
---------------
Planning Artifacts:
Completion Artifacts:
oslo
----
Planning Artifacts:
Completion Artifacts:
Packaging-deb
-------------
Planning Artifacts:
Completion Artifacts:
Packaging-rpm
-------------
Planning Artifacts:
Completion Artifacts:
Puppet OpenStack
----------------
Planning Artifacts:
Completion Artifacts:
Quality Assurance
-----------------
Planning Artifacts:
Completion Artifacts:
rally
-----
Planning Artifacts:
Completion Artifacts:
RefStack
--------
Planning Artifacts:
Completion Artifacts:
Release Management
------------------
Planning Artifacts:
Completion Artifacts:
requirements
------------
Planning Artifacts:
Completion Artifacts:
sahara
------
Planning Artifacts:
Completion Artifacts:
searchlight
-----------
Planning Artifacts:
Completion Artifacts:
Security
--------
Planning Artifacts:
Completion Artifacts:
senlin
------
Planning Artifacts:
Completion Artifacts:
solum
-----
Planning Artifacts:
Completion Artifacts:
Stable branch maintenance
-------------------------
Planning Artifacts:
Completion Artifacts:
storlets
--------
Planning Artifacts:
Completion Artifacts:
swift
-----
Planning Artifacts:
Completion Artifacts:
tacker
------
Planning Artifacts:
Completion Artifacts:
Telemetry
---------
Planning Artifacts:
Completion Artifacts:
tricircle
---------
Planning Artifacts:
Completion Artifacts:
tripleo
-------
Planning Artifacts:
Completion Artifacts:
trove
-----
Planning Artifacts:
Completion Artifacts:
vitrage
-------
Planning Artifacts:
Completion Artifacts:
watcher
-------
Planning Artifacts:
Completion Artifacts:
winstackers
-----------
Planning Artifacts:
Completion Artifacts:
zaqar
-----
Planning Artifacts:
Completion Artifacts:
zun
---
Planning Artifacts:
Completion Artifacts: