zuul
/
zuul
Code Issues Proposed changes

Remove internal Swift support 

The zuul v3 architecture does not require that this be supported
as an internal special case.  Remove it for now in the expectation
that we will implement similar functionality using the auth system
and Ansible roles.

Change-Id: I9c1e05a716ce8cfb5f66c4688db110c320d33b4b
This commit is contained in:
James E. Blair 2017-03-07 13:08:47 -08:00
parent 9e784526e4
commit 92e953a807
13 changed files with 3 additions and 525 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

30
doc/source/launchers.rst
View File

@ -11,8 +11,6 @@
.. _`Turbo-Hipster Documentation`:
http://turbo-hipster.rtfd.org/
.. _FormPost: http://docs.openstack.org/developer/swift/misc.html#module-swift.common.middleware.formpost
.. _launchers:
Launchers
@ -138,34 +136,6 @@ And finally a reference being altered::
Your jobs can check whether the parameters are ``000000`` to act
differently on each kind of event.
Swift parameters
~~~~~~~~~~~~~~~~
If swift information has been configured for the job zuul will also
provide signed credentials for the builder to upload results and
assets into containers using the `FormPost`_ middleware.
Each zuul container/instruction set will contain each of the following
parameters where $NAME is the ``name`` defined in the layout.
*SWIFT_$NAME_URL*
The swift destination URL. This will be the entire URL including
the AUTH, container and path prefix (folder).
*SWIFT_$NAME_HMAC_BODY*
The information signed in the HMAC body. The body is as follows::
PATH TO OBJECT PREFIX (excluding domain)
BLANK LINE (zuul implements no form redirect)
MAX FILE SIZE
MAX FILE COUNT
SIGNATURE EXPIRY
*SWIFT_$NAME_SIGNATURE*
The HMAC body signed with the configured key.
*SWIFT_$NAME_LOGSERVER_PREFIX*
The URL to prepend to the object path when returning the results
from a build.
Gearman
-------

112
doc/source/zuul.rst
View File

@ -172,76 +172,6 @@ can be found on the :doc:`merger` page.
Path to PID lock file for the merger process.
``pidfile=/var/run/zuul-merger/merger.pid``
.. _swift:
swift
"""""
To send (optional) swift upload instructions this section must be
present. Multiple destinations can be defined in the :ref:`jobs` section
of the layout.
If you are sending the temp-url-key or fetching the x-storage-url, you
will need the python-swiftclient module installed.
**X-Account-Meta-Temp-Url-Key** (optional)
This is the key used to sign the HMAC message. If you do not set a
key Zuul will generate one automatically.
**Send-Temp-Url-Key** (optional)
Zuul can send the X-Account-Meta-Temp-Url-Key to swift for you if
you have set up the appropriate credentials in ``authurl`` below.
This isn't necessary if you know and have set your
X-Account-Meta-Temp-Url-Key.
If set, Zuul requires the python-swiftclient module.
``default: true``
**X-Storage-Url** (optional)
The storage URL is the destination to upload files into. If you do
not set this the ``authurl`` credentials are used to fetch the url
from swift and Zuul will requires the python-swiftclient module.
**authurl** (optional)
The (keystone) Auth URL for swift.
``For example, https://identity.api.rackspacecloud.com/v2.0/``
This is required if you have Send-Temp-Url-Key set to ``True`` or
if you have not supplied the X-Storage-Url.
Any of the `swiftclient connection parameters`_ can also be defined
here by the same name. Including the os_options by their key name (
``for example tenant_id``)
.. _swiftclient connection parameters: http://docs.openstack.org/developer/python-swiftclient/swiftclient.html#module-swiftclient.client
**region_name** (optional)
The region name holding the swift container
``For example, SYD``
Each destination defined by the :ref:`jobs` will have the following
default values that it may overwrite.
**default_container** (optional)
Container name to place the log into
``For example, logs``
**default_expiry** (optional)
How long the signed destination should be available for
``default: 7200 (2hrs)``
**default_max_file_size** (optional)
The maximum size of an individual file
``default: 104857600 (100MB)``
**default_max_file_count** (optional)
The maximum number of separate files to allow
``default: 10``
**default_logserver_prefix**
Provide a URL to the CDN or logserver app so that a worker knows
what URL to return. The worker should return the logserver_prefix
url and the object path.
``For example: http://logs.example.org/server.app?obj=``
.. _connection:
connection ArbitraryName
@ -786,48 +716,6 @@ each job as it builds a list from the project specification.
**tags (optional)**
A list of arbitrary strings which will be associated with the job.
**swift**
If :ref:`swift` is configured then each job can define a destination
container for the builder to place logs and/or assets into. Multiple
containers can be listed for each job by providing a unique ``name``.
*name*
Set an identifying name for the container. This is used in the
parameter key sent to the builder. For example if it ``logs`` then
one of the parameters sent will be ``SWIFT_logs_CONTAINER``
(case-sensitive).
Each of the defaults defined in :ref:`swift` can be overwritten as:
*container* (optional)
Container name to place the log into
``For example, logs``
*expiry* (optional)
How long the signed destination should be available for
*max-file-size** (optional)
The maximum size of an individual file
*max_file_size** (optional, deprecated)
A deprecated alternate spelling of *max-file-size*.
*max-file-count* (optional)
The maximum number of separate files to allow
*max_file_count* (optional, deprecated)
A deprecated alternate spelling of *max-file-count*.
*logserver-prefix*
Provide a URL to the CDN or logserver app so that a worker knows
what URL to return.
``For example: http://logs.example.org/server.app?obj=``
The worker should return the logserver-prefix url and the object
path as the URL in the results data packet.
*logserver_prefix* (deprecated)
A deprecated alternate spelling of *logserver-prefix*.
Here is an example of setting the failure message for jobs that check
whether a change merges cleanly::

1
test-requirements.txt
View File

@ -6,7 +6,6 @@ sphinxcontrib-blockdiag>=1.1.0
fixtures>=0.3.14
python-keystoneclient>=0.4.2
python-subunit
python-swiftclient>=1.6
testrepository>=0.0.17
testtools>=0.9.32
sphinxcontrib-programoutput

21
tests/base.py
View File

@ -32,7 +32,6 @@ from six import StringIO
import socket
import string
import subprocess
import swiftclient
import sys
import tempfile
import threading
@ -60,7 +59,6 @@ import zuul.webapp
import zuul.rpclistener
import zuul.launcher.server
import zuul.launcher.client
import zuul.lib.swift
import zuul.lib.connections
import zuul.merger.client
import zuul.merger.merger
@ -904,18 +902,6 @@ class FakeSMTP(object):
return True
class FakeSwiftClientConnection(swiftclient.client.Connection):
def post_account(self, headers):
# Do nothing
pass
def get_auth(self):
# Returns endpoint and (unused) auth token
endpoint = os.path.join('https://storage.example.org', 'V1',
'AUTH_account')
return endpoint, ''
class FakeNodepool(object):
REQUEST_ROOT = '/nodepool/requests'
NODE_ROOT = '/nodepool/nodes'
@ -1332,11 +1318,6 @@ class ZuulTestCase(BaseTestCase):
self.sched = zuul.scheduler.Scheduler(self.config)
self.useFixture(fixtures.MonkeyPatch('swiftclient.client.Connection',
FakeSwiftClientConnection))
self.swift = zuul.lib.swift.Swift(self.config)
self.event_queues = [
self.sched.result_event_queue,
self.sched.trigger_event_queue,
@ -1367,7 +1348,7 @@ class ZuulTestCase(BaseTestCase):
self.builds = self.launch_server.running_builds
self.launch_client = zuul.launcher.client.LaunchClient(
self.config, self.sched, self.swift)
self.config, self.sched)
self.merge_client = zuul.merger.client.MergeClient(
self.config, self.sched)
self.nodepool = zuul.nodepool.Nodepool(self.sched)

59
tests/fixtures/layout-swift.yaml vendored
View File

@ -1,59 +0,0 @@
pipelines:
- name: check
manager: IndependentPipelineManager
trigger:
gerrit:
- event: patchset-created
success:
gerrit:
verified: 1
failure:
gerrit:
verified: -1
- name: post
manager: IndependentPipelineManager
trigger:
gerrit:
- event: ref-updated
ref: ^(?!refs/).*$
- name: gate
manager: DependentPipelineManager
failure-message: Build failed. For information on how to proceed, see http://wiki.example.org/Test_Failures
trigger:
gerrit:
- event: comment-added
approval:
- approved: 1
success:
gerrit:
verified: 2
submit: true
failure:
gerrit:
verified: -2
start:
gerrit:
verified: 0
precedence: high
jobs:
- name: ^.*$
swift:
- name: logs
- name: ^.*-merge$
swift:
- name: logs
container: merge_logs
failure-message: Unable to merge change
- name: test-test
swift:
- name: MOSTLY
container: stash
projects:
- name: org/project
gate:
- test-merge
- test-test

10
tests/fixtures/zuul-connections-bad-sql.conf vendored
View File

@ -12,16 +12,6 @@ git_user_email=zuul@example.com
git_user_name=zuul
zuul_url=http://zuul.example.com/p
[swift]
authurl=https://identity.api.example.org/v2.0/
user=username
key=password
tenant_name=" "
default_container=logs
region_name=EXP
logserver_prefix=http://logs.example.org/server.app/
[connection review_gerrit]
driver=gerrit
server=review.example.com

10
tests/fixtures/zuul-connections-multiple-gerrits.conf vendored
View File

@ -15,16 +15,6 @@ zuul_url=http://zuul.example.com/p
[launcher]
git_dir=/tmp/zuul-test/launcher-git
[swift]
authurl=https://identity.api.example.org/v2.0/
user=username
key=password
tenant_name=" "
default_container=logs
region_name=EXP
logserver_prefix=http://logs.example.org/server.app/
[connection review_gerrit]
driver=gerrit
server=review.example.com

10
tests/fixtures/zuul-connections-same-gerrit.conf vendored
View File

@ -15,16 +15,6 @@ zuul_url=http://zuul.example.com/p
[launcher]
git_dir=/tmp/zuul-test/launcher-git
[swift]
authurl=https://identity.api.example.org/v2.0/
user=username
key=password
tenant_name=" "
default_container=logs
region_name=EXP
logserver_prefix=http://logs.example.org/server.app/
[connection review_gerrit]
driver=gerrit
server=review.example.com

46
tests/unit/test_scheduler.py
View File

@ -3583,52 +3583,6 @@ For CI problems and help debugging, contact ci@example.org"""
self.assertNotIn('logs.example.com', B.messages[1])
self.assertNotIn('SKIPPED', B.messages[1])
@skip("Disabled for early v3 development")
def test_swift_instructions(self):
"Test that the correct swift instructions are sent to the workers"
self.updateConfigLayout(
'tests/fixtures/layout-swift.yaml')
self.sched.reconfigure(self.config)
self.registerJobs()
self.launch_server.hold_jobs_in_build = True
A = self.fake_gerrit.addFakeChange('org/project', 'master', 'A')
A.addApproval('code-review', 2)
self.fake_gerrit.addEvent(A.addApproval('approved', 1))
self.waitUntilSettled()
self.assertEqual(
"https://storage.example.org/V1/AUTH_account/merge_logs/1/1/1/"
"gate/test-merge/",
self.builds[0].parameters['SWIFT_logs_URL'][:-7])
self.assertEqual(5,
len(self.builds[0].parameters['SWIFT_logs_HMAC_BODY'].
split('\n')))
self.assertIn('SWIFT_logs_SIGNATURE', self.builds[0].parameters)
self.assertEqual(
"https://storage.example.org/V1/AUTH_account/logs/1/1/1/"
"gate/test-test/",
self.builds[1].parameters['SWIFT_logs_URL'][:-7])
self.assertEqual(5,
len(self.builds[1].parameters['SWIFT_logs_HMAC_BODY'].
split('\n')))
self.assertIn('SWIFT_logs_SIGNATURE', self.builds[1].parameters)
self.assertEqual(
"https://storage.example.org/V1/AUTH_account/stash/1/1/1/"
"gate/test-test/",
self.builds[1].parameters['SWIFT_MOSTLY_URL'][:-7])
self.assertEqual(5,
len(self.builds[1].
parameters['SWIFT_MOSTLY_HMAC_BODY'].split('\n')))
self.assertIn('SWIFT_MOSTLY_SIGNATURE', self.builds[1].parameters)
self.launch_server.hold_jobs_in_build = False
self.launch_server.release()
self.waitUntilSettled()
def test_client_get_running_jobs(self):
"Test that the RPC client can get a list of running jobs"
self.launch_server.hold_jobs_in_build = True

6
zuul/cmd/scheduler.py
View File

@ -127,7 +127,6 @@ class Scheduler(zuul.cmd.ZuulApp):
import zuul.launcher.client
import zuul.merger.client
import zuul.nodepool
import zuul.lib.swift
import zuul.webapp
import zuul.rpclistener
import zuul.zk
@ -141,11 +140,8 @@ class Scheduler(zuul.cmd.ZuulApp):
self.log = logging.getLogger("zuul.Scheduler")
self.sched = zuul.scheduler.Scheduler(self.config)
# TODO(jhesketh): Move swift into a connection?
self.swift = zuul.lib.swift.Swift(self.config)
gearman = zuul.launcher.client.LaunchClient(self.config, self.sched,
self.swift)
gearman = zuul.launcher.client.LaunchClient(self.config, self.sched)
merger = zuul.merger.client.MergeClient(self.config, self.sched)
nodepool = zuul.nodepool.Nodepool(self.sched)

12
zuul/configloader.py
View File

@ -122,20 +122,8 @@ class NodeSetParser(object):
class JobParser(object):
@staticmethod
def getSchema():
swift_tmpurl = {vs.Required('name'): str,
'container': str,
'expiry': int,
'max_file_size': int,
'max-file-size': int,
'max_file_count': int,
'max-file-count': int,
'logserver_prefix': str,
'logserver-prefix': str,
}
auth = {'secrets': to_list(str),
'inherit': bool,
'swift-tmpurl': to_list(swift_tmpurl),
}
node = {vs.Required('name'): str,

43
zuul/launcher/client.py
View File

@ -17,7 +17,6 @@ import gear
import json
import logging
import os
import six
import time
import threading
from uuid import uuid4
@ -149,10 +148,9 @@ class LaunchClient(object):
log = logging.getLogger("zuul.LaunchClient")
negative_function_cache_ttl = 5
def __init__(self, config, sched, swift):
def __init__(self, config, sched):
self.config = config
self.sched = sched
self.swift = swift
self.builds = {}
self.meta_jobs = {} # A list of meta-jobs like stop or describe
@ -211,42 +209,6 @@ class LaunchClient(object):
self.log.debug("Function %s is not registered" % name)
return False
def updateBuildParams(self, job, item, params):
"""Allow the job to modify and add build parameters"""
# NOTE(jhesketh): The params need to stay in a key=value data pair
# as workers cannot necessarily handle lists.
if 'swift' in job.auth and self.swift.connection:
for name, s in job.swift.items():
swift_instructions = {}
s_config = {}
s_config.update((k, v.format(item=item, job=job,
change=item.change))
if isinstance(v, six.string_types)
else (k, v)
for k, v in s.items())
(swift_instructions['URL'],
swift_instructions['HMAC_BODY'],
swift_instructions['SIGNATURE']) = \
self.swift.generate_form_post_middleware_params(
params['LOG_PATH'], **s_config)
if 'logserver_prefix' in s_config:
swift_instructions['LOGSERVER_PREFIX'] = \
s_config['logserver_prefix']
elif self.config.has_option('swift',
'default_logserver_prefix'):
swift_instructions['LOGSERVER_PREFIX'] = \
self.config.get('swift', 'default_logserver_prefix')
# Create a set of zuul instructions for each instruction-set
# given in the form of NAME_PARAMETER=VALUE
for key, value in swift_instructions.items():
params['_'.join(['SWIFT', name, key])] = value
def launch(self, job, item, pipeline, dependent_items=[]):
uuid = str(uuid4().hex)
self.log.info(
@ -309,9 +271,6 @@ class LaunchClient(object):
params['BASE_LOG_PATH'] = item.change.getBasePath()
params['LOG_PATH'] = destination_path
# Allow the job to update the params
self.updateBuildParams(job, item, params)
# This is what we should be heading toward for parameters:
# required:

168
zuul/lib/swift.py
View File

@ -1,168 +0,0 @@
# Copyright 2014 Rackspace Australia
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import hmac
from hashlib import sha1
import logging
from time import time
import os
import random
import six
from six.moves import urllib
import string
class Swift(object):
log = logging.getLogger("zuul.Swift")
def __init__(self, config):
self.config = config
self.connection = False
if self.config.has_option('swift', 'X-Account-Meta-Temp-Url-Key'):
self.secure_key = self.config.get('swift',
'X-Account-Meta-Temp-Url-Key')
else:
self.secure_key = ''.join(
random.choice(string.ascii_uppercase + string.digits)
for x in range(20)
)
self.storage_url = ''
if self.config.has_option('swift', 'X-Storage-Url'):
self.storage_url = self.config.get('swift', 'X-Storage-Url')
try:
if self.config.has_section('swift'):
if (not self.config.has_option('swift', 'Send-Temp-Url-Key')
or self.config.getboolean('swift',
'Send-Temp-Url-Key')):
self.connect()
# Tell swift of our key
headers = {}
headers['X-Account-Meta-Temp-Url-Key'] = self.secure_key
self.connection.post_account(headers)
if not self.config.has_option('swift', 'X-Storage-Url'):
self.connect()
self.storage_url = self.connection.get_auth()[0]
except Exception as e:
self.log.warning("Unable to set up swift. Signed storage URL is "
"likely to be wrong. %s" % e)
def connect(self):
if not self.connection:
authurl = self.config.get('swift', 'authurl')
user = (self.config.get('swift', 'user')
if self.config.has_option('swift', 'user') else None)
key = (self.config.get('swift', 'key')
if self.config.has_option('swift', 'key') else None)
retries = (self.config.get('swift', 'retries')
if self.config.has_option('swift', 'retries') else 5)
preauthurl = (self.config.get('swift', 'preauthurl')
if self.config.has_option('swift', 'preauthurl')
else None)
preauthtoken = (self.config.get('swift', 'preauthtoken')
if self.config.has_option('swift', 'preauthtoken')
else None)
snet = (self.config.get('swift', 'snet')
if self.config.has_option('swift', 'snet') else False)
starting_backoff = (self.config.get('swift', 'starting_backoff')
if self.config.has_option('swift',
'starting_backoff')
else 1)
max_backoff = (self.config.get('swift', 'max_backoff')
if self.config.has_option('swift', 'max_backoff')
else 64)
tenant_name = (self.config.get('swift', 'tenant_name')
if self.config.has_option('swift', 'tenant_name')
else None)
auth_version = (self.config.get('swift', 'auth_version')
if self.config.has_option('swift', 'auth_version')
else 2.0)
cacert = (self.config.get('swift', 'cacert')
if self.config.has_option('swift', 'cacert') else None)
insecure = (self.config.get('swift', 'insecure')
if self.config.has_option('swift', 'insecure')
else False)
ssl_compression = (self.config.get('swift', 'ssl_compression')
if self.config.has_option('swift',
'ssl_compression')
else True)
available_os_options = ['tenant_id', 'auth_token', 'service_type',
'endpoint_type', 'tenant_name',
'object_storage_url', 'region_name']
os_options = {}
for os_option in available_os_options:
if self.config.has_option('swift', os_option):
os_options[os_option] = self.config.get('swift', os_option)
import swiftclient
self.connection = swiftclient.client.Connection(
authurl=authurl, user=user, key=key, retries=retries,
preauthurl=preauthurl, preauthtoken=preauthtoken, snet=snet,
starting_backoff=starting_backoff, max_backoff=max_backoff,
tenant_name=tenant_name, os_options=os_options,
auth_version=auth_version, cacert=cacert, insecure=insecure,
ssl_compression=ssl_compression)
def generate_form_post_middleware_params(self, destination_prefix='',
**kwargs):
"""Generate the FormPost middleware params for the given settings"""
# Define the available settings and their defaults
settings = {
'container': '',
'expiry': 7200,
'max_file_size': 104857600,
'max_file_count': 10,
'file_path_prefix': ''
}
for key, default in six.iteritems(settings):
# TODO(jeblair): Remove the following two lines after a
# deprecation period for the underscore variants of the
# settings in YAML.
if key in kwargs:
settings[key] = kwargs[key]
# Since we prefer '-' rather than '_' in YAML, look up
# keys there using hyphens. Continue to use underscores
# everywhere else.
altkey = key.replace('_', '-')
if altkey in kwargs:
settings[key] = kwargs[altkey]
elif self.config.has_option('swift', 'default_' + key):
settings[key] = self.config.get('swift', 'default_' + key)
# TODO: these are always strings; some should be converted
# to ints.
expires = int(time() + int(settings['expiry']))
redirect = ''
url = os.path.join(self.storage_url, settings['container'],
settings['file_path_prefix'],
destination_prefix)
u = urllib.parse.urlparse(url)
hmac_body = '%s\n%s\n%s\n%s\n%s' % (u.path, redirect,
settings['max_file_size'],
settings['max_file_count'],
expires)
signature = hmac.new(self.secure_key, hmac_body, sha1).hexdigest()
return url, hmac_body, signature