openstack
/
sahara-tests
Code Issues Proposed changes

Feature sets for sahara-scenario 

Implement feature sets for scenario tests. A feature
- automatically loads additional default files if available
  when -p and -v are used;
- selects also specific EDP jobs which matches the specified
  tags, in addition to EDP jobs without tags.

For more details, check the referenced story and also the
original spec:
https://specs.openstack.org/openstack/sahara-specs/specs/sahara-tests/sahara-scenario-feature-sets.html

Story: 2004691
Task: 28702
Change-Id: Ic3e2e5d3f37d5bb7002605dcd2de3fdd892dc8bc
This commit is contained in:
Luigi Toscano 2019-01-18 17:08:49 +01:00
parent 23af1543a2
commit 1518565156
8 changed files with 380 additions and 13 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

37
doc/source/scenario.rst
View File

@ -93,6 +93,31 @@ version is not necessary):
$ tox -e venv -- sahara-scenario -p vanilla -v 2.7.1
..
Different OpenStack releases may require different configuration for the
same set of plugin and versions. If you use the plugin and version flag,
if you want to use the configuration file for a specific OpenStack release
supported by sahara-scenario, you can specify also the ``-r RELEASE``
argument, where ``RELEASE`` is the official name of the OpenStack release.
By default only default configuration files for the specified plugin and
version (and release, if any) are included. Also, if any job configuration
is included, only jobs not tagged with any features will be executed.
In order to enable feature-specific configuration settings, pass
the list of requested features through the ``--feature`` (``-f``) parameter.
The parameter makes sure that:
* additional base configuration file which are feature-specific are included;
* in addition to non-tagged jobs, jobs which are tagged with the specified
features are included too.
Example:
.. sourcecode:: console
$ tox -e venv -- sahara-scenario -p vanilla -v 2.7.1 -f s3 -f myfeature -r rocky
..
Create the YAML and/or the YAML mako template files for scenario tests
``etc/scenario/simple-testcase.yaml``.
You can take a look at sample YAML files `How to write scenario files`_.
@ -164,7 +189,11 @@ Optional arguments
| --release, | specify Sahara release |
| -r RELEASE | |
+-------------------+----------------------------+
| --report | write results to file |
| --report | write results to file |
+-------------------+----------------------------+
| --feature, | list of features |
| -f FEAT1 | that should be enabled |
| [-f FEAT2 ...] | |
+-------------------+----------------------------+
| --count COUNT | specify count of runs |
+-------------------+----------------------------+
@ -449,10 +478,12 @@ This sections is an array-type.
- ['run_jobs', 'scale', 'run_jobs']
- array of checks
* - edp_jobs_flow
- string
- string, list
-
-
- name of edp job flow
- name of jobs defined under edp_jobs_flow be executed on the cluster;
if list, each item may be a dict with fields
``name`` (string) and ``features`` (list), or a string
* - hdfs_username
- string
-

14
etc/scenario/simple-testcase.yaml
View File

@ -71,3 +71,17 @@ edp_jobs_flow:
configs:
edp.streaming.mapper: /bin/cat
edp.streaming.reducer: /usr/bin/wc
- type: Pig
features:
- manila
input_datasource:
type: manila
source: etc/edp-examples/edp-pig/top-todoers/data/input
output_datasource:
type: manila
destination: /user/hadoop/edp-output
main_lib:
type: manila
source: etc/edp-examples/edp-pig/top-todoers/example.pig
configs:
dfs.replication: 1

11
releasenotes/notes/scenario-feature-sets-ac3e0e9e40b236bb.yaml Normal file
View File

@ -0,0 +1,11 @@
---
features:
- |
sahara-scenario supports feature sets. When passing specific
feature tags to sahara-scenario, additional job templates and
EDP jobs marked with those tags will be loaded.
fixes:
- |
When passing the plugin/version parameters to sahara-scenario,
users can now specify additional YAML templates which will be merged
to the default YAMLs, instead of being ignored.

8
sahara_tests/scenario/runner.py
View File

@ -124,6 +124,8 @@ def get_base_parser():
nargs='?', help='Specify Sahara release')
parser.add_argument('--report', default=False, action='store_true',
help='Write results of test to file')
parser.add_argument('--feature', '-f', default=[],
nargs='?', help='Set of features to enable')
parser.add_argument('--count', default=1, nargs='?', type=valid_count,
help='Specify count of runs current cases.')
return parser
@ -155,12 +157,14 @@ def main():
version = args.plugin_version
release = args.release
report = args.report
features = args.feature
count = args.count
auth_values = utils.get_auth_values(cloud_config, args)
scenario_arguments = utils.get_default_templates(plugin, version, release,
scenario_arguments)
scenario_arguments,
features)
files = get_scenario_files(scenario_arguments)
@ -171,7 +175,7 @@ def main():
params_for_login = {'credentials': auth_values}
config = utils.generate_config(files, template_variables, params_for_login,
verbose_run)
verbose_run, features)
# validate config
validation.validate(config)

83
sahara_tests/scenario/utils.py
View File

@ -102,7 +102,8 @@ def get_templates_variables(files, variable_file, verbose_run, scenario_args,
return template_variables
def generate_config(files, template_variables, auth_values, verbose_run):
def generate_config(files, template_variables, auth_values, verbose_run,
features_list=None):
config = {'credentials': {},
'network': {},
'clusters': [],
@ -125,6 +126,50 @@ def generate_config(files, template_variables, auth_values, verbose_run):
else:
raise ValueError('Job flow exist')
config['credentials'].update(auth_values['credentials'])
# filter out the jobs depending on the features, if any
unknown_jobs = []
if features_list is None:
features_list = []
for cluster in config['clusters']:
if cluster.get('edp_jobs_flow'):
filtered_jobs = []
# The jobs associated to a cluster may be a single value (string)
# or a list of values; handle both cases.
cluster_jobs_list = cluster['edp_jobs_flow']
if isinstance(cluster_jobs_list, six.string_types):
cluster_jobs_list = [cluster_jobs_list]
for job_item in cluster_jobs_list:
if isinstance(job_item, dict):
job = job_item['name']
else:
job = job_item
# get the list of features, if defined
job_features = set()
if isinstance(job_item, dict):
job_features = set(job_item.get('features', []))
# If a job has no features associated, it is always used.
# Otherwise, it should be used only if any of its features
# matches any of the features requested,
if (not job_features or
(features_list is not None and
job_features.intersection(features_list))):
# the job is relevant for the configuration,
# so it must be defined; if it is not, it will break,
# so take note of the name
if job not in config['edp_jobs_flow']:
unknown_jobs.append(job)
continue
# job defined, it can be used
filtered_jobs.append(job)
cluster['edp_jobs_flow'] = filtered_jobs
if unknown_jobs:
# Some jobs which are listed in some clusters
# are not defined, stop here
raise ValueError('Unknown jobs: %s' % (unknown_jobs))
if verbose_run:
six.print_("Generated configuration:\n%s" % (
yaml.safe_dump(config,
@ -134,7 +179,10 @@ def generate_config(files, template_variables, auth_values, verbose_run):
return config
def get_default_templates(plugin, version, release, scenario_arguments):
def get_default_templates(plugin, version, release, scenario_arguments,
features=None):
all_templates = []
templates_location = TEST_TEMPLATE_DIR
if release is not None:
templates_location = os.path.join(TEST_TEMPLATE_DIR, release)
@ -146,10 +194,33 @@ def get_default_templates(plugin, version, release, scenario_arguments):
template = "%s-%s.yaml.mako" % (plugin, version)
else:
raise ValueError("Please, specify version for plugin via '-v'")
DEFAULT_TEMPLATE_VARS.append(os.path.join(templates_location,
template))
scenario_arguments = DEFAULT_TEMPLATE_VARS
return scenario_arguments
# find the templates for each features, if they exist
feature_templates_vars = []
if features:
default_templates_base = []
for default_template in DEFAULT_TEMPLATE_VARS:
if default_template.endswith('.yaml.mako'):
default_templates_base.append(
default_template[:-len('.yaml.mako')])
# for each default template, look for a corresponding
# <file>_<feature>.yaml.mako
for feature in features:
for default_template_base in default_templates_base:
template_feature = '%s_%s.yaml.mako' % (
default_template_base, feature)
if os.path.exists(template_feature):
feature_templates_vars.append(template_feature)
# return a combination of: default templates, version-specific
# templates, feature-based templates
all_templates = DEFAULT_TEMPLATE_VARS + [os.path.join(
templates_location, template)] + feature_templates_vars
# it makes sense that all the other templates passed as arguments
# are always added at the end
all_templates += scenario_arguments
return all_templates
def get_auth_values(cloud_config, args):

18
sahara_tests/scenario/validation.py
View File

@ -320,7 +320,23 @@ SCHEMA = {
}
},
"edp_jobs_flow": {
"type": ["string", "array"]
"type": ["string", "array"],
"items": {
"type": ["string", "object"],
"properties": {
"name": {
"type": "string",
},
"features": {
"type": "array",
"items": {
"type": "string",
"minLength": 1
}
}
},
"required": ["name", "features"],
}
},
"retain_resources": {
"type": "boolean"

215
sahara_tests/unit/scenario/test_utils.py Normal file
View File

@ -0,0 +1,215 @@
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import os
import mock
import testtools
from sahara_tests.scenario import utils
class UtilsGenerateConfigTest(testtools.TestCase):
def setUp(self):
super(UtilsGenerateConfigTest, self).setUp()
self._cluster_config = {
"clusters": [{
"edp_jobs_flow": [
"pig_all",
{
"name": "pig_feat2",
"features": ["feat2"]
},
{
"name": "pig_featmulti",
"features": ["feat1", "feat2"]
},
]
}],
"edp_jobs_flow": {
"pig_all": [
{
"type": "Pig",
"input_datasource": {
"type": "swift",
"source": "sahara_tests/scenario/defaults/"
"edp-examples/edp-pig/"
"top-todoers/data/input"
},
"output_datasource": {
"type": "hdfs",
"destination": "/user/hadoop/edp-output"
},
"main_lib": {
"type": "swift",
"source": "sahara_tests/scenario/defaults/"
"edp-examples/edp-pig/"
"top-todoers/example.pig"
}
}
],
"pig_feat2": [
{
"type": "Pig",
"input_datasource": {
"type": "swift",
"source": "sahara_tests/scenario/defaults/"
"edp-examples/edp-pig/"
"top-todoers/data/input"
},
"output_datasource": {
"type": "hdfs",
"destination": "/user/hadoop/edp-output"
},
"main_lib": {
"type": "swift",
"source": "sahara_tests/scenario/defaults/"
"edp-examples/edp-pig/"
"top-todoers/example.pig"
}
}
],
"pig_featmulti": [
{
"type": "Pig",
"input_datasource": {
"type": "swift",
"source": "sahara_tests/scenario/defaults/"
"edp-examples/edp-pig/"
"top-todoers/data/input"
},
"output_datasource": {
"type": "hdfs",
"destination": "/user/hadoop/edp-output"
},
"main_lib": {
"type": "swift",
"source": "sahara_tests/scenario/defaults/"
"edp-examples/edp-pig/"
"top-todoers/example.pig"
}
}
]
}
}
@mock.patch('sahara_tests.scenario.utils.read_scenario_config')
def test_generate_config_feature(self, m_readscenarioconfig):
"""Check the generate_config method when features are specified."""
m_readscenarioconfig.return_value = self._cluster_config
# "template_variables" can be empty because read_scenario_config,
# which is its only users, is a mock variable.
# "files" is used to loop over the read keys, so at one fake
# file name is needed.
result = utils.generate_config(['dummyfile.yaml'], None,
{'credentials': {}}, False,
features_list=['feat1'])
self.assertIn('clusters', result)
self.assertEqual(set(result['clusters'][0]['edp_jobs_flow']),
set(["pig_all", "pig_featmulti"]))
@mock.patch('sahara_tests.scenario.utils.read_scenario_config')
def test_generate_config_nofeatures(self, m_readscenarioconfig):
"""Check the generate_config method when features are specified."""
m_readscenarioconfig.return_value = self._cluster_config
# "template_variables" can be empty because read_scenario_config,
# which is its only users, is a mock variable.
# "files" is used to loop over the read keys, so at one fake
# file name is needed.
result = utils.generate_config(['dummyfile.yaml'], None,
{'credentials': {}}, False,
features_list=[])
self.assertIn('clusters', result)
self.assertEqual(set(result['clusters'][0]['edp_jobs_flow']),
set(["pig_all"]))
@mock.patch('sahara_tests.scenario.utils.read_scenario_config')
def test_generate_config_singlejob_str(self, m_readscenarioconfig):
"""Check the generate_config method when the cluster runs only
a single job defined as string and not as list."""
cluster_config_singlejob_str = self._cluster_config
cluster_config_singlejob_str['clusters'][0]["edp_jobs_flow"] = \
'pig_all'
m_readscenarioconfig.return_value = cluster_config_singlejob_str
# "template_variables" can be empty because read_scenario_config,
# which is its only users, is a mock variable.
# "files" is used to loop over the read keys, so at one fake
# file name is needed.
result = utils.generate_config(['dummyfile.yaml'], None,
{'credentials': {}}, False,
features_list=[])
self.assertIn('clusters', result)
self.assertEqual(set(result['clusters'][0]['edp_jobs_flow']),
set(["pig_all"]))
@mock.patch('sahara_tests.scenario.utils.read_scenario_config')
def test_generate_config_unknownjob(self, m_readscenarioconfig):
"""Check the generate_config method when an unknown job is specified,
thus leading to an exception."""
cluster_config_unknownjob = self._cluster_config
cluster_config_unknownjob['clusters'][0]["edp_jobs_flow"].append(
'unknown_job')
m_readscenarioconfig.return_value = cluster_config_unknownjob
# "template_variables" can be empty because read_scenario_config,
# which is its only users, is a mock variable.
# "files" is used to loop over the read keys, so at one fake
# file name is needed.
self.assertRaises(ValueError, utils.generate_config,
['dummyfile.yaml'], None, {'credentials': {}},
False, features_list=[])
class UtilsTemplatesTest(testtools.TestCase):
def test_get_default_templates_noversion_nofeatures(self):
"""Check the list of automatically discovered templates
when the plugin does not require versions and
there are no features specified."""
found_templates = utils.get_default_templates('fake', None, None,
['conffile.yaml'])
expected_templates = (
utils.DEFAULT_TEMPLATE_VARS +
[os.path.join(utils.TEST_TEMPLATE_DIR, 'fake.yaml.mako'),
'conffile.yaml']
)
self.assertListEqual(found_templates, expected_templates)
def test_get_default_templates_missingversion(self):
"""Check the list of automatically discovered templates
when the plugin requires a version but it was not specified."""
self.assertRaises(ValueError, utils.get_default_templates,
'vanilla', None, None, ['conffile.yaml'])
@mock.patch('os.path.exists', side_effect=[True, False, True, True])
def test_get_default_templates_version_release_features(self, m_exist):
"""Check the list of automatically discovered templates
when the plugin requires a version, a release is specified and
there are features specified."""
found_templates = utils.get_default_templates('vanilla', '2.7.1',
'rocky',
['conffile.yaml'],
['feat1', 'feat2'])
expected_templates = (
utils.DEFAULT_TEMPLATE_VARS +
[os.path.join(utils.TEST_TEMPLATE_DIR, 'rocky',
'vanilla-2.7.1.yaml.mako'),
os.path.join(utils.TEST_TEMPLATE_DIR,
'credentials_feat1.yaml.mako'),
os.path.join(utils.TEST_TEMPLATE_DIR,
'credentials_feat2.yaml.mako'),
os.path.join(utils.TEST_TEMPLATE_DIR,
'edp_feat2.yaml.mako'),
'conffile.yaml']
)
self.assertListEqual(found_templates, expected_templates)

7
sahara_tests/unit/scenario/vanilla2_7_1.yaml
View File

@ -46,7 +46,11 @@ clusters:
- run_jobs
- scale
- run_jobs
edp_jobs_flow: test_flow
edp_jobs_flow:
- test_flow
- name: test_flow_hive
features:
- testhive
edp_batching: 1
retain_resources: true
@ -94,6 +98,7 @@ edp_jobs_flow:
output_datasource:
type: hdfs
destination: /user/hadoop/edp-output
test_flow_hive:
- type: Hive
input_datasource:
type: swift