openstack
/
deb-python-os-apply-config
Code Issues Proposed changes

Retire Packaging Deb project repos 

This commit is part of a series to retire the Packaging Deb
project. Step 2 is to remove all content from the project
repos, replacing it with a README notification where to find
ongoing work, and how to recover the repo if needed at some
future point (as in
https://docs.openstack.org/infra/manual/drivers.html#retiring-a-project).

Change-Id: Ie925a35847b69cac5762e1bba205b1a1364c21a1
This commit is contained in:
Tony Breeds 2017-09-12 16:04:15 -06:00
parent c2e15c8424
commit cb4c456861
44 changed files with 14 additions and 2117 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
.coveragerc
View File

@ -1,7 +0,0 @@
[run]
branch = True
source = os_apply_config
omit = os_apply_config/tests/*,os_apply_config/openstack/*
[report]
ignore_errors = True

45
.gitignore vendored
View File

@ -1,45 +0,0 @@
*.py[cod]
# C extensions
*.so
# Packages
*.egg
*.egg-info
dist
build
eggs
parts
bin
var
sdist
develop-eggs
.installed.cfg
lib
lib64
# Installer logs
pip-log.txt
# Unit test / coverage reports
.coverage
cover
.testrepository
.tox
nosetests.xml
# Translations
*.mo
# Mr Developer
.mr.developer.cfg
.project
.pydevproject
# OpenStack Generated Files
AUTHORS
ChangeLog
# Editors
*~
*.swp

4
.gitreview
View File

@ -1,4 +0,0 @@
[gerrit]
host=review.openstack.org
port=29418
project=openstack/os-apply-config.git

4
.testr.conf
View File

@ -1,4 +0,0 @@
[DEFAULT]
test_command=${PYTHON:-python} -m subunit.run discover -t ./ . $LISTOPT $IDOPTION
test_id_option=--load-list $IDFILE
test_list_option=--list

176
LICENSE
View File

@ -1,176 +0,0 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.

7
MANIFEST.in
View File

@ -1,7 +0,0 @@
include AUTHORS
include ChangeLog
include README.md
exclude .gitignore
exclude .gitreview
global-exclude *.pyc

14
README Normal file
View File

@ -0,0 +1,14 @@
This project is no longer maintained.
The contents of this repository are still available in the Git
source code management system. To see the contents of this
repository before it reached its end of life, please check out the
previous commit with "git checkout HEAD^1".
For ongoing work on maintaining OpenStack packages in the Debian
distribution, please see the Debian OpenStack packaging team at
https://wiki.debian.org/OpenStack/.
For any further questions, please email
openstack-dev@lists.openstack.org or join #openstack-dev on
Freenode.

148
README.rst
View File

@ -1,148 +0,0 @@
========================
Team and repository tags
========================
.. image:: http://governance.openstack.org/badges/os-apply-config.svg
:target: http://governance.openstack.org/reference/tags/index.html
.. Change things from this point on
===============
os-apply-config
===============
-----------------------------------------------
Apply configuration from cloud metadata (JSON)
-----------------------------------------------
What does it do?
================
It turns metadata from one or more JSON files like this::
{"keystone": {"database": {"host": "127.0.0.1", "user": "keystone", "password": "foobar"}}}
into service config files like this::
[sql]
connection = mysql://keystone:foobar@127.0.0.1/keystone
...other settings...
Usage
=====
Just pass it the path to a directory tree of templates::
sudo os-apply-config -t /home/me/my_templates
By default it will read config files according to the contents of
the file `/var/lib/os-collect-config/os_config_files.json`. In
order to remain backward compatible it will also fall back to
/var/run/os-collect-config/os_config_files.json, but the fallback
path is deprecated and will be removed in a later release. The main
path can be changed with the command line switch `--os-config-files`,
or the environment variable `OS_CONFIG_FILES_PATH`. The list can
also be overridden with the environment variable `OS_CONFIG_FILES`.
If overriding with `OS_CONFIG_FILES`, the paths are expected to be colon,
":", separated. Each json file referred to must have a mapping as their
root structure. Keys in files mentioned later in the list will override
keys in earlier files from this list. For example::
OS_CONFIG_FILES=/tmp/ec2.json:/tmp/cfn.json os-apply-config
This will read `ec2.json` and `cfn.json`, and if they have any
overlapping keys, the value from `cfn.json` will be used. That will
populate the tree for any templates found in the template path. See
https://git.openstack.org/cgit/openstack/os-collect-config for a
program that will automatically collect data and populate this list.
You can also override `OS_CONFIG_FILES` with the `--metadata` command
line option, specifying it multiple times instead of colon separating
the list.
`os-apply-config` will also always try to read metadata in the old
legacy paths first to populate the tree. These paths can be changed
with `--fallback-metadata`.
Templates
=========
The template directory structure should mimic a root filesystem, and
contain templates for only those files you want configured. For
example::
~/my_templates$ tree
.
+-- etc
+-- keystone
| +-- keystone.conf
+-- mysql
+-- mysql.conf
An example tree can be found `here <http://git.openstack.org/cgit/openstack/tripleo-image-elements/tree/elements/keystone/os-apply-config>`_.
If a template is executable it will be treated as an *executable
template*. Otherwise, it will be treated as a *mustache template*.
Mustache Templates
------------------
If you don't need any logic, just some string substitution, use a
mustache template.
Metadata settings are accessed with dot ('.') notation::
[sql]
connection = mysql://{{keystone.database.user}}:{{keystone.database.password}}@{{keystone.database.host}}/keystone
Executable Templates
--------------------
Configuration requiring logic is expressed in executable templates.
An executable template is a script which accepts configuration as a
JSON string on standard in, and writes a config file to standard out.
The script should exit non-zero if it encounters a problem, so that
os-apply-config knows what's up.
The output of the script will be written to the path corresponding to
the executable template's path in the template tree::
#!/usr/bin/env ruby
require 'json'
params = JSON.parse STDIN.read
puts "connection = mysql://#{c['keystone']['database']['user']}:#{c['keystone']['database']['password']}@#{c['keystone']['database']['host']}/keystone"
You could even embed mustache in a heredoc, and use that::
#!/usr/bin/env ruby
require 'json'
require 'mustache'
params = JSON.parse STDIN.read
template = <<-eos
[sql]
connection = mysql://{{keystone.database.user}}:{{keystone.database.password}}@{{keystone.database.host}}/keystone
[log]
...
eos
# tweak params here...
puts Mustache.render(template, params)
Quick Start
===========
::
# install it
sudo pip install -U git+git://git.openstack.org/openstack/os-apply-config.git
# grab example templates
git clone git://git.openstack.org/openstack/tripleo-image-elements /tmp/config
# run it
os-apply-config -t /tmp/config/elements/nova/os-apply-config/ -m /tmp/config/elements/seed-stack-config/config.json -o /tmp/config_output

0
os_apply_config/__init__.py
View File

385
os_apply_config/apply_config.py
View File

@ -1,385 +0,0 @@
# Copyright (c) 2013 Hewlett-Packard Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import argparse
import json
import logging
import os
import subprocess
import sys
import tempfile
from pystache import context
import yaml
from os_apply_config import collect_config
from os_apply_config import config_exception as exc
from os_apply_config import oac_file
from os_apply_config import renderers
from os_apply_config import value_types
from os_apply_config import version
DEFAULT_TEMPLATES_DIR = '/usr/libexec/os-apply-config/templates'
def templates_dir():
"""Determine the default templates directory path
If the OS_CONFIG_APPLIER_TEMPLATES environment variable has been set,
use its value.
Otherwise, select a default path based on which directories exist on the
system, preferring the newer paths but still allowing the old ones for
backwards compatibility.
"""
templates_dir = os.environ.get('OS_CONFIG_APPLIER_TEMPLATES', None)
if templates_dir is None:
templates_dir = '/opt/stack/os-apply-config/templates'
if not os.path.isdir(templates_dir):
# Backwards compat with the old name.
templates_dir = '/opt/stack/os-config-applier/templates'
if (os.path.isdir(templates_dir) and
not os.path.isdir(DEFAULT_TEMPLATES_DIR)):
logging.warning('Template directory %s is deprecated. The '
'recommended location for template files is %s',
templates_dir, DEFAULT_TEMPLATES_DIR)
else:
templates_dir = DEFAULT_TEMPLATES_DIR
return templates_dir
TEMPLATES_DIR = templates_dir()
OS_CONFIG_FILES_PATH = os.environ.get(
'OS_CONFIG_FILES_PATH', '/var/lib/os-collect-config/os_config_files.json')
OS_CONFIG_FILES_PATH_OLD = '/var/run/os-collect-config/os_config_files.json'
CONTROL_FILE_SUFFIX = ".oac"
def install_config(
config_path, template_root, output_path, validate, subhash=None,
fallback_metadata=None):
config = strip_hash(
collect_config.collect_config(config_path, fallback_metadata), subhash)
tree = build_tree(template_paths(template_root), config)
if not validate:
for path, obj in tree.items():
write_file(os.path.join(
output_path, strip_prefix('/', path)), obj)
def _extract_key(config_path, key, fallback_metadata=None):
config = collect_config.collect_config(config_path, fallback_metadata)
keys = key.split('.')
for key in keys:
try:
config = config[key]
if config is None:
raise TypeError()
except (KeyError, TypeError):
try:
if type(config) == list:
config = config[int(key)]
continue
except (IndexError, ValueError):
pass
return None
return config
def print_key(
config_path, key, type_name, default=None, fallback_metadata=None):
config = collect_config.collect_config(config_path, fallback_metadata)
config = _extract_key(config_path, key, fallback_metadata)
if config is None:
if default is not None:
print(str(default))
return
else:
raise exc.ConfigException(
'key %s does not exist in %s' % (key, config_path))
value_types.ensure_type(str(config), type_name)
if isinstance(config, (dict, list, bool)):
print(json.dumps(config))
else:
print(str(config))
def boolean_key(metadata, key, fallback_metadata):
config = _extract_key(metadata, key, fallback_metadata)
if not isinstance(config, bool):
return -1
if config:
return 0
else:
return 1
def write_file(path, obj):
if not obj.allow_empty and len(obj.body) == 0:
if os.path.exists(path):
logger.info("deleting %s", path)
os.unlink(path)
else:
logger.info("not creating empty %s", path)
return
logger.info("writing %s", path)
if os.path.exists(path):
stat = os.stat(path)
mode, uid, gid = stat.st_mode, stat.st_uid, stat.st_gid
else:
mode, uid, gid = 0o644, -1, -1
mode = obj.mode or mode
if obj.owner is not None:
uid = obj.owner
if obj.group is not None:
gid = obj.group
d = os.path.dirname(path)
os.path.exists(d) or os.makedirs(d)
with tempfile.NamedTemporaryFile(dir=d, delete=False) as newfile:
if type(obj.body) == str:
obj.body = obj.body.encode('utf-8')
newfile.write(obj.body)
os.chmod(newfile.name, mode)
os.chown(newfile.name, uid, gid)
os.rename(newfile.name, path)
def build_tree(templates, config):
"""Return a map of filenames to OacFiles."""
res = {}
for in_file, out_file in templates:
try:
body = render_template(in_file, config)
ctrl_file = in_file + CONTROL_FILE_SUFFIX
ctrl_dict = {}
if os.path.isfile(ctrl_file):
with open(ctrl_file) as cf:
ctrl_body = cf.read()
ctrl_dict = yaml.safe_load(ctrl_body) or {}
if not isinstance(ctrl_dict, dict):
raise exc.ConfigException(
"header is not a dict: %s" % in_file)
res[out_file] = oac_file.OacFile(body, **ctrl_dict)
except exc.ConfigException as e:
e.args += in_file,
raise
return res
def render_template(template, config):
if is_executable(template):
return render_executable(template, config)
else:
try:
return render_moustache(open(template).read(), config)
except context.KeyNotFoundError as e:
raise exc.ConfigException(
"key '%s' from template '%s' does not exist in metadata file."
% (e.key, template))
except Exception as e:
logger.error("%s", e)
raise exc.ConfigException(
"could not render moustache template %s" % template)
def is_executable(path):
return os.path.isfile(path) and os.access(path, os.X_OK)
def render_moustache(text, config):
r = renderers.JsonRenderer(missing_tags='ignore')
return r.render(text, config)
def render_executable(path, config):
p = subprocess.Popen([path],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
stdout, stderr = p.communicate(json.dumps(config).encode('utf-8'))
p.wait()
if p.returncode != 0:
raise exc.ConfigException(
"config script failed: %s\n\nwith output:\n\n%s" %
(path, stdout + stderr))
return stdout.decode('utf-8')
def template_paths(root):
res = []
for cur_root, _subdirs, files in os.walk(root):
for f in files:
if f.endswith(CONTROL_FILE_SUFFIX):
continue
inout = (os.path.join(cur_root, f), os.path.join(
strip_prefix(root, cur_root), f))
res.append(inout)
return res
def strip_prefix(prefix, s):
return s[len(prefix):] if s.startswith(prefix) else s
def strip_hash(h, keys):
if not keys:
return h
for k in keys.split('.'):
if k in h and isinstance(h[k], dict):
h = h[k]
else:
raise exc.ConfigException(
"key '%s' does not correspond to a hash in the metadata file"
% keys)
return h
def parse_opts(argv):
parser = argparse.ArgumentParser(
description='Reads and merges JSON configuration files specified'
' by colon separated environment variable OS_CONFIG_FILES, unless'
' overridden by command line option --metadata. If no files are'
' specified this way, falls back to legacy behavior of searching'
' the fallback metadata path for a single config file.')
parser.add_argument('-t', '--templates', metavar='TEMPLATE_ROOT',
help="""path to template root directory (default:
%(default)s)""",
default=TEMPLATES_DIR)
parser.add_argument('-o', '--output', metavar='OUT_DIR',
help='root directory for output (default:%(default)s)',
default='/')
parser.add_argument('-m', '--metadata', metavar='METADATA_FILE', nargs='*',
help='Overrides environment variable OS_CONFIG_FILES.'
' Specify multiple times, rather than separate files'
' with ":".',
default=[])
parser.add_argument('--fallback-metadata', metavar='FALLBACK_METADATA',
nargs='*', help='Files to search when OS_CONFIG_FILES'
' is empty. (default: %(default)s)',
default=['/var/cache/heat-cfntools/last_metadata',
'/var/lib/heat-cfntools/cfn-init-data',
'/var/lib/cloud/data/cfn-init-data'])
parser.add_argument(
'-v', '--validate', help='validate only. do not write files',
default=False, action='store_true')
parser.add_argument(
'--print-templates', default=False, action='store_true',
help='Print templates root and exit.')
parser.add_argument('-s', '--subhash',
help='use the sub-hash named by this key,'
' instead of the full metadata hash')
parser.add_argument('--key', metavar='KEY', default=None,
help='print the specified key and exit.'
' (may be used with --type and --key-default)')
parser.add_argument('--type', default='default',
help='exit with error if the specified --key does not'
' match type. Valid types are'
' <int|default|netaddress|netdevice|dsn|'
'swiftdevices|raw>')
parser.add_argument('--key-default',
help='This option only affects running with --key.'
' Print this if key is not found. This value is'
' not subject to type restrictions. If --key is'
' specified and no default is specified, program'
' exits with an error on missing key.')
parser.add_argument('--boolean-key',
help='This option is incompatible with --key.'
' Use this to evaluate whether a value is'
' boolean true or false. The return code of the'
' command will be 0 for true, 1 for false, and -1'
' for non-boolean values.')
parser.add_argument('--version', action='version',
version=version.version_info.version_string())
parser.add_argument('--os-config-files',
default=OS_CONFIG_FILES_PATH,
help='Set path to os_config_files.json')
opts = parser.parse_args(argv[1:])
return opts
def load_list_from_json(json_file):
json_obj = []
if os.path.exists(json_file):
with open(json_file) as ocf:
json_obj = json.loads(ocf.read())
if not isinstance(json_obj, list):
raise ValueError("No list defined in json file: %s" % json_file)
return json_obj
def main(argv=sys.argv):
opts = parse_opts(argv)
if opts.print_templates:
print(opts.templates)
return 0
if not opts.metadata:
if 'OS_CONFIG_FILES' in os.environ:
opts.metadata = os.environ['OS_CONFIG_FILES'].split(':')
else:
opts.metadata = load_list_from_json(opts.os_config_files)
if ((not opts.metadata and opts.os_config_files ==
OS_CONFIG_FILES_PATH)):
logger.warning('DEPRECATED: falling back to %s' %
OS_CONFIG_FILES_PATH_OLD)
opts.metadata = load_list_from_json(OS_CONFIG_FILES_PATH_OLD)
if opts.key and opts.boolean_key:
logger.warning('--key is not compatible with --boolean-key.'
' --boolean-key ignored.')
try:
if opts.templates is None:
raise exc.ConfigException('missing option --templates')
if opts.key:
print_key(opts.metadata,
opts.key,
opts.type,
opts.key_default,
opts.fallback_metadata)
elif opts.boolean_key:
return boolean_key(opts.metadata,
opts.boolean_key,
opts.fallback_metadata)
else:
install_config(opts.metadata, opts.templates, opts.output,
opts.validate, opts.subhash, opts.fallback_metadata)
logger.info("success")
except exc.ConfigException as e:
logger.error(e)
return 1
return 0
# logging
LOG_FORMAT = '[%(asctime)s] [%(levelname)s] %(message)s'
DATE_FORMAT = '%Y/%m/%d %I:%M:%S %p'
def add_handler(logger, handler):
handler.setFormatter(logging.Formatter(LOG_FORMAT, datefmt=DATE_FORMAT))
logger.addHandler(handler)
logger = logging.getLogger('os-apply-config')
logger.setLevel(logging.INFO)
add_handler(logger, logging.StreamHandler())
if os.geteuid() == 0:
add_handler(logger, logging.FileHandler('/var/log/os-apply-config.log'))
if __name__ == '__main__':
sys.exit(main(sys.argv))

70
os_apply_config/collect_config.py
View File

@ -1,70 +0,0 @@
# Copyright (c) 2013 Hewlett-Packard Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import copy
import json
import os
from os_apply_config import config_exception as exc
def read_configs(config_files):
'''Generator yields data from any existing file in list config_files.'''
for input_path in [x for x in config_files if x]:
if os.path.exists(input_path):
try:
with open(input_path) as input_file:
yield((input_file.read(), input_path))
except IOError as e:
raise exc.ConfigException('Could not open %s for reading. %s' %
(input_path, e))
def parse_configs(config_data):
'''Generator yields parsed json for each item passed in config_data.'''
for input_data, input_path in config_data:
try:
yield(json.loads(input_data))
except ValueError:
raise exc.ConfigException('Could not parse metadata file: %s' %
input_path)
def _deep_merge_dict(a, b):
if not isinstance(b, dict):
return b
new_dict = copy.deepcopy(a)
for k, v in iter(b.items()):
if k in new_dict and isinstance(new_dict[k], dict):
new_dict[k] = _deep_merge_dict(new_dict[k], v)
else:
new_dict[k] = copy.deepcopy(v)
return new_dict
def merge_configs(parsed_configs):
'''Returns deep-merged dict from passed list of dicts.'''
final_conf = {}
for conf in parsed_configs:
if conf:
final_conf = _deep_merge_dict(final_conf, conf)
return final_conf
def collect_config(os_config_files, fallback_paths=None):
'''Convenience method to read, parse, and merge all paths.'''
if fallback_paths:
os_config_files = fallback_paths + os_config_files
return merge_configs(parse_configs(read_configs(os_config_files)))

18
os_apply_config/config_exception.py
View File

@ -1,18 +0,0 @@
# Copyright (c) 2013 Hewlett-Packard Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
class ConfigException(Exception):
pass

146
os_apply_config/oac_file.py
View File

@ -1,146 +0,0 @@
# Copyright (c) 2014 Hewlett-Packard Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import grp
import pwd
import six
from os_apply_config import config_exception as exc
class OacFile(object):
DEFAULTS = {
'allow_empty': True,
'mode': None,
'owner': None,
'group': None,
}
def __init__(self, body, **kwargs):
super(OacFile, self).__init__()
self.body = body
for k, v in six.iteritems(self.DEFAULTS):
setattr(self, '_' + k, v)
for k, v in six.iteritems(kwargs):
if not hasattr(self, k):
raise exc.ConfigException(
"unrecognised file control key '%s'" % (k))
setattr(self, k, v)
def __eq__(self, other):
if type(other) is type(self):
return self.__dict__ == other.__dict__
return False
def __ne__(self, other):
return not self.__eq__(other)
def __repr__(self):
a = ["OacFile(%s" % repr(self.body)]
for key, default in six.iteritems(self.DEFAULTS):
value = getattr(self, key)
if value != default:
a.append("%s=%s" % (key, repr(value)))
return ", ".join(a) + ")"
def set(self, key, value):
"""Allows setting attrs as an expression rather than a statement."""
setattr(self, key, value)
return self
@property
def allow_empty(self):
"""Returns allow_empty.
If True and body='', no file will be created and any existing
file will be deleted.
"""
return self._allow_empty
@allow_empty.setter
def allow_empty(self, value):
if type(value) is not bool:
raise exc.ConfigException(
"allow_empty requires Boolean, got: '%s'" % value)
self._allow_empty = value
return self
@property
def mode(self):
"""The permissions to set on the file, EG 0755."""
return self._mode
@mode.setter
def mode(self, v):
"""Pass in the mode to set on the file.
EG 0644. Must be between 0 and 0777, the sticky bit is not supported.
"""
if type(v) is not int:
raise exc.ConfigException("mode '%s' is not numeric" % v)
if not 0 <= v <= 0o777:
raise exc.ConfigException("mode '%#o' out of range" % v)
self._mode = v
@property
def owner(self):
"""The UID to set on the file, EG 'rabbitmq' or '501'."""
return self._owner
@owner.setter
def owner(self, v):
"""Pass in the UID to set on the file.
EG 'rabbitmq' or 501.
"""
try:
if type(v) is int:
user = pwd.getpwuid(v)
elif type(v) is str:
user = pwd.getpwnam(v)
else:
raise exc.ConfigException(
"owner '%s' must be a string or int" % v)
except KeyError:
raise exc.ConfigException(
"owner '%s' not found in passwd database" % v)
self._owner = user[2]
@property
def group(self):
"""The GID to set on the file, EG 'rabbitmq' or '501'."""
return self._group
@group.setter
def group(self, v):
"""Pass in the GID to set on the file.
EG 'rabbitmq' or 501.
"""
try:
if type(v) is int:
group = grp.getgrgid(v)
elif type(v) is str:
group = grp.getgrnam(v)
else:
raise exc.ConfigException(
"group '%s' must be a string or int" % v)
except KeyError:
raise exc.ConfigException(
"group '%s' not found in group database" % v)
self._group = group[2]

41
os_apply_config/renderers.py
View File

@ -1,41 +0,0 @@
# Copyright (c) 2013 Hewlett-Packard Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import json
import pystache
class JsonRenderer(pystache.Renderer):
def __init__(self,
file_encoding=None,
string_encoding=None,
decode_errors=None,
search_dirs=None,
file_extension=None,
escape=None,
partials=None,
missing_tags=None):
# json would be html escaped otherwise
if escape is None:
escape = lambda u: u
return super(JsonRenderer, self).__init__(file_encoding,
string_encoding,
decode_errors, search_dirs,
file_extension, escape,
partials, missing_tags)
def str_coerce(self, val):
return json.dumps(val)

0
os_apply_config/tests/__init__.py
View File

1
os_apply_config/tests/chown_templates/group.gid
View File

@ -1 +0,0 @@
lorem gido

1
os_apply_config/tests/chown_templates/group.gid.oac
View File

@ -1 +0,0 @@
group: 0

1
os_apply_config/tests/chown_templates/group.name
View File

@ -1 +0,0 @@
namo gido

1
os_apply_config/tests/chown_templates/group.name.oac
View File

@ -1 +0,0 @@
group: root

1
os_apply_config/tests/chown_templates/owner.name
View File

@ -1 +0,0 @@
namo uido

1
os_apply_config/tests/chown_templates/owner.name.oac
View File

@ -1 +0,0 @@
owner: root

1
os_apply_config/tests/chown_templates/owner.uid
View File

@ -1 +0,0 @@
lorem uido

1
os_apply_config/tests/chown_templates/owner.uid.oac
View File

@ -1 +0,0 @@
owner: 0

0
os_apply_config/tests/templates/etc/control/allow_empty
View File

1
os_apply_config/tests/templates/etc/control/allow_empty.oac
View File

@ -1 +0,0 @@
allow_empty: false

1
os_apply_config/tests/templates/etc/control/empty
View File

@ -1 +0,0 @@
foo

1
os_apply_config/tests/templates/etc/control/empty.oac
View File

@ -1 +0,0 @@
# comment

1
os_apply_config/tests/templates/etc/control/mode
View File

@ -1 +0,0 @@
lorem modus

1
os_apply_config/tests/templates/etc/control/mode.oac
View File

@ -1 +0,0 @@
mode: 0755

8
os_apply_config/tests/templates/etc/glance/script.conf
View File

@ -1,8 +0,0 @@
#!/usr/bin/env python
from __future__ import print_function
import json
import sys
params = json.loads(sys.stdin.read())
x = params["x"]
if x is None: raise Exception("undefined: x")
print(x)

2
os_apply_config/tests/templates/etc/keystone/keystone.conf
View File

@ -1,2 +0,0 @@
[foo]
database = {{database.url}}

430
os_apply_config/tests/test_apply_config.py
View File

@ -1,430 +0,0 @@
# Copyright (c) 2013 Hewlett-Packard Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import atexit
import json
import os
import tempfile
import fixtures
import mock
import testtools
from os_apply_config import apply_config
from os_apply_config import config_exception as exc
from os_apply_config import oac_file
# example template tree
TEMPLATES = os.path.join(os.path.dirname(__file__), 'templates')
# config for example tree
CONFIG = {
"x": "foo",
"y": False,
"z": None,
"btrue": True,
"bfalse": False,
"database": {
"url": "sqlite:///blah"
},
"l": [1, 2],
}
# config for example tree - with subhash
CONFIG_SUBHASH = {
"OpenStack::Config": {
"x": "foo",
"database": {
"url": "sqlite:///blah"
}
}
}
# expected output for example tree
OUTPUT = {
"/etc/glance/script.conf": oac_file.OacFile(
"foo\n"),
"/etc/keystone/keystone.conf": oac_file.OacFile(
"[foo]\ndatabase = sqlite:///blah\n"),
"/etc/control/empty": oac_file.OacFile(
"foo\n"),
"/etc/control/allow_empty": oac_file.OacFile(
"").set('allow_empty', False),
"/etc/control/mode": oac_file.OacFile(
"lorem modus\n").set('mode', 0o755),
}
TEMPLATE_PATHS = OUTPUT.keys()
# expected output for chown tests
# separated out to avoid needing to mock os.chown for most tests
CHOWN_TEMPLATES = os.path.join(os.path.dirname(__file__), 'chown_templates')
CHOWN_OUTPUT = {
"owner.uid": oac_file.OacFile("lorem uido\n").set('owner', 0),
"owner.name": oac_file.OacFile("namo uido\n").set('owner', 0),
"group.gid": oac_file.OacFile("lorem gido\n").set('group', 0),
"group.name": oac_file.OacFile("namo gido\n").set('group', 0),
}
def main_path():
return (
os.path.dirname(os.path.realpath(__file__)) +
'/../os_apply_config.py')
def template(relpath):
return os.path.join(TEMPLATES, relpath[1:])
class TestRunOSConfigApplier(testtools.TestCase):
"""Tests the commandline options."""
def setUp(self):
super(TestRunOSConfigApplier, self).setUp()
self.useFixture(fixtures.NestedTempfile())
self.stdout = self.useFixture(fixtures.StringStream('stdout')).stream
self.useFixture(fixtures.MonkeyPatch('sys.stdout', self.stdout))
stderr = self.useFixture(fixtures.StringStream('stderr')).stream
self.useFixture(fixtures.MonkeyPatch('sys.stderr', stderr))
self.logger = self.useFixture(
fixtures.FakeLogger(name="os-apply-config"))
fd, self.path = tempfile.mkstemp()
with os.fdopen(fd, 'w') as t:
t.write(json.dumps(CONFIG))
t.flush()
def test_print_key(self):
self.assertEqual(0, apply_config.main(
['os-apply-config.py', '--metadata', self.path, '--key',
'database.url', '--type', 'raw']))
self.stdout.seek(0)
self.assertEqual(CONFIG['database']['url'],
self.stdout.read().strip())
self.assertEqual('', self.logger.output)
def test_print_key_json_dict(self):
self.assertEqual(0, apply_config.main(
['os-apply-config.py', '--metadata', self.path, '--key',
'database', '--type', 'raw']))
self.stdout.seek(0)
self.assertEqual(CONFIG['database'],
json.loads(self.stdout.read().strip()))
self.assertEqual('', self.logger.output)
def test_print_key_json_list(self):
self.assertEqual(0, apply_config.main(
['os-apply-config.py', '--metadata', self.path, '--key',
'l', '--type', 'raw']))
self.stdout.seek(0)
self.assertEqual(CONFIG['l'],
json.loads(self.stdout.read().strip()))
self.assertEqual('', self.logger.output)
def test_print_non_string_key(self):
self.assertEqual(0, apply_config.main(
['os-apply-config.py', '--metadata', self.path, '--key',
'y', '--type', 'raw']))
self.stdout.seek(0)
self.assertEqual("false",
self.stdout.read().strip())
self.assertEqual('', self.logger.output)
def test_print_null_key(self):
self.assertEqual(0, apply_config.main(
['os-apply-config.py', '--metadata', self.path, '--key',
'z', '--type', 'raw', '--key-default', '']))
self.stdout.seek(0)
self.assertEqual('', self.stdout.read().strip())
self.assertEqual('', self.logger.output)
def test_print_key_missing(self):
self.assertEqual(1, apply_config.main(
['os-apply-config.py', '--metadata', self.path, '--key',
'does.not.exist']))
self.assertIn('does not exist', self.logger.output)
def test_print_key_missing_default(self):
self.assertEqual(0, apply_config.main(
['os-apply-config.py', '--metadata', self.path, '--key',
'does.not.exist', '--key-default', '']))
self.stdout.seek(0)
self.assertEqual('', self.stdout.read().strip())
self.assertEqual('', self.logger.output)
def test_print_key_wrong_type(self):
self.assertEqual(1, apply_config.main(
['os-apply-config.py', '--metadata', self.path, '--key',
'x', '--type', 'int']))
self.assertIn('cannot interpret value', self.logger.output)
def test_print_key_from_list(self):
self.assertEqual(0, apply_config.main(
['os-apply-config.py', '--metadata', self.path, '--key',
'l.0', '--type', 'int']))
self.stdout.seek(0)
self.assertEqual(str(CONFIG['l'][0]),
self.stdout.read().strip())
self.assertEqual('', self.logger.output)
def test_print_key_from_list_missing(self):
self.assertEqual(1, apply_config.main(
['os-apply-config.py', '--metadata', self.path, '--key',
'l.2', '--type', 'int']))
self.assertIn('does not exist', self.logger.output)
def test_print_key_from_list_missing_default(self):
self.assertEqual(0, apply_config.main(
['os-apply-config.py', '--metadata', self.path, '--key',
'l.2', '--type', 'int', '--key-default', '']))
self.stdout.seek(0)
self.assertEqual('', self.stdout.read().strip())
self.assertEqual('', self.logger.output)
def test_print_templates(self):
apply_config.main(['os-apply-config', '--print-templates'])
self.stdout.seek(0)
self.assertEqual(
self.stdout.read().strip(), apply_config.TEMPLATES_DIR)
self.assertEqual('', self.logger.output)
def test_boolean_key(self):
rcode = apply_config.main(['os-apply-config', '--metadata',
self.path, '--boolean-key', 'btrue'])
self.assertEqual(0, rcode)
rcode = apply_config.main(['os-apply-config', '--metadata',
self.path, '--boolean-key', 'bfalse'])
self.assertEqual(1, rcode)
rcode = apply_config.main(['os-apply-config', '--metadata',
self.path, '--boolean-key', 'x'])
self.assertEqual(-1, rcode)
def test_boolean_key_and_key(self):
rcode = apply_config.main(['os-apply-config', '--metadata',
self.path, '--boolean-key', 'btrue',
'--key', 'x'])
self.assertEqual(0, rcode)
self.stdout.seek(0)
self.assertEqual(self.stdout.read().strip(), 'foo')
self.assertIn('--boolean-key ignored', self.logger.output)
def test_os_config_files(self):
with tempfile.NamedTemporaryFile() as fake_os_config_files:
with tempfile.NamedTemporaryFile() as fake_config:
fake_config.write(json.dumps(CONFIG).encode('utf-8'))
fake_config.flush()
fake_os_config_files.write(
json.dumps([fake_config.name]).encode('utf-8'))
fake_os_config_files.flush()
apply_config.main(['os-apply-config',
'--key', 'database.url',
'--type', 'raw',
'--os-config-files',
fake_os_config_files.name])
self.stdout.seek(0)
self.assertEqual(
CONFIG['database']['url'], self.stdout.read().strip())
class OSConfigApplierTestCase(testtools.TestCase):
def setUp(self):
super(OSConfigApplierTestCase, self).setUp()
self.logger = self.useFixture(fixtures.FakeLogger('os-apply-config'))
self.useFixture(fixtures.NestedTempfile())
def write_config(self, config):
fd, path = tempfile.mkstemp()
with os.fdopen(fd, 'w') as t:
t.write(json.dumps(config))
t.flush()
return path
def check_output_file(self, tmpdir, path, obj):
full_path = os.path.join(tmpdir, path[1:])
if obj.allow_empty:
assert os.path.exists(full_path), "%s doesn't exist" % path
self.assertEqual(obj.body, open(full_path).read())
else:
assert not os.path.exists(full_path), "%s exists" % path
def test_install_config(self):
path = self.write_config(CONFIG)
tmpdir = tempfile.mkdtemp()
apply_config.install_config([path], TEMPLATES, tmpdir, False)
for path, obj in OUTPUT.items():
self.check_output_file(tmpdir, path, obj)
def test_install_config_subhash(self):
tpath = self.write_config(CONFIG_SUBHASH)
tmpdir = tempfile.mkdtemp()
apply_config.install_config(
[tpath], TEMPLATES, tmpdir, False, 'OpenStack::Config')
for path, obj in OUTPUT.items():
self.check_output_file(tmpdir, path, obj)
def test_delete_if_not_allowed_empty(self):
path = self.write_config(CONFIG)
tmpdir = tempfile.mkdtemp()
template = "/etc/control/allow_empty"
target_file = os.path.join(tmpdir, template[1:])
# Touch the file
os.makedirs(os.path.dirname(target_file))
open(target_file, 'a').close()
apply_config.install_config([path], TEMPLATES, tmpdir, False)
# File should be gone
self.assertFalse(os.path.exists(target_file))
def test_respect_file_permissions(self):
path = self.write_config(CONFIG)
tmpdir = tempfile.mkdtemp()
template = "/etc/keystone/keystone.conf"
target_file = os.path.join(tmpdir, template[1:])
os.makedirs(os.path.dirname(target_file))
# File doesn't exist, use the default mode (644)
apply_config.install_config([path], TEMPLATES, tmpdir, False)
self.assertEqual(0o100644, os.stat(target_file).st_mode)
self.assertEqual(OUTPUT[template].body, open(target_file).read())
# Set a different mode:
os.chmod(target_file, 0o600)
apply_config.install_config([path], TEMPLATES, tmpdir, False)
# The permissions should be preserved
self.assertEqual(0o100600, os.stat(target_file).st_mode)
self.assertEqual(OUTPUT[template].body, open(target_file).read())
def test_build_tree(self):
tree = apply_config.build_tree(
apply_config.template_paths(TEMPLATES), CONFIG)
self.assertEqual(OUTPUT, tree)
def test_render_template(self):
# execute executable files, moustache non-executables
self.assertEqual("abc\n", apply_config.render_template(template(
"/etc/glance/script.conf"), {"x": "abc"}))
self.assertRaises(
exc.ConfigException,
apply_config.render_template,
template("/etc/glance/script.conf"), {})
def test_render_template_bad_template(self):
tdir = self.useFixture(fixtures.TempDir())
bt_path = os.path.join(tdir.path, 'bad_template')
with open(bt_path, 'w') as bt:
bt.write("{{#foo}}bar={{bar}}{{/bar}}")
e = self.assertRaises(exc.ConfigException,
apply_config.render_template,
bt_path, {'foo': [{'bar':
'abc'}]})
self.assertIn('could not render moustache template', str(e))
self.assertIn('Section end tag mismatch', self.logger.output)
def test_render_moustache(self):
self.assertEqual(
"ab123cd",
apply_config.render_moustache("ab{{x.a}}cd", {"x": {"a": "123"}}))
def test_render_moustache_bad_key(self):
self.assertEqual(u'', apply_config.render_moustache("{{badkey}}", {}))
def test_render_executable(self):
params = {"x": "foo"}
self.assertEqual("foo\n", apply_config.render_executable(
template("/etc/glance/script.conf"), params))
def test_render_executable_failure(self):
self.assertRaises(
exc.ConfigException,
apply_config.render_executable,
template("/etc/glance/script.conf"), {})
def test_template_paths(self):
expected = list(map(lambda p: (template(p), p), TEMPLATE_PATHS))
actual = apply_config.template_paths(TEMPLATES)
expected.sort(key=lambda tup: tup[1])
actual.sort(key=lambda tup: tup[1])
self.assertEqual(expected, actual)
def test_strip_hash(self):
h = {'a': {'b': {'x': 'y'}}, "c": [1, 2, 3]}
self.assertEqual({'x': 'y'}, apply_config.strip_hash(h, 'a.b'))
self.assertRaises(exc.ConfigException,
apply_config.strip_hash, h, 'a.nonexistent')
self.assertRaises(exc.ConfigException,
apply_config.strip_hash, h, 'a.c')
def test_load_list_from_json(self):
def mkstemp():
fd, path = tempfile.mkstemp()
atexit.register(
lambda: os.path.exists(path) and os.remove(path))
return (fd, path)
def write_contents(fd, contents):
with os.fdopen(fd, 'w') as t:
t.write(contents)
t.flush()
fd, path = mkstemp()
load_list = apply_config.load_list_from_json
self.assertRaises(ValueError, load_list, path)
write_contents(fd, json.dumps(["/tmp/config.json"]))
json_obj = load_list(path)
self.assertEqual(["/tmp/config.json"], json_obj)
os.remove(path)
self.assertEqual([], load_list(path))
fd, path = mkstemp()
write_contents(fd, json.dumps({}))
self.assertRaises(ValueError, load_list, path)
def test_default_templates_dir_current(self):
default = '/usr/libexec/os-apply-config/templates'
with mock.patch('os.path.isdir', lambda x: x == default):
self.assertEqual(default, apply_config.templates_dir())
def test_default_templates_dir_deprecated(self):
default = '/opt/stack/os-apply-config/templates'
with mock.patch('os.path.isdir', lambda x: x == default):
self.assertEqual(default, apply_config.templates_dir())
def test_default_templates_dir_old_deprecated(self):
default = '/opt/stack/os-config-applier/templates'
with mock.patch('os.path.isdir', lambda x: x == default):
self.assertEqual(default, apply_config.templates_dir())
def test_default_templates_dir_both(self):
default = '/usr/libexec/os-apply-config/templates'
deprecated = '/opt/stack/os-apply-config/templates'
with mock.patch('os.path.isdir', lambda x: (x == default or
x == deprecated)):
self.assertEqual(default, apply_config.templates_dir())
def test_control_mode(self):
path = self.write_config(CONFIG)
tmpdir = tempfile.mkdtemp()
template = "/etc/control/mode"
target_file = os.path.join(tmpdir, template[1:])
apply_config.install_config([path], TEMPLATES, tmpdir, False)
self.assertEqual(0o100755, os.stat(target_file).st_mode)
@mock.patch('os.chown')
def test_control_chown(self, chown_mock):
path = self.write_config(CONFIG)
tmpdir = tempfile.mkdtemp()
apply_config.install_config([path], CHOWN_TEMPLATES, tmpdir, False)
chown_mock.assert_has_calls([mock.call(mock.ANY, 0, -1), # uid
mock.call(mock.ANY, 0, -1), # username
mock.call(mock.ANY, -1, 0), # gid
mock.call(mock.ANY, -1, 0)], # groupname
any_order=True)

121
os_apply_config/tests/test_collect_config.py
View File

@ -1,121 +0,0 @@
# Copyright (c) 2013 Hewlett-Packard Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import json
import os
import fixtures
import testtools
from os_apply_config import collect_config
from os_apply_config import config_exception as exc
class OCCTestCase(testtools.TestCase):
def test_collect_config(self):
conflict_configs = [('ec2', {'local-ipv4': '192.0.2.99',
'instance-id': 'feeddead'}),
('cfn', {'foo': {'bar': 'foo-bar'},
'local-ipv4': '198.51.100.50'})]
config_files = []
tdir = self.useFixture(fixtures.TempDir())
for name, config in conflict_configs:
path = os.path.join(tdir.path, '%s.json' % name)
with open(path, 'w') as out:
out.write(json.dumps(config))
config_files.append(path)
config = collect_config.collect_config(config_files)
self.assertEqual(
{'local-ipv4': '198.51.100.50',
'instance-id': 'feeddead',
'foo': {'bar': 'foo-bar'}}, config)
def test_collect_config_fallback(self):
tdir = self.useFixture(fixtures.TempDir())
with open(os.path.join(tdir.path, 'does_exist.json'), 'w') as t:
t.write(json.dumps({'a': 1}))
noexist_path = os.path.join(tdir.path, 'does_not_exist.json')
config = collect_config.collect_config([], [noexist_path, t.name])
self.assertEqual({'a': 1}, config)
with open(os.path.join(tdir.path, 'does_exist_new.json'), 'w') as t2:
t2.write(json.dumps({'a': 2}))
config = collect_config.collect_config([t2.name], [t.name])
self.assertEqual({'a': 2}, config)
config = collect_config.collect_config([], [t.name, noexist_path])
self.assertEqual({'a': 1}, config)
self.assertEqual({},
collect_config.collect_config([], [noexist_path]))
self.assertEqual({},
collect_config.collect_config([]))
def test_failed_read(self):
tdir = self.useFixture(fixtures.TempDir())
unreadable_path = os.path.join(tdir.path, 'unreadable.json')
with open(unreadable_path, 'w') as u:
u.write(json.dumps({}))
os.chmod(unreadable_path, 0o000)
self.assertRaises(
exc.ConfigException,
lambda: list(collect_config.read_configs([unreadable_path])))
def test_bad_json(self):
tdir = self.useFixture(fixtures.TempDir())
bad_json_path = os.path.join(tdir.path, 'bad.json')
self.assertRaises(
exc.ConfigException,
lambda: list(collect_config.parse_configs([('{', bad_json_path)])))
class TestMergeConfigs(testtools.TestCase):
def test_merge_configs_noconflict(self):
noconflict_configs = [{'a': '1'},
{'b': 'Y'}]
result = collect_config.merge_configs(noconflict_configs)
self.assertEqual({'a': '1',
'b': 'Y'}, result)
def test_merge_configs_conflict(self):
conflict_configs = [{'a': '1'}, {'a': 'Z'}]
result = collect_config.merge_configs(conflict_configs)
self.assertEqual({'a': 'Z'}, result)
def test_merge_configs_deep_conflict(self):
deepconflict_conf = [{'a': '1'},
{'b': {'x': 'foo-bar', 'y': 'tribbles'}},
{'b': {'x': 'shazam'}}]
result = collect_config.merge_configs(deepconflict_conf)
self.assertEqual({'a': '1',
'b': {'x': 'shazam', 'y': 'tribbles'}}, result)
def test_merge_configs_type_conflict(self):
type_conflict = [{'a': 1}, {'a': [7, 8, 9]}]
result = collect_config.merge_configs(type_conflict)
self.assertEqual({'a': [7, 8, 9]}, result)
def test_merge_configs_list_conflict(self):
list_conflict = [{'a': [1, 2, 3]},
{'a': [4, 5, 6]}]
result = collect_config.merge_configs(list_conflict)
self.assertEqual({'a': [4, 5, 6]}, result)
def test_merge_configs_empty_notdict(self):
list_conflict = [[], {'a': '1'}, '', None, {'b': '2'}, {}]
result = collect_config.merge_configs(list_conflict)
self.assertEqual({'a': '1', 'b': '2'}, result)

38
os_apply_config/tests/test_json_renderer.py
View File

@ -1,38 +0,0 @@
# Copyright (c) 2013 Hewlett-Packard Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import json
import testtools
from testtools import content
from os_apply_config import renderers
TEST_JSON = '{"a":{"b":[1,2,3,"foo"],"c": "the quick brown fox"}}'
class JsonRendererTestCase(testtools.TestCase):
def test_json_renderer(self):
context = json.loads(TEST_JSON)
x = renderers.JsonRenderer()
result = x.render('{{a.b}}', context)
self.addDetail('result', content.text_content(result))
result_structure = json.loads(result)
desire_structure = json.loads('[1,2,3,"foo"]')
self.assertEqual(desire_structure, result_structure)
result = x.render('{{a.c}}', context)
self.addDetail('result', content.text_content(result))
self.assertEqual(u'the quick brown fox', result)

90
os_apply_config/tests/test_oac_file.py
View File

@ -1,90 +0,0 @@
# Copyright (c) 2013 Hewlett-Packard Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import grp
import pwd
import testtools
from os_apply_config import config_exception as exc
from os_apply_config import oac_file
class OacFileTestCase(testtools.TestCase):
def test_mode_string(self):
oacf = oac_file.OacFile('')
mode = '0644'
try:
oacf.mode = mode
except exc.ConfigException as e:
self.assertIn("mode '%s' is not numeric" % mode, str(e))
def test_mode_range(self):
oacf = oac_file.OacFile('')
for mode in [-1, 0o1000]:
try:
oacf.mode = mode
except exc.ConfigException as e:
self.assertTrue("mode '%#o' out of range" % mode in str(e),
"mode: %#o" % mode)
for mode in [0, 0o777]:
oacf.mode = mode
def test_owner_positive(self):
oacf = oac_file.OacFile('')
users = pwd.getpwall()
for name in [user[0] for user in users]:
oacf.owner = name
for uid in [user[2] for user in users]:
oacf.owner = uid
def test_owner_negative(self):
oacf = oac_file.OacFile('')
try:
user = -1
oacf.owner = user
except exc.ConfigException as e:
self.assertIn(
"owner '%s' not found in passwd database" % user, str(e))
try:
user = "za"
oacf.owner = user
except exc.ConfigException as e:
self.assertIn(
"owner '%s' not found in passwd database" % user, str(e))
def test_group_positive(self):
oacf = oac_file.OacFile('')
groups = grp.getgrall()
for name in [group[0] for group in groups]:
oacf.group = name
for gid in [group[2] for group in groups]:
oacf.group = gid
def test_group_negative(self):
oacf = oac_file.OacFile('')
try:
group = -1
oacf.group = group
except exc.ConfigException as e:
self.assertIn(
"group '%s' not found in group database" % group, str(e))
try:
group = "za"
oacf.group = group
except exc.ConfigException as e:
self.assertIn(
"group '%s' not found in group database" % group, str(e))

158
os_apply_config/tests/test_value_type.py
View File

@ -1,158 +0,0 @@
# Copyright (c) 2013 Hewlett-Packard Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import testtools
from os_apply_config import config_exception
from os_apply_config import value_types
class ValueTypeTestCase(testtools.TestCase):
def test_unknown_type(self):
self.assertRaises(
ValueError, value_types.ensure_type, "foo", "badtype")
def test_int(self):
self.assertEqual("123", value_types.ensure_type("123", "int"))
def test_default(self):
self.assertEqual("foobar",
value_types.ensure_type("foobar", "default"))
self.assertEqual("x86_64",
value_types.ensure_type("x86_64", "default"))
def test_default_bad(self):
self.assertRaises(config_exception.ConfigException,
value_types.ensure_type, "foo\nbar", "default")
def test_default_empty(self):
self.assertEqual('',
value_types.ensure_type('', 'default'))
def test_raw_empty(self):
self.assertEqual('',
value_types.ensure_type('', 'raw'))
def test_net_address_ipv4(self):
self.assertEqual('192.0.2.1', value_types.ensure_type('192.0.2.1',
'netaddress'))
def test_net_address_cidr(self):
self.assertEqual('192.0.2.0/24',
value_types.ensure_type('192.0.2.0/24', 'netaddress'))
def test_ent_address_ipv6(self):
self.assertEqual('::', value_types.ensure_type('::', 'netaddress'))
self.assertEqual('2001:db8::2:1', value_types.ensure_type(
'2001:db8::2:1', 'netaddress'))
def test_net_address_dns(self):
self.assertEqual('host.0domain-name.test',
value_types.ensure_type('host.0domain-name.test',
'netaddress'))
def test_net_address_empty(self):
self.assertEqual('', value_types.ensure_type('', 'netaddress'))
def test_net_address_bad(self):
self.assertRaises(config_exception.ConfigException,
value_types.ensure_type, "192.0.2.1;DROP TABLE foo",
'netaddress')
def test_netdevice(self):
self.assertEqual('eth0',
value_types.ensure_type('eth0', 'netdevice'))
def test_netdevice_dash(self):
self.assertEqual('br-ctlplane',
value_types.ensure_type('br-ctlplane', 'netdevice'))
def test_netdevice_alias(self):
self.assertEqual('eth0:1',
value_types.ensure_type('eth0:1', 'netdevice'))
def test_netdevice_bad(self):
self.assertRaises(config_exception.ConfigException,
value_types.ensure_type, "br-tun; DROP TABLE bar",
'netdevice')
def test_dsn_nopass(self):
test_dsn = 'mysql://user@host/db'
self.assertEqual(test_dsn, value_types.ensure_type(test_dsn, 'dsn'))
def test_dsn(self):
test_dsn = 'mysql://user:pass@host/db'
self.assertEqual(test_dsn, value_types.ensure_type(test_dsn, 'dsn'))
def test_dsn_set_variables(self):
test_dsn = 'mysql://user:pass@host/db?charset=utf8'
self.assertEqual(test_dsn, value_types.ensure_type(test_dsn, 'dsn'))
def test_dsn_sqlite_memory(self):
test_dsn = 'sqlite://'
self.assertEqual(test_dsn, value_types.ensure_type(test_dsn, 'dsn'))
def test_dsn_sqlite_file(self):
test_dsn = 'sqlite:///tmp/foo.db'
self.assertEqual(test_dsn, value_types.ensure_type(test_dsn, 'dsn'))
def test_dsn_bad(self):
self.assertRaises(config_exception.ConfigException,
value_types.ensure_type,
"mysql:/user:pass@host/db?charset=utf8", 'dsn')
self.assertRaises(config_exception.ConfigException,
value_types.ensure_type,
"mysql://user:pass@host/db?charset=utf8;DROP TABLE "
"foo", 'dsn')
def test_swiftdevices_single(self):
test_swiftdevices = 'r1z1-127.0.0.1:%PORT%/d1'
self.assertEqual(test_swiftdevices, value_types.ensure_type(
test_swiftdevices,
'swiftdevices'))
def test_swiftdevices_multi(self):
test_swiftdevices = 'r1z1-127.0.0.1:%PORT%/d1,r1z1-127.0.0.1:%PORT%/d2'
self.assertEqual(test_swiftdevices, value_types.ensure_type(
test_swiftdevices,
'swiftdevices'))
def test_swiftdevices_blank(self):
test_swiftdevices = ''
self.assertRaises(config_exception.ConfigException,
value_types.ensure_type,
test_swiftdevices,
'swiftdevices')
def test_swiftdevices_bad(self):
test_swiftdevices = 'rz1-127.0.0.1:%PORT%/d1'
self.assertRaises(config_exception.ConfigException,
value_types.ensure_type,
test_swiftdevices,
'swiftdevices')
def test_username(self):
for test_username in ['guest', 'guest_13-42']:
self.assertEqual(test_username, value_types.ensure_type(
test_username,
'username'))
def test_username_bad(self):
for test_username in ['guest`ls`', 'guest$PASSWD', 'guest 2']:
self.assertRaises(config_exception.ConfigException,
value_types.ensure_type,
test_username,
'username')

44
os_apply_config/value_types.py
View File

@ -1,44 +0,0 @@
# Copyright (c) 2013 Hewlett-Packard Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import re
from os_apply_config import config_exception
TYPES = {
"int": "^[0-9]+$",
"default": "^[A-Za-z0-9_]*$",
"netaddress": "^[A-Za-z0-9/.:-]*$",
"netdevice": "^[A-Za-z0-9/.:-]*$",
"dsn": "(?#driver)^[a-zA-Z0-9]+://"
"(?#username[:password])([a-zA-Z0-9+_-]+(:[^@]+)?)?"
"(?#@host or file)(@?[a-zA-Z0-9/_.-]+)?"
"(?#/dbname)(/[a-zA-Z0-9_-]+)?"
"(?#?variable=value)(\?[a-zA-Z0-9=_-]+)?$",
"swiftdevices": "^(r\d+z\d+-[A-Za-z0-9.-_]+:%PORT%/[^,]+,?)+$",
"username": "^[A-Za-z0-9_-]+$",
"raw": ""
}
def ensure_type(string_value, type_name='default'):
if type_name not in TYPES:
raise ValueError(
"requested validation of unknown type: %s" % type_name)
if not re.match(TYPES[type_name], string_value):
exception = config_exception.ConfigException
raise exception("cannot interpret value '%s' as type %s" % (
string_value, type_name))
return string_value

18
os_apply_config/version.py
View File

@ -1,18 +0,0 @@
# Copyright (c) 2013 Hewlett-Packard Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import pbr.version
version_info = pbr.version.VersionInfo('os-apply-config')

9
requirements.txt
View File

@ -1,9 +0,0 @@
# The order of packages is significant, because pip processes them in the order
# of appearance. Changing the order has an impact on the overall integration
# process, which may cause wedges in the gate later.
pbr!=2.1.0,>=2.0.0 # Apache-2.0
anyjson>=0.3.3 # BSD
pystache # MIT
PyYAML>=3.10.0 # MIT
six>=1.9.0 # MIT

31
setup.cfg
View File

@ -1,31 +0,0 @@
[metadata]
name = os-apply-config
author = OpenStack
author-email = openstack-dev@lists.openstack.org
summary = Config files from cloud metadata
description-file =
README.rst
home-page = http://git.openstack.org/cgit/openstack/os-apply-config
classifier =
Development Status :: 4 - Beta
Environment :: Console
Environment :: OpenStack
Intended Audience :: Developers
Intended Audience :: Information Technology
License :: OSI Approved :: Apache Software License
Operating System :: OS Independent
Programming Language :: Python
[files]
packages =
os_apply_config
[entry_points]
console_scripts =
os-config-applier = os_apply_config.apply_config:main
os-apply-config = os_apply_config.apply_config:main
[egg_info]
tag_build =
tag_date = 0
tag_svn_revision = 0

29
setup.py
View File

@ -1,29 +0,0 @@
# Copyright (c) 2013 Hewlett-Packard Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# THIS FILE IS MANAGED BY THE GLOBAL REQUIREMENTS REPO - DO NOT EDIT
import setuptools
# In python < 2.7.4, a lazy loading of package `pbr` will break
# setuptools if some other modules registered functions in `atexit`.
# solution from: http://bugs.python.org/issue15881#msg170215
try:
import multiprocessing # noqa
except ImportError:
pass
setuptools.setup(
setup_requires=['pbr>=2.0.0'],
pbr=True)

13
test-requirements.txt
View File

@ -1,13 +0,0 @@
# The order of packages is significant, because pip processes them in the order
# of appearance. Changing the order has an impact on the overall integration
# process, which may cause wedges in the gate later.
hacking!=0.13.0,<0.14,>=0.12.0 # Apache-2.0
coverage!=4.4,>=4.0 # Apache-2.0
fixtures>=3.0.0 # Apache-2.0/BSD
mock>=2.0 # BSD
python-subunit>=0.0.18 # Apache-2.0/BSD
sphinx>=1.6.2 # BSD
testrepository>=0.0.18 # Apache-2.0/BSD
testscenarios>=0.4 # Apache-2.0/BSD
testtools>=1.4.0 # MIT

30
tools/tox_install.sh
View File

@ -1,30 +0,0 @@
#!/usr/bin/env bash
# Client constraint file contains this client version pin that is in conflict
# with installing the client from source. We should remove the version pin in
# the constraints file before applying it for from-source installation.
CONSTRAINTS_FILE="$1"
shift 1
set -e
# NOTE(tonyb): Place this in the tox enviroment's log dir so it will get
# published to logs.openstack.org for easy debugging.
localfile="$VIRTUAL_ENV/log/upper-constraints.txt"
if [[ "$CONSTRAINTS_FILE" != http* ]]; then
CONSTRAINTS_FILE="file://$CONSTRAINTS_FILE"
fi
# NOTE(tonyb): need to add curl to bindep.txt if the project supports bindep
curl "$CONSTRAINTS_FILE" --insecure --progress-bar --output "$localfile"
pip install -c"$localfile" openstack-requirements
# This is the main purpose of the script: Allow local installation of
# the current repo. It is listed in constraints file and thus any
# install will be constrained and we need to unconstrain it.
edit-constraints "$localfile" -- "$CLIENT_NAME"
pip install -c"$localfile" -U "$@"
exit $?

32
tox.ini
View File

@ -1,32 +0,0 @@
[tox]
minversion = 2.0
skipsdist = True
envlist = py27,pep8
[testenv]
usedevelop = True
install_command = {toxinidir}/tools/tox_install.sh {env:UPPER_CONSTRAINTS_FILE:https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt} {opts} {packages}
setenv = VIRTUAL_ENV={envdir}
BRANCH_NAME=master
CLIENT_NAME=os-apply-config
deps = -r{toxinidir}/requirements.txt
-r{toxinidir}/test-requirements.txt
commands =
python setup.py testr --slowest --testr-args='{posargs}'
[tox:jenkins]
sitepackages = True
[testenv:pep8]
commands = flake8
[testenv:cover]
commands =
python setup.py test --coverage --coverage-package-name=os_apply_config
[testenv:venv]
commands = {posargs}
[flake8]
exclude = .venv,.tox,dist,doc,*.egg
show-source = true