diff --git a/CONTRIBUTING.rst b/CONTRIBUTING.rst new file mode 100644 index 0000000..81d0448 --- /dev/null +++ b/CONTRIBUTING.rst @@ -0,0 +1,19 @@ +If you would like to contribute to the development of OpenStack, +you must follow the steps in this page: + + http://docs.openstack.org/infra/manual/developers.html + +Once those steps have been completed, changes to OpenStack +should be submitted for review via the Gerrit tool, following +the workflow documented at: + + http://docs.openstack.org/infra/manual/developers.html#development-workflow + +Pull requests submitted through GitHub will be ignored. + +Bugs should be filed on Launchpad, not GitHub: + + https://bugs.launchpad.net/meteos + + + diff --git a/HACKING.rst b/HACKING.rst new file mode 100644 index 0000000..c3990d0 --- /dev/null +++ b/HACKING.rst @@ -0,0 +1,11 @@ +Meteos Style Commandments +============================ + +- Step 1: Read the OpenStack Style Commandments + http://docs.openstack.org/developer/hacking/ +- Step 2: Read on + + +Meteos Specific Commandments +------------------------------- + diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..68c771a --- /dev/null +++ b/LICENSE @@ -0,0 +1,176 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + diff --git a/README.rst b/README.rst new file mode 100644 index 0000000..4f7f984 --- /dev/null +++ b/README.rst @@ -0,0 +1,27 @@ +====== +Meteos +====== + +You have come across an OpenStack Machine Learning service. It has +identified itself as "Meteos." It was abstracted from the Manila +project. + +* Wiki: https://wiki.openstack.org/Meteos +* Developer docs: http://docs.openstack.org/developer/meteos + +Getting Started +--------------- + +If you'd like to run from the master branch, you can clone the git repo: + + git clone https://github.com/openstack/meteos.git + +For developer information please see +`HACKING.rst `_ + +You can raise bugs here http://bugs.launchpad.net/meteos + +Python client +------------- + +https://github.com/openstack/python-meteosclient.git diff --git a/babel.cfg b/babel.cfg new file mode 100644 index 0000000..15cd6cb --- /dev/null +++ b/babel.cfg @@ -0,0 +1,2 @@ +[python: **.py] + diff --git a/devstack/README.rst b/devstack/README.rst new file mode 100644 index 0000000..21aca85 --- /dev/null +++ b/devstack/README.rst @@ -0,0 +1,21 @@ +====================== + Enabling in Devstack +====================== + +1. Download DevStack + +2. Add this repo as an external repository in ``local.conf`` + +.. sourcecode:: bash + + [[local|localrc]] + enable_plugin meteos git://git.openstack.org/openstack/meteos + +Optionally, a git refspec may be provided as follows: + +.. sourcecode:: bash + + [[local|localrc]] + enable_plugin meteos git://git.openstack.org/openstack/meteos + +3. run ``stack.sh`` diff --git a/devstack/exercise.sh b/devstack/exercise.sh new file mode 100644 index 0000000..ef4794b --- /dev/null +++ b/devstack/exercise.sh @@ -0,0 +1,50 @@ +#!/usr/bin/env bash + +# Sanity check that Meteos started if enabled + +echo "*********************************************************************" +echo "Begin DevStack Exercise: $0" +echo "*********************************************************************" + +# This script exits on an error so that errors don't compound and you see +# only the first error that occurred. +set -o errexit + +# Print the commands being run so that we can see the command that triggers +# an error. It is also useful for following allowing as the install occurs. +set -o xtrace + + +# Settings +# ======== + +# Keep track of the current directory +EXERCISE_DIR=$(cd $(dirname "$0") && pwd) +TOP_DIR=$(cd $EXERCISE_DIR/..; pwd) + +# Import common functions +source $TOP_DIR/functions + +# Import configuration +source $TOP_DIR/openrc + +# Import exercise configuration +source $TOP_DIR/exerciserc + +is_service_enabled meteos || exit 55 + +if is_ssl_enabled_service "meteos" ||\ + is_ssl_enabled_service "meteos-api" ||\ + is_service_enabled tls-proxy; then + METEOS_SERVICE_PROTOCOL="https" +fi + +METEOS_SERVICE_PROTOCOL=${METEOS_SERVICE_PROTOCOL:-$SERVICE_PROTOCOL} + +$CURL_GET $METEOS_SERVICE_PROTOCOL://$SERVICE_HOST:8989/ 2>/dev/null \ + | grep -q 'Auth' || die $LINENO "Meteos API isn't functioning!" + +set +o xtrace +echo "*********************************************************************" +echo "SUCCESS: End DevStack Exercise: $0" +echo "*********************************************************************" diff --git a/devstack/plugin.sh b/devstack/plugin.sh new file mode 100644 index 0000000..9624055 --- /dev/null +++ b/devstack/plugin.sh @@ -0,0 +1,187 @@ +#!/bin/bash +# +# lib/meteos + +# Dependencies: +# ``functions`` file +# ``DEST``, ``DATA_DIR``, ``STACK_USER`` must be defined + +# ``stack.sh`` calls the entry points in this order: +# +# install_meteos +# install_python_meteosclient +# configure_meteos +# start_meteos +# stop_meteos +# cleanup_meteos + +# Save trace setting +XTRACE=$(set +o | grep xtrace) +set +o xtrace + + +# Functions +# --------- + +# create_meteos_accounts() - Set up common required meteos accounts +# +# Tenant User Roles +# ------------------------------ +# service meteos admin +function create_meteos_accounts { + + create_service_user "meteos" + + get_or_create_service "meteos" "machine-learning" "Meteos Machine Learning" + get_or_create_endpoint "machine-learning" \ + "$REGION_NAME" \ + "$METEOS_SERVICE_PROTOCOL://$METEOS_SERVICE_HOST:$METEOS_SERVICE_PORT/v1/\$(tenant_id)s" \ + "$METEOS_SERVICE_PROTOCOL://$METEOS_SERVICE_HOST:$METEOS_SERVICE_PORT/v1/\$(tenant_id)s" \ + "$METEOS_SERVICE_PROTOCOL://$METEOS_SERVICE_HOST:$METEOS_SERVICE_PORT/v1/\$(tenant_id)s" +} + +# cleanup_meteos() - Remove residual data files, anything left over from +# previous runs that would need to clean up. +function cleanup_meteos { + + # Cleanup auth cache dir + sudo rm -rf $METEOS_AUTH_CACHE_DIR +} + +# configure_meteos() - Set config files, create data dirs, etc +function configure_meteos { + sudo install -d -o $STACK_USER $METEOS_CONF_DIR + + if [[ -f $METEOS_DIR/etc/meteos/policy.json ]]; then + cp -p $METEOS_DIR/etc/meteos/policy.json $METEOS_CONF_DIR + fi + + cp -p $METEOS_DIR/etc/meteos/api-paste.ini $METEOS_CONF_DIR + + # Create auth cache dir + sudo install -d -o $STACK_USER -m 700 $METEOS_AUTH_CACHE_DIR + rm -rf $METEOS_AUTH_CACHE_DIR/* + + configure_auth_token_middleware \ + $METEOS_CONF_FILE meteos $METEOS_AUTH_CACHE_DIR + + # Set admin user parameters needed for trusts creation + iniset $METEOS_CONF_FILE \ + keystone_authtoken admin_tenant_name $SERVICE_TENANT_NAME + iniset $METEOS_CONF_FILE keystone_authtoken admin_user meteos + iniset $METEOS_CONF_FILE \ + keystone_authtoken admin_password $SERVICE_PASSWORD + + iniset_rpc_backend meteos $METEOS_CONF_FILE DEFAULT + + # Set configuration to send notifications + iniset $METEOS_CONF_FILE DEFAULT debug $ENABLE_DEBUG_LOG_LEVEL + + iniset $METEOS_CONF_FILE DEFAULT plugins $METEOS_ENABLED_PLUGINS + + iniset $METEOS_CONF_FILE \ + database connection `database_connection_url meteos` + + # Format logging + if [ "$LOG_COLOR" == "True" ] && [ "$SYSLOG" == "False" ]; then + setup_colorized_logging $METEOS_CONF_FILE DEFAULT + fi + + recreate_database meteos + $METEOS_BIN_DIR/meteos-manage \ + --config-file $METEOS_CONF_FILE db sync +} + +# install_meteos() - Collect source and prepare +function install_meteos { + setup_develop $METEOS_DIR +} + +# install_python_meteosclient() - Collect source and prepare +function install_python_meteosclient { + git_clone $METEOSCLIENT_REPO $METEOSCLIENT_DIR $METEOSCLIENT_BRANCH + setup_develop $METEOSCLIENT_DIR +} + +# start_meteos() - Start running processes, including screen +function start_meteos { + local service_port=$METEOS_SERVICE_PORT + local service_protocol=$METEOS_SERVICE_PROTOCOL + + run_process meteos-all "$METEOS_BIN_DIR/meteos-all \ + --config-file $METEOS_CONF_FILE" + run_process meteos-api "$METEOS_BIN_DIR/meteos-api \ + --config-file $METEOS_CONF_FILE" + run_process meteos-eng "$METEOS_BIN_DIR/meteos-engine \ + --config-file $METEOS_CONF_FILE" + + echo "Waiting for Meteos to start..." + if ! wait_for_service $SERVICE_TIMEOUT \ + $service_protocol://$METEOS_SERVICE_HOST:$service_port; then + die $LINENO "Meteos did not start" + fi +} + +# configure_tempest_for_meteos() - Tune Tempest configuration for Meteos +function configure_tempest_for_meteos { + if is_service_enabled tempest; then + iniset $TEMPEST_CONFIG service_available meteos True + iniset $TEMPEST_CONFIG data-processing-feature-enabled plugins $METEOS_ENABLED_PLUGINS + fi +} + +# stop_meteos() - Stop running processes +function stop_meteos { + # Kill the Meteos screen windows + stop_process meteos-all + stop_process meteos-api + stop_process meteos-eng +} + +# is_meteos_enabled. This allows is_service_enabled meteos work +# correctly throughout devstack. +function is_meteos_enabled { + if is_service_enabled meteos-api || \ + is_service_enabled meteos-eng || \ + is_service_enabled meteos-all; then + return 0 + else + return 1 + fi +} + +# Dispatcher for Meteos plugin +if is_service_enabled meteos; then + if [[ "$1" == "stack" && "$2" == "install" ]]; then + echo_summary "Installing meteos" + install_meteos + install_python_meteosclient + cleanup_meteos + elif [[ "$1" == "stack" && "$2" == "post-config" ]]; then + echo_summary "Configuring meteos" + configure_meteos + create_meteos_accounts + elif [[ "$1" == "stack" && "$2" == "extra" ]]; then + echo_summary "Initializing meteos" + start_meteos + elif [[ "$1" == "stack" && "$2" == "test-config" ]]; then + echo_summary "Configuring tempest" + configure_tempest_for_meteos + fi + + if [[ "$1" == "unstack" ]]; then + stop_meteos + fi + + if [[ "$1" == "clean" ]]; then + cleanup_meteos + fi +fi + + +# Restore xtrace +$XTRACE + +# Local variables: +# mode: shell-script +# End: diff --git a/devstack/settings b/devstack/settings new file mode 100644 index 0000000..9d6203f --- /dev/null +++ b/devstack/settings @@ -0,0 +1,37 @@ +#!/bin/bash + +# Settings needed for the Meteos plugin +# ------------------------------------- + +# Set up default directories +METEOSCLIENT_DIR=$DEST/python-meteosclient +METEOS_DIR=$DEST/meteos + +METEOSCLIENT_REPO=${METEOSCLIENT_REPO:-\ +${GIT_BASE}/openstack/python-meteosclient.git} +METEOSCLIENT_BRANCH=${METEOSCLIENT_BRANCH:-master} + +METEOS_CONF_DIR=${METEOS_CONF_DIR:-/etc/meteos} +METEOS_CONF_FILE=${METEOS_CONF_DIR}/meteos.conf + +# TODO(slukjanov): Should we append meteos to SSL_ENABLED_SERVICES? + +if is_ssl_enabled_service "meteos" || is_service_enabled tls-proxy; then + METEOS_SERVICE_PROTOCOL="https" +fi +METEOS_SERVICE_HOST=${METEOS_SERVICE_HOST:-$SERVICE_HOST} +METEOS_SERVICE_PORT=${METEOS_SERVICE_PORT:-8989} +METEOS_SERVICE_PROTOCOL=${METEOS_SERVICE_PROTOCOL:-$SERVICE_PROTOCOL} +METEOS_ENDPOINT_TYPE=${METEOS_ENDPOINT_TYPE:-adminURL} + +METEOS_AUTH_CACHE_DIR=${METEOS_AUTH_CACHE_DIR:-/var/cache/meteos} + +METEOS_BIN_DIR=$(get_python_exec_prefix) + +METEOS_ENABLE_DISTRIBUTED_PERIODICS=${METEOS_ENABLE_DISTRIBUTED_PERIODICS:-\ +True} + +# Tell Tempest this project is present +TEMPEST_SERVICES+=,meteos + +enable_service meteos-api meteos-eng diff --git a/etc/meteos/README.meteos.conf b/etc/meteos/README.meteos.conf new file mode 100644 index 0000000..a4a7d31 --- /dev/null +++ b/etc/meteos/README.meteos.conf @@ -0,0 +1,4 @@ +To generate the sample meteos.conf file, run the following +command from the top level of the meteos directory: + +tox -egenconfig diff --git a/etc/meteos/api-paste.ini b/etc/meteos/api-paste.ini new file mode 100644 index 0000000..574527d --- /dev/null +++ b/etc/meteos/api-paste.ini @@ -0,0 +1,49 @@ +############# +# OpenStack # +############# + +[composite:osapi_learning] +use = call:meteos.api:root_app_factory +/: apiversions +/v1: openstack_learning_api + +[composite:openstack_learning_api] +use = call:meteos.api.middleware.auth:pipeline_factory +noauth = cors faultwrap http_proxy_to_wsgi sizelimit noauth api +keystone = cors faultwrap http_proxy_to_wsgi sizelimit authtoken keystonecontext api +keystone_nolimit = cors faultwrap http_proxy_to_wsgi sizelimit authtoken keystonecontext api + +[filter:faultwrap] +paste.filter_factory = meteos.api.middleware.fault:FaultWrapper.factory + +[filter:noauth] +paste.filter_factory = meteos.api.middleware.auth:NoAuthMiddleware.factory + +[filter:sizelimit] +paste.filter_factory = oslo_middleware.sizelimit:RequestBodySizeLimiter.factory + +[filter:http_proxy_to_wsgi] +paste.filter_factory = oslo_middleware.http_proxy_to_wsgi:HTTPProxyToWSGI.factory + +[app:api] +paste.app_factory = meteos.api.v1.router:APIRouter.factory + +[pipeline:apiversions] +pipeline = cors faultwrap http_proxy_to_wsgi oslearningversionapp + +[app:oslearningversionapp] +paste.app_factory = meteos.api.versions:VersionsRouter.factory + +########## +# Engine # +########## + +[filter:keystonecontext] +paste.filter_factory = meteos.api.middleware.auth:MeteosKeystoneContext.factory + +[filter:authtoken] +paste.filter_factory = keystonemiddleware.auth_token:filter_factory + +[filter:cors] +paste.filter_factory = oslo_middleware.cors:filter_factory +oslo_config_project = meteos diff --git a/etc/meteos/api-paste.ini.orig b/etc/meteos/api-paste.ini.orig new file mode 100644 index 0000000..574527d --- /dev/null +++ b/etc/meteos/api-paste.ini.orig @@ -0,0 +1,49 @@ +############# +# OpenStack # +############# + +[composite:osapi_learning] +use = call:meteos.api:root_app_factory +/: apiversions +/v1: openstack_learning_api + +[composite:openstack_learning_api] +use = call:meteos.api.middleware.auth:pipeline_factory +noauth = cors faultwrap http_proxy_to_wsgi sizelimit noauth api +keystone = cors faultwrap http_proxy_to_wsgi sizelimit authtoken keystonecontext api +keystone_nolimit = cors faultwrap http_proxy_to_wsgi sizelimit authtoken keystonecontext api + +[filter:faultwrap] +paste.filter_factory = meteos.api.middleware.fault:FaultWrapper.factory + +[filter:noauth] +paste.filter_factory = meteos.api.middleware.auth:NoAuthMiddleware.factory + +[filter:sizelimit] +paste.filter_factory = oslo_middleware.sizelimit:RequestBodySizeLimiter.factory + +[filter:http_proxy_to_wsgi] +paste.filter_factory = oslo_middleware.http_proxy_to_wsgi:HTTPProxyToWSGI.factory + +[app:api] +paste.app_factory = meteos.api.v1.router:APIRouter.factory + +[pipeline:apiversions] +pipeline = cors faultwrap http_proxy_to_wsgi oslearningversionapp + +[app:oslearningversionapp] +paste.app_factory = meteos.api.versions:VersionsRouter.factory + +########## +# Engine # +########## + +[filter:keystonecontext] +paste.filter_factory = meteos.api.middleware.auth:MeteosKeystoneContext.factory + +[filter:authtoken] +paste.filter_factory = keystonemiddleware.auth_token:filter_factory + +[filter:cors] +paste.filter_factory = oslo_middleware.cors:filter_factory +oslo_config_project = meteos diff --git a/etc/meteos/logging_sample.conf b/etc/meteos/logging_sample.conf new file mode 100644 index 0000000..2c6c3e3 --- /dev/null +++ b/etc/meteos/logging_sample.conf @@ -0,0 +1,73 @@ +[loggers] +keys = root, meteos + +[handlers] +keys = stderr, stdout, watchedfile, syslog, null + +[formatters] +keys = default + +[logger_root] +level = WARNING +handlers = null + +[logger_meteos] +level = INFO +handlers = stderr +qualname = meteos + +[logger_amqplib] +level = WARNING +handlers = stderr +qualname = amqplib + +[logger_sqlalchemy] +level = WARNING +handlers = stderr +qualname = sqlalchemy +# "level = INFO" logs SQL queries. +# "level = DEBUG" logs SQL queries and results. +# "level = WARNING" logs neither. (Recommended for production systems.) + +[logger_boto] +level = WARNING +handlers = stderr +qualname = boto + +[logger_suds] +level = INFO +handlers = stderr +qualname = suds + +[logger_eventletwsgi] +level = WARNING +handlers = stderr +qualname = eventlet.wsgi.server + +[handler_stderr] +class = StreamHandler +args = (sys.stderr,) +formatter = default + +[handler_stdout] +class = StreamHandler +args = (sys.stdout,) +formatter = default + +[handler_watchedfile] +class = handlers.WatchedFileHandler +args = ('meteos.log',) +formatter = default + +[handler_syslog] +class = handlers.SysLogHandler +args = ('/dev/log', handlers.SysLogHandler.LOG_USER) +formatter = default + +[handler_null] +class = meteos.common.openstack.NullHandler +formatter = default +args = () + +[formatter_default] +format = %(message)s diff --git a/etc/meteos/meteos.conf b/etc/meteos/meteos.conf new file mode 100644 index 0000000..0c6f311 --- /dev/null +++ b/etc/meteos/meteos.conf @@ -0,0 +1,39 @@ +[DEFAULT] + +debug = True +verbose = True +log_dir = /var/log/meteos +rpc_backend = rabbit + +host = 10.0.0.6 +hostname = meteos-vm +osapi_learning_listen_port = 8989 +osapi_learning_workers = 4 +meteos-engine_workers = 4 +api_paste_config=api-paste.ini + +[sahara] +auth_url = http://192.168.0.4:5000/v2.0 + +[database] +connection = sqlite:///etc/meteos/meteos.sqlite3 + +[keystone_authtoken] + +auth_uri = http://192.168.0.4:5000/v2.0 +identity_uri=http://192.168.0.4:35357 +admin_tenant_name=services +admin_password=9396d9e9e57545d8 +admin_user=sahara + +[oslo_messaging_rabbit] + +rabbit_host = 192.168.0.4 +rabbit_port = 5672 +rabbit_use_ssl = False +rabbit_userid = guest +rabbit_password = guest + +[engine] + +worker = 4 diff --git a/etc/meteos/policy.json b/etc/meteos/policy.json new file mode 100644 index 0000000..010c2f8 --- /dev/null +++ b/etc/meteos/policy.json @@ -0,0 +1,13 @@ +{ + "context_is_admin": "role:admin", + "admin_or_owner": "is_admin:True or project_id:%(project_id)s", + "default": "rule:admin_or_owner", + + "admin_api": "is_admin:True", + + "learning:create": "", + "learning:delete": "rule:default", + "learning:get": "rule:default", + "learning:get_all": "rule:default", + "learning:update": "rule:default" +} diff --git a/etc/meteos/rootwrap.conf b/etc/meteos/rootwrap.conf new file mode 100644 index 0000000..b156f68 --- /dev/null +++ b/etc/meteos/rootwrap.conf @@ -0,0 +1,27 @@ +# Configuration for meteos-rootwrap +# This file should be owned by (and only-writeable by) the root user + +[DEFAULT] +# List of directories to load filter definitions from (separated by ','). +# These directories MUST all be only writeable by root ! +filters_path=/etc/meteos/rootwrap.d,/usr/learning/meteos/rootwrap + +# List of directories to search executables in, in case filters do not +# explicitely specify a full path (separated by ',') +# If not specified, defaults to system PATH environment variable. +# These directories MUST all be only writeable by root ! +exec_dirs=/sbin,/usr/sbin,/bin,/usr/bin,/usr/local/sbin,/usr/local/bin,/usr/lpp/mmfs/bin + +# Enable logging to syslog +# Default value is False +use_syslog=False + +# Which syslog facility to use. +# Valid values include auth, authpriv, syslog, user0, user1... +# Default value is 'syslog' +syslog_log_facility=syslog + +# Which messages to log. +# INFO means log all usage +# ERROR means only log unsuccessful attempts +syslog_log_level=ERROR diff --git a/etc/meteos/rootwrap.d/learning.filters b/etc/meteos/rootwrap.d/learning.filters new file mode 100644 index 0000000..90736ca --- /dev/null +++ b/etc/meteos/rootwrap.d/learning.filters @@ -0,0 +1,20 @@ +# meteos-rootwrap command filters for share nodes +# This file should be owned by (and only-writeable by) the root user + +[Filters] +# meteos/utils.py : 'chown', '%s', '%s' +chown: CommandFilter, chown, root +# meteos/utils.py : 'cat', '%s' +cat: CommandFilter, cat, root + +# meteos/share/drivers/lvm.py: 'rmdir', '%s' +rmdir: CommandFilter, rmdir, root + +# meteos/share/drivers/helpers.py: 'cp', '%s', '%s' +cp: CommandFilter, cp, root + +# meteos/share/drivers/helpers.py: 'service', '%s', '%s' +service: CommandFilter, service, root + +# meteos/share/drivers/glusterfs.py: 'rm', '-rf', '%s' +rm: CommandFilter, rm, root diff --git a/etc/oslo-config-generator/meteos.conf b/etc/oslo-config-generator/meteos.conf new file mode 100644 index 0000000..08baff1 --- /dev/null +++ b/etc/oslo-config-generator/meteos.conf @@ -0,0 +1,9 @@ +[DEFAULT] +output_file = etc/meteos/meteos.conf.sample +namespace = meteos +namespace = oslo.messaging +namespace = oslo.middleware.cors +namespace = oslo.middleware.http_proxy_to_wsgi +namespace = oslo.db +namespace = oslo.db.concurrency +namespace = keystonemiddleware.auth_token diff --git a/meteos/__init__.py b/meteos/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/meteos/api/__init__.py b/meteos/api/__init__.py new file mode 100644 index 0000000..d91f63a --- /dev/null +++ b/meteos/api/__init__.py @@ -0,0 +1,21 @@ +# Copyright 2010 United States Government as represented by the +# Administrator of the National Aeronautics and Space Administration. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +import paste.urlmap + + +def root_app_factory(loader, global_conf, **local_conf): + return paste.urlmap.urlmap_factory(loader, global_conf, **local_conf) diff --git a/meteos/api/auth.py b/meteos/api/auth.py new file mode 100644 index 0000000..0503e3f --- /dev/null +++ b/meteos/api/auth.py @@ -0,0 +1,38 @@ +# Copyright (c) 2013 OpenStack, LLC. +# +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +from oslo_log import log + +from meteos.api.middleware import auth +from meteos.i18n import _LW + +LOG = log.getLogger(__name__) + + +class MeteosKeystoneContext(auth.MeteosKeystoneContext): + def __init__(self, application): + LOG.warning(_LW('meteos.api.auth:MeteosKeystoneContext is deprecated. ' + 'Please use ' + 'meteos.api.middleware.auth:MeteosKeystoneContext ' + 'instead.')) + super(MeteosKeystoneContext, self).__init__(application) + + +def pipeline_factory(loader, global_conf, **local_conf): + LOG.warning(_LW('meteos.api.auth:pipeline_factory is deprecated. ' + 'Please use meteos.api.middleware.auth:pipeline_factory ' + 'instead.')) + auth.pipeline_factory(loader, global_conf, **local_conf) diff --git a/meteos/api/common.py b/meteos/api/common.py new file mode 100644 index 0000000..298faf4 --- /dev/null +++ b/meteos/api/common.py @@ -0,0 +1,318 @@ +# Copyright 2010 OpenStack LLC. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +import os +import re + +from oslo_config import cfg +from oslo_log import log +from six.moves.urllib import parse +import webob + +from meteos.api.openstack import api_version_request as api_version +from meteos.api.openstack import versioned_method +from meteos.i18n import _ + +api_common_opts = [ + cfg.IntOpt( + 'osapi_max_limit', + default=1000, + help='The maximum number of items returned in a single response from ' + 'a collection resource.'), + cfg.StrOpt( + 'osapi_learning_base_URL', + help='Base URL to be presented to users in links to the Learning API'), +] + +CONF = cfg.CONF +CONF.register_opts(api_common_opts) +LOG = log.getLogger(__name__) + + +# Regex that matches alphanumeric characters, periods, hypens, +# colons and underscores: +# ^ assert position at start of the string +# [\w\.\-\:\_] match expression +# $ assert position at end of the string +VALID_KEY_NAME_REGEX = re.compile(r"^[\w\.\-\:\_]+$", re.UNICODE) + + +def validate_key_names(key_names_list): + """Validate each item of the list to match key name regex.""" + for key_name in key_names_list: + if not VALID_KEY_NAME_REGEX.match(key_name): + return False + return True + + +def get_pagination_params(request): + """Return marker, limit tuple from request. + + :param request: `wsgi.Request` possibly containing 'marker' and 'limit' + GET variables. 'marker' is the id of the last element + the client has seen, and 'limit' is the maximum number + of items to return. If 'limit' is not specified, 0, or + > max_limit, we default to max_limit. Negative values + for either marker or limit will cause + exc.HTTPBadRequest() exceptions to be raised. + + """ + params = {} + if 'limit' in request.GET: + params['limit'] = _get_limit_param(request) + if 'marker' in request.GET: + params['marker'] = _get_marker_param(request) + return params + + +def _get_limit_param(request): + """Extract integer limit from request or fail.""" + try: + limit = int(request.GET['limit']) + except ValueError: + msg = _('limit param must be an integer') + raise webob.exc.HTTPBadRequest(explanation=msg) + if limit < 0: + msg = _('limit param must be positive') + raise webob.exc.HTTPBadRequest(explanation=msg) + return limit + + +def _get_marker_param(request): + """Extract marker ID from request or fail.""" + return request.GET['marker'] + + +def limited(items, request, max_limit=CONF.osapi_max_limit): + """Return a slice of items according to requested offset and limit. + + :param items: A sliceable entity + :param request: ``wsgi.Request`` possibly containing 'offset' and 'limit' + GET variables. 'offset' is where to start in the list, + and 'limit' is the maximum number of items to return. If + 'limit' is not specified, 0, or > max_limit, we default + to max_limit. Negative values for either offset or limit + will cause exc.HTTPBadRequest() exceptions to be raised. + :kwarg max_limit: The maximum number of items to return from 'items' + """ + try: + offset = int(request.GET.get('offset', 0)) + except ValueError: + msg = _('offset param must be an integer') + raise webob.exc.HTTPBadRequest(explanation=msg) + + try: + limit = int(request.GET.get('limit', max_limit)) + except ValueError: + msg = _('limit param must be an integer') + raise webob.exc.HTTPBadRequest(explanation=msg) + + if limit < 0: + msg = _('limit param must be positive') + raise webob.exc.HTTPBadRequest(explanation=msg) + + if offset < 0: + msg = _('offset param must be positive') + raise webob.exc.HTTPBadRequest(explanation=msg) + + limit = min(max_limit, limit or max_limit) + range_end = offset + limit + return items[offset:range_end] + + +def limited_by_marker(items, request, max_limit=CONF.osapi_max_limit): + """Return a slice of items according to the requested marker and limit.""" + params = get_pagination_params(request) + + limit = params.get('limit', max_limit) + marker = params.get('marker') + + limit = min(max_limit, limit) + start_index = 0 + if marker: + start_index = -1 + for i, item in enumerate(items): + if 'flavorid' in item: + if item['flavorid'] == marker: + start_index = i + 1 + break + elif item['id'] == marker or item.get('uuid') == marker: + start_index = i + 1 + break + if start_index < 0: + msg = _('marker [%s] not found') % marker + raise webob.exc.HTTPBadRequest(explanation=msg) + range_end = start_index + limit + return items[start_index:range_end] + + +def remove_version_from_href(href): + """Removes the first api version from the href. + + Given: 'http://www.meteos.com/v1.1/123' + Returns: 'http://www.meteos.com/123' + + Given: 'http://www.meteos.com/v1.1' + Returns: 'http://www.meteos.com' + + """ + parsed_url = parse.urlsplit(href) + url_parts = parsed_url.path.split('/', 2) + + # NOTE: this should match vX.X or vX + expression = re.compile(r'^v([0-9]+|[0-9]+\.[0-9]+)(/.*|$)') + if expression.match(url_parts[1]): + del url_parts[1] + + new_path = '/'.join(url_parts) + + if new_path == parsed_url.path: + msg = 'href %s does not contain version' % href + LOG.debug(msg) + raise ValueError(msg) + + parsed_url = list(parsed_url) + parsed_url[2] = new_path + return parse.urlunsplit(parsed_url) + + +def dict_to_query_str(params): + # TODO(throughnothing): we should just use urllib.urlencode instead of this + # But currently we don't work with urlencoded url's + param_str = "" + for key, val in params.items(): + param_str = param_str + '='.join([str(key), str(val)]) + '&' + + return param_str.rstrip('&') + + +class ViewBuilder(object): + """Model API responses as dictionaries.""" + + _collection_name = None + _detail_version_modifiers = [] + + def _get_links(self, request, identifier): + return [{"rel": "self", + "href": self._get_href_link(request, identifier), }, + {"rel": "bookmark", + "href": self._get_bookmark_link(request, identifier), }] + + def _get_next_link(self, request, identifier): + """Return href string with proper limit and marker params.""" + params = request.params.copy() + params["marker"] = identifier + prefix = self._update_link_prefix(request.application_url, + CONF.osapi_learning_base_URL) + url = os.path.join(prefix, + request.environ["meteos.context"].project_id, + self._collection_name) + return "%s?%s" % (url, dict_to_query_str(params)) + + def _get_href_link(self, request, identifier): + """Return an href string pointing to this object.""" + prefix = self._update_link_prefix(request.application_url, + CONF.osapi_learning_base_URL) + return os.path.join(prefix, + request.environ["meteos.context"].project_id, + self._collection_name, + str(identifier)) + + def _get_bookmark_link(self, request, identifier): + """Create a URL that refers to a specific resource.""" + base_url = remove_version_from_href(request.application_url) + base_url = self._update_link_prefix(base_url, + CONF.osapi_learning_base_URL) + return os.path.join(base_url, + request.environ["meteos.context"].project_id, + self._collection_name, + str(identifier)) + + def _get_collection_links(self, request, items, id_key="uuid"): + """Retrieve 'next' link, if applicable.""" + links = [] + limit = int(request.params.get("limit", 0)) + if limit and limit == len(items): + last_item = items[-1] + if id_key in last_item: + last_item_id = last_item[id_key] + else: + last_item_id = last_item["id"] + links.append({ + "rel": "next", + "href": self._get_next_link(request, last_item_id), + }) + return links + + def _update_link_prefix(self, orig_url, prefix): + if not prefix: + return orig_url + url_parts = list(parse.urlsplit(orig_url)) + prefix_parts = list(parse.urlsplit(prefix)) + url_parts[0:2] = prefix_parts[0:2] + return parse.urlunsplit(url_parts) + + def update_versioned_resource_dict(self, request, resource_dict, resource): + """Updates the given resource dict for the given request version. + + This method calls every method, that is applicable to the request + version, in _detail_version_modifiers. + """ + for method_name in self._detail_version_modifiers: + method = getattr(self, method_name) + if request.api_version_request.matches_versioned_method(method): + request_context = request.environ['meteos.context'] + method.func(self, request_context, resource_dict, resource) + + @classmethod + def versioned_method(cls, min_ver, max_ver=None, experimental=False): + """Decorator for versioning API methods. + + :param min_ver: string representing minimum version + :param max_ver: optional string representing maximum version + :param experimental: flag indicating an API is experimental and is + subject to change or removal at any time + """ + + def decorator(f): + obj_min_ver = api_version.APIVersionRequest(min_ver) + if max_ver: + obj_max_ver = api_version.APIVersionRequest(max_ver) + else: + obj_max_ver = api_version.APIVersionRequest() + + # Add to list of versioned methods registered + func_name = f.__name__ + new_func = versioned_method.VersionedMethod( + func_name, obj_min_ver, obj_max_ver, experimental, f) + + return new_func + + return decorator + + +def remove_invalid_options(context, search_options, allowed_search_options): + """Remove search options that are not valid for non-admin API/context.""" + if context.is_admin: + # Allow all options + return + # Otherwise, strip out all unknown options + unknown_options = [opt for opt in search_options + if opt not in allowed_search_options] + bad_options = ", ".join(unknown_options) + LOG.debug("Removing options '%(bad_options)s' from query", + {"bad_options": bad_options}) + for opt in unknown_options: + del search_options[opt] diff --git a/meteos/api/contrib/__init__.py b/meteos/api/contrib/__init__.py new file mode 100644 index 0000000..5c12530 --- /dev/null +++ b/meteos/api/contrib/__init__.py @@ -0,0 +1,37 @@ +# Copyright 2011 Justin Santa Barbara +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +"""Contrib contains extensions that are shipped with meteos. + +It can't be called 'extensions' because that causes namespacing problems. + +""" + +from oslo_config import cfg +from oslo_log import log + +from meteos.api import extensions + +CONF = cfg.CONF +LOG = log.getLogger(__name__) + + +def standard_extensions(ext_mgr): + extensions.load_standard_extensions(ext_mgr, LOG, __path__, __package__) + + +def select_extensions(ext_mgr): + extensions.load_standard_extensions(ext_mgr, LOG, __path__, __package__, + CONF.osapi_learning_ext_list) diff --git a/meteos/api/middleware/__init__.py b/meteos/api/middleware/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/meteos/api/middleware/auth.py b/meteos/api/middleware/auth.py new file mode 100644 index 0000000..3f79e80 --- /dev/null +++ b/meteos/api/middleware/auth.py @@ -0,0 +1,150 @@ +# Copyright 2010 OpenStack LLC. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. +""" +Common Auth Middleware. + +""" +import os + +from oslo_config import cfg +from oslo_log import log +from oslo_serialization import jsonutils +import webob.dec +import webob.exc + +from meteos.api.openstack import wsgi +from meteos import context +from meteos.i18n import _ +from meteos import wsgi as base_wsgi + +use_forwarded_for_opt = cfg.BoolOpt( + 'use_forwarded_for', + default=False, + help='Treat X-Forwarded-For as the canonical remote address. ' + 'Only enable this if you have a sanitizing proxy.') + +CONF = cfg.CONF +CONF.register_opt(use_forwarded_for_opt) +LOG = log.getLogger(__name__) + + +def pipeline_factory(loader, global_conf, **local_conf): + """A paste pipeline replica that keys off of auth_strategy.""" + pipeline = local_conf[CONF.auth_strategy] + if not CONF.api_rate_limit: + limit_name = CONF.auth_strategy + '_nolimit' + pipeline = local_conf.get(limit_name, pipeline) + pipeline = pipeline.split() + filters = [loader.get_filter(n) for n in pipeline[:-1]] + app = loader.get_app(pipeline[-1]) + filters.reverse() + for filter in filters: + app = filter(app) + return app + + +class InjectContext(base_wsgi.Middleware): + """Add a 'meteos.context' to WSGI environ.""" + + def __init__(self, context, *args, **kwargs): + self.context = context + super(InjectContext, self).__init__(*args, **kwargs) + + @webob.dec.wsgify(RequestClass=base_wsgi.Request) + def __call__(self, req): + req.environ['meteos.context'] = self.context + return self.application + + +class MeteosKeystoneContext(base_wsgi.Middleware): + """Make a request context from keystone headers.""" + + @webob.dec.wsgify(RequestClass=base_wsgi.Request) + def __call__(self, req): + user_id = req.headers.get('X_USER') + user_id = req.headers.get('X_USER_ID', user_id) + if user_id is None: + LOG.debug("Neither X_USER_ID nor X_USER found in request") + return webob.exc.HTTPUnauthorized() + # get the roles + roles = [r.strip() for r in req.headers.get('X_ROLE', '').split(',')] + if 'X_TENANT_ID' in req.headers: + # This is the new header since Keystone went to ID/Name + project_id = req.headers['X_TENANT_ID'] + else: + # This is for legacy compatibility + project_id = req.headers['X_TENANT'] + + # Get the auth token + auth_token = req.headers.get('X_AUTH_TOKEN', + req.headers.get('X_STORAGE_TOKEN')) + + # Build a context, including the auth_token... + remote_address = req.remote_addr + if CONF.use_forwarded_for: + remote_address = req.headers.get('X-Forwarded-For', remote_address) + + service_catalog = None + if req.headers.get('X_SERVICE_CATALOG') is not None: + try: + catalog_header = req.headers.get('X_SERVICE_CATALOG') + service_catalog = jsonutils.loads(catalog_header) + except ValueError: + raise webob.exc.HTTPInternalServerError( + _('Invalid service catalog json.')) + + ctx = context.RequestContext(user_id, + project_id, + roles=roles, + auth_token=auth_token, + remote_address=remote_address, + service_catalog=service_catalog) + + req.environ['meteos.context'] = ctx + return self.application + + +class NoAuthMiddleware(base_wsgi.Middleware): + """Return a fake token if one isn't specified.""" + + @webob.dec.wsgify(RequestClass=wsgi.Request) + def __call__(self, req): + if 'X-Auth-Token' not in req.headers: + user_id = req.headers.get('X-Auth-User', 'admin') + project_id = req.headers.get('X-Auth-Project-Id', 'admin') + os_url = os.path.join(req.url, project_id) + res = webob.Response() + # NOTE(vish): This is expecting and returning Auth(1.1), whereas + # keystone uses 2.0 auth. We should probably allow + # 2.0 auth here as well. + res.headers['X-Auth-Token'] = '%s:%s' % (user_id, project_id) + res.headers['X-Server-Management-Url'] = os_url + res.content_type = 'text/plain' + res.status = '204' + return res + + token = req.headers['X-Auth-Token'] + user_id, _sep, project_id = token.partition(':') + project_id = project_id or user_id + remote_address = getattr(req, 'remote_address', '127.0.0.1') + if CONF.use_forwarded_for: + remote_address = req.headers.get('X-Forwarded-For', remote_address) + ctx = context.RequestContext(user_id, + project_id, + is_admin=True, + remote_address=remote_address) + + req.environ['meteos.context'] = ctx + return self.application diff --git a/meteos/api/middleware/fault.py b/meteos/api/middleware/fault.py new file mode 100644 index 0000000..056cbcd --- /dev/null +++ b/meteos/api/middleware/fault.py @@ -0,0 +1,74 @@ +# Copyright 2010 United States Government as represented by the +# Administrator of the National Aeronautics and Space Administration. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +from oslo_log import log +import six +import webob.dec +import webob.exc + +from meteos.api.openstack import wsgi +from meteos.i18n import _LE, _LI +from meteos import utils +from meteos import wsgi as base_wsgi + +LOG = log.getLogger(__name__) + + +class FaultWrapper(base_wsgi.Middleware): + """Calls down the middleware stack, making exceptions into faults.""" + + _status_to_type = {} + + @staticmethod + def status_to_type(status): + if not FaultWrapper._status_to_type: + for clazz in utils.walk_class_hierarchy(webob.exc.HTTPError): + FaultWrapper._status_to_type[clazz.code] = clazz + return FaultWrapper._status_to_type.get( + status, webob.exc.HTTPInternalServerError)() + + def _error(self, inner, req): + LOG.exception(_LE("Caught error: %s"), six.text_type(inner)) + + safe = getattr(inner, 'safe', False) + headers = getattr(inner, 'headers', None) + status = getattr(inner, 'code', 500) + if status is None: + status = 500 + + msg_dict = dict(url=req.url, status=status) + LOG.info(_LI("%(url)s returned with HTTP %(status)d"), msg_dict) + outer = self.status_to_type(status) + if headers: + outer.headers = headers + # NOTE(johannes): We leave the explanation empty here on + # purpose. It could possibly have sensitive information + # that should not be returned back to the user. See + # bugs 868360 and 874472 + # NOTE(eglynn): However, it would be over-conservative and + # inconsistent with the EC2 API to hide every exception, + # including those that are safe to expose, see bug 1021373 + if safe: + outer.explanation = '%s: %s' % (inner.__class__.__name__, + six.text_type(inner)) + return wsgi.Fault(outer) + + @webob.dec.wsgify(RequestClass=wsgi.Request) + def __call__(self, req): + try: + return req.get_response(self.application) + except Exception as ex: + return self._error(ex, req) diff --git a/meteos/api/openstack/__init__.py b/meteos/api/openstack/__init__.py new file mode 100644 index 0000000..32b5c02 --- /dev/null +++ b/meteos/api/openstack/__init__.py @@ -0,0 +1,81 @@ +# Copyright (c) 2013 OpenStack, LLC. +# +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +""" +WSGI middleware for OpenStack API controllers. +""" + +from oslo_log import log +import routes + +from meteos.api.openstack import wsgi +from meteos.i18n import _, _LW +from meteos import wsgi as base_wsgi + +LOG = log.getLogger(__name__) + + +class APIMapper(routes.Mapper): + def routematch(self, url=None, environ=None): + if url is "": + result = self._match("", environ) + return result[0], result[1] + return routes.Mapper.routematch(self, url, environ) + + +class ProjectMapper(APIMapper): + def resource(self, member_name, collection_name, **kwargs): + if 'parent_resource' not in kwargs: + kwargs['path_prefix'] = '{project_id}/' + else: + parent_resource = kwargs['parent_resource'] + p_collection = parent_resource['collection_name'] + p_member = parent_resource['member_name'] + kwargs['path_prefix'] = '{project_id}/%s/:%s_id' % (p_collection, + p_member) + routes.Mapper.resource(self, + member_name, + collection_name, + **kwargs) + + +class APIRouter(base_wsgi.Router): + """Routes requests on the API to the appropriate controller and method.""" + ExtensionManager = None # override in subclasses + + @classmethod + def factory(cls, global_config, **local_config): + """Simple paste factory, :class:`meteos.wsgi.Router` doesn't have.""" + return cls() + + def __init__(self, ext_mgr=None): + mapper = ProjectMapper() + self.resources = {} + self._setup_routes(mapper, ext_mgr) + super(APIRouter, self).__init__(mapper) + + def _setup_routes(self, mapper, ext_mgr): + raise NotImplementedError + + +class FaultWrapper(base_wsgi.Middleware): + def __init__(self, application): + LOG.warning(_LW('meteos.api.openstack:FaultWrapper is deprecated. ' + 'Please use ' + 'meteos.api.middleware.fault:FaultWrapper instead.')) + # Avoid circular imports from here. + from meteos.api.middleware import fault + super(FaultWrapper, self).__init__(fault.FaultWrapper(application)) diff --git a/meteos/api/openstack/api_version_request.py b/meteos/api/openstack/api_version_request.py new file mode 100644 index 0000000..f0ce3d1 --- /dev/null +++ b/meteos/api/openstack/api_version_request.py @@ -0,0 +1,169 @@ +# Copyright 2014 IBM Corp. +# Copyright 2015 Clinton Knight +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +import re + +from meteos.api.openstack import versioned_method +from meteos import exception +from meteos.i18n import _ +from meteos import utils + +# Define the minimum and maximum version of the API across all of the +# REST API. The format of the version is: +# X.Y where: +# +# - X will only be changed if a significant backwards incompatible API +# change is made which affects the API as whole. That is, something +# that is only very very rarely incremented. +# +# - Y when you make any change to the API. Note that this includes +# semantic changes which may not affect the input or output formats or +# even originate in the API code layer. We are not distinguishing +# between backwards compatible and backwards incompatible changes in +# the versioning system. It must be made clear in the documentation as +# to what is a backwards compatible change and what is a backwards +# incompatible one. + +# +# You must update the API version history string below with a one or +# two line description as well as update rest_api_version_history.rst +REST_API_VERSION_HISTORY = """ + + REST API Version History: + + * 1.0 - Initial version. Includes all V1 APIs and extensions in Mitaka. +""" + +# The minimum and maximum versions of the API supported +# The default api version request is defined to be the +# the minimum version of the API supported. +_MIN_API_VERSION = "1.0" +_MAX_API_VERSION = "1.0" +DEFAULT_API_VERSION = _MIN_API_VERSION + + +# NOTE(cyeoh): min and max versions declared as functions so we can +# mock them for unittests. Do not use the constants directly anywhere +# else. +def min_api_version(): + return APIVersionRequest(_MIN_API_VERSION) + + +def max_api_version(): + return APIVersionRequest(_MAX_API_VERSION) + + +class APIVersionRequest(utils.ComparableMixin): + """This class represents an API Version Request. + + This class includes convenience methods for manipulation + and comparison of version numbers as needed to implement + API microversions. + """ + + def __init__(self, version_string=None, experimental=False): + """Create an API version request object.""" + self._ver_major = None + self._ver_minor = None + self._experimental = experimental + + if version_string is not None: + match = re.match(r"^([1-9]\d*)\.([1-9]\d*|0)$", + version_string) + if match: + self._ver_major = int(match.group(1)) + self._ver_minor = int(match.group(2)) + else: + raise exception.InvalidAPIVersionString(version=version_string) + + def __str__(self): + """Debug/Logging representation of object.""" + return ("API Version Request Major: %(major)s, Minor: %(minor)s" + % {'major': self._ver_major, 'minor': self._ver_minor}) + + def is_null(self): + return self._ver_major is None and self._ver_minor is None + + def _cmpkey(self): + """Return the value used by ComparableMixin for rich comparisons.""" + return self._ver_major, self._ver_minor + + @property + def experimental(self): + return self._experimental + + @experimental.setter + def experimental(self, value): + if type(value) != bool: + msg = _('The experimental property must be a bool value.') + raise exception.InvalidParameterValue(err=msg) + self._experimental = value + + def matches_versioned_method(self, method): + """Compares this version to that of a versioned method.""" + + if type(method) != versioned_method.VersionedMethod: + msg = _('An API version request must be compared ' + 'to a VersionedMethod object.') + raise exception.InvalidParameterValue(err=msg) + + return self.matches(method.start_version, + method.end_version, + method.experimental) + + def matches(self, min_version, max_version, experimental=False): + """Compares this version to the specified min/max range. + + Returns whether the version object represents a version + greater than or equal to the minimum version and less than + or equal to the maximum version. + + If min_version is null then there is no minimum limit. + If max_version is null then there is no maximum limit. + If self is null then raise ValueError. + + :param min_version: Minimum acceptable version. + :param max_version: Maximum acceptable version. + :param experimental: Whether to match experimental APIs. + :returns: boolean + """ + + if self.is_null(): + raise ValueError + # NOTE(cknight): An experimental request should still match a + # non-experimental API, so the experimental check isn't just + # looking for equality. + if not self.experimental and experimental: + return False + if max_version.is_null() and min_version.is_null(): + return True + elif max_version.is_null(): + return min_version <= self + elif min_version.is_null(): + return self <= max_version + else: + return min_version <= self <= max_version + + def get_string(self): + """Returns a string representation of this object. + + If this method is used to create an APIVersionRequest, + the resulting object will be an equivalent request. + """ + if self.is_null(): + raise ValueError + return ("%(major)s.%(minor)s" % + {'major': self._ver_major, 'minor': self._ver_minor}) diff --git a/meteos/api/openstack/rest_api_version_history.rst b/meteos/api/openstack/rest_api_version_history.rst new file mode 100644 index 0000000..ae37be6 --- /dev/null +++ b/meteos/api/openstack/rest_api_version_history.rst @@ -0,0 +1,14 @@ +REST API Version History +======================== + +This documents the changes made to the REST API with every +microversion change. The description for each version should be a +verbose one which has enough information to be suitable for use in +user documentation. + +1.0 +--- + The 1.0 Meteos API includes all v1 core APIs existing prior to + the introduction of microversions. The /v1 URL is used to call + 1.0 APIs, and microversions headers sent to this endpoint are + ignored. diff --git a/meteos/api/openstack/urlmap.py b/meteos/api/openstack/urlmap.py new file mode 100644 index 0000000..8ff1dfe --- /dev/null +++ b/meteos/api/openstack/urlmap.py @@ -0,0 +1,29 @@ +# Copyright (c) 2013 OpenStack, LLC. +# +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +from oslo_log import log + +from meteos.api import urlmap +from meteos.i18n import _LW + +LOG = log.getLogger(__name__) + + +def urlmap_factory(loader, global_conf, **local_conf): + LOG.warning(_LW('meteos.api.openstack.urlmap:urlmap_factory ' + 'is deprecated. ' + 'Please use meteos.api.urlmap:urlmap_factory instead.')) + urlmap.urlmap_factory(loader, global_conf, **local_conf) diff --git a/meteos/api/openstack/versioned_method.py b/meteos/api/openstack/versioned_method.py new file mode 100644 index 0000000..28ab516 --- /dev/null +++ b/meteos/api/openstack/versioned_method.py @@ -0,0 +1,49 @@ +# Copyright 2014 IBM Corp. +# Copyright 2015 Clinton Knight +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +from meteos import utils + + +class VersionedMethod(utils.ComparableMixin): + + def __init__(self, name, start_version, end_version, experimental, func): + """Versioning information for a single method. + + Minimum and maximums are inclusive. + + :param name: Name of the method + :param start_version: Minimum acceptable version + :param end_version: Maximum acceptable_version + :param experimental: True if method is experimental + :param func: Method to call + """ + self.name = name + self.start_version = start_version + self.end_version = end_version + self.experimental = experimental + self.func = func + + def __str__(self): + args = { + 'name': self.name, + 'start': self.start_version, + 'end': self.end_version + } + return ("Version Method %(name)s: min: %(start)s, max: %(end)s" % args) + + def _cmpkey(self): + """Return the value used by ComparableMixin for rich comparisons.""" + return self.start_version diff --git a/meteos/api/openstack/wsgi.py b/meteos/api/openstack/wsgi.py new file mode 100644 index 0000000..de1a001 --- /dev/null +++ b/meteos/api/openstack/wsgi.py @@ -0,0 +1,1343 @@ +# Copyright 2011 OpenStack LLC. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +import functools +import inspect +import math +import time + +from oslo_log import log +from oslo_serialization import jsonutils +from oslo_utils import strutils +import six +import webob +import webob.exc + +from meteos.api.openstack import api_version_request as api_version +from meteos.api.openstack import versioned_method +from meteos.common import constants +from meteos import exception +from meteos.i18n import _, _LE, _LI +from meteos import policy +from meteos import wsgi + +LOG = log.getLogger(__name__) + +SUPPORTED_CONTENT_TYPES = ( + 'application/json', +) + +_MEDIA_TYPE_MAP = { + 'application/json': 'json', +} + +# name of attribute to keep version method information +VER_METHOD_ATTR = 'versioned_methods' + +# Name of header used by clients to request a specific version +# of the REST API +API_VERSION_REQUEST_HEADER = 'X-OpenStack-Meteos-API-Version' +EXPERIMENTAL_API_REQUEST_HEADER = 'X-OpenStack-Meteos-API-Experimental' + +V1_SCRIPT_NAME = '/v1' + + +class Request(webob.Request): + """Add some OpenStack API-specific logic to the base webob.Request.""" + + def __init__(self, *args, **kwargs): + super(Request, self).__init__(*args, **kwargs) + self._resource_cache = {} + if not hasattr(self, 'api_version_request'): + self.api_version_request = api_version.APIVersionRequest() + + def cache_resource(self, resource_to_cache, id_attribute='id', name=None): + """Cache the given resource. + + Allow API methods to cache objects, such as results from a DB query, + to be used by API extensions within the same API request. + + The resource_to_cache can be a list or an individual resource, + but ultimately resources are cached individually using the given + id_attribute. + + Different resources types might need to be cached during the same + request, they can be cached using the name parameter. For example: + + Controller 1: + request.cache_resource(db_volumes, 'volumes') + request.cache_resource(db_volume_types, 'types') + Controller 2: + db_volumes = request.cached_resource('volumes') + db_type_1 = request.cached_resource_by_id('1', 'types') + + If no name is given, a default name will be used for the resource. + + An instance of this class only lives for the lifetime of a + single API request, so there's no need to implement full + cache management. + """ + if not isinstance(resource_to_cache, list): + resource_to_cache = [resource_to_cache] + if not name: + name = self.path + cached_resources = self._resource_cache.setdefault(name, {}) + for resource in resource_to_cache: + cached_resources[resource[id_attribute]] = resource + + def cached_resource(self, name=None): + """Get the cached resources cached under the given resource name. + + Allow an API extension to get previously stored objects within + the same API request. + + Note that the object data will be slightly stale. + + :returns: a dict of id_attribute to the resource from the cached + resources, an empty map if an empty collection was cached, + or None if nothing has been cached yet under this name + """ + if not name: + name = self.path + if name not in self._resource_cache: + # Nothing has been cached for this key yet + return None + return self._resource_cache[name] + + def cached_resource_by_id(self, resource_id, name=None): + """Get a resource by ID cached under the given resource name. + + Allow an API extension to get a previously stored object + within the same API request. This is basically a convenience method + to lookup by ID on the dictionary of all cached resources. + + Note that the object data will be slightly stale. + + :returns: the cached resource or None if the item is not in the cache + """ + resources = self.cached_resource(name) + if not resources: + # Nothing has been cached yet for this key yet + return None + return resources.get(resource_id) + + def cache_db_items(self, key, items, item_key='id'): + """Cache db items. + + Allow API methods to store objects from a DB query to be + used by API extensions within the same API request. + An instance of this class only lives for the lifetime of a + single API request, so there's no need to implement full + cache management. + """ + self.cache_resource(items, item_key, key) + + def get_db_items(self, key): + """Get db item by key. + + Allow an API extension to get previously stored objects within + the same API request. + Note that the object data will be slightly stale. + """ + return self.cached_resource(key) + + def get_db_item(self, key, item_key): + """Get db item by key and item key. + + Allow an API extension to get a previously stored object + within the same API request. + Note that the object data will be slightly stale. + """ + return self.get_db_items(key).get(item_key) + + def cache_db_learning_types(self, learning_types): + self.cache_db_items('learning_types', learning_types, 'id') + + def cache_db_learning_type(self, learning_type): + self.cache_db_items('learning_types', [learning_type], 'id') + + def get_db_learning_types(self): + return self.get_db_items('learning_types') + + def get_db_learning_type(self, learning_type_id): + return self.get_db_item('learning_types', learning_type_id) + + def best_match_content_type(self): + """Determine the requested response content-type.""" + if 'meteos.best_content_type' not in self.environ: + # Calculate the best MIME type + content_type = None + + # Check URL path suffix + parts = self.path.rsplit('.', 1) + if len(parts) > 1: + possible_type = 'application/' + parts[1] + if possible_type in SUPPORTED_CONTENT_TYPES: + content_type = possible_type + + if not content_type: + content_type = self.accept.best_match(SUPPORTED_CONTENT_TYPES) + + self.environ['meteos.best_content_type'] = (content_type or + 'application/json') + + return self.environ['meteos.best_content_type'] + + def get_content_type(self): + """Determine content type of the request body. + + Does not do any body introspection, only checks header. + """ + if "Content-Type" not in self.headers: + return None + + allowed_types = SUPPORTED_CONTENT_TYPES + content_type = self.content_type + + if content_type not in allowed_types: + raise exception.InvalidContentType(content_type=content_type) + + return content_type + + def set_api_version_request(self): + """Set API version request based on the request header information. + + Microversions starts with /v2, so if a client sends a /v1 URL, then + ignore the headers and request 1.0 APIs. + """ + + if not self.script_name: + self.api_version_request = api_version.APIVersionRequest() + elif self.script_name == V1_SCRIPT_NAME: + self.api_version_request = api_version.APIVersionRequest('1.0') + else: + if API_VERSION_REQUEST_HEADER in self.headers: + hdr_string = self.headers[API_VERSION_REQUEST_HEADER] + self.api_version_request = api_version.APIVersionRequest( + hdr_string) + + # Check that the version requested is within the global + # minimum/maximum of supported API versions + if not self.api_version_request.matches( + api_version.min_api_version(), + api_version.max_api_version()): + raise exception.InvalidGlobalAPIVersion( + req_ver=self.api_version_request.get_string(), + min_ver=api_version.min_api_version().get_string(), + max_ver=api_version.max_api_version().get_string()) + + else: + self.api_version_request = api_version.APIVersionRequest( + api_version.DEFAULT_API_VERSION) + + # Check if experimental API was requested + if EXPERIMENTAL_API_REQUEST_HEADER in self.headers: + self.api_version_request.experimental = strutils.bool_from_string( + self.headers[EXPERIMENTAL_API_REQUEST_HEADER]) + + +class ActionDispatcher(object): + """Maps method name to local methods through action name.""" + + def dispatch(self, *args, **kwargs): + """Find and call local method.""" + action = kwargs.pop('action', 'default') + action_method = getattr(self, six.text_type(action), self.default) + return action_method(*args, **kwargs) + + def default(self, data): + raise NotImplementedError() + + +class TextDeserializer(ActionDispatcher): + """Default request body deserialization.""" + + def deserialize(self, datastring, action='default'): + return self.dispatch(datastring, action=action) + + def default(self, datastring): + return {} + + +class JSONDeserializer(TextDeserializer): + + def _from_json(self, datastring): + try: + return jsonutils.loads(datastring) + except ValueError: + msg = _("cannot understand JSON") + raise exception.MalformedRequestBody(reason=msg) + + def default(self, datastring): + return {'body': self._from_json(datastring)} + + +class DictSerializer(ActionDispatcher): + """Default request body serialization.""" + + def serialize(self, data, action='default'): + return self.dispatch(data, action=action) + + def default(self, data): + return "" + + +class JSONDictSerializer(DictSerializer): + """Default JSON request body serialization.""" + + def default(self, data): + return six.b(jsonutils.dumps(data)) + + +def serializers(**serializers): + """Attaches serializers to a method. + + This decorator associates a dictionary of serializers with a + method. Note that the function attributes are directly + manipulated; the method is not wrapped. + """ + + def decorator(func): + if not hasattr(func, 'wsgi_serializers'): + func.wsgi_serializers = {} + func.wsgi_serializers.update(serializers) + return func + return decorator + + +def deserializers(**deserializers): + """Attaches deserializers to a method. + + This decorator associates a dictionary of deserializers with a + method. Note that the function attributes are directly + manipulated; the method is not wrapped. + """ + + def decorator(func): + if not hasattr(func, 'wsgi_deserializers'): + func.wsgi_deserializers = {} + func.wsgi_deserializers.update(deserializers) + return func + return decorator + + +def response(code): + """Attaches response code to a method. + + This decorator associates a response code with a method. Note + that the function attributes are directly manipulated; the method + is not wrapped. + """ + + def decorator(func): + func.wsgi_code = code + return func + return decorator + + +class ResponseObject(object): + """Bundles a response object with appropriate serializers. + + Object that app methods may return in order to bind alternate + serializers with a response object to be serialized. Its use is + optional. + """ + + def __init__(self, obj, code=None, headers=None, **serializers): + """Binds serializers with an object. + + Takes keyword arguments akin to the @serializer() decorator + for specifying serializers. Serializers specified will be + given preference over default serializers or method-specific + serializers on return. + """ + + self.obj = obj + self.serializers = serializers + self._default_code = 200 + self._code = code + self._headers = headers or {} + self.serializer = None + self.media_type = None + + def __getitem__(self, key): + """Retrieves a header with the given name.""" + + return self._headers[key.lower()] + + def __setitem__(self, key, value): + """Sets a header with the given name to the given value.""" + + self._headers[key.lower()] = value + + def __delitem__(self, key): + """Deletes the header with the given name.""" + + del self._headers[key.lower()] + + def _bind_method_serializers(self, meth_serializers): + """Binds method serializers with the response object. + + Binds the method serializers with the response object. + Serializers specified to the constructor will take precedence + over serializers specified to this method. + + :param meth_serializers: A dictionary with keys mapping to + response types and values containing + serializer objects. + """ + + # We can't use update because that would be the wrong + # precedence + for mtype, serializer in meth_serializers.items(): + self.serializers.setdefault(mtype, serializer) + + def get_serializer(self, content_type, default_serializers=None): + """Returns the serializer for the wrapped object. + + Returns the serializer for the wrapped object subject to the + indicated content type. If no serializer matching the content + type is attached, an appropriate serializer drawn from the + default serializers will be used. If no appropriate + serializer is available, raises InvalidContentType. + """ + + default_serializers = default_serializers or {} + + try: + mtype = _MEDIA_TYPE_MAP.get(content_type, content_type) + if mtype in self.serializers: + return mtype, self.serializers[mtype] + else: + return mtype, default_serializers[mtype] + except (KeyError, TypeError): + raise exception.InvalidContentType(content_type=content_type) + + def preserialize(self, content_type, default_serializers=None): + """Prepares the serializer that will be used to serialize. + + Determines the serializer that will be used and prepares an + instance of it for later call. This allows the serializer to + be accessed by extensions for, e.g., template extension. + """ + + mtype, serializer = self.get_serializer(content_type, + default_serializers) + self.media_type = mtype + self.serializer = serializer() + + def attach(self, **kwargs): + """Attach slave templates to serializers.""" + + if self.media_type in kwargs: + self.serializer.attach(kwargs[self.media_type]) + + def serialize(self, request, content_type, default_serializers=None): + """Serializes the wrapped object. + + Utility method for serializing the wrapped object. Returns a + webob.Response object. + """ + + if self.serializer: + serializer = self.serializer + else: + _mtype, _serializer = self.get_serializer(content_type, + default_serializers) + serializer = _serializer() + + response = webob.Response() + response.status_int = self.code + for hdr, value in self._headers.items(): + response.headers[hdr] = six.text_type(value) + response.headers['Content-Type'] = six.text_type(content_type) + if self.obj is not None: + response.body = serializer.serialize(self.obj) + + return response + + @property + def code(self): + """Retrieve the response status.""" + + return self._code or self._default_code + + @property + def headers(self): + """Retrieve the headers.""" + + return self._headers.copy() + + +def action_peek_json(body): + """Determine action to invoke.""" + + try: + decoded = jsonutils.loads(body) + except ValueError: + msg = _("cannot understand JSON") + raise exception.MalformedRequestBody(reason=msg) + + # Make sure there's exactly one key... + if len(decoded) != 1: + msg = _("too many body keys") + raise exception.MalformedRequestBody(reason=msg) + + # Return the action and the decoded body... + return list(decoded.keys())[0] + + +class ResourceExceptionHandler(object): + """Context manager to handle Resource exceptions. + + Used when processing exceptions generated by API implementation + methods (or their extensions). Converts most exceptions to Fault + exceptions, with the appropriate logging. + """ + + def __enter__(self): + return None + + def __exit__(self, ex_type, ex_value, ex_traceback): + if not ex_value: + return True + + if isinstance(ex_value, exception.NotAuthorized): + msg = six.text_type(ex_value) + raise Fault(webob.exc.HTTPForbidden(explanation=msg)) + elif isinstance(ex_value, exception.VersionNotFoundForAPIMethod): + raise + elif isinstance(ex_value, exception.Invalid): + raise Fault(exception.ConvertedException( + code=ex_value.code, explanation=six.text_type(ex_value))) + elif isinstance(ex_value, TypeError): + exc_info = (ex_type, ex_value, ex_traceback) + LOG.error(_LE('Exception handling resource: %s'), + ex_value, exc_info=exc_info) + raise Fault(webob.exc.HTTPBadRequest()) + elif isinstance(ex_value, Fault): + LOG.info(_LI("Fault thrown: %s"), six.text_type(ex_value)) + raise ex_value + elif isinstance(ex_value, webob.exc.HTTPException): + LOG.info(_LI("HTTP exception thrown: %s"), six.text_type(ex_value)) + raise Fault(ex_value) + + # We didn't handle the exception + return False + + +class Resource(wsgi.Application): + """WSGI app that handles (de)serialization and controller dispatch. + + WSGI app that reads routing information supplied by RoutesMiddleware + and calls the requested action method upon its controller. All + controller action methods must accept a 'req' argument, which is the + incoming wsgi.Request. If the operation is a PUT or POST, the controller + method must also accept a 'body' argument (the deserialized request body). + They may raise a webob.exc exception or return a dict, which will be + serialized by requested content type. + + Exceptions derived from webob.exc.HTTPException will be automatically + wrapped in Fault() to provide API friendly error responses. + """ + support_api_request_version = True + + def __init__(self, controller, action_peek=None, **deserializers): + """init method of Resource. + + :param controller: object that implement methods created by routes lib + :param action_peek: dictionary of routines for peeking into an action + request body to determine the desired action + """ + + self.controller = controller + + default_deserializers = dict(json=JSONDeserializer) + default_deserializers.update(deserializers) + + self.default_deserializers = default_deserializers + self.default_serializers = dict(json=JSONDictSerializer) + + self.action_peek = dict(json=action_peek_json) + self.action_peek.update(action_peek or {}) + + # Copy over the actions dictionary + self.wsgi_actions = {} + if controller: + self.register_actions(controller) + + # Save a mapping of extensions + self.wsgi_extensions = {} + self.wsgi_action_extensions = {} + + def register_actions(self, controller): + """Registers controller actions with this resource.""" + + actions = getattr(controller, 'wsgi_actions', {}) + for key, method_name in actions.items(): + self.wsgi_actions[key] = getattr(controller, method_name) + + def register_extensions(self, controller): + """Registers controller extensions with this resource.""" + + extensions = getattr(controller, 'wsgi_extensions', []) + for method_name, action_name in extensions: + # Look up the extending method + extension = getattr(controller, method_name) + + if action_name: + # Extending an action... + if action_name not in self.wsgi_action_extensions: + self.wsgi_action_extensions[action_name] = [] + self.wsgi_action_extensions[action_name].append(extension) + else: + # Extending a regular method + if method_name not in self.wsgi_extensions: + self.wsgi_extensions[method_name] = [] + self.wsgi_extensions[method_name].append(extension) + + def get_action_args(self, request_environment): + """Parse dictionary created by routes library.""" + + # NOTE(Vek): Check for get_action_args() override in the + # controller + if hasattr(self.controller, 'get_action_args'): + return self.controller.get_action_args(request_environment) + + try: + args = request_environment['wsgiorg.routing_args'][1].copy() + except (KeyError, IndexError, AttributeError): + return {} + + try: + del args['controller'] + except KeyError: + pass + + try: + del args['format'] + except KeyError: + pass + + return args + + def get_body(self, request): + try: + content_type = request.get_content_type() + except exception.InvalidContentType: + LOG.debug("Unrecognized Content-Type provided in request") + return None, '' + + if not content_type: + LOG.debug("No Content-Type provided in request") + return None, '' + + if len(request.body) <= 0: + LOG.debug("Empty body provided in request") + return None, '' + + return content_type, request.body + + def deserialize(self, meth, content_type, body): + meth_deserializers = getattr(meth, 'wsgi_deserializers', {}) + try: + mtype = _MEDIA_TYPE_MAP.get(content_type, content_type) + if mtype in meth_deserializers: + deserializer = meth_deserializers[mtype] + else: + deserializer = self.default_deserializers[mtype] + except (KeyError, TypeError): + raise exception.InvalidContentType(content_type=content_type) + + return deserializer().deserialize(body) + + def pre_process_extensions(self, extensions, request, action_args): + # List of callables for post-processing extensions + post = [] + + for ext in extensions: + if inspect.isgeneratorfunction(ext): + response = None + + # If it's a generator function, the part before the + # yield is the preprocessing stage + try: + with ResourceExceptionHandler(): + gen = ext(req=request, **action_args) + response = next(gen) + except Fault as ex: + response = ex + + # We had a response... + if response: + return response, [] + + # No response, queue up generator for post-processing + post.append(gen) + else: + # Regular functions only perform post-processing + post.append(ext) + + # Run post-processing in the reverse order + return None, reversed(post) + + def post_process_extensions(self, extensions, resp_obj, request, + action_args): + for ext in extensions: + response = None + if inspect.isgenerator(ext): + # If it's a generator, run the second half of + # processing + try: + with ResourceExceptionHandler(): + response = ext.send(resp_obj) + except StopIteration: + # Normal exit of generator + continue + except Fault as ex: + response = ex + else: + # Regular functions get post-processing... + try: + with ResourceExceptionHandler(): + response = ext(req=request, resp_obj=resp_obj, + **action_args) + except exception.VersionNotFoundForAPIMethod: + # If an attached extension (@wsgi.extends) for the + # method has no version match its not an error. We + # just don't run the extends code + continue + except Fault as ex: + response = ex + + # We had a response... + if response: + return response + + return None + + @webob.dec.wsgify(RequestClass=Request) + def __call__(self, request): + """WSGI method that controls (de)serialization and method dispatch.""" + + LOG.info(_LI("%(method)s %(url)s") % {"method": request.method, + "url": request.url}) + if self.support_api_request_version: + # Set the version of the API requested based on the header + try: + request.set_api_version_request() + except exception.InvalidAPIVersionString as e: + return Fault(webob.exc.HTTPBadRequest( + explanation=six.text_type(e))) + except exception.InvalidGlobalAPIVersion as e: + return Fault(webob.exc.HTTPNotAcceptable( + explanation=six.text_type(e))) + + # Identify the action, its arguments, and the requested + # content type + action_args = self.get_action_args(request.environ) + action = action_args.pop('action', None) + content_type, body = self.get_body(request) + accept = request.best_match_content_type() + + # NOTE(Vek): Splitting the function up this way allows for + # auditing by external tools that wrap the existing + # function. If we try to audit __call__(), we can + # run into troubles due to the @webob.dec.wsgify() + # decorator. + return self._process_stack(request, action, action_args, + content_type, body, accept) + + def _process_stack(self, request, action, action_args, + content_type, body, accept): + """Implement the processing stack.""" + + # Get the implementing method + try: + meth, extensions = self.get_method(request, action, + content_type, body) + except (AttributeError, TypeError): + return Fault(webob.exc.HTTPNotFound()) + except KeyError as ex: + msg = _("There is no such action: %s") % ex.args[0] + return Fault(webob.exc.HTTPBadRequest(explanation=msg)) + except exception.MalformedRequestBody: + msg = _("Malformed request body") + return Fault(webob.exc.HTTPBadRequest(explanation=msg)) + + if body: + msg = ("Action: '%(action)s', calling method: %(meth)s, body: " + "%(body)s") % {'action': action, + 'body': six.text_type(body), + 'meth': six.text_type(meth)} + LOG.debug(strutils.mask_password(msg)) + else: + LOG.debug("Calling method '%(meth)s'", + {'meth': six.text_type(meth)}) + + # Now, deserialize the request body... + try: + if content_type: + contents = self.deserialize(meth, content_type, body) + else: + contents = {} + except exception.InvalidContentType: + msg = _("Unsupported Content-Type") + return Fault(webob.exc.HTTPBadRequest(explanation=msg)) + except exception.MalformedRequestBody: + msg = _("Malformed request body") + return Fault(webob.exc.HTTPBadRequest(explanation=msg)) + + # Update the action args + action_args.update(contents) + + project_id = action_args.pop("project_id", None) + context = request.environ.get('meteos.context') + if (context and project_id and (project_id != context.project_id)): + msg = _("Malformed request url") + return Fault(webob.exc.HTTPBadRequest(explanation=msg)) + + # Run pre-processing extensions + response, post = self.pre_process_extensions(extensions, + request, action_args) + + if not response: + try: + with ResourceExceptionHandler(): + action_result = self.dispatch(meth, request, action_args) + except Fault as ex: + response = ex + + if not response: + # No exceptions; convert action_result into a + # ResponseObject + resp_obj = None + if type(action_result) is dict or action_result is None: + resp_obj = ResponseObject(action_result) + elif isinstance(action_result, ResponseObject): + resp_obj = action_result + else: + response = action_result + + # Run post-processing extensions + if resp_obj: + _set_request_id_header(request, resp_obj) + # Do a preserialize to set up the response object + serializers = getattr(meth, 'wsgi_serializers', {}) + resp_obj._bind_method_serializers(serializers) + if hasattr(meth, 'wsgi_code'): + resp_obj._default_code = meth.wsgi_code + resp_obj.preserialize(accept, self.default_serializers) + + # Process post-processing extensions + response = self.post_process_extensions(post, resp_obj, + request, action_args) + + if resp_obj and not response: + response = resp_obj.serialize(request, accept, + self.default_serializers) + + try: + msg_dict = dict(url=request.url, status=response.status_int) + msg = _("%(url)s returned with HTTP %(status)d") % msg_dict + except AttributeError as e: + msg_dict = dict(url=request.url, e=e) + msg = _("%(url)s returned a fault: %(e)s") % msg_dict + + LOG.info(msg) + + if hasattr(response, 'headers'): + for hdr, val in response.headers.items(): + # Headers must be utf-8 strings + response.headers[hdr] = six.text_type(val) + + if not request.api_version_request.is_null(): + response.headers[API_VERSION_REQUEST_HEADER] = ( + request.api_version_request.get_string()) + if request.api_version_request.experimental: + response.headers[EXPERIMENTAL_API_REQUEST_HEADER] = ( + request.api_version_request.experimental) + response.headers['Vary'] = API_VERSION_REQUEST_HEADER + + return response + + def get_method(self, request, action, content_type, body): + """Look up the action-specific method and its extensions.""" + + # Look up the method + try: + if not self.controller: + meth = getattr(self, action) + else: + meth = getattr(self.controller, action) + except AttributeError: + if (not self.wsgi_actions or + action not in ['action', 'create', 'delete']): + # Propagate the error + raise + else: + return meth, self.wsgi_extensions.get(action, []) + + if action == 'action': + # OK, it's an action; figure out which action... + mtype = _MEDIA_TYPE_MAP.get(content_type) + action_name = self.action_peek[mtype](body) + LOG.debug("Action body: %s" % body) + else: + action_name = action + + # Look up the action method + return (self.wsgi_actions[action_name], + self.wsgi_action_extensions.get(action_name, [])) + + def dispatch(self, method, request, action_args): + """Dispatch a call to the action-specific method.""" + + try: + return method(req=request, **action_args) + except exception.VersionNotFoundForAPIMethod: + # We deliberately don't return any message information + # about the exception to the user so it looks as if + # the method is simply not implemented. + return Fault(webob.exc.HTTPNotFound()) + + +def action(name): + """Mark a function as an action. + + The given name will be taken as the action key in the body. + + This is also overloaded to allow extensions to provide + non-extending definitions of create and delete operations. + """ + + def decorator(func): + func.wsgi_action = name + return func + return decorator + + +def extends(*args, **kwargs): + """Indicate a function extends an operation. + + Can be used as either:: + + @extends + def index(...): + pass + + or as:: + + @extends(action='resize') + def _action_resize(...): + pass + """ + + def decorator(func): + # Store enough information to find what we're extending + func.wsgi_extends = (func.__name__, kwargs.get('action')) + return func + + # If we have positional arguments, call the decorator + if args: + return decorator(*args) + + # OK, return the decorator instead + return decorator + + +class ControllerMetaclass(type): + """Controller metaclass. + + This metaclass automates the task of assembling a dictionary + mapping action keys to method names. + """ + + def __new__(mcs, name, bases, cls_dict): + """Adds the wsgi_actions dictionary to the class.""" + + # Find all actions + actions = {} + extensions = [] + versioned_methods = None + # start with wsgi actions from base classes + for base in bases: + actions.update(getattr(base, 'wsgi_actions', {})) + + if base.__name__ == "Controller": + # NOTE(cyeoh): This resets the VER_METHOD_ATTR attribute + # between API controller class creations. This allows us + # to use a class decorator on the API methods that doesn't + # require naming explicitly what method is being versioned as + # it can be implicit based on the method decorated. It is a bit + # ugly. + if VER_METHOD_ATTR in base.__dict__: + versioned_methods = getattr(base, VER_METHOD_ATTR) + delattr(base, VER_METHOD_ATTR) + + for key, value in cls_dict.items(): + if not callable(value): + continue + if getattr(value, 'wsgi_action', None): + actions[value.wsgi_action] = key + elif getattr(value, 'wsgi_extends', None): + extensions.append(value.wsgi_extends) + + # Add the actions and extensions to the class dict + cls_dict['wsgi_actions'] = actions + cls_dict['wsgi_extensions'] = extensions + if versioned_methods: + cls_dict[VER_METHOD_ATTR] = versioned_methods + + return super(ControllerMetaclass, mcs).__new__(mcs, name, bases, + cls_dict) + + +@six.add_metaclass(ControllerMetaclass) +class Controller(object): + """Default controller.""" + + _view_builder_class = None + + def __init__(self, view_builder=None): + """Initialize controller with a view builder instance.""" + if view_builder: + self._view_builder = view_builder + elif self._view_builder_class: + self._view_builder = self._view_builder_class() + else: + self._view_builder = None + + def __getattribute__(self, key): + + def version_select(*args, **kwargs): + """Select and call the matching version of the specified method. + + Look for the method which matches the name supplied and version + constraints and calls it with the supplied arguments. + + :returns: Returns the result of the method called + :raises: VersionNotFoundForAPIMethod if there is no method which + matches the name and version constraints + """ + + # The first arg to all versioned methods is always the request + # object. The version for the request is attached to the + # request object + if len(args) == 0: + version_request = kwargs['req'].api_version_request + else: + version_request = args[0].api_version_request + + func_list = self.versioned_methods[key] + for func in func_list: + if version_request.matches_versioned_method(func): + # Update the version_select wrapper function so + # other decorator attributes like wsgi.response + # are still respected. + functools.update_wrapper(version_select, func.func) + return func.func(self, *args, **kwargs) + + # No version match + raise exception.VersionNotFoundForAPIMethod( + version=version_request) + + try: + version_meth_dict = object.__getattribute__(self, VER_METHOD_ATTR) + except AttributeError: + # No versioning on this class + return object.__getattribute__(self, key) + + if (version_meth_dict and + key in object.__getattribute__(self, VER_METHOD_ATTR)): + return version_select + + return object.__getattribute__(self, key) + + # NOTE(cyeoh): This decorator MUST appear first (the outermost + # decorator) on an API method for it to work correctly + @classmethod + def api_version(cls, min_ver, max_ver=None, experimental=False): + """Decorator for versioning API methods. + + Add the decorator to any method which takes a request object + as the first parameter and belongs to a class which inherits from + wsgi.Controller. + + :param min_ver: string representing minimum version + :param max_ver: optional string representing maximum version + :param experimental: flag indicating an API is experimental and is + subject to change or removal at any time + """ + + def decorator(f): + obj_min_ver = api_version.APIVersionRequest(min_ver) + if max_ver: + obj_max_ver = api_version.APIVersionRequest(max_ver) + else: + obj_max_ver = api_version.APIVersionRequest() + + # Add to list of versioned methods registered + func_name = f.__name__ + new_func = versioned_method.VersionedMethod( + func_name, obj_min_ver, obj_max_ver, experimental, f) + + func_dict = getattr(cls, VER_METHOD_ATTR, {}) + if not func_dict: + setattr(cls, VER_METHOD_ATTR, func_dict) + + func_list = func_dict.get(func_name, []) + if not func_list: + func_dict[func_name] = func_list + func_list.append(new_func) + # Ensure the list is sorted by minimum version (reversed) + # so later when we work through the list in order we find + # the method which has the latest version which supports + # the version requested. + # TODO(cyeoh): Add check to ensure that there are no overlapping + # ranges of valid versions as that is ambiguous + func_list.sort(reverse=True) + + return f + + return decorator + + @staticmethod + def authorize(arg): + """Decorator for checking the policy on API methods. + + Add this decorator to any API method which takes a request object + as the first parameter and belongs to a class which inherits from + wsgi.Controller. The class must also have a class member called + 'resource_name' which specifies the resource for the policy check. + + Can be used in any of the following forms + @authorize + @authorize('my_action_name') + + :param arg: Can either be the function being decorated or a str + containing the 'action' for the policy check. If no action name is + provided, the function name is assumed to be the action name. + """ + action_name = None + + def decorator(f): + @functools.wraps(f) + def wrapper(self, req, *args, **kwargs): + action = action_name or f.__name__ + context = req.environ['meteos.context'] + try: + policy.check_policy(context, self.resource_name, action) + except exception.PolicyNotAuthorized: + raise webob.exc.HTTPForbidden() + return f(self, req, *args, **kwargs) + return wrapper + + if callable(arg): + return decorator(arg) + else: + action_name = arg + return decorator + + @staticmethod + def is_valid_body(body, entity_name): + if not (body and entity_name in body): + return False + + def is_dict(d): + try: + d.get(None) + return True + except AttributeError: + return False + + if not is_dict(body[entity_name]): + return False + + return True + + +class AdminActionsMixin(object): + """Mixin class for API controllers with admin actions.""" + + body_attributes = { + 'status': 'reset_status', + 'replica_state': 'reset_replica_state', + 'task_state': 'reset_task_state', + } + + valid_statuses = { + 'status': set([ + constants.STATUS_CREATING, + constants.STATUS_AVAILABLE, + constants.STATUS_DELETING, + constants.STATUS_ERROR, + constants.STATUS_ERROR_DELETING, + ]), + } + + def _update(self, *args, **kwargs): + raise NotImplementedError() + + def _get(self, *args, **kwargs): + raise NotImplementedError() + + def _delete(self, *args, **kwargs): + raise NotImplementedError() + + def validate_update(self, body, status_attr='status'): + update = {} + try: + update[status_attr] = body[status_attr] + except (TypeError, KeyError): + msg = _("Must specify '%s'") % status_attr + raise webob.exc.HTTPBadRequest(explanation=msg) + if update[status_attr] not in self.valid_statuses[status_attr]: + expl = (_("Invalid state. Valid states: %s.") % + ", ".join(six.text_type(i) for i in + self.valid_statuses[status_attr])) + raise webob.exc.HTTPBadRequest(explanation=expl) + return update + + @Controller.authorize('reset_status') + def _reset_status(self, req, id, body, status_attr='status'): + """Reset the status_attr specified on the resource.""" + context = req.environ['meteos.context'] + body_attr = self.body_attributes[status_attr] + update = self.validate_update( + body.get(body_attr, body.get('-'.join(('os', body_attr)))), + status_attr=status_attr) + msg = "Updating %(resource)s '%(id)s' with '%(update)r'" + LOG.debug(msg, {'resource': self.resource_name, 'id': id, + 'update': update}) + try: + self._update(context, id, update) + except exception.NotFound as e: + raise webob.exc.HTTPNotFound(six.text_type(e)) + return webob.Response(status_int=202) + + @Controller.authorize('force_delete') + def _force_delete(self, req, id, body): + """Delete a resource, bypassing the check for status.""" + context = req.environ['meteos.context'] + try: + resource = self._get(context, id) + except exception.NotFound as e: + raise webob.exc.HTTPNotFound(six.text_type(e)) + self._delete(context, resource, force=True) + return webob.Response(status_int=202) + + +class Fault(webob.exc.HTTPException): + """Wrap webob.exc.HTTPException to provide API friendly response.""" + + _fault_names = {400: "badRequest", + 401: "unauthorized", + 403: "forbidden", + 404: "itemNotFound", + 405: "badMethod", + 409: "conflictingRequest", + 413: "overLimit", + 415: "badMediaType", + 501: "notImplemented", + 503: "serviceUnavailable"} + + def __init__(self, exception): + """Create a Fault for the given webob.exc.exception.""" + self.wrapped_exc = exception + self.status_int = exception.status_int + + @webob.dec.wsgify(RequestClass=Request) + def __call__(self, req): + """Generate a WSGI response based on the exception passed to ctor.""" + # Replace the body with fault details. + code = self.wrapped_exc.status_int + fault_name = self._fault_names.get(code, "computeFault") + fault_data = { + fault_name: { + 'code': code, + 'message': self.wrapped_exc.explanation}} + if code == 413: + retry = self.wrapped_exc.headers['Retry-After'] + fault_data[fault_name]['retryAfter'] = retry + + if not req.api_version_request.is_null(): + self.wrapped_exc.headers[API_VERSION_REQUEST_HEADER] = ( + req.api_version_request.get_string()) + if req.api_version_request.experimental: + self.wrapped_exc.headers[EXPERIMENTAL_API_REQUEST_HEADER] = ( + req.api_version_request.experimental) + self.wrapped_exc.headers['Vary'] = API_VERSION_REQUEST_HEADER + + content_type = req.best_match_content_type() + serializer = { + 'application/json': JSONDictSerializer(), + }[content_type] + + self.wrapped_exc.body = serializer.serialize(fault_data) + self.wrapped_exc.content_type = content_type + _set_request_id_header(req, self.wrapped_exc.headers) + + return self.wrapped_exc + + def __str__(self): + return self.wrapped_exc.__str__() + + +def _set_request_id_header(req, headers): + context = req.environ.get('meteos.context') + if context: + headers['x-compute-request-id'] = context.request_id + + +class OverLimitFault(webob.exc.HTTPException): + """Rate-limited request response.""" + + def __init__(self, message, details, retry_time): + """Initialize new `OverLimitFault` with relevant information.""" + hdrs = OverLimitFault._retry_after(retry_time) + self.wrapped_exc = webob.exc.HTTPRequestEntityTooLarge(headers=hdrs) + self.content = { + "overLimitFault": { + "code": self.wrapped_exc.status_int, + "message": message, + "details": details, + }, + } + + @staticmethod + def _retry_after(retry_time): + delay = int(math.ceil(retry_time - time.time())) + retry_after = delay if delay > 0 else 0 + headers = {'Retry-After': '%d' % retry_after} + return headers + + @webob.dec.wsgify(RequestClass=Request) + def __call__(self, request): + """Wrap the exception. + + Wrap the exception with a serialized body conforming to our + error format. + """ + content_type = request.best_match_content_type() + + serializer = { + 'application/json': JSONDictSerializer(), + }[content_type] + + content = serializer.serialize(self.content) + self.wrapped_exc.body = content + + return self.wrapped_exc diff --git a/meteos/api/urlmap.py b/meteos/api/urlmap.py new file mode 100644 index 0000000..51595c7 --- /dev/null +++ b/meteos/api/urlmap.py @@ -0,0 +1,291 @@ +# Copyright 2011 OpenStack LLC. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +import re +try: + from urllib.request import parse_http_list # noqa +except ImportError: + from urllib2 import parse_http_list # noqa + +import paste.urlmap + +from meteos.api.openstack import wsgi + + +_quoted_string_re = r'"[^"\\]*(?:\\.[^"\\]*)*"' +_option_header_piece_re = re.compile( + r';\s*([^\s;=]+|%s)\s*' + r'(?:=\s*([^;]+|%s))?\s*' % + (_quoted_string_re, _quoted_string_re)) + + +def unquote_header_value(value): + """Unquotes a header value. + + This does not use the real unquoting but what browsers are actually + using for quoting. + + :param value: the header value to unquote. + """ + if value and value[0] == value[-1] == '"': + # this is not the real unquoting, but fixing this so that the + # RFC is met will result in bugs with internet explorer and + # probably some other browsers as well. IE for example is + # uploading files with "C:\foo\bar.txt" as filename + value = value[1:-1] + return value + + +def parse_list_header(value): + """Parse lists as described by RFC 2068 Section 2. + + In particular, parse comma-separated lists where the elements of + the list may include quoted-strings. A quoted-string could + contain a comma. A non-quoted string could have quotes in the + middle. Quotes are removed automatically after parsing. + + The return value is a standard :class:`list`: + + >>> parse_list_header('token, "quoted value"') + ['token', 'quoted value'] + + :param value: a string with a list header. + :return: :class:`list` + """ + result = [] + for item in parse_http_list(value): + if item[:1] == item[-1:] == '"': + item = unquote_header_value(item[1:-1]) + result.append(item) + return result + + +def parse_options_header(value): + """Parse header into content type and options. + + Parse a ``Content-Type`` like header into a tuple with the content + type and the options: + + >>> parse_options_header('Content-Type: text/html; mimetype=text/html') + ('Content-Type:', {'mimetype': 'text/html'}) + + :param value: the header to parse. + :return: (str, options) + """ + def _tokenize(string): + for match in _option_header_piece_re.finditer(string): + key, value = match.groups() + key = unquote_header_value(key) + if value is not None: + value = unquote_header_value(value) + yield key, value + + if not value: + return '', {} + + parts = _tokenize(';' + value) + name = next(parts)[0] + extra = dict(parts) + return name, extra + + +class Accept(object): + def __init__(self, value): + self._content_types = [parse_options_header(v) for v in + parse_list_header(value)] + + def best_match(self, supported_content_types): + # FIXME: Should we have a more sophisticated matching algorithm that + # takes into account the version as well? + best_quality = -1 + best_content_type = None + best_params = {} + best_match = '*/*' + + for content_type in supported_content_types: + for content_mask, params in self._content_types: + try: + quality = float(params.get('q', 1)) + except ValueError: + continue + + if quality < best_quality: + continue + elif best_quality == quality: + if best_match.count('*') <= content_mask.count('*'): + continue + + if self._match_mask(content_mask, content_type): + best_quality = quality + best_content_type = content_type + best_params = params + best_match = content_mask + + return best_content_type, best_params + + def content_type_params(self, best_content_type): + """Find parameters in Accept header for given content type.""" + for content_type, params in self._content_types: + if best_content_type == content_type: + return params + + return {} + + def _match_mask(self, mask, content_type): + if '*' not in mask: + return content_type == mask + if mask == '*/*': + return True + mask_major = mask[:-2] + content_type_major = content_type.split('/', 1)[0] + return content_type_major == mask_major + + +def urlmap_factory(loader, global_conf, **local_conf): + if 'not_found_app' in local_conf: + not_found_app = local_conf.pop('not_found_app') + else: + not_found_app = global_conf.get('not_found_app') + if not_found_app: + not_found_app = loader.get_app(not_found_app, global_conf=global_conf) + urlmap = URLMap(not_found_app=not_found_app) + for path, app_name in local_conf.items(): + path = paste.urlmap.parse_path_expression(path) + app = loader.get_app(app_name, global_conf=global_conf) + urlmap[path] = app + return urlmap + + +class URLMap(paste.urlmap.URLMap): + def _match(self, host, port, path_info): + """Find longest match for a given URL path.""" + for (domain, app_url), app in self.applications: + if domain and domain != host and domain != host + ':' + port: + continue + if (path_info == app_url or path_info.startswith(app_url + '/')): + return app, app_url + + return None, None + + def _set_script_name(self, app, app_url): + def wrap(environ, start_response): + environ['SCRIPT_NAME'] += app_url + return app(environ, start_response) + + return wrap + + def _munge_path(self, app, path_info, app_url): + def wrap(environ, start_response): + environ['SCRIPT_NAME'] += app_url + environ['PATH_INFO'] = path_info[len(app_url):] + return app(environ, start_response) + + return wrap + + def _path_strategy(self, host, port, path_info): + """Check path suffix for MIME type and path prefix for API version.""" + mime_type = app = app_url = None + + parts = path_info.rsplit('.', 1) + if len(parts) > 1: + possible_type = 'application/' + parts[1] + if possible_type in wsgi.SUPPORTED_CONTENT_TYPES: + mime_type = possible_type + + parts = path_info.split('/') + if len(parts) > 1: + possible_app, possible_app_url = self._match(host, port, path_info) + # Don't use prefix if it ends up matching default + if possible_app and possible_app_url: + app_url = possible_app_url + app = self._munge_path(possible_app, path_info, app_url) + + return mime_type, app, app_url + + def _content_type_strategy(self, host, port, environ): + """Check Content-Type header for API version.""" + app = None + params = parse_options_header(environ.get('CONTENT_TYPE', ''))[1] + if 'version' in params: + app, app_url = self._match(host, port, '/v' + params['version']) + if app: + app = self._set_script_name(app, app_url) + + return app + + def _accept_strategy(self, host, port, environ, supported_content_types): + """Check Accept header for best matching MIME type and API version.""" + accept = Accept(environ.get('HTTP_ACCEPT', '')) + + app = None + + # Find the best match in the Accept header + mime_type, params = accept.best_match(supported_content_types) + if 'version' in params: + app, app_url = self._match(host, port, '/v' + params['version']) + if app: + app = self._set_script_name(app, app_url) + + return mime_type, app + + def __call__(self, environ, start_response): + host = environ.get('HTTP_HOST', environ.get('SERVER_NAME')).lower() + if ':' in host: + host, port = host.split(':', 1) + else: + if environ['wsgi.url_scheme'] == 'http': + port = '80' + else: + port = '443' + + path_info = environ['PATH_INFO'] + path_info = self.normalize_url(path_info, False)[1] + + # The API version is determined in one of three ways: + # 1) URL path prefix (eg /v1.1/tenant/servers/detail) + # 2) Content-Type header (eg application/json;version=1.1) + # 3) Accept header (eg application/json;q=0.8;version=1.1) + + # Meteos supports only application/json as MIME type for the responses. + supported_content_types = list(wsgi.SUPPORTED_CONTENT_TYPES) + + mime_type, app, app_url = self._path_strategy(host, port, path_info) + + if not app: + app = self._content_type_strategy(host, port, environ) + + if not mime_type or not app: + possible_mime_type, possible_app = self._accept_strategy( + host, port, environ, supported_content_types) + if possible_mime_type and not mime_type: + mime_type = possible_mime_type + if possible_app and not app: + app = possible_app + + if not mime_type: + mime_type = 'application/json' + + if not app: + # Didn't match a particular version, probably matches default + app, app_url = self._match(host, port, path_info) + if app: + app = self._munge_path(app, path_info, app_url) + + if app: + environ['meteos.best_content_type'] = mime_type + return app(environ, start_response) + + environ['paste.urlmap_object'] = self + return self.not_found_application(environ, start_response) diff --git a/meteos/api/v1/__init__.py b/meteos/api/v1/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/meteos/api/v1/datasets.py b/meteos/api/v1/datasets.py new file mode 100644 index 0000000..07cf8bc --- /dev/null +++ b/meteos/api/v1/datasets.py @@ -0,0 +1,156 @@ +# Copyright 2013 NetApp +# All Rights Reserved. +# Copyright (c) 2016 NEC Corporation. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +"""The datasets api.""" + +import ast +import re +import string + +from oslo_log import log +from oslo_utils import strutils +from oslo_utils import uuidutils +import six +import webob +from webob import exc + +from meteos.api import common +from meteos.api.openstack import wsgi +from meteos.api.views import datasets as dataset_views +from meteos import exception +from meteos.i18n import _, _LI +from meteos import engine + +LOG = log.getLogger(__name__) + + +class DatasetController(wsgi.Controller, wsgi.AdminActionsMixin): + + """The Datasets API v1 controller for the OpenStack API.""" + resource_name = 'dataset' + _view_builder_class = dataset_views.ViewBuilder + + def __init__(self): + super(self.__class__, self).__init__() + self.engine_api = engine.API() + + def show(self, req, id): + """Return data about the given dataset.""" + context = req.environ['meteos.context'] + + try: + dataset = self.engine_api.get_dataset(context, id) + except exception.NotFound: + raise exc.HTTPNotFound() + + return self._view_builder.detail(req, dataset) + + def delete(self, req, id): + """Delete a dataset.""" + context = req.environ['meteos.context'] + + LOG.info(_LI("Delete dataset with id: %s"), id, context=context) + + try: + self.engine_api.delete_dataset(context, id) + except exception.NotFound: + raise exc.HTTPNotFound() + except exception.InvalidLearning as e: + raise exc.HTTPForbidden(explanation=six.text_type(e)) + + return webob.Response(status_int=202) + + def index(self, req): + """Returns a summary list of datasets.""" + return self._get_datasets(req, is_detail=False) + + def detail(self, req): + """Returns a detailed list of datasets.""" + return self._get_datasets(req, is_detail=True) + + def _get_datasets(self, req, is_detail): + """Returns a list of datasets, transformed through view builder.""" + context = req.environ['meteos.context'] + + search_opts = {} + search_opts.update(req.GET) + + # Remove keys that are not related to dataset attrs + search_opts.pop('limit', None) + search_opts.pop('offset', None) + sort_key = search_opts.pop('sort_key', 'created_at') + sort_dir = search_opts.pop('sort_dir', 'desc') + + datasets = self.engine_api.get_all_datasets( + context, search_opts=search_opts, sort_key=sort_key, + sort_dir=sort_dir) + + limited_list = common.limited(datasets, req) + + if is_detail: + datasets = self._view_builder.detail_list(req, limited_list) + else: + datasets = self._view_builder.summary_list(req, limited_list) + return datasets + + def create(self, req, body): + """Creates a new dataset.""" + context = req.environ['meteos.context'] + + if not self.is_valid_body(body, 'dataset'): + raise exc.HTTPUnprocessableEntity() + + dataset = body['dataset'] + + LOG.debug("Create dataset with request: %s", dataset) + + try: + experiment = self.engine_api.get_experiment( + context, dataset['experiment_id']) + template = self.engine_api.get_template( + context, experiment.template_id) + except exception.NotFound: + raise exc.HTTPNotFound() + + display_name = dataset.get('display_name') + display_description = dataset.get('display_description') + method = dataset.get('method') + experiment_id = dataset.get('experiment_id') + source_dataset_url = dataset.get('source_dataset_url') + params = dataset.get('params') + swift_tenant = dataset.get('swift_tenant') + swift_username = dataset.get('swift_username') + swift_password = dataset.get('swift_password') + + new_dataset = self.engine_api.create_dataset(context, + display_name, + display_description, + method, + source_dataset_url, + params, + template.id, + template.job_template_id, + experiment_id, + experiment.cluster_id, + swift_tenant, + swift_username, + swift_password) + + return self._view_builder.detail(req, new_dataset) + + +def create_resource(): + return wsgi.Resource(DatasetController()) diff --git a/meteos/api/v1/experiments.py b/meteos/api/v1/experiments.py new file mode 100644 index 0000000..0cd6958 --- /dev/null +++ b/meteos/api/v1/experiments.py @@ -0,0 +1,144 @@ +# Copyright 2013 NetApp +# All Rights Reserved. +# Copyright (c) 2016 NEC Corporation. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +"""The experiments api.""" + +import ast +import re +import string + +from oslo_log import log +from oslo_utils import strutils +from oslo_utils import uuidutils +import six +import webob +from webob import exc + +from meteos.api import common +from meteos.api.openstack import wsgi +from meteos.api.views import experiments as experiment_views +from meteos import exception +from meteos.i18n import _, _LI +from meteos import engine + +LOG = log.getLogger(__name__) + + +class ExperimentController(wsgi.Controller, wsgi.AdminActionsMixin): + + """The Experiments API v1 controller for the OpenStack API.""" + resource_name = 'experiment' + _view_builder_class = experiment_views.ViewBuilder + + def __init__(self): + super(self.__class__, self).__init__() + self.engine_api = engine.API() + + def show(self, req, id): + """Return data about the given experiment.""" + context = req.environ['meteos.context'] + + try: + experiment = self.engine_api.get_experiment(context, id) + except exception.NotFound: + raise exc.HTTPNotFound() + + return self._view_builder.detail(req, experiment) + + def delete(self, req, id): + """Delete a experiment.""" + context = req.environ['meteos.context'] + + LOG.info(_LI("Delete experiment with id: %s"), id, context=context) + + try: + self.engine_api.delete_experiment(context, id) + except exception.NotFound: + raise exc.HTTPNotFound() + except exception.InvalidLearning as e: + raise exc.HTTPForbidden(explanation=six.text_type(e)) + + return webob.Response(status_int=202) + + def index(self, req): + """Returns a summary list of experiments.""" + return self._get_experiments(req, is_detail=False) + + def detail(self, req): + """Returns a detailed list of experiments.""" + return self._get_experiments(req, is_detail=True) + + def _get_experiments(self, req, is_detail): + """Returns a list of experiments, transformed through view builder.""" + context = req.environ['meteos.context'] + + search_opts = {} + search_opts.update(req.GET) + + # Remove keys that are not related to experiment attrs + search_opts.pop('limit', None) + search_opts.pop('offset', None) + sort_key = search_opts.pop('sort_key', 'created_at') + sort_dir = search_opts.pop('sort_dir', 'desc') + + experiments = self.engine_api.get_all_experiments( + context, search_opts=search_opts, sort_key=sort_key, + sort_dir=sort_dir) + + limited_list = common.limited(experiments, req) + + if is_detail: + experiments = self._view_builder.detail_list(req, limited_list) + else: + experiments = self._view_builder.summary_list(req, limited_list) + return experiments + + def create(self, req, body): + """Creates a new experiment.""" + context = req.environ['meteos.context'] + + if not self.is_valid_body(body, 'experiment'): + raise exc.HTTPUnprocessableEntity() + + experiment = body['experiment'] + + LOG.debug("Create experiment with request: %s", experiment) + + try: + self.engine_api.get_template(context, experiment['template_id']) + except exception.NotFound: + raise exc.HTTPNotFound() + + display_name = experiment.get('display_name') + display_description = experiment.get('display_description') + template_id = experiment.get('template_id') + key_name = experiment.get('key_name') + neutron_management_network = experiment.get( + 'neutron_management_network') + + new_experiment = self.engine_api.create_experiment( + context, + display_name, + display_description, + template_id, + key_name, + neutron_management_network) + + return self._view_builder.detail(req, new_experiment) + + +def create_resource(): + return wsgi.Resource(ExperimentController()) diff --git a/meteos/api/v1/learnings.py b/meteos/api/v1/learnings.py new file mode 100644 index 0000000..e1d9ff7 --- /dev/null +++ b/meteos/api/v1/learnings.py @@ -0,0 +1,155 @@ +# Copyright 2013 NetApp +# All Rights Reserved. +# Copyright (c) 2016 NEC Corporation. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +"""The learnings api.""" + +import ast +import re +import string + +from oslo_log import log +from oslo_utils import strutils +from oslo_utils import uuidutils +import six +import webob +from webob import exc + +from meteos.api import common +from meteos.api.openstack import wsgi +from meteos.api.views import learnings as learning_views +from meteos import exception +from meteos.i18n import _, _LI +from meteos import engine + +LOG = log.getLogger(__name__) + + +class LearningController(wsgi.Controller, wsgi.AdminActionsMixin): + + """The Learnings API v1 controller for the OpenStack API.""" + resource_name = 'learning' + _view_builder_class = learning_views.ViewBuilder + + def __init__(self): + super(self.__class__, self).__init__() + self.engine_api = engine.API() + + def show(self, req, id): + """Return data about the given learning.""" + context = req.environ['meteos.context'] + + try: + learning = self.engine_api.get_learning(context, id) + except exception.NotFound: + raise exc.HTTPNotFound() + + return self._view_builder.detail(req, learning) + + def delete(self, req, id): + """Delete a learning.""" + context = req.environ['meteos.context'] + + LOG.info(_LI("Delete learning with id: %s"), id, context=context) + + try: + self.engine_api.delete_learning(context, id) + except exception.NotFound: + raise exc.HTTPNotFound() + except exception.InvalidLearning as e: + raise exc.HTTPForbidden(explanation=six.text_type(e)) + + return webob.Response(status_int=202) + + def index(self, req): + """Returns a summary list of learnings.""" + return self._get_learnings(req, is_detail=False) + + def detail(self, req): + """Returns a detailed list of learnings.""" + return self._get_learnings(req, is_detail=True) + + def _get_learnings(self, req, is_detail): + """Returns a list of learnings, transformed through view builder.""" + context = req.environ['meteos.context'] + + search_opts = {} + search_opts.update(req.GET) + + # Remove keys that are not related to learning attrs + search_opts.pop('limit', None) + search_opts.pop('offset', None) + sort_key = search_opts.pop('sort_key', 'created_at') + sort_dir = search_opts.pop('sort_dir', 'desc') + + learnings = self.engine_api.get_all_learnings( + context, search_opts=search_opts, sort_key=sort_key, + sort_dir=sort_dir) + + limited_list = common.limited(learnings, req) + + if is_detail: + learnings = self._view_builder.detail_list(req, limited_list) + else: + learnings = self._view_builder.summary_list(req, limited_list) + return learnings + + def create(self, req, body): + """Creates a new learning.""" + context = req.environ['meteos.context'] + + if not self.is_valid_body(body, 'learning'): + raise exc.HTTPUnprocessableEntity() + + learning = body['learning'] + + LOG.debug("Create learning with request: %s", learning) + + try: + self.engine_api.get_experiment(context, learning['experiment_id']) + except exception.NotFound: + raise exc.HTTPNotFound() + + display_name = learning.get('display_name') + display_description = learning.get('display_description') + experiment_id = learning.get('experiment_id') + model_id = learning.get('model_id') + method = learning.get('method') + args = learning.get('args') + + try: + experiment = self.engine_api.get_experiment(context, experiment_id) + template = self.engine_api.get_template( + context, experiment.template_id) + except exception.NotFound: + raise exc.HTTPNotFound() + + new_learning = self.engine_api.create_learning( + context, + display_name, + display_description, + model_id, + method, + args, + template.id, + template.job_template_id, + experiment_id, + experiment.cluster_id) + + return self._view_builder.detail(req, new_learning) + + +def create_resource(): + return wsgi.Resource(LearningController()) diff --git a/meteos/api/v1/models.py b/meteos/api/v1/models.py new file mode 100644 index 0000000..d70d5e1 --- /dev/null +++ b/meteos/api/v1/models.py @@ -0,0 +1,158 @@ +# Copyright 2013 NetApp +# All Rights Reserved. +# Copyright (c) 2016 NEC Corporation. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +"""The models api.""" + +import ast +import re +import string + +from oslo_log import log +from oslo_utils import strutils +from oslo_utils import uuidutils +import six +import webob +from webob import exc + +from meteos.api import common +from meteos.api.openstack import wsgi +from meteos.api.views import models as model_views +from meteos import exception +from meteos.i18n import _, _LI +from meteos import engine + +LOG = log.getLogger(__name__) + + +class ModelController(wsgi.Controller, wsgi.AdminActionsMixin): + + """The Models API v1 controller for the OpenStack API.""" + resource_name = 'model' + _view_builder_class = model_views.ViewBuilder + + def __init__(self): + super(self.__class__, self).__init__() + self.engine_api = engine.API() + + def show(self, req, id): + """Return data about the given model.""" + context = req.environ['meteos.context'] + + try: + model = self.engine_api.get_model(context, id) + except exception.NotFound: + raise exc.HTTPNotFound() + + return self._view_builder.detail(req, model) + + def delete(self, req, id): + """Delete a model.""" + context = req.environ['meteos.context'] + + LOG.info(_LI("Delete model with id: %s"), id, context=context) + + try: + self.engine_api.delete_model(context, id) + except exception.NotFound: + raise exc.HTTPNotFound() + except exception.InvalidLearning as e: + raise exc.HTTPForbidden(explanation=six.text_type(e)) + + return webob.Response(status_int=202) + + def index(self, req): + """Returns a summary list of models.""" + return self._get_models(req, is_detail=False) + + def detail(self, req): + """Returns a detailed list of models.""" + return self._get_models(req, is_detail=True) + + def _get_models(self, req, is_detail): + """Returns a list of models, transformed through view builder.""" + context = req.environ['meteos.context'] + + search_opts = {} + search_opts.update(req.GET) + + # Remove keys that are not related to model attrs + search_opts.pop('limit', None) + search_opts.pop('offset', None) + sort_key = search_opts.pop('sort_key', 'created_at') + sort_dir = search_opts.pop('sort_dir', 'desc') + + models = self.engine_api.get_all_models( + context, search_opts=search_opts, sort_key=sort_key, + sort_dir=sort_dir) + + limited_list = common.limited(models, req) + + if is_detail: + models = self._view_builder.detail_list(req, limited_list) + else: + models = self._view_builder.summary_list(req, limited_list) + return models + + def create(self, req, body): + """Creates a new model.""" + context = req.environ['meteos.context'] + + if not self.is_valid_body(body, 'model'): + raise exc.HTTPUnprocessableEntity() + + model = body['model'] + + LOG.debug("Create model with request: %s", model) + + try: + experiment = self.engine_api.get_experiment( + context, model['experiment_id']) + template = self.engine_api.get_template( + context, experiment.template_id) + except exception.NotFound: + raise exc.HTTPNotFound() + + display_name = model.get('display_name') + display_description = model.get('display_description') + experiment_id = model.get('experiment_id') + source_dataset_url = model.get('source_dataset_url') + dataset_format = model.get('dataset_format', 'csv') + model_type = model.get('model_type') + model_params = model.get('model_params') + swift_tenant = model.get('swift_tenant') + swift_username = model.get('swift_username') + swift_password = model.get('swift_password') + + new_model = self.engine_api.create_model(context, + display_name, + display_description, + source_dataset_url, + dataset_format, + model_type, + model_params, + template.id, + template.job_template_id, + experiment_id, + experiment.cluster_id, + swift_tenant, + swift_username, + swift_password) + + return self._view_builder.detail(req, new_model) + + +def create_resource(): + return wsgi.Resource(ModelController()) diff --git a/meteos/api/v1/router.py b/meteos/api/v1/router.py new file mode 100644 index 0000000..1524ebe --- /dev/null +++ b/meteos/api/v1/router.py @@ -0,0 +1,73 @@ +# Copyright 2011 OpenStack LLC. +# Copyright 2011 United States Government as represented by the +# Administrator of the National Aeronautics and Space Administration. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +""" +WSGI middleware for OpenStack Learning API v1. +""" + +import meteos.api.openstack +from meteos.api.v1 import learnings +from meteos.api.v1 import experiments +from meteos.api.v1 import templates +from meteos.api.v1 import datasets +from meteos.api.v1 import models +from meteos.api import versions + + +class APIRouter(meteos.api.openstack.APIRouter): + """Route API requests. + + Routes requests on the OpenStack API to the appropriate controller + and method. + """ + def _setup_routes(self, mapper, ext_mgr): + self.resources['versions'] = versions.create_resource() + mapper.connect("versions", "/", + controller=self.resources['versions'], + action='index') + + mapper.redirect("", "/") + + self.resources['templates'] = templates.create_resource() + mapper.resource("template", "templates", + controller=self.resources['templates'], + collection={'detail': 'GET'}, + member={'action': 'POST'}) + + self.resources['experiments'] = experiments.create_resource() + mapper.resource("experiment", "experiments", + controller=self.resources['experiments'], + collection={'detail': 'GET'}, + member={'action': 'POST'}) + + self.resources['learnings'] = learnings.create_resource() + mapper.resource("learning", "learnings", + controller=self.resources['learnings'], + collection={'detail': 'GET'}, + member={'action': 'POST'}) + + self.resources['datasets'] = datasets.create_resource() + mapper.resource("dataset", "datasets", + controller=self.resources['datasets'], + collection={'detail': 'GET'}, + member={'action': 'POST'}) + + self.resources['models'] = models.create_resource() + mapper.resource("model", "models", + controller=self.resources['models'], + collection={'detail': 'GET'}, + member={'action': 'POST'}) diff --git a/meteos/api/v1/templates.py b/meteos/api/v1/templates.py new file mode 100644 index 0000000..c5834c6 --- /dev/null +++ b/meteos/api/v1/templates.py @@ -0,0 +1,145 @@ +# Copyright 2013 NetApp +# All Rights Reserved. +# Copyright (c) 2016 NEC Corporation. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +"""The templates api.""" + +import ast +import re +import string + +from oslo_log import log +from oslo_utils import strutils +from oslo_utils import uuidutils +import six +import webob +from webob import exc + +from meteos.api import common +from meteos.api.openstack import wsgi +from meteos.api.views import templates as template_views +from meteos import exception +from meteos.i18n import _, _LI +from meteos import engine + +LOG = log.getLogger(__name__) + + +class TemplateController(wsgi.Controller, wsgi.AdminActionsMixin): + + """The Templates API v1 controller for the OpenStack API.""" + resource_name = 'template' + _view_builder_class = template_views.ViewBuilder + + def __init__(self): + super(self.__class__, self).__init__() + self.engine_api = engine.API() + + def show(self, req, id): + """Return data about the given template.""" + context = req.environ['meteos.context'] + + try: + template = self.engine_api.get_template(context, id) + except exception.NotFound: + raise exc.HTTPNotFound() + + return self._view_builder.detail(req, template) + + def delete(self, req, id): + """Delete a template.""" + context = req.environ['meteos.context'] + + LOG.info(_LI("Delete template with id: %s"), id, context=context) + + try: + self.engine_api.delete_template(context, id) + except exception.NotFound: + raise exc.HTTPNotFound() + except exception.InvalidLearning as e: + raise exc.HTTPForbidden(explanation=six.text_type(e)) + + return webob.Response(status_int=202) + + def index(self, req): + """Returns a summary list of templates.""" + return self._get_templates(req, is_detail=False) + + def detail(self, req): + """Returns a detailed list of templates.""" + return self._get_templates(req, is_detail=True) + + def _get_templates(self, req, is_detail): + """Returns a list of templates, transformed through view builder.""" + context = req.environ['meteos.context'] + + search_opts = {} + search_opts.update(req.GET) + + # Remove keys that are not related to template attrs + search_opts.pop('limit', None) + search_opts.pop('offset', None) + sort_key = search_opts.pop('sort_key', 'created_at') + sort_dir = search_opts.pop('sort_dir', 'desc') + + templates = self.engine_api.get_all_templates( + context, search_opts=search_opts, sort_key=sort_key, + sort_dir=sort_dir) + + limited_list = common.limited(templates, req) + + if is_detail: + templates = self._view_builder.detail_list(req, limited_list) + else: + templates = self._view_builder.summary_list(req, limited_list) + return templates + + def create(self, req, body): + """Creates a new template.""" + context = req.environ['meteos.context'] + + if not self.is_valid_body(body, 'template'): + raise exc.HTTPUnprocessableEntity() + + template = body['template'] + + LOG.debug("Create template with request: %s", template) + + kwargs = { + 'image_id': template.get('image_id'), + 'master_nodes_num': template.get('master_nodes_num'), + 'master_flavor_id': template.get('master_flavor_id'), + 'worker_nodes_num': template.get('worker_nodes_num'), + 'worker_flavor_id': template.get('worker_flavor_id'), + 'spark_version': template.get('spark_version'), + 'floating_ip_pool': template.get('floating_ip_pool'), + } + + display_name = template.get('display_name') + display_description = template.get('display_description') + + try: + new_template = self.engine_api.create_template(context, + display_name, + display_description, + **kwargs) + except exception.NotFound: + raise exc.HTTPNotFound() + + return self._view_builder.detail(req, new_template) + + +def create_resource(): + return wsgi.Resource(TemplateController()) diff --git a/meteos/api/versions.py b/meteos/api/versions.py new file mode 100644 index 0000000..3596349 --- /dev/null +++ b/meteos/api/versions.py @@ -0,0 +1,97 @@ +# Copyright 2010 OpenStack LLC. +# Copyright 2015 Clinton Knight +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +import copy + +from oslo_config import cfg + +from meteos.api import openstack +from meteos.api.openstack import api_version_request +from meteos.api.openstack import wsgi +from meteos.api.views import versions as views_versions + +CONF = cfg.CONF + +_LINKS = [{ + 'rel': 'describedby', + 'type': 'text/html', + 'href': 'http://docs.openstack.org/', +}] + +_MEDIA_TYPES = [{ + 'base': 'application/json', + 'type': 'application/vnd.openstack.learning+json;version=1', +}] + +_KNOWN_VERSIONS = { + 'v1.0': { + 'id': 'v1.0', + 'status': 'SUPPORTED', + 'version': '', + 'min_version': '', + 'updated': '2015-08-27T11:33:21Z', + 'links': _LINKS, + 'media-types': _MEDIA_TYPES, + }, +} + + +class VersionsRouter(openstack.APIRouter): + """Route versions requests.""" + + def _setup_routes(self, mapper, ext_mgr): + self.resources['versions'] = create_resource() + mapper.connect('versions', '/', + controller=self.resources['versions'], + action='all') + mapper.redirect('', '/') + + +class VersionsController(wsgi.Controller): + + def __init__(self): + super(VersionsController, self).__init__(None) + + @wsgi.Controller.api_version('1.0', '1.0') + def index(self, req): + """Return versions supported prior to the microversions epoch.""" + builder = views_versions.get_view_builder(req) + known_versions = copy.deepcopy(_KNOWN_VERSIONS) + known_versions.pop('v2.0') + return builder.build_versions(known_versions) + + @wsgi.Controller.api_version('2.0') # noqa + def index(self, req): # pylint: disable=E0102 + """Return versions supported after the start of microversions.""" + builder = views_versions.get_view_builder(req) + known_versions = copy.deepcopy(_KNOWN_VERSIONS) + known_versions.pop('v1.0') + return builder.build_versions(known_versions) + + # NOTE (cknight): Calling the versions API without + # /v1 or /v2 in the URL will lead to this unversioned + # method, which should always return info about all + # available versions. + @wsgi.response(300) + def all(self, req): + """Return all known versions.""" + builder = views_versions.get_view_builder(req) + known_versions = copy.deepcopy(_KNOWN_VERSIONS) + return builder.build_versions(known_versions) + + +def create_resource(): + return wsgi.Resource(VersionsController()) diff --git a/meteos/api/views/__init__.py b/meteos/api/views/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/meteos/api/views/datasets.py b/meteos/api/views/datasets.py new file mode 100644 index 0000000..229a5a0 --- /dev/null +++ b/meteos/api/views/datasets.py @@ -0,0 +1,80 @@ +# Copyright 2013 OpenStack LLC. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +from meteos.api import common + + +class ViewBuilder(common.ViewBuilder): + + """Model a server API response as a python dictionary.""" + + _collection_name = 'datasets' + _detail_version_modifiers = [] + + def summary_list(self, request, datasets): + """Show a list of datasets without many details.""" + return self._list_view(self.summary, request, datasets) + + def detail_list(self, request, datasets): + """Detailed view of a list of datasets.""" + return self._list_view(self.detail, request, datasets) + + def summary(self, request, dataset): + """Generic, non-detailed view of a dataset.""" + return { + 'dataset': { + 'id': dataset.get('id'), + 'source_dataset_url': dataset.get('source_dataset_url'), + 'name': dataset.get('display_name'), + 'description': dataset.get('display_description'), + 'status': dataset.get('status'), + 'created_at': dataset.get('created_at'), + 'links': self._get_links(request, dataset['id']) + } + } + + def detail(self, request, dataset): + """Detailed view of a single dataset.""" + context = request.environ['meteos.context'] + + dataset_dict = { + 'id': dataset.get('id'), + 'created_at': dataset.get('created_at'), + 'status': dataset.get('status'), + 'name': dataset.get('display_name'), + 'description': dataset.get('display_description'), + 'user_id': dataset.get('user_id'), + 'project_id': dataset.get('project_id'), + 'head': dataset.get('head'), + 'stderr': dataset.get('stderr'), + } + + self.update_versioned_resource_dict(request, dataset_dict, dataset) + + return {'dataset': dataset_dict} + + def _list_view(self, func, request, datasets): + """Provide a view for a list of datasets.""" + datasets_list = [func(request, dataset)['dataset'] + for dataset in datasets] + datasets_links = self._get_collection_links(request, + datasets, + self._collection_name) + datasets_dict = dict(datasets=datasets_list) + + if datasets_links: + datasets_dict['datasets_links'] = datasets_links + + return datasets_dict diff --git a/meteos/api/views/experiments.py b/meteos/api/views/experiments.py new file mode 100644 index 0000000..f6fe953 --- /dev/null +++ b/meteos/api/views/experiments.py @@ -0,0 +1,80 @@ +# Copyright 2013 OpenStack LLC. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +from meteos.api import common + + +class ViewBuilder(common.ViewBuilder): + + """Model a server API response as a python dictionary.""" + + _collection_name = 'experiments' + _detail_version_modifiers = [] + + def summary_list(self, request, experiments): + """Show a list of experiments without many details.""" + return self._list_view(self.summary, request, experiments) + + def detail_list(self, request, experiments): + """Detailed view of a list of experiments.""" + return self._list_view(self.detail, request, experiments) + + def summary(self, request, experiment): + """Generic, non-detailed view of a experiment.""" + return { + 'experiment': { + 'id': experiment.get('id'), + 'name': experiment.get('display_name'), + 'description': experiment.get('display_description'), + 'status': experiment.get('status'), + 'created_at': experiment.get('created_at'), + 'links': self._get_links(request, experiment['id']) + } + } + + def detail(self, request, experiment): + """Detailed view of a single experiment.""" + context = request.environ['meteos.context'] + + experiment_dict = { + 'id': experiment.get('id'), + 'created_at': experiment.get('created_at'), + 'status': experiment.get('status'), + 'name': experiment.get('display_name'), + 'description': experiment.get('display_description'), + 'project_id': experiment.get('project_id'), + 'user_id': experiment.get('user_id'), + 'key_name': experiment.get('key_name'), + 'management_network': experiment.get('neutron_management_network'), + } + + self.update_versioned_resource_dict( + request, experiment_dict, experiment) + + return {'experiment': experiment_dict} + + def _list_view(self, func, request, experiments): + """Provide a view for a list of experiments.""" + experiments_list = [func(request, experiment)['experiment'] + for experiment in experiments] + experiments_links = self._get_collection_links(request, + experiments, + self._collection_name) + experiments_dict = dict(experiments=experiments_list) + + if experiments_links: + experiments_dict['experiments_links'] = experiments_links + + return experiments_dict diff --git a/meteos/api/views/learnings.py b/meteos/api/views/learnings.py new file mode 100644 index 0000000..c265b12 --- /dev/null +++ b/meteos/api/views/learnings.py @@ -0,0 +1,84 @@ +# Copyright 2013 OpenStack LLC. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +from meteos.api import common + + +class ViewBuilder(common.ViewBuilder): + + """Model a server API response as a python dictionary.""" + + _collection_name = 'learnings' + _detail_version_modifiers = [] + + def summary_list(self, request, learnings): + """Show a list of learnings without many details.""" + return self._list_view(self.summary, request, learnings) + + def detail_list(self, request, learnings): + """Detailed view of a list of learnings.""" + return self._list_view(self.detail, request, learnings) + + def summary(self, request, learning): + """Generic, non-detailed view of a learning.""" + return { + 'learning': { + 'id': learning.get('id'), + 'name': learning.get('display_name'), + 'description': learning.get('display_description'), + 'status': learning.get('status'), + 'type': learning.get('model_type'), + 'args': learning.get('args'), + 'stdout': learning.get('stdout'), + 'created_at': learning.get('created_at'), + 'links': self._get_links(request, learning['id']) + } + } + + def detail(self, request, learning): + """Detailed view of a single learning.""" + context = request.environ['meteos.context'] + + learning_dict = { + 'id': learning.get('id'), + 'created_at': learning.get('created_at'), + 'status': learning.get('status'), + 'name': learning.get('display_name'), + 'description': learning.get('display_description'), + 'user_id': learning.get('user_id'), + 'project_id': learning.get('project_id'), + 'stdout': learning.get('stdout'), + 'stderr': learning.get('stderr'), + 'method': learning.get('method'), + 'args': learning.get('args'), + } + + self.update_versioned_resource_dict(request, learning_dict, learning) + + return {'learning': learning_dict} + + def _list_view(self, func, request, learnings): + """Provide a view for a list of learnings.""" + learnings_list = [func(request, learning)['learning'] + for learning in learnings] + learnings_links = self._get_collection_links(request, + learnings, + self._collection_name) + learnings_dict = dict(learnings=learnings_list) + + if learnings_links: + learnings_dict['learnings_links'] = learnings_links + + return learnings_dict diff --git a/meteos/api/views/models.py b/meteos/api/views/models.py new file mode 100644 index 0000000..00afd44 --- /dev/null +++ b/meteos/api/views/models.py @@ -0,0 +1,82 @@ +# Copyright 2013 OpenStack LLC. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +from meteos.api import common + + +class ViewBuilder(common.ViewBuilder): + + """Model a server API response as a python dictionary.""" + + _collection_name = 'models' + _detail_version_modifiers = [] + + def summary_list(self, request, models): + """Show a list of models without many details.""" + return self._list_view(self.summary, request, models) + + def detail_list(self, request, models): + """Detailed view of a list of models.""" + return self._list_view(self.detail, request, models) + + def summary(self, request, model): + """Generic, non-detailed view of a model.""" + return { + 'model': { + 'id': model.get('id'), + 'source_dataset_url': model.get('source_dataset_url'), + 'name': model.get('display_name'), + 'description': model.get('display_description'), + 'type': model.get('model_type'), + 'status': model.get('status'), + 'created_at': model.get('created_at'), + 'links': self._get_links(request, model['id']) + } + } + + def detail(self, request, model): + """Detailed view of a single model.""" + context = request.environ['meteos.context'] + + model_dict = { + 'id': model.get('id'), + 'created_at': model.get('created_at'), + 'status': model.get('status'), + 'name': model.get('display_name'), + 'description': model.get('display_description'), + 'user_id': model.get('user_id'), + 'project_id': model.get('project_id'), + 'type': model.get('model_type'), + 'params': model.get('model_params'), + 'stdout': model.get('stdout'), + 'stderr': model.get('stderr'), + } + + self.update_versioned_resource_dict(request, model_dict, model) + + return {'model': model_dict} + + def _list_view(self, func, request, models): + """Provide a view for a list of models.""" + models_list = [func(request, model)['model'] for model in models] + models_links = self._get_collection_links(request, + models, + self._collection_name) + models_dict = dict(models=models_list) + + if models_links: + models_dict['models_links'] = models_links + + return models_dict diff --git a/meteos/api/views/templates.py b/meteos/api/views/templates.py new file mode 100644 index 0000000..9bd8b03 --- /dev/null +++ b/meteos/api/views/templates.py @@ -0,0 +1,85 @@ +# Copyright 2013 OpenStack LLC. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +from meteos.api import common + + +class ViewBuilder(common.ViewBuilder): + + """Model a server API response as a python dictionary.""" + + _collection_name = 'templates' + _detail_version_modifiers = [] + + def summary_list(self, request, templates): + """Show a list of templates without many details.""" + return self._list_view(self.summary, request, templates) + + def detail_list(self, request, templates): + """Detailed view of a list of templates.""" + return self._list_view(self.detail, request, templates) + + def summary(self, request, template): + """Generic, non-detailed view of a template.""" + return { + 'template': { + 'id': template.get('id'), + 'name': template.get('display_name'), + 'description': template.get('display_description'), + 'master_nodes': template.get('master_nodes_num'), + 'worker_nodes': template.get('worker_nodes_num'), + 'spark_version': template.get('spark_version'), + 'status': template.get('status'), + 'links': self._get_links(request, template['id']) + } + } + + def detail(self, request, template): + """Detailed view of a single template.""" + context = request.environ['meteos.context'] + + template_dict = { + 'id': template.get('id'), + 'created_at': template.get('created_at'), + 'status': template.get('status'), + 'name': template.get('display_name'), + 'description': template.get('display_description'), + 'user_id': template.get('user_id'), + 'project_id': template.get('project_id'), + 'master_nodes': template.get('master_nodes_num'), + 'master_flavor': template.get('master_flavor_id'), + 'worker_nodes': template.get('worker_nodes_num'), + 'worker_flavor': template.get('worker_flavor_id'), + 'spark_version': template.get('spark_version'), + 'cluster_id': template.get('cluster_id'), + } + + self.update_versioned_resource_dict(request, template_dict, template) + + return {'template': template_dict} + + def _list_view(self, func, request, templates): + """Provide a view for a list of templates.""" + templates_list = [func(request, template)['template'] + for template in templates] + templates_links = self._get_collection_links(request, + templates, + self._collection_name) + templates_dict = dict(templates=templates_list) + + if templates_links: + templates_dict['templates_links'] = templates_links + + return templates_dict diff --git a/meteos/api/views/versions.py b/meteos/api/views/versions.py new file mode 100644 index 0000000..b2bddcd --- /dev/null +++ b/meteos/api/views/versions.py @@ -0,0 +1,66 @@ +# Copyright 2010-2011 OpenStack LLC. +# Copyright 2015 Clinton Knight +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +import copy +import re + +from six.moves import urllib + + +def get_view_builder(req): + return ViewBuilder(req.application_url) + + +_URL_SUFFIX = {'v1.0': 'v1', 'v2.0': 'v2'} + + +class ViewBuilder(object): + def __init__(self, base_url): + """Initialize ViewBuilder. + + :param base_url: url of the root wsgi application + """ + self.base_url = base_url + + def build_versions(self, versions): + views = [self._build_version(versions[key]) + for key in sorted(list(versions.keys()))] + return dict(versions=views) + + def _build_version(self, version): + view = copy.deepcopy(version) + view['links'] = self._build_links(version) + return view + + def _build_links(self, version_data): + """Generate a container of links that refer to the provided version.""" + links = copy.deepcopy(version_data.get('links', {})) + version = _URL_SUFFIX.get(version_data['id']) + links.append({'rel': 'self', + 'href': self._generate_href(version=version)}) + return links + + def _generate_href(self, version='v1', path=None): + """Create a URL that refers to a specific version_number.""" + base_url = self._get_base_url_without_version() + href = urllib.parse.urljoin(base_url, version).rstrip('/') + '/' + if path: + href += path.lstrip('/') + return href + + def _get_base_url_without_version(self): + """Get the base URL with out the /v1 suffix.""" + return re.sub('v[1-9]+/?$', '', self.base_url) diff --git a/meteos/cluster/__init__.py b/meteos/cluster/__init__.py new file mode 100644 index 0000000..385a02d --- /dev/null +++ b/meteos/cluster/__init__.py @@ -0,0 +1,33 @@ +# Copyright 2014 Mirantis Inc. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +import oslo_config.cfg +import oslo_utils.importutils + +_volume_opts = [ + oslo_config.cfg.StrOpt('cluster_api_class', + default='meteos.cluster.sahara.API', + help='The full class name of the ' + 'Cluster API class to use.'), +] + +oslo_config.cfg.CONF.register_opts(_volume_opts) + + +def API(): + importutils = oslo_utils.importutils + cluster_api_class = oslo_config.cfg.CONF.cluster_api_class + cls = importutils.import_class(cluster_api_class) + return cls() diff --git a/meteos/cluster/binary/meteos-script-1.6.0.py b/meteos/cluster/binary/meteos-script-1.6.0.py new file mode 100644 index 0000000..69486e3 --- /dev/null +++ b/meteos/cluster/binary/meteos-script-1.6.0.py @@ -0,0 +1,386 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# Copyright 2016 NEC Corpocation All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + + +import base64 +import sys +import uuid +import socket +from ast import literal_eval +from numpy import array +from math import sqrt +from pyspark import SparkContext + +from pyspark.mllib.linalg import SparseVector +from pyspark.mllib.classification import LogisticRegressionWithSGD +from pyspark.mllib.classification import LogisticRegressionModel +from pyspark.mllib.clustering import KMeans, KMeansModel +from pyspark.mllib.recommendation import ALS, MatrixFactorizationModel, Rating +from pyspark.mllib.regression import LabeledPoint +from pyspark.mllib.regression import LinearRegressionWithSGD +from pyspark.mllib.regression import LinearRegressionModel +from pyspark.mllib.tree import DecisionTree, DecisionTreeModel +from pyspark.mllib.util import MLUtils + + +class ModelController(object): + + """Class defines interface of Model Controller.""" + + def __init__(self): + super(ModelController, self).__init__() + + def create_model(self, data, params): + """Is called to create mode.""" + raise NotImplementedError() + + def create_model_libsvm(self, data, params): + """Is called to create mode.""" + raise NotImplementedError() + + def load_model(self, context, path): + """Is called to load mode.""" + raise NotImplementedError() + + def predict(self, context, params): + """Is called to predict value.""" + raise NotImplementedError() + + def predict_libsvm(self, context, params): + """Is called to predict value.""" + raise NotImplementedError() + + def parsePoint(self, line): + values = [float(s) for s in line.split(',')] + if values[0] == -1: + values[0] = 0 + return LabeledPoint(values[0], values[1:]) + +class KMeansModelController(ModelController): + + def __init__(self): + super(KMeansModelController, self).__init__() + + def create_model(self, data, params): + + numClasses = params.get('numClasses', 2) + numIterations = params.get('numIterations', 10) + runs = params.get('runs', 10) + mode = params.get('mode', 'random') + + parsedData = data.map( + lambda line: array([float(x) for x in line.split(',')])) + + return KMeans.train(parsedData, + numClasses, + maxIterations=numIterations, + runs=runs, + initializationMode=mode) + + def load_model(self, context, path): + return KMeansModel.load(context, path) + + def predict(self, model, params): + return model.predict(params.split(',')) + + +class RecommendationController(ModelController): + + def __init__(self): + super(RecommendationController, self).__init__() + + def create_model(self, data, params): + + # Build the recommendation model using Alternating Least Squares + rank = params.get('rank', 10) + numIterations = params.get('numIterations', 10) + + ratings = data.map(lambda l: l.split(','))\ + .map(lambda l: Rating(int(l[0]), int(l[1]), float(l[2]))) + + return ALS.train(ratings, rank, numIterations) + + def load_model(self, context, path): + return MatrixFactorizationModel.load(context, path) + + def predict(self, model, params): + + parsedData = params.split(',') + return model.predict(parsedData[0], parsedData[1]) + + +class LinearRegressionModelController(ModelController): + + def __init__(self): + super(LinearRegressionModelController, self).__init__() + + def create_model(self, data, params): + + iterations = params.get('numIterations', 10) + step = params.get('step', 0.00000001) + + points = data.map(self.parsePoint) + return LinearRegressionWithSGD.train(points, + iterations=iterations, + step=step) + + def create_model_libsvm(self, data, params): + + iterations = params.get('numIterations', 10) + step = params.get('step', 0.00000001) + + return LinearRegressionWithSGD.train(data, + iterations=iterations, + step=step) + + def load_model(self, context, path): + return LinearRegressionModel.load(context, path) + + def predict(self, model, params): + return model.predict(params.split(',')) + + def predict_libsvm(self, model, params): + return self.predict(model, params) + + +class LogisticRegressionModelController(ModelController): + + def __init__(self): + super(LogisticRegressionModelController, self).__init__() + + def create_model(self, data, params): + + numIterations = params.get('numIterations', 10) + + points = data.map(self.parsePoint) + return LogisticRegressionWithSGD.train(points, numIterations) + + def create_model_libsvm(self, data, params): + + numIterations = params.get('numIterations', 10) + + return LogisticRegressionWithSGD.train(data, numIterations) + + def load_model(self, context, path): + return LogisticRegressionModel.load(context, path) + + def predict(self, model, params): + return model.predict(params.split(',')) + + def predict_libsvm(self, model, params): + return self.predict(model, params) + + +class DecisionTreeModelController(ModelController): + + def __init__(self): + super(DecisionTreeModelController, self).__init__() + + def _parse_to_libsvm(self, param): + + index_l = [] + value_l = [] + + param_l = param.split(' ') + param_len = str(len(param_l) * 2) + + for p in param_l: + index_l.append(str(int(p.split(':')[0]) - 1)) + value_l.append(p.split(':')[1]) + + index = ','.join(index_l) + value = ','.join(value_l) + + parsed_str = '(' + param_len + ', [' + index + '],[' + value + '])' + + return SparseVector.parse(parsed_str) + + def create_model_libsvm(self, data, params): + + impurity = params.get('impurity', 'variance') + maxDepth = params.get('maxDepth', 5) + maxBins = params.get('maxBins', 32) + + return DecisionTree.trainRegressor(data, + categoricalFeaturesInfo={}, + impurity=impurity, + maxDepth=maxDepth, + maxBins=maxBins) + + def load_model(self, context, path): + return DecisionTreeModel.load(context, path) + + def predict(self, model, params): + return model.predict(params.split(',')) + + def predict_libsvm(self, model, params): + parsed_params = self._parse_to_libsvm(params) + return model.predict(parsed_params) + + +class MeteosSparkController(object): + + def init_context(self): + + self.base_hostname = socket.gethostname().split(".")[0] + master_node = 'spark://' + self.base_hostname + ':7077' + self.context = SparkContext(master_node, 'INFO') + + def parse_args(self, args): + + self.id = args[3] + decoded_args = base64.b64decode(args[4]) + self.job_args = literal_eval(decoded_args) + + self.datapath = 'data-' + self.id + self.modelpath = 'model-' + self.id + + def init_model_controller(self): + + model_type = self.job_args['model']['type'] + + if model_type == 'KMeans': + self.controller = KMeansModelController() + elif model_type == 'Recommendation': + self.controller = RecommendationController() + elif model_type == 'LogisticRegression': + self.controller = LogisticRegressionModelController() + elif model_type == 'LinearRegression': + self.controller = LinearRegressionModelController() + elif model_type == 'DecisionTreeRegression': + self.controller = DecisionTreeModelController() + + def save_data(self, collect=True): + + if collect: + self.data.collect() + self.data.saveAsTextFile(self.datapath) + print self.data.take(10) + + def load_data(self): + + source_dataset_url = self.job_args['source_dataset_url'] + + if source_dataset_url.count('swift'): + swift = self.job_args['swift'] + tenant = swift['tenant'] + username = swift['username'] + password = swift['password'] + container_name = source_dataset_url.split('/')[2] + object_name = source_dataset_url.split('/')[3] + + prefix = 'fs.swift.service.sahara' + hconf = self.context._jsc.hadoopConfiguration() + hconf.set(prefix + '.tenant', tenant) + hconf.set(prefix + '.username', username) + hconf.set(prefix + '.password', password) + hconf.setInt(prefix + ".http.port", 8080) + + self.data = self._load_data('swift://' + container_name + '.sahara/' + object_name) + else: + dataset_path = 'data-' + source_dataset_url.split('/')[2] + self.data = self._load_data(dataset_path) + + def _load_data(self, path): + + dataset_format = self.job_args.get('dataset_format') + + if dataset_format == 'libsvm': + return MLUtils.loadLibSVMFile(self.context, path) + else: + return self.context.textFile(path).cache() + + def create_and_save_model(self): + + model_params = self.job_args['model']['params'] + params = base64.b64decode(model_params) + list_params = literal_eval(params) + + dataset_format = self.job_args.get('dataset_format') + + if dataset_format == 'libsvm': + self.model = self.controller.create_model_libsvm(self.data, list_params) + else: + self.model = self.controller.create_model(self.data, list_params) + + self.model.save(self.context, self.modelpath) + + def download_dataset(self): + + self.load_data() + self.save_data() + + def parse_dataset(self): + + self.load_data() + + dataset_param = self.job_args['dataset']['params'] + params = base64.b64decode(dataset_param) + list_params = literal_eval(params) + + cmd = '' + + for param in list_params: + cmd = cmd + '.' + param['method'] + '(' + param['args'] + ')' + + exec('self.data = self.data' + cmd) + self.save_data() + + def create_model(self): + + self.load_data() + self.create_and_save_model() + + def predict(self): + + predict_params = self.job_args['learning']['params'] + params = base64.b64decode(predict_params) + + self.model = self.controller.load_model(self.context, self.modelpath) + + dataset_format = self.job_args.get('dataset_format') + + if dataset_format == 'libsvm': + self.output = self.controller.predict_libsvm(self.model, params) + else: + self.output = self.controller.predict(self.model, params) + + print(self.output) + + +if __name__ == '__main__': + + meteos = MeteosSparkController() + meteos.parse_args(sys.argv) + meteos.init_model_controller() + meteos.init_context() + + getattr(meteos, meteos.job_args['method'])() diff --git a/meteos/cluster/sahara.py b/meteos/cluster/sahara.py new file mode 100644 index 0000000..78b05fb --- /dev/null +++ b/meteos/cluster/sahara.py @@ -0,0 +1,177 @@ +# Copyright 2014 Mirantis Inc. +# All Rights Reserved. +# Copyright (c) 2016 NEC Corporation. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +""" +Handles all requests relating to volumes + sahara. +""" + +import copy +import os + +from osc_lib import exceptions as sahara_exception +from saharaclient import client as sahara_client +from keystoneauth1 import loading as ks_loading +from oslo_config import cfg +import six + +from meteos.common import client_auth +from meteos.common.config import core_opts +import meteos.context as ctxt +from meteos.db import base +from meteos import exception +from meteos.i18n import _ + +SAHARA_GROUP = 'sahara' + +sahara_opts = [ + cfg.StrOpt('auth_url', + default='http://localhost:5000/v2.0', + help='Identity service URL.', + deprecated_group='DEFAULT') +] + +CONF = cfg.CONF +CONF.register_opts(core_opts) +CONF.register_opts(sahara_opts, SAHARA_GROUP) +ks_loading.register_session_conf_options(CONF, SAHARA_GROUP) +ks_loading.register_auth_conf_options(CONF, SAHARA_GROUP) + + +def list_opts(): + return client_auth.AuthClientLoader.list_opts(SAHARA_GROUP) + + +def saharaclient(context): + deprecated_opts_for_v2 = { + 'auth_url': CONF.sahara.auth_url, + 'token': context.auth_token, + 'tenant_id': context.tenant, + } + AUTH_OBJ = client_auth.AuthClientLoader( + client_class=sahara_client.Client, + exception_module=sahara_exception, + cfg_group=SAHARA_GROUP, + deprecated_opts_for_v2=deprecated_opts_for_v2, + url=CONF.sahara.auth_url, + token=context.auth_token) + return AUTH_OBJ.get_client(context) + + +class API(base.Base): + + """API for interacting with the data processing manager.""" + + def image_set(self, context, id, user_name): + item = saharaclient(context).images.update_image(id, user_name) + return item.image['id'] + + def image_tags_add(self, context, id, data): + saharaclient(context).images.update_tags(id, data) + + def image_remove(self, context, id): + saharaclient(context).images.unregister_image(id) + + def create_node_group_template(self, context, name, plugin_name, version, + flavor_id, node_processes, floating_ip_pool, + auto_security_group): + item = saharaclient(context).node_group_templates.create( + name, + plugin_name, + version, + flavor_id, + node_processes=node_processes, + floating_ip_pool=floating_ip_pool, + auto_security_group=auto_security_group) + + return item.id + + def delete_node_group_template(self, context, id): + saharaclient(context).node_group_templates.delete(id) + + def create_cluster_template(self, context, name, plugin_name, + version, node_groups): + item = saharaclient(context).cluster_templates.create( + name, + plugin_name, + version, + node_groups=node_groups) + + return item.id + + def delete_cluster_template(self, context, id): + saharaclient(context).cluster_templates.delete(id) + + def get_job_binary_data(self, context, id): + item = saharaclient(context).job_binary_internals.get(id) + return item.id + + def create_job_binary_data(self, context, name, data): + item = saharaclient(context).job_binary_internals.create(name, data) + return item.id + + def delete_job_binary_data(self, context, id): + saharaclient(context).job_binary_internals.delete(id) + + def create_job_binary(self, context, name, url): + item = saharaclient(context).job_binaries.create(name, url) + return item.id + + def delete_job_binary(self, context, id): + saharaclient(context).job_binaries.delete(id) + + def create_job_template(self, context, name, type, mains): + item = saharaclient(context).jobs.create(name, type, mains=mains) + return item.id + + def delete_job_template(self, context, id): + saharaclient(context).jobs.delete(id) + + def get_node_groups(self, context, id): + item = saharaclient(context).clusters.get(id) + return item.node_groups + + def create_cluster(self, context, name, plugin, version, image_id, + template_id, keypair, neutron_management_network): + + item = saharaclient(context).clusters.create( + name, + plugin, + version, + cluster_template_id=template_id, + default_image_id=image_id, + user_keypair_id=keypair, + net_id=neutron_management_network) + + return item.id + + def delete_cluster(self, context, id): + saharaclient(context).clusters.delete(id) + + def get_cluster(self, context, id): + item = saharaclient(context).clusters.get(id) + return item + + def job_create(self, context, job_template_id, cluster_id, configs): + item = saharaclient(context).job_executions.create( + job_template_id, cluster_id, configs=configs) + return item.id + + def job_delete(self, context, id): + saharaclient(context).job_executions.delete(id) + + def get_job(self, context, id): + item = saharaclient(context).job_executions.get(id) + return item diff --git a/meteos/cmd/__init__.py b/meteos/cmd/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/meteos/cmd/api.py b/meteos/cmd/api.py new file mode 100644 index 0000000..7283547 --- /dev/null +++ b/meteos/cmd/api.py @@ -0,0 +1,54 @@ +#!/usr/bin/env python + +# Copyright 2010 United States Government as represented by the +# Administrator of the National Aeronautics and Space Administration. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Starter script for meteos OS API.""" + +import eventlet +eventlet.monkey_patch() + +import sys + +from oslo_config import cfg +from oslo_log import log + +from meteos import i18n +i18n.enable_lazy() + +from meteos.common import config # Need to register global_opts # noqa +from meteos import service +from meteos import utils +from meteos import version + +CONF = cfg.CONF + + +def main(): + log.register_options(CONF) + CONF(sys.argv[1:], project='meteos', + version=version.version_string()) + log.setup(CONF, "meteos") + utils.monkey_patch() + + launcher = service.process_launcher() + server = service.WSGIService('osapi_learning') + launcher.launch_service(server, workers=server.workers or 1) + launcher.wait() + + +if __name__ == '__main__': + main() diff --git a/meteos/cmd/engine.py b/meteos/cmd/engine.py new file mode 100644 index 0000000..4730316 --- /dev/null +++ b/meteos/cmd/engine.py @@ -0,0 +1,60 @@ +#!/usr/bin/env python + +# Copyright 2013 NetApp +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +"""Starter script for meteos Learning.""" + +import eventlet +eventlet.monkey_patch() + +import sys + +from oslo_config import cfg +from oslo_log import log + +from meteos import i18n +i18n.enable_lazy() + +from meteos.common import config # Need to register global_opts # noqa +from meteos import service +from meteos import utils +from meteos import version + +CONF = cfg.CONF + + +def main(): + log.register_options(CONF) + CONF(sys.argv[1:], project='meteos', + version=version.version_string()) + log.setup(CONF, "meteos") + utils.monkey_patch() + launcher = service.process_launcher() + if CONF.enabled_learning_backends: + for backend in CONF.enabled_learning_backends: + host = "%s@%s" % (CONF.host, backend) + server = service.Service.create(host=host, + service_name=backend, + binary='meteos-engine') + launcher.launch_service(server) + else: + server = service.Service.create(binary='meteos-engine') + launcher.launch_service(server) + launcher.wait() + + +if __name__ == '__main__': + main() diff --git a/meteos/cmd/manage.py b/meteos/cmd/manage.py new file mode 100644 index 0000000..f59c4e2 --- /dev/null +++ b/meteos/cmd/manage.py @@ -0,0 +1,465 @@ +#!/usr/bin/env python + +# Copyright (c) 2011 X.commerce, a business unit of eBay Inc. +# Copyright 2010 United States Government as represented by the +# Administrator of the National Aeronautics and Space Administration. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +# Interactive shell based on Django: +# +# Copyright (c) 2005, the Lawrence Journal-World +# All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions are met: +# +# 1. Redistributions of source code must retain the above copyright notice, +# this list of conditions and the following disclaimer. +# +# 2. Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# +# 3. Neither the name of Django nor the names of its contributors may be +# used to endorse or promote products derived from this software without +# specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +# A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +# OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +# SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +# LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +# DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +# THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +""" + CLI interface for meteos management. +""" + +from __future__ import print_function + +import os +import sys + +from meteos import i18n +i18n.enable_lazy() + +from oslo_config import cfg +from oslo_log import log +from oslo_utils import uuidutils + +from meteos.common import config # Need to register global_opts # noqa +from meteos import context +from meteos import db +from meteos.db import migration +from meteos.i18n import _ +from meteos import utils +from meteos import version + +CONF = cfg.CONF + + +# Decorators for actions +def args(*args, **kwargs): + def _decorator(func): + func.__dict__.setdefault('args', []).insert(0, (args, kwargs)) + return func + return _decorator + + +def param2id(object_id): + """Helper function to convert various id types to internal id. + + args: [object_id], e.g. 'vol-0000000a' or 'volume-0000000a' or '10' + """ + if uuidutils.is_uuid_like(object_id): + return object_id + elif '-' in object_id: + # FIXME(ja): mapping occurs in nova? + pass + else: + return int(object_id) + + +class ShellCommands(object): + + def bpython(self): + """Runs a bpython shell. + + Falls back to Ipython/python shell if unavailable + """ + self.run('bpython') + + def ipython(self): + """Runs an Ipython shell. + + Falls back to Python shell if unavailable + """ + self.run('ipython') + + def python(self): + """Runs a python shell. + + Falls back to Python shell if unavailable + """ + self.run('python') + + @args('--shell', dest="shell", + metavar='', + help='Python shell') + def run(self, shell=None): + """Runs a Python interactive interpreter.""" + if not shell: + shell = 'bpython' + + if shell == 'bpython': + try: + import bpython + bpython.embed() + except ImportError: + shell = 'ipython' + if shell == 'ipython': + try: + from IPython import embed + embed() + except ImportError: + # Ipython < 0.11 + try: + import IPython + + # Explicitly pass an empty list as arguments, because + # otherwise IPython would use sys.argv from this script. + shell = IPython.Shell.IPShell(argv=[]) + shell.mainloop() + except ImportError: + # no IPython module + shell = 'python' + + if shell == 'python': + import code + try: + # Try activating rlcompleter, because it's handy. + import readline + except ImportError: + pass + else: + # We don't have to wrap the following import in a 'try', + # because we already know 'readline' was imported successfully. + import rlcompleter # noqa + readline.parse_and_bind("tab:complete") + code.interact() + + @args('--path', required=True, help='Script path') + def script(self, path): + """Runs the script from the specified path with flags set properly. + + arguments: path + """ + exec(compile(open(path).read(), path, 'exec'), locals(), globals()) + + +class HostCommands(object): + + """List hosts.""" + + @args('zone', nargs='?', default=None, + help='Availability Zone (default: %(default)s)') + def list(self, zone=None): + """Show a list of all physical hosts. Filter by zone. + + args: [zone] + """ + print("%-25s\t%-15s" % (_('host'), _('zone'))) + ctxt = context.get_admin_context() + services = db.service_get_all(ctxt) + if zone: + services = [ + s for s in services if s['availability_zone']['name'] == zone] + hosts = [] + for srv in services: + if not [h for h in hosts if h['host'] == srv['host']]: + hosts.append(srv) + + for h in hosts: + print("%-25s\t%-15s" % (h['host'], h['availability_zone']['name'])) + + +class DbCommands(object): + + """Class for managing the database.""" + + def __init__(self): + pass + + @args('version', nargs='?', default=None, + help='Database version') + def sync(self, version=None): + """Sync the database up to the most recent version.""" + return migration.upgrade(version) + + def version(self): + """Print the current database version.""" + print(migration.version()) + + # NOTE(imalinovskiy): + # Meteos init migration hardcoded here, + # because alembic has strange behaviour: + # downgrade base = downgrade from head(001) -> base(001) + # = downgrade from 001 -> (empty) [ERROR] + # downgrade 001 = downgrade from head(001)->001 + # = do nothing [OK] + @args('version', nargs='?', default='001', + help='Version to downgrade') + def downgrade(self, version=None): + """Downgrade database to the given version.""" + return migration.downgrade(version) + + @args('--message', help='Revision message') + @args('--autogenerate', help='Autogenerate migration from schema') + def revision(self, message, autogenerate): + """Generate new migration.""" + return migration.revision(message, autogenerate) + + @args('version', nargs='?', default=None, + help='Version to stamp version table with') + def stamp(self, version=None): + """Stamp the version table with the given version.""" + return migration.stamp(version) + + +class VersionCommands(object): + + """Class for exposing the codebase version.""" + + def list(self): + print(version.version_string()) + + def __call__(self): + self.list() + + +class ConfigCommands(object): + + """Class for exposing the flags defined by flag_file(s).""" + + def list(self): + for key, value in CONF.items(): + if value is not None: + print('%s = %s' % (key, value)) + + +class GetLogCommands(object): + + """Get logging information.""" + + def errors(self): + """Get all of the errors from the log files.""" + error_found = 0 + if CONF.log_dir: + logs = [x for x in os.listdir(CONF.log_dir) if x.endswith('.log')] + for file in logs: + log_file = os.path.join(CONF.log_dir, file) + lines = [line.strip() for line in open(log_file, "r")] + lines.reverse() + print_name = 0 + for index, line in enumerate(lines): + if line.find(" ERROR ") > 0: + error_found += 1 + if print_name == 0: + print(log_file + ":-") + print_name = 1 + print("Line %d : %s" % (len(lines) - index, line)) + if error_found == 0: + print("No errors in logfiles!") + + @args('num_entries', nargs='?', type=int, default=10, + help='Number of entries to list (default: %(default)d)') + def syslog(self, num_entries=10): + """Get of the meteos syslog events.""" + entries = int(num_entries) + count = 0 + log_file = '' + if os.path.exists('/var/log/syslog'): + log_file = '/var/log/syslog' + elif os.path.exists('/var/log/messages'): + log_file = '/var/log/messages' + else: + print("Unable to find system log file!") + sys.exit(1) + lines = [line.strip() for line in open(log_file, "r")] + lines.reverse() + print("Last %s meteos syslog entries:-" % (entries)) + for line in lines: + if line.find("meteos") > 0: + count += 1 + print("%s" % (line)) + if count == entries: + break + + if count == 0: + print("No meteos entries in syslog!") + + +class ServiceCommands(object): + + """Methods for managing services.""" + + def list(self): + """Show a list of all meteos services.""" + ctxt = context.get_admin_context() + services = db.service_get_all(ctxt) + print_format = "%-16s %-36s %-16s %-10s %-5s %-10s" + print(print_format % ( + _('Binary'), + _('Host'), + _('Zone'), + _('Status'), + _('State'), + _('Updated At')) + ) + for svc in services: + alive = utils.service_is_up(svc) + art = ":-)" if alive else "XXX" + status = 'enabled' + if svc['disabled']: + status = 'disabled' + print(print_format % ( + svc['binary'], + svc['host'].partition('.')[0], + svc['availability_zone']['name'], + status, + art, + svc['updated_at'], + )) + + +CATEGORIES = { + 'config': ConfigCommands, + 'db': DbCommands, + 'host': HostCommands, + 'logs': GetLogCommands, + 'service': ServiceCommands, + 'shell': ShellCommands, + 'version': VersionCommands +} + + +def methods_of(obj): + """Get all callable methods of an object that don't start with underscore. + + Returns a list of tuples of the form (method_name, method). + """ + result = [] + for i in dir(obj): + if callable(getattr(obj, i)) and not i.startswith('_'): + result.append((i, getattr(obj, i))) + return result + + +def add_command_parsers(subparsers): + for category in CATEGORIES: + command_object = CATEGORIES[category]() + + parser = subparsers.add_parser(category) + parser.set_defaults(command_object=command_object) + + category_subparsers = parser.add_subparsers(dest='action') + + for (action, action_fn) in methods_of(command_object): + parser = category_subparsers.add_parser(action) + + action_kwargs = [] + for args, kwargs in getattr(action_fn, 'args', []): + parser.add_argument(*args, **kwargs) + + parser.set_defaults(action_fn=action_fn) + parser.set_defaults(action_kwargs=action_kwargs) + + +category_opt = cfg.SubCommandOpt('category', + title='Command categories', + handler=add_command_parsers) + + +def get_arg_string(args): + arg = None + if args[0] == '-': + # (Note)zhiteng: args starts with CONF.oparser.prefix_chars + # is optional args. Notice that cfg module takes care of + # actual ArgParser so prefix_chars is always '-'. + if args[1] == '-': + # This is long optional arg + arg = args[2:] + else: + arg = args[1:] + else: + arg = args + + return arg + + +def fetch_func_args(func): + fn_args = [] + for args, kwargs in getattr(func, 'args', []): + arg = get_arg_string(args[0]) + fn_args.append(getattr(CONF.category, arg)) + + return fn_args + + +def main(): + """Parse options and call the appropriate class/method.""" + CONF.register_cli_opt(category_opt) + script_name = sys.argv[0] + if len(sys.argv) < 2: + print(_("\nOpenStack meteos version: %(version)s\n") % + {'version': version.version_string()}) + print(script_name + " category action []") + print(_("Available categories:")) + for category in CATEGORIES: + print("\t%s" % category) + sys.exit(2) + + try: + log.register_options(CONF) + CONF(sys.argv[1:], project='meteos', + version=version.version_string()) + log.setup(CONF, "meteos") + except cfg.ConfigFilesNotFoundError: + cfgfile = CONF.config_file[-1] if CONF.config_file else None + if cfgfile and not os.access(cfgfile, os.R_OK): + st = os.stat(cfgfile) + print(_("Could not read %s. Re-running with sudo") % cfgfile) + try: + os.execvp('sudo', ['sudo', '-u', '#%s' % st.st_uid] + sys.argv) + except Exception: + print(_('sudo failed, continuing as if nothing happened')) + + print(_('Please re-run meteos-manage as root.')) + sys.exit(2) + + fn = CONF.category.action_fn + + fn_args = fetch_func_args(fn) + fn(*fn_args) + + +if __name__ == '__main__': + main() diff --git a/meteos/common/__init__.py b/meteos/common/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/meteos/common/client_auth.py b/meteos/common/client_auth.py new file mode 100644 index 0000000..2ed45c6 --- /dev/null +++ b/meteos/common/client_auth.py @@ -0,0 +1,107 @@ +# Copyright 2016 SAP SE +# All Rights Reserved +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +import copy + +from keystoneauth1 import loading as ks_loading +from keystoneauth1.loading._plugins.identity import v2 +from oslo_config import cfg +from oslo_log import log + +from meteos import exception +from meteos.i18n import _, _LW + +CONF = cfg.CONF +LOG = log.getLogger(__name__) + +"""Helper class to support keystone v2 and v3 for clients + +Builds auth and session context before instantiation of the actual +client. In order to build this context a dedicated config group is +needed to load all needed parameters dynamically. + + +""" + + +class AuthClientLoader(object): + + def __init__(self, client_class, exception_module, cfg_group, + deprecated_opts_for_v2=None, url=None, token=None): + self.client_class = client_class + self.exception_module = exception_module + self.group = cfg_group + self.admin_auth = None + self.conf = CONF + self.session = None + self.auth_plugin = None + self.deprecated_opts_for_v2 = deprecated_opts_for_v2 + self.url = url + self.token = token + + @staticmethod + def list_opts(group): + """Generates a list of config option for a given group + + :param group: group name + :return: list of auth default configuration + """ + opts = copy.deepcopy(ks_loading.register_session_conf_options( + CONF, group)) + opts.insert(0, ks_loading.get_auth_common_conf_options()[0]) + + for plugin_option in ks_loading.get_auth_plugin_conf_options( + 'password'): + found = False + for option in opts: + if option.name == plugin_option.name: + found = True + break + if not found: + opts.append(plugin_option) + opts.sort(key=lambda x: x.name) + return [(group, opts)] + + def _load_auth_plugin(self): + if self.admin_auth: + return self.admin_auth + self.auth_plugin = ks_loading.load_auth_from_conf_options( + CONF, self.group) + + self.auth_plugin = v2.Token().load_from_options( + **self.deprecated_opts_for_v2) + + if self.auth_plugin: + return self.auth_plugin + + msg = _('Cannot load auth plugin for %s') % self.group + raise self.exception_module.Unauthorized(message=msg) + + def get_client(self, context, admin=False, **kwargs): + """Get's the client with the correct auth/session context + + """ + if not self.session: + self.session = ks_loading.load_session_from_conf_options( + self.conf, self.group) + + if not self.admin_auth: + self.admin_auth = self._load_auth_plugin() + auth_plugin = self.admin_auth + + return self.client_class(version='1.0', + session=self.session, + auth=auth_plugin, + **kwargs) diff --git a/meteos/common/config.py b/meteos/common/config.py new file mode 100644 index 0000000..1897aff --- /dev/null +++ b/meteos/common/config.py @@ -0,0 +1,178 @@ +# Copyright 2010 United States Government as represented by the +# Administrator of the National Aeronautics and Space Administration. +# All Rights Reserved. +# Copyright 2012 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +"""Command-line flag library. + +Emulates gflags by wrapping cfg.ConfigOpts. + +The idea is to move fully to cfg eventually, and this wrapper is a +stepping stone. + +""" + +import socket + +from oslo_config import cfg +from oslo_log import log +from oslo_middleware import cors +from oslo_utils import netutils +import six + +from meteos.common import constants +from meteos import exception +from meteos.i18n import _ + +CONF = cfg.CONF +log.register_options(CONF) + + +core_opts = [ + cfg.StrOpt('api_paste_config', + default="api-paste.ini", + help='File name for the paste.deploy config for meteos-api.'), + cfg.StrOpt('state_path', + default='/var/lib/meteos', + help="Top-level directory for maintaining meteos's state."), + cfg.StrOpt('os_region_name', + help='Region name of this node.'), +] + +debug_opts = [ +] + +CONF.register_cli_opts(core_opts) +CONF.register_cli_opts(debug_opts) + +global_opts = [ + cfg.StrOpt('my_ip', + default=netutils.get_my_ipv4(), + sample_default='', + help='IP address of this host.'), + cfg.StrOpt('scheduler_topic', + default='meteos-scheduler', + help='The topic scheduler nodes listen on.'), + cfg.StrOpt('learning_topic', + default='meteos-engine', + help='The topic learning nodes listen on.'), + cfg.StrOpt('data_topic', + default='meteos-data', + help='The topic data nodes listen on.'), + cfg.BoolOpt('api_rate_limit', + default=True, + help='Whether to rate limit the API.'), + cfg.ListOpt('osapi_learning_ext_list', + default=[], + help='Specify list of extensions to load when using osapi_' + 'learning_extension option with meteos.api.contrib.' + 'select_extensions.'), + cfg.ListOpt('osapi_learning_extension', + default=['meteos.api.contrib.standard_extensions'], + help='The osapi learning extensions to load.'), + cfg.StrOpt('sqlite_db', + default='meteos.sqlite', + help='The filename to use with sqlite.'), + cfg.BoolOpt('sqlite_synchronous', + default=True, + help='If passed, use synchronous mode for sqlite.'), + cfg.IntOpt('sql_idle_timeout', + default=3600, + help='Timeout before idle SQL connections are reaped.'), + cfg.IntOpt('sql_max_retries', + default=10, + help='Maximum database connection retries during startup. ' + '(setting -1 implies an infinite retry count).'), + cfg.IntOpt('sql_retry_interval', + default=10, + help='Interval between retries of opening a SQL connection.'), + cfg.StrOpt('engine_manager', + default='meteos.engine.manager.LearningManager', + help='Full class name for the learning manager.'), + cfg.StrOpt('host', + default=socket.gethostname(), + sample_default='', + help='Name of this node. This can be an opaque identifier. ' + 'It is not necessarily a hostname, FQDN, or IP address.'), + # NOTE(vish): default to nova for compatibility with nova installs + cfg.StrOpt('storage_availability_zone', + default='nova', + help='Availability zone of this node.'), + cfg.StrOpt('default_learning_type', + help='Default learning type to use.'), + cfg.ListOpt('memcached_servers', + help='Memcached servers or None for in process cache.'), + cfg.StrOpt('learning_usage_audit_period', + default='month', + help='Time period to generate learning usages for. ' + 'Time period must be hour, day, month or year.'), + cfg.StrOpt('root_helper', + default='sudo', + help='Deprecated: command to use for running commands as ' + 'root.'), + cfg.StrOpt('rootwrap_config', + help='Path to the rootwrap configuration file to use for ' + 'running commands as root.'), + cfg.BoolOpt('monkey_patch', + default=False, + help='Whether to log monkey patching.'), + cfg.ListOpt('monkey_patch_modules', + default=[], + help='List of modules or decorators to monkey patch.'), + cfg.IntOpt('service_down_time', + default=60, + help='Maximum time since last check-in for up service.'), + cfg.StrOpt('learning_api_class', + default='meteos.engine.api.API', + help='The full class name of the learning API class to use.'), + cfg.StrOpt('auth_strategy', + default='keystone', + help='The strategy to use for auth. Supports noauth, keystone, ' + 'and deprecated.'), + cfg.ListOpt('enabled_learning_backends', + help='A list of learning backend names to use. These backend ' + 'names should be backed by a unique [CONFIG] group ' + 'with its options.'), +] + +CONF.register_opts(global_opts) + + +def set_middleware_defaults(): + """Update default configuration options for oslo.middleware.""" + # CORS Defaults + # TODO(krotscheck): Update with https://review.openstack.org/#/c/285368/ + cfg.set_defaults(cors.CORS_OPTS, + allow_headers=['X-Auth-Token', + 'X-OpenStack-Request-ID', + 'X-Openstack-Meteos-Api-Version', + 'X-OpenStack-Meteos-API-Experimental', + 'X-Identity-Status', + 'X-Roles', + 'X-Service-Catalog', + 'X-User-Id', + 'X-Tenant-Id'], + expose_headers=['X-Auth-Token', + 'X-OpenStack-Request-ID', + 'X-Openstack-Meteos-Api-Version', + 'X-OpenStack-Meteos-API-Experimental', + 'X-Subject-Token', + 'X-Service-Token'], + allow_methods=['GET', + 'PUT', + 'POST', + 'DELETE', + 'PATCH'] + ) diff --git a/meteos/common/constants.py b/meteos/common/constants.py new file mode 100644 index 0000000..4a18903 --- /dev/null +++ b/meteos/common/constants.py @@ -0,0 +1,28 @@ +# Copyright 2013 OpenStack Foundation +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +STATUS_NEW = 'new' +STATUS_CREATING = 'creating' +STATUS_DELETING = 'deleting' +STATUS_DELETED = 'deleted' +STATUS_ERROR = 'error' +STATUS_ERROR_DELETING = 'error_deleting' +STATUS_AVAILABLE = 'available' +STATUS_ACTIVE = 'active' +STATUS_INACTIVE = 'inactive' +STATUS_UPDATING = 'updating' +STATUS_SAHARA_ACTIVE = 'Active' +STATUS_JOB_SUCCESS = 'SUCCEEDED' +STATUS_JOB_ERROR = 'DONEWITHERROR' diff --git a/meteos/context.py b/meteos/context.py new file mode 100644 index 0000000..b2cfcb3 --- /dev/null +++ b/meteos/context.py @@ -0,0 +1,151 @@ +# Copyright 2011 OpenStack LLC. +# Copyright 2010 United States Government as represented by the +# Administrator of the National Aeronautics and Space Administration. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +"""RequestContext: context for requests that persist through all of meteos.""" + +import copy + +from oslo_context import context +from oslo_log import log +from oslo_utils import timeutils +import six + +from meteos.i18n import _, _LW +from meteos import policy + +LOG = log.getLogger(__name__) + + +class RequestContext(context.RequestContext): + """Security context and request information. + + Represents the user taking a given action within the system. + + """ + + def __init__(self, user_id, project_id, is_admin=None, read_deleted="no", + roles=None, remote_address=None, timestamp=None, + request_id=None, auth_token=None, overwrite=True, + quota_class=None, service_catalog=None, **kwargs): + """Initialize RequestContext. + + :param read_deleted: 'no' indicates deleted records are hidden, 'yes' + indicates deleted records are visible, 'only' indicates that + *only* deleted records are visible. + + :param overwrite: Set to False to ensure that the greenthread local + copy of the index is not overwritten. + + :param kwargs: Extra arguments that might be present, but we ignore + because they possibly came in from older rpc messages. + """ + + user = kwargs.pop('user', None) + tenant = kwargs.pop('tenant', None) + super(RequestContext, self).__init__( + auth_token=auth_token, + user=user_id or user, + tenant=project_id or tenant, + domain=kwargs.pop('domain', None), + user_domain=kwargs.pop('user_domain', None), + project_domain=kwargs.pop('project_domain', None), + is_admin=is_admin, + read_only=kwargs.pop('read_only', False), + show_deleted=kwargs.pop('show_deleted', False), + request_id=request_id, + resource_uuid=kwargs.pop('resource_uuid', None), + overwrite=overwrite, + roles=roles) + + kwargs.pop('user_identity', None) + if kwargs: + LOG.warning(_LW('Arguments dropped when creating context: %s.'), + str(kwargs)) + self.user_id = self.user + self.project_id = self.tenant + + if self.is_admin is None: + self.is_admin = policy.check_is_admin(self.roles) + elif self.is_admin and 'admin' not in self.roles: + self.roles.append('admin') + self.read_deleted = read_deleted + self.remote_address = remote_address + if not timestamp: + timestamp = timeutils.utcnow() + if isinstance(timestamp, six.string_types): + timestamp = timeutils.parse_strtime(timestamp) + self.timestamp = timestamp + if service_catalog: + self.service_catalog = [s for s in service_catalog + if s.get('type') in ('compute', 'volume')] + else: + self.service_catalog = [] + + self.quota_class = quota_class + + def _get_read_deleted(self): + return self._read_deleted + + def _set_read_deleted(self, read_deleted): + if read_deleted not in ('no', 'yes', 'only'): + raise ValueError(_("read_deleted can only be one of 'no', " + "'yes' or 'only', not %r") % read_deleted) + self._read_deleted = read_deleted + + def _del_read_deleted(self): + del self._read_deleted + + read_deleted = property(_get_read_deleted, _set_read_deleted, + _del_read_deleted) + + def to_dict(self): + values = super(RequestContext, self).to_dict() + values.update({ + 'user_id': getattr(self, 'user_id', None), + 'project_id': getattr(self, 'project_id', None), + 'read_deleted': getattr(self, 'read_deleted', None), + 'remote_address': getattr(self, 'remote_address', None), + 'timestamp': self.timestamp.isoformat() if hasattr( + self, 'timestamp') else None, + 'quota_class': getattr(self, 'quota_class', None), + 'service_catalog': getattr(self, 'service_catalog', None)}) + return values + + @classmethod + def from_dict(cls, values): + return cls(**values) + + def elevated(self, read_deleted=None, overwrite=False): + """Return a version of this context with admin flag set.""" + ctx = copy.deepcopy(self) + ctx.is_admin = True + + if 'admin' not in ctx.roles: + ctx.roles.append('admin') + + if read_deleted is not None: + ctx.read_deleted = read_deleted + + return ctx + + +def get_admin_context(read_deleted="no"): + return RequestContext(user_id=None, + project_id=None, + is_admin=True, + read_deleted=read_deleted, + overwrite=False) diff --git a/meteos/db/__init__.py b/meteos/db/__init__.py new file mode 100644 index 0000000..aeb54de --- /dev/null +++ b/meteos/db/__init__.py @@ -0,0 +1,20 @@ +# Copyright 2010 United States Government as represented by the +# Administrator of the National Aeronautics and Space Administration. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. +""" +DB abstraction for Meteos +""" + +from meteos.db.api import * # noqa diff --git a/meteos/db/api.py b/meteos/db/api.py new file mode 100644 index 0000000..eb7a02d --- /dev/null +++ b/meteos/db/api.py @@ -0,0 +1,328 @@ +# Copyright (c) 2011 X.commerce, a business unit of eBay Inc. +# Copyright 2010 United States Government as represented by the +# Administrator of the National Aeronautics and Space Administration. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +"""Defines interface for DB access. + +The underlying driver is loaded as a :class:`LazyPluggable`. + +Functions in this module are imported into the meteos.db namespace. Call these +functions from meteos.db namespace, not the meteos.db.api namespace. + +All functions in this module return objects that implement a dictionary-like +interface. Currently, many of these objects are sqlalchemy objects that +implement a dictionary interface. However, a future goal is to have all of +these objects be simple dictionaries. + + +**Related Flags** + +:backend: string to lookup in the list of LazyPluggable backends. + `sqlalchemy` is the only supported backend right now. + +:connection: string specifying the sqlalchemy connection to use, like: + `sqlite:///var/lib/meteos/meteos.sqlite`. + +:enable_new_services: when adding a new service to the database, is it in the + pool of available hardware (Default: True) + +""" +from oslo_config import cfg +from oslo_db import api as db_api + +db_opts = [ + cfg.StrOpt('db_backend', + default='sqlalchemy', + help='The backend to use for database.'), + cfg.BoolOpt('enable_new_services', + default=True, + help='Services to be added to the available pool on create.'), + cfg.StrOpt('learning_name_template', + default='learning-%s', + help='Template string to be used to generate learning names.'), + cfg.StrOpt('learning_snapshot_name_template', + default='learning-snapshot-%s', + help='Template string to be used to generate learning snapshot ' + 'names.'), +] + +CONF = cfg.CONF +CONF.register_opts(db_opts) + +_BACKEND_MAPPING = {'sqlalchemy': 'meteos.db.sqlalchemy.api'} +IMPL = db_api.DBAPI.from_config(cfg.CONF, backend_mapping=_BACKEND_MAPPING, + lazy=True) + + +def authorize_project_context(context, project_id): + """Ensures a request has permission to access the given project.""" + return IMPL.authorize_project_context(context, project_id) + + +def authorize_quota_class_context(context, class_name): + """Ensures a request has permission to access the given quota class.""" + return IMPL.authorize_quota_class_context(context, class_name) + + +# + + +def service_destroy(context, service_id): + """Destroy the service or raise if it does not exist.""" + return IMPL.service_destroy(context, service_id) + + +def service_get(context, service_id): + """Get a service or raise if it does not exist.""" + return IMPL.service_get(context, service_id) + + +def service_get_by_host_and_topic(context, host, topic): + """Get a service by host it's on and topic it listens to.""" + return IMPL.service_get_by_host_and_topic(context, host, topic) + + +def service_get_all(context, disabled=None): + """Get all services.""" + return IMPL.service_get_all(context, disabled) + + +def service_get_all_by_topic(context, topic): + """Get all services for a given topic.""" + return IMPL.service_get_all_by_topic(context, topic) + + +def service_get_all_learning_sorted(context): + """Get all learning services sorted by learning count. + + :returns: a list of (Service, learning_count) tuples. + + """ + return IMPL.service_get_all_learning_sorted(context) + + +def service_get_by_args(context, host, binary): + """Get the state of an service by node name and binary.""" + return IMPL.service_get_by_args(context, host, binary) + + +def service_create(context, values): + """Create a service from the values dictionary.""" + return IMPL.service_create(context, values) + + +def service_update(context, service_id, values): + """Set the given properties on an service and update it. + + Raises NotFound if service does not exist. + + """ + return IMPL.service_update(context, service_id, values) + + +# + + +def experiment_create(context, experiment_values): + """Create new experiment.""" + return IMPL.experiment_create(context, experiment_values) + + +def experiment_update(context, experiment_id, values): + """Update experiment fields.""" + return IMPL.experiment_update(context, experiment_id, values) + + +def experiment_get(context, experiment_id): + """Get experiment by id.""" + return IMPL.experiment_get(context, experiment_id) + + +def experiment_get_all(context, filters=None, sort_key=None, sort_dir=None): + """Get all experiments.""" + return IMPL.experiment_get_all( + context, filters=filters, sort_key=sort_key, sort_dir=sort_dir, + ) + + +def experiment_get_all_by_project(context, project_id, filters=None, + sort_key=None, sort_dir=None): + """Returns all experiments with given project ID.""" + return IMPL.experiment_get_all_by_project( + context, project_id, filters=filters, + sort_key=sort_key, sort_dir=sort_dir, + ) + + +def experiment_delete(context, experiment_id): + """Delete experiment.""" + return IMPL.experiment_delete(context, experiment_id) + + +# + + +def template_create(context, template_values): + """Create new template.""" + return IMPL.template_create(context, template_values) + + +def template_update(context, template_id, values): + """Update template fields.""" + return IMPL.template_update(context, template_id, values) + + +def template_get(context, template_id): + """Get template by id.""" + return IMPL.template_get(context, template_id) + + +def template_get_all(context, filters=None, sort_key=None, sort_dir=None): + """Get all templates.""" + return IMPL.template_get_all( + context, filters=filters, sort_key=sort_key, sort_dir=sort_dir, + ) + + +def template_get_all_by_project(context, project_id, filters=None, + sort_key=None, sort_dir=None): + """Returns all templates with given project ID.""" + return IMPL.template_get_all_by_project( + context, project_id, filters=filters, + sort_key=sort_key, sort_dir=sort_dir, + ) + + +def template_delete(context, template_id): + """Delete template.""" + return IMPL.template_delete(context, template_id) + + +# + + +def dataset_create(context, dataset_values): + """Create new dataset.""" + return IMPL.dataset_create(context, dataset_values) + + +def dataset_update(context, dataset_id, values): + """Update dataset fields.""" + return IMPL.dataset_update(context, dataset_id, values) + + +def dataset_get(context, dataset_id): + """Get dataset by id.""" + return IMPL.dataset_get(context, dataset_id) + + +def dataset_get_all(context, filters=None, sort_key=None, sort_dir=None): + """Get all datasets.""" + return IMPL.dataset_get_all( + context, filters=filters, sort_key=sort_key, sort_dir=sort_dir, + ) + + +def dataset_get_all_by_project(context, project_id, filters=None, + sort_key=None, sort_dir=None): + """Returns all datasets with given project ID.""" + return IMPL.dataset_get_all_by_project( + context, project_id, filters=filters, + sort_key=sort_key, sort_dir=sort_dir, + ) + + +def dataset_delete(context, dataset_id): + """Delete dataset.""" + return IMPL.dataset_delete(context, dataset_id) + + +# + + +def model_create(context, model_values): + """Create new model.""" + return IMPL.model_create(context, model_values) + + +def model_update(context, model_id, values): + """Update model fields.""" + return IMPL.model_update(context, model_id, values) + + +def model_get(context, model_id): + """Get model by id.""" + return IMPL.model_get(context, model_id) + + +def model_get_all(context, filters=None, sort_key=None, sort_dir=None): + """Get all models.""" + return IMPL.model_get_all( + context, filters=filters, sort_key=sort_key, sort_dir=sort_dir, + ) + + +def model_get_all_by_project(context, project_id, filters=None, + sort_key=None, sort_dir=None): + """Returns all models with given project ID.""" + return IMPL.model_get_all_by_project( + context, project_id, filters=filters, + sort_key=sort_key, sort_dir=sort_dir, + ) + + +def model_delete(context, model_id): + """Delete model.""" + return IMPL.model_delete(context, model_id) + + +# + + +def learning_create(context, learning_values): + """Create new learning.""" + return IMPL.learning_create(context, learning_values) + + +def learning_update(context, learning_id, values): + """Update learning fields.""" + return IMPL.learning_update(context, learning_id, values) + + +def learning_get(context, learning_id): + """Get learning by id.""" + return IMPL.learning_get(context, learning_id) + + +def learning_get_all(context, filters=None, sort_key=None, sort_dir=None): + """Get all learnings.""" + return IMPL.learning_get_all( + context, filters=filters, sort_key=sort_key, sort_dir=sort_dir, + ) + + +def learning_get_all_by_project(context, project_id, filters=None, + sort_key=None, sort_dir=None): + """Returns all learnings with given project ID.""" + return IMPL.learning_get_all_by_project( + context, project_id, filters=filters, + sort_key=sort_key, sort_dir=sort_dir, + ) + + +def learning_delete(context, learning_id): + """Delete learning.""" + return IMPL.learning_delete(context, learning_id) diff --git a/meteos/db/base.py b/meteos/db/base.py new file mode 100644 index 0000000..405a34c --- /dev/null +++ b/meteos/db/base.py @@ -0,0 +1,37 @@ +# Copyright 2010 United States Government as represented by the +# Administrator of the National Aeronautics and Space Administration. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +"""Base class for classes that need modular database access.""" + +from oslo_config import cfg +from oslo_utils import importutils + +db_driver_opt = cfg.StrOpt('db_driver', + default='meteos.db', + help='Driver to use for database access.') + +CONF = cfg.CONF +CONF.register_opt(db_driver_opt) + + +class Base(object): + """DB driver is injected in the init method.""" + + def __init__(self, db_driver=None): + super(Base, self).__init__() + if not db_driver: + db_driver = CONF.db_driver + self.db = importutils.import_module(db_driver) # pylint: disable=C0103 diff --git a/meteos/db/migration.py b/meteos/db/migration.py new file mode 100644 index 0000000..0a596c3 --- /dev/null +++ b/meteos/db/migration.py @@ -0,0 +1,48 @@ +# Copyright 2010 United States Government as represented by the +# Administrator of the National Aeronautics and Space Administration. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +"""Database setup and migration commands.""" + +from meteos import utils + + +IMPL = utils.LazyPluggable( + 'db_backend', sqlalchemy='meteos.db.migrations.alembic.migration') + + +def upgrade(version): + """Upgrade database to 'version' or the most recent version.""" + return IMPL.upgrade(version) + + +def downgrade(version): + """Downgrade database to 'version' or to initial state.""" + return IMPL.downgrade(version) + + +def version(): + """Display the current database version.""" + return IMPL.version() + + +def stamp(version): + """Stamp database with 'version' or the most recent version.""" + return IMPL.stamp(version) + + +def revision(message, autogenerate): + """Generate new migration script.""" + return IMPL.revision(message, autogenerate) diff --git a/meteos/db/migrations/__init__.py b/meteos/db/migrations/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/meteos/db/migrations/alembic.ini b/meteos/db/migrations/alembic.ini new file mode 100644 index 0000000..d39bdf9 --- /dev/null +++ b/meteos/db/migrations/alembic.ini @@ -0,0 +1,59 @@ +# A generic, single database configuration. + +[alembic] +# path to migration scripts +script_location = %(here)s/alembic + +# template used to generate migration files +# file_template = %%(rev)s_%%(slug)s + +# max length of characters to apply to the +# "slug" field +#truncate_slug_length = 40 + +# set to 'true' to run the environment during +# the 'revision' command, regardless of autogenerate +# revision_environment = false + +# set to 'true' to allow .pyc and .pyo files without +# a source .py file to be detected as revisions in the +# versions/ directory +# sourceless = false + +#sqlalchemy.url = driver://user:pass@localhost/dbname + + +# Logging configuration +[loggers] +keys = root,sqlalchemy,alembic + +[handlers] +keys = console + +[formatters] +keys = generic + +[logger_root] +level = WARN +handlers = console +qualname = + +[logger_sqlalchemy] +level = WARN +handlers = +qualname = sqlalchemy.engine + +[logger_alembic] +level = INFO +handlers = +qualname = alembic + +[handler_console] +class = StreamHandler +args = (sys.stderr,) +level = NOTSET +formatter = generic + +[formatter_generic] +format = %(levelname)-5.5s [%(name)s] %(message)s +datefmt = %H:%M:%S diff --git a/meteos/db/migrations/alembic/__init__.py b/meteos/db/migrations/alembic/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/meteos/db/migrations/alembic/env.py b/meteos/db/migrations/alembic/env.py new file mode 100644 index 0000000..7f7a7ce --- /dev/null +++ b/meteos/db/migrations/alembic/env.py @@ -0,0 +1,41 @@ +# Copyright 2014 Mirantis Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +from __future__ import with_statement + +from alembic import context + +from meteos.db.sqlalchemy import api as db_api +from meteos.db.sqlalchemy import models as db_models + + +def run_migrations_online(): + """Run migrations in 'online' mode. + + In this scenario we need to create an Engine + and associate a connection with the context. + """ + engine = db_api.get_engine() + connection = engine.connect() + target_metadata = db_models.MeteosBase.metadata + context.configure(connection=connection, # pylint: disable=E1101 + target_metadata=target_metadata) + try: + with context.begin_transaction(): # pylint: disable=E1101 + context.run_migrations() # pylint: disable=E1101 + finally: + connection.close() + + +run_migrations_online() diff --git a/meteos/db/migrations/alembic/migration.py b/meteos/db/migrations/alembic/migration.py new file mode 100644 index 0000000..6da398a --- /dev/null +++ b/meteos/db/migrations/alembic/migration.py @@ -0,0 +1,84 @@ +# Copyright 2014 Mirantis Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +import os + +import alembic +from alembic import config as alembic_config +import alembic.migration as alembic_migration +from oslo_config import cfg + +from meteos.db.sqlalchemy import api as db_api + +CONF = cfg.CONF + + +def _alembic_config(): + path = os.path.join(os.path.dirname(__file__), os.pardir, 'alembic.ini') + config = alembic_config.Config(path) + return config + + +def version(): + """Current database version. + + :returns: Database version + :rtype: string + """ + engine = db_api.get_engine() + with engine.connect() as conn: + context = alembic_migration.MigrationContext.configure(conn) + return context.get_current_revision() + + +def upgrade(revision): + """Upgrade database. + + :param version: Desired database version + :type version: string + """ + return alembic.command.upgrade(_alembic_config(), revision or 'head') + + +def downgrade(revision): + """Downgrade database. + + :param version: Desired database version + :type version: string + """ + return alembic.command.downgrade(_alembic_config(), revision or 'base') + + +def stamp(revision): + """Stamp database with provided revision. + + Don't run any migrations. + + :param revision: Should match one from repository or head - to stamp + database with most recent revision + :type revision: string + """ + return alembic.command.stamp(_alembic_config(), revision or 'head') + + +def revision(message=None, autogenerate=False): + """Create template for migration. + + :param message: Text that will be used for migration title + :type message: string + :param autogenerate: If True - generates diff based on current database + state + :type autogenerate: bool + """ + return alembic.command.revision(_alembic_config(), message, autogenerate) diff --git a/meteos/db/migrations/alembic/script.py.mako b/meteos/db/migrations/alembic/script.py.mako new file mode 100644 index 0000000..1189a94 --- /dev/null +++ b/meteos/db/migrations/alembic/script.py.mako @@ -0,0 +1,34 @@ +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +"""${message} + +Revision ID: ${up_revision} +Revises: ${down_revision} +Create Date: ${create_date} + +""" + +# revision identifiers, used by Alembic. +revision = ${repr(up_revision)} +down_revision = ${repr(down_revision)} + +from alembic import op +import sqlalchemy as sa +${imports if imports else ""} + +def upgrade(): + ${upgrades if upgrades else "pass"} + + +def downgrade(): + ${downgrades if downgrades else "pass"} diff --git a/meteos/db/migrations/alembic/versions/001_meteos_init.py b/meteos/db/migrations/alembic/versions/001_meteos_init.py new file mode 100644 index 0000000..303b72c --- /dev/null +++ b/meteos/db/migrations/alembic/versions/001_meteos_init.py @@ -0,0 +1,228 @@ +# Copyright 2012 OpenStack LLC. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +"""meteos_init + +Revision ID: 001 +Revises: None +Create Date: 2016-09-27 17:51:57.077203 + +""" + +# revision identifiers, used by Alembic. +revision = '001' +down_revision = None + +from alembic import op +from oslo_log import log +from sqlalchemy import Boolean, Column, DateTime, ForeignKey, Text +from sqlalchemy import Integer, MetaData, String, Table, UniqueConstraint + +from meteos.i18n import _LE + +LOG = log.getLogger(__name__) + + +def upgrade(): + migrate_engine = op.get_bind().engine + meta = MetaData() + meta.bind = migrate_engine + + services = Table( + 'services', meta, + Column('created_at', DateTime), + Column('updated_at', DateTime), + Column('deleted_at', DateTime), + Column('deleted', Integer, default=0), + Column('id', Integer, primary_key=True, nullable=False), + Column('host', String(length=255)), + Column('binary', String(length=255)), + Column('topic', String(length=255)), + Column('report_count', Integer, nullable=False), + Column('disabled', Boolean), + Column('availability_zone', String(length=255)), + mysql_engine='InnoDB', + mysql_charset='utf8' + ) + + templates = Table( + 'templates', meta, + Column('created_at', DateTime), + Column('updated_at', DateTime), + Column('deleted_at', DateTime), + Column('deleted', String(length=36), default='False'), + Column('id', String(length=36), primary_key=True, nullable=False), + Column('user_id', String(length=255)), + Column('project_id', String(length=255)), + Column('status', String(length=255)), + Column('scheduled_at', DateTime), + Column('launched_at', DateTime), + Column('terminated_at', DateTime), + Column('display_name', String(length=255)), + Column('display_description', String(length=255)), + Column('sahara_image_id', String(length=36)), + Column('master_node_id', String(length=36)), + Column('slave_node_id', String(length=36)), + Column('binary_data_id', String(length=36)), + Column('binary_id', String(length=36)), + Column('cluster_template_id', String(length=36)), + Column('job_template_id', String(length=36)), + Column('master_flavor_id', String(length=36)), + Column('master_nodes_num', Integer), + Column('worker_flavor_id', String(length=36)), + Column('worker_nodes_num', Integer), + Column('spark_version', String(length=36)), + Column('floating_ip_pool', String(length=36)), + mysql_engine='InnoDB', + mysql_charset='utf8' + ) + + experiments = Table( + 'experiments', meta, + Column('created_at', DateTime), + Column('updated_at', DateTime), + Column('deleted_at', DateTime), + Column('deleted', String(length=36), default='False'), + Column('id', String(length=36), primary_key=True, nullable=False), + Column('user_id', String(length=255)), + Column('project_id', String(length=255)), + Column('status', String(length=255)), + Column('scheduled_at', DateTime), + Column('launched_at', DateTime), + Column('terminated_at', DateTime), + Column('display_name', String(length=255)), + Column('display_description', String(length=255)), + Column('template_id', String(length=36)), + Column('cluster_id', String(length=36)), + Column('key_name', String(length=36)), + Column('neutron_management_network', String(length=36)), + mysql_engine='InnoDB', + mysql_charset='utf8' + ) + + data_sets = Table( + 'data_sets', meta, + Column('created_at', DateTime), + Column('updated_at', DateTime), + Column('deleted_at', DateTime), + Column('deleted', String(length=36), default='False'), + Column('id', String(length=36), primary_key=True, nullable=False), + Column('source_dataset_url', String(length=255)), + Column('user_id', String(length=255)), + Column('project_id', String(length=255)), + Column('experiment_id', String(length=36)), + Column('cluster_id', String(length=36)), + Column('job_id', String(length=36)), + Column('status', String(length=255)), + Column('scheduled_at', DateTime), + Column('launched_at', DateTime), + Column('terminated_at', DateTime), + Column('display_name', String(length=255)), + Column('display_description', String(length=255)), + Column('container_name', String(length=255)), + Column('object_name', String(length=255)), + Column('user', String(length=255)), + Column('password', String(length=255)), + Column('head', Text), + Column('stderr', Text), + mysql_engine='InnoDB', + mysql_charset='utf8' + ) + + models = Table( + 'models', meta, + Column('created_at', DateTime), + Column('updated_at', DateTime), + Column('deleted_at', DateTime), + Column('deleted', String(length=36), default='False'), + Column('id', String(length=36), primary_key=True, nullable=False), + Column('source_dataset_url', String(length=255)), + Column('dataset_format', String(length=255)), + Column('user_id', String(length=255)), + Column('project_id', String(length=255)), + Column('experiment_id', String(length=36)), + Column('cluster_id', String(length=36)), + Column('job_id', String(length=36)), + Column('status', String(length=255)), + Column('scheduled_at', DateTime), + Column('launched_at', DateTime), + Column('terminated_at', DateTime), + Column('display_name', String(length=255)), + Column('display_description', String(length=255)), + Column('model_type', String(length=255)), + Column('model_params', String(length=255)), + Column('stdout', Text), + Column('stderr', Text), + mysql_engine='InnoDB', + mysql_charset='utf8' + ) + + learnings = Table( + 'learnings', meta, + Column('created_at', DateTime), + Column('updated_at', DateTime), + Column('deleted_at', DateTime), + Column('deleted', String(length=36), default='False'), + Column('id', String(length=36), primary_key=True, nullable=False), + Column('model_id', String(length=36)), + Column('model_type', String(length=255)), + Column('user_id', String(length=255)), + Column('project_id', String(length=255)), + Column('experiment_id', String(length=36)), + Column('cluster_id', String(length=36)), + Column('job_id', String(length=36)), + Column('status', String(length=255)), + Column('scheduled_at', DateTime), + Column('launched_at', DateTime), + Column('terminated_at', DateTime), + Column('display_name', String(length=255)), + Column('display_description', String(length=255)), + Column('method', String(length=255)), + Column('args', String(length=255)), + Column('stdout', Text), + Column('stderr', Text), + mysql_engine='InnoDB', + mysql_charset='utf8' + ) + + # create all tables + # Take care on create order for those with FK dependencies + tables = [services, templates, learnings, experiments, data_sets, models] + + for table in tables: + if not table.exists(): + try: + table.create() + except Exception: + LOG.info(repr(table)) + LOG.exception(_LE('Exception while creating table.')) + raise + + if migrate_engine.name == "mysql": + tables = ["services", "learnings"] + + migrate_engine.execute("SET foreign_key_checks = 0") + for table in tables: + migrate_engine.execute( + "ALTER TABLE %s CONVERT TO CHARACTER SET utf8" % table) + migrate_engine.execute("SET foreign_key_checks = 1") + migrate_engine.execute( + "ALTER DATABASE %s DEFAULT CHARACTER SET utf8" % + migrate_engine.url.database) + migrate_engine.execute("ALTER TABLE %s Engine=InnoDB" % table) + + +def downgrade(): + raise NotImplementedError('Downgrade from initial Meteos install is not' + ' supported.') diff --git a/meteos/db/migrations/utils.py b/meteos/db/migrations/utils.py new file mode 100644 index 0000000..14bda0f --- /dev/null +++ b/meteos/db/migrations/utils.py @@ -0,0 +1,21 @@ +# Copyright 2015 Mirantis Inc. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +import sqlalchemy as sa + + +def load_table(name, connection): + return sa.Table(name, sa.MetaData(), autoload=True, + autoload_with=connection) diff --git a/meteos/db/sqlalchemy/__init__.py b/meteos/db/sqlalchemy/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/meteos/db/sqlalchemy/api.py b/meteos/db/sqlalchemy/api.py new file mode 100644 index 0000000..3633bde --- /dev/null +++ b/meteos/db/sqlalchemy/api.py @@ -0,0 +1,915 @@ +# Copyright (c) 2011 X.commerce, a business unit of eBay Inc. +# Copyright 2010 United States Government as represented by the +# Administrator of the National Aeronautics and Space Administration. +# Copyright (c) 2014 Mirantis, Inc. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +"""Implementation of SQLAlchemy backend.""" + +import copy +import datetime +from functools import wraps +import sys +import uuid +import warnings + +# NOTE(uglide): Required to override default oslo_db Query class +import meteos.db.sqlalchemy.query # noqa + +from oslo_config import cfg +from oslo_db import api as oslo_db_api +from oslo_db import exception as db_exception +from oslo_db import options as db_options +from oslo_db.sqlalchemy import session +from oslo_db.sqlalchemy import utils as db_utils +from oslo_log import log +from oslo_utils import timeutils +from oslo_utils import uuidutils +import six +from sqlalchemy import and_ +from sqlalchemy import or_ +from sqlalchemy.orm import joinedload +from sqlalchemy.sql.expression import true +from sqlalchemy.sql import func + +from meteos.common import constants +from meteos.db.sqlalchemy import models +from meteos import exception +from meteos.i18n import _, _LE, _LW + +CONF = cfg.CONF + +LOG = log.getLogger(__name__) + +_DEFAULT_QUOTA_NAME = 'default' +PER_PROJECT_QUOTAS = [] + +_FACADE = None + +_DEFAULT_SQL_CONNECTION = 'sqlite://' +db_options.set_defaults(cfg.CONF, + connection=_DEFAULT_SQL_CONNECTION) + + +def _create_facade_lazily(): + global _FACADE + if _FACADE is None: + _FACADE = session.EngineFacade.from_config(cfg.CONF) + return _FACADE + + +def get_engine(): + facade = _create_facade_lazily() + return facade.get_engine() + + +def get_session(**kwargs): + facade = _create_facade_lazily() + return facade.get_session(**kwargs) + + +def get_backend(): + """The backend is this module itself.""" + + return sys.modules[__name__] + + +def is_admin_context(context): + """Indicates if the request context is an administrator.""" + if not context: + warnings.warn(_('Use of empty request context is deprecated'), + DeprecationWarning) + raise Exception('die') + return context.is_admin + + +def is_user_context(context): + """Indicates if the request context is a normal user.""" + if not context: + return False + if context.is_admin: + return False + if not context.user_id or not context.project_id: + return False + return True + + +def authorize_project_context(context, project_id): + """Ensures a request has permission to access the given project.""" + if is_user_context(context): + if not context.project_id: + raise exception.NotAuthorized() + elif context.project_id != project_id: + raise exception.NotAuthorized() + + +def authorize_user_context(context, user_id): + """Ensures a request has permission to access the given user.""" + if is_user_context(context): + if not context.user_id: + raise exception.NotAuthorized() + elif context.user_id != user_id: + raise exception.NotAuthorized() + + +def authorize_quota_class_context(context, class_name): + """Ensures a request has permission to access the given quota class.""" + if is_user_context(context): + if not context.quota_class: + raise exception.NotAuthorized() + elif context.quota_class != class_name: + raise exception.NotAuthorized() + + +def require_admin_context(f): + """Decorator to require admin request context. + + The first argument to the wrapped function must be the context. + + """ + @wraps(f) + def wrapper(*args, **kwargs): + if not is_admin_context(args[0]): + raise exception.AdminRequired() + return f(*args, **kwargs) + return wrapper + + +def require_context(f): + """Decorator to require *any* user or admin context. + + This does no authorization for user or project access matching, see + :py:func:`authorize_project_context` and + :py:func:`authorize_user_context`. + + The first argument to the wrapped function must be the context. + + """ + @wraps(f) + def wrapper(*args, **kwargs): + if not is_admin_context(args[0]) and not is_user_context(args[0]): + raise exception.NotAuthorized() + return f(*args, **kwargs) + return wrapper + + +def model_query(context, model, *args, **kwargs): + """Query helper that accounts for context's `read_deleted` field. + + :param context: context to query under + :param model: model to query. Must be a subclass of ModelBase. + :param session: if present, the session to use + :param read_deleted: if present, overrides context's read_deleted field. + :param project_only: if present and context is user-type, then restrict + query to match the context's project_id. + """ + session = kwargs.get('session') or get_session() + read_deleted = kwargs.get('read_deleted') or context.read_deleted + project_only = kwargs.get('project_only') + kwargs = dict() + + if project_only and not context.is_admin: + kwargs['project_id'] = context.project_id + if read_deleted in ('no', 'n', False): + kwargs['deleted'] = False + elif read_deleted in ('yes', 'y', True): + kwargs['deleted'] = True + + return db_utils.model_query( + model=model, session=session, args=args, **kwargs) + + +def exact_filter(query, model, filters, legal_keys): + """Applies exact match filtering to a query. + + Returns the updated query. Modifies filters argument to remove + filters consumed. + + :param query: query to apply filters to + :param model: model object the query applies to, for IN-style + filtering + :param filters: dictionary of filters; values that are lists, + tuples, sets, or frozensets cause an 'IN' test to + be performed, while exact matching ('==' operator) + is used for other values + :param legal_keys: list of keys to apply exact filtering to + """ + + filter_dict = {} + + # Walk through all the keys + for key in legal_keys: + # Skip ones we're not filtering on + if key not in filters: + continue + + # OK, filtering on this key; what value do we search for? + value = filters.pop(key) + + if isinstance(value, (list, tuple, set, frozenset)): + # Looking for values in a list; apply to query directly + column_attr = getattr(model, key) + query = query.filter(column_attr.in_(value)) + else: + # OK, simple exact match; save for later + filter_dict[key] = value + + # Apply simple exact matches + if filter_dict: + query = query.filter_by(**filter_dict) + + return query + + +def ensure_dict_has_id(model_dict): + if not model_dict.get('id'): + model_dict['id'] = uuidutils.generate_uuid() + return model_dict + + +def _sync_learnings(context, project_id, user_id, session): + (learnings, gigs) = learning_data_get_for_project(context, + project_id, + user_id, + session=session) + return {'learnings': learnings} + + +QUOTA_SYNC_FUNCTIONS = { + '_sync_learnings': _sync_learnings, +} + + +# + + +@require_admin_context +def service_destroy(context, service_id): + session = get_session() + with session.begin(): + service_ref = service_get(context, service_id, session=session) + service_ref.soft_delete(session) + + +@require_admin_context +def service_get(context, service_id, session=None): + result = model_query( + context, + models.Service, + session=session).\ + filter_by(id=service_id).\ + first() + if not result: + raise exception.ServiceNotFound(service_id=service_id) + + return result + + +@require_admin_context +def service_get_all(context, disabled=None): + query = model_query(context, models.Service) + + if disabled is not None: + query = query.filter_by(disabled=disabled) + + return query.all() + + +@require_admin_context +def service_get_all_by_topic(context, topic): + return model_query( + context, models.Service, read_deleted="no").\ + filter_by(disabled=False).\ + filter_by(topic=topic).\ + all() + + +@require_admin_context +def service_get_by_host_and_topic(context, host, topic): + result = model_query( + context, models.Service, read_deleted="no").\ + filter_by(disabled=False).\ + filter_by(host=host).\ + filter_by(topic=topic).\ + first() + if not result: + raise exception.ServiceNotFound(service_id=host) + return result + + +@require_admin_context +def _service_get_all_topic_subquery(context, session, topic, subq, label): + sort_value = getattr(subq.c, label) + return model_query(context, models.Service, + func.coalesce(sort_value, 0), + session=session, read_deleted="no").\ + filter_by(topic=topic).\ + filter_by(disabled=False).\ + outerjoin((subq, models.Service.host == subq.c.host)).\ + order_by(sort_value).\ + all() + + +@require_admin_context +def service_get_all_learning_sorted(context): + session = get_session() + with session.begin(): + topic = CONF.learning_topic + label = 'learning_gigabytes' + subq = model_query(context, models.Share, + func.sum(models.Share.size).label(label), + session=session, read_deleted="no").\ + join(models.ShareInstance, + models.ShareInstance.learning_id == models.Share.id).\ + group_by(models.ShareInstance.host).\ + subquery() + return _service_get_all_topic_subquery(context, + session, + topic, + subq, + label) + + +@require_admin_context +def service_get_by_args(context, host, binary): + result = model_query(context, models.Service).\ + filter_by(host=host).\ + filter_by(binary=binary).\ + first() + + if not result: + raise exception.HostBinaryNotFound(host=host, binary=binary) + + return result + + +@require_admin_context +def service_create(context, values): + session = get_session() + + service_ref = models.Service() + service_ref.update(values) + if not CONF.enable_new_services: + service_ref.disabled = True + + with session.begin(): + service_ref.save(session) + return service_ref + + +@require_admin_context +@oslo_db_api.wrap_db_retry(max_retries=5, retry_on_deadlock=True) +def service_update(context, service_id, values): + session = get_session() + + with session.begin(): + service_ref = service_get(context, service_id, session=session) + service_ref.update(values) + service_ref.save(session=session) + + +# + + +def _experiment_get_query(context, session=None): + if session is None: + session = get_session() + return model_query(context, models.Experiment, session=session) + + +@require_context +def experiment_get(context, experiment_id, session=None): + result = _experiment_get_query( + context, session).filter_by(id=experiment_id).first() + + if result is None: + raise exception.NotFound() + + return result + + +@require_context +def experiment_create(context, experiment_values): + values = copy.deepcopy(experiment_values) + values = ensure_dict_has_id(values) + + session = get_session() + experiment_ref = models.Experiment() + experiment_ref.update(values) + + with session.begin(): + experiment_ref.save(session=session) + + # NOTE(u_glide): Do so to prevent errors with relationships + return experiment_get(context, experiment_ref['id'], session=session) + + +@require_context +@oslo_db_api.wrap_db_retry(max_retries=5, retry_on_deadlock=True) +def experiment_update(context, experiment_id, update_values): + session = get_session() + values = copy.deepcopy(update_values) + + with session.begin(): + experiment_ref = experiment_get( + context, experiment_id, session=session) + + experiment_ref.update(values) + experiment_ref.save(session=session) + return experiment_ref + + +def _experiment_get_all_with_filters(context, project_id=None, filters=None, + sort_key=None, sort_dir=None): + if not sort_key: + sort_key = 'created_at' + if not sort_dir: + sort_dir = 'desc' + query = ( + _experiment_get_query(context).join() + ) + + # Apply filters + if not filters: + filters = {} + + # Apply sorting + if sort_dir.lower() not in ('desc', 'asc'): + msg = _("Wrong sorting data provided: sort key is '%(sort_key)s' " + "and sort direction is '%(sort_dir)s'.") % { + "sort_key": sort_key, "sort_dir": sort_dir} + raise exception.InvalidInput(reason=msg) + + def apply_sorting(model, query): + sort_attr = getattr(model, sort_key) + sort_method = getattr(sort_attr, sort_dir.lower()) + return query.order_by(sort_method()) + + try: + query = apply_sorting(models.Experiment, query) + except AttributeError: + msg = _("Wrong sorting key provided - '%s'.") % sort_key + raise exception.InvalidInput(reason=msg) + + # Returns list of experiments that satisfy filters. + query = query.all() + return query + + +@require_context +def experiment_get_all_by_project(context, project_id, filters=None, + sort_key=None, sort_dir=None): + """Returns list of experiments with given project ID.""" + query = _experiment_get_all_with_filters( + context, project_id=project_id, filters=filters, + sort_key=sort_key, sort_dir=sort_dir, + ) + return query + + +@require_context +def experiment_delete(context, experiment_id): + session = get_session() + + with session.begin(): + experiment_ref = experiment_get(context, experiment_id, session) + experiment_ref.soft_delete(session=session) + + +# + + +def _template_get_query(context, session=None): + if session is None: + session = get_session() + return model_query(context, models.Template, session=session) + + +@require_context +def template_get(context, template_id, session=None): + result = _template_get_query( + context, session).filter_by(id=template_id).first() + + if result is None: + raise exception.NotFound() + + return result + + +@require_context +def template_create(context, template_values): + values = copy.deepcopy(template_values) + values = ensure_dict_has_id(values) + + session = get_session() + template_ref = models.Template() + template_ref.update(values) + + with session.begin(): + template_ref.save(session=session) + + # NOTE(u_glide): Do so to prevent errors with relationships + return template_get(context, template_ref['id'], session=session) + + +@require_context +@oslo_db_api.wrap_db_retry(max_retries=5, retry_on_deadlock=True) +def template_update(context, template_id, update_values): + session = get_session() + values = copy.deepcopy(update_values) + + with session.begin(): + template_ref = template_get(context, template_id, session=session) + + template_ref.update(values) + template_ref.save(session=session) + return template_ref + + +def _template_get_all_with_filters(context, project_id=None, filters=None, + sort_key=None, sort_dir=None): + if not sort_key: + sort_key = 'created_at' + if not sort_dir: + sort_dir = 'desc' + query = ( + _template_get_query(context).join() + ) + + # Apply filters + if not filters: + filters = {} + + # Apply sorting + if sort_dir.lower() not in ('desc', 'asc'): + msg = _("Wrong sorting data provided: sort key is '%(sort_key)s' " + "and sort direction is '%(sort_dir)s'.") % { + "sort_key": sort_key, "sort_dir": sort_dir} + raise exception.InvalidInput(reason=msg) + + def apply_sorting(model, query): + sort_attr = getattr(model, sort_key) + sort_method = getattr(sort_attr, sort_dir.lower()) + return query.order_by(sort_method()) + + try: + query = apply_sorting(models.Template, query) + except AttributeError: + msg = _("Wrong sorting key provided - '%s'.") % sort_key + raise exception.InvalidInput(reason=msg) + + # Returns list of templates that satisfy filters. + query = query.all() + return query + + +@require_context +def template_get_all_by_project(context, project_id, filters=None, + sort_key=None, sort_dir=None): + """Returns list of templates with given project ID.""" + query = _template_get_all_with_filters( + context, project_id=project_id, filters=filters, + sort_key=sort_key, sort_dir=sort_dir, + ) + return query + + +@require_context +def template_delete(context, template_id): + session = get_session() + + with session.begin(): + template_ref = template_get(context, template_id, session) + template_ref.soft_delete(session=session) + + +# + + +def _dataset_get_query(context, session=None): + if session is None: + session = get_session() + return model_query(context, models.Dataset, session=session) + + +@require_context +def dataset_get(context, dataset_id, session=None): + result = _dataset_get_query( + context, session).filter_by(id=dataset_id).first() + + if result is None: + raise exception.NotFound() + + return result + + +@require_context +def dataset_create(context, dataset_values): + values = copy.deepcopy(dataset_values) + values = ensure_dict_has_id(values) + + session = get_session() + dataset_ref = models.Dataset() + dataset_ref.update(values) + + with session.begin(): + dataset_ref.save(session=session) + + # NOTE(u_glide): Do so to prevent errors with relationships + return dataset_get(context, dataset_ref['id'], session=session) + + +@require_context +@oslo_db_api.wrap_db_retry(max_retries=5, retry_on_deadlock=True) +def dataset_update(context, dataset_id, update_values): + session = get_session() + values = copy.deepcopy(update_values) + + with session.begin(): + dataset_ref = dataset_get(context, dataset_id, session=session) + + dataset_ref.update(values) + dataset_ref.save(session=session) + return dataset_ref + + +def _dataset_get_all_with_filters(context, project_id=None, filters=None, + sort_key=None, sort_dir=None): + if not sort_key: + sort_key = 'created_at' + if not sort_dir: + sort_dir = 'desc' + query = ( + _dataset_get_query(context).join() + ) + + # Apply filters + if not filters: + filters = {} + + # Apply sorting + if sort_dir.lower() not in ('desc', 'asc'): + msg = _("Wrong sorting data provided: sort key is '%(sort_key)s' " + "and sort direction is '%(sort_dir)s'.") % { + "sort_key": sort_key, "sort_dir": sort_dir} + raise exception.InvalidInput(reason=msg) + + def apply_sorting(model, query): + sort_attr = getattr(model, sort_key) + sort_method = getattr(sort_attr, sort_dir.lower()) + return query.order_by(sort_method()) + + try: + query = apply_sorting(models.Dataset, query) + except AttributeError: + msg = _("Wrong sorting key provided - '%s'.") % sort_key + raise exception.InvalidInput(reason=msg) + + # Returns list of datasets that satisfy filters. + query = query.all() + return query + + +@require_context +def dataset_get_all_by_project(context, project_id, filters=None, + sort_key=None, sort_dir=None): + """Returns list of datasets with given project ID.""" + query = _dataset_get_all_with_filters( + context, project_id=project_id, filters=filters, + sort_key=sort_key, sort_dir=sort_dir, + ) + return query + + +@require_context +def dataset_delete(context, dataset_id): + session = get_session() + + with session.begin(): + dataset_ref = dataset_get(context, dataset_id, session) + dataset_ref.soft_delete(session=session) + + +# + + +def _model_get_query(context, session=None): + if session is None: + session = get_session() + return model_query(context, models.Model, session=session) + + +@require_context +def model_get(context, model_id, session=None): + result = _model_get_query(context, session).filter_by(id=model_id).first() + + if result is None: + raise exception.NotFound() + + return result + + +@require_context +def model_create(context, model_values): + values = copy.deepcopy(model_values) + values = ensure_dict_has_id(values) + + session = get_session() + model_ref = models.Model() + model_ref.update(values) + + with session.begin(): + model_ref.save(session=session) + + # NOTE(u_glide): Do so to prevent errors with relationships + return model_get(context, model_ref['id'], session=session) + + +@require_context +@oslo_db_api.wrap_db_retry(max_retries=5, retry_on_deadlock=True) +def model_update(context, model_id, update_values): + session = get_session() + values = copy.deepcopy(update_values) + + with session.begin(): + model_ref = model_get(context, model_id, session=session) + + model_ref.update(values) + model_ref.save(session=session) + return model_ref + + +def _model_get_all_with_filters(context, project_id=None, filters=None, + sort_key=None, sort_dir=None): + if not sort_key: + sort_key = 'created_at' + if not sort_dir: + sort_dir = 'desc' + query = ( + _model_get_query(context).join() + ) + + # Apply filters + if not filters: + filters = {} + + # Apply sorting + if sort_dir.lower() not in ('desc', 'asc'): + msg = _("Wrong sorting data provided: sort key is '%(sort_key)s' " + "and sort direction is '%(sort_dir)s'.") % { + "sort_key": sort_key, "sort_dir": sort_dir} + raise exception.InvalidInput(reason=msg) + + def apply_sorting(model, query): + sort_attr = getattr(model, sort_key) + sort_method = getattr(sort_attr, sort_dir.lower()) + return query.order_by(sort_method()) + + try: + query = apply_sorting(models.Model, query) + except AttributeError: + msg = _("Wrong sorting key provided - '%s'.") % sort_key + raise exception.InvalidInput(reason=msg) + + # Returns list of models that satisfy filters. + query = query.all() + return query + + +@require_context +def model_get_all_by_project(context, project_id, filters=None, + sort_key=None, sort_dir=None): + """Returns list of models with given project ID.""" + query = _model_get_all_with_filters( + context, project_id=project_id, filters=filters, + sort_key=sort_key, sort_dir=sort_dir, + ) + return query + + +@require_context +def model_delete(context, model_id): + session = get_session() + + with session.begin(): + model_ref = model_get(context, model_id, session) + model_ref.soft_delete(session=session) + + +# + + +def _learning_get_query(context, session=None): + if session is None: + session = get_session() + return model_query(context, models.Learning, session=session) + + +@require_context +def learning_get(context, learning_id, session=None): + result = _learning_get_query( + context, session).filter_by(id=learning_id).first() + + if result is None: + raise exception.NotFound() + + return result + + +@require_context +def learning_create(context, learning_values): + values = copy.deepcopy(learning_values) + values = ensure_dict_has_id(values) + + session = get_session() + learning_ref = models.Learning() + learning_ref.update(values) + + with session.begin(): + learning_ref.save(session=session) + + # NOTE(u_glide): Do so to prevent errors with relationships + return learning_get(context, learning_ref['id'], session=session) + + +@require_context +@oslo_db_api.wrap_db_retry(max_retries=5, retry_on_deadlock=True) +def learning_update(context, learning_id, update_values): + session = get_session() + values = copy.deepcopy(update_values) + + with session.begin(): + learning_ref = learning_get(context, learning_id, session=session) + + learning_ref.update(values) + learning_ref.save(session=session) + return learning_ref + + +def _learning_get_all_with_filters(context, project_id=None, filters=None, + sort_key=None, sort_dir=None): + if not sort_key: + sort_key = 'created_at' + if not sort_dir: + sort_dir = 'desc' + query = ( + _learning_get_query(context).join() + ) + + # Apply filters + if not filters: + filters = {} + + # Apply sorting + if sort_dir.lower() not in ('desc', 'asc'): + msg = _("Wrong sorting data provided: sort key is '%(sort_key)s' " + "and sort direction is '%(sort_dir)s'.") % { + "sort_key": sort_key, "sort_dir": sort_dir} + raise exception.InvalidInput(reason=msg) + + def apply_sorting(learning, query): + sort_attr = getattr(learning, sort_key) + sort_method = getattr(sort_attr, sort_dir.lower()) + return query.order_by(sort_method()) + + try: + query = apply_sorting(models.Learning, query) + except AttributeError: + msg = _("Wrong sorting key provided - '%s'.") % sort_key + raise exception.InvalidInput(reason=msg) + + # Returns list of learnings that satisfy filters. + query = query.all() + return query + + +@require_context +def learning_get_all_by_project(context, project_id, filters=None, + sort_key=None, sort_dir=None): + """Returns list of learnings with given project ID.""" + query = _learning_get_all_with_filters( + context, project_id=project_id, filters=filters, + sort_key=sort_key, sort_dir=sort_dir, + ) + return query + + +@require_context +def learning_delete(context, learning_id): + session = get_session() + + with session.begin(): + learning_ref = learning_get(context, learning_id, session) + learning_ref.soft_delete(session=session) diff --git a/meteos/db/sqlalchemy/models.py b/meteos/db/sqlalchemy/models.py new file mode 100644 index 0000000..3ea9682 --- /dev/null +++ b/meteos/db/sqlalchemy/models.py @@ -0,0 +1,200 @@ +# Copyright (c) 2011 X.commerce, a business unit of eBay Inc. +# Copyright 2010 United States Government as represented by the +# Administrator of the National Aeronautics and Space Administration. +# Copyright 2011 Piston Cloud Computing, Inc. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. +""" +SQLAlchemy models for Meteos data. +""" + +from oslo_config import cfg +from oslo_db.sqlalchemy import models +from oslo_log import log +from sqlalchemy import Column, Integer, String, schema +from sqlalchemy.ext.declarative import declarative_base +from sqlalchemy import orm +from sqlalchemy import ForeignKey, DateTime, Boolean, Enum, Text + +from meteos.common import constants + +CONF = cfg.CONF +BASE = declarative_base() + +LOG = log.getLogger(__name__) + + +class MeteosBase(models.ModelBase, + models.TimestampMixin, + models.SoftDeleteMixin): + + """Base class for Meteos Models.""" + __table_args__ = {'mysql_engine': 'InnoDB'} + metadata = None + + def to_dict(self): + model_dict = {} + for k, v in self.items(): + if not issubclass(type(v), MeteosBase): + model_dict[k] = v + return model_dict + + def soft_delete(self, session, update_status=False, + status_field_name='status'): + """Mark this object as deleted.""" + if update_status: + setattr(self, status_field_name, constants.STATUS_DELETED) + + return super(MeteosBase, self).soft_delete(session) + + +class Service(BASE, MeteosBase): + + """Represents a running service on a host.""" + + __tablename__ = 'services' + id = Column(Integer, primary_key=True) + host = Column(String(255)) # , ForeignKey('hosts.id')) + binary = Column(String(255)) + topic = Column(String(255)) + report_count = Column(Integer, nullable=False, default=0) + disabled = Column(Boolean, default=False) + + +class Template(BASE, MeteosBase): + + __tablename__ = 'templates' + id = Column(String(36), primary_key=True) + deleted = Column(String(36), default='False') + user_id = Column(String(255)) + project_id = Column(String(255)) + + display_name = Column(String(255)) + display_description = Column(String(255)) + + sahara_image_id = Column(String(36)) + master_node_id = Column(String(36)) + slave_node_id = Column(String(36)) + binary_data_id = Column(String(36)) + binary_id = Column(String(36)) + cluster_template_id = Column(String(36)) + job_template_id = Column(String(36)) + + master_flavor_id = Column(String(36)) + worker_flavor_id = Column(String(36)) + master_nodes_num = Column(Integer) + worker_nodes_num = Column(Integer) + floating_ip_pool = Column(String(36)) + spark_version = Column(String(36)) + + status = Column(String(255)) + launched_at = Column(DateTime) + + +class Experiment(BASE, MeteosBase): + + __tablename__ = 'experiments' + id = Column(String(36), primary_key=True) + deleted = Column(String(36), default='False') + user_id = Column(String(255)) + project_id = Column(String(255)) + + display_name = Column(String(255)) + display_description = Column(String(255)) + template_id = Column(String(36)) + cluster_id = Column(String(36)) + key_name = Column(String(36)) + neutron_management_network = Column(String(36)) + + status = Column(String(255)) + launched_at = Column(DateTime) + + +class Dataset(BASE, MeteosBase): + + __tablename__ = 'data_sets' + id = Column(String(36), primary_key=True) + source_dataset_url = Column(String(255)) + deleted = Column(String(36), default='False') + user_id = Column(String(255)) + project_id = Column(String(255)) + experiment_id = Column(String(36)) + cluster_id = Column(String(36)) + job_id = Column(String(36)) + + display_name = Column(String(255)) + display_description = Column(String(255)) + + container_name = Column(String(255)) + object_name = Column(String(255)) + user = Column(String(255)) + password = Column(String(255)) + + status = Column(String(255)) + launched_at = Column(DateTime) + + head = Column(Text) + stderr = Column(Text) + + +class Model(BASE, MeteosBase): + + __tablename__ = 'models' + id = Column(String(36), primary_key=True) + source_dataset_url = Column(String(255)) + dataset_format = Column(String(255)) + deleted = Column(String(36), default='False') + user_id = Column(String(255)) + project_id = Column(String(255)) + experiment_id = Column(String(36)) + cluster_id = Column(String(36)) + job_id = Column(String(36)) + + display_name = Column(String(255)) + display_description = Column(String(255)) + + model_type = Column(String(255)) + model_params = Column(String(255)) + + status = Column(String(255)) + launched_at = Column(DateTime) + + stdout = Column(Text) + stderr = Column(Text) + + +class Learning(BASE, MeteosBase): + + __tablename__ = 'learnings' + id = Column(String(36), primary_key=True) + model_id = Column(String(36)) + model_type = Column(String(255)) + deleted = Column(String(36), default='False') + user_id = Column(String(255)) + project_id = Column(String(255)) + experiment_id = Column(String(36)) + cluster_id = Column(String(36)) + job_id = Column(String(36)) + + display_name = Column(String(255)) + display_description = Column(String(255)) + + method = Column(String(255)) + args = Column(String(255)) + + status = Column(String(255)) + launched_at = Column(DateTime) + + stdout = Column(Text) + stderr = Column(Text) diff --git a/meteos/db/sqlalchemy/query.py b/meteos/db/sqlalchemy/query.py new file mode 100644 index 0000000..1abc0f8 --- /dev/null +++ b/meteos/db/sqlalchemy/query.py @@ -0,0 +1,40 @@ +# Copyright 2015 Mirantis Inc. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +from oslo_db.sqlalchemy import orm +import sqlalchemy + +from meteos.common import constants + + +class Query(orm.Query): + def soft_delete(self, synchronize_session='evaluate', update_status=False, + status_field_name='status'): + if update_status: + setattr(self, status_field_name, constants.STATUS_DELETED) + + return super(Query, self).soft_delete(synchronize_session) + + +def get_maker(engine, autocommit=True, expire_on_commit=False): + """Return a SQLAlchemy sessionmaker using the given engine.""" + return sqlalchemy.orm.sessionmaker(bind=engine, + class_=orm.Session, + autocommit=autocommit, + expire_on_commit=expire_on_commit, + query_cls=Query) + +# NOTE(uglide): Monkey patch oslo_db get_maker() function to use custom Query +orm.get_maker = get_maker diff --git a/meteos/engine/__init__.py b/meteos/engine/__init__.py new file mode 100644 index 0000000..7725e72 --- /dev/null +++ b/meteos/engine/__init__.py @@ -0,0 +1,25 @@ +# Copyright 2010 United States Government as represented by the +# Administrator of the National Aeronautics and Space Administration. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +# Importing full names to not pollute the namespace and cause possible +# collisions with use of 'from meteos.engine import ' elsewhere. +import oslo_utils.importutils as import_utils + +from meteos.common import config + +CONF = config.CONF + +API = import_utils.import_class(CONF.learning_api_class) diff --git a/meteos/engine/api.py b/meteos/engine/api.py new file mode 100644 index 0000000..f980081 --- /dev/null +++ b/meteos/engine/api.py @@ -0,0 +1,470 @@ +# Copyright 2010 United States Government as represented by the +# Administrator of the National Aeronautics and Space Administration. +# All Rights Reserved. +# Copyright (c) 2015 Tom Barron. All rights reserved. +# Copyright (c) 2015 Mirantis Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +""" +Handles all requests relating to learnings. +""" + +from oslo_config import cfg +from oslo_log import log +from oslo_utils import excutils +from oslo_utils import strutils +from oslo_utils import timeutils +import six + +from meteos.common import constants +from meteos.db import base +from meteos import exception +from meteos.i18n import _, _LE, _LI, _LW +from meteos import policy +from meteos.engine import rpcapi as engine_rpcapi +from meteos import utils + +LOG = log.getLogger(__name__) + + +class API(base.Base): + + """API for interacting with the learning manager.""" + + def __init__(self, db_driver=None): + self.engine_rpcapi = engine_rpcapi.LearningAPI() + super(API, self).__init__(db_driver) + + def get_all_templates(self, context, search_opts=None, + sort_key='created_at', sort_dir='desc'): + policy.check_policy(context, 'template', 'get_all') + + if search_opts is None: + search_opts = {} + + LOG.debug("Searching for templates by: %s", six.text_type(search_opts)) + + project_id = context.project_id + + templates = self.db.template_get_all_by_project(context, project_id, + sort_key=sort_key, + sort_dir=sort_dir) + + if search_opts: + results = [] + for s in templates: + # values in search_opts can be only strings + if all(s.get(k, None) == v for k, v in search_opts.items()): + results.append(s) + templates = results + return templates + + def get_template(self, context, template_id): + rv = self.db.template_get(context, template_id) + return rv + + def create_template(self, context, name, description, image_id=None, + master_nodes_num=None, master_flavor_id=None, + worker_nodes_num=None, worker_flavor_id=None, + spark_version=None, floating_ip_pool=None): + """Create new Expariment.""" + policy.check_policy(context, 'template', 'create') + + template = {'id': None, + 'user_id': context.user_id, + 'project_id': context.project_id, + 'display_name': name, + 'display_description': description, + 'image_id': image_id, + 'master_nodes_num': master_nodes_num, + 'master_flavor_id': master_flavor_id, + 'worker_nodes_num': worker_nodes_num, + 'worker_flavor_id': worker_flavor_id, + 'spark_version': spark_version, + 'floating_ip_pool': floating_ip_pool, + } + + try: + result = self.db.template_create(context, template) + self.engine_rpcapi.create_template(context, result) + except Exception: + with excutils.save_and_reraise_exception(): + self.db.template_delete(context, result['id']) + + # Retrieve the learning with instance details + template = self.db.template_get(context, result['id']) + + return template + + def delete_template(self, context, id, force=False): + """Delete template.""" + + policy.check_policy(context, 'template', 'delete') + + template = self.db.template_get(context, id) + + statuses = (constants.STATUS_AVAILABLE, constants.STATUS_ERROR, + constants.STATUS_INACTIVE) + if not (force or template['status'] in statuses): + msg = _("Learning status must be one of %(statuses)s") % { + "statuses": statuses} + raise exception.InvalidLearning(reason=msg) + + result = self.engine_rpcapi.delete_template(context, id) + + def get_all_experiments(self, context, search_opts=None, + sort_key='created_at', sort_dir='desc'): + + policy.check_policy(context, 'experiment', 'get_all') + + if search_opts is None: + search_opts = {} + + LOG.debug("Searching for experiments by: %s", + six.text_type(search_opts)) + + project_id = context.project_id + + experiments = self.db.experiment_get_all_by_project( + context, project_id, + sort_key=sort_key, sort_dir=sort_dir) + + if search_opts: + results = [] + for s in experiments: + # values in search_opts can be only strings + if all(s.get(k, None) == v for k, v in search_opts.items()): + results.append(s) + experiments = results + return experiments + + def get_experiment(self, context, experiment_id): + rv = self.db.experiment_get(context, experiment_id) + return rv + + def create_experiment(self, context, name, description, template_id, + key_name, neutron_management_network): + """Create new Experiment.""" + policy.check_policy(context, 'experiment', 'create') + + experiment = {'id': None, + 'user_id': context.user_id, + 'project_id': context.project_id, + 'display_name': name, + 'display_description': description, + 'template_id': template_id, + 'key_name': key_name, + 'neutron_management_network': neutron_management_network, + } + + try: + result = self.db.experiment_create(context, experiment) + self.engine_rpcapi.create_experiment(context, result) + updates = {'status': constants.STATUS_CREATING} + + LOG.info(_LI("Accepted creation of experiment %s."), result['id']) + self.db.experiment_update(context, result['id'], updates) + + except Exception: + with excutils.save_and_reraise_exception(): + self.db.experiment_delete(context, result['id']) + + # Retrieve the learning with instance details + experiment = self.db.experiment_get(context, result['id']) + + return experiment + + def delete_experiment(self, context, id, force=False): + """Delete experiment.""" + + policy.check_policy(context, 'experiment', 'delete') + + experiment = self.db.experiment_get(context, id) + + statuses = (constants.STATUS_AVAILABLE, constants.STATUS_ERROR, + constants.STATUS_INACTIVE) + if not (force or experiment['status'] in statuses): + msg = _("Learning status must be one of %(statuses)s") % { + "statuses": statuses} + raise exception.InvalidLearning(reason=msg) + + result = self.engine_rpcapi.delete_experiment(context, id) + + def get_all_datasets(self, context, search_opts=None, + sort_key='created_at', sort_dir='desc'): + + policy.check_policy(context, 'dataset', 'get_all') + + if search_opts is None: + search_opts = {} + + LOG.debug("Searching for datasets by: %s", six.text_type(search_opts)) + + project_id = context.project_id + + datasets = self.db.dataset_get_all_by_project(context, + project_id, + sort_key=sort_key, + sort_dir=sort_dir) + + if search_opts: + results = [] + for s in datasets: + # values in search_opts can be only strings + if all(s.get(k, None) == v for k, v in search_opts.items()): + results.append(s) + datasets = results + return datasets + + def get_dataset(self, context, dataset_id): + rv = self.db.dataset_get(context, dataset_id) + return rv + + def create_dataset(self, context, name, description, method, + source_dataset_url, params, template_id, + job_template_id, experiment_id, cluster_id, + swift_tenant, swift_username, swift_password): + """Create a Dataset""" + policy.check_policy(context, 'dataset', 'create') + + dataset = {'id': None, + 'display_name': name, + 'display_description': description, + 'method': method, + 'source_dataset_url': source_dataset_url, + 'user_id': context.user_id, + 'project_id': context.project_id, + 'experiment_id': experiment_id, + 'cluster_id': cluster_id, + 'params': params, + 'cluster_id': cluster_id + } + + try: + result = self.db.dataset_create(context, dataset) + result['template_id'] = template_id + result['job_template_id'] = job_template_id + result['swift_tenant'] = swift_tenant + result['swift_username'] = swift_username + result['swift_password'] = swift_password + self.engine_rpcapi.create_dataset(context, result) + updates = {'status': constants.STATUS_CREATING} + + LOG.info(_LI("Accepted parsing of dataset %s."), result['id']) + self.db.dataset_update(context, result['id'], updates) + except Exception: + with excutils.save_and_reraise_exception(): + self.db.dataset_delete(context, result['id']) + + # Retrieve the learning with instance details + dataset = self.db.dataset_get(context, result['id']) + + return dataset + + def delete_dataset(self, context, id, force=False): + """Delete dataset.""" + + policy.check_policy(context, 'dataset', 'delete') + + dataset = self.db.dataset_get(context, id) + + statuses = (constants.STATUS_AVAILABLE, constants.STATUS_ERROR, + constants.STATUS_INACTIVE) + if not (force or dataset['status'] in statuses): + msg = _("Learning status must be one of %(statuses)s") % { + "statuses": statuses} + raise exception.InvalidLearning(reason=msg) + + result = self.engine_rpcapi.delete_dataset(context, + dataset['cluster_id'], + dataset['job_id'], + id) + + def get_all_models(self, context, search_opts=None, sort_key='created_at', + sort_dir='desc'): + policy.check_policy(context, 'model', 'get_all') + + if search_opts is None: + search_opts = {} + + LOG.debug("Searching for models by: %s", six.text_type(search_opts)) + + project_id = context.project_id + + models = self.db.model_get_all_by_project(context, + project_id, + sort_key=sort_key, + sort_dir=sort_dir) + + if search_opts: + results = [] + for s in models: + # values in search_opts can be only strings + if all(s.get(k, None) == v for k, v in search_opts.items()): + results.append(s) + models = results + return models + + def get_model(self, context, model_id): + rv = self.db.model_get(context, model_id) + return rv + + def create_model(self, context, name, description, source_dataset_url, + dataset_format, model_type, model_params, template_id, + job_template_id, experiment_id, cluster_id, + swift_tenant, swift_username, swift_password): + """Create a Model""" + policy.check_policy(context, 'model', 'create') + + model = {'id': None, + 'display_name': name, + 'display_description': description, + 'source_dataset_url': source_dataset_url, + 'dataset_format': dataset_format, + 'user_id': context.user_id, + 'project_id': context.project_id, + 'model_type': model_type, + 'model_params': model_params, + 'experiment_id': experiment_id, + 'cluster_id': cluster_id + } + + try: + result = self.db.model_create(context, model) + result['job_template_id'] = job_template_id + result['template_id'] = template_id + result['swift_tenant'] = swift_tenant + result['swift_username'] = swift_username + result['swift_password'] = swift_password + self.engine_rpcapi.create_model(context, result) + updates = {'status': constants.STATUS_CREATING} + + LOG.info(_LI("Accepted creation of model %s."), result['id']) + self.db.model_update(context, result['id'], updates) + except Exception: + with excutils.save_and_reraise_exception(): + self.db.model_delete(context, result['id']) + + # Retrieve the learning with instance details + model = self.db.model_get(context, result['id']) + + return model + + def delete_model(self, context, id, force=False): + """Delete model.""" + + policy.check_policy(context, 'model', 'delete') + + model = self.db.model_get(context, id) + + statuses = (constants.STATUS_AVAILABLE, constants.STATUS_ERROR, + constants.STATUS_INACTIVE) + if not (force or model['status'] in statuses): + msg = _("Learning status must be one of %(statuses)s") % { + "statuses": statuses} + raise exception.InvalidLearning(reason=msg) + + result = self.engine_rpcapi.delete_model(context, + model['cluster_id'], + model['job_id'], + id) + + def get_all_learnings(self, context, search_opts=None, + sort_key='created_at', sort_dir='desc'): + policy.check_policy(context, 'learning', 'get_all') + + if search_opts is None: + search_opts = {} + + LOG.debug("Searching for learnings by: %s", six.text_type(search_opts)) + + project_id = context.project_id + + learnings = self.db.learning_get_all_by_project(context, + project_id, + sort_key=sort_key, + sort_dir=sort_dir) + + if search_opts: + results = [] + for s in learnings: + # values in search_opts can be only strings + if all(s.get(k, None) == v for k, v in search_opts.items()): + results.append(s) + learnings = results + return learnings + + def get_learning(self, context, learning_id): + rv = self.db.learning_get(context, learning_id) + return rv + + def create_learning(self, context, name, description, model_id, method, + args, template_id, job_template_id, + experiment_id, cluster_id): + """Create a Learning""" + policy.check_policy(context, 'learning', 'create') + model = self.db.model_get(context, model_id) + + learning = {'id': None, + 'display_name': name, + 'display_description': description, + 'model_id': model_id, + 'model_type': model.model_type, + 'user_id': context.user_id, + 'project_id': context.project_id, + 'method': method, + 'args': args, + 'job_template_id': job_template_id, + 'experiment_id': experiment_id, + 'cluster_id': cluster_id + } + + try: + result = self.db.learning_create(context, learning) + result['template_id'] = template_id + result['job_template_id'] = job_template_id + result['cluster_id'] = cluster_id + result['dataset_format'] = model.dataset_format + self.engine_rpcapi.create_learning(context, result) + updates = {'status': constants.STATUS_CREATING} + + LOG.info(_LI("Accepted creation of learning %s."), result['id']) + self.db.learning_update(context, result['id'], updates) + except Exception: + with excutils.save_and_reraise_exception(): + self.db.learning_delete(context, result['id']) + + # Retrieve the learning with instance details + learning = self.db.learning_get(context, result['id']) + + return learning + + def delete_learning(self, context, id, force=False): + """Delete learning.""" + + policy.check_policy(context, 'learning', 'delete') + + learning = self.db.learning_get(context, id) + + statuses = (constants.STATUS_AVAILABLE, constants.STATUS_ERROR, + constants.STATUS_INACTIVE) + if not (force or learning['status'] in statuses): + msg = _("Learning status must be one of %(statuses)s") % { + "statuses": statuses} + raise exception.InvalidLearning(reason=msg) + + result = self.engine_rpcapi.delete_learning(context, + learning['cluster_id'], + learning['job_id'], + id) diff --git a/meteos/engine/configuration.py b/meteos/engine/configuration.py new file mode 100644 index 0000000..2567244 --- /dev/null +++ b/meteos/engine/configuration.py @@ -0,0 +1,81 @@ +#!/usr/bin/env python + +# Copyright (c) 2012 Rackspace Hosting +# Copyright (c) 2013 NetApp +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +""" +Configuration support for all drivers. + +This module allows support for setting configurations either from default +or from a particular CONF group, to be able to set multiple configurations +for a given set of values. + +For instance, two generic configurations can be set by naming them in groups as + + [generic1] + learning_backend_name=generic-backend-1 + ... + + [generic2] + learning_backend_name=generic-backend-2 + ... + +And the configuration group name will be passed in so that all calls to +configuration.volume_group within that instance will be mapped to the proper +named group. + +This class also ensures the implementation's configuration is grafted into the +option group. This is due to the way cfg works. All cfg options must be defined +and registered in the group in which they are used. +""" + +from oslo_config import cfg + +CONF = cfg.CONF + + +class Configuration(object): + + def __init__(self, learning_opts, config_group=None): + """Graft config values into config group. + + This takes care of grafting the implementation's config values + into the config group. + """ + self.config_group = config_group + + # set the local conf so that __call__'s know what to use + if self.config_group: + self._ensure_config_values(learning_opts) + self.local_conf = CONF._get(self.config_group) + else: + self.local_conf = CONF + + def _ensure_config_values(self, learning_opts): + CONF.register_opts(learning_opts, + group=self.config_group) + + def append_config_values(self, learning_opts): + self._ensure_config_values(learning_opts) + + def safe_get(self, value): + try: + return self.__getattr__(value) + except cfg.NoSuchOptError: + return None + + def __getattr__(self, value): + return getattr(self.local_conf, value) diff --git a/meteos/engine/driver.py b/meteos/engine/driver.py new file mode 100644 index 0000000..3dbcacd --- /dev/null +++ b/meteos/engine/driver.py @@ -0,0 +1,127 @@ +# Copyright 2012 NetApp +# Copyright 2015 Mirantis inc. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. +""" +Drivers for learnings. + +""" + +import six +import time + +from oslo_config import cfg +from oslo_log import log + +from meteos import exception +from meteos.i18n import _, _LE +from meteos import utils + +LOG = log.getLogger(__name__) + +ssh_opts = [ + cfg.StrOpt( + 'ssh_user', + default='ubuntu', + help='SSH login user.'), + cfg.StrOpt( + 'ssh_password', + default='ubuntu', + help='SSH login password.'), + cfg.IntOpt( + 'ssh_port', + default=22, + help='SSH connection port number.'), + cfg.IntOpt( + 'ssh_conn_timeout', + default=60, + help='Backend server SSH connection timeout.'), + cfg.IntOpt( + 'ssh_min_pool_conn', + default=1, + help='Minimum number of connections in the SSH pool.'), + cfg.IntOpt( + 'ssh_max_pool_conn', + default=10, + help='Maximum number of connections in the SSH pool.'), +] + +CONF = cfg.CONF +CONF.register_opts(ssh_opts) + + +class LearningDriver(object): + + """Class defines interface of NAS driver.""" + + def __init__(self, driver_handles_learning_servers, *args, **kwargs): + """Implements base functionality for learning drivers. + + :param driver_handles_learning_servers: expected boolean value or + tuple/list/set of boolean values. + There are two possible approaches for learning drivers in Meteos. + First is when learning driver is able to handle learning-servers + and second when not. + Drivers can support either both (indicated by a tuple/set/list with + (True, False)) or only one of these approaches. So, it is allowed + to be 'True' when learning driver does support handling of learning + servers and allowed to be 'False' when it does support usage of + unhandled learning-servers that are not tracked by Meteos. + Learning drivers are allowed to work only in one of two possible + driver modes, that is why only one should be chosen. + :param config_opts: tuple, list or set of config option lists + that should be registered in driver's configuration right after + this attribute is created. Useful for usage with mixin classes. + """ + super(LearningDriver, self).__init__() + self.configuration = kwargs.get('configuration', None) + self.initialized = False + self._stats = {} + + self.pools = [] + + for config_opt_set in kwargs.get('config_opts', []): + self.configuration.append_config_values(config_opt_set) + + def create_template(self, context, request_specs): + """Is called to create template.""" + raise NotImplementedError() + + def delete_template(self, context, request_specs): + """Is called to delete template.""" + raise NotImplementedError() + + def create_experiment(self, context, request_specs): + """Is called to create experimnet.""" + raise NotImplementedError() + + def delete_experiment(self, context, request_specs): + """Is called to delete experimnet.""" + raise NotImplementedError() + + def create_dataset(self, context, request_specs): + """Is called to create dataset.""" + raise NotImplementedError() + + def delete_dataset(self, context, request_specs): + """Is called to delete dataset.""" + raise NotImplementedError() + + def create_model(self, context, request_specs): + """Is called to create model.""" + raise NotImplementedError() + + def delete_model(self, context, request_specs): + """Is called to delete model.""" + raise NotImplementedError() diff --git a/meteos/engine/drivers/__init__.py b/meteos/engine/drivers/__init__.py new file mode 100644 index 0000000..075c627 --- /dev/null +++ b/meteos/engine/drivers/__init__.py @@ -0,0 +1,22 @@ +# Copyright 2012 OpenStack LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +""" +:mod:`meteos.engine.driver` -- Meteos Learning Drivers +===================================================== + +.. automodule:: meteos.engine.driver + :platform: Unix + :synopsis: Module containing all the Meteos Learning drivers. +""" diff --git a/meteos/engine/drivers/generic.py b/meteos/engine/drivers/generic.py new file mode 100644 index 0000000..1d45a74 --- /dev/null +++ b/meteos/engine/drivers/generic.py @@ -0,0 +1,468 @@ +# Copyright (c) 2014 NetApp, Inc. +# All Rights Reserved. +# Copyright (c) 2016 NEC Corporation. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +"""Generic Driver for learnings.""" + +import base64 +from eventlet import greenthread +import os +import random +import time + +from oslo_concurrency import processutils +from oslo_config import cfg +from oslo_log import log +from oslo_utils import excutils +from oslo_utils import importutils +from oslo_utils import units +import retrying +import six + +from meteos.common import constants as const +from meteos import context +from meteos import exception +from meteos.i18n import _, _LE, _LI, _LW +from meteos.engine import driver +from meteos import utils +from meteos import cluster + +LOG = log.getLogger(__name__) + +learning_opts = [ + cfg.IntOpt( + 'create_experiment_timeout', + default=600, + help="Time to wait for creating experiment (seconds)."), + cfg.IntOpt( + 'execute_job_timeout', + default=600, + help='Timeout for executing job (seconds).'), + cfg.IntOpt( + 'api_retry_interval', + default=10, + help='The number of seconds to wait before retrying the request.'), +] + + +CONF = cfg.CONF +CONF.register_opts(learning_opts) + +SAHARA_GROUP = 'sahara' +PLUGIN_NAME = 'spark' +SPARK_USER_NAME = 'ubuntu' +METEOS_JOB_TYPE = 'Spark' + + +class GenericLearningDriver(driver.LearningDriver): + + """Executes commands relating to Learnings.""" + + def __init__(self, *args, **kwargs): + """Do initialization.""" + super(GenericLearningDriver, self).__init__( + [False, True], *args, **kwargs) + self.admin_context = context.get_admin_context() + self.cluster_api = cluster.API() + self.sshpool = None + + def _run_ssh(self, ip, cmd_list, check_exit_code=True, attempts=1): + utils.check_ssh_injection(cmd_list) + + ssh_conn_timeout = self.configuration.ssh_conn_timeout + ssh_port = self.configuration.ssh_port + ssh_user = self.configuration.ssh_user + ssh_password = self.configuration.ssh_password + min_size = self.configuration.ssh_min_pool_conn + max_size = self.configuration.ssh_max_pool_conn + command = ' '. join(cmd_list) + + if not self.sshpool: + self.sshpool = utils.SSHPool( + ip, + ssh_port, + ssh_conn_timeout, + ssh_user, + ssh_password, + min_size=min_size, + max_size=max_size) + last_exception = None + try: + with self.sshpool.item() as ssh: + while attempts > 0: + attempts -= 1 + try: + return processutils.ssh_execute( + ssh, + command, + check_exit_code=check_exit_code) + except Exception as e: + LOG.error(e) + last_exception = e + greenthread.sleep(random.randint(20, 500) / 100.0) + try: + raise processutils.ProcessExecutionError( + exit_code=last_exception.exit_code, + stdout=last_exception.stdout, + stderr=last_exception.stderr, + cmd=last_exception.cmd) + except AttributeError: + raise processutils.ProcessExecutionError( + exit_code=-1, + stdout="", + stderr="Error running SSH command", + cmd=command) + + except Exception: + with excutils.save_and_reraise_exception(): + LOG.error(_LE("Error running SSH command: %s"), command) + + def _delete_hdfs_dir(self, context, cluster_id, dir_name): + + cluster = self.cluster_api.get_cluster(context, cluster_id) + node_groups = cluster.node_groups + + for node in node_groups: + if 'master' in node['node_processes']: + ip = node['instances'][0]['management_ip'] + + path = '/user/ubuntu/' + dir_name + + cmd = ['sudo', '-u', 'hdfs', 'hadoop', 'fs', '-rm', '-r', path] + + try: + self._run_ssh(ip, cmd) + except Exception: + pass + + def create_template(self, context, request_specs): + """Creates Template.""" + + image_id = request_specs['image_id'] + master_nodes_num = request_specs['master_nodes_num'] + master_flavor_id = request_specs['master_flavor_id'] + worker_nodes_num = request_specs['worker_nodes_num'] + worker_flavor_id = request_specs['worker_flavor_id'] + floating_ip_pool = request_specs['floating_ip_pool'] + spark_version = request_specs['spark_version'] + + master_node_name = 'master-tmpl-' + request_specs['id'] + slave_node_name = 'slave-tmpl-' + request_specs['id'] + cluster_node_name = 'cluster-tmpl-' + request_specs['id'] + job_binary_name = 'meteos-' + request_specs['id'] + '.py' + job_name = 'meteos-job-' + request_specs['id'] + + sahara_image_id = self.cluster_api.image_set( + context, image_id, SPARK_USER_NAME) + self.cluster_api.image_tags_add( + context, sahara_image_id, ['spark', spark_version]) + + master_node_id = self.cluster_api.create_node_group_template( + context, master_node_name, PLUGIN_NAME, spark_version, + master_flavor_id, ['master', 'namenode'], floating_ip_pool, True) + + slave_node_id = self.cluster_api.create_node_group_template( + context, slave_node_name, PLUGIN_NAME, spark_version, + worker_flavor_id, ["slave", "datanode"], floating_ip_pool, True) + + cluster_node_groups = [ + { + "name": "master", + "node_group_template_id": master_node_id, + "count": master_nodes_num + }, + { + "name": "workers", + "node_group_template_id": slave_node_id, + "count": worker_nodes_num + }] + + cluster_template_id = self.cluster_api.create_cluster_template( + context, + cluster_node_name, + PLUGIN_NAME, spark_version, + cluster_node_groups) + + filename = 'meteos-script-' + spark_version + '.py' + filepath = os.path.dirname(__file__) + '/../../cluster/binary/' + filename + + data = utils.file_open(filepath) + + binary_data_id = self.cluster_api.create_job_binary_data( + context, job_binary_name, data) + + binary_url = 'internal-db://' + binary_data_id + binary_id = self.cluster_api.create_job_binary( + context, job_binary_name, binary_url) + mains = [binary_id] + job_template_id = self.cluster_api.create_job_template( + context, job_name, METEOS_JOB_TYPE, mains) + + response = {'sahara_image_id': sahara_image_id, + 'master_node_id': master_node_id, + 'slave_node_id': slave_node_id, + 'binary_data_id': binary_data_id, + 'binary_id': binary_id, + 'cluster_template_id': cluster_template_id, + 'job_template_id': job_template_id} + + return response + + def delete_template(self, context, request_specs): + """Delete Template.""" + + self.cluster_api.delete_job_template( + context, request_specs['job_template_id']) + self.cluster_api.delete_job_binary(context, request_specs['binary_id']) + self.cluster_api.delete_job_binary_data( + context, request_specs['binary_data_id']) + self.cluster_api.delete_cluster_template( + context, request_specs['cluster_template_id']) + self.cluster_api.delete_node_group_template( + context, request_specs['slave_node_id']) + self.cluster_api.delete_node_group_template( + context, request_specs['master_node_id']) + # self.cluster_api.image_remove(context, + # request_specs['sahara_image_id']) + + def create_experiment(self, context, request_specs, + image_id, cluster_id, spark_version): + """Creates Experiment.""" + + cluster_name = 'cluster-' + request_specs['id'][0:8] + key_name = request_specs['key_name'] + neutron_management_network = request_specs[ + 'neutron_management_network'] + + cluster_id = self.cluster_api.create_cluster( + context, + cluster_name, + PLUGIN_NAME, + spark_version, + image_id, + cluster_id, + key_name, + neutron_management_network) + + return cluster_id + + def delete_experiment(self, context, id): + """Delete Experiment.""" + self.cluster_api.delete_cluster(context, id) + + def wait_for_cluster_create(self, context, id): + + starttime = time.time() + deadline = starttime + self.configuration.create_experiment_timeout + interval = self.configuration.api_retry_interval + tries = 0 + + while True: + cluster = self.cluster_api.get_cluster(context, id) + + if cluster.status == const.STATUS_SAHARA_ACTIVE: + break + + tries += 1 + now = time.time() + if now > deadline: + msg = _("Timeout trying to create experiment " + "%s") % id + raise exception.Invalid(reason=msg) + + LOG.debug("Waiting for cluster to complete: Current status: %s", + cluster.status) + time.sleep(interval) + + def get_job_result(self, context, job_id, template_id, cluster_id): + + stdout = "" + stderr = "" + + starttime = time.time() + deadline = starttime + self.configuration.execute_job_timeout + interval = self.configuration.api_retry_interval + tries = 0 + + while True: + job = self.cluster_api.get_job(context, job_id) + + if job.info['status'] == const.STATUS_JOB_SUCCESS: + stdout = self._get_job_result(context, + template_id, + cluster_id, + job_id) + break + elif job.info['status'] == const.STATUS_JOB_ERROR: + stderr = self._get_job_result(context, + template_id, + cluster_id, + job_id) + break + + tries += 1 + now = time.time() + if now > deadline: + msg = _("Timeout trying to create experiment " + "%s") % job_id + raise exception.Invalid(reason=msg) + + LOG.debug("Waiting for job to complete: Current status: %s", + job.info['status']) + time.sleep(interval) + return stdout, stderr + + def create_dataset(self, context, request_specs): + """Create Dataset.""" + + job_args = {} + + job_template_id = request_specs['job_template_id'] + cluster_id = request_specs['cluster_id'] + source_dataset_url = request_specs['source_dataset_url'] + + job_args['method'] = request_specs['method'] + '_dataset' + job_args['source_dataset_url'] = source_dataset_url + + dataset_args = {'params': request_specs['params']} + job_args['dataset'] = dataset_args + + swift_args = {} + + if source_dataset_url.count('swift'): + swift_args['tenant'] = request_specs['swift_tenant'] + swift_args['username'] = request_specs['swift_username'] + swift_args['password'] = request_specs['swift_password'] + + job_args['swift'] = swift_args + + model_args = {'type': None} + job_args['model'] = model_args + + LOG.debug("Execute job with args: %s", job_args) + + configs = {'configs': {'edp.java.main_class': 'sahara.dummy', + 'edp.spark.adapt_for_swift': True}, + 'args': [request_specs['id'], + base64.b64encode(str(job_args))]} + + result = self.cluster_api.job_create( + context, job_template_id, cluster_id, configs) + + return result + + def delete_dataset(self, context, cluster_id, job_id, id): + """Delete Dataset.""" + + dir_name = 'data-' + id + + self._delete_hdfs_dir(context, cluster_id, dir_name) + self.cluster_api.job_delete(context, job_id) + + def create_model(self, context, request_specs): + """Create Model.""" + + job_args = {} + + job_template_id = request_specs['job_template_id'] + cluster_id = request_specs['cluster_id'] + source_dataset_url = request_specs['source_dataset_url'] + + job_args['method'] = 'create_model' + job_args['source_dataset_url'] = source_dataset_url + job_args['dataset_format'] = request_specs['dataset_format'] + + model_args = {'type': request_specs['model_type'], + 'params': request_specs['model_params']} + job_args['model'] = model_args + + swift_args = {} + + if source_dataset_url.count('swift'): + swift_args['tenant'] = request_specs['swift_tenant'] + swift_args['username'] = request_specs['swift_username'] + swift_args['password'] = request_specs['swift_password'] + + job_args['swift'] = swift_args + + LOG.debug("Execute job with args: %s", job_args) + + configs = {'configs': {'edp.java.main_class': 'sahara.dummy', + 'edp.spark.adapt_for_swift': True}, + 'args': [request_specs['id'], + base64.b64encode(str(job_args))]} + + result = self.cluster_api.job_create( + context, job_template_id, cluster_id, configs) + + return result + + def delete_model(self, context, cluster_id, job_id, id): + """Delete Model.""" + + dir_name = 'model-' + id + + self._delete_hdfs_dir(context, cluster_id, dir_name) + self.cluster_api.job_delete(context, job_id) + + def create_learning(self, context, request_specs): + """Create Learning.""" + + job_args = {} + + job_template_id = request_specs['job_template_id'] + cluster_id = request_specs['cluster_id'] + job_args['method'] = request_specs['method'] + job_args['dataset_format'] = request_specs['dataset_format'] + + model_args = {'type': request_specs['model_type']} + job_args['model'] = model_args + + learning_args = {'params': request_specs['args']} + job_args['learning'] = learning_args + + LOG.debug("Execute job with args: %s", job_args) + + configs = {'configs': {'edp.java.main_class': 'sahara.dummy', + 'edp.spark.adapt_for_swift': True}, + 'args': [request_specs['model_id'], + base64.b64encode(str(job_args))]} + + result = self.cluster_api.job_create( + context, job_template_id, cluster_id, configs) + + return result + + def delete_learning(self, context, cluster_id, job_id, id): + """Delete Learning.""" + + self.cluster_api.job_delete(context, job_id) + + def _get_job_result(self, context, template_id, cluster_id, job_id): + + result = {} + cluster = self.cluster_api.get_cluster(context, cluster_id) + node_groups = cluster.node_groups + + for node in node_groups: + if 'master' in node['node_processes']: + ip = node['instances'][0]['management_ip'] + + path = '/tmp/spark-edp/meteos-job-' + \ + template_id + '/' + job_id + '/stdout' + + stdout, stderr = self._run_ssh(ip, ['cat', path]) + + return stdout diff --git a/meteos/engine/manager.py b/meteos/engine/manager.py new file mode 100644 index 0000000..c4cc30c --- /dev/null +++ b/meteos/engine/manager.py @@ -0,0 +1,340 @@ +# Copyright (c) 2014 NetApp Inc. +# All Rights Reserved. +# Copyright (c) 2016 NEC Corporation. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. +"""NAS learning manager managers creating learnings and access rights. + +**Related Flags** + +:learning_driver: Used by :class:`LearningManager`. +""" + +import copy +import datetime +import functools + +from oslo_config import cfg +from oslo_log import log +from oslo_serialization import jsonutils +from oslo_service import periodic_task +from oslo_utils import excutils +from oslo_utils import importutils +from oslo_utils import strutils +from oslo_utils import timeutils +import six + +from meteos.common import constants +from meteos import context +from meteos import exception +from meteos.i18n import _, _LE, _LI, _LW +from meteos import manager +from meteos.engine import api +import meteos.engine.configuration +from meteos.engine import rpcapi as engine_rpcapi +from meteos import utils + +LOG = log.getLogger(__name__) + +engine_manager_opts = [ + cfg.StrOpt('learning_driver', + default='meteos.engine.drivers.generic.GenericLearningDriver', + help='Driver to use for learning creation.'), +] + +CONF = cfg.CONF +CONF.register_opts(engine_manager_opts) +CONF.import_opt('periodic_interval', 'meteos.service') + + +class LearningManager(manager.Manager): + + """Manages Learning resources.""" + + RPC_API_VERSION = '1.0' + + def __init__(self, learning_driver=None, service_name=None, + *args, **kwargs): + """Load the driver from args, or from flags.""" + self.configuration = meteos.engine.configuration.Configuration( + engine_manager_opts, + config_group=service_name) + super(LearningManager, self).__init__(*args, **kwargs) + + if not learning_driver: + learning_driver = self.configuration.learning_driver + + self.driver = importutils.import_object( + learning_driver, configuration=self.configuration, + ) + + def _update_status(self, context, resource_name, + id, job_id, stdout, stderr): + + if stderr: + status = constants.STATUS_ERROR + LOG.error(_LI("Fail to create %s %s."), resource_name, id) + else: + status = constants.STATUS_AVAILABLE + LOG.info(_LI("%s %s created successfully."), resource_name, id) + + updates = { + 'status': status, + 'job_id': job_id, + 'launched_at': timeutils.utcnow(), + 'stderr': stderr, + } + + if resource_name == 'DataSet': + updates['head'] = stdout + self.db.dataset_update(context, id, updates) + + elif resource_name == 'Model': + updates['stdout'] = stdout + self.db.model_update(context, id, updates) + + elif resource_name == 'Learning': + updates['stdout'] = stdout.rstrip('\n') + self.db.learning_update(context, id, updates) + + def create_template(self, context, request_spec=None): + """Creates a template.""" + context = context.elevated() + + LOG.debug("Create template with request: %s", request_spec) + + try: + response = self.driver.create_template( + context, request_spec) + + except Exception as e: + with excutils.save_and_reraise_exception(): + LOG.error(_LE("template %s failed on creation."), + request_spec['id']) + self.db.template_update( + context, request_spec['id'], + {'status': constants.STATUS_ERROR} + ) + + LOG.info(_LI("template %s created successfully."), + request_spec['id']) + + updates = response + updates['status'] = constants.STATUS_AVAILABLE + updates['launched_at'] = timeutils.utcnow() + + self.db.template_update(context, request_spec['id'], updates) + + def delete_template(self, context, id=None): + """Deletes a template.""" + context = context.elevated() + + try: + template = self.db.template_get(context, id) + self.driver.delete_template(context, template) + + except Exception as e: + with excutils.save_and_reraise_exception(): + LOG.error(_LE("Template %s failed on deletion."), id) + self.db.template_update( + context, id, + {'status': constants.STATUS_ERROR_DELETING} + ) + + LOG.info(_LI("Template %s deleted successfully."), id) + self.db.template_delete(context, id) + + def create_experiment(self, context, request_spec=None): + """Creates a Experiment.""" + context = context.elevated() + + LOG.debug("Create experiment with request: %s", request_spec) + + try: + template = self.db.template_get( + context, request_spec['template_id']) + cluster_id = self.driver.create_experiment( + context, request_spec, template.sahara_image_id, + template.cluster_template_id, template.spark_version) + self.driver.wait_for_cluster_create(context, cluster_id) + + experiment = request_spec + + except Exception as e: + with excutils.save_and_reraise_exception(): + LOG.error(_LE("Experiment %s failed on creation."), + request_spec['id']) + self.db.experiment_update( + context, request_spec['id'], + {'status': constants.STATUS_ERROR} + ) + + LOG.info(_LI("Experiment %s created successfully."), + experiment['id']) + updates = { + 'status': constants.STATUS_AVAILABLE, + 'launched_at': timeutils.utcnow(), + 'cluster_id': cluster_id, + } + self.db.experiment_update(context, experiment['id'], updates) + + def delete_experiment(self, context, id=None): + """Deletes a experiment.""" + context = context.elevated() + + try: + experiment = self.db.experiment_get(context, id) + self.driver.delete_experiment(context, experiment['cluster_id']) + + except Exception as e: + with excutils.save_and_reraise_exception(): + LOG.error(_LE("Experiment %s failed on deletion."), id) + self.db.experiment_update( + context, id, + {'status': constants.STATUS_ERROR_DELETING} + ) + + LOG.info(_LI("Experiment %s deleted successfully."), id) + self.db.experiment_delete(context, id) + + def create_dataset(self, context, request_spec=None): + """Create a Dataset.""" + context = context.elevated() + + LOG.debug("Create dataset with request: %s", request_spec) + + try: + job_id = self.driver.create_dataset(context, request_spec) + stdout, stderr = self.driver.get_job_result( + context, + job_id, + request_spec['template_id'], + request_spec['cluster_id']) + + except Exception as e: + with excutils.save_and_reraise_exception(): + LOG.error(_LE("Dataset %s failed on creation."), + request_spec['id']) + self.db.dataset_update( + context, request_spec['id'], + {'status': constants.STATUS_ERROR} + ) + + self._update_status(context, 'DataSet', request_spec['id'], + job_id, stdout, stderr) + + def delete_dataset(self, context, cluster_id=None, job_id=None, id=None): + """Deletes a Dataset.""" + context = context.elevated() + + try: + self.driver.delete_dataset(context, cluster_id, job_id, id) + + except Exception as e: + with excutils.save_and_reraise_exception(): + LOG.error(_LE("Dataset %s failed on deletion."), id) + self.db.dataset_update( + context, id, + {'status': constants.STATUS_ERROR_DELETING} + ) + + LOG.info(_LI("Dataset %s deleted successfully."), id) + self.db.dataset_delete(context, id) + + def create_model(self, context, request_spec=None): + """Create a Model.""" + context = context.elevated() + + LOG.debug("Create model with request: %s", request_spec) + + try: + job_id = self.driver.create_model(context, request_spec) + stdout, stderr = self.driver.get_job_result( + context, + job_id, + request_spec['template_id'], + request_spec['cluster_id']) + + except Exception as e: + with excutils.save_and_reraise_exception(): + LOG.error(_LE("Model %s failed on creation."), + request_spec['id']) + self.db.model_update( + context, request_spec['id'], + {'status': constants.STATUS_ERROR} + ) + + self._update_status(context, 'Model', request_spec['id'], + job_id, stdout, stderr) + + def delete_model(self, context, cluster_id=None, job_id=None, id=None): + """Deletes a Model.""" + context = context.elevated() + + try: + self.driver.delete_model(context, cluster_id, job_id, id) + + except Exception as e: + with excutils.save_and_reraise_exception(): + LOG.error(_LE("Model %s failed on deletion."), id) + self.db.model_update( + context, id, + {'status': constants.STATUS_ERROR_DELETING} + ) + + LOG.info(_LI("Model %s deleted successfully."), id) + self.db.model_delete(context, id) + + def create_learning(self, context, request_spec=None): + """Create a Learning.""" + context = context.elevated() + + LOG.debug("Create learning with request: %s", request_spec) + + try: + job_id = self.driver.create_learning(context, request_spec) + stdout, stderr = self.driver.get_job_result( + context, + job_id, + request_spec['template_id'], + request_spec['cluster_id']) + + except Exception as e: + with excutils.save_and_reraise_exception(): + LOG.error(_LE("Learning %s failed on creation."), + request_spec['id']) + self.db.learning_update( + context, request_spec['id'], + {'status': constants.STATUS_ERROR} + ) + + self._update_status(context, 'Learning', request_spec['id'], + job_id, stdout, stderr) + + def delete_learning(self, context, cluster_id=None, job_id=None, id=None): + """Deletes a Learning.""" + context = context.elevated() + + try: + self.driver.delete_learning(context, cluster_id, job_id, id) + + except Exception as e: + with excutils.save_and_reraise_exception(): + LOG.error(_LE("Learning %s failed on deletion."), id) + self.db.learning_update( + context, id, + {'status': constants.STATUS_ERROR_DELETING} + ) + + self.db.learning_delete(context, id) + LOG.info(_LI("Learning %s deleted successfully."), id) diff --git a/meteos/engine/rpcapi.py b/meteos/engine/rpcapi.py new file mode 100644 index 0000000..513e320 --- /dev/null +++ b/meteos/engine/rpcapi.py @@ -0,0 +1,118 @@ +# Copyright 2012, Intel, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +""" +Client side of the learning RPC API. +""" + +from oslo_config import cfg +import oslo_messaging as messaging +from oslo_serialization import jsonutils + +from meteos import rpc + +CONF = cfg.CONF + + +class LearningAPI(object): + """Client side of the learning rpc API. + + API version history: + + 1.0 - Initial version. + """ + + BASE_RPC_API_VERSION = '1.0' + + def __init__(self, topic=None): + super(LearningAPI, self).__init__() + target = messaging.Target(topic=CONF.learning_topic, + version=self.BASE_RPC_API_VERSION) + self._client = rpc.get_client(target, version_cap='1.12') + + @staticmethod + def make_msg(method, **kwargs): + return method, kwargs + + def call(self, ctxt, msg, version=None, timeout=None): + method, kwargs = msg + + if version is not None: + client = self._client.prepare(version=version) + else: + client = self._client + + if timeout is not None: + client = client.prepare(timeout=timeout) + + return client.call(ctxt, method, **kwargs) + + def cast(self, ctxt, msg, version=None): + method, kwargs = msg + if version is not None: + client = self._client.prepare(version=version) + else: + client = self._client + return client.cast(ctxt, method, **kwargs) + + def create_template(self, context, request_spec): + request_spec_p = jsonutils.to_primitive(request_spec) + return self.call(context, self.make_msg('create_template', + request_spec=request_spec_p)) + + def delete_template(self, context, id): + return self.call(context, self.make_msg('delete_template', + id=id)) + + def create_experiment(self, context, request_spec): + request_spec_p = jsonutils.to_primitive(request_spec) + return self.cast(context, self.make_msg('create_experiment', + request_spec=request_spec_p)) + + def delete_experiment(self, context, id): + return self.cast(context, self.make_msg('delete_experiment', + id=id)) + + def create_dataset(self, context, request_spec): + request_spec_p = jsonutils.to_primitive(request_spec) + return self.cast(context, self.make_msg('create_dataset', + request_spec=request_spec_p)) + + def delete_dataset(self, context, cluster_id, job_id, id): + return self.call(context, self.make_msg('delete_dataset', + cluster_id=cluster_id, + job_id=job_id, + id=id)) + + def create_model(self, context, request_spec): + request_spec_p = jsonutils.to_primitive(request_spec) + return self.cast(context, self.make_msg('create_model', + request_spec=request_spec_p)) + + def delete_model(self, context, cluster_id, job_id, id): + return self.call(context, self.make_msg('delete_model', + cluster_id=cluster_id, + job_id=job_id, + id=id)) + + def create_learning(self, context, request_spec): + request_spec_p = jsonutils.to_primitive(request_spec) + return self.cast(context, self.make_msg('create_learning', + request_spec=request_spec_p)) + + def delete_learning(self, context, cluster_id, job_id, id): + return self.call(context, self.make_msg('delete_learning', + cluster_id=cluster_id, + job_id=job_id, + id=id)) diff --git a/meteos/exception.py b/meteos/exception.py new file mode 100644 index 0000000..e82716e --- /dev/null +++ b/meteos/exception.py @@ -0,0 +1,194 @@ +# Copyright 2010 United States Government as represented by the +# Administrator of the National Aeronautics and Space Administration. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +"""Meteos base exception handling. + +Includes decorator for re-raising Meteos-type exceptions. + +SHOULD include dedicated exception logging. + +""" +import re + +from oslo_concurrency import processutils +from oslo_config import cfg +from oslo_log import log +import six +import webob.exc + +from meteos.i18n import _ +from meteos.i18n import _LE + +LOG = log.getLogger(__name__) + +exc_log_opts = [ + cfg.BoolOpt('fatal_exception_format_errors', + default=False, + help='Whether to make exception message format errors fatal.'), +] + +CONF = cfg.CONF +CONF.register_opts(exc_log_opts) + + +ProcessExecutionError = processutils.ProcessExecutionError + + +class ConvertedException(webob.exc.WSGIHTTPException): + def __init__(self, code=400, title="", explanation=""): + self.code = code + self.title = title + self.explanation = explanation + super(ConvertedException, self).__init__() + + +class Error(Exception): + pass + + +class MeteosException(Exception): + """Base Meteos Exception + + To correctly use this class, inherit from it and define + a 'message' property. That message will get printf'd + with the keyword arguments provided to the constructor. + + """ + message = _("An unknown exception occurred.") + code = 500 + headers = {} + safe = False + + def __init__(self, message=None, detail_data={}, **kwargs): + self.kwargs = kwargs + self.detail_data = detail_data + + if 'code' not in self.kwargs: + try: + self.kwargs['code'] = self.code + except AttributeError: + pass + for k, v in self.kwargs.items(): + if isinstance(v, Exception): + self.kwargs[k] = six.text_type(v) + + if not message: + try: + message = self.message % kwargs + + except Exception: + # kwargs doesn't match a variable in the message + # log the issue and the kwargs + LOG.exception(_LE('Exception in string format operation.')) + for name, value in kwargs.items(): + LOG.error(_LE("%(name)s: %(value)s"), { + 'name': name, 'value': value}) + if CONF.fatal_exception_format_errors: + raise + else: + # at least get the core message out if something happened + message = self.message + elif isinstance(message, Exception): + message = six.text_type(message) + + if re.match('.*[^\.]\.\.$', message): + message = message[:-1] + self.msg = message + super(MeteosException, self).__init__(message) + + +class Conflict(MeteosException): + message = _("%(err)s") + code = 409 + + +class Invalid(MeteosException): + message = _("Unacceptable parameters.") + code = 400 + + +class InvalidRequest(Invalid): + message = _("The request is invalid.") + + +class InvalidResults(Invalid): + message = _("The results are invalid.") + + +class InvalidInput(Invalid): + message = _("Invalid input received: %(reason)s.") + + +class InvalidContentType(Invalid): + message = _("Invalid content type %(content_type)s.") + + +class InvalidParameterValue(Invalid): + message = _("%(err)s") + + +class InvalidUUID(Invalid): + message = _("Expected a uuid but received %(uuid)s.") + + +class InvalidLearning(Invalid): + message = _("Invalid learning: %(reason)s.") + + +class NotAuthorized(MeteosException): + message = _("Not authorized.") + code = 403 + + +class NotFound(MeteosException): + message = _("Resource could not be found.") + code = 404 + safe = True + + +class VersionNotFoundForAPIMethod(Invalid): + message = _("API version %(version)s is not supported on this method.") + + +class HostBinaryNotFound(NotFound): + message = _("Could not find binary %(binary)s on host %(host)s.") + + +class MalformedRequestBody(MeteosException): + message = _("Malformed message body: %(reason)s.") + + +class NotAuthorized(MeteosException): + message = _("Not authorized.") + code = 403 + + +class AdminRequired(NotAuthorized): + message = _("User does not have admin privileges.") + + +class PolicyNotAuthorized(NotAuthorized): + message = _("Policy doesn't allow %(action)s to be performed.") + + +class DriverNotInitialized(MeteosException): + message = _("Share driver '%(driver)s' not initialized.") + + +class Duplicated(MeteosException): + message = _("Duplicate entry") + code = 409 + safe = True diff --git a/meteos/hacking/__init__.py b/meteos/hacking/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/meteos/hacking/checks.py b/meteos/hacking/checks.py new file mode 100644 index 0000000..dcb379d --- /dev/null +++ b/meteos/hacking/checks.py @@ -0,0 +1,360 @@ +# Copyright (c) 2012, Cloudscaling +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +import ast +import re +import six + +import pep8 + + +""" +Guidelines for writing new hacking checks + + - Use only for Meteos specific tests. OpenStack general tests + should be submitted to the common 'hacking' module. + - Pick numbers in the range M3xx. Find the current test with + the highest allocated number and then pick the next value. + - Keep the test method code in the source file ordered based + on the M3xx value. + - List the new rule in the top level HACKING.rst file + - Add test cases for each new rule to meteos/tests/test_hacking.py + +""" + +UNDERSCORE_IMPORT_FILES = [] + +log_translation = re.compile( + r"(.)*LOG\.(audit|error|info|critical|exception)\(\s*('|\")") +log_translation_LC = re.compile( + r"(.)*LOG\.(critical)\(\s*(_\(|'|\")") +log_translation_LE = re.compile( + r"(.)*LOG\.(error|exception)\(\s*(_\(|'|\")") +log_translation_LI = re.compile( + r"(.)*LOG\.(info)\(\s*(_\(|'|\")") +log_translation_LW = re.compile( + r"(.)*LOG\.(warning|warn)\(\s*(_\(|'|\")") +translated_log = re.compile( + r"(.)*LOG\.(audit|error|info|warn|warning|critical|exception)" + "\(\s*_\(\s*('|\")") +string_translation = re.compile(r"[^_]*_\(\s*('|\")") +underscore_import_check = re.compile(r"(.)*import _$") +underscore_import_check_multi = re.compile(r"(.)*import (.)*_, (.)*") +# We need this for cases where they have created their own _ function. +custom_underscore_check = re.compile(r"(.)*_\s*=\s*(.)*") +oslo_namespace_imports = re.compile(r"from[\s]*oslo[.](.*)") +dict_constructor_with_list_copy_re = re.compile(r".*\bdict\((\[)?(\(|\[)") +assert_no_xrange_re = re.compile(r"\s*xrange\s*\(") +assert_True = re.compile(r".*assertEqual\(True, .*\)") +assert_None = re.compile(r".*assertEqual\(None, .*\)") + + +class BaseASTChecker(ast.NodeVisitor): + + """Provides a simple framework for writing AST-based checks. + + Subclasses should implement visit_* methods like any other AST visitor + implementation. When they detect an error for a particular node the + method should call ``self.add_error(offending_node)``. Details about + where in the code the error occurred will be pulled from the node + object. + + Subclasses should also provide a class variable named CHECK_DESC to + be used for the human readable error message. + + """ + + CHECK_DESC = 'No check message specified' + + def __init__(self, tree, filename): + """This object is created automatically by pep8. + + :param tree: an AST tree + :param filename: name of the file being analyzed + (ignored by our checks) + """ + self._tree = tree + self._errors = [] + + def run(self): + """Called automatically by pep8.""" + self.visit(self._tree) + return self._errors + + def add_error(self, node, message=None): + """Add an error caused by a node to the list of errors for pep8.""" + message = message or self.CHECK_DESC + error = (node.lineno, node.col_offset, message, self.__class__) + self._errors.append(error) + + def _check_call_names(self, call_node, names): + if isinstance(call_node, ast.Call): + if isinstance(call_node.func, ast.Name): + if call_node.func.id in names: + return True + return False + + +def no_translate_debug_logs(logical_line, filename): + """Check for 'LOG.debug(_(' + + As per our translation policy, + https://wiki.openstack.org/wiki/LoggingStandards#Log_Translation + we shouldn't translate debug level logs. + + * This check assumes that 'LOG' is a logger. + * Use filename so we can start enforcing this in specific folders instead + of needing to do so all at once. + + M319 + """ + if logical_line.startswith("LOG.debug(_("): + yield(0, "M319 Don't translate debug level logs") + + +class CheckLoggingFormatArgs(BaseASTChecker): + + """Check for improper use of logging format arguments. + + LOG.debug("Volume %s caught fire and is at %d degrees C and climbing.", + ('volume1', 500)) + + The format arguments should not be a tuple as it is easy to miss. + + """ + + CHECK_DESC = 'M310 Log method arguments should not be a tuple.' + LOG_METHODS = [ + 'debug', 'info', + 'warn', 'warning', + 'error', 'exception', + 'critical', 'fatal', + 'trace', 'log' + ] + + def _find_name(self, node): + """Return the fully qualified name or a Name or Attribute.""" + if isinstance(node, ast.Name): + return node.id + elif (isinstance(node, ast.Attribute) + and isinstance(node.value, (ast.Name, ast.Attribute))): + method_name = node.attr + obj_name = self._find_name(node.value) + if obj_name is None: + return None + return obj_name + '.' + method_name + elif isinstance(node, six.string_types): + return node + else: # could be Subscript, Call or many more + return None + + def visit_Call(self, node): + """Look for the 'LOG.*' calls.""" + # extract the obj_name and method_name + if isinstance(node.func, ast.Attribute): + obj_name = self._find_name(node.func.value) + if isinstance(node.func.value, ast.Name): + method_name = node.func.attr + elif isinstance(node.func.value, ast.Attribute): + obj_name = self._find_name(node.func.value) + method_name = node.func.attr + else: # could be Subscript, Call or many more + return super(CheckLoggingFormatArgs, self).generic_visit(node) + + # obj must be a logger instance and method must be a log helper + if (obj_name != 'LOG' + or method_name not in self.LOG_METHODS): + return super(CheckLoggingFormatArgs, self).generic_visit(node) + + # the call must have arguments + if not len(node.args): + return super(CheckLoggingFormatArgs, self).generic_visit(node) + + # any argument should not be a tuple + for arg in node.args: + if isinstance(arg, ast.Tuple): + self.add_error(arg) + + return super(CheckLoggingFormatArgs, self).generic_visit(node) + + +def validate_log_translations(logical_line, physical_line, filename): + # Translations are not required in the test and tempest + # directories. + if ("meteos/tests" in filename or "meteos_tempest_tests" in filename or + "contrib/tempest" in filename): + return + if pep8.noqa(physical_line): + return + msg = "M327: LOG.critical messages require translations `_LC()`!" + if log_translation_LC.match(logical_line): + yield (0, msg) + msg = ("M328: LOG.error and LOG.exception messages require translations " + "`_LE()`!") + if log_translation_LE.match(logical_line): + yield (0, msg) + msg = "M329: LOG.info messages require translations `_LI()`!" + if log_translation_LI.match(logical_line): + yield (0, msg) + msg = "M330: LOG.warning messages require translations `_LW()`!" + if log_translation_LW.match(logical_line): + yield (0, msg) + msg = "M331: Log messages require translations!" + if log_translation.match(logical_line): + yield (0, msg) + + +def check_explicit_underscore_import(logical_line, filename): + """Check for explicit import of the _ function + + We need to ensure that any files that are using the _() function + to translate logs are explicitly importing the _ function. We + can't trust unit test to catch whether the import has been + added so we need to check for it here. + """ + + # Build a list of the files that have _ imported. No further + # checking needed once it is found. + if filename in UNDERSCORE_IMPORT_FILES: + pass + elif (underscore_import_check.match(logical_line) or + underscore_import_check_multi.match(logical_line) or + custom_underscore_check.match(logical_line)): + UNDERSCORE_IMPORT_FILES.append(filename) + elif (translated_log.match(logical_line) or + string_translation.match(logical_line)): + yield(0, "M323: Found use of _() without explicit import of _ !") + + +class CheckForStrUnicodeExc(BaseASTChecker): + + """Checks for the use of str() or unicode() on an exception. + + This currently only handles the case where str() or unicode() + is used in the scope of an exception handler. If the exception + is passed into a function, returned from an assertRaises, or + used on an exception created in the same scope, this does not + catch it. + """ + + CHECK_DESC = ('M325 str() and unicode() cannot be used on an ' + 'exception. Remove or use six.text_type()') + + def __init__(self, tree, filename): + super(CheckForStrUnicodeExc, self).__init__(tree, filename) + self.name = [] + self.already_checked = [] + + # Python 2 + def visit_TryExcept(self, node): + for handler in node.handlers: + if handler.name: + self.name.append(handler.name.id) + super(CheckForStrUnicodeExc, self).generic_visit(node) + self.name = self.name[:-1] + else: + super(CheckForStrUnicodeExc, self).generic_visit(node) + + # Python 3 + def visit_ExceptHandler(self, node): + if node.name: + self.name.append(node.name) + super(CheckForStrUnicodeExc, self).generic_visit(node) + self.name = self.name[:-1] + else: + super(CheckForStrUnicodeExc, self).generic_visit(node) + + def visit_Call(self, node): + if self._check_call_names(node, ['str', 'unicode']): + if node not in self.already_checked: + self.already_checked.append(node) + if isinstance(node.args[0], ast.Name): + if node.args[0].id in self.name: + self.add_error(node.args[0]) + super(CheckForStrUnicodeExc, self).generic_visit(node) + + +class CheckForTransAdd(BaseASTChecker): + + """Checks for the use of concatenation on a translated string. + + Translations should not be concatenated with other strings, but + should instead include the string being added to the translated + string to give the translators the most information. + """ + + CHECK_DESC = ('M326 Translated messages cannot be concatenated. ' + 'String should be included in translated message.') + + TRANS_FUNC = ['_', '_LI', '_LW', '_LE', '_LC'] + + def visit_BinOp(self, node): + if isinstance(node.op, ast.Add): + if self._check_call_names(node.left, self.TRANS_FUNC): + self.add_error(node.left) + elif self._check_call_names(node.right, self.TRANS_FUNC): + self.add_error(node.right) + super(CheckForTransAdd, self).generic_visit(node) + + +def check_oslo_namespace_imports(logical_line, physical_line, filename): + if pep8.noqa(physical_line): + return + if re.match(oslo_namespace_imports, logical_line): + msg = ("M333: '%s' must be used instead of '%s'.") % ( + logical_line.replace('oslo.', 'oslo_'), + logical_line) + yield(0, msg) + + +def dict_constructor_with_list_copy(logical_line): + msg = ("M336: Must use a dict comprehension instead of a dict constructor" + " with a sequence of key-value pairs." + ) + if dict_constructor_with_list_copy_re.match(logical_line): + yield (0, msg) + + +def no_xrange(logical_line): + if assert_no_xrange_re.match(logical_line): + yield(0, "M337: Do not use xrange().") + + +def validate_assertTrue(logical_line): + if re.match(assert_True, logical_line): + msg = ("M313: Unit tests should use assertTrue(value) instead" + " of using assertEqual(True, value).") + yield(0, msg) + + +def validate_assertIsNone(logical_line): + if re.match(assert_None, logical_line): + msg = ("M312: Unit tests should use assertIsNone(value) instead" + " of using assertEqual(None, value).") + yield(0, msg) + + +def factory(register): + register(validate_log_translations) + register(check_explicit_underscore_import) + register(no_translate_debug_logs) + register(CheckForStrUnicodeExc) + register(CheckLoggingFormatArgs) + register(CheckForTransAdd) + register(check_oslo_namespace_imports) + register(dict_constructor_with_list_copy) + register(no_xrange) + register(validate_assertTrue) + register(validate_assertIsNone) diff --git a/meteos/i18n.py b/meteos/i18n.py new file mode 100644 index 0000000..31372ae --- /dev/null +++ b/meteos/i18n.py @@ -0,0 +1,50 @@ +# Copyright 2014 IBM Corp. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +"""oslo.i18n integration module. + +See http://docs.openstack.org/developer/oslo.i18n/usage.html . + +""" + +import oslo_i18n + +DOMAIN = 'meteos' + +_translators = oslo_i18n.TranslatorFactory(domain=DOMAIN) + +# The primary translation function using the well-known name "_" +_ = _translators.primary + +# Translators for log levels. +# +# The abbreviated names are meant to reflect the usual use of a short +# name like '_'. The "L" is for "log" and the other letter comes from +# the level. +_LI = _translators.log_info +_LW = _translators.log_warning +_LE = _translators.log_error +_LC = _translators.log_critical + + +def enable_lazy(): + return oslo_i18n.enable_lazy() + + +def translate(value, user_locale): + return oslo_i18n.translate(value, user_locale) + + +def get_available_languages(): + return oslo_i18n.get_available_languages(DOMAIN) diff --git a/meteos/manager.py b/meteos/manager.py new file mode 100644 index 0000000..3a4042a --- /dev/null +++ b/meteos/manager.py @@ -0,0 +1,114 @@ +# Copyright 2010 United States Government as represented by the +# Administrator of the National Aeronautics and Space Administration. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +"""Base Manager class. + +Managers are responsible for a certain aspect of the system. It is a logical +grouping of code relating to a portion of the system. In general other +components should be using the manager to make changes to the components that +it is responsible for. + +For example, other components that need to deal with volumes in some way, +should do so by calling methods on the VolumeManager instead of directly +changing fields in the database. This allows us to keep all of the code +relating to volumes in the same place. + +We have adopted a basic strategy of Smart managers and dumb data, which means +rather than attaching methods to data objects, components should call manager +methods that act on the data. + +Methods on managers that can be executed locally should be called directly. If +a particular method must execute on a remote host, this should be done via rpc +to the service that wraps the manager + +Managers should be responsible for most of the db access, and +non-implementation specific data. Anything implementation specific that can't +be generalized should be done by the Driver. + +In general, we prefer to have one manager with multiple drivers for different +implementations, but sometimes it makes sense to have multiple managers. You +can think of it this way: Abstract different overall strategies at the manager +level(FlatNetwork vs VlanNetwork), and different implementations at the driver +level(LinuxNetDriver vs CiscoNetDriver). + +Managers will often provide methods for initial setup of a host or periodic +tasks to a wrapping service. + +This module provides Manager, a base class for managers. + +""" + +from oslo_config import cfg +from oslo_log import log +from oslo_service import periodic_task + +from meteos.db import base +from meteos import version + +CONF = cfg.CONF +LOG = log.getLogger(__name__) + + +class PeriodicTasks(periodic_task.PeriodicTasks): + def __init__(self): + super(PeriodicTasks, self).__init__(CONF) + + +class Manager(base.Base, PeriodicTasks): + + @property + def RPC_API_VERSION(self): + """Redefine this in child classes.""" + raise NotImplementedError + + @property + def target(self): + """This property is used by oslo_messaging. + + https://wiki.openstack.org/wiki/Oslo/Messaging#API_Version_Negotiation + """ + if not hasattr(self, '_target'): + import oslo_messaging as messaging + self._target = messaging.Target(version=self.RPC_API_VERSION) + return self._target + + def __init__(self, host=None, db_driver=None): + if not host: + host = CONF.host + self.host = host + self.additional_endpoints = [] + super(Manager, self).__init__(db_driver) + + def periodic_tasks(self, context, raise_on_error=False): + """Tasks to be run at a periodic interval.""" + return self.run_periodic_tasks(context, raise_on_error=raise_on_error) + + def init_host(self): + """Handle initialization if this is a standalone service. + + Child classes should override this method. + + """ + pass + + def service_version(self, context): + return version.version_string() + + def service_config(self, context): + config = {} + for key in CONF: + config[key] = CONF.get(key, None) + return config diff --git a/meteos/opts.py b/meteos/opts.py new file mode 100644 index 0000000..7a07233 --- /dev/null +++ b/meteos/opts.py @@ -0,0 +1,75 @@ +# Copyright (c) 2014 SUSE Linux Products GmbH. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +__all__ = [ + 'list_opts' +] + +import copy +import itertools + +import oslo_concurrency.opts +import oslo_log._options +import oslo_middleware.opts +import oslo_policy.opts + +import meteos.api.common +import meteos.api.middleware.auth +import meteos.common.config +import meteos.db.api +import meteos.db.base +import meteos.exception +import meteos.service +import meteos.engine.api +import meteos.engine.driver +import meteos.engine.drivers.generic +import meteos.engine.manager +import meteos.cluster.sahara +import meteos.wsgi + + +# List of *all* options in [DEFAULT] namespace of meteos. +# Any new option list or option needs to be registered here. +_global_opt_lists = [ + # Keep list alphabetically sorted + meteos.api.common.api_common_opts, + [meteos.api.middleware.auth.use_forwarded_for_opt], + meteos.common.config.core_opts, + meteos.common.config.debug_opts, + meteos.common.config.global_opts, + meteos.cluster.sahara.sahara_opts, + meteos.db.api.db_opts, + [meteos.db.base.db_driver_opt], + meteos.exception.exc_log_opts, + meteos.service.service_opts, + meteos.engine.driver.ssh_opts, + meteos.engine.drivers.generic.learning_opts, + meteos.engine.manager.engine_manager_opts, + meteos.wsgi.eventlet_opts, + meteos.wsgi.socket_opts, +] + +_opts = [ + (None, list(itertools.chain(*_global_opt_lists))) +] + +_opts.extend(oslo_concurrency.opts.list_opts()) +_opts.extend(oslo_log._options.list_opts()) +_opts.extend(oslo_middleware.opts.list_opts()) +_opts.extend(oslo_policy.opts.list_opts()) + + +def list_opts(): + """Return a list of oslo.config options available in Meteos.""" + return [(m, copy.deepcopy(o)) for m, o in _opts] diff --git a/meteos/policy.py b/meteos/policy.py new file mode 100644 index 0000000..ef910a2 --- /dev/null +++ b/meteos/policy.py @@ -0,0 +1,113 @@ +# Copyright (c) 2011 OpenStack, LLC. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +"""Policy Engine For Meteos""" + +import functools + +from oslo_config import cfg +from oslo_policy import policy + +from meteos import exception + +CONF = cfg.CONF +_ENFORCER = None + + +def reset(): + global _ENFORCER + if _ENFORCER: + _ENFORCER.clear() + _ENFORCER = None + + +def init(policy_path=None): + global _ENFORCER + if not _ENFORCER: + _ENFORCER = policy.Enforcer(CONF) + if policy_path: + _ENFORCER.policy_path = policy_path + _ENFORCER.load_rules() + + +def enforce(context, action, target, do_raise=True): + """Verifies that the action is valid on the target in this context. + + :param context: meteos context + :param action: string representing the action to be checked + this should be colon separated for clarity. + i.e. ``compute:create_instance``, + ``compute:attach_volume``, + ``volume:attach_volume`` + :param target: dictionary representing the object of the action + for object creation this should be a dictionary representing the + location of the object e.g. ``{'project_id': context.project_id}`` + :param do_raise: Whether to raise an exception if check fils. + + :returns: When ``do_raise`` is ``False``, returns a value that + evaluates as ``True`` or ``False`` depending on whether + the policy allows action on the target. + + :raises: meteos.exception.PolicyNotAuthorized if verification fails + and ``do_raise`` is ``True``. + + """ + init() + if not isinstance(context, dict): + context = context.to_dict() + + # Add the exception arguments if asked to do a raise + extra = {} + if do_raise: + extra.update(exc=exception.PolicyNotAuthorized, action=action, + do_raise=do_raise) + return _ENFORCER.enforce(action, target, context, **extra) + + +def check_is_admin(roles): + """Whether or not roles contains 'admin' role according to policy setting. + + """ + init() + + # include project_id on target to avoid KeyError if context_is_admin + # policy definition is missing, and default admin_or_owner rule + # attempts to apply. Since our credentials dict does not include a + # project_id, this target can never match as a generic rule. + target = {'project_id': ''} + credentials = {'roles': roles} + return _ENFORCER.enforce("context_is_admin", target, credentials) + + +def wrap_check_policy(resource): + """Check policy corresponding to the wrapped methods prior to execution.""" + def check_policy_wraper(func): + @functools.wraps(func) + def wrapped(self, context, target_obj, *args, **kwargs): + check_policy(context, resource, func.__name__, target_obj) + return func(self, context, target_obj, *args, **kwargs) + + return wrapped + return check_policy_wraper + + +def check_policy(context, resource, action, target_obj=None): + target = { + 'project_id': context.project_id, + 'user_id': context.user_id, + } + target.update(target_obj or {}) + _action = '%s:%s' % (resource, action) + enforce(context, _action, target) diff --git a/meteos/rpc.py b/meteos/rpc.py new file mode 100644 index 0000000..3b396d1 --- /dev/null +++ b/meteos/rpc.py @@ -0,0 +1,153 @@ +# Copyright 2013 Red Hat, Inc. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +__all__ = [ + 'init', + 'cleanup', + 'set_defaults', + 'add_extra_exmods', + 'clear_extra_exmods', + 'get_allowed_exmods', + 'RequestContextSerializer', + 'get_client', + 'get_server', + 'get_notifier', + 'TRANSPORT_ALIASES', +] + +from oslo_config import cfg +import oslo_messaging as messaging +from oslo_serialization import jsonutils + +import meteos.context +import meteos.exception + +CONF = cfg.CONF +TRANSPORT = None +NOTIFIER = None + +ALLOWED_EXMODS = [ + meteos.exception.__name__, +] +EXTRA_EXMODS = [] + +# NOTE(flaper87): The meteos.openstack.common.rpc entries are +# for backwards compat with Havana rpc_backend configuration +# values. The meteos.rpc entries are for compat with Folsom values. +TRANSPORT_ALIASES = { + 'meteos.openstack.common.rpc.impl_kombu': 'rabbit', + 'meteos.openstack.common.rpc.impl_qpid': 'qpid', + 'meteos.openstack.common.rpc.impl_zmq': 'zmq', + 'meteos.rpc.impl_kombu': 'rabbit', + 'meteos.rpc.impl_qpid': 'qpid', + 'meteos.rpc.impl_zmq': 'zmq', +} + + +def init(conf): + global TRANSPORT, NOTIFIER + exmods = get_allowed_exmods() + TRANSPORT = messaging.get_transport(conf, + allowed_remote_exmods=exmods, + aliases=TRANSPORT_ALIASES) + + serializer = RequestContextSerializer(JsonPayloadSerializer()) + NOTIFIER = messaging.Notifier(TRANSPORT, serializer=serializer) + + +def initialized(): + return None not in [TRANSPORT, NOTIFIER] + + +def cleanup(): + global TRANSPORT, NOTIFIER + assert TRANSPORT is not None + assert NOTIFIER is not None + TRANSPORT.cleanup() + TRANSPORT = NOTIFIER = None + + +def set_defaults(control_exchange): + messaging.set_transport_defaults(control_exchange) + + +def add_extra_exmods(*args): + EXTRA_EXMODS.extend(args) + + +def clear_extra_exmods(): + del EXTRA_EXMODS[:] + + +def get_allowed_exmods(): + return ALLOWED_EXMODS + EXTRA_EXMODS + + +class JsonPayloadSerializer(messaging.NoOpSerializer): + + @staticmethod + def serialize_entity(context, entity): + return jsonutils.to_primitive(entity, convert_instances=True) + + +class RequestContextSerializer(messaging.Serializer): + + def __init__(self, base): + self._base = base + + def serialize_entity(self, context, entity): + if not self._base: + return entity + return self._base.serialize_entity(context, entity) + + def deserialize_entity(self, context, entity): + if not self._base: + return entity + return self._base.deserialize_entity(context, entity) + + def serialize_context(self, context): + return context.to_dict() + + def deserialize_context(self, context): + return meteos.context.RequestContext.from_dict(context) + + +def get_transport_url(url_str=None): + return messaging.TransportURL.parse(CONF, url_str, TRANSPORT_ALIASES) + + +def get_client(target, version_cap=None, serializer=None): + assert TRANSPORT is not None + serializer = RequestContextSerializer(serializer) + return messaging.RPCClient(TRANSPORT, + target, + version_cap=version_cap, + serializer=serializer) + + +def get_server(target, endpoints, serializer=None): + assert TRANSPORT is not None + serializer = RequestContextSerializer(serializer) + return messaging.get_rpc_server(TRANSPORT, + target, + endpoints, + executor='eventlet', + serializer=serializer) + + +def get_notifier(service=None, host=None, publisher_id=None): + assert NOTIFIER is not None + if not publisher_id: + publisher_id = "%s.%s" % (service, host or CONF.host) + return NOTIFIER.prepare(publisher_id=publisher_id) diff --git a/meteos/service.py b/meteos/service.py new file mode 100644 index 0000000..b4175f3 --- /dev/null +++ b/meteos/service.py @@ -0,0 +1,379 @@ +# Copyright 2010 United States Government as represented by the +# Administrator of the National Aeronautics and Space Administration. +# Copyright 2011 Justin Santa Barbara +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +"""Generic Node base class for all workers that run on hosts.""" + +import inspect +import os +import random + +from oslo_config import cfg +from oslo_log import log +import oslo_messaging as messaging +from oslo_service import loopingcall +from oslo_service import service +from oslo_utils import importutils + +from meteos import context +from meteos import db +from meteos import exception +from meteos.i18n import _, _LE, _LI, _LW +from meteos import rpc +from meteos import version +from meteos import wsgi + +LOG = log.getLogger(__name__) + +service_opts = [ + cfg.IntOpt('report_interval', + default=10, + help='Seconds between nodes reporting state to datastore.'), + cfg.IntOpt('periodic_interval', + default=60, + help='Seconds between running periodic tasks.'), + cfg.IntOpt('periodic_fuzzy_delay', + default=60, + help='Range of seconds to randomly delay when starting the ' + 'periodic task scheduler to reduce stampeding. ' + '(Disable by setting to 0)'), + cfg.StrOpt('osapi_learning_listen', + default="::", + help='IP address for OpenStack Learning API to listen on.'), + cfg.PortOpt('osapi_learning_listen_port', + default=8989, + help='Port for OpenStack Learning API to listen on.'), + cfg.IntOpt('osapi_learning_workers', + default=1, + help='Number of workers for OpenStack Learning API service.'), +] + +CONF = cfg.CONF +CONF.register_opts(service_opts) + + +class Service(service.Service): + """Service object for binaries running on hosts. + + A service takes a manager and enables rpc by listening to queues based + on topic. It also periodically runs tasks on the manager and reports + it state to the database services table. + """ + + def __init__(self, host, binary, topic, manager, report_interval=None, + periodic_interval=None, periodic_fuzzy_delay=None, + service_name=None, *args, **kwargs): + super(Service, self).__init__() + if not rpc.initialized(): + rpc.init(CONF) + self.host = host + self.binary = binary + self.topic = topic + self.manager_class_name = manager + manager_class = importutils.import_class(self.manager_class_name) + self.manager = manager_class(host=self.host, + service_name=service_name, + *args, **kwargs) + self.report_interval = report_interval + self.periodic_interval = periodic_interval + self.periodic_fuzzy_delay = periodic_fuzzy_delay + self.saved_args, self.saved_kwargs = args, kwargs + self.timers = [] + + def start(self): + version_string = version.version_string() + LOG.info(_LI('Starting %(topic)s node (version %(version_string)s)'), + {'topic': self.topic, 'version_string': version_string}) + self.model_disconnected = False + ctxt = context.get_admin_context() + try: + service_ref = db.service_get_by_args(ctxt, + self.host, + self.binary) + self.service_id = service_ref['id'] + except exception.NotFound: + self._create_service_ref(ctxt) + + LOG.debug("Creating RPC server for service %s.", self.topic) + + target = messaging.Target(topic=self.topic, server=self.host) + endpoints = [self.manager] + endpoints.extend(self.manager.additional_endpoints) + self.rpcserver = rpc.get_server(target, endpoints) + self.rpcserver.start() + + self.manager.init_host() + if self.report_interval: + pulse = loopingcall.FixedIntervalLoopingCall(self.report_state) + pulse.start(interval=self.report_interval, + initial_delay=self.report_interval) + self.timers.append(pulse) + + if self.periodic_interval: + if self.periodic_fuzzy_delay: + initial_delay = random.randint(0, self.periodic_fuzzy_delay) + else: + initial_delay = None + + periodic = loopingcall.FixedIntervalLoopingCall( + self.periodic_tasks) + periodic.start(interval=self.periodic_interval, + initial_delay=initial_delay) + self.timers.append(periodic) + + def _create_service_ref(self, context): + zone = CONF.storage_availability_zone + service_ref = db.service_create(context, + {'host': self.host, + 'binary': self.binary, + 'topic': self.topic, + 'report_count': 0, + 'availability_zone': zone}) + self.service_id = service_ref['id'] + + def __getattr__(self, key): + manager = self.__dict__.get('manager', None) + return getattr(manager, key) + + @classmethod + def create(cls, host=None, binary=None, topic=None, manager=None, + report_interval=None, periodic_interval=None, + periodic_fuzzy_delay=None, service_name=None): + """Instantiates class and passes back application object. + + :param host: defaults to CONF.host + :param binary: defaults to basename of executable + :param topic: defaults to bin_name - 'meteos-' part + :param manager: defaults to CONF._manager + :param report_interval: defaults to CONF.report_interval + :param periodic_interval: defaults to CONF.periodic_interval + :param periodic_fuzzy_delay: defaults to CONF.periodic_fuzzy_delay + + """ + if not host: + host = CONF.host + if not binary: + binary = os.path.basename(inspect.stack()[-1][1]) + if not topic: + topic = binary + if not manager: + subtopic = topic.rpartition('meteos-')[2] + manager = CONF.get('%s_manager' % subtopic, None) + if report_interval is None: + report_interval = CONF.report_interval + if periodic_interval is None: + periodic_interval = CONF.periodic_interval + if periodic_fuzzy_delay is None: + periodic_fuzzy_delay = CONF.periodic_fuzzy_delay + service_obj = cls(host, binary, topic, manager, + report_interval=report_interval, + periodic_interval=periodic_interval, + periodic_fuzzy_delay=periodic_fuzzy_delay, + service_name=service_name) + + return service_obj + + def kill(self): + """Destroy the service object in the datastore.""" + self.stop() + try: + db.service_destroy(context.get_admin_context(), self.service_id) + except exception.NotFound: + LOG.warning(_LW('Service killed that has no database entry.')) + + def stop(self): + # Try to shut the connection down, but if we get any sort of + # errors, go ahead and ignore them.. as we're shutting down anyway + try: + self.rpcserver.stop() + except Exception: + pass + for x in self.timers: + try: + x.stop() + except Exception: + pass + self.timers = [] + + super(Service, self).stop() + + def wait(self): + for x in self.timers: + try: + x.wait() + except Exception: + pass + + def periodic_tasks(self, raise_on_error=False): + """Tasks to be run at a periodic interval.""" + ctxt = context.get_admin_context() + self.manager.periodic_tasks(ctxt, raise_on_error=raise_on_error) + + def report_state(self): + """Update the state of this service in the datastore.""" + ctxt = context.get_admin_context() + zone = CONF.storage_availability_zone + state_catalog = {} + try: + try: + service_ref = db.service_get(ctxt, self.service_id) + except exception.NotFound: + LOG.debug('The service database object disappeared, ' + 'Recreating it.') + self._create_service_ref(ctxt) + service_ref = db.service_get(ctxt, self.service_id) + + state_catalog['report_count'] = service_ref['report_count'] + 1 + + db.service_update(ctxt, + self.service_id, state_catalog) + + # TODO(termie): make this pattern be more elegant. + if getattr(self, 'model_disconnected', False): + self.model_disconnected = False + LOG.error(_LE('Recovered model server connection!')) + + # TODO(vish): this should probably only catch connection errors + except Exception: # pylint: disable=W0702 + if not getattr(self, 'model_disconnected', False): + self.model_disconnected = True + LOG.exception(_LE('model server went away')) + + +class WSGIService(service.ServiceBase): + """Provides ability to launch API from a 'paste' configuration.""" + + def __init__(self, name, loader=None): + """Initialize, but do not start the WSGI server. + + :param name: The name of the WSGI server given to the loader. + :param loader: Loads the WSGI application using the given name. + :returns: None + + """ + self.name = name + self.manager = self._get_manager() + self.loader = loader or wsgi.Loader() + if not rpc.initialized(): + rpc.init(CONF) + self.app = self.loader.load_app(name) + self.host = getattr(CONF, '%s_listen' % name, "0.0.0.0") + self.port = getattr(CONF, '%s_listen_port' % name, 0) + self.workers = getattr(CONF, '%s_workers' % name, None) + if self.workers is not None and self.workers < 1: + LOG.warning( + _LW("Value of config option %(name)s_workers must be integer " + "greater than 1. Input value ignored.") % {'name': name}) + # Reset workers to default + self.workers = None + self.server = wsgi.Server(name, + self.app, + host=self.host, + port=self.port) + + def _get_manager(self): + """Initialize a Manager object appropriate for this service. + + Use the service name to look up a Manager subclass from the + configuration and initialize an instance. If no class name + is configured, just return None. + + :returns: a Manager instance, or None. + + """ + fl = '%s_manager' % self.name + if fl not in CONF: + return None + + manager_class_name = CONF.get(fl, None) + if not manager_class_name: + return None + + manager_class = importutils.import_class(manager_class_name) + return manager_class() + + def start(self): + """Start serving this service using loaded configuration. + + Also, retrieve updated port number in case '0' was passed in, which + indicates a random port should be used. + + :returns: None + + """ + if self.manager: + self.manager.init_host() + self.server.start() + self.port = self.server.port + + def stop(self): + """Stop serving this API. + + :returns: None + + """ + self.server.stop() + + def wait(self): + """Wait for the service to stop serving this API. + + :returns: None + + """ + self.server.wait() + + def reset(self): + """Reset server greenpool size to default. + + :returns: None + """ + self.server.reset() + + +def process_launcher(): + return service.ProcessLauncher(CONF) + + +# NOTE(vish): the global launcher is to maintain the existing +# functionality of calling service.serve + +# service.wait +_launcher = None + + +def serve(server, workers=None): + global _launcher + if _launcher: + raise RuntimeError(_('serve() can only be called once')) + _launcher = service.launch(CONF, server, workers=workers) + + +def wait(): + LOG.debug('Full set of CONF:') + for flag in CONF: + flag_get = CONF.get(flag, None) + # hide flag contents from log if contains a password + # should use secret flag when switch over to openstack-common + if ("_password" in flag or "_key" in flag or + (flag == "sql_connection" and "mysql:" in flag_get)): + LOG.debug('%(flag)s : FLAG SET ', {"flag": flag}) + else: + LOG.debug('%(flag)s : %(flag_get)s', + {"flag": flag, "flag_get": flag_get}) + try: + _launcher.wait() + except KeyboardInterrupt: + _launcher.stop() + rpc.cleanup() diff --git a/meteos/test.py b/meteos/test.py new file mode 100644 index 0000000..8fb1013 --- /dev/null +++ b/meteos/test.py @@ -0,0 +1,354 @@ +# Copyright 2010 United States Government as represented by the +# Administrator of the National Aeronautics and Space Administration. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +"""Base classes for our unit tests. + +Allows overriding of flags for use of fakes, and some black magic for +inline callbacks. + +""" + +import os +import shutil +import uuid + +import fixtures +import mock +from oslo_concurrency import lockutils +from oslo_config import cfg +from oslo_config import fixture as config_fixture +import oslo_i18n +from oslo_messaging import conffixture as messaging_conffixture +import oslotest.base as base_test + +from meteos.db import migration +from meteos.db.sqlalchemy import api as db_api +from meteos.db.sqlalchemy import models as db_models +from meteos import rpc +from meteos import service +from meteos.tests import conf_fixture +from meteos.tests import fake_notifier + +test_opts = [ + cfg.StrOpt('sqlite_clean_db', + default='clean.sqlite', + help='File name of clean sqlite database.'), +] + +CONF = cfg.CONF +CONF.register_opts(test_opts) + +_DB_CACHE = None + + +class Database(fixtures.Fixture): + + def __init__(self, db_session, db_migrate, sql_connection, sqlite_db, + sqlite_clean_db): + self.sql_connection = sql_connection + self.sqlite_db = sqlite_db + self.sqlite_clean_db = sqlite_clean_db + self.engine = db_session.get_engine() + self.engine.dispose() + conn = self.engine.connect() + if sql_connection == "sqlite://": + self.setup_sqlite(db_migrate) + else: + testdb = os.path.join(CONF.state_path, sqlite_db) + db_migrate.upgrade('head') + if os.path.exists(testdb): + return + if sql_connection == "sqlite://": + conn = self.engine.connect() + self._DB = "".join(line for line in conn.connection.iterdump()) + self.engine.dispose() + else: + cleandb = os.path.join(CONF.state_path, sqlite_clean_db) + shutil.copyfile(testdb, cleandb) + + def setUp(self): + super(Database, self).setUp() + if self.sql_connection == "sqlite://": + conn = self.engine.connect() + conn.connection.executescript(self._DB) + self.addCleanup(self.engine.dispose) # pylint: disable=E1101 + else: + shutil.copyfile( + os.path.join(CONF.state_path, self.sqlite_clean_db), + os.path.join(CONF.state_path, self.sqlite_db), + ) + + def setup_sqlite(self, db_migrate): + if db_migrate.version(): + return + db_models.BASE.metadata.create_all(self.engine) + db_migrate.stamp('head') + + +class TestCase(base_test.BaseTestCase): + + """Test case base class for all unit tests.""" + + def setUp(self): + """Run before each test method to initialize test environment.""" + super(TestCase, self).setUp() + + oslo_i18n.enable_lazy(enable=False) + conf_fixture.set_defaults(CONF) + CONF([], default_config_files=[]) + + global _DB_CACHE + if not _DB_CACHE: + _DB_CACHE = Database( + db_api, + migration, + sql_connection=CONF.database.connection, + sqlite_db=CONF.sqlite_db, + sqlite_clean_db=CONF.sqlite_clean_db, + ) + self.useFixture(_DB_CACHE) + + self.injected = [] + self._services = [] + self.flags(fatal_exception_format_errors=True) + # This will be cleaned up by the NestedTempfile fixture + lock_path = self.useFixture(fixtures.TempDir()).path + self.fixture = self.useFixture(config_fixture.Config(lockutils.CONF)) + self.fixture.config(lock_path=lock_path, group='oslo_concurrency') + self.fixture.config( + disable_process_locking=True, group='oslo_concurrency') + + rpc.add_extra_exmods('meteos.tests') + self.addCleanup(rpc.clear_extra_exmods) + self.addCleanup(rpc.cleanup) + + self.messaging_conf = messaging_conffixture.ConfFixture(CONF) + self.messaging_conf.transport_driver = 'fake' + self.messaging_conf.response_timeout = 15 + self.useFixture(self.messaging_conf) + rpc.init(CONF) + + mock.patch('keystoneauth1.loading.load_auth_from_conf_options').start() + + fake_notifier.stub_notifier(self) + + def tearDown(self): + """Runs after each test method to tear down test environment.""" + super(TestCase, self).tearDown() + # Reset any overridden flags + CONF.reset() + + # Stop any timers + for x in self.injected: + try: + x.stop() + except AssertionError: + pass + + # Kill any services + for x in self._services: + try: + x.kill() + except Exception: + pass + + # Delete attributes that don't start with _ so they don't pin + # memory around unnecessarily for the duration of the test + # suite + for key in [k for k in self.__dict__.keys() if k[0] != '_']: + del self.__dict__[key] + + def flags(self, **kw): + """Override flag variables for a test.""" + for k, v in kw.items(): + CONF.set_override(k, v, enforce_type=True) + + def start_service(self, name, host=None, **kwargs): + host = host and host or uuid.uuid4().hex + kwargs.setdefault('host', host) + kwargs.setdefault('binary', 'meteos-%s' % name) + svc = service.Service.create(**kwargs) + svc.start() + self._services.append(svc) + return svc + + def mock_object(self, obj, attr_name, new_attr=None, **kwargs): + """Use python mock to mock an object attribute + + Mocks the specified objects attribute with the given value. + Automatically performs 'addCleanup' for the mock. + + """ + if not new_attr: + new_attr = mock.Mock() + patcher = mock.patch.object(obj, attr_name, new_attr, **kwargs) + patcher.start() + self.addCleanup(patcher.stop) + return new_attr + + def mock_class(self, class_name, new_val=None, **kwargs): + """Use python mock to mock a class + + Mocks the specified objects attribute with the given value. + Automatically performs 'addCleanup' for the mock. + + """ + if not new_val: + new_val = mock.Mock() + patcher = mock.patch(class_name, new_val, **kwargs) + patcher.start() + self.addCleanup(patcher.stop) + return new_val + + # Useful assertions + def assertDictMatch(self, d1, d2, approx_equal=False, tolerance=0.001): + """Assert two dicts are equivalent. + + This is a 'deep' match in the sense that it handles nested + dictionaries appropriately. + + NOTE: + + If you don't care (or don't know) a given value, you can specify + the string DONTCARE as the value. This will cause that dict-item + to be skipped. + + """ + def raise_assertion(msg): + d1str = str(d1) + d2str = str(d2) + base_msg = ('Dictionaries do not match. %(msg)s d1: %(d1str)s ' + 'd2: %(d2str)s' % + {"msg": msg, "d1str": d1str, "d2str": d2str}) + raise AssertionError(base_msg) + + d1keys = set(d1.keys()) + d2keys = set(d2.keys()) + if d1keys != d2keys: + d1only = d1keys - d2keys + d2only = d2keys - d1keys + raise_assertion('Keys in d1 and not d2: %(d1only)s. ' + 'Keys in d2 and not d1: %(d2only)s' % + {"d1only": d1only, "d2only": d2only}) + + for key in d1keys: + d1value = d1[key] + d2value = d2[key] + try: + error = abs(float(d1value) - float(d2value)) + within_tolerance = error <= tolerance + except (ValueError, TypeError): + # If both values aren't convertible to float, just ignore + # ValueError if arg is a str, TypeError if it's something else + # (like None) + within_tolerance = False + + if hasattr(d1value, 'keys') and hasattr(d2value, 'keys'): + self.assertDictMatch(d1value, d2value) + elif 'DONTCARE' in (d1value, d2value): + continue + elif approx_equal and within_tolerance: + continue + elif d1value != d2value: + raise_assertion("d1['%(key)s']=%(d1value)s != " + "d2['%(key)s']=%(d2value)s" % + { + "key": key, + "d1value": d1value, + "d2value": d2value + }) + + def assertDictListMatch(self, L1, L2, approx_equal=False, tolerance=0.001): + """Assert a list of dicts are equivalent.""" + def raise_assertion(msg): + L1str = str(L1) + L2str = str(L2) + base_msg = ('List of dictionaries do not match: %(msg)s ' + 'L1: %(L1str)s L2: %(L2str)s' % + {"msg": msg, "L1str": L1str, "L2str": L2str}) + raise AssertionError(base_msg) + + L1count = len(L1) + L2count = len(L2) + if L1count != L2count: + raise_assertion('Length mismatch: len(L1)=%(L1count)d != ' + 'len(L2)=%(L2count)d' % + {"L1count": L1count, "L2count": L2count}) + + for d1, d2 in zip(L1, L2): + self.assertDictMatch(d1, d2, approx_equal=approx_equal, + tolerance=tolerance) + + def assertSubDictMatch(self, sub_dict, super_dict): + """Assert a sub_dict is subset of super_dict.""" + self.assertTrue(set(sub_dict.keys()).issubset(set(super_dict.keys()))) + for k, sub_value in sub_dict.items(): + super_value = super_dict[k] + if isinstance(sub_value, dict): + self.assertSubDictMatch(sub_value, super_value) + elif 'DONTCARE' in (sub_value, super_value): + continue + else: + self.assertEqual(sub_value, super_value) + + def assertIn(self, a, b, *args, **kwargs): + """Python < v2.7 compatibility. Assert 'a' in 'b'.""" + try: + f = super(TestCase, self).assertIn + except AttributeError: + self.assertTrue(a in b, *args, **kwargs) + else: + f(a, b, *args, **kwargs) + + def assertNotIn(self, a, b, *args, **kwargs): + """Python < v2.7 compatibility. Assert 'a' NOT in 'b'.""" + try: + f = super(TestCase, self).assertNotIn + except AttributeError: + self.assertFalse(a in b, *args, **kwargs) + else: + f(a, b, *args, **kwargs) + + def assertIsInstance(self, a, b, *args, **kwargs): + """Python < v2.7 compatibility.""" + try: + f = super(TestCase, self).assertIsInstance + except AttributeError: + self.assertIsInstance(a, b) + else: + f(a, b, *args, **kwargs) + + def assertIsNone(self, a, *args, **kwargs): + """Python < v2.7 compatibility.""" + try: + f = super(TestCase, self).assertIsNone + except AttributeError: + self.assertTrue(a is None) + else: + f(a, *args, **kwargs) + + def _dict_from_object(self, obj, ignored_keys): + if ignored_keys is None: + ignored_keys = [] + return {k: v for k, v in obj.iteritems() + if k not in ignored_keys} + + def _assertEqualListsOfObjects(self, objs1, objs2, ignored_keys=None): + obj_to_dict = lambda o: self._dict_from_object(o, ignored_keys) + sort_key = lambda d: [d[k] for k in sorted(d)] + conv_and_sort = lambda obj: sorted(map(obj_to_dict, obj), key=sort_key) + + self.assertEqual(conv_and_sort(objs1), conv_and_sort(objs2)) diff --git a/meteos/testing/README.rst b/meteos/testing/README.rst new file mode 100644 index 0000000..78cdf8d --- /dev/null +++ b/meteos/testing/README.rst @@ -0,0 +1,53 @@ +======================================= +OpenStack Meteos Testing Infrastructure +======================================= + +A note of clarification is in order, to help those who are new to testing in +OpenStack Meteos: + +- actual unit tests are created in the "tests" directory; +- the "testing" directory is used to house the infrastructure needed to support + testing in OpenStack Meteos. + +This README file attempts to provide current and prospective contributors with +everything they need to know in order to start creating unit tests and +utilizing the convenience code provided in meteos.testing. + +Writing Unit Tests +------------------ + +- All new unit tests are to be written in python-mock. +- Old tests that are still written in mox should be updated to use python-mock. + Usage of mox has been deprecated for writing Meteos unit tests. +- use addCleanup in favor of tearDown + +test.TestCase +------------- +The TestCase class from meteos.test (generally imported as test) will +automatically manage self.stubs using the stubout module. +They will automatically verify and clean up during the tearDown step. + +If using test.TestCase, calling the super class setUp is required and +calling the super class tearDown is required to be last if tearDown +is overridden. + +Running Tests +------------- + +In the root of the Meteos source code run the run_tests.sh script. This will +offer to create a virtual environment and populate it with dependencies. +If you don't have dependencies installed that are needed for compiling Meteos's +direct dependencies, you'll have to use your operating system's method of +installing extra dependencies. To get help using this script execute it with +the -h parameter to get options `./run_tests.sh -h` + +Tests and assertRaises +---------------------- +When asserting that a test should raise an exception, test against the +most specific exception possible. An overly broad exception type (like +Exception) can mask errors in the unit test itself. + +Example:: + + self.assertRaises(exception.InstanceNotFound, db.instance_get_by_uuid, + elevated, instance_uuid) diff --git a/meteos/tests/__init__.py b/meteos/tests/__init__.py new file mode 100644 index 0000000..9c1c10a --- /dev/null +++ b/meteos/tests/__init__.py @@ -0,0 +1,27 @@ +# Copyright 2010 United States Government as represented by the +# Administrator of the National Aeronautics and Space Administration. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +""" +:mod:`meteos.tests` -- Meteos Unittests +===================================================== + +.. automodule:: meteos.tests + :platform: Unix +""" + +import eventlet + +eventlet.monkey_patch() diff --git a/meteos/utils.py b/meteos/utils.py new file mode 100644 index 0000000..41deb5c --- /dev/null +++ b/meteos/utils.py @@ -0,0 +1,400 @@ +# Copyright 2010 United States Government as represented by the +# Administrator of the National Aeronautics and Space Administration. +# Copyright 2011 Justin Santa Barbara +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +"""Utilities and helper functions.""" + +import contextlib +import errno +import functools +import inspect +import os +import pyclbr +import random +import re +import shutil +import socket +import sys +import tempfile +import time + +from eventlet import pools +import netaddr +from oslo_concurrency import lockutils +from oslo_concurrency import processutils +from oslo_config import cfg +from oslo_log import log +from oslo_utils import importutils +from oslo_utils import netutils +from oslo_utils import timeutils +import paramiko +import retrying +import six + +from meteos.common import constants +from meteos.db import api as db_api +from meteos import exception +from meteos.i18n import _ + +CONF = cfg.CONF +LOG = log.getLogger(__name__) + +synchronized = lockutils.synchronized_with_prefix('meteos-') + + +def _get_root_helper(): + return 'sudo meteos-rootwrap %s' % CONF.rootwrap_config + + +def execute(*cmd, **kwargs): + """Convenience wrapper around oslo's execute() function.""" + if 'run_as_root' in kwargs and 'root_helper' not in kwargs: + kwargs['root_helper'] = _get_root_helper() + return processutils.execute(*cmd, **kwargs) + + +def trycmd(*args, **kwargs): + """Convenience wrapper around oslo's trycmd() function.""" + if 'run_as_root' in kwargs and 'root_helper' not in kwargs: + kwargs['root_helper'] = _get_root_helper() + return processutils.trycmd(*args, **kwargs) + + +class SSHPool(pools.Pool): + """A simple eventlet pool to hold ssh connections.""" + + def __init__(self, ip, port, conn_timeout, login, password=None, + privatekey=None, *args, **kwargs): + self.ip = ip + self.port = port + self.login = login + self.password = password + self.conn_timeout = conn_timeout if conn_timeout else None + self.path_to_private_key = privatekey + super(SSHPool, self).__init__(*args, **kwargs) + + def create(self): + ssh = paramiko.SSHClient() + ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy()) + look_for_keys = True + if self.path_to_private_key: + self.path_to_private_key = os.path.expanduser( + self.path_to_private_key) + look_for_keys = False + elif self.password: + look_for_keys = False + try: + ssh.connect(self.ip, + port=self.port, + username=self.login, + password=self.password, + key_filename=self.path_to_private_key, + look_for_keys=look_for_keys, + timeout=self.conn_timeout) + # Paramiko by default sets the socket timeout to 0.1 seconds, + # ignoring what we set through the sshclient. This doesn't help for + # keeping long lived connections. Hence we have to bypass it, by + # overriding it after the transport is initialized. We are setting + # the sockettimeout to None and setting a keepalive packet so that, + # the server will keep the connection open. All that does is send + # a keepalive packet every ssh_conn_timeout seconds. + if self.conn_timeout: + transport = ssh.get_transport() + transport.sock.settimeout(None) + transport.set_keepalive(self.conn_timeout) + return ssh + except Exception as e: + msg = _("Check whether private key or password are correctly " + "set. Error connecting via ssh: %s") % e + LOG.error(msg) + raise exception.SSHException(msg) + + def get(self): + """Return an item from the pool, when one is available. + + This may cause the calling greenthread to block. Check if a + connection is active before returning it. For dead connections + create and return a new connection. + """ + if self.free_items: + conn = self.free_items.popleft() + if conn: + if conn.get_transport().is_active(): + return conn + else: + conn.close() + return self.create() + if self.current_size < self.max_size: + created = self.create() + self.current_size += 1 + return created + return self.channel.get() + + def remove(self, ssh): + """Close an ssh client and remove it from free_items.""" + ssh.close() + ssh = None + if ssh in self.free_items: + self.free_items.pop(ssh) + if self.current_size > 0: + self.current_size -= 1 + + +def check_ssh_injection(cmd_list): + ssh_injection_pattern = ['`', '$', '|', '||', ';', '&', '&&', '>', '>>', + '<'] + + # Check whether injection attacks exist + for arg in cmd_list: + arg = arg.strip() + + # Check for matching quotes on the ends + is_quoted = re.match('^(?P[\'"])(?P.*)(?P=quote)$', arg) + if is_quoted: + # Check for unescaped quotes within the quoted argument + quoted = is_quoted.group('quoted') + if quoted: + if (re.match('[\'"]', quoted) or + re.search('[^\\\\][\'"]', quoted)): + raise exception.SSHInjectionThreat(command=cmd_list) + else: + # We only allow spaces within quoted arguments, and that + # is the only special character allowed within quotes + if len(arg.split()) > 1: + raise exception.SSHInjectionThreat(command=cmd_list) + + # Second, check whether danger character in command. So the shell + # special operator must be a single argument. + for c in ssh_injection_pattern: + if c not in arg: + continue + + result = arg.find(c) + if not result == -1: + if result == 0 or not arg[result - 1] == '\\': + raise exception.SSHInjectionThreat(command=cmd_list) + + +class LazyPluggable(object): + """A pluggable backend loaded lazily based on some value.""" + + def __init__(self, pivot, **backends): + self.__backends = backends + self.__pivot = pivot + self.__backend = None + + def __get_backend(self): + if not self.__backend: + backend_name = CONF[self.__pivot] + if backend_name not in self.__backends: + raise exception.Error(_('Invalid backend: %s') % backend_name) + + backend = self.__backends[backend_name] + if isinstance(backend, tuple): + name = backend[0] + fromlist = backend[1] + else: + name = backend + fromlist = backend + + self.__backend = __import__(name, None, None, fromlist) + LOG.debug('backend %s', self.__backend) + return self.__backend + + def __getattr__(self, key): + backend = self.__get_backend() + return getattr(backend, key) + + +def monkey_patch(): + """Patch decorator. + + If the Flags.monkey_patch set as True, + this function patches a decorator + for all functions in specified modules. + You can set decorators for each modules + using CONF.monkey_patch_modules. + The format is "Module path:Decorator function". + Example: 'meteos.api.ec2.cloud:' \ + meteos.openstack.common.notifier.api.notify_decorator' + + Parameters of the decorator is as follows. + (See meteos.openstack.common.notifier.api.notify_decorator) + + name - name of the function + function - object of the function + """ + # If CONF.monkey_patch is not True, this function do nothing. + if not CONF.monkey_patch: + return + # Get list of modules and decorators + for module_and_decorator in CONF.monkey_patch_modules: + module, decorator_name = module_and_decorator.split(':') + # import decorator function + decorator = importutils.import_class(decorator_name) + __import__(module) + # Retrieve module information using pyclbr + module_data = pyclbr.readmodule_ex(module) + for key in module_data.keys(): + # set the decorator for the class methods + if isinstance(module_data[key], pyclbr.Class): + clz = importutils.import_class("%s.%s" % (module, key)) + # NOTE(vponomaryov): we need to distinguish class methods types + # for py2 and py3, because the concept of 'unbound methods' has + # been removed from the python3.x + if six.PY3: + member_type = inspect.isfunction + else: + member_type = inspect.ismethod + for method, func in inspect.getmembers(clz, member_type): + setattr( + clz, method, + decorator("%s.%s.%s" % (module, key, method), func)) + # set the decorator for the function + if isinstance(module_data[key], pyclbr.Function): + func = importutils.import_class("%s.%s" % (module, key)) + setattr(sys.modules[module], key, + decorator("%s.%s" % (module, key), func)) + + +def file_open(filename): + """Open file + + see built-in file() documentation for more details + + Note: The reason this is kept in a separate module is to easily + be able to provide a stub module that doesn't alter system + state at all (for unit tests) + """ + + try: + fd = open(filename) + except IOError as e: + if e.errno != errno.ENOENT: + raise + result = False + else: + data = fd.read() + fd.close() + + return data + + +def service_is_up(service): + """Check whether a service is up based on last heartbeat.""" + last_heartbeat = service['updated_at'] or service['created_at'] + # Timestamps in DB are UTC. + tdelta = timeutils.utcnow() - last_heartbeat + elapsed = tdelta.total_seconds() + return abs(elapsed) <= CONF.service_down_time + + +def validate_service_host(context, host): + service = db_api.service_get_by_host_and_topic(context, host, + 'meteos-engine') + if not service_is_up(service): + raise exception.ServiceIsDown(service=service['host']) + + return service + + +def walk_class_hierarchy(clazz, encountered=None): + """Walk class hierarchy, yielding most derived classes first.""" + if not encountered: + encountered = [] + for subclass in clazz.__subclasses__(): + if subclass not in encountered: + encountered.append(subclass) + # drill down to leaves first + for subsubclass in walk_class_hierarchy(subclass, encountered): + yield subsubclass + yield subclass + + +class IsAMatcher(object): + def __init__(self, expected_value=None): + self.expected_value = expected_value + + def __eq__(self, actual_value): + return isinstance(actual_value, self.expected_value) + + +class ComparableMixin(object): + def _compare(self, other, method): + try: + return method(self._cmpkey(), other._cmpkey()) + except (AttributeError, TypeError): + # _cmpkey not implemented, or return different type, + # so I can't compare with "other". + return NotImplemented + + def __lt__(self, other): + return self._compare(other, lambda s, o: s < o) + + def __le__(self, other): + return self._compare(other, lambda s, o: s <= o) + + def __eq__(self, other): + return self._compare(other, lambda s, o: s == o) + + def __ge__(self, other): + return self._compare(other, lambda s, o: s >= o) + + def __gt__(self, other): + return self._compare(other, lambda s, o: s > o) + + def __ne__(self, other): + return self._compare(other, lambda s, o: s != o) + + +def require_driver_initialized(func): + @functools.wraps(func) + def wrapper(self, *args, **kwargs): + # we can't do anything if the driver didn't init + if not self.driver.initialized: + driver_name = self.driver.__class__.__name__ + raise exception.DriverNotInitialized(driver=driver_name) + return func(self, *args, **kwargs) + return wrapper + + +def wait_for_access_update(context, db, learning_instance, + migration_wait_access_rules_timeout): + starttime = time.time() + deadline = starttime + migration_wait_access_rules_timeout + tries = 0 + + while True: + instance = db.learning_instance_get(context, learning_instance['id']) + + if instance['access_rules_status'] == constants.STATUS_ACTIVE: + break + + tries += 1 + now = time.time() + if instance['access_rules_status'] == constants.STATUS_ERROR: + msg = _("Failed to update access rules" + " on learning instance %s") % learning_instance['id'] + raise exception.LearningMigrationFailed(reason=msg) + elif now > deadline: + msg = _("Timeout trying to update access rules" + " on learning instance %(learning_id)s. Timeout " + "was %(timeout)s seconds.") % { + 'learning_id': learning_instance['id'], + 'timeout': migration_wait_access_rules_timeout} + raise exception.LearningMigrationFailed(reason=msg) + else: + time.sleep(tries ** 2) diff --git a/meteos/version.py b/meteos/version.py new file mode 100644 index 0000000..a46bf04 --- /dev/null +++ b/meteos/version.py @@ -0,0 +1,23 @@ +# Copyright 2011 OpenStack LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +from pbr import version as pbr_version + +METEOS_VENDOR = "OpenStack Foundation" +METEOS_PRODUCT = "OpenStack Meteos" +METEOS_PACKAGE = None # OS distro package version suffix + +loaded = False +version_info = pbr_version.VersionInfo('meteos') +version_string = version_info.version_string diff --git a/meteos/wsgi.py b/meteos/wsgi.py new file mode 100644 index 0000000..c7c9bc5 --- /dev/null +++ b/meteos/wsgi.py @@ -0,0 +1,551 @@ +# Copyright 2010 United States Government as represented by the +# Administrator of the National Aeronautics and Space Administration. +# Copyright 2010 OpenStack LLC. +# All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); you may +# not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT +# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the +# License for the specific language governing permissions and limitations +# under the License. + +"""Utility methods for working with WSGI servers.""" + +from __future__ import print_function + +import errno +import os +import socket +import ssl +import sys +import time + +import eventlet +import eventlet.wsgi +import greenlet +from oslo_config import cfg +from oslo_log import log +from oslo_service import service +from oslo_utils import excutils +from oslo_utils import netutils +from paste import deploy +import routes.middleware +import webob.dec +import webob.exc + +from meteos.common import config +from meteos import exception +from meteos.i18n import _, _LE, _LI + +socket_opts = [ + cfg.IntOpt('backlog', + default=4096, + help="Number of backlog requests to configure the socket " + "with."), + cfg.BoolOpt('tcp_keepalive', + default=True, + help="Sets the value of TCP_KEEPALIVE (True/False) for each " + "server socket."), + cfg.IntOpt('tcp_keepidle', + default=600, + help="Sets the value of TCP_KEEPIDLE in seconds for each " + "server socket. Not supported on OS X."), + cfg.IntOpt('tcp_keepalive_interval', + help="Sets the value of TCP_KEEPINTVL in seconds for each " + "server socket. Not supported on OS X."), + cfg.IntOpt('tcp_keepalive_count', + help="Sets the value of TCP_KEEPCNT for each " + "server socket. Not supported on OS X."), + cfg.StrOpt('ssl_ca_file', + help="CA certificate file to use to verify " + "connecting clients."), + cfg.StrOpt('ssl_cert_file', + help="Certificate file to use when starting " + "the server securely."), + cfg.StrOpt('ssl_key_file', + help="Private key file to use when starting " + "the server securely."), +] + +eventlet_opts = [ + cfg.IntOpt('max_header_line', + default=16384, + help="Maximum line size of message headers to be accepted. " + "Option max_header_line may need to be increased when " + "using large tokens (typically those generated by the " + "Keystone v3 API with big service catalogs)."), + cfg.IntOpt('client_socket_timeout', + default=900, + help="Timeout for client connections socket operations. " + "If an incoming connection is idle for this number of " + "seconds it will be closed. A value of '0' means " + "wait forever."), + cfg.BoolOpt('wsgi_keep_alive', + default=True, + help='If False, closes the client socket connection ' + 'explicitly. Setting it to True to maintain backward ' + 'compatibility. Recommended setting is set it to False.'), +] + +CONF = cfg.CONF +CONF.register_opts(socket_opts) +CONF.register_opts(eventlet_opts) + +LOG = log.getLogger(__name__) + + +class Server(service.ServiceBase): + """Server class to manage a WSGI server, serving a WSGI application.""" + + default_pool_size = 1000 + + def __init__(self, name, app, host=None, port=None, pool_size=None, + protocol=eventlet.wsgi.HttpProtocol, backlog=128): + """Initialize, but do not start, a WSGI server. + + :param name: Pretty name for logging. + :param app: The WSGI application to serve. + :param host: IP address to serve the application. + :param port: Port number to server the application. + :param pool_size: Maximum number of eventlets to spawn concurrently. + :returns: None + + """ + eventlet.wsgi.MAX_HEADER_LINE = CONF.max_header_line + self.client_socket_timeout = CONF.client_socket_timeout + self.name = name + self.app = app + self._host = host or "0.0.0.0" + self._port = port or 0 + self._server = None + self._socket = None + self._protocol = protocol + self.pool_size = pool_size or self.default_pool_size + self._pool = eventlet.GreenPool(self.pool_size) + self._logger = log.getLogger("eventlet.wsgi.server") + + if backlog < 1: + raise exception.InvalidInput( + reason='The backlog must be more than 1') + + bind_addr = (host, port) + # TODO(dims): eventlet's green dns/socket module does not actually + # support IPv6 in getaddrinfo(). We need to get around this in the + # future or monitor upstream for a fix + try: + info = socket.getaddrinfo(bind_addr[0], + bind_addr[1], + socket.AF_UNSPEC, + socket.SOCK_STREAM)[0] + family = info[0] + bind_addr = info[-1] + except Exception: + family = socket.AF_INET + + cert_file = CONF.ssl_cert_file + key_file = CONF.ssl_key_file + ca_file = CONF.ssl_ca_file + self._use_ssl = cert_file or key_file + + if cert_file and not os.path.exists(cert_file): + raise RuntimeError(_("Unable to find cert_file : %s") % cert_file) + + if ca_file and not os.path.exists(ca_file): + raise RuntimeError(_("Unable to find ca_file : %s") % ca_file) + + if key_file and not os.path.exists(key_file): + raise RuntimeError(_("Unable to find key_file : %s") % key_file) + + if self._use_ssl and (not cert_file or not key_file): + raise RuntimeError(_("When running server in SSL mode, you must " + "specify both a cert_file and key_file " + "option value in your configuration file")) + + retry_until = time.time() + 30 + while not self._socket and time.time() < retry_until: + try: + self._socket = eventlet.listen( + bind_addr, backlog=backlog, family=family) + except socket.error as err: + if err.args[0] != errno.EADDRINUSE: + raise + eventlet.sleep(0.1) + + if not self._socket: + raise RuntimeError(_("Could not bind to %(host)s:%(port)s " + "after trying for 30 seconds") % + {'host': host, 'port': port}) + + (self._host, self._port) = self._socket.getsockname()[0:2] + LOG.info(_LI("%(name)s listening on %(_host)s:%(_port)s"), + {'name': self.name, '_host': self._host, '_port': self._port}) + + def start(self): + """Start serving a WSGI application. + + :returns: None + :raises: meteos.exception.InvalidInput + + """ + # The server socket object will be closed after server exits, + # but the underlying file descriptor will remain open, and will + # give bad file descriptor error. So duplicating the socket object, + # to keep file descriptor usable. + + config.set_middleware_defaults() + dup_socket = self._socket.dup() + + netutils.set_tcp_keepalive( + dup_socket, + tcp_keepalive=CONF.tcp_keepalive, + tcp_keepidle=CONF.tcp_keepidle, + tcp_keepalive_interval=CONF.tcp_keepalive_interval, + tcp_keepalive_count=CONF.tcp_keepalive_count + ) + + if self._use_ssl: + try: + ssl_kwargs = { + 'server_side': True, + 'certfile': CONF.ssl_cert_file, + 'keyfile': CONF.ssl_key_file, + 'cert_reqs': ssl.CERT_NONE, + } + + if CONF.ssl_ca_file: + ssl_kwargs['ca_certs'] = CONF.ssl_ca_file + ssl_kwargs['cert_reqs'] = ssl.CERT_REQUIRED + + dup_socket = ssl.wrap_socket(dup_socket, + **ssl_kwargs) + + dup_socket.setsockopt(socket.SOL_SOCKET, + socket.SO_REUSEADDR, 1) + + except Exception: + with excutils.save_and_reraise_exception(): + LOG.error( + _LE("Failed to start %(name)s on %(_host)s:%(_port)s " + "with SSL support."), + {"name": self.name, "_host": self._host, + "_port": self._port} + ) + + wsgi_kwargs = { + 'func': eventlet.wsgi.server, + 'sock': dup_socket, + 'site': self.app, + 'protocol': self._protocol, + 'custom_pool': self._pool, + 'log': self._logger, + 'socket_timeout': self.client_socket_timeout, + 'keepalive': CONF.wsgi_keep_alive, + } + + self._server = eventlet.spawn(**wsgi_kwargs) + + @property + def host(self): + return self._host + + @property + def port(self): + return self._port + + def stop(self): + """Stop this server. + + This is not a very nice action, as currently the method by which a + server is stopped is by killing its eventlet. + + :returns: None + + """ + LOG.info(_LI("Stopping WSGI server.")) + if self._server is not None: + # Resize pool to stop new requests from being processed + self._pool.resize(0) + self._server.kill() + + def wait(self): + """Block, until the server has stopped. + + Waits on the server's eventlet to finish, then returns. + + :returns: None + + """ + try: + if self._server is not None: + self._pool.waitall() + self._server.wait() + except greenlet.GreenletExit: + LOG.info(_LI("WSGI server has stopped.")) + + def reset(self): + """Reset server greenpool size to default. + + :returns: None + """ + self._pool.resize(self.pool_size) + + +class Request(webob.Request): + pass + + +class Application(object): + """Base WSGI application wrapper. Subclasses need to implement __call__.""" + + @classmethod + def factory(cls, global_config, **local_config): + """Used for paste app factories in paste.deploy config files. + + Any local configuration (that is, values under the [app:APPNAME] + section of the paste config) will be passed into the `__init__` method + as kwargs. + + A hypothetical configuration would look like: + + [app:wadl] + latest_version = 1.3 + paste.app_factory = meteos.api.fancy_api:Wadl.factory + + which would result in a call to the `Wadl` class as + + import meteos.api.fancy_api + fancy_api.Wadl(latest_version='1.3') + + You could of course re-implement the `factory` method in subclasses, + but using the kwarg passing it shouldn't be necessary. + + """ + return cls(**local_config) + + def __call__(self, environ, start_response): + r"""Subclasses will probably want to implement __call__ like this: + + @webob.dec.wsgify(RequestClass=Request) + def __call__(self, req): + # Any of the following objects work as responses: + + # Option 1: simple string + res = 'message\n' + + # Option 2: a nicely formatted HTTP exception page + res = exc.HTTPForbidden(detail='Nice try') + + # Option 3: a webob Response object (in case you need to play with + # headers, or you want to be treated like an iterable, or or or) + res = Response(); + res.app_iter = open('somefile') + + # Option 4: any wsgi app to be run next + res = self.application + + # Option 5: you can get a Response object for a wsgi app, too, to + # play with headers etc + res = req.get_response(self.application) + + # You can then just return your response... + return res + # ... or set req.response and return None. + req.response = res + + See the end of http://pythonpaste.org/webob/modules/dec.html + for more info. + + """ + raise NotImplementedError(_('You must implement __call__')) + + +class Middleware(Application): + """Base WSGI middleware. + + These classes require an application to be + initialized that will be called next. By default the middleware will + simply call its wrapped app, or you can override __call__ to customize its + behavior. + + """ + + @classmethod + def factory(cls, global_config, **local_config): + """Used for paste app factories in paste.deploy config files. + + Any local configuration (that is, values under the [filter:APPNAME] + section of the paste config) will be passed into the `__init__` method + as kwargs. + + A hypothetical configuration would look like: + + [filter:analytics] + redis_host = 127.0.0.1 + paste.filter_factory = meteos.api.analytics:Analytics.factory + + which would result in a call to the `Analytics` class as + + import meteos.api.analytics + analytics.Analytics(app_from_paste, redis_host='127.0.0.1') + + You could of course re-implement the `factory` method in subclasses, + but using the kwarg passing it shouldn't be necessary. + + """ + def _factory(app): + return cls(app, **local_config) + return _factory + + def __init__(self, application): + self.application = application + + def process_request(self, req): + """Called on each request. + + If this returns None, the next application down the stack will be + executed. If it returns a response then that response will be returned + and execution will stop here. + + """ + return None + + def process_response(self, response): + """Do whatever you'd like to the response.""" + return response + + @webob.dec.wsgify(RequestClass=Request) + def __call__(self, req): + response = self.process_request(req) + if response: + return response + response = req.get_response(self.application) + return self.process_response(response) + + +class Debug(Middleware): + """Helper class for debugging a WSGI application. + + Can be inserted into any WSGI application chain to get information + about the request and response. + + """ + + @webob.dec.wsgify(RequestClass=Request) + def __call__(self, req): + print(('*' * 40) + ' REQUEST ENVIRON') + for key, value in req.environ.items(): + print(key, '=', value) + print() + resp = req.get_response(self.application) + + print(('*' * 40) + ' RESPONSE HEADERS') + for (key, value) in resp.headers.items(): + print(key, '=', value) + print() + + resp.app_iter = self.print_generator(resp.app_iter) + + return resp + + @staticmethod + def print_generator(app_iter): + """Iterator that prints the contents of a wrapper string.""" + print(('*' * 40) + ' BODY') + for part in app_iter: + sys.stdout.write(part.decode()) + sys.stdout.flush() + yield part + print() + + +class Router(object): + """WSGI middleware that maps incoming requests to WSGI apps.""" + + def __init__(self, mapper): + """Create a router for the given routes.Mapper. + + Each route in `mapper` must specify a 'controller', which is a + WSGI app to call. You'll probably want to specify an 'action' as + well and have your controller be an object that can route + the request to the action-specific method. + + Examples: + mapper = routes.Mapper() + sc = ServerController() + + # Explicit mapping of one route to a controller+action + mapper.connect(None, '/svrlist', controller=sc, action='list') + + # Actions are all implicitly defined + mapper.resource('server', 'servers', controller=sc) + + # Pointing to an arbitrary WSGI app. You can specify the + # {path_info:.*} parameter so the target app can be handed just that + # section of the URL. + mapper.connect(None, '/v1.0/{path_info:.*}', controller=BlogApp()) + + """ + self.map = mapper + self._router = routes.middleware.RoutesMiddleware(self._dispatch, + self.map) + + @webob.dec.wsgify(RequestClass=Request) + def __call__(self, req): + """Route the incoming request to a controller based on self.map. + + If no match, return a 404. + + """ + return self._router + + @staticmethod + @webob.dec.wsgify(RequestClass=Request) + def _dispatch(req): + """Dispatch the request to the appropriate controller. + + Called by self._router after matching the incoming request to a route + and putting the information into req.environ. Either returns 404 + or the routed WSGI app's response. + + """ + match = req.environ['wsgiorg.routing_args'][1] + if not match: + return webob.exc.HTTPNotFound() + app = match['controller'] + return app + + +class Loader(object): + """Used to load WSGI applications from paste configurations.""" + + def __init__(self, config_path=None): + """Initialize the loader, and attempt to find the config. + + :param config_path: Full or relative path to the paste config. + :returns: None + + """ + config_path = config_path or CONF.api_paste_config + self.config_path = CONF.find_file(config_path) + if not self.config_path: + raise exception.ConfigNotFound(path=config_path) + + def load_app(self, name): + """Return the paste URLMap wrapped WSGI application. + + :param name: Name of the application to load. + :returns: Paste URLMap object wrapping the requested application. + :raises: `meteos.exception.PasteAppNotFound` + + """ + try: + return deploy.loadapp("config:%s" % self.config_path, name=name) + except LookupError as err: + LOG.error(err) + raise exception.PasteAppNotFound(name=name, path=self.config_path) diff --git a/pylintrc b/pylintrc new file mode 100644 index 0000000..f01333c --- /dev/null +++ b/pylintrc @@ -0,0 +1,38 @@ +# The format of this file isn't really documented; just use --generate-rcfile + +[Messages Control] +# NOTE(justinsb): We might want to have a 2nd strict pylintrc in future +# C0111: Don't require docstrings on every method +# W0511: TODOs in code comments are fine. +# W0142: *args and **kwargs are fine. +# W0622: Redefining id is fine. +disable=C0111,W0511,W0142,W0622 + +[Basic] +# Variable names can be 1 to 31 characters long, with lowercase and underscores +variable-rgx=[a-z_][a-z0-9_]{0,30}$ + +# Argument names can be 2 to 31 characters long, with lowercase and underscores +argument-rgx=[a-z_][a-z0-9_]{1,30}$ + +# Method names should be at least 3 characters long +# and be lowecased with underscores +method-rgx=([a-z_][a-z0-9_]{2,50}|setUp|tearDown)$ + +# Module names matching manila-* are ok (files in bin/) +module-rgx=(([a-z_][a-z0-9_]*)|([A-Z][a-zA-Z0-9]+)|(manila-[a-z0-9_-]+))$ + +# Don't require docstrings on tests. +no-docstring-rgx=((__.*__)|([tT]est.*)|setUp|tearDown)$ + +[Design] +max-public-methods=100 +min-public-methods=0 +max-args=6 + +[Variables] + +# List of additional names supposed to be defined in builtins. Remember that +# you should avoid to define new builtins when possible. +# _ is used by our localization +additional-builtins=_ diff --git a/releasenotes/notes/.placeholder b/releasenotes/notes/.placeholder new file mode 100644 index 0000000..e69de29 diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..d106c67 --- /dev/null +++ b/requirements.txt @@ -0,0 +1,45 @@ +# The order of packages is significant, because pip processes them in the order +# of appearance. Changing the order has an impact on the overall integration +# process, which may cause wedges in the gate later. + +# pbr should be first +pbr>=1.6 # Apache-2.0 + +alembic>=0.8.4 # MIT +Babel>=2.3.4 # BSD +eventlet!=0.18.3,>=0.18.2 # MIT +greenlet>=0.3.2 # MIT +lxml>=2.3 # BSD +netaddr!=0.7.16,>=0.7.13 # BSD +oslo.config>=3.14.0 # Apache-2.0 +oslo.context>=2.9.0 # Apache-2.0 +oslo.db!=4.13.1,!=4.13.2,>=4.10.0 # Apache-2.0 +oslo.i18n>=2.1.0 # Apache-2.0 +oslo.log>=3.11.0 # Apache-2.0 +oslo.messaging>=5.2.0 # Apache-2.0 +oslo.middleware>=3.0.0 # Apache-2.0 +oslo.policy>=1.9.0 # Apache-2.0 +oslo.rootwrap>=5.0.0 # Apache-2.0 +oslo.serialization>=1.10.0 # Apache-2.0 +oslo.service>=1.10.0 # Apache-2.0 +oslo.utils>=3.16.0 # Apache-2.0 +oslo.concurrency>=3.8.0 # Apache-2.0 +osprofiler>=1.4.0 # Apache-2.0 +paramiko>=2.0 # LGPLv2.1+ +Paste # MIT +PasteDeploy>=1.5.0 # MIT +pyparsing>=2.0.1 # MIT +python-neutronclient>=5.1.0 # Apache-2.0 +keystoneauth1>=2.10.0 # Apache-2.0 +keystonemiddleware!=4.5.0,>=4.2.0 # Apache-2.0 +requests>=2.10.0 # Apache-2.0 +retrying!=1.3.0,>=1.2.3 # Apache-2.0 +Routes!=2.0,!=2.1,!=2.3.0,>=1.12.3;python_version=='2.7' # MIT +Routes!=2.0,!=2.3.0,>=1.12.3;python_version!='2.7' # MIT +six>=1.9.0 # MIT +SQLAlchemy<1.1.0,>=1.0.10 # MIT +stevedore>=1.16.0 # Apache-2.0 +python-cinderclient!=1.7.0,!=1.7.1,>=1.6.0 # Apache-2.0 +python-novaclient!=2.33.0,>=2.29.0 # Apache-2.0 +python-saharaclient>=0.14.0 # Apache-2.0 +WebOb>=1.2.3 # MIT diff --git a/setup.cfg b/setup.cfg new file mode 100644 index 0000000..50104c5 --- /dev/null +++ b/setup.cfg @@ -0,0 +1,67 @@ +[metadata] +name = meteos +summary = Machine Learning for OpenStack +description-file = + README.rst +author = OpenStack +author-email = openstack-dev@lists.openstack.org +home-page = http://www.openstack.org/ +classifier = + Environment :: OpenStack + Intended Audience :: Information Technology + Intended Audience :: System Administrators + License :: OSI Approved :: Apache Software License + Operating System :: POSIX :: Linux + Programming Language :: Python + Programming Language :: Python :: 2 + Programming Language :: Python :: 2.7 + +[global] +setup-hooks = + pbr.hooks.setup_hook + +[files] +packages = + meteos +[entry_points] +console_scripts = + meteos-api = meteos.cmd.api:main + meteos-manage = meteos.cmd.manage:main + meteos-rootwrap = oslo_rootwrap.cmd:main + meteos-engine = meteos.cmd.engine:main +# These are for backwards compat with Havana notification_driver configuration values +oslo_messaging.notify.drivers = +oslo.config.opts = + meteos = meteos.opts:list_opts +oslo.config.opts.defaults = + meteos = meteos.common.config:set_middleware_defaults + +[build_sphinx] +all_files = 1 +build-dir = doc/build +source-dir = doc/source + +[egg_info] +tag_build = +tag_date = 0 +tag_svn_revision = 0 + +[compile_catalog] +directory = meteos/locale +domain = meteos + +[update_catalog] +domain = meteos +output_dir = meteos/locale +input_file = meteos/locale/meteos.pot + +[extract_messages] +keywords = _ gettext ngettext l_ lazy_gettext +mapping_file = babel.cfg +output_file = meteos/locale/meteos.pot + +[wheel] +universal = 1 + +[pbr] +warnerrors = true diff --git a/setup.py b/setup.py new file mode 100644 index 0000000..782bb21 --- /dev/null +++ b/setup.py @@ -0,0 +1,29 @@ +# Copyright (c) 2013 Hewlett-Packard Development Company, L.P. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or +# implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# THIS FILE IS MANAGED BY THE GLOBAL REQUIREMENTS REPO - DO NOT EDIT +import setuptools + +# In python < 2.7.4, a lazy loading of package `pbr` will break +# setuptools if some other modules registered functions in `atexit`. +# solution from: http://bugs.python.org/issue15881#msg170215 +try: + import multiprocessing # noqa +except ImportError: + pass + +setuptools.setup( + setup_requires=['pbr>=1.8'], + pbr=True) diff --git a/test-requirements.txt b/test-requirements.txt new file mode 100644 index 0000000..bc17d0d --- /dev/null +++ b/test-requirements.txt @@ -0,0 +1,32 @@ +# The order of packages is significant, because pip processes them in the order +# of appearance. Changing the order has an impact on the overall integration +# process, which may cause wedges in the gate later. + +# hacking should be first +hacking<0.11,>=0.10.2 + +bashate>=0.2 # Apache-2.0 +coverage>=3.6 # Apache-2.0 +ddt>=1.0.1 # MIT +fixtures>=3.0.0 # Apache-2.0/BSD +mock>=2.0 # BSD +iso8601>=0.1.11 # MIT +oslotest>=1.10.0 # Apache-2.0 +oslosphinx!=3.4.0,>=2.5.0 # Apache-2.0 +openstackdocstheme>=1.5.0 # Apache-2.0 + +# Do not remove 'PyMySQL' and 'psycopg2' dependencies. They are used +# by oslo_db lib for running MySQL and PostgreSQL DB migration tests. +# See http://docs.openstack.org/developer/oslo.db/contributing.html#how-to-run-unit-tests +PyMySQL!=0.7.7,>=0.6.2 # MIT License +psycopg2>=2.5 # LGPL/ZPL + +python-subunit>=0.0.18 # Apache-2.0/BSD +requests-mock>=1.0 # Apache-2.0 +sphinx!=1.3b1,<1.3,>=1.2.1 # BSD +os-api-ref>=1.0.0 # Apache-2.0 +os-testr>=0.7.0 # Apache-2.0 +testrepository>=0.0.18 # Apache-2.0/BSD +testresources>=0.2.4 # Apache-2.0/BSD +testtools>=1.4.0 # MIT +reno>=1.8.0 # Apache2 diff --git a/tox.ini b/tox.ini new file mode 100644 index 0000000..fbd7da0 --- /dev/null +++ b/tox.ini @@ -0,0 +1,106 @@ +[tox] +minversion = 2.0 +skipsdist = True +envlist = py34,py27,pep8 + +[testenv] +setenv = VIRTUAL_ENV={envdir} +usedevelop = True +install_command = pip install -c{env:UPPER_CONSTRAINTS_FILE:https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt} {opts} {packages} +whitelist_externals = find +deps = -r{toxinidir}/requirements.txt + -r{toxinidir}/test-requirements.txt +commands = + find . -type f -name "*.py[c|o]" -delete + ostestr {posargs} + +[testenv:releasenotes] +commands = sphinx-build -a -E -W -d releasenotes/build/doctrees \ + -b html releasenotes/source releasenotes/build/html + +[testenv:install-guide] +# builds to install-guide/build/html +commands = sphinx-build -a -E -W -d install-guide/build/doctrees -b html install-guide/source install-guide/build/html + +[testenv:debug] +commands = oslo_debug_helper {posargs} + +[testenv:debug-py27] +basepython = python2.7 +commands = oslo_debug_helper {posargs} + +[testenv:debug-py34] +basepython = python3.4 +commands = oslo_debug_helper {posargs} + +[testenv:pep8] +commands = + flake8 {posargs} + # Run bashate during pep8 runs to ensure violations are caught by + # the check and gate queues. + bashate -i E006,E042,E043 \ + tools/enable-pre-commit-hook.sh \ + contrib/ci/pre_test_hook.sh \ + contrib/ci/post_test_hook.sh \ + devstack/plugin.sh \ + devstack/upgrade/from-mitaka/upgrade-meteos \ + devstack/upgrade/resources.sh \ + devstack/upgrade/shutdown.sh \ + devstack/upgrade/upgrade.sh \ + tools/cover.sh \ + tools/check_logging.sh \ + run_tests.sh + {toxinidir}/tools/check_exec.py {toxinidir}/meteos + {toxinidir}/tools/check_logging.sh {toxinidir}/meteos + +[testenv:genconfig] +whitelist_externals = bash +commands = + oslo-config-generator --config-file etc/oslo-config-generator/meteos.conf + +[testenv:venv] +commands = {posargs} + +[testenv:docs] +commands = python setup.py build_sphinx + +[testenv:cover] +commands = {toxinidir}/tools/cover.sh {posargs} + +[testenv:fast8] +commands = + {toxinidir}/tools/fast8.sh + +[testenv:pylint] +deps = -r{toxinidir}/requirements.txt + pylint==0.26.0 +whitelist_externals = bash +commands = bash tools/lintstack.sh + +[testenv:lint] +deps = -r{toxinidir}/requirements.txt + pylint==0.26.0 +commands = python tools/lintstack.py check + +[testenv:api-ref] +# This environment is called from CI scripts to test and publish +# the API Ref to developer.openstack.org. +whitelist_externals = rm +commands = + rm -rf api-ref/build + sphinx-build -W -b html -d api-ref/build/doctrees api-ref/source api-ref/build/html + +[testenv:dbrevision] +deps = -r{toxinidir}/requirements.txt +commands = alembic -c meteos/db/migrations/alembic.ini revision -m ""{posargs} + +[flake8] +# Following checks are ignored on purpose: +ignore = +builtins = _ +exclude = .git,.tox,.testrepository,.venv,build,cover,dist,doc,*egg,api-ref/build,*/source/conf.py + +[hacking] +import_exceptions = + meteos.i18n +local-check-factory = meteos.hacking.checks.factory