From 8974baa6efbea872b491636796c239d6beafeabb Mon Sep 17 00:00:00 2001 From: Andrey Kurilin Date: Wed, 7 Feb 2018 15:37:32 +0200 Subject: [PATCH] Release notes for 0.11.0 Change-Id: I0d26a082dd50a59dff281ea6cbe42115d75363ad --- doc/release_notes/archive/v0.11.0.rst | 263 ++++++++++++++++++++++ doc/release_notes/latest.rst | 2 +- doc/specs/implemented/deployment_type.rst | 216 ++++++++++++++++++ doc/specs/implemented/osprofiler.rst | 162 +++++++++++++ 4 files changed, 642 insertions(+), 1 deletion(-) create mode 100644 doc/release_notes/archive/v0.11.0.rst create mode 100644 doc/specs/implemented/deployment_type.rst create mode 100644 doc/specs/implemented/osprofiler.rst diff --git a/doc/release_notes/archive/v0.11.0.rst b/doc/release_notes/archive/v0.11.0.rst new file mode 100644 index 00000000..234d87e3 --- /dev/null +++ b/doc/release_notes/archive/v0.11.0.rst @@ -0,0 +1,263 @@ +============= +Rally v0.11.0 +============= + +Overview +-------- + ++------------------+-----------------------+ +| Release date | **02/15/2018** | ++------------------+-----------------------+ + +* Stabilize Rally releases (see requirements section) +* Publishing docker images is returned! +* Environment introduction (see Deployment section) + +Details +------- + +Requirements +~~~~~~~~~~~~ + +As for long time we tried to make our releases stable and workable even after +a year from release date. For this purpose, upper limits for all our +requirements were used. Such solution helped to make releases more stable, but +it did not work well and created more pain than a profit. +The main issue was related to new releases of not-direct rally +dependencies which can be incompatible which each other. + +From Rally 0.11.0 we are stopping adding upper version limits for our +`requirements +`_ (this does +not apply to cases when we sure that some new releases of dependency broke us +for sure). + +As alternative solution, the `upper-constraints file +`_ is +selected. It is a file with a list of packages and their versions which we used +in our CI while testing. Versions from this list are suggested to use in +production. +Please note, that it also contains not direct Rally dependencies (dependencies +of rally dependencies and dependencies of dependencies of rally dependencies +as well) and packages which possibly doesn't relate to Rally at all. + +The example ou usage: + + .. console:: bash + + $ git clone https://github.com/openstack/rally && cd rally + $ pip install . --constraint upper-constraints.txt + +Logging +~~~~~~~ + +The default value of ``use_stderr`` option of Rally config is changed to +**True**. It is done to ensure that json output of rally commands will not be +messed with logging and integration with third-party tools and scripts should +become simpler. + +Docker +~~~~~~ + +Unfortunately, we lost access to `rallyforge organization at DockerHub +`_, so our docker images were not +published anywhere officially for some time. Docker Support did not help us a +lot. At least, the original repo is removed now and we can start new +organization at DockerHub from the scratch. + +The new docker images for Rally+OpenStack plugins with an updated HowTo you +can find at `xrally/xrally-openstack repo +`_. It contains all Rally +releases + latest tag which maps to master branch. + + +Command Line Interface +~~~~~~~~~~~~~~~~~~~~~~ + +* Introduce `rally db ensure + `_ + command. It is going to create Rally DB if it doesn't exist, otherwise it + does nothing. + +* Various improvements for `rally db + `_. + Such as printing results of operations as well as connection string to DB, + hiding credentials by default, etc. + +* Various changes in returning error codes. Error codes become different for + different exceptions. Also, `rally task start + `_ + began to return 1 in case of any runtime issues and 2 if any workload doesn't + pass it's SLA. + +* The new CLI for new component (see ``Environments & Deployments`` section + for more details) - `rally env + `_ + +Environments & Deployments +~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Deployment Component is fundamental part of Rally which is used by Task and +Verification components. Whereas Task and Verification components experienced +redesigns (some parts were redesigned even several times), Deployment component +was only extended over the time. +Currently, we are at the point when further extending requires much work and +user experience become worth. It is a hard decision, as like we had done with +Verification component in Rally 0.8, the Deployment component is re-written +from the scratch. To be clear, the new version of the component has not only +the new code, but the new name as well - it is the Environment. + +We are at the stage when Rally is suitable not only for OpenStack but for +various platforms and systems. The Environment is some entity against which +Rally should perform load. The specific Environment can include credentials for +OpenStack or for Kubernetes or for both. The Environment is a way to describe +the complex system/cloud which all of us have. As well it can be used for +simple systems as easy as for complex. + +If you are regular Rally user which tests OpenStack clusters, nothing is +changing for you at this moment. `rally deployment +`_ +CLI still works and it is not even deprecated. It will be kept for a while. +But you can start looking at our new CLI `rally env +`_ . + +As for writing plugins for external platforms - we are working on updating our +documentary. + +Task component +~~~~~~~~~~~~~~ + +* The json results are extended with context execution stats (i.e when and + which context was executed, how much time setup and cleanup methods took, + etc). Also, `volumes@openstack + `_ + and + `volume_types@openstack + `_ + are ported to include detailed stats of executed actions. In further + releases, all existing contexts will be adopted to the similar behavior. + +* Better OSProfiler integration + + Rally environment&deployment config began accept + ``profiler_conn_str`` property which is used to generate OSProfiler native + html-report and to embed it to Rally's html-report. + +* HTML report is extended to include a timestamp of failures. + +Plugins +~~~~~~~ + +As it mentioned above, the Deployment Component will be replaced soon. Such +decision led to abandoning all deployment-related plugins (engines, providers, +etc). + + +**Scenarios**: + +* *NEW!!* + + `NovaServers.boot_server_attach_volume_and_list_attachments + `_ + + +* *UPDATED!!* + + - Extend several Nova&Neutron related scenarios with + ``create_floating_ip_args`` parameter + + `NovaServers.boot_and_associate_floating_ip + `_ + `NovaServers.boot_server_associate_and_dissociate_floating_ip + `_ + + - Modify Bgpvpn scenarios to test true bgpvpn + + All Bgpvpn scenarios began to boot a server to add active port in the + network associated to the bgpvpn which allows to test not only the record in + the database, but true bgpvpn + + +**Contexts**: + +*UPDATED!!* + +`network@openstack +`_ +context is extended with ability to specify external router information. + +Fixed bugs +~~~~~~~~~~ + +* [backported into 0.10.1][deployment] Suppress deprecation warning about an + old format in case of using `--fromenv option of rally deployment create + `_ + +* [backported into 0.10.1][deployment] Failure `rally deployment show + `_ + while displaying the information about deployment with a config in an old + format. + +* [backported into 0.10.1][task] New json report processed the hook results in + a wrong way + + `Launchpad bug-report #1734336 + `_ + +* [backported into 0.10.1][task] Failure while generating trends reports in + case of failures in setup method of any context + + `Launchpad bug-report #1732193 + `_ + +* [backported into 0.10.1][task] Failure to export results in ElasticSearch 5.x + cluster in case of extra ``/`` in the end of destination url. + +* [deployment] OpenStack deployment creation with `--fromenv + `_ + option used old deprecated format. + +* [verify] Rally did not support creating verifiers from Gerrit/SSH source. + + `Launchpad bug-report #1737529 + `_ + +* [task][openstack] Removing default security group in users@openstack context + did not take into account that neutron can return multiple resources in some + configuration instead of one security group which relates to requested + tenant. + +* [task][openstack] Existing openstack users get their roles un-assigned if + they are equal to what roles@openstack context is configured to assign. + + `Launchpad bug-report #1720270 + `_ + +* [task][openstack] Validation step ignores roles@openstack context and + marks as "invalid" valid cases + + Some actions in openstack can be performed only if the user has specific + role. Since these roles can be different in different OpenStack installations + Rally has `roles@openstack context + `_ + context which can assign roles to the users. + Validation step did not check for specified roles in workload config and made + wrong assumption about accessibility of resources + + `Launchpad bug-report #1539878 + `_ + +* [task][openstack] Wrong identifiers were used for filtering Mistral resources + while cleanup step. + +* [task][openstack] `NovaServers.boot_and_live_migrate_server + `_ + does wrong target host selection + + `Launchpad bug-report #1734914 + `_ + +Thanks +~~~~~~ + + 2 Everybody! diff --git a/doc/release_notes/latest.rst b/doc/release_notes/latest.rst index bd84f787..2a846ea9 120000 --- a/doc/release_notes/latest.rst +++ b/doc/release_notes/latest.rst @@ -1 +1 @@ -./archive/v0.10.1.rst \ No newline at end of file +archive/v0.11.0.rst \ No newline at end of file diff --git a/doc/specs/implemented/deployment_type.rst b/doc/specs/implemented/deployment_type.rst new file mode 100644 index 00000000..bbf0f4b5 --- /dev/null +++ b/doc/specs/implemented/deployment_type.rst @@ -0,0 +1,216 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +============================ +Rally Deployment Unification +============================ + +Make Rally be able to examine any software through the API, +unbound it from OpenStack. + + +Problem description +=================== + +Rally is able to examine only system that use Keystone as a authentication +services, which limits sphere where Rally is suitable. + +At the moment to run Rally Task or Rally Verify you must specify OpenStack +deployment which contains credentials of it. These credentials are used in +Rally Task & Verify for different setups and validations. + +Rally is not able to store more than one credential for one deployment, so +it is impossible to support multi-scenario runs related to different systems. + + +Proposed change +=============== + +* Modify 'Deployment' database model to be able to store credentials + of many different systems, adding type of system. + +Now we have model Deployment with admin and users columns, +which are credentials for Keystone (tight coupled with OpenStack). + +There is next model now: + +.. code-block:: python + + class Deployment(BASE, RallyBase): + ... + admin = sa.Column(types.PickleType, nullable=True) + users = sa.Column(types.PickleType, default=[], nullable=False) + ... + +and values of columns in DB something like that: + +``admin = {admin_creds} or None`` + +``users = [{user_creds1}, {user_creds2}, ...] or []`` + +We need to decouple deployment from OpenStack and +make credentials more flexible, we describe it in one column named +``credentials``, where we can store special structure containing credentials +for many different systems, including type of credentials for each. + +.. code-block:: python + + class Deployment(BASE, RallyBase): + ... + credentials = sa.Column(types.PickleType, default=[], nullable=False) + ... + +So, for current OpenStack credentials we will have next data +in credentials column in DB after migration: + +.. code-block:: python + + credentials = [ + [ + "openstack", + {admin: {admin_creds} or None, + users: [{user_creds1}, {user_creds2}, ...] or []} + ], + ] + +and for multi-credentials deployment: + +.. code-block:: python + + credentials = [ + [ + "openstack", + {admin: {admin_creds} or None, + users: [{user_creds1}, {user_creds2}, ...] or []} + ], + [ + "zabbix", + {"url": "example.com", "login": "admin", "password": "admin"} + ] + ] + +Future summarized schema in DB: +``credentials = [[, ], ... ]`` + +To implement this point we need to write db migration, tests for it +and write adapters for credentials get/create/update methods, +mostly for support backward compatibility in ``rally.api`` module methods. + +* Get rid of ``rally.common.objects.credential.Credential`` class + and fix it usages mostly in ``rally.osclients`` if needed. + +Refactor all usages of passing ``rally.common.objects.credential.Credential`` +to ``rally.osclients.OSClient``, make possible to take dict as credentials +for ``rally.osclients.OSClient`` class, initialise +``rally.plugins.openstack.credentials.OpenStackCredentials`` class +in ``OSClient`` ``__init__`` method. + +Base class for credentials will be inherited from plugins.Plugin +and must implement validation method, +it will be placed in ``rally.plugins.common.credentials``: + +.. code-block:: python + + @six.add_metaclass(abc.ABCMeta) + @plugin.configure(name="base_credentials", schema="{...}") + class Credentials(plugin.Plugin): + def __init__(self, credentials): + self.validate(credentials) + super(Credentials, self).__setattr__("credentials", credentials) + + def __getattr__(self, item): + if item in self.__dict__: + return self.__dict__[item] + return self.credentials[item] + + def __setattr__(self, key, value): + self.credentials[key] = value + + def to_dict(self): + return self.credentials.copy() + + def validate(self, obj): + jsonschema.validate(obj, self._meta_get("schema")) + +and we need to add child for openstack credentials, +it will be placed in ``rally.plugins.openstack.credentials``: + +.. code-block:: python + + openstack_credentials_schema = { + "type": "object", + + "properties": { + "auth_url": {"type": "string"}, + "username": {"type": "string"}, + "password": {"type": "string"}, + }, + "required": ["auth_url", "username", "password"] + } + + @plugin.configure(name="openstack_credentials", + schema=openstack_credentials_schema) + class OpenStackCredentials(Credentials): + pass + +Replace usage of ``rally.common.objects.credential.Credential`` to +``rally.plugins.openstack.credentials.OpenStackCredentials`` +in ``rally.osclients`` + +* Update cli to show deployment type in output of 'rally deployment list'. + +Make possible to show deployments list in case of multi-scenario as: + +.. code-block:: shell + + > rally deployment list # (in case of many deployments) + + uuid | name | created_at | type | credential + -------+--------+------------+-----------+--------------------------------- + | | 21-02-2016 | openstack | {"admin": {...}, "users": [...]} + | zabbix | {"login": "login", "psw": "..."} + + +Alternatives +------------ + +None + + +Implementation +============== + + + +Assignee(s) +----------- + +Primary assignee: + rpromyshlennikov aka Rodion Promyshlennikov (rpromyshlennikov@mirantis.com) + + +Work Items +---------- + +- Change Deployment db model class +- Write migrations +- Make adapters for credentials get/create/update methods for temporary + support changed data format +- Remove all usages of passing ``rally.common.objects.credential.Credential`` + to ``rally.osclients.OSClient`` +- Create new plugin based class for credentials +- Write subclass of rally.plugins.common.credentials.Credential + for OpenStack credentials with proper validation of them +- Migrate to new credentials class +- Remove ``rally.common.objects.credential.Credential`` class +- Improve CLI-client to make possible show multi-credentials deployments. +- Feature refactoring: remove adapters after + "Multi Scenario support" implementation. + +Dependencies +============ + +None diff --git a/doc/specs/implemented/osprofiler.rst b/doc/specs/implemented/osprofiler.rst new file mode 100644 index 00000000..41dbaa43 --- /dev/null +++ b/doc/specs/implemented/osprofiler.rst @@ -0,0 +1,162 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +.. + This template should be in ReSTructured text. The filename in the git + repository should match the launchpad URL, for example a URL of + https://blueprints.launchpad.net/rally/+spec/awesome-thing should be named + awesome-thing.rst . Please do not delete any of the sections in this + template. If you have nothing to say for a whole section, just write: None + For help with syntax, see http://www.sphinx-doc.org/en/stable/rest.html + To test out your formatting, see http://www.tele3.cz/jbar/rest/rest.html + +================================= +OSProfiler integration into Rally +================================= + +The OSProfiler is a distributed trace toolkit library. It provides pythonic +helpers to do trace generation to avoid repeated code to trace WSGI, RPC, DB, +and other important places... + +Integration OSProfiler into Rally can help to dig into concurrency problems of +OpenStack which is a huge ecosystem of cooperative services. + +Problem description +=================== + +Rally Framework provides a powerful interface to generate real, big, load for +the deployment. Such load can kill the cloud, specifically OpenStack. There is +no way to identify reasons and bottlenecks without parsing timestamps and logs. +To fix that issue embedding profiling into each of workload iteration can help +to display wide picture of where we were in that particular moment when +something went wrong. + +Proposed change +=============== + +Two facts about OSProfiler which are the start point for the proposed changes: + +* HMAC key is used as a secret identifier while profiling +* Initialization of profiling is made in thread safe mode. Profiling of one + iteration should not influence profiling another one + +Storing secret key +------------------ + +The HMAC key is not something that will be changed from one task to another. +It is specific thing for the deployment, like authentication url or other +credentials. That is why Rally deployment config is the best place to store +such information. + +Since OSProfiler is OpenStack specific tool, we need to extend OpenStack +credentials model in Rally to support new argument. It should be done in two +places: validation (by modifying jsonschema [0]_) and the place where +credentials are store (specific class [1]_ [2]_). + +Initialization profiling +------------------------ + +As it was mentioned before, we need to initialize OSProfiler per iteration. +OSProfiler is made in thread safe mode [3]_, so we should not have problem from +that side. + +Initialization of OSProfiler is quite simple. + + .. code-block:: python + + from osprofiler import profiler + + profiler.init(HMAC_KEY) + + +As for the place where to initialize OSProfiler in Rally, constructor of +scenario is a good choice. First of all, we have a separate class for OpenStack +scenarios [4]_ which means that integration with OSProfiler there will not +affect all other platforms. Another reason for using constructor is that we +initialize new instance of scenario class for each iterations. + +Storing profiling results +------------------------- + +OSProfiler sends to collector a message at every trace point. We should not +care about supported OSProfiler backends and use only OSProfiler as +entry-point. + +The full trace can be obtained via trace-id after profiling is initialized. + + .. code-block:: python + + from osprofiler import profiler + + trace_id = profiler.get().get_base_id() + +At the first step of integration OSProfiler in Rally, let's store that trace-id +just like simple text. It will allow to show trace-id in Rally HTML and JSON +reports. + + .. code-block:: python + + self.add_output(complete={"title": "OSProfiler Trace-ID", + "chart_plugin": "TextArea", + "data": [trace_id]}) + +We can execute these lines in the same place where we initialize OSProfiler. + +In future, we should develop a separate chart that will embed OSProfiler html +report as a separate tab in the Rally report. + +Enabling profiling +------------------ + +Enabling/disabling profiling should be done via rally configuration file: + +* It is common place for storing different kinds of options. +* There is planned feature that will able to re-set config options via + deployment config or task file. + +The default value of that options should be True. In case of missing HMAC key +in credentials, attempt to initialize OSProfiler should not be started. + +Alternatives +------------ + +Here [5]_ you can find the answer to that section. + + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + Andrey Kurilin + + +Work Items +---------- + +* Extend OpenStack credentials +* Add new configuration option to Rally +* Extend OpenStack scenario base class to initialize OSProfiler and store + trace id + + +Dependencies +============ + +None + + +References +========== + +.. [0] https://github.com/openstack/rally/blob/a5691d7850b5abd7ea707730f0d48d75116d88d3/rally/plugins/openstack/credential.py#L154 +.. [1] https://github.com/openstack/rally/blob/a5691d7850b5abd7ea707730f0d48d75116d88d3/rally/plugins/openstack/credential.py#L26 +.. [2] https://github.com/openstack/rally/blob/a5691d7850b5abd7ea707730f0d48d75116d88d3/rally/plugins/openstack/credential.py#L161 +.. [3] https://github.com/openstack/osprofiler/blob/1.8.0/osprofiler/profiler.py#L29-L30 +.. [4] https://github.com/openstack/rally/blob/a5691d7850b5abd7ea707730f0d48d75116d88d3/rally/plugins/openstack/scenario.py#L28-L55 +.. [5] https://docs.openstack.org/osprofiler/latest/user/background.html#why-not-cprofile-and-etc