Cluster conductor creates trusts for all drivers, but does not clean
them up. The Heat driver has previously performed this action.
This change moves the lifecycle of trust and certificate creation
to the Conductor, so drivers do not need to clean up resources they
didn't create.
Change-Id: I2b3e99589d2d3069191d0727406601f0647a9722
This period job has been deprecated since Change
I3ca0f2e96fe63870406cc5323f08fa018ac6e8be in Rocky/Stein.
As it defaults to disabled, it causes logs like the following to be sent
over and over again.
Running periodic task MagnumPeriodicTasks._send_cluster_metrics
Skip sending cluster metrics _send_cluster_metrics
Remove the code totally as it has basically been a noop for a few
cycles.
Change-Id: Ib9142ab17d562b1d7ccf1409a9e0d934585a094d
Trust token can be deleted outside of magnum,
But when trust token not found, the periodic update status job will
stay in inprogress unless another cluster action triggered.
Propose to use admin context when trust can not be found in periodic
update status job.
Story: 2010232
Task: 46031
Change-Id: I9cc9a0e654fb26ebec517e3413a592ac6613777c
The "spacing" parameter of peirod jobs sync_cluster_status and
sync_cluster_health_status are placed incorrectly. This patch
fixes it.
Task: 41200
Story: 2007683
Change-Id: I9a129241d5f31d84d93da047c654da21604eff66
The default 10 seconds health polling interval is too frequent for most of
the cases. Now it has been changed to 60s. A new config
`health_polling_interval` is added to make the interval configurable.
Cloud admin can totally disable the health polling by set a negative value
for the config.
Task: 39795
Story: 2007683
Change-Id: Iad30487b8c119e94ee21d75f53fb86eb23dff365
Magnum is sending notifications like cluster create but has no
details regarding the cluster, like cluster UUID. Notifications
from other OpenStack projects contain full detailed information
(e.g. instance UUID in Nova instance create notification).
Detailed notifications are important for other OpenStack
projects like Searchlight or third party projects that cache
information regarding OpenStack objects or have custom actions
running on notification. Caching systems can efficiently update
one single object (e.g. cluster), while without notifications
they need to periodically retrieve object list, which is
inefficient.
Change-Id: I820fbe0659222ba31baf43ca09d2bbb0030ed61f
Story: #2006297
Task: 36009
This changes the existing cluster APIs and the cluster conductor to
take into consideration nodegroups:
* create: now creates the default nodegroups for the cluster
* update: updates the default nodegroups of the cluster
* delete: deletes also the nodegroups that belong to the cluster
* cluster_resize: takes into account the nodegroup provided by the API
story: 2005266
Change-Id: I5478c83ca316f8f09625607d5ae9d9f3c02eb65a
Calling Kubernetes native API to update the cluster health status
so that it can used for cluster auto healing.
Task: 24593
Story: 2002742
Change-Id: Ia76eeeb2f1734dff38d9660c804d7d2d0f65b9fb
Cleaning up comments and logging to make sure they properly adhere
to Openstack standards.
* Consistently use """ instead of ''' for comments.
* Always lazy-load logging parameters.
* Fixed bad log line in cert_manager.
Change-Id: I547f5dfa61609a899aef9b1470be8d8a6d8e4b81
Currently, Magnum is running periodic tasks to collect k8s cluster
metrics to message bus. Unfortunately, it's collecting pods info
only from "default" namespace which makes this function useless.
What's more, even Magnum can get all pods from all namespaces, it
doesn't make much sense to keep this function in Magnum. Because
operators only care about the health of cluster nodes. If they
want to know the status of pods, they can use heapster or other
tools to get that.
Task: 22619
Story: 1775116
Change-Id: I3ca0f2e96fe63870406cc5323f08fa018ac6e8be
Added configuration parameter, send_cluster_metrics, to magnum.conf
with default value of True. If set to True, periodic tasks will pull
COE data and send to ceilometer. This parameter can be set to False to
disable periodic collection of data to avoid unnecessary load from the
cluster.
Closes-Bug: #1668330
Related-Bug: #1746510
Change-Id: I9945293e7b2b52731f6e220d0925c1f6ad097caa
* Add osprofiler wsgi middleware. This middleware is used for 2 things:
1) It checks that person who wants to trace is trusted and knows
secret HMAC key.
2) It starts tracing in case of proper trace headers
and adds first wsgi trace point, with info about HTTP request
* Add initialization of osprofiler at start of service
Currently that includes oslo.messaging notifer instance creation
to send Ceilometer backend notifications.
* Traces HTTP/RPC/DB API calls
Demo: https://hieulq.github.io/cluster-create-false-new-html.html
Co-Authored-By: Hieu LE <hieulq@vn.fujitsu.com>
Implements: blueprint osprofiler-support-in-magnum
Change-Id: I7d68995aab81d365433950aada078ef1fcd5469b
Move cluster status change notifications into the
periodic task so that drivers do not have to have
any knowledge of Magnum notification strategy.
Change-Id: I5c71dd780f7bd6d4b683e491f5b4ce22cecb396c
Partial-Blueprint: bp-driver-consolodation
This is an alternative implementation to:
https://review.openstack.org/#/c/397961
This version implements an earlier proposal from the
spec that adds a driver method for synchronizing
cluster state. This method is optional so that drivers
that do not wish to leverage the existing periodic
synchronization task can do so in whatever manner
they wish and Magnum will not force them to do anything
unnecessarily.
1. add an update_cluster_status method to the driver
interface
2. implment update_cluster_status for Heat drivers
using the existing tested logic
3. Remove cluster status updates from the cluster conductor
in favor of the periodic sync_cluster_status task - this
should avoid timeouts and race conditions possible in the
previous implementation
4. Update the periodic sync_cluster_status method to use
the driver to update cluster status rather than calling
Heat directly
Change-Id: Iae0ec7af2542343cc51e85f0efd21086d693e540
Partial-Blueprint: bp-driver-consolodation
This is patch 3 of 3 to change the internal usage of the terms
Bay and BayModel. This patch updates Bay to Cluster in DB and
Object as well as all the usages. No functionality should be
changed by this patch, just naming and db updates.
Change-Id: Ife04b0f944ded03ca932d70e09e6766d09cf5d9f
Implements: blueprint rename-bay-to-cluster
This patch set modifies lines which are importing objects
instead of modules. As per openstack import guide lines, user should
import modules in a file not objects.
http://docs.openstack.org/developer/hacking/#imports
Closes-Bug: #1620161
Change-Id: I7ec9022a6b1cec36c678a2cec2a1856e70a51c68
This is the first of several patches to add new Cluster commands
that will replace the Bay terminalogy in Magnum. This patch adds
the new Cluster and ClusterTemplate commands in addition to the
Bay and Baymodel commands. Additional patches will be created
for client, docs, and additional functional tests.
Change-Id: Ie686281a6f98a1a9931158d2a79eee6ac21ed9a1
Implements: blueprint rename-bay-to-cluster
There is a rollback mechanism in heat after the stack
update failed. There should be a rollback mechanism in
magnum after bay update failed.
This patch add new microversion 1.3 to add rollback
support for Magnum bay, user can enable rollback on bay
update failure by specifying microversion 1.3 in header(
{'OpenStack-API-Version': 'container-infra 1.3'}) and
passing 'rollback=True'(http://XXX/v1/bays/XXX/?rollback=True)
when issuing bay update reqeust.
Change-Id: Idd02769f98078702404a11dc9f7a3339ce4e22eb
Partially-Implements: blueprint bay-rollback-on-update-failure
Added i18n translation to all log messages. Also added
hacking check for this to ensure in future all log
messages are translated.
Change-Id: I77fbb18ffe47b12232125cc1fa7aae97ea9b1214
Closes-Bug: #1603759
The periodic task unneccessarily lists Heat stacks in the
global tenant (across all tenants) which the Magnum service
user may lack permission for. Also, the most restrictive way
to let it use global stack-list is chose a Keystone role and
open that operation to any user in any project holding that
role.
This commit substitutes a direct lookup of all bays' stack_id
attributes for this global stack list. This direct lookup will
yield the same net result. In order to get the neccessary
permissions it will use each bay's stored Keystone trust to
act on behalf of the bay's creating user.
Co-Authored-By: Jiri Suchomel <jiri.suchomel@suse.com>
Closes-Bug: #1589955
Change-Id: I67b176c137c463e37e037970cc4e468d51db30c9
There are files containing string format arguments inside logging
messages. Using logging function parameters should be preferred.
Change-Id: Id558f66de13146f6ae76a7a69f49721b6c3d6257
Closes-Bug: #1321274
Updated import order as per the OpenStack import standards [1].
Modules under magnum/common/pythonk8sclient are not covered in
this patch because this code is generated via a third party
tool "swagger-codegen" which does not follows openstack
standards yet.
[1] http://docs.openstack.org/developer/hacking/#import-order-template
Change-Id: Ia87f3e797ee627bf3c04d2bae9639fd305b41b0d
Liberty tech-debt-bug
Magnum service was initially utilizing periodic task within magnum
conductor. This commit lays out independent periodic framework for
magnum service, by decoupling from periodic task.
Change-Id: Ie0e725b75069309a9fa783f77c1a454f9dae549e
Partial-Bug: 1500998
Not removing existing try-catch block for sync_bay_status and other
periodic tasks. Rather leaving a note with the pointers and explanation
for developers to add new periodic task.
Change-Id: I2560d3961c19d73ab78a8e9812d69742c6bd2117
Closes-Bug:1496094
The cyclomatic complexity of sync_bay_status was greater than 10 and
very hard to comprehend. This patch reduces the complexity and makes the
task more open to refactoring and maintence.
Partial-Bug: #1501331
Change-Id: If415aa7710b586b0429ed1bc5077976791b424d1
* Implement a bay monitoring framework that provides APIs to pull data from
bay, and compute metrics based on the data.
* Add support for computing memory utilization of a swarm bay.
* Add a periodic task that sends metrics to ceilometer.
Change-Id: I32730e116c59a4e5f9d1ccb466d464c2fc1f6193
Partially-Implements: blueprint autoscale-bay
Partially-Implements: blueprint magnum-notifications
To support 'magnum service-list' after 'nova service-list', we need to
introduce periodic status update functionality for internal services.
Change-Id: Ia0c09222405c87cb61e5de4a43ba345ae3405b50
Partially-Implements: blueprint magnum-service-list
Closes-bug: #1492501
In periodic task, when bay status is found to be different with
corresponding stack, or the corresponding stack cannot be found in
heat, status reason should be synced as well as status. Otherwise,
status reason will be meaningless in some situation.
Change-Id: I27d6a7ecb0c74bd4d4a2bb2bfcd81d7632cd2147
Closes-Bug: #1488709
In before, bay and x509keypair supported list resources from all
tenants. It is desirable to generalize this capability while reducing
duplicated codes. Therefore, moving 'all_tenants' options to context.
Change-Id: Icfe31a6f2ac2e21fa7f377e244764f10892d25c7
Partially-Implements: blueprint autoscale-bay
Statuses are currently defined as a StringField when they should
be defined as an Enum object. This fix changes these objects from
using StringField by extending the Enum object in
oslo.versionedobjects
Change-Id: I12760e1d72f4af9931ee97bbad298f92f785118f
Partial-Bug: #1489136
The filters in method 'sync_bay_status' in periodic.py should be a dict,
instead of a list.
Change-Id: Id12e8768abb17b70bbc92d951b0d95e765d94e38
Closes-Bug: #1487812
In order to run several magnum-conductor, we should make periodic
tasks can run simultaneously. We should not destroy a bay twice.
Change-Id: Ie3c66442839197196f532734e67d810749ec5052
Partial-Implements: blueprint horizontal-scale
This patch makes some general cleanup changes to help strings to tidy
up various areas of the code. No functional impacts result from the
changes within this patch.
Closes-Bug: #1477837
Change-Id: I25a81059e974ac0083bb439f1cd37fb82157b3bb
This patch adds periodic task `sync_bay_status` which will be used for syncing
bay's status from heat stack, we will pull bays which status in
[bay_status.CREATE_IN_PROGRESS,
bay_status.UPDATE_IN_PROGRESS,
bay_status.DELETE_IN_PROGRESS]
which are all in a temporary status, and try to sync up the status with heat's
stack.
status changes will be like this:
bay_status stack_status sync up bay_status to
---------- ------------ ---------------------
CREATE_IN_PROGRESS CREATE_COMPLETE CREATE_COMPLETE
UPDATE_IN_PROGRESS UPDATE_COMPLETE UPDATE_COMPLETE
DELETE_IN_PROGRESS DELETE_COMPLETE DELETE_COMPLETE
CREATE_IN_PROGRESS CREATE_FAILED CREATE_FAILED
UPDATE_IN_PROGRESS UPDATE_FAILED UPDATE_FAILED
DELETE_IN_PROGRESS DELETE_FAILED DELETE_FAILED
CREATE_IN_PROGRESS Not Found CREATE_FAILED
UPDATE_IN_PROGRESS Not Found UPDATE_FAILED
DELETE_IN_PROGRESS Not Found destroy
Partial-Implements: blueprint add-periodic-task
Co-Authored-By: ShaoHe Feng <shaohe.feng@intel.com>
Change-Id: Ie9cc4d3f03c7938a8d988010604da79c9b8a22fd
This patch adds periodic timer to conductor service.
Any new periodic tasks need to be added to magnum/service/periodic.py
We add 2 default configure options to support periodic task:
'periodic_enable', default=True
'periodic_interval_max', default=60
These 2 options are for rpc_service.
PS:
We don't need to supervisor timers because timer itself won't have any
exception, and it is not a thread, which means can not be killed.
Partial-Implements: blueprint add-periodic-task
Co-Authored-By: ShaoHe Feng <shaohe.feng@intel.com>
Change-Id: I132d9cc89b26baf249b62b713ccd3776f0078d7d
Dale Smith