Commit Graph

2447 Commits

Author SHA1 Message Date
James Page bc5922c684
Fix oslo.db >= 15.0.0 compatibility
Minimal refactor of SQLAlchemy api module to be compatible with
oslo.db >= 15.0.0 where autocommit behaviour was dropped.

Closes-Bug: #2056181
Change-Id: I33be53f647faae2aad30a43c10980df950d5d7c2
2024-03-27 09:41:23 +00:00
Ghanshyam Mann 9d58a6d457 Update python classifier in setup.cfg
As per the current release tested runtime, we test
python version from 3.8 to 3.11 so updating the
same in python classifier in setup.cfg

Change-Id: Ie010eea38eb0861699b60f16dfd3e2e95ae33709
2024-01-09 19:22:04 -08:00
Lucian Petrut c95ce4ec17 Add MAAS support
At the moment, Watcher can use a single bare metal provisioning
service: Openstack Ironic.

We're now adding support for Canonical's MAAS service [1], which
is commonly used along with Juju [2] to deploy Openstack.

In order to do so, we're building a metal client abstraction, with
concrete implementations for Ironic and MAAS. We'll pick the MAAS
client if the MAAS url is provided, otherwise defaulting to Ironic.

For now, we aren't updating the baremetal model collector since it
doesn't seem to be used by any of the existing Watcher strategy
implementations.

[1] https://maas.io/docs
[2] https://juju.is/docs

Implements: blueprint maas-support

Change-Id: I6861995598f6c542fa9c006131f10203f358e0a6
2023-12-11 10:21:33 +00:00
Zuul 9492c2190e Merge "vm workload consolidation: use actual host metrics" 2023-12-01 01:51:39 +00:00
Lucian Petrut 808f1bcee3 Update action json schema
Power-off actions created by the energy saving strategy include
a resource name property, which currently isn't part of the
action json schema. For this reason, json schema validation fails.

  Additional properties are not allowed ('resource_name' was unexpected)

We'll update the json schema, including the resource name property.

Change-Id: I924d36732a917c0be98b08c2f4128e9136356215
2023-11-15 01:11:56 +00:00
Lucian Petrut 3b224b5629 Fix object tests
A couple of object tests are failing, probably after a dependency
bump.

watcher.objects.base.objects is mocked, so the registered object
version isn't properly retrieved, leading to a type error:

    File "/mnt/data/workspace/watcher/watcher/tests/objects/test_objects.py",
    line 535, in test_hook_chooses_newer_properly
      reg.registration_hook(MyObj, 0)
    File "/mnt/data/workspace/watcher/watcher/objects/base.py",
    line 46, in registration_hook
      cur_version = versionutils.convert_version_to_tuple(
    File "/home/ubuntu/openstack_venv/lib/python3.10/site-packages/oslo_utils/versionutils.py",
    line 91, in convert_version_to_tuple
      version_str = re.sub(r'(\d+)(a|alpha|b|beta|rc)\d+$', '\\1', version_str)
    File "/usr/lib/python3.10/re.py", line 209, in sub
      return _compile(pattern, flags).sub(repl, string, count)
  TypeError: expected string or bytes-like object

We'll solve the issue by setting the VERSION attribute against
the mock object.

Change-Id: Ifeb38b98f1d702908531de5fc5c846bd1c53de4b
2023-11-14 10:38:40 +00:00
Lucian Petrut 424e9a76af vm workload consolidation: use actual host metrics
The "vm workload consolidation" strategy is summing up instance
usage in order to estimate host usage.

The problem is that some infrastructure services (e.g. OVS or Ceph
clients) may also use a significant amount of resources, which
would be ignored. This can impact Watcher's ability to detect
overloaded nodes and correctly rebalance the workload.

This commit will use the host metrics, if available. The proposed
implementation uses the maximum value between the host metric
and the sum of the instance metrics.

Note that we're holding a dict of host metric deltas in order to
account for planned migrations.

Change-Id: I82f474ee613f6c9a7c0a9d24a05cba41d2f68edb
2023-10-27 21:54:42 +03:00
Zuul 40e93407c7 Merge "Handle deprecated "cpu_util" metric" 2023-10-27 09:47:38 +00:00
Zuul 721aec1cb6 Merge "vm workload consolidation: allow cold migrations" 2023-10-27 09:47:36 +00:00
Zuul 8a3ee8f931 Merge "Improve vm_consolidation logging" 2023-10-27 09:20:13 +00:00
Lucian Petrut 00fea975e2 Handle deprecated "cpu_util" metric
The "cpu_util" metric has been deprecated a few years ago.
We'll obtain the same result by converting the cumulative cpu
time to a percentage, leveraging the rate of change aggregation.

Change-Id: I18fe0de6f74c785e674faceea0c48f44055818fe
2023-10-24 10:47:23 +00:00
Lucian Petrut fd6562382e Avoid performing retries in case of missing resources
There may be no available metrics for instances that are stopped
or were recently spawned. This makes retries unnecessary and time
consuming.

For this reason, we'll ignore gnocchi MetricNotFound errors.

Change-Id: I79cd03bf04db634b931d6dfd32d5150f58e82044
2023-10-23 14:14:21 +00:00
Lucian Petrut ec90891636 Improve vm_consolidation logging
We're adding a few info log messages in order to trace the
"vm consolidation" strategy more easily.

Change-Id: I8ce1a9dd173733f1b801839d3ad0c1269c4306bb
2023-10-23 14:10:02 +00:00
Lucian Petrut 7336a48057 vm workload consolidation: allow cold migrations
Although Watcher supports cold migrations, the vm workload
consolidation workflow only allows live migrations to be
performed.

We'll remove this unnecessary limitation so that stopped instances
could be cold migrated.

Change-Id: I4b41550f2255560febf8586722a0e02045c3a486
2023-10-23 13:03:18 +00:00
Lucian Petrut 922478fbda Unblock the CI gate
The Nova collector json schema validation started [1][2] failing after
the jsonschema upper constraint was bumped from 4.17.3 to 4.19.1 [3].

The reason is that jsonschema v4.18.0a1 switched to a reference
resolving library [4], which treats the aggregate "id" as a jsonschema
id and expects it to be a string [5]. For this reason, we're now getting
AttributeError exceptions.

As a workaround, we'll rename the "id" ref element as "host_aggr_id".

Also, the watcher-tempest-multinode job is configured to use Focal,
which is no longer supported by Devstack [6]. That being considered,
we'll switch to Ubuntu Jammy (22.04).

While at it, we're disabling Cinder Backup, which isn't used while
testing Watched. It currently causes Devstack failures since it
uses the Swift backend by default, which is disabled.

[1] https://paste.opendev.org/raw/bjQ1uIdbDMnmA1UEhxLL/
[2] https://paste.opendev.org/raw/bNgxqulBwBLYB7tNhrU4/
[3] ab0dcbdda2
[4] https://github.com/python-jsonschema/jsonschema/releases/tag/v4.18.0a1
[5] c23a5dc1c9/referencing/jsonschema.py (L54-L55C18)
[6] https://paste.openstack.org/raw/bSoSyXgbtmq6d9768HQn/

Change-Id: I300620c2ec4857b1e0d402a9b57a637f576eeb24
2023-10-23 09:21:55 +03:00
OpenStack Release Bot 9f0eca2343 Update master for stable/2023.2
Add file to the reno documentation build to show release notes for
stable/2023.2.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/2023.2.

Sem-Ver: feature
Change-Id: I8a0c75ce5a4e5ae5cccd8eb1cb0325747a619122
2023-09-14 01:24:43 +00:00
Zuul 1e11c490a7 Merge "Add timeout option for Grafana request" 2023-08-29 11:21:46 +00:00
Zuul 8a7a8db661 Merge "Imported Translations from Zanata" 2023-08-28 06:21:40 +00:00
BubaVV 0610070e59 Add timeout option for Grafana request
Implemented config option to setup Grafana API request timeout

Change-Id: I8cbf8ce22f199fe22c0b162ba1f419169881f193
2023-08-23 17:46:19 +03:00
OpenStack Proposal Bot a0997a0423 Imported Translations from Zanata
For more information about this automatic import see:
https://docs.openstack.org/i18n/latest/reviewing-translation-import.html

Change-Id: I37201577bd8d9c53db8ce6700f47d911359da6d2
2023-08-14 04:24:29 +00:00
chenker 4ea3eada3e Fix watcher comment
Change-Id: I4512cf1032e08934886d5e3ca858b3e05c3da76c
2023-08-13 00:00:12 +00:00
Zuul cd1c0f3054 Merge "Imported Translations from Zanata" 2023-03-08 07:04:33 +00:00
OpenStack Proposal Bot 684350977d Imported Translations from Zanata
For more information about this automatic import see:
https://docs.openstack.org/i18n/latest/reviewing-translation-import.html

Change-Id: I4ee251e6d37a1b955c22dc6fdc04c1a08c9ae9b8
2023-03-02 03:28:31 +00:00
OpenStack Release Bot d28630b759 Update master for stable/2023.1
Add file to the reno documentation build to show release notes for
stable/2023.1.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/2023.1.

Sem-Ver: feature
Change-Id: Ia585893e7fef42e9991a2b81f604d1ff28c0a5ad
2023-02-28 13:31:08 +00:00
Zuul f7fbaf46a2 Merge "Use new get_rpc_client API from oslo.messaging" 2023-02-09 01:25:15 +00:00
Zuul e7cda537e7 Merge "Modify saving_energy log info" 2023-02-07 12:18:58 +00:00
chenker c7be34fbaa update saving_energy docs
Change-Id: I3b0c86911a8d32912c2de2e2392af9539b8d9be0
2023-02-07 10:27:54 +00:00
chenker 52da088011 Modify saving_energy log info
Change-Id: I84879a453aa3ff78917d1136c62978b9d0e606de
2023-02-07 10:20:04 +00:00
Tobias Urdin 6ac3a6febf Fix passenv in tox.ini
Change-Id: If1ddb1d48eeb96191bcbfadd1a5e14f4350a02e4
2023-02-07 08:02:20 +00:00
Tobias Urdin e36b77ad6d Use new get_rpc_client API from oslo.messaging
Use the new API that is consistent with
the existing API instead of instantiating the client
class directly.

This was introduced in release 14.1.0 here [1] and
added into oslo.messaging here [2]

[1] https://review.opendev.org/c/openstack/requirements/+/869340
[2] https://review.opendev.org/c/openstack/oslo.messaging/+/862419

Change-Id: I43c399a0c68473e40b8b71e9617c8334a439e675
2023-01-19 20:50:26 +00:00
Thierry Carrez 6003322711 Move queue declaration to project level
This moves the watcher queue declaration from the pipeline level
(where it is no longer valid) to the project level.

https: //lists.openstack.org/pipermail/openstack-discuss/2022-May/028603.html
Change-Id: I06923abb00f7eecd59587f44cd1f6a069e88a9fc
2022-09-26 14:19:58 +02:00
Zuul f4ffca01b8 Merge "Switch to 2023.1 Python3 unit tests and generic template name" 2022-09-16 06:36:21 +00:00
Alfredo Moralejo 5d70c207cd Fix compatibility with oslo.db 12.1.0
oslo.db 12.1.0 has changed the default value for the 'autocommit'
parameter of 'LegacyEngineFacade' from 'True' to 'False'. This is a
necessary step to ensure compatibility with SQLAlchemy 2.0. However, we
are currently relying on the autocommit behavior and need changes to
explicitly manage sessions. Until that happens, we need to override the
default.

Co-Authored-By: Stephen Finucane <stephenfin@redhat.com>
Change-Id: I7db39d958d087322bfa0aad70dfbd04de9228dd7
2022-09-15 16:52:41 +02:00
OpenStack Release Bot 0b2e641d00 Switch to 2023.1 Python3 unit tests and generic template name
This is an automatically generated patch to ensure unit testing
is in place for all the of the tested runtimes for antelope. Also,
updating the template name to generic one.

See also the PTI in governance [1].

[1]: https://governance.openstack.org/tc/reference/project-testing-interface.html

Change-Id: Ide6c6c398f8e6cdd590c6620a752ad802a1f5cf8
2022-09-13 12:30:33 +00:00
OpenStack Release Bot ff84b052a5 Update master for stable/zed
Add file to the reno documentation build to show release notes for
stable/zed.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/zed.

Sem-Ver: feature
Change-Id: I1726e33a14038712dbb9fd5e5c0cddf8ad872e69
2022-09-13 12:30:32 +00:00
Zuul a43b040ebc Merge "Imported Translations from Zanata" 2022-08-30 10:44:52 +00:00
Zuul 749fa2507a Merge "Tests: fix requirements for unit tests" 2022-08-30 08:15:05 +00:00
OpenStack Proposal Bot 76d61362ee Imported Translations from Zanata
For more information about this automatic import see:
https://docs.openstack.org/i18n/latest/reviewing-translation-import.html

Change-Id: I95133dece6fdaf931dfed64015806430ba8d04f0
2022-08-29 04:12:15 +00:00
wangjiaqi07 c55143bc21 remove unicode from code
Change-Id: I747445d482a2fb40c2f39139c5fd2a0cb26c27bc
2022-08-19 14:17:10 +08:00
suzhengwei 7609df3370 Tests: fix requirements for unit tests
Add WebTest to test-requirements which used to be imported as a
transitive requirement via pecan, but the latest release of
pecan dropped this dependency. So make this requirement explicit.

Related-Bug: #1982110
Change-Id: I4852be23b489257aaa56d3fa22d27f72bcabf919
2022-07-28 16:14:13 +08:00
chenker b57eac12cb Watcher DB upgrde compatibility consideration for add_apscheduler_jobs
Change-Id: I8896ff5731bb8c1bf88a5d7b926bd2a884100ea8
2022-04-28 02:21:06 +00:00
OpenStack Release Bot ac6911d3c4 Add Python3 zed unit tests
This is an automatically generated patch to ensure unit testing
is in place for all the of the tested runtimes for zed.

See also the PTI in governance [1].

[1]: https://governance.openstack.org/tc/reference/project-testing-interface.html

Change-Id: I5cf874842550de18ff777b909fd28e2c32e6d530
2022-03-10 12:14:06 +00:00
OpenStack Release Bot 23c2010681 Update master for stable/yoga
Add file to the reno documentation build to show release notes for
stable/yoga.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/yoga.

Sem-Ver: feature
Change-Id: Ic7c275b38fef9afc29577f81fe92546bb94b2930
2022-03-10 12:14:04 +00:00
zhurong 01d74d0a87 Remove ceilometerclient dependecy
Change-Id: Ifa0f2493aa8414a29dc2722b6636a33bc5808be6
2022-01-07 05:48:22 +00:00
OpenStack Release Bot e4fab0ce7f Add Python3 yoga unit tests
This is an automatically generated patch to ensure unit testing
is in place for all the of the tested runtimes for yoga.

See also the PTI in governance [1].

[1]: https://governance.openstack.org/tc/reference/project-testing-interface.html

Change-Id: I328b3ccb76153fa0dbb4d174dd976412be049200
2021-09-15 17:14:09 +00:00
OpenStack Release Bot 76ecaaeb3a Update master for stable/xena
Add file to the reno documentation build to show release notes for
stable/xena.

Use pbr instruction to increment the minor version number
automatically so that master versions are higher than the versions on
stable/xena.

Sem-Ver: feature
Change-Id: If1c02305a153575c6a550844b0c6f45b74ea5ef3
2021-09-15 17:14:07 +00:00
chenker 6dd2f2a9c1 BugFix: Prevent float type variables from being passed to random
>>> random.sample([5,10], 1.3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.6/random.py", line 321, in sample
    result = [None] * k
TypeError: can't multiply sequence by non-int of type 'float'

Change-Id: Ifa5dca06f07220512579e4fe3c5c741aeffc71cc
2021-08-23 01:58:52 +00:00
Takashi Kajinami a993849928 Use Block Storage API v3 instead of API v2
Block Storage API v2 was deprecated during Pike cycle and is being
removed during Xena cycle, and current v3 API should be used instead.

Change-Id: Ia5247742b31f5f07186ef908588f0972d3ac609f
2021-07-27 11:04:16 +09:00
Zuul 6dbac1f6ae Merge "Add watcher dashboard to devstack documentation" 2021-07-13 14:31:16 +00:00
sue c28756c48b use HTTPStatus instead of direct code
Python introduced http.HTTPStatus since version 3.5,
and Wallaby has targeted a minimum version of python 3.6.

Change-Id: I45f732f0f59b8fae831bb6c07f4fdd98cdd7409a
2021-07-09 11:02:36 +02:00