Commit Graph

81 Commits

Author SHA1 Message Date
Sergiy Markin 6b317dda05 Airflow stable 2.8.2
This PS bumps up Airflow version to the latest
2.8.2 and also bumps up openstack dependences to
Antelope 2023.1

Change-Id: I61b5b39079f9c3360f5dfaafb1f961b629795c1c
2024-02-26 20:59:52 +00:00
Sergiy Markin 1e180a3f51 Airflow stable 2.8.1
Change-Id: I7667fc5cfc5e02c360d0e964de75cf5d5e6fe825
2024-02-21 22:23:55 +00:00
Ruslan Aliev 47ced6d7de Allow creation of network bonding with a single interface
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
Change-Id: I7a5f51541051abbeada3bea1608394bb4e374115
2024-02-16 13:44:43 -06:00
Sergiy Markin f99abfa433 Airflow stable 2.6.2
This PS updates python modules and code to match Airflow 2.6.2:

- bionic py36 gates  were removed
- python code corrected to match new modules versions
- selection of python modules versions was perfrmed based on
  airflow-2.6.2 constraints
- postgresql image updated to 14.8

Change-Id: Ibdcc75e600166c20b842508aa5539587cca466f0
2023-08-29 00:50:54 +00:00
Sergiy Markin 73d22a488f Restored ubuntu_bionic image build
This PS restores image build for ubuntu_bionic and adds appropriate
gates to keep it tested by appropriate functional and integrational
tests.

Change-Id: I695fb42efff29aeda737f2b9c1eaeb84b95aa57c
2023-05-24 18:37:21 +00:00
Sergiy Markin d00eaf0303 Drydock focal related upgrades
This PS implements the following changes:
- switches freeze approach to requirements-direct.txt and
  requirements-frozen.txt files
- adjusts code tabulation style according to  yapf recommendations
- replaces deprecated usage of responce.body attribute with
  responce.text
- fixes integration tests in controlled by Makefile + tox
- uplifts Helm to v3.9.4

Change-Id: I751db72eb8f670825382f11a36657112faeb169a
2023-04-26 22:32:49 +00:00
Sergiy Markin 415a8b52c5 [focal] Python modules sync with Airship project
- uplifted some python modules
- fixed tox4 requirements
- added focal build node as a default one
- added bindep.txt and bindep role to playbooks and docker image build process
- changes Makefile to reflect GoLang and dependency management changes
- upgraded Helm to v3 for chart build process
- uplifted postgresql version to 14.6
- fixed deprecated falcon.API - replaced with falcon.APP
- fixed upstream docker image publishing process

Change-Id: I307d72bb7680f6f5c71e42ad30666cf786420460
2023-04-08 08:20:22 +00:00
Ruslan Aliev 98f3d886d8 Add retries to set/get boot device actions
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
Change-Id: I3c1f12fdad2bbd13503c85ae3ea54bc281f40207
2023-03-21 00:04:22 -05:00
Ruslan Aliev 389db640e4 Change MAAS api endpoints according to v3.0.0
Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
Change-Id: I9daa8b5b8d76f2da939ce0dffdb3471ba7cd8947
2023-03-10 01:44:09 -06:00
Markin, Sergiy (sm515x) b431f7a2cf [NIC Card Auto-Detect] NIC Card Auto-Detect
This patch is an addition to this patchset:

https://review.opendev.org/c/airship/drydock/+/850579

It adds more debug messages.

Change-Id: I01061934f5584276e461ef7a7f4c7a87c30755af
2022-08-02 14:54:27 +00:00
Markin, Sergiy (sm515x) afdfa33099 [NIC Card Auto-Detect] NIC Card Auto-Detect
This patch adds an ability to match several NIC addresses by one
hardware profile by using regex as the address. The regex expression has
to have regex: prefix in order to be recognized.

Change-Id: I0bb067fb1783725e4ac485683eb898d5fc2d7bf2
2022-07-20 20:09:43 +00:00
SPEARS, DUSTIN (ds443n) a171f3c7a5 Add postgresql retention cronjob
Adding cronjob to purge the drydock DB based on retention day value. Additionally adding drydock API endpoint for purging the tasks and result_message tables and running vacuum full on drydock DB.

Change-Id: Ibcce61ecdafa637ca3ffec654152060aae26d4b8
2022-05-18 10:55:26 -04:00
Phil Sphicas e3f984d92b Fix schema typo
Spelling correction in the BaremetalNode and HostProfile schemas.

Change-Id: I83ec294dec63a520dba892ed4dee5afa69134012
2021-11-10 08:54:19 -08:00
Phil Sphicas 611e98de3f Use OOB driver creds for MAAS
During MAAS enlistment (and commissioning), an IPMI account (named
"maas" by default) is created on each node, which MAAS then uses for
power management.

This change allows MAAS to use the same credentials as the ones used by
the OOB driver, by overwriting the power parameters for the discovered
nodes. This includes the power type, so if the node is configured to use
Redfish, then Drydock will update a MAAS node discovered as IPMI to use
Redfish instead.

It also provides an option to instruct MAAS not to recreate IPMI
credentials during commissioning, which is passed through to the MAAS
API. Setting this to true is only supported in MAAS 2.7 or later [0].

The two maasdriver configuration options are introduced in drydock.conf,
along with their default values:

    [maasdriver]
    use_node_oob_params = false
    skip_bmc_config = false

These options do not prevent MAAS from creating the IPMI account during
enlistment - this would require addition MAAS customization.

0: 8842d0bfd3

Change-Id: I24d3bc3b1cc94907d73bc247de3fc06dd4750ab1
2021-07-30 16:39:06 +00:00
Phil Sphicas 3cbeb29f2d Increase ThreadPoolExecutor max_workers
The existing max_workers setting for the ThreadPoolExecutor caps the
number of nodes that can be deployed concurrently to 16. This change
allows more threads to be run in parallel, allowing the operator to
deploy more nodes at once.

Change-Id: I6d1dfa7ad8f5e03a8328311e6b59ee116a250462
2021-07-28 22:56:13 +00:00
Phil Sphicas 292e94ee2c Avoid expensive MAAS API calls
The MAAS API call "GET /MAAS/api/2.0/machines/" retrieves information
about every machine known to MAAS, which is very slow. The API supports
filtering based on hostname and mac_address (among others), and querying
for power parameters for all nodes at once.

This change modifies identify_baremetal_node to avoid calling refresh on
the full machine list.

Also, the refresh method of ResourceCollectionBase is updated to allow
passing of params, which can be used to take advantage of the filtering.
Note that a filtered call to refresh overwrites the resources collection
to only contain the returned values.

Most calls to Machines.refresh() aren't really needed at all - they are
replaced with a call to Machines.empty_refresh(), which will still make
sure that the API endpoint is accessible but return an empty collection.
(This may get removed entirely in the future.)

Change-Id: Ie58c45e1790c5c827d9d47f5582214ca519946de
2021-07-28 22:56:13 +00:00
DeJaeger, Darren (dd118r) 3ca7978bc7 Drydock request factory tuning
This patchset tunes the Drydock MAAS request factory to:
a) Implement retries for requests toward MAAS_URL/api/2.0/
b) Bumps the request timeout slightly

In addition, restricts threaded actions towards nodes to
using the same MAAS client, effectively rate limiting calls
to the MAAS api.

Change-Id: I2e66105ae332adaed62c9c3bc8cddc63e1f7bf23
2021-07-27 05:44:19 +00:00
anthony.bellino b04304094f Gate Fixes for Drydock
* pep8 fixes for airship-drydock-omni-test
* Install requirements sequential for ubuntu_xenial image
* Install older version of pip<21.0 for ubuntu_xenial image

Change-Id: If494b6abc1f8b85d96da4cad2da8606b8b2a1352
2021-05-17 17:43:00 -07:00
Zuul c7583638e1 Merge "Accelerate YAML operations with LibYAML" 2020-09-28 16:58:34 +00:00
Phil Sphicas caa7f22b50 Accelerate YAML operations with LibYAML
Patch PyYAML (via the pylibyaml library) to automatically enable the
LibYAML parser and emitter, which are faster than the Python versions.

https://pypi.org/project/pylibyaml/

Change-Id: Iaddc0f30ed99b1f9a999f5365e9e8bf43349b82f
2020-09-25 08:07:49 +00:00
Rick Bartra 96db568d4b Update pip package versions in preparation of pip 20.3
When pip is upgraded to 20.3, the pip dependency resolver is much more
strict and will no longer install a combination of packages that is mutually
inconsistent[0].

These changes account for the fact that Shipyard imports Armada, Drydock,
Promenade, and Deckhand. Having said that, with pip 20.3, the pip
packages amongst those projects cannot conflict. A follow-up change may
be needed if more conflicts are found.

[0] https://pip.pypa.io/en/latest/user_guide/#changes-to-the-pip-dependency-resolver-in-20-2-2020

Change-Id: I89c6dc728824f00f964c794142766012c407c4ed
2020-09-23 17:17:28 -04:00
Mahmoudi, Ahmad (am495p) 5063769796 Added some validations loggs for site model
During drydock node deployments, sometimes MaaS node deployment for
some nodes fails when the node tries to pull the node bootaction
files, using drydock api.
Drydock api call fails with `500 Internal Server Error`, when
drydock tries to create the booaction files for the node. The logs,
however do no provide any additional clues on what caused drydock to
fail. This issue does not happen always, and subsequent site updates
will most of the deploy the failed nodes.

The additional checks and logs are added to help pinpoint the root cause
of the 500 return code, if/when this issue heppens again.

This ps also, uplifted `MarkupSafe` pip library from 1.0 to 1.1.1 to
address the issue with MarkupSafe and latest version of setuptools
described here: https://github.com/pallets/markupsafe/issues/116

Change-Id: I08a088d9690d8d9dd1f771dc5e84d1eb02fbd39f
2020-07-24 18:57:21 +00:00
Alexander Hughes 6a9c6b092b Fix image build checks missing setuptools
Use apt to install python3-pip, and use pip3 in event system has
both pip2 and pip3 installed. Use apt to install setuptools for
Ansible's consumption.

Change-Id: Id80c809c636abe41a1cbb4d465f82ed1e8e0e9d7
Signed-off-by: Alexander Hughes <Alexander.Hughes@pm.me>
2020-06-19 13:32:58 +00:00
Phil Sphicas 93f593f568 Support binary prefixes for node storage size
This change allows node storage sizes to be specified using binary
prefixes (MiB, GiB, TiB) in addition to the existing supported formats
(MB, GB, TB).

Change-Id: Idef88b648a75bad87625acf1d73af011480cc0b9
2020-06-02 07:11:13 +00:00
Phil Sphicas a75704b8cb fix: prevent multiple IP assignments
A recent change[0] to address PEP8 issues resulted in an unintended
behavior modification, in some cases resulting in MAAS allocation of
multiple IP addresses to the same NIC.

This reverts to the original code logic.

[0] 1755930331

Change-Id: I6dccd1b60c414e3aa966085e81dc0b61244e9814
2020-05-30 20:26:40 +00:00
Alexander Hughes 1755930331 Address PEP8 Failures in Drydock
Flake8 version recently updated to include new PEP8 rules. Some of
the codebase is not compliant with the new rules.

Change-Id: I0f5b3d41ee54ff0d9ffa05f733f98c7e34f0f258
Signed-off-by: Alexander Hughes <Alexander.Hughes@pm.me>
2020-05-12 18:27:06 +00:00
Alexander Hughes 80c82d8957 Uplift pyyaml and requests
Automatic security alerts were created for pyyaml==3.12 and
requests==2.19.1 suggesting these packages be upgraded to 2.20.0 and
5.1 respectively.

Vulnerabilities addressed:
CVE-2018-18074 on requests package
CVE-2017-18342 on PyYAML package

Change-Id: Iff5bc11d60c2724fef0bb8b2552e17573c79dc9f
Signed-off-by: Alexander Hughes <Alexander.Hughes@pm.me>
2020-03-04 23:11:31 +00:00
Ahmad Mahmoudi d0acfe77cc (fix) Address uwsgi segmentation fault issue
With Ubuntu bionic base image for drydock docker image, uwsig crashes
with segmentation fault when it tries to load/import the psycopg2 package.
The reason for this is that uwsgi and psycopg2 packages are built with
incopatible ssl libraries.

Upgrading uwsgi and psycopg2 to address this issue for bionic based
images.

Change-Id: I3d0bfb96c19849f5c2925366f8712bf47985df67
2020-02-26 15:09:22 +00:00
Zuul b7d4b12116 Merge "Improve error logging during validation" 2019-10-11 18:03:02 +00:00
Scott Hussey 45d470069e Improve error logging during validation
- Currently several failure paths won't log any messages
  when doing a site validation. Add these messages

- Also, for validation steps that are dependent on external resources
  make the resource inaccessibility a warning rather than a failure.

Change-Id: I431ed188e2f6cd3fc3fa41ae2729f3a099fdfbf5
2019-09-21 21:59:53 -05:00
Mahmoudi, Ahmad (am495p) a6fa70dff8 Allow Deploy Allocated Node
Updated the maas diver to allow node deploy in Allocated state.

Change-Id: Ic68279e641431e61ec714d94867bd7bbaea083c6
2019-08-26 22:03:44 -05:00
Zuul eb28cf3aec Merge "Support MAAS having dead rack controllers" 2019-05-30 20:01:01 +00:00
BARTRA, RICK fdb6dcaca6 Add CIDR validation for MAAS
MAAS only accepts CIDR IPs that do not have host bits set otherwise
MAAS sees the CIDR as a second network. This commit adds a Drydock
validation that checks if the CIDR has host bits and also suggests
which CIDR to use if the provided one is not acceptable to MAAS.

Change-Id: Ib6d4d8277d0e1634524426a08e138e39fb37f14b
2019-04-16 21:34:10 +00:00
Smruti Soumitra Khuntia d12aa27712 End user logging for audit traceabilty
Changes for Client to support new end user header
and add end user name to logs.

Change-Id: Iea1e42eafa573960735415ce337a1558b864edfc
2019-04-01 10:07:34 +00:00
Jared Miller 4a9f9d7ab5 Disable tgtd service on maas-rack-controller
tgt service is exposed in maas-rack-controller as a root-owned process
We are not utilizing tgt so let's just disable it.

Needed for this: https://review.openstack.org/#/c/639432/

Change-Id: I89da1c033baa72ac9c2755af9eeff6b8cce47517
2019-03-15 15:16:04 -04:00
Scott Hussey 7504c2f907 (fix) Support non-present networks
- Drydock should support defined networks that MAAS cannot
  see. This fixes an issue that caused this use-case to fail
  by ensuring that the MAAS client models with no internal
  resource IDs work.

Change-Id: I1a20d4730e94eee7268ff0cc3451e4b459a1e62b
2019-02-19 11:26:08 -06:00
Roman Gorshunov 02ae2bf0ee Fix: flake8 tests failing due to invalid types comparison
Change-Id: I7266c0740652da40858a5a34cec73452ed0e10e6
2019-02-11 18:16:12 +01:00
Scott Hussey 0428e30251 Support MAAS having dead rack controllers
- Ensure that picking a rack controller only uses
  healthy controllers

Change-Id: I23b1955d270d3b3b51e7043e5d96dc5018b8b25f
2019-02-04 15:56:42 -06:00
pd2839 d93d6d5a0a Added line no to drydock message format for troubeshooting
Change-Id: I88c8cd6a56771cf6e97cfa874dc63963e247a890
2019-01-08 20:41:45 +00:00
zhouxinyong 197a1e082f Remove the repetition words in readme.md
Change-Id: I79c024e432292aad0a0cbc1218699909d5af9432
2019-01-04 20:17:03 +08:00
Zuul c76fda6ac4 Merge "Validate mountpoint in a HostProfile" 2018-12-26 15:28:40 +00:00
Zuul c39a4ede1a Merge "Add Redfish as OOB driver" 2018-12-21 14:36:22 +00:00
Scott Hussey 54ea0e1374 Workaround MAAS race condition
- If two threads concurrently update the power parameters
  of VMs on the same libvirt host, it seems to leave MAAS
  with broken state. Add locking in Drydock so that Drydock
  does not do this concurrently.

Change-Id: I85575f4ba48152b4dc79c646871f33b69f845ab9
2018-12-17 11:29:52 -06:00
Hemanth Nakkina da0c7e831e Add Redfish as OOB driver
This patch implements Refish as new OOB driver for Drydock.
All the existing Drydock Orchestrator actions are implemented.

Change-Id: I31d653fb41189a18c34cfafb0f490ca4f4d661b5
2018-12-08 21:19:30 +05:30
Zuul 485f919822 Merge "(fix) Use endpoint for MAAS URL" 2018-12-04 14:25:20 +00:00
Scott Hussey a2418241ce (fix) Use endpoint for MAAS URL
- Instead of forcing a user to provide the full URL for the MAAS API,
  instead use the endpoints pattern and render the URI via HTK templates.
- Add secret name to chart to support HTK ingress
- Install libyaml to take advantage of faster parsing by pyyaml
- Add exception logging when node compiling fails.
- Add caching of parsed design to gain efficiency
- Add TLS certificate secret for use by the ingress document

Change-Id: I5a2dbc415483c336d38d67edcebdfc5812f7bb0c
2018-11-27 12:04:38 -06:00
melissaml 3470f89519 omit the twice occured words in orchestrator.py
Change-Id: I50fb7b96cb0ee6bee670fd3b85007e99ef0aabc9
2018-11-14 17:57:43 +08:00
zhouxinyong 8e8b7d6c7b delete the duplicate words in networkLink3.yaml
Change-Id: I21315ec670cc6f4d68e6dfef41e9676541cde47a
2018-11-13 09:58:59 +08:00
zhouxinyong 314921f7c9 remove the repetition words in fullsite_networks.yaml
Change-Id: I2abb2bed6dc418bcf23c7a0c1dbaed027763ca37
2018-11-13 09:56:42 +08:00
zhouxinyong d73f6cbf73 remove the repetition words in multidoc.yaml
Change-Id: Ida718c82ca57eb7b995019fc7aa7676201d74a04
2018-11-13 09:55:09 +08:00