Upon investigation, we have identified an opportunity to enhance the
messaging within the sysinv.log file.
Currently, during the initialization process, DNSMasq is initiated
following the readiness of the management_ip (IPv4/IPv6) and the
file system.
However, during the initial unlock phase following a fresh install,
we observed delays in certain services receiving DNS query results
despite DNSMasq being operational.
This issue predominantly manifests during the first unlock
post-installation, affecting services such as sysinv-api and
sysinv-conductor.
Consequently, error messages are logged in the sysinv.log as these
services encounter difficulties.
Fortunately, this issue does not adversely impact system functionality
as the SM effectively restarts these services.
To mitigate these error messages and ensure smoother operation,
we've introduced a new dependency within the SM.
Now, the sysinv-api service will commence only after DNSMasq is
operational, thereby reducing the likelihood of DNS-related issues
during startup.
However, due to constraints related to host-swact, a similar adjustment
cannot be made for sysinv-conductor at this time.
Additionally, we refined the DNSMasq start script to return only upon
successful resolution of a DNS query, streamlining its initialization
process.
Test Plan
AIO-SX fresh install
AIO-DX fresh install
AIO-DX host-swact
Story: 2010722
Task: 50220
Change-Id: I546a85861415d9b12c9073625ca5d2db6ebfa2e6
Signed-off-by: Fabiano Correa Mercer <fabiano.correamercer@windriver.com>
This commit adds support for creating and deleting a controller
filesystem. At the moment, only the creation of the controller
fs 'ceph-float' is allowed, used to establish ceph-specific
storage on controllers for Rook Ceph support.
The states below were introduced:
- 'drbd_fs_creating_in_progress': Status when creation is in
progress. The standby controller must be locked when using
the fs command.
- 'drbd_fs_creating_on_unlock': Status when using the create fs
command after bootstrap and before the first unlock of
controller-0.
- 'drbd_fs_deleting_in_progress': Status when deletion is in
progress. The standby controller must be locked when using
the fs command.
- 'drbd_fs_update_error': Status that indicates that there was a
failure in creation/deletion, with the possibility of retry
to be successful.
* A new alarm was added to FM for controller-fs: 800.105.
Test Plan:
PASS: AIO-SX / AIO-DX / Standard -> fresh install with Ceph Bare
Metal using designer build with topic changes + Check that
there is no interference or errors.
PASS: AIO-DX -> Standby controller locked and ceph-rook as
storage-backend + controller-fs add ceph-float=<size> +
checking if everything is created correctly: lv, drbd and
SM services.
PASS: AIO-DX -> After bootstrap, add ceph-rook as storage backend
+ use controller-fs add ceph=<size> + check if controller_fs
went to creation state on unlock + continue installation and
check if after unlock controller-1 if everything is created
correctly.
PASS: AIO-DX -> with the ceph filesystem created, modify (resize)
the new filesystem and some of the default ones, checking
that is working properly.
PASS: AIO-DX -> Lock/unlock + swact tests.
PASS: AIO-DX -> Standby controller locked + controllerfs-delete
ceph + checking if everything is deleted correctly: lv, drbd
and SM services.
PASS: Force operation to go to the state "drbd_fs_update_error" +
Check if the alarm 800.105 is raised + retry the command that
failed + Verify that the execution was successful and the
alarm was cleared.
Depends-On: https://review.opendev.org/c/starlingx/stx-puppet/+/919078
Story: 2011117
Task: 50073
Change-Id: I57dd9669ad8cddea81ec0692cd11435dae27dce4
Co-Authored-By: Robert Church <robert.church@windriver.com>
Signed-off-by: Gabriel de Araújo Cabral <gabriel.cabral@windriver.com>
This commit introduces changes to make ceph-mon API available
only for systems that have Ceph as the storage backend.
Test Plan:
PASS: AIO-SX/AIO-DX/Standard -> Fresh install with Ceph bare
metal.
PASS: In the above systems, Check that all ceph-mon commands do
not have restrictions on the system's storage backend and
are working properly.
PASS: AIO-DX -> Fresh install with Rook Ceph as storage-backend
+ Check if the restriction is present when attempting to
create a ceph-mon.
PASS: AIO-DX -> Fresh install without any storage backend +
Check if the restriction is present when attempting to
create a ceph-mon.
Story: 2011117
Task: 50098
Change-Id: I700cee85f574869e87c091fb3b9a478a3b569e0a
Signed-off-by: Gabriel de Araújo Cabral <gabriel.cabral@windriver.com>
This update made various improvments to the ipsec-config's ocf script
to make it more resillient to error conditions. Particularly when the
swanctl.conf points to the wrong version of config file, the
ipsec-config service will correct it based on whether the host is
active or standby controller.
Test Plan:
PASS: In a DX system, swact controllers back and forth, verify after
swact, swanctl.conf points to the correct version of swanctl
config file, IPsec SAs established, both controllers are in
unlocked|enabled|available state.
PASS: In a DX system, lock standby controller, then force reboot active
controller by "reboot -f", verify that the swanctl.conf points to
the swanct_standby.conf when it boots up, and then it is
corrected to point to swanctl_active.conf by ipsec-config when SM
services start up, also IPsec SAs are established between
controllers. Eventually both controllers are in
unlocked|enabled|available, system is stable with no reboot.
PASS: In a DX system, reboot both controllers, verify that the
swanctl.conf points to the swanct_standby.conf when they boot up,
then the symlink on active controller is corrected by
ipsec-config when SM services start up, also IPsec SAs are
established between controllers. Eventually both controllers are
in unlocked|enabled|available, system is stable with no reboot.
PASS: In a DX system, manually link a wrong config file to
swanctl.conf, verify that the ipsec-config service correct it.
Story: 2010940
Task: 50198
Change-Id: I9fc028eb7a0ed296f9dd47b92c2b53302e845ac3
Signed-off-by: Andy Ning <andy.ning@windriver.com>
Namespace not found error message is shown in sysinv.log when an application that has a specific namespace is uploaded.
This log is expected and has no impact on the system. This
happens due to the command "kubectl -k <manifest_dir>
--dry-run=server". If the application has a specific namespace that
the system does not yet have, the command shows this error. But
there is nothing to worry about, as the namespace is created within
the apply process.
To avoid misinterpretations, this log has been suppressed.
Test Plan:
PASS: Add the new log to kube_app.py and restart sysinv.
PASS: During the application upload process, the log that says the
namespace was not found no longer appears.
Closes-Bug: 2067070
Change-Id: I6e56c338c9789c686381c330be65ec109a3f0b50
Signed-off-by: David Bastos <david.barbosabastos@windriver.com>
In STX9, python is upgraded to 3.9. Yaml load function requires specific
loader to deal with 'tuple' type. This fix is to replace all
yaml.safe_load(f) with yaml.load(f, Loader=yaml.FullLoader) as per [1].
[1] https://github.com/yaml/pyyaml/blob/main/lib/yaml/loader.py#L51
TCs:
Passed upgrade DC system controller.
Passed upgrade standalone DX system.
Closes Bug: 2045695
Change-Id: Ic5a0a51163b5bdd4e5d2c3c31c8dda4ab0de06e1
Signed-off-by: Bin Qian <bin.qian@windriver.com>
Cgtsclient uses post method of request module to upload files and
does not use, verify and cert options of post method. This cause
failure when doing insecure upload like in license-install command.
This change updates cgtsclient to use verify and cert options of
post method, when using upload commands.
Test Plan:
PASS: Verify upload with SessionClient using verify & cert options
PASS: Verify upload with HTTPClient using verify & cert options
PASS: Verify insecure, ca-file, cert-file & key-file in cgtsclient
Closes-Bug: 2067447
Change-Id: Icb9aca3d69ddd2e8b7882f0ce6907f18feccd28e
Signed-off-by: Joseph Vazhappilly <joseph.vazhappillypaily@windriver.com>
The controller_config init script is updated to symlink swanctl.conf to
the standby controller version of IPsec config file during reboot.
(swanctl_standby.conf). This makes the symlink correct for only one
controller reboot case (eg, active controller forcely reboot, but when
it comes up it will be standby controller). For rare cases such as
active controller reboot while standby controller is locked, the
symlink on the active controller will be corrected by ipsec-config SM
service when both controllers boots up.
Test Plan:
PASS: In a DX system, force reboot active controller by "reboot -f",
during the controller reboot, verify IPsec SAs are established
and puppet manifests are successfully applied. After the
controller boots up, verify it's unlocked|enabled|available, and
stable with no reboot.
PASS: In a DX system, lock standby controller, then force reboot active
controller by "reboot -f", verify that the swanctl.conf points to
the swanct_standby.conf when it boots up, and then it is
corrected to point to swanctl_active.conf by ipsec-config when SM
services start up, also IPsec SAs are established between
controllers. Eventually both controllers are in
unlocked|enabled|available, system is stable with no reboot.
Story: 2010940
Task: 50195
Change-Id: I1e860b4b4f0f75ff2f8bde81d612d70cb80fb312
Signed-off-by: Andy Ning <andy.ning@windriver.com>
This commit fixes the support for StarlingX applications that do not
provide any plugins, also known as generic apps. Two changes were
required to fix that regression:
* Check if the app is generic when retrieving its charts. A previous
commit [1] introduced a check to verify whether charts are enabled,
which caused false negatives because generic apps do not provide
plugins to tell if they are enabled or disabled. In light of that,
all charts provided by generic apps should be considered as enabled.
* Swap the order that the "_get_list_of_charts" and
"generate_helm_application_overrides" methods are called during the
application apply process. The call to
"generate_helm_application_overrides" requires application charts to
be passed as a parameter, so "_get_list_of_charts" should be called
first.
In addition, an error log message was added to cover the scenario where
no application charts are found. That will provide better information
for developers that eventually makes mistakes when structuring their
apps. Otherwise, the apply process would silently abort with no clear
error.
[1] 967eedadb7
Test Plan:
PASS: build-pkgs -a && build-image.
PASS: AIO-SX fresh install.
PASS: Upload/apply/remove/delete generic app.
PASS: Upload dell-storage app.
Enable csm-replication replication chart.
Apply user overrides.
Apply dell-storage.
Confirm that the enabled chart was installed.
Remove/Delete dell-storage.
Closes-bug: 2067430
Change-Id: Ib6fbd53209e19b12c5dbedc3584e04f8261363f4
Signed-off-by: Igor Soares <Igor.PiresSoares@windriver.com>
Convert keystone database migration to a migration script at major
release deploy activate, to replace the logic in [1].
[1]: https://opendev.org/starlingx/config/src/branch/master/controllerconfig/controllerconfig/controllerconfig/upgrades/management.py#L340-L354
Story: 2010676
Task: 50081
TCs:
Passed: USM major release deploy activate completed successfully
Passed: USM major release deploy activate failed
Passed: USM major release deploy activate after activate failed
Change-Id: Ie6529733c2db231db03d7253a8a716cb0899fd8a
Signed-off-by: Bin Qian <bin.qian@windriver.com>
Region name was not being correctly updated to match correct value.
This lead to the VIM services having the wrong region name and failing.
TEST PLAN
PASS: On SX subcloud (with UUID region name), verify VIM config for
correct region names
Story: 2011045
Task: 50208
Change-Id: I896b1bf8898886ee5962bb4cce6e955a70a91ff7
Signed-off-by: Joshua Kraitberg <joshua.kraitberg@windriver.com>
This commit establishes stricter rules for validating Kubernetes
version fields in applications' metadata files. The general rule that
such fields should be strings was maintained. Now, in addition to that,
they should follow specific string formats. This targets both "minimum"
and "maximum" fields under the "supported_k8s_version" section.
Allowed formats and examples for "supported_k8s_version:minimum" and
"supported_k8s_version:maximum" fields:
* major.minor.patch (e.g. 1.29.2)
* vmajor.minor.patch (e.g. v1.29.2)
* vmajor.minor (e.g. v1.29 - this is interpreted as 1.29.0)
Examples of unsupported formats:
* major (e.g. 1)
* vmajor (e.g. v1)
* major.minor (e.g. 1.29)
* major.minor. (e.g. 1.29.)
* vmajor.minor. (e.g. v1.29.)
Test Plan
PASS: build-pkgs -a && build-image
PASS: AIO-SX fresh install
PASS: Upload/apply platform-integ-apps.
Confirm that no errors were reported.
Check if default values were correctly saved to the database.
PASS: Edit platform-integ-apps metadata to contain
"supported_k8s_version:minimum: 1.24.4" and
"supported_k8s_version:maximum: 1.29.2".
Upload/apply platform-integ-apps.
Confirm that no errors were reported.
Check if values were correctly saved to the database.
PASS: Edit platform-integ-apps metadata to contain
"supported_k8s_version:minimum: v1.24.4" and
"supported_k8s_version:maximum: v1.29.2".
Upload/apply platform-integ-apps.
Confirm that no errors were reported.
Check if values were correctly saved to the database.
PASS: Edit platform-integ-apps metadata to contain
"supported_k8s_version:minimum: v1.24" and
"supported_k8s_version:maximum: v1.29".
Upload/apply platform-integ-apps.
Confirm that no errors were reported.
Check if values were saved as 1.24.0 and 1.29.0 in the database.
PASS: Edit platform-integ-apps metadata to contain
"supported_k8s_version:minimum: 1.24" and
"supported_k8s_version:maximum: 1.29".
Upload/apply platform-integ-apps.
Confirm that the upload was rejected.
PASS: Edit platform-integ-apps metadata to contain
"supported_k8s_version:minimum: 1.24." and
"supported_k8s_version:maximum: 1.29.".
Upload platform platform-integ-apps.
Confirm that the upload was rejected.
PASS: Edit platform-integ-apps metadata to contain
"supported_k8s_version:minimum: 1.24.4.." and
"supported_k8s_version:maximum: 1.29.2.".
Confirm that the upload was rejected.
PASS: Edit platform-integ-apps metadata to contain
"supported_k8s_version:minimum: v1" and
"supported_k8s_version:maximum: v2".
Confirm that the upload was rejected.
PASS: Edit platform-integ-apps metadata to contain
"supported_k8s_version:minimum: 1" and
"supported_k8s_version:maximum: 2".
Confirm that the upload was rejected.
Story: 2010929
Task: 50179
Change-Id: I1acb6706c11f12d2eeb6ea855e69d332192a805d
Signed-off-by: Igor Soares <Igor.PiresSoares@windriver.com>
This commit is an addition to the proxy API [1], to added semantic check
for USM upgrade to restrict host-swact and host-unlock.
This commit also removed the check that prevents upgrade and downgrade
when new hardware is detected during upgrade procedure.
Test Plan:
PASS: run the upgrade on DX with USM, observed the restrictions take place
Task: 49798
Story: 2010676
[1] https://review.opendev.org/c/starlingx/config/+/914974
Change-Id: I8ded9faf7691ce849d51ef39f7598f287c6f1ca4
Signed-off-by: Bin Qian <bin.qian@windriver.com>
Introducing a new service-parameter for password rules in
security_compliance section of /etc/keystone/keystone.conf
New service parameters :
| service | section | name
identity security_compliance password_expires_days
The 'password_expires_days' will allow user to custom the expiry period
of the password in keystone users.
In order for changes to take effect, the new configuration needs to be
applied with:
system service-parameter-apply identity
Test Plan:
PASS: Run full deploy of an .iso.
PASS: Verify that 'password_expires_days' works with an integer.
PASS: Verify that 'password_expires_days' show up in
'system service-parameter-list' in section identity.
PASS: Verify that 'password_expires_days' is modified with
'system service-parameter-apply'.
Story: 2011084
Task: 49824
Depends-On: https://review.opendev.org/c/starlingx/stx-puppet/+/920056
Change-Id: I758208b28dddb1127bcc496ae1bb6907ebc2d125
Signed-off-by: Karla Felix <karla.karolinenogueirafelix@windriver.com>
This commit removes the mgmt_ip field from the database i_host table.
The mgmt_ip is necessary for MTCE and other calls, this value is
returned by the database now, using the address name and network type
by the function: get_address_by_host_networktype.
Test Plan:
PASS: AIO-SX installation
PASS: AIO-SX mgmt reconfiguration
PASS: AIO-DX installation
PASS: Standard installation
PASS: DC install
PASS: DC AIO-SX mgmt reconfiguration
PASS: Sanity for AIO-SX, AIO-DX, DC
Story: 2010722
Task: 49835
Signed-off-by: Teresa Ho <teresa.ho@windriver.com>
Change-Id: I577fa51df261846292f2e9dff2aeba5fff431848
After executing "system oam-modify oam_ip=<x.x.x.x>" command,
"updated_at" field is always "None" and is not updating.
This commit contain the code changes to update the sysninv database
where it populates the "updated_at" column in networks table
during "system oam-modify" command.
Test Plan:
PASS: Developer iso is successfully deployed in Simplex VM with
current code changes.
PASS: Validated "updated_at" field after executing "system oam-modify"
command."updated_at" field is updating correctly.
PASS: Validated "system oam-show" command to check the "updated_at"
filed is correctly displayed.
PASS: Validated the networks table where the "updated_at" column
is correctly populated.
Sample output:
[sysadmin@controller-0 ~(keystone_admin)]$ system oam-show
+----------------+--------------------------------------+
| Property | Value |
+----------------+--------------------------------------+
| created_at | 2024-05-20T02:24:02.684818+00:00 |
| isystem_uuid | 30e6d1f3-bc0e-4b59-9533-8edc061e7c63 |
| oam_gateway_ip | 10.20.7.1 |
| oam_ip | 10.20.7.7 |
| oam_subnet | 10.20.7.0/24 |
| updated_at | 2024-05-20T03:20:46.853689+00:00 |
| uuid | dc04244e-ee74-4003-9293-2e1ded380a05 |
+----------------+--------------------------------------+
Kindly refer launchpad bug for details.
Closes-Bug: #2066156
Change-Id: I529fbcdc0fb1530add7e8f2ddd703f37badd71da
Signed-off-by: sshaikh1 <sirin.shaikh@windriver.com>
Introduce an additional check to block Kubernetes upgrades on the
kube-upgrade-start step if applied apps marked to be post updated are
incompatible with the target k8s version. This will prevent
incompatibility gaps in the upgrade process, i.e. avoiding that the
post-update stage is reached while running an incompatible app.
Application developers can leverage the auto update feature and the
pre-update step in case incompatible apps need to be updated before
deploying a new Kubernetes version.
Test Plan:
PASS: build-pkgs -a && build-image.
PASS: Deploy platform with Kubernetes 1.24.
Apply a metrics-server version containing
supported_k8s_version:minimum: 1.24 and
supported_k8s_version:maximum: 1.27.
Add a new metrics-server bundle containing
supported_k8s_version:minimum: 1.28 and
supported_k8s_version:maximum: 1.29.
Start Kubernetes upgrade to 1.28.
Confirm that the upgrade was blocked.
PASS: Deploy platform with Kubernetes 1.24.
Apply a metrics-server version containing
supported_k8s_version:minimum: 1.24 and
supported_k8s_version:maximum: 1.28.
Add a new metrics-server bundle containing
supported_k8s_version:minimum: 1.28 and
supported_k8s_version:maximum: 1.29.
Start Kubernetes upgrade to 1.28.
Confirm that metrics-server is updated during post-update step.
Closes-Bug: 2066042
Change-Id: Ie5d7e872ae49f833bf1cf1a4eb80a7f50dad8920
Signed-off-by: Igor Soares <Igor.PiresSoares@windriver.com>
Starting on stx-11, patching will be managed by the
USM feature, so all patch related health-checks and
information should be gathered via USM APIs.
This commit changes the patch current health checks
to point to USM APIs. However, there is not an endpoint
that returns per-host data, and deploy host-list only
returns data if there is a deployment in progress, so
there will be a followup commit to change this logic
again if an endpoint is created to return data as the
legacy sw-patch query-hosts command.
This commit does not affect legacy upgrades, since
on legacy upgrade the health-check code runs on the
FROM release code base.
Test Plan
PASS: run 'deploy precheck' and verify the output
PASS: run 'deploy precheck' on a system with [Fail]
patch current health check, apply the fix and
verify the output
Story: 2010676
Task: 50107
Signed-off-by: Heitor Matsui <heitorvieira.matsui@windriver.com>
Change-Id: I69c630468543ac99d8ba79f21646573a91140616
This commit creates the update_users, create_roles
and create_projects methods in
openstack_config_endpoints.py, which is required in [1]
in order to set the required services, roles and users
during keystone bootstrap.
[1]: https://review.opendev.org/c/starlingx/ansible-playbooks/+/915284
Test plan:
Note that all of the test cases were performed with the changes from
[1].
1. PASS: Verify the services, roles and users were created after
the keystone bootstrap.
2. PASS: Validate that the admin and sysinv users have the
ignore_lockout_failure_attempts set to true.
3. PASS: Validate the sql dump of the keystone database generated in
a subcloud deployment in relation to the one generated before the
changes from [1].
Story: 2011035
Task: 49966
Change-Id: I5be50bec1174a451d11e4dbc2eff0b01fc182576
Signed-off-by: Raphael Lima <Raphael.Lima@windriver.com>
Some of the k8s Opaque type secrets data has None values, this
is failing the "system k8s-certificate-list" command.This fix
addresses this issue by checking the "data" field in the
secret
Test Cases:
Pass: Have a secret with no data, run "system k8s-certificate-
list" comamnd and verify it is listing the certificates
Closes-bug: 2065926
Change-Id: I85ea4341d71016c6064b22860f761b3f77f1619c
Signed-off-by: amantri <ayyappa.mantri@windriver.com>
Script introduced by [1] broke USM deploy start due to not
parsing the fourth parameter passed to the scripts during
the data migration.
This commit fixes the issue.
[1] https://review.opendev.org/c/starlingx/config/+/919624
Test Plan
PASS: execute deploy start successfully
Closes-bug: 2065910
Change-Id: I03a9e055bc487b423c385ee7fb69fc31396ac734
Signed-off-by: Heitor Matsui <heitorvieira.matsui@windriver.com>
This change added ipsec-auth client invocation in controller_config,
worker_config and storage_config init scripts that will run during
first reboot after installation, to configure and enable IPsec for the
node.
Note that IPsec for the first controller is configured and enabled by
bootstrap ansible playbook. So the invocation of ipsec-client is
skipped in controller_config.
Test Plan:
PASS: DX system, install controller-0, bootstrap and unlock, verify
IPsec is configured and enabled.
PASS: Install controller-1, verify IPsec is configured and enabled
after first reboot, SAs are established, and controller-1 is
online.
PASS: Install a worker node, verify IPsec is configured and enabled
after first reboot, SAs are establishe, and the worker node is
online.
PASS: After controller-1 and worker hosts are unlocked, verify SAs are
established among all hosts, and all nodes are in unlocked,
enabled and available states.
PASS: DC system with SX subcloud, verify System Controller and subcloud
are deployed successfully. In central cloud, SAs are established
among all hosts, all nodes are in unlocked, enabled and available
states.
Verify subcloud are online, managed, and all resource are in
in-sync states.
Verfiy user can ssh to subcloud.
Story: 2010940
Task: 50021
Depends-On: https://review.opendev.org/c/starlingx/ansible-playbooks/+/917868
Signed-off-by: Andy Ning <andy.ning@windriver.com>
Change-Id: I5572b4b50238c0c5e76cc04cabd24078e9defa5b
This commit updates various codes related to the "protecting mgmt
network with IPsec" feature, to support stx 9 to stx 10 upgrade. With
this changeset, the IPsec feature will not break current upgrade. There
will be follow-up changes to fully support stx 9 to stx 10 upgrade with
the IPsec feature.
Test Plan: (AIO DX system)
PASS: Packages build, image build.
PASS: Without IPsec feature enabled, upgrade from stx 9 to stx 10,
verify all steps are successful.
PASS: With Ipsec feature enabled, upgrade from stx 9 to stx 10, verify
all steps are successful.
(IPsec is not configured and enabled during upgrade)
Story: 2010940
Task: 50095
Change-Id: Id708f540f39ed98a9e05cc12a71a5a4fa3d5bfa5
Signed-off-by: Andy Ning <andy.ning@windriver.com>
Zuul