Commit Graph

619 Commits

Author SHA1 Message Date
Zuul 3c3a3ecb62 Merge "Insert ostree commit id into metadata" 2024-03-26 16:15:07 +00:00
Zuul a159c6ee62 Merge "Include missing data migration steps on deploy start" 2024-03-26 15:57:08 +00:00
Zuul f99c9d28c5 Merge "Restrict software commands to keystone admin role" 2024-03-26 13:25:30 +00:00
Luis Eduardo Bonatti a6c5490d1a Fix deploy precheck returning blank error message
This commit fix the deploy precheck <release> command output which
is returning one line with an blank "Error:" when the output is correct.

Test Plan:
PASS: Deploy precheck <release> returning without error.
PASS: Deploy precheck <release> returning expected error.
PASS: Deploy start failed with expected error.
PASS: Deploy start

Note: Exit code 3 was chosen based on exit codes with special meaning
which the 1 is for catchall general errors and 2 for misuse of shell
builtins. The number 3 is not allocated with special meaning so it
was chosen for unhealthy precheck.

Closes-Bug: 2056106

Change-Id: Ifed48157f7810eec2881d8a9e011eae8941f3427
Signed-off-by: Luis Eduardo Bonatti <LuizEduardo.Bonatti@windriver.com>
2024-03-25 09:48:57 -03:00
Joseph Vazhappilly 169b84bc6c Restrict software commands to keystone admin role
When using Keystone auth for software cli, only user with 'admin' role
is allowed to run any commands. When using software cli without 'sudo',
all software commands require user with 'admin' role.

This review also update the exception handling and error reporting.

Test Plan:
PASS: A Keystone user in the 'admin' project with 'admin' role should
      be able to run ALL 'software' commands WITHOUT SUDO
PASS: A Keystone user in the 'admin' project with only 'member' and/or
      'reader' role should NOT be able to run ANY 'software' commands
      WITHOUT SUDO

Story: 2010676
Task: 49754

Change-Id: I46653021b1a82bccded5eb870dc0907cd5c2351b
Signed-off-by: Joseph Vazhappilly <joseph.vazhappillypaily@windriver.com>
2024-03-25 03:24:32 -04:00
Zuul 6485390ce1 Merge "Implement versioned deploy precheck script" 2024-03-22 20:14:21 +00:00
Heitor Matsui f88638a374 Include missing data migration steps on deploy start
There are additional steps needed to enable "deploy host"
operation for major release upgrades. This commit adds these
additional steps, which are:

1. Create TO release platform config directory, done during
   upgrade-start on legacy upgrade procedure
2. Create TO release rabbitmq directory, done by upgrade
   manifest after controller-1 reinstall on legacy upgrade

The commit also fixes some issues:

1. shell-utils logging functions logging nowhere after
   being sourced by other scripts
2. sync-controllers-feed script was only syncing the
   ostree_repo directory instead of the full feed content
3. major release deployment scripts ran from different
   places, now all scripts are executed from the TO release
   feed directory, or from checked out TO release ostree
   repo in case of chroot
4. change umount command from chroot_mounts to lazy umount

Note: Since the "deploy host" endpoint for major release
deployment is not yet implemented, the test plan will have
test cases that simulate the "deploy host" operation.

Test Plan
PASS: simulate "deploy host" successfully for AIO-SX
PASS: simulate "deploy host" successfully for AIO-DX

Depends-on: https://review.opendev.org/c/starlingx/config/+/913715

Story: 2010676
Task: 49703

Change-Id: Ib6ae49b3590a1e50acb305ac7482e28bcc4de403
Signed-off-by: Heitor Matsui <heitorvieira.matsui@windriver.com>
2024-03-22 15:12:14 -03:00
Charles Short 94bff293cf Insert ostree commit id into metadata
Insert ostree commit id into metadata file after apt-ostree
runs. "previous_commit" tag is for commit id before apt-ostree
is executed and "commit" tag is for latest commit id after it
is executed.

Test Plan:
PASS: Run software deploy start for patch and check commit id
      is inserted in metadata file

Story: 2010676
Task: 49753
Change-Id: I8fc3c33430188449d852770824e3ecd765583dd7
Signed-off-by: Charles Short <charles.short@windriver.com>
Signed-off-by: sshathee <shunmugam.shatheesh@windriver.com>
2024-03-22 01:59:35 -04:00
junfeng-li 1d78ce4cec Fix upload-dir respond missing info
This commit is to fix 'software upload-dir' not having
respond that contains uploaded release info.

Test Plan:

PASS:  upload files using 'software upload-dir'

Task: 49634
Story: 2010676
Change-Id: I635554fdbdb80fe31a38d1170202405fe6f32d3a
Signed-off-by: junfeng-li <junfeng.li@windriver.com>
2024-03-21 19:51:44 +00:00
Zuul b0afdd45f1 Merge "Add deletion constraint" 2024-03-21 17:10:04 +00:00
Zuul e2a0423bbd Merge "Fix: load-import not feed controller-1" 2024-03-21 13:41:46 +00:00
Bin Qian 229acb985f Add deletion constraint
This change adds checks before deleting software releases:
1. software release is available or unavailable
2. When it is on a system controller, the release is not being used by a
   subcloud
This change also update the following:
1. removed the exception handling in controller level, moved to
   exception hook
2. CLI code to display HTTP error, only handles 500 status code, by
   displaying message from API, all other 4xx, 5xx status code display
   HTTP error directly.
3. ensure CLI return 1 for unsuccessful requets (status code 500)
4. fixed some minor issues

Story: 2010676
Task: 49657

TCs:
     passed: observe delection rejected because of release not found,
release is not in available or unavailable state.
     passed: delete an available release
     passed: on system controller, successfully delete scenarios
     passed: (simulated) on system controller with subcloud, delete
     release used by subcloud is rejected

Change-Id: I306b1d8604113b92d907384844e8e8107835a463
Signed-off-by: Bin Qian <bin.qian@windriver.com>
2024-03-20 22:41:48 +00:00
Heitor Matsui 2c27382eba Fix upload-dir logic, return and help text
Currently upload-dir logic was uploading only the files from
the last directory passed as argument to the command, and it's
help text and output was divergent from the similar "software
upload" command.

This commit fixes the logic, allowing uploading from multiple
directories correctly, and fixes the help text and output to
align with "software upload".

Test Plan
PASS: show help text for "software upload-dir"
PASS: upload one directory containing patches
PASS: upload one directory containing iso + patches
PASS: upload multiple directories containing patches
PASS: upload multiple directories containing iso + patches

Story: 2010676
Task: 49693

Change-Id: I5886e8c5c55355e24ec471c4ae47e91ec3c84dfd
Signed-off-by: Heitor Matsui <heitorvieira.matsui@windriver.com>
2024-03-19 21:28:30 -03:00
Zuul 75de9f16ff Merge "Fix deploy host for in-service patch" 2024-03-19 18:04:59 +00:00
Lindley Werner c8a7941a21 Fix: load-import not feed controller-1
When doing load-import operation, the /var/www/pages/feed/ files must
be replicated in both controllers. The files were being created in
controller-0, but they were not copied to controller-1.

This fix added a way to get all the folders from one controller to
another and sync them with ostree the explanation is in the following
commit: I42c274079631a3c197015e636e03de1bc96de28b

Test-plan:
PASS: After a load-import, the inactive controller should have the
feed repo created.

Closes-bug: 2045321

Change-Id: I260951461d2c19550e9f57ad7ab9ec66a25de5bb
Signed-off-by: Lindley Werner <Lindley.Vieira@windriver.com>
2024-03-19 14:53:39 -03:00
Lindley Werner 831d3d3e93 Fix deploy host for in-service patch
'software deploy host <in-service-patch-release>' should NOT fail if
there is no restart-script (post-deploy script).
If the rsync does not find the scripts, does not raise an exception.

Test-plan:
PASS: software deploy host <in-service-patch-release> successful

Closes-bug: 2058393

Change-Id: I1b8cac9e0401c3f64c7334139c62bf272e9aeb56
Signed-off-by: Lindley Werner <Lindley.Vieira@windriver.com>
2024-03-19 11:40:34 -03:00
Luis Eduardo Bonatti 32250d9b59 Implement versioned deploy precheck script
This commit is to include a versioned deploy precheck script into
/opt/software/scripts/rel-<sw_version> to be able to run the correct
precheck code for a specific release.

Along with this commit, the precheck api is changed to use the new
location of the script, and the precheck script is changed to add
support to patch-only prechecks, and as a consequence, minor wording
changes were done to return more accurate messages to the user.

1. For the iso upload scenario:
The upload process will copy all scripts under
<iso_root>/upgrades/software-deploy to /opt/software/rel-<ver>/scripts

2. For the patch upload scenario:
The upload process will check if patch contains the deploy-precheck
script. If it does, then the script is copied to
/opt/software/rel-<ver>/scripts, if not then a symlink will be created
to the patch 'required patch' versioned precheck script.

Notes:
- iso (prepatched or not) will always come with deploy-precheck script
- <ver> assumes the format MM.mm.pp

Test Plan:
PASS: Upload multiples patches, both with and without precheck
      scripts, and verify the versioned directories are created
      and the precheck script is created as expected
PASS: Run deploy precheck for an iso release and verify the upgrade
      precheck output is returned as expected
PASS: Run deploy precheck for a patch release and verify the patch
      precheck output is returned as expected

Depends-on: https://review.opendev.org/c/starlingx/metal/+/911595

Story: 2010676
Task: 49263

Change-Id: I04ff89d43579fd71592f7ec534db57a1ead79483
Signed-off-by: Luis Eduardo Bonatti <LuizEduardo.Bonatti@windriver.com>
Co-signed-off-by: Heitor Matsui <heitorvieira.matsui@windriver.com>
2024-03-14 09:41:28 -03:00
Joseph Vazhappilly 0cd1d59425 Add support for https in USM software client
This change add support for https with SSL protocol and certificate.
The USM client can work with either insecure (disable SSL/TLS
certificate verification) or with SSL certificate. The client is
also modified to support sessions and versions. These changes are
adapted from cgtsclient.

This adds three authorization modes, [token, keystone & local-root].

In token mode, a keystone token and software-url is used for auth.
Eg: $ software \
        --software-url "http://192.168.204.1:5497" \
        --os-auth-token "${TOKEN}" list

In keystone mode, sourced keystone configs in env is used for auth.
Eg: $ source /etc/platform/openrc; software list

In local-root mode, authorization is by privileged user (root/sudo)
of the controller where software application is running.
Eg: $ sudo software list

Optional arguments specific to https:
  -k, --insecure
  --cert-file CERT_FILE
  --key-file KEY_FILE
  --ca-file CA_FILE

Example usage for insecure connection:
  software -k list

Story: 2010676
Task: 49666

Test Plan:
PASS: Verify software cli output for http endpoints
PASS: Verify software cli output for https endpoints

Change-Id: I2e2ff115b8d03cddb02e026da84f389918238dab
Signed-off-by: Joseph Vazhappilly <joseph.vazhappillypaily@windriver.com>
2024-03-14 06:58:50 -04:00
Zuul 202751d57b Merge "Fix pxe files copy and permission during upload" 2024-03-13 21:11:42 +00:00
junfeng-li f2a4f93908 Deploy state periodical sync
This commit is to allow active controller periodically sending deploy
state message to the software agent on its peer controller.  The
interval is set to 30 seconds.

Test Plan:

PASS: build and deploy the iso
PASS: start new deployment, file is synced in both controllers

Task: 49655
Story: 2010676
Change-Id: Ie95c5a7d45b3d88331569ca52d64d40a4f39d6c3
Signed-off-by: junfeng-li <junfeng.li@windriver.com>
2024-03-12 17:04:19 +00:00
Heitor Matsui 720b10f07c Fix pxe files copy and permission during upload
This commit fixes an issue that was preventing load import
script from copying the TO release pxe files correctly from
the load, and also gives pxe-update-*.sh script execution
permission to avoid permission denied errors during host-unlock.

Test Plan
PASS: upload TO release load, verify the TO release pxe.cfg.linux
      files are copied to /var and that /etc/pxe-update script is
      updated to 755

Story: 2010676
Task: 49698

Signed-off-by: Heitor Matsui <heitorvieira.matsui@windriver.com>
Change-Id: I222b484d3a28c603ed8d7c42d0405481086735f0
2024-03-11 12:00:19 -03:00
Zuul 306ea5f631 Merge "Deploy host-list implementation" 2024-03-08 13:04:36 +00:00
Luis Eduardo Bonatti 378e2010fe Deploy host-list implementation
This commit add some changes on deploy host-list.
Adds a function to query the hostnames from sysinv
to deploy host-list entities during deploy start.
Changes endpoint to GET verb, the endpoint return in case
of no deployment in progress it will an empty list
and at CLI will print "No deploy in progress." In case
there is a deployment in progress the CLI will behave
the same but the endpoint will return the data below:

[{'hostname': '<hostname>',
  'software_release': '<sw_version>',
  'target_release': '<sw_version>',
  'reboot_required': 'str<true/false>',
  'host_state': '<host_deploy_state>'}]

This commit also changes the wait_for_install_complete function
to follow the new state logic.

Note: Software deploy host is affected by this change related
to states and will need a future commit regarding state changes
during deploy start and deploy host itself.

Test Plan:
PASS: Software deploy host-list with/without deployment in progress.
PASS: Deploy_host creation/update/get/delete.
PASS: Collect hostnames to deploy host entities during deploy start.

Story: 2010676
Task: 49586

Change-Id: I7b03df30fd8e326637a3ffc031e0fdf543cb6356
Signed-off-by: Luis Eduardo Bonatti <LuizEduardo.Bonatti@windriver.com>
2024-03-06 11:10:29 -03:00
Luis Eduardo Bonatti 98cd083cff Deploy show implement
This commit add some changes to deploy show endpoint, the name was
changed to just deploy with GET verb and also changes the deploy to
be saved as a list of dict to attend the API requirements. Now the
api accepts from_release and to_release as optional params, in case
it is provided the endpoint will return a dict otherwise will return
a list of dict.

Test Plan:
PASS: Create deploy
PASS: Update deploy
PASS: Software deploy start
PASS: Software deploy show

Story: 2010676
Task: 49645

Change-Id: I68d243c05da88c7eecf2d866c7202c3c7be51a2b
Signed-off-by: Luis Eduardo Bonatti <LuizEduardo.Bonatti@windriver.com>
2024-03-05 12:27:56 -03:00
junfeng-li 88e95d5c1f Fix in_sync_controller endpoint respond decode err
This is to remove the unnecessary routing based on request
method in which the respond can't be decoded properly by
the sm-api

Test Plan:

PASS: run system host-swact

Task: 49661
Story: 2010676
Change-Id: Ife8862b7d5666de3a3dafff582dbb4c27e1adafa
Signed-off-by: junfeng-li <junfeng.li@windriver.com>
2024-03-04 21:48:32 +00:00
Zuul e2112227d6 Merge "Create 2nd thread to handle USM REST API requests" 2024-03-04 20:56:55 +00:00
Hugo Brito 047555f6ff Fix constraints file in tox.ini
The constraints file used for tox.ini was removed. We need to
update the file to use the StarlingX Debian constraints file.

Test Plan:
PASS - Run tox command

Closes-bug: 2055734

Change-Id: I306be11f6edc4538cbb3f7a164bac9e1ad08501f
Signed-off-by: Hugo Brito <hugo.brito@windriver.com>
2024-03-01 19:18:51 -03:00
Bin Qian b700e3f4a1 Create 2nd thread to handle USM REST API requests
This change is to create 2nd thread to provide concurrent service. In a
different commit [1], the haproxy is to be configured to distribute the
slow requests to the 2nd thread, and the fast requests to the primiary
thread.

TCs:
    passed: concurrent keystone requests of "software upload/
    deploy precheck/deploy start" and "software list/deploy show/
    deploy host-list"
    passed: keystone authenticated "software deploy precheck"
    request completed.

Story: 2010676
Task: 49647

[1] https://review.opendev.org/c/starlingx/stx-puppet/+/910644

Change-Id: I0e8e8ac1b5177f1bbf40e047335c075b0a471fc1
Signed-off-by: Bin Qian <bin.qian@windriver.com>
2024-03-01 14:39:33 +00:00
Zuul 1fb44f6b28 Merge "Sync with peer controller deploy state changes" 2024-02-29 20:02:18 +00:00
Zuul 25420d18a8 Merge "Improve logging for deploy python scripts" 2024-02-29 13:54:56 +00:00
Heitor Matsui e414d85716 Remove duplicate load import script copy
The usm_load_import script is currently being copied to two
different places needlessly during the upload process for a
major release: (/etc/software and /opt/software/rel-<ver>/bin).

This commit keeps only the copy to the versioned directory under
/opt/software/rel-<ver> and also changes the code to use the
script from this location accordingly.

Test Plan
PASS: run 'software upload' successfully

Story: 2010676
Task: 49624

Relates-to: https://review.opendev.org/c/starlingx/update/+/910027

Change-Id: I0178007721505ff8e2c2244964692b79e8aafea9
Signed-off-by: Heitor Matsui <heitorvieira.matsui@windriver.com>
2024-02-28 17:38:51 -03:00
Zuul 557b80922e Merge "Improve logging for deploy start shell scripts" 2024-02-28 18:06:51 +00:00
Zuul 7b7d541db7 Merge "Remove invalid checks from deploy precheck" 2024-02-28 18:06:46 +00:00
Heitor Matsui af0a86e357 Improve logging for deploy start shell scripts
This commit improves logging during deploy start by:
1. Creating a common module to be sourced by shell scripts to
   load general-use functions, thus reducing code duplication
   between the scripts
2. Replacing plain "echo" commands on the scripts by logging
   functions present on the common module
3. Adding timestamps to the log messages
4. Centralizing all scripts logs into software.log, favouring
   the troubleshooting, now that log lines contain timestamps
   and the process/script that generated them

This commit also deletes the ostree_mounts.yaml file since
it would be used by apt-ostree integration, which was dropped.

Test Plan
PASS: run deploy start successfully and verify that deploy start
      log messages are logged with the expected format

Story: 2010676
Task: 49607

Change-Id: I0bdebde8147faa5b29a642e35bfaf26e9862ed0a
Signed-off-by: Heitor Matsui <heitorvieira.matsui@windriver.com>
2024-02-28 11:07:03 -03:00
Heitor Matsui 3e1485dd05 Improve logging for deploy python scripts
This commit improves logging during "deploy start" by:
1. Creating a function on upgrade_utils to configure logging
2. Replacing basic log configuration with the function from
   previous item
3. Configuring software/utilities/migrate.py module to
   effectively log in a file
4. Suppressing meaningless SQL commands output generated
   during databases migration

Test Plan
PASS: run deploy start/upload, verify the logs from the
      scripts are created as expected

Story: 2010676
Task: 49620

Depends-on: https://review.opendev.org/c/starlingx/update/+/910262

Signed-off-by: Heitor Matsui <heitorvieira.matsui@windriver.com>
Change-Id: Ib78bdcd1dd98b0f221985a429b6e9eeddeba837b
2024-02-28 11:00:50 -03:00
Luis Eduardo Bonatti cc3c1e896e Sync with peer controller deploy state changes
This commit create a generic method to be used when a deploy state
is changed. For every modification on the file it will call the message
class to send the deploy state dataset to peer controller.

Test Plan:

PASS: Deploy state change sent to peer controller on deploy start

Depends-on: https://review.opendev.org/c/starlingx/update/+/904367

Story: 2010676
Task: 49335

Change-Id: Idc06074f059c2f0d683b5f291e594f6a1555d6d5
Signed-off-by: Luis Eduardo Bonatti <LuizEduardo.Bonatti@windriver.com>
2024-02-27 12:12:22 -03:00
Heitor Matsui 4788433459 Remove invalid checks from deploy precheck
The deploy-precheck script is now stored in a versioned
directory under /opt/software/rel-<ver>, so there is no
need to check if the feed directory exists.

This commit removes deprecated variables and checks that are
not needed as result of the versioning of the release scripts
directory.

Test Plan
PASS: run "software deploy precheck" successfully

Story: 2010676
Task: 49633

Change-Id: I42b00143234f4aa5a3c871bbd6900f3bfce43c8d
Signed-off-by: Heitor Matsui <heitorvieira.matsui@windriver.com>
2024-02-27 11:56:13 -03:00
Zuul e96e621b6d Merge "Improve software upload help text" 2024-02-27 14:41:42 +00:00
Zuul 9f7bddff25 Merge "deploy state changed update" 2024-02-26 21:41:45 +00:00
Zuul e740273a59 Merge "Deploy state sync on swact" 2024-02-26 21:33:09 +00:00
Heitor Matsui d5de91e301 Improve software upload help text
This commit changes the wording on software upload command
to improve the feedback to the user.

Test Plan
PASS: run "software upload -h" and verify the output

Story: 2010676
Task: 49589

Change-Id: I8df2e67ef20627e96399d797c998e00c38eab849
Signed-off-by: Heitor Matsui <heitorvieira.matsui@windriver.com>
2024-02-26 14:27:48 -03:00
Zuul c0d9a2ec0d Merge "Only allow one iso file being uploaded" 2024-02-23 15:10:09 +00:00
junfeng-li deae981e5f Deploy state sync on swact
This commit adds a new USM REST API endpoint that
provides the controllers deployment sync status during
the upgrade.

Depends-on: https://review.opendev.org/c/starlingx/update/+/904367

Test Plan:

PASS: sent request to get the response of the sync status

Task: 49426
Story: 2010676
Change-Id: Ib661f707686f13cbb52a5dbb67f636a9504ce8c9
Signed-off-by: junfeng-li <junfeng.li@windriver.com>
2024-02-23 14:56:43 +00:00
Bin Qian 1b462be12a deploy state changed update
After major release data migration completes, update deploy state to
start-done for successful migration or start-failed if not.
Ths deploy state update from sub process is sent to software_controller
via udp message. Only the updates from a valid agent will be accepted.

story: 2010676
Task: 49590

TCs:
     pass: Update deploy state to start-done after data migration
completes.
     pass: Update deploy state to start-failed after data migration
fails.

Change-Id: If4bfdfea168374bd11185243cffc20aa3af9c160
Signed-off-by: Bin Qian <bin.qian@windriver.com>
2024-02-23 13:43:09 +00:00
Zuul 5d89fc63f4 Merge "Fix package name fetch during start/delete" 2024-02-21 20:14:45 +00:00
Zuul 5539b5784f Merge "Add package listing option for "software show"" 2024-02-21 18:07:44 +00:00
Luis Eduardo Bonatti 9c990c4d5e Create message class to sync deploy state between controllers
This commit creates classes to handle the communication and sync
of deploy state between controllers and also add a operation
counter of deploy states.

This commit also synced with peer controller when deploy starts.

Test Plan:

PASS: Software.json of peer controller updated on deploy start.
PASS: Software.json of synced folder on peer controller matches
active controller on deploy start.
PASS: Software.json of peer controller updated with deploy state
change.
PASS: deploy state synced to peer in DX system when software deploy
start succeed.

Depends-on: https://review.opendev.org/c/starlingx/update/+/904362

Story: 2010676
Task: 49325

Change-Id: Id69b15e38402b5314657de963f5b69f164e2c351
Signed-off-by: Luis Eduardo Bonatti <LuizEduardo.Bonatti@windriver.com>
2024-02-16 15:51:49 +00:00
Zuul ce3dc63c3e Merge "Create release_data wrapper classes" 2024-02-16 15:14:07 +00:00
Heitor Matsui 1182e86c08 Fix package name fetch during start/delete
The apt-ostree command expect to receive only the name of the
package to install/delete from the package archive repository,
and due to [1] the packages are stored with name+version,
leading to apt-ostree failure.

This commit splits name from version from the package list so
that apt-ostree failures are fixed.

[1] https://review.opendev.org/c/starlingx/update/+/909046

Test Plan
PASS: install a patch release successfully
PASS: delete a patch release and verify the package is deleted
      from apt-ostree package repository successfully

Story: 2010676
Task: TBD

Change-Id: Id7f8b8379aa0e870f1fa136cb2e629f020dde7f3
Signed-off-by: Heitor Matsui <heitorvieira.matsui@windriver.com>
2024-02-16 09:26:15 -03:00
junfeng-li b662112b99 Only allow one iso file being uploaded
When using 'software upload-dir' to upload files ,
this commit will only allow one iso file or one signature
to be uploaded in the directory.  Multiple iso/signature
files in the directory to be uploaded won't be permitted
in order to prevent not enough disk space in /scratch
directory in the active controller and wrong signature
files being used.

Test Plan:

PASS: upload multiple iso files in the directory.
PASS: upload one iso file in the directory.
PASS: upload multiple sig files in the directory.
PASS: upload one sig file in the directory.

Task: 49576
Story: 2010676

Change-Id: Iba5af49ca50b4c836157027e140095a68ee5341b
Signed-off-by: junfeng-li <junfeng.li@windriver.com>
2024-02-15 21:12:08 +00:00