In Python3.11, the argparse module became stricter. While the following
code was fine in previous versions, it will now raise an
argparse.ArgumentError because we added the same subparser twice:
import argparse
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers(title='Commands')
subparsers.add_parser('foo')
subparsers.add_parser('foo')
subparsers.add_parser('bar')
parser.parse_args()
This mistake was silently ignore in Python3.10, which explains why this
was never an issue for glance-manage.
Closes-Bug: #1982426
Change-Id: I3a88ab5d2e67a1553f03b0b8ba44efbef976ffb0
When we want to purge all deleted rows
we use "glance-manage db purge" or
"glance-manage purge_images_table"
then we never know how many deleted rows are still in a table.
So we need to launch the command many times until
the command reports that 0 rows were deleted in every table.
It's inconvenient.
The patch introduced new valid value for "max_rows".
If "max_rows" equals -1 all deleted rows are purged.
Closes-Bug: 1702445
Change-Id: Ia9811d93b1c0b67a1b29a9e8ee3e0fc098c57d4a
This has been in place since Ocata. If any version since then has been
deployed, this will have already been deployed. Time to drop this.
UpgradeImpact
Change-Id: I5412d78c63cf3381782f5c6fc059641489f89053
Implements: blueprint remove-sqlalchemy-migrate
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
This patch removes majority of the registry and it's related
endpoints and config options that has been deprecated for
removal in various releases.
Change-Id: I75014bd50bf382efebe56bd89c20ffefbdde25f5
The `glance manage purge-images-table` is supposed to be used
only if really required due to OSSN-0075. And the advice is
not to purge any freshly deleted rows due to the possible
ID collision of still in use images. This change increases
the default of "age_in_days" to 180 from it's original 30 to
stress this out. User can still define smaller age if they
must.
Change-Id: Idb13288b72b31940a357fc9b11db2d6bcd396261
Modified the current ``glance-manage db purge`` command to eliminate images
table from purging the records. Added new command
``glance-manage db purge_images_table`` to purge the records from images
table.
DocImpact
SecurityImpact
Change-Id: Ie6641659b54543ed9f96c393d664e52a26bfaf6a
Implements: blueprint mitigate-ossn-0075
Recently this change [1] in glance-manage db_sync is internally
using Expand, Migrate and Contract. EMC is explicitly used for
online migration for which glance uses triggers to sync data
between old columns and new columns. DB Sync is used for
offline migartion for which adding triggers is not required.
Made provision to execute triggers explicitly in case of
online migration (EMC pattern) and skip the same in
case of offline migration (db sync).
[1] https://review.openstack.org/#/c/433934/
Closes-Bug: #1749640
Change-Id: I816c73405dd61d933182ad5efc24445a0add4eea
After recent change in manage command [1] deploying glance with
postgres using stack.sh fails as db_sync internally uses E-M-C
and it has support only for mysql.
Added flag online_migration which will be False in case of
db_sync so that enigne check will be eliminated and True in case
of running expand, migrate, contract to perform validatio check on
engine type.
[1] https://review.openstack.org/#/c/433934/
Closes-Bug: #1749466
Change-Id: Ifddeff139e212564118520f3150db198ab1c94f4
If you run expand and migrate commands for the second time,
it should return a user friendly message instead of attempting
to upgrade db again.
Added a check to confirm if expand and migrate are already in
sync and return a user friendly message.
Closes-Bug: #1745360
Change-Id: Iaf2e8ae2004db03f9b7498a2c498360fec096066
After recent change in manage command [1]
deploying glance with postgres using stack.sh
fails as db_sync internally uses E-M-C and it
has support only for mysql.
Made provision to perform offline migration
if database is postgresql.
[1] https://review.openstack.org/#/c/433934/
Change-Id: Ie6c66da8b26930d783526a4b92b77baf493825f9
Closes-Bug: #bug/1747869
This patch adds a new "glance-manage db check" command
which will check the current state of the users upgrade
repos and relay info back to the user if the user has any
outstanding db upgrades left to run with appropriate exit code.
Co-Authored-By: Bhagyashri Shewale <bhagyashri.shewale@nttdata.com>
Implements: Ie1e2fec2361765ddf23da897abcf0e12e682612e
Change-Id: I1e0b02d615690f65a17b4ccfe4e4a72cc9e15ada
If you return anything as string to sys.exit method, the system exit
status is one (i.e. failure).
Added user friendly print message and returned without any exit code.
Closes-Bug: #1747690
Change-Id: I107cc1ad4b8d6d0ebee1127fc1e57db55c7cd839
Modified 'glance-manage db_sync' operation to use
expand, migrate and contract operations.
Added test queens scripts for testing purpose only.
This patch removes use of "monolithic" upgrade scripts
and resolve the issue while upgrading from ocata to pike.
Co-Authored-By: Shilpa Devharakar <Shilpa.Devharakar@nttdata.com>
Closes-Bug: #1723001
Change-Id: I2653560d637a6696f936b49e87f16326fd601dfe
Oslo.config deprecated parameter enforce_type and change its
default value to True in Ifa552de0a994e40388cbc9f7dbaa55700ca276b0.
Remove the usage of it to avoid DeprecationWarning: "Using the
'enforce_type' argument is deprecated in version '4.0' and will be
removed in version '5.0': The argument enforce_type has changed its
default value to True and then will be removed completely."
Change-Id: If045d76574823f29c58b39a104354abc1407cb8d
Related--Bug: #1517839
'glance-manage db purge' command fails with DBReferenceError due
to FK constraint failure and exits with stack-trace on the command
prompt.
Made changes to give user-friendly error message to the user as
well as log appropriate error message in glance-manage logs
instead of stack-trace.
Co-author-by: Dinesh Bhor <dinesh.bhor@nttdata.com>
Change-Id: I52e56b69f1b78408018c837d71d75c6df3df9e71
Closes-Bug: #1554412
This patch adds equivalent expand/migrate/contract migration scripts for
Community Images. The expand migration
'ocata_expand01_add_visibility.py' creates a new migration branch 'expand'
from the last known migration i.e., 'mitaka02'. Similarly, the contract
migration 'ocata_contract01_drop_is_public.py' creates another new
migration branch called 'contract' from the last known migration.
The data migration 'ocata_migrate01_community_images.py' migrates
all rows in the database at once. There is possibility of performance
degradation while the data migrations are running.
Change-Id: I34f5623d6804e9fe594e6b5b196ea4a162578196
Partially-Implements: blueprint database-strategy-for-rolling-upgrades
Co-Authored-By: Hemanth Makkapati <hemanth.makkapati@rackspace.com>
Depends-On: Ie839e0f240436dce7b151de5b464373516ff5a64
The parent of this patch introduced the change to Alembic-based
migrations. This commit builds on top of that by adding expand,
migrate and contract commands to the glance-manage tool.
Appropriate documentation is updated and failing tests are adjusted
to accomodate the new database versioning schema.
Data migrations are expected to be run in the background with older
Glance services being active during the upgrade process.
Partially-Implements: blueprint database-strategy-for-rolling-upgrades
Co-Authored-By: Hemanth Makkapati <hemanth.makkapati@rackspace.com>
Change-Id: Ie839e0f240436dce7b151de5b464373516ff5a64
Depends-On: I77921366a05ba6f9841143af89c1f4059d8454c6
This change proposes the use of Alembic to manage Glance migrations.
* Introduce new directory ``alembic_migrations`` under
``glance/db/sqlalchemy``. This directory is the home for all glance
migrations henceforth. All the migration scripts reside under
``versions`` directory.
* All the migrations up to Liberty are consolidated into one migration
called ``liberty_initial`` as those migrations are not supported
any more. Mitaka migrations are retained but under a different naming
convention.
* All the glance manage db commands are changed appropriately. They now
use alembic to perform operations such as ``version``, ``upgrade``,
``sync`` and ``version_control``.
* The database versions are not numerical any more. They are the revision
ID of the last migration applied on the database. Since we don't
support migrations before Mitaka, the Liberty version ``42`` will now
appear as ``liberty``. Migration ``43`` and ``44`` in Mitaka appear as
``mitaka01`` and ``mitaka02`` respectively.
* When one performs a ``sync`` or ``upgrade`` command, the database is
first stamped with an equivalent alembic version before upgrading.
* The older migration scripts are retained so that users can correlate
with the new migrations. Also, it is probably safe to retain them until
the alembic migrations become stable. Similarly, the ``migrate_version``
table is not removed yet.
Partially-Implements: blueprint alembic-migrations
Change-Id: Ie8594ff339a13bf190aefa308f54e97ee20ecfa2
Co-Authored-By: Alexander Bashmakov <alexander.bashmakov@intel.com>
Depends-On: I1596499529af249bc48dfe859bbd31e90c48a5e0
Adding ability to purge records less than 1 day old, using the
glance-manage db_purge utility.
Closes-Bug: #1643287
Change-Id: Ibaea583d49bd5d09ad2e6bf99d2c0efaac5cb4ec
The 3.17.0 release of oslo.log changes the default value of "use_stderr"
to False (from True). We had tests that asserted that our glance-*
commands used stderr to provide information to administrators. This
works to preserve that default value for operators by overriding the
default to True when we're running in one of those commands.
We deliberately excluded glance-api, glance-glare, and glance-registry
because glance-api and glance-registry are usually not run manually and
they don't need to log to stderr by default and glance-glare is
deprecated.
Change-Id: Ia054f8f637c826280722c0d2a6811fdbc0cf54ed
As downgrade are not supported after Kilo, we should remove them now.
Roll backs can be performed as mentioned in the below link:
http://docs.openstack.org/ops-guide/ops-upgrades.html#rolling-back-a-failed-upgrade
The DB downgrades were deprecated in Glance Mitaka release by commit
e3366afdfb.
Change-Id: I937d15d93f16a3e44a50e6ff1a469098eab67c79
Implements: blueprint remove-db-downgrade
Currently 'glance-manage db purge' fails if given a
very large number for max_rows.
Moved and renamed validate_mysql_int() from
glance.common.utils to glance.db.sqlalchemy.api as
_validate_db_int() since it is related to database
only so that it can be used to validate max_rows
for maximum limit.
Closes-Bug: #1543937
Change-Id: Id16694807c180632c1785e9b1ebe8d1c79d885ab
Moved validation for age_in_days and max_rows from
glance.db.sqlalchemy.api to glance.cmd.manage since
code from purge_deleted_rows() method of
glance.db.sqlalchemy.api is unreachable as it is
throwing ValueError exception for invalid input from
glance.cmd.manage itself.
Related-Bug: #1543937
Change-Id: I4e976fde2a78fd901c233966adc350a0ded41841
The default values needed for glance's implementation of cors
middleware have been moved from paste.ini into the configuration
hooks provided by oslo.config. Furthermore, these values have been
added to glance's default configuration parsing. This ensures
that if a value remains unset in glance-api.conf, it will be set to use
sane defaults, and that an operator modifying the configuration
file will be presented with a default set of necessary sane headers.
Change-Id: I3c9d267b6224d6c7e5cc2c41cb51fb7e363c4955
Closes-Bug: 1551836
Import i18n functions from module directly and do not use
global module variables like _ = i18n._. It makes code shorter
and cleaner. It also to detect cases when these functions are not
used in module.
Change-Id: Iaa593ac1f2dd15cbcad049bef6ba68f1cfa610da
DB downgrade doesn't be supported in OpenStack now. Glance will
remove it in N release. In M cycle, we should add a warning to users.
Closes-bug: #1501233
Implements: blueprint remove-db-downgrade
Change-Id: I2791d8421abc0ad6c4905d5ddaa3fa99f264e333
* Replace glance.common.utils.exception_to_str() with
oslo_utils.encodeutils.exception_to_unicode().
* Remove exception_to_str() from glance.common.utils
* Add "from oslo_utils import encodeutils" and remove "from
glance.common import utils" (when utils is no more used)
* glance/api/v2/images.py: replace utils.exception_to_str() with
six.text_type() to cast the oslo_i18n message to Unicode
Glance exception_to_str() function was not compatible with Python 3,
whereas exception_to_unicode() is compatible with Python 3, it's in Oslo
common libraries and it's well tested.
This patch was first generated by a tool (modified version of sixer),
and then fixed to respect the PEP8 (especially the constraint of 80
columns).
Change-Id: I86008c8adc0c5664f96573c1015cc15e2d06e3e2
The glance-manage command isn't very helpful when it is run without any
arguments. The current output looks like following:
$ glance-manage
usage: glance-manage [options] <cmd>
glance-manage: error: too few arguments
This patch extends the default output to look similar to what can be
seen in the cinder-manage tool. The new output is:
$ glance-manage
/usr/local/bin/glance-manage category action [<args>]
Available categories:
db
Change-Id: I2c56ccf5a28cdaf9fdfb03d79763b34dc096a858
Closes-Bug: #1394316
Currently there is no way to upgrade metadata definitions to the
newest set. This change extends existing command:
glance-manage db load_metadefs.
The extension allows user to merge metadata definitions that are
stored in files with data that exists in database. By default it
prefers existing data over new (--merge), but it can be combined
with other options to change this logic. Use --prefer_new flag so
it will prefer new data over existing data in database or
--overwrite so it will drop every namespace (and attached
resources) found in both, database and file.
By default glance-manage db load_metadefs works the same way it
worked before extension. To enable new logic user needs to
provide at least --merge option (or combine it with other two
options)
Implements: blueprint metadefs-upgrade-by-json-file
Change-Id: I55fa6640142db5110deb88d9ecd8507e7f533c58
oslo_db was moved out of the oslo namespace in
oslo.db>=1.4.0.
Change-Id: I13620d26ff12c55b2294b3b051669e0e98513a10
Related-to: blueprint drop-namespace-packages
oslo_config was moved out of the oslo namespace in oslo.config>=1.6.0.
Related-to: blueprint drop-namespace-packages
Change-Id: I30ecbf8f9de77496fcb643c7ad9738d79ad359f0
As specified in the blueprint, we are only going to set log_file in the
configuration file for now, and we read it last in glance.cmd.manage.
This allows glance-manage to use the appropriate log-file and still
retain the settings it needs in glance-api.conf and glance-registry.conf
DocImpact
Change-Id: I87595a29634e6ffda4e1581d42a92dfe6f84044b
Implements: blueprint create-glance-manage-conf
Closes-bug: #1391211
Changing all Glance files to use i18n instead of gettextutils for log
translations.
Finishes bp refactoring-glance-logging
Change-Id: I90d6ab4f7e09f4e7591921f9526de6370ebdb789
This is a deprecated option only should be used for migration. As we
scheduled before, now glance removes it out from Juno release.
Change-Id: Ice4ade659307ee5e635a75d6637b4664ee431c20
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
As a common approach most projects used now and Oslo preferred, this
change enabled sample configuration file generation mechanism for
each Glance services.
This change, as an enhancement, allows generating separated sample
configuration files for each Glance major services, e.g.:
etc/glance-api.conf.sample
etc/glance-cache.conf.sample
etc/glance-manage.conf.sample
etc/glance-registry.conf.sample
etc/glance-scrubber.conf.sample
It is different than I94d486d6686815c45705a7a9b00fb26062e1eb63
which only supports generating an unified sample configuration
file to including all Glance available options.
This mechanism not only can help auditing by packager, milestone
maintainer or developer as a function (testenv) of tox, but also
those separated configuration files could make deployment be easy.
And it helps keeping sample configuration files be update with
code change.
The change added "genconfigs" function as a tox testenv (-egenconfigs),
and it dependes on oslo-config-generator function of oslo.config.
The change doesn't introduce those sample files Glance repo, so
next step is to investigate if we can generate them in gate
automatically when a change was merged.
Related-Change-Id: I15686708fc9460948a58cfea3d18dae40ba1fda9
Related-Change-Id: Iae31856d5886ee78786972d80c7c103c3460a2b3
Related-Change-Id: I76043b08e2872867e5af2a5ac902e4d092fda5c8
Closes-Bug: #1300546
Closes-Bug: #1361963
Change-Id: Ibe03a3fe80b96ca32acb1a6bea7e38e6075951bb
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
Cyril Roelandt