This has been in place since Ocata. If any version since then has been
deployed, this will have already been deployed. Time to drop this.
UpgradeImpact
Change-Id: I5412d78c63cf3381782f5c6fc059641489f89053
Implements: blueprint remove-sqlalchemy-migrate
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib.
Change-Id: I44e7b6f76e2d12f620ec602afc77ce11ba6b9d9a
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
As glance is changing metadata value of location from 'backend' to
'store', adding migration script which will update old image locations
to use 'store' as a metadata value.
NOTE:
Bump master (train development) to 19
Change-Id: I1386c535bc8ff4519e6b0bb879026b05c930b791
Sem-Ver: api-break
In the bug, a user tried setting a devstack password with a "@" in it.
As it turns out, sqlalchmey turns the connection-string into a
sqlalchemy.engine.url.URL object [1] which returns a RFC1738 quoted
string.
However, alembic's set_main_option [2] uses python
string-interpolation which interprets '%' characters. This means you
end up with an interpolation traceback when using any quoted character
(':@/') in a user/password (more likely password).
Avoid this by ensuring the URL is safe for python interpolation in
set_main_option by replacing '%' -> '%%'.
I convinced myself this is safe because sqlalchemy correctly parses
the quoted and unquoted versions just the same
---
>>> str(sqlalchemy.engine.url.make_url('mysql+pymysql://foo:crazy:@/pw@/moo'))
'mysql+pymysql://foo:crazy%3A%40%2Fpw@/moo'
>>> str(sqlalchemy.engine.url.make_url('mysql+pymysql://foo:crazy%3A%40%2Fpw@/moo'))
'mysql+pymysql://foo:crazy%3A%40%2Fpw@/moo'
---
A test is added
[1] https://github.com/zzzeek/sqlalchemy/blob/master/lib/sqlalchemy/engine/url.py
[2] http://alembic.zzzcomputing.com/en/latest/api/config.html#alembic.config.Config.set_main_option
Change-Id: I3ef7e3e539e35ce040573f2044ab6eb3c990200a
Closes-Bug: #1695299
After recent change in manage command [1] deploying glance with
postgres using stack.sh fails as db_sync internally uses E-M-C
and it has support only for mysql.
Added flag online_migration which will be False in case of
db_sync so that enigne check will be eliminated and True in case
of running expand, migrate, contract to perform validatio check on
engine type.
[1] https://review.openstack.org/#/c/433934/
Closes-Bug: #1749466
Change-Id: Ifddeff139e212564118520f3150db198ab1c94f4
If you run expand and migrate commands for the second time,
it should return a user friendly message instead of attempting
to upgrade db again.
Added a check to confirm if expand and migrate are already in
sync and return a user friendly message.
Closes-Bug: #1745360
Change-Id: Iaf2e8ae2004db03f9b7498a2c498360fec096066
This patch adds a new "glance-manage db check" command
which will check the current state of the users upgrade
repos and relay info back to the user if the user has any
outstanding db upgrades left to run with appropriate exit code.
Co-Authored-By: Bhagyashri Shewale <bhagyashri.shewale@nttdata.com>
Implements: Ie1e2fec2361765ddf23da897abcf0e12e682612e
Change-Id: I1e0b02d615690f65a17b4ccfe4e4a72cc9e15ada
We have a ton of DeprecationWarning messages in our unit test runs.
Most of these are out of our control from third party libs. This
adds a WarningsFixture to limit warning output to once per test
run. In local py35 unit testing, this went from 14549 warnings to
8913.
Also including ignorning a policy 'is_admin' deprecation warning
that was added before a clear plan or replacement had been put
in place. Other projects have added this rather than fixing it
at the source since it is currently being reworked.
Based on work previously done in Nova and Cinder.
Change-Id: I4d97f74ed37b7b0e9a613ecfe33c4b26216ca768
The parent of this patch introduced the change to Alembic-based
migrations. This commit builds on top of that by adding expand,
migrate and contract commands to the glance-manage tool.
Appropriate documentation is updated and failing tests are adjusted
to accomodate the new database versioning schema.
Data migrations are expected to be run in the background with older
Glance services being active during the upgrade process.
Partially-Implements: blueprint database-strategy-for-rolling-upgrades
Co-Authored-By: Hemanth Makkapati <hemanth.makkapati@rackspace.com>
Change-Id: Ie839e0f240436dce7b151de5b464373516ff5a64
Depends-On: I77921366a05ba6f9841143af89c1f4059d8454c6
This change proposes the use of Alembic to manage Glance migrations.
* Introduce new directory ``alembic_migrations`` under
``glance/db/sqlalchemy``. This directory is the home for all glance
migrations henceforth. All the migration scripts reside under
``versions`` directory.
* All the migrations up to Liberty are consolidated into one migration
called ``liberty_initial`` as those migrations are not supported
any more. Mitaka migrations are retained but under a different naming
convention.
* All the glance manage db commands are changed appropriately. They now
use alembic to perform operations such as ``version``, ``upgrade``,
``sync`` and ``version_control``.
* The database versions are not numerical any more. They are the revision
ID of the last migration applied on the database. Since we don't
support migrations before Mitaka, the Liberty version ``42`` will now
appear as ``liberty``. Migration ``43`` and ``44`` in Mitaka appear as
``mitaka01`` and ``mitaka02`` respectively.
* When one performs a ``sync`` or ``upgrade`` command, the database is
first stamped with an equivalent alembic version before upgrading.
* The older migration scripts are retained so that users can correlate
with the new migrations. Also, it is probably safe to retain them until
the alembic migrations become stable. Similarly, the ``migrate_version``
table is not removed yet.
Partially-Implements: blueprint alembic-migrations
Change-Id: Ie8594ff339a13bf190aefa308f54e97ee20ecfa2
Co-Authored-By: Alexander Bashmakov <alexander.bashmakov@intel.com>
Depends-On: I1596499529af249bc48dfe859bbd31e90c48a5e0
As downgrade are not supported after Kilo, we should remove them now.
Roll backs can be performed as mentioned in the below link:
http://docs.openstack.org/ops-guide/ops-upgrades.html#rolling-back-a-failed-upgrade
The DB downgrades were deprecated in Glance Mitaka release by commit
e3366afdfb.
Change-Id: I937d15d93f16a3e44a50e6ff1a469098eab67c79
Implements: blueprint remove-db-downgrade
while running unit tests using run_tests.sh,
we can see quite an amount of unwanted log
messages and stdout prints as part of
the unit test results on the console.
This basically makes it difficult to go
through the test results and figure whats happening.
This patch fixes the cause for those
extraneous log messages caused due to
a incorrect fixture usage in test_manage.py
and also by mocking sys.stdout at the respective unit tests.
Closes-bug: #1496898
Change-Id: If986691cb3f36ba357a5c0df34b6df7bf2d8ff86
Currently there is no way to upgrade metadata definitions to the
newest set. This change extends existing command:
glance-manage db load_metadefs.
The extension allows user to merge metadata definitions that are
stored in files with data that exists in database. By default it
prefers existing data over new (--merge), but it can be combined
with other options to change this logic. Use --prefer_new flag so
it will prefer new data over existing data in database or
--overwrite so it will drop every namespace (and attached
resources) found in both, database and file.
By default glance-manage db load_metadefs works the same way it
worked before extension. To enable new logic user needs to
provide at least --merge option (or combine it with other two
options)
Implements: blueprint metadefs-upgrade-by-json-file
Change-Id: I55fa6640142db5110deb88d9ecd8507e7f533c58
oslo_db was moved out of the oslo namespace in
oslo.db>=1.4.0.
Change-Id: I13620d26ff12c55b2294b3b051669e0e98513a10
Related-to: blueprint drop-namespace-packages
The tests generate a large amount of extraneous logging output.
This due to the manage utility, when tested via test_manage,
setting up logging in it's own way. Monkey patching the log.setup
call to a no-op during tests prevents this.
Also fixed an incorrect usage of assert which was causing a
warning to be generated.
Change-Id: I6b8724142c0812d84e697e280acc0264620f6a2b
This is a deprecated option only should be used for migration. As we
scheduled before, now glance removes it out from Juno release.
Change-Id: Ice4ade659307ee5e635a75d6637b4664ee431c20
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
Implements: blueprint metadata-schema-catalog
A common API hosted by the Glance service for vendors, admins,
services, and users to meaningfully define available key / value
pair and tag metadata. The intent is to enable better metadata
collaboration across artifacts, services, and projects for
OpenStack users.
This is about the definition of the available metadata that can
be used on different types of resources (images, artifacts,
volumes, flavors, aggregates, etc). A definition includes the
properties type, its key, it's description, and it's constraints.
This catalogue will not store the values for specific instance
properties.
Change-Id: Ib1c1abf80879fb6dcd5ee30c7d2bc65b0ba720d5
DocImpact
Co-Authored-By: Lakshmi N Sampath <lakshmi.sampath@hp.com>
Co-Authored-By: Wayne Okuma <wayne.okuma@hp.com>
Co-Authored-By: Travis Tripp <travis.tripp@hp.com>
Co-Authored-By: Pawel Koniszewski <pawel.koniszewski@intel.com>
Co-Authored-By: Michal Jastrzebski <michal.jastrzebski@intel.com>
Co-Authored-By: Michal Dulko <michal.dulko@intel.com>
Some unittests replace methods by mock objects, so they are leave mocked
when unittest finished. It can cause a lot of issues, because when we
try to get access to these attributes in the next test, we get mock
object instead of real attribute. Use @mock.patch.object decorator to
get attributes `unmocked` after test.
Change-Id: I23b3926a7671e0669315836da787ec88a812d67b
Replace common oslo code glance.openstack.common.db
by usage of oslo.db library and remove common code.
Change-Id: I27216e03763f72e8665121d4440ba2828349eac5
Adding a deprecated option to skip the DB sanity check to db_sync
interface, to give deployer a chance to execute migration script.
Without this change all migration scripts will be blocked before get
execution.
DocImpact
Change-Id: If887ac6b814879a0140dc1065a060233aee7bc82
Partial-Bug: #1279000
Related-Id: I7f271d846141ac72dde3fb0d12159b125018eb2c
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
This sync contains commit ``ce69e7f Don't store engine instances in oslo.db ``
which removed global engine from oslo.db code. So, added code
for work with sqla engines and sessions to Glance - get_engine()
and get_session() functions in glance.db.sqlalchemy.api module.
Also this remove database slave connection and tpool features, because
they was removed from oslo code and seeps to be unused in Glance.
Full list of changes:
7959826 db: move all options into database group
dda24eb Introduce mysql_sql_mode option, remove old warning
0b5af67 Introduce a method to set any MySQL session SQL mode
8dccc7b Handle ibm_db_sa DBDuplicateEntry integrity errors
0f24d82 Fix migration.db_version when no tables
ac84a40 Update log translation domains
c0d357b Add model_query() to db.sqlalchemy.utils module
84254fc Fix a small typo in api.py
b8a676c Remove CONF.database.connection default value
86707cd Remove None for dict.get()
0545121 Fix duplicating of SQL queries in logs
fcf517d Update oslo log messages with translation domains
fa05b7c Restore the ability to load the DB backend lazily
630d395 Don't use cfg.CONF in oslo.db
ce69e7f Don't store engine instances in oslo.db
35dc1d7 py3kcompat: remove
b4f72b2 Don't raise MySQL 2013 'Lost connection' errors
271adfb Format sql in db.sqlalchemy.session docstring
0334cb3 Handle exception messages with six.text_type
eff69ce Drop dependency on log from oslo db code
7a11a04 Automatic retry db.api query if db connection lost
11f2add Clean up docstring in db.sqlalchemy.session
1b5147f Only enable MySQL TRADITIONAL mode if we're running against MySQL
39e1c5c Move db tests base.py to common code
986dafd Fix parsing of UC errors in sqlite 3.7.16+/3.8.2+
9a203e6 Use dialect rather than a particular DB API driver
1779029 Move helper DB functions to db.sqlalchemy.utils
bcf6d5e Small edits on help strings
ae01e9a Transition from migrate to alembic
70ebb19 Fix mocking of utcnow() for model datetime cols
7aa94df Add a db check for CHARSET=utf8
aff0171 Remove "vim: tabstop=4 shiftwidth=4 softtabstop=4" from headers
fa0f36f Fix database connection string is secret
8575d87 Removed copyright from empty files
d08d27f Fix the obsolete exception message
8b2b0b7 Use hacking import_exceptions for gettextutils._
9bc593e Add docstring for exception handlers of session
855644a Removal of _REPOSITORY global variable.
ea6caf9 Remove string.lowercase usage
a33989e Remove eventlet tpool from common db.api
e40903b Database hook enabling traditional mode at MySQL
f2115a0 Replace xrange in for loop with range
c802fa6 SQLAlchemy error patterns improved
1c1f199 Remove unused import
97d8cf4 Remove lazy loading of database backend
Co-authored-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
Related to blueprint db-use-oslo-common-code
Change-Id: Ia73abba8309ccc2ad10a0f636b410984c6b6e5d8
Stephen Finucane