These are no longer used and no longer necessary. Remove them.
Change-Id: Ibd7d46aefb3794dd342c8afb40dbb8d4ce00a976
Implements: blueprint remove-sqlalchemy-migrate
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
We want to remove the old sqlalchemy-migrate based migrations. Move the
only remaining module we need from this up a level so we can 'rm -rf'
the entire directory in a future change.
Change-Id: I3b1a1668c48e3136fd91b1418e00c38deec76bb5
Implements: blueprint remove-sqlalchemy-migrate
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
We were capped at a very old version of hacking. Hacking itself caps the
various linters it uses to remain consistent, so our pep8 job was not
checking quite a bit that current versions have added.
This raises that limit to the latest to get up to the level of other
projects and addresses the errors the updated linters uncovered.
Change-Id: I89a9d73fbd59606a649e26077acebc5c42873d67
Co-authored-by: Sean McGinnis <sean.mcginnis@gmail.com>
The query added with CI change [0] was using the python 'and'
instead of sqlalchemy's 'and_()'. This patch changes that.
This patch also further refines the query to look specifically
for visility 'private' instead of '<> public' for accuracy during
setting of visibility to 'shared'.
[0] I94bc7708b291ce37319539e27b3e88c9a17e1a9f
Change-Id: I6851aa0e5ca8cecaff518609c14cd528bca95ade
This change replaces the existing boolean 'is_public' column for
the 'images' table with enum 'visibility' column featuring the
four explicit visibility values - public, private, shared,
and community.
This change also implements and enables all backend code to
utilize the new values.
Co-Authored-By: Timothy Symanczyk <timothy_symanczyk@symantec.com>
Co-Authored-By: Dharini Chandrasekar <dharini.chandrasekar@intel.com>
Implements: blueprint community-level-v2-image-sharing
Closes-Bug: #1394299
Closes-Bug: #1452443
Depends-On: I6e3268f3712cbc0aadb51d204c694023b92d55a5
Change-Id: I94bc7708b291ce37319539e27b3e88c9a17e1a9f
As downgrade are not supported after Kilo, we should remove them now.
Roll backs can be performed as mentioned in the below link:
http://docs.openstack.org/ops-guide/ops-upgrades.html#rolling-back-a-failed-upgrade
The DB downgrades were deprecated in Glance Mitaka release by commit
e3366afdfb.
Change-Id: I937d15d93f16a3e44a50e6ff1a469098eab67c79
Implements: blueprint remove-db-downgrade
The metadata definitions in etc/metadefs allow each namespace to be associated
with a resource type in OpenStack. Now that Horizon is supporting adding
metadata to instances (just got in during the mitaka cycle - so unrealeased),
I realized that we used OS::Nova::Instance instead of OS::Nova::Server in
Glance. This doesn’t align with Heat [0] or Searchlight [1].
There are a couple of metadef files that have OS::Nova::Instance that need to
change to OS::Nova::Server. I see also that OS::Nova:Instance is in one of
the db scripts. That script simply adds some initial "resource types" to the
database. [3]. It should be noted that there is no hard dependency on that
resource type to be in the DB script. You can add new resource types at any
time via API or JSON files and they are automatically added.
I'm not sure if the change to the db script needs to be done in a different
patch or not, but that can easily be accommodated.
See bug for additional links.
Change-Id: I196ce1d9a62a61027ccd444b17a30b2c018d9c84
Closes-Bug: 1537903
This change introduces glance.common.timeutils that provides the
timeutils previously consumed from oslo_utils.
Oslo is deprecating some timeutils functionality which of Glance
depends on. Suggested replacement (isoformat) would break glance APIs
so it's cleaner to carry this functionality in Glance rather than
re-invent the wheel.
Co-Authored-By: Erno Kuvaja <jokke@usr.fi>
Co-Authored-By: Sabari Kumar Murugesan <smurugesan@vmware.com>
Change-Id: I91e1cc9a273249fd88749cecf21200f3f5e2bab1
Fix all assert problems raised by Bandit. Asserts are potentially
problematic, since Python optimization sometimes removes them, so code
needs to remain safe and functional without the assert.
Two asserts are safe to skip, so they are deleted for improved error
messages. Three asserts are probably necessary, and are converted to
exceptions. Two asserts are probably necessary, and are instead made to
fail safely, and `# nosec` is added to the assert line.
This also enables the assert test in bandit's configuration.
Change-Id: Ic69a204ceb15cac234c6b6bca3d950256a98016d
Partial-bug: 1511862
Import i18n functions from module directly and do not use
global module variables like _ = i18n._. It makes code shorter
and cleaner. It also to detect cases when these functions are not
used in module.
Change-Id: Iaa593ac1f2dd15cbcad049bef6ba68f1cfa610da
Provide support in Glance API for querying the image list for
created_at and updated_at times using guidance from the API Working
Group recommendations for filtering. Filtering is applied at the DB
layer.
DocImpact
ApiImpact
MitakaPriority
Change-Id: Ie94295bb82779ec17ab773928c71ae4a9ee8fbcc
Implements bp: v2-additional-filtering
The 042_add_changes_to_reinstall_unique_metadef_constraints.py file was
checked in with file permissions of 775. This should be restricted to
644.
Change-Id: I4b8359046e3a592c6a73676d30f22da7f07e3299
Closes-Bug: 1494895
Backend store-specific requirements are not hard requirements, since
the choice of backend store is up to the operator. The
python-swiftclient and oslo.vmware modules are herein moved out of
requirements.txt and instead listed as optional requirements using
the extras functionality in setup.cfg.
Removed unnecessary import of swift backend in db migration, to allow
python-swiftclient to move out of hard requirements.
Cleaned up other requirements that are no longer used.
Change-Id: I9edbf8a614b5caaf38b4c59bd864b7baa55bf36b
Depends-On: I94d75e3d52c9e3e9f7a741a079d25b1bdff29b27
Closes-Bug: #1475737
Sometime during Kilo, the unique constraints on
metadef_namespaces (namespace)
metadef_objects(namespace_id, name)
metadef_properties(namespace_id, name)
metadef_tags(namespace_id, name)
metadef_resource_types(name)
were replaced with non-unique indices.
I believe this was done erroneously to make the migrate_repo/versions/scripts
match the db/sqlalchemy/models_metadef.py definitions. Unfortunately, the
schema scripts were correct with the unique constraints and what should have
changed was models_metadef.py.
This bug, puts one more migrate script in place which will rename any
duplicate records it finds to make them unique and then re-establishes
the unique constraints while dropping the non-unique indices. It also,
fixes models_metadef.py and adds in tests to attempt to create duplicates
which should result in an HTTPConflict.
Change-Id: Idf3569a27d64abea3ed6ec92fb77b36a4d6d5fd5
Closes-Bug: 1468946
oslo.db mandates all tables should be created with utf-8 charset
when using MySQL DB. We added an option (deprecated) to skip this
sanity check and it was subsequently removed in Liberty.
This patch fixes existing migration scripts to create tables with
utf-8 charset. Since oslo.db does a sanity check before running
the migrations, it's safe to assume that only new tables created
since the previous migration will be affected.
Closes-bug: #1279000
Change-Id: Ia31dc9e5d4494d1b9633f916fa6e2704d33dffb0
* Replace glance.common.utils.exception_to_str() with
oslo_utils.encodeutils.exception_to_unicode().
* Remove exception_to_str() from glance.common.utils
* Add "from oslo_utils import encodeutils" and remove "from
glance.common import utils" (when utils is no more used)
* glance/api/v2/images.py: replace utils.exception_to_str() with
six.text_type() to cast the oslo_i18n message to Unicode
Glance exception_to_str() function was not compatible with Python 3,
whereas exception_to_unicode() is compatible with Python 3, it's in Oslo
common libraries and it's well tested.
This patch was first generated by a tool (modified version of sixer),
and then fixed to respect the PEP8 (especially the constraint of 80
columns).
Change-Id: I86008c8adc0c5664f96573c1015cc15e2d06e3e2
The sqlite migration 37 created duplicate unique constraints on
the image_properties table, one unnamed and one called
ix_image_properties_image_id_name.
This causes the unit tests to fail with sqlalchemy 1.0+ since it
catches the problem when it compares the model to the schema.
The fix is to only create the named constraint.
Change-Id: I7ee19f08f2c107987e763466dc6e3b07185598c9
Closes-Bug: #1467008
When using db2 as the database backend, it encounters problem
when running db_sync in step 039 due to "SQL0669N A system
required index cannot be dropped explicitly", which need to
drop the constraint first then recreate the index. And this
patch also fixes the problem of dropping some indexes which
only are created in non-db2 databases, meanwhile, it will not
break the logic for all other database backends.
Closes-Bug: #1434440
Change-Id: I628bfa9ab1e4e07b71b49fc0e5a6087679b642a6
Adds a Data Model (Tables: Artifact, ArtifactDependecy, ArtifactTag,
ArtifactProperty, ArtifactBlob, ArtifactBlobLocation), Migrations and DB
API for SQLAlchemy. Adds wrappers for simple and registry DB drivers.
Adds appropriate unittests to DB API tests and to migration tests.
Implements-blueprint: artifact-repository
Co-Authored-By: Alexander Tivelkov <ativelkov@mirantis.com>
Co-Authored-By: Mike Fedosin <mfedosin@mirantis.com>
Co-Authored-By: Inessa Vasilevskaya <ivasilevskaya@mirantis.com>
Change-Id: Ia491a58956101a1c40f1bca95cd9efe432f13dce
Adds new migration to fix an issue with an index created by
mysql.
Also includes patch I00da67ab06cd0a3c46aa8a6cd815d0559408c2f0,
because without it tests won't pass.
Change-Id: Iffb54d5ee404889d49c40dbe0401d61e55723872
Co-Authored-By: Mike Fedosin <mfedosin@mirantis.com>
Co-Authored-By: Andreas Jaeger <aj@suse.de>
Co-Authored-By: Oleksii Chuprykov <ochuprykov@mirantis.com>
Correct migrations to satisfy models_metadef.
Add nullable=False for json_schema in metadef_object
and metadef_property according to changes in
Change-Id: I2c574210f8d62c77a438afab83ff80f3e5bd2fe7
Set updated_at nullable=True due to problems
with setting default value NOW() for datetime
type for mysql (the same as in models.py).
Closes-Bug: #1365436
Change-Id: I5399923621913724914499aad9148a9410faa4ef
It's not pythonic to use parentheses in conditional statements which can
fit into single line. Following patch removes redundant parentheses from
the code.
Change-Id: I01544931cc49b250b8224a593c64d7b59b576799
Glance SQL scripts assume too much about the environment and the way in
which they are executed by sqlalchemy-migrate. Particularly, they manage
transactions on their own even though sqlalchemy-migrate do it itself,
plus it assumes that scripts are executed in one go (even though
standard Python DB-API 2.0 does not mention multi-statement SQL commands
to be executed on DB cursors.
Also fix comments to be actual SQL comments and not C comments
https://blueprints.launchpad.net/oslo.db/+spec/enable-mysql-connector
Change-Id: I10c58b3af75d3ab9153a8bbd2a539bf1577de328
Closes-Bug: 1368391
Closes-Bug: 1410494
oslo_db was moved out of the oslo namespace in
oslo.db>=1.4.0.
Change-Id: I13620d26ff12c55b2294b3b051669e0e98513a10
Related-to: blueprint drop-namespace-packages
oslo_config was moved out of the oslo namespace in oslo.config>=1.6.0.
Related-to: blueprint drop-namespace-packages
Change-Id: I30ecbf8f9de77496fcb643c7ad9738d79ad359f0
We must have correct models i.e. models that correspond
a database schema to use sqlalchemy features.
Update models.py and add migration script to correct
database schema and get rid of the difference between
schema and models.
Partial-Bug: #1365436
Change-Id: I9b4571906d39bcdb35048caa58d16ad5e888cce4
The builtin _ should not longer be used. Instead _
should be explicitly assigned from glance.i18n.
This patch does the following:
(1) adds these explict assignments to any
glance file that was missing them,
(2) removes the defintion of _ as a builtin from
tox.ini so that a pep8 failure will occur
if _ is not defined,
(3) removes calls to install().
This removes the last direct use of gettextutils by glance.
The indirect uses, via openstack/common, will be removed
in the next sync with oslo.
Change-Id: Ie54e8a67e747e022cc01022dfad35d89686bdfc4
Changing all Glance files to use i18n instead of gettextutils for log
translations.
Finishes bp refactoring-glance-logging
Change-Id: I90d6ab4f7e09f4e7591921f9526de6370ebdb789
The metadef_properties and metadef_objects tables both have
a column named schema. Unfortunately, schema is a reserved word
in some relational database products, including MySQL and PostgreSQL.
The metadef_properties.schema and metadef_objects.schema
columns should be renamed to a non reserved word.
Change-Id: I9c1b497d2b09b9282a83bd8c19c32edfa4dd159f
Closes-Bug: 1378968
The MetadefProperty and MetadefObject ORM classes currently specify the
JSON schema columns as type Text. It is preferred to use the
JSONEncodedDict Type Decorator instead. This fix also includes necessary
code changes to remove JSON encoding/decoding that was previously done
in other layers. Fixes for unit tests involving the schema columns are
also included.
Change-Id: I2c574210f8d62c77a438afab83ff80f3e5bd2fe7
Closes-Bug: 1368479
Refactored migration tests to use OpportunisticTestCase, removed
unused code and ``test_migrations.conf`` file.
The main feature of this approach is to create a new database with
random name for each migration test. This will avoid migration tests of
race conditions and reduce tests intersection. After this change, database
``openstack_citest`` will be used only for initial connection to the database.
``test_migrations.conf`` file not required anymore, because we create test
database for migration test, so we no longer need to keep database credentials.
Partial-Bug: #1368274
Change-Id: Ib1d6dfae62cc60f814c01d07adc53f68e7c234f6
This commits removes the old `store` package from glance and adopts the
usage of the new glance.store library. The library was designed to
preserve backwards compatibility as much as possible. In fact, most of
the changes in this patch are related to function args ordering and not
function renames or workflow changes.
Some changes that are worth mentioning:
1. Glance store doesn't rely on a global config object. All config
options must be explicitly registered.
2. All store operations now accepted an optional context. This is a
fallout from the context not being required in the `Store` constructor
anymore.
3. Store drivers are behind a private package called `_drivers` and
they're not suppose to be accessed directly. Instead, functions like
`get_store_from_scheme` should be used.
4. Stores are disabled by default
5. All the store specific options are under the `glance_store` group.
DocImpact:
The old store related configuration options have been moved under the
`glance_store` section. However, the old options will go through a
deprecation path. That is, they'll still be read from the `DEFAULT`
section to give deployers enough time to update their config files.
In k-2, the deprecated options will be completely obsolete.
Closes-bug: #1291848
Implements-blueprint: create-store-package
Change-Id: Iaacc70993ad5da292b93de42bbecda73d53b19fd
Implements: blueprint metadata-schema-catalog
A common API hosted by the Glance service for vendors, admins,
services, and users to meaningfully define available key / value
pair and tag metadata. The intent is to enable better metadata
collaboration across artifacts, services, and projects for
OpenStack users.
This is about the definition of the available metadata that can
be used on different types of resources (images, artifacts,
volumes, flavors, aggregates, etc). A definition includes the
properties type, its key, it's description, and it's constraints.
This catalogue will not store the values for specific instance
properties.
Change-Id: I01cc63df3d3abe383c94cfb54598868b4bb729bb
DocImpact
Co-Authored-By: Lakshmi N Sampath <lakshmi.sampath@hp.com>
Co-Authored-By: Wayne Okuma <wayne.okuma@hp.com>
Co-Authored-By: Travis Tripp <travis.tripp@hp.com>
Co-Authored-By: Pawel Koniszewski <pawel.koniszewski@intel.com>
Co-Authored-By: Michal Jastrzebski <michal.jastrzebski@intel.com>
Co-Authored-By: Michal Dulko <michal.dulko@intel.com>
Since the move to oslo.db, sqlalchemy no longer raises
ProgrammingError. Instead, it is wrapped in a
oslo.db.exception.DBError. This causes migration to fail on older
postgres, which relies on catching this error to use the correct
constraint name.
Change-Id: Icf7b20eb4e09f3c91aa3df3bcea497f810a6f907
006_mysql_downgrade didn't correctly quote 'key' column of
'image_properties' table when creating index, the index wasn't created
as needed, and then operator could not run any upgrade script
successfully since 006_mysql_upgrade script could not to handle this
unexpected db status.
Note, this issue doesn't impact one-way upgrade migration operation,
since bug existed in downgrade script. But once operator do a
reciprocating migration operation, this issue will rise up as a blocker.
Closes-bug: 1351413
Change-Id: Ie1ed1b7f06b8a0aeede70c3d4dd061c857e448da
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
This change fixes translations to be using correct translation functions.
Contains few log level drops for too high logging levels.
This PS does not address logging changes for modules under glance/store/
due to the new repo request. After stores are moved to their own repo
new PS will be submitted for those.
This PS does not touch the test code.
Separate PS will be submitted for log lever raises withing codebase
as requested during the spec review.
Related to bp refactoring-glance-logging
Change-Id: I683c35face8d5b7056ef7fedcd75deb63dbbf8e2
According to https://wiki.openstack.org/wiki/Python3 dict.iteritems()
should be replaced with six.iteritems(dict).
Change-Id: Id8a87cf2ba68b37507f9238a48e8a224612a7b90
Stephen Finucane