Added expand script and model to define `cached_images` and
`node_reference` database tables.
Related blueprint centralized-cache-db
Change-Id: I9991a4e945bd4d231e942096445fd5374ff7ff2a
Added 3 new columns 'image_id', 'request_id' and 'user_id' to task
table. This is a base patch for adding new API to show task
information to the user.
Partially-Implements: blueprint messages-api
Change-Id: I4946c25375f1dc54d8f550f21c97bdf5b3005877
Added new boolean column "os_hidden" in images table. Images where
"os_hidden" = True will be omitted from the image list presented
to the user. This will apply to all image visibilities. However,
the images will continue to be discoverable. User can use
filter "os_hidden=true" in GET v2/images call to see all hidden
images.
Implements: blueprint hidden-images
Change-Id: If8f02ca94fdb8e1ac7a81853cd392988900172d1
This change replaces the existing boolean 'is_public' column for
the 'images' table with enum 'visibility' column featuring the
four explicit visibility values - public, private, shared,
and community.
This change also implements and enables all backend code to
utilize the new values.
Co-Authored-By: Timothy Symanczyk <timothy_symanczyk@symantec.com>
Co-Authored-By: Dharini Chandrasekar <dharini.chandrasekar@intel.com>
Implements: blueprint community-level-v2-image-sharing
Closes-Bug: #1394299
Closes-Bug: #1452443
Depends-On: I6e3268f3712cbc0aadb51d204c694023b92d55a5
Change-Id: I94bc7708b291ce37319539e27b3e88c9a17e1a9f
This code fixes the situation when several
models are mismatched for SQLite because of
type inconsistencies between Integer and
BigInteger in sqlalchemy.
Change-Id: I52b3a0158db8e3dc48f19509d1f9f80420ee40ea
Closes-bug: #1526804
Closes-bug: #1526675
This change introduces glance.common.timeutils that provides the
timeutils previously consumed from oslo_utils.
Oslo is deprecating some timeutils functionality which of Glance
depends on. Suggested replacement (isoformat) would break glance APIs
so it's cleaner to carry this functionality in Glance rather than
re-invent the wheel.
Co-Authored-By: Erno Kuvaja <jokke@usr.fi>
Co-Authored-By: Sabari Kumar Murugesan <smurugesan@vmware.com>
Change-Id: I91e1cc9a273249fd88749cecf21200f3f5e2bab1
Provide support in Glance API for querying the image list for
created_at and updated_at times using guidance from the API Working
Group recommendations for filtering. Filtering is applied at the DB
layer.
DocImpact
ApiImpact
MitakaPriority
Change-Id: Ie94295bb82779ec17ab773928c71ae4a9ee8fbcc
Implements bp: v2-additional-filtering
Newer versions of oslo_db added a sanity check to ensure that table's
charsets are utf8. This patch makes this change explicit and re-enables
the sanity check that was disabled in Ifffbdd50c38a1d25215faa6a966c2d2a63935711
Change-Id: Icbc41bd3964a6124d3e0d906cb85944fdc312844
Closes-bug: #1411489
The Oslo libraries have moved all of their code out of the 'oslo'
namespace package into per-library packages. The namespace package was
retained during kilo for backwards compatibility, but will be removed by
the liberty-2 milestone. This change removes the use of the namespace
package, replacing it with the new package names.
The patches in the libraries will be put on hold until application
patches have landed, or L2, whichever comes first. At that point, new
versions of the libraries without namespace packages will be released as
a major version update.
Please merge this patch, or an equivalent, before L2 to avoid problems
with those library releases.
Blueprint: remove-namespace-packages
https://blueprints.launchpad.net/oslo-incubator/+spec/remove-namespace-packages
Change-Id: Ifa8baab33cdb3e606cf175a8c29c3a4ef6c44480
oslo_db was moved out of the oslo namespace in
oslo.db>=1.4.0.
Change-Id: I13620d26ff12c55b2294b3b051669e0e98513a10
Related-to: blueprint drop-namespace-packages
We must have correct models i.e. models that correspond
a database schema to use sqlalchemy features.
Update models.py and add migration script to correct
database schema and get rid of the difference between
schema and models.
Partial-Bug: #1365436
Change-Id: I9b4571906d39bcdb35048caa58d16ad5e888cce4
Replace common oslo code glance.openstack.common.db
by usage of oslo.db library and remove common code.
Change-Id: I27216e03763f72e8665121d4440ba2828349eac5
The set_time_override function in timeutils was written as a
helper function to mock utcnow for unittests before 'mock' was
generally used. Now that we have mock and fixture, we no longer
need to use it.
Change-Id: I203f7369943b948e0462becbddcba65b2d1576f0
Partial-Bug: #1266962
This sync contains commit ``ce69e7f Don't store engine instances in oslo.db ``
which removed global engine from oslo.db code. So, added code
for work with sqla engines and sessions to Glance - get_engine()
and get_session() functions in glance.db.sqlalchemy.api module.
Also this remove database slave connection and tpool features, because
they was removed from oslo code and seeps to be unused in Glance.
Full list of changes:
7959826 db: move all options into database group
dda24eb Introduce mysql_sql_mode option, remove old warning
0b5af67 Introduce a method to set any MySQL session SQL mode
8dccc7b Handle ibm_db_sa DBDuplicateEntry integrity errors
0f24d82 Fix migration.db_version when no tables
ac84a40 Update log translation domains
c0d357b Add model_query() to db.sqlalchemy.utils module
84254fc Fix a small typo in api.py
b8a676c Remove CONF.database.connection default value
86707cd Remove None for dict.get()
0545121 Fix duplicating of SQL queries in logs
fcf517d Update oslo log messages with translation domains
fa05b7c Restore the ability to load the DB backend lazily
630d395 Don't use cfg.CONF in oslo.db
ce69e7f Don't store engine instances in oslo.db
35dc1d7 py3kcompat: remove
b4f72b2 Don't raise MySQL 2013 'Lost connection' errors
271adfb Format sql in db.sqlalchemy.session docstring
0334cb3 Handle exception messages with six.text_type
eff69ce Drop dependency on log from oslo db code
7a11a04 Automatic retry db.api query if db connection lost
11f2add Clean up docstring in db.sqlalchemy.session
1b5147f Only enable MySQL TRADITIONAL mode if we're running against MySQL
39e1c5c Move db tests base.py to common code
986dafd Fix parsing of UC errors in sqlite 3.7.16+/3.8.2+
9a203e6 Use dialect rather than a particular DB API driver
1779029 Move helper DB functions to db.sqlalchemy.utils
bcf6d5e Small edits on help strings
ae01e9a Transition from migrate to alembic
70ebb19 Fix mocking of utcnow() for model datetime cols
7aa94df Add a db check for CHARSET=utf8
aff0171 Remove "vim: tabstop=4 shiftwidth=4 softtabstop=4" from headers
fa0f36f Fix database connection string is secret
8575d87 Removed copyright from empty files
d08d27f Fix the obsolete exception message
8b2b0b7 Use hacking import_exceptions for gettextutils._
9bc593e Add docstring for exception handlers of session
855644a Removal of _REPOSITORY global variable.
ea6caf9 Remove string.lowercase usage
a33989e Remove eventlet tpool from common db.api
e40903b Database hook enabling traditional mode at MySQL
f2115a0 Replace xrange in for loop with range
c802fa6 SQLAlchemy error patterns improved
1c1f199 Remove unused import
97d8cf4 Remove lazy loading of database backend
Co-authored-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
Related to blueprint db-use-oslo-common-code
Change-Id: Ia73abba8309ccc2ad10a0f636b410984c6b6e5d8
In order to distinguish file size from the virtual image size, we need
to add a new field to the Image model to represent that. This patch adds
a new field called `virtual_size` to the Image model and the migration
for it.
The existing size attribute is kept as is in order to keep backwards
compatibility. Most of the users consider that attribute to be the
physical image size, which is what we're trying to achieve here.
The API implementation will be done in a follow-up patch.
Partially-Implements bp: split-image-size
Change-Id: Ie968e8bcfaeda79de003fe8fd7a3b58c3e45f592
Adding a status field to image's each location property, each location
status can be 'active', 'pending_delete' and 'deleted'.
Under location's status information Scrubber service can make cleanup
based on DB records also but not a dedicated queue-file for each image.
This is first part of this change which covered DB core migration.
Partially-Implements BP: image-location-status
Change-Id: I013b70d55ef14a3ceaca962f9bc48296cb8b2552
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
Every call to json.loads/json.dumps have been changed to
jsonutils.loads/jsonutils.dumps respectively. Import json has been removed
also replaced by import glance.openstack.common.jsonutils
654d80b416dc5f413cb791aa838ec8688bf7da44 Create openstack.common.jsonutils
Change-Id: I8ef580e5eb91526dfaef37050ce1f5c6d88d56b5
Closes-bug: #1257922
Use common oslo database code for work with session, connection, engines
instead of glance implementations.
Work with config options modified due to new common code config.
Removed unused glance code and tests for it.
Details:
glance/cmd/manage.py
- removed load load the glance-registry config. We load sql connection
in common code now.
glance/db/sqlalchemy/api.py
- removed functions _wrap_db_error(), _is_db_connection_error(),
_get_maker(), get_engine(), _ping_listener() and config
`sql_connection_opt`. We have all these thigs in module
`glance.openstack.common.db.sqlalchemy`.
- modified functions setup_db_env(), add_cli_options(), clear_db_env().
- functions _get_session() and get_engine() now call same functions
from module `glance.openstack.common.db.sqlalchemy`.
- replaced sqlalchemy exception `sqlalchemy.exc.IntegrityError` to
database duplicate exception - `db_exc.DBDuplicateEntry`.
Removed redundant tests - we use common code , so a lot of features was
already tested in Oslo.
Global variable _ENGINE was replaced by common function get_engine().
DocImpact
New database config options came from Oslo. See Table below
-------------------------------------------------------------------
Database config options ([group] option)
-----------------------------|-------------------------------------
Glance | Oslo
-----------------------------|-------------------------------------
[DEFAULT] sql_connection | [database] connection
[DEFAULT] sql_idle_timeout | [database] idle_timeout
[DEFAULT] sql_max_retries | [database] max_retries
[DEFAULT] sql_retry_interval | [database] retry_interval
[DEFAULT] db_auto_create | NONE
[DEFAULT] sqlalchemy_debug | [database] connection_debug
[DEFAULT] use_tpool | [database] use_tpool
NONE | [database] slave_connection
NONE | [database] min_pool_size
NONE | [database] max_pool_size
NONE | [database] slave_connection
NONE | [database] min_pool_size
NONE | [database] max_pool_size
NONE | [database] max_overflow
NONE | [database] connection_trace
NONE | [database] pool_timeout
-------------------------------------------------------------------
blueprint db-use-oslo-common-code
Change-Id: I3ff976545b1a82ff8df780e34128fcaf6f892b8c
Each project should directly use the standard uuid module.
uuidutils will be deprecated/removed in this cycle.
This patch replaces every uuidutils.generate_uuid() with
str(uuid.uuid4()) and uuidutils.is_uuid_like()
with utils.is_uuid_like().
Change-Id: I43642d4f1e137c14134b3d544e367b504b9851ac
Closes-Bug: #1253497
The tasks table scheme in migration for adding tasks to Glance DB has a
flag for disabling from setting a NULL owner.
In order to keep the behavior consistent across the glance codebase,
this patch adds owner as not nullable in the models.py.
Fixes bug: 1257949
Change-Id: Id9fff4277f427b7d83d80b5a0a8029968f70e0df
- move 'input', 'message' and 'result' columns to table 'task_info'
- introduce migration script for the above db changes.
- appropriate tests are included
Co-authored-by: Nikhil Komawar <nikhil.komawar@rackspace.com>
partly implements bp async-glance-workers
Change-Id: I9867d609f4729572b72a44d1f05e353acf6c98d9
Modified common model class - added common `models.ModelBase` and
`models.TimestampMixin` classes as base class instead of object.
Renamed base model class from `ModelBase` to `GlanceBase`.
Removed duplicated code.
blueprint db-use-oslo-common-code
Change-Id: I6038adcfcb6b985736ac8d9374ac0a2fe1cdcc96
Instead of globally ignoring Pyflakes and Hacking
warnings, blacklist explitly only those that trigger
numerously. Fix the rest alongway and start to gate
on those that are now passing.
Change-Id: Ia19dc708cf0ccab2da5b46d1652f16e901499c24
Add tasks data model and table for async worker to
store the tasks data persistently.
Partially implement blueprint async-glance-workers
Change-Id: I2b2d70295da4905bf75cffc2b2eadec174d1abfc
Sometimes, user want to query images by tag since it's an
intuitive way to distinct different image. The REST API
would be like: v2/images?tag=x86&tag=64bit
docImpact
Implement bp image-query-by-tag
Change-Id: I42946fa838ed896823e17fd4c51995209d9d2ad3
Models should be the same as the state in DB after migration
blueprint db-sync-models-with-migrations
Change-Id: Iea57866206776c14677dfa78e868143b6fb153a9
checksum image property will be indexed so that users can search for an
image by specifying the checksum
Change-Id: I31543afed31512f8f5f613640012bc7c1d7ea993
Implements: blueprint index-using-checksum-image-property
This patch adds a column to the image_locations table. The column
will store location specific information. As a location is added
to an image there may be some meta data about that information that
needs to be stored with it. For example, if it is a file:// URL,
information about the NFS server (host name, mount point) would be
needed in order for the URL to be useful.
blueprint: direct-url-meta-data
blueprint: multiple-image-locations
Change-Id: I6a96163abc8e4388e84cae2a8952f89d433210ff
This patch implements registry's API v2 based on the blueprint
registry-api-v2.
Major changes:
* The new registry API is implemented using a small rpc protocol
over http. Resources must be registered and the exposed methods
will be automatically discovered (they must start with rpc_).
* Registry's API now returns data as returned by the db-api. The
idea behind this is to provide a db-like api that can be used as a
db-driver leaving to glance-api the responsibility for cleaning
up what will go out to the client.
Minor changes:
* Split unit/v1/test_api.py in 2 separate test files (for
consistency with v2 unit)
* Added test_registry to unit/v2/
Implements blueprint registry-api-v2
Change-Id: Ia9f6eb60f52bb093e807a8a89d4df89fb80c4336
Currently, the db_api defines some functions that are used for accessing
images data but some of them are for internal use only. In an effort to
improve consistency, I've made some of those functions - the ones that
are not accessed outside the db_api - private by prepending their names
with an "_".
Most of this functions were defined within sqlalchemy and as such didn't
required any change in the simple driver, others though, required a
change in both drivers, i.e. get_session.
This patch implements part of the blueprint registry-api-v2.
NOTE: During the review, try to find functions that should be private
and haven't been changed.
Change-Id: I6342dc94ebf2a35d6bc9bd7698b81a94ffa13373
Fixes bug #1098700.
Previously, deleting then re-creating an image member failed with
500 ServerError due to the registry service failing to commit
the update to the image_members table.
The problem arose because the declared uniqueness constraint on that
table only spanned the image_id and member columns, but did not take
into account that deleted rows are left in situ with the deleted_at
and deleted columns set as appropriate. Hence the unique constraint
was violated by the addition of the new row.
We modify the uniqueness constraint via a migration script to take
account of the deleted_at column also (NULL for live memberships).
Note that the new ability to recreate an image membership cannot be
be asserted in a functional test, as these tests are based on sqlite
which does not support 'ALTER TABLE DROP CONSTRAINT'.
Change-Id: I175801ba95ecd9295791bb12e9096f59efa19c02
This index was added in migration 008. However, it was not on the model,
which meant any autocreated db would not have it.
Fixes bug 1154777
Change-Id: I34590ffebaf547be9a12e5a85216cda145d9e8db
Fix running test_migration solely.
api.py depends on models.py that depends on api.py.
Import statements in these files are such that api.py
must be imported before models.py. That's why usually
test_migrations doesn't fail when all tests are launched.
However, thus test_migration depends on test running order
making testing procedure unreliable, including parallel testing.
Move glance.db.sqlalchemy.api import to the only place
where it's required in models.py
Change-Id: Ib20dde8edd3473595aac1ca8bbaa5a0d3116a697
Fixes: bug #1154058
Migration 19 moves image location data from images.location to
the new image_locations table. The sqlalchemy db driver has been
updated to look at this new table, but the complexity of multiple
locations is not yet exposed to the reset of the system.
Related to bp multiple-image-locations
Change-Id: I6c4c684f522d9b41b9930e7f79a4e6fb3e3f5c39
Abhishek Kekane