Added required database API methods for cache related
operations.
Related blueprint centralized-cache-db
Change-Id: Iae1f1927566f4bda74c5aeccf464df820cfae9d4
This aligns the metadef DB APIs with the rest of the APIs and simplifies
our life when we decide to move off the legacy engine facade in the
future.
Change-Id: Ifc1605e8e89c48b071ea505e40a9488df208fb37
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Align '_image_update' with 'image_update'. This caught me out once or
twice.
Change-Id: I78e005b51485a224e59c075a00798de5fa5a3896
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
We already got rid of implicit autocommit. Now get rid of explicit
autocommit. This is far more involved than the prior patch but the basic
strategy here is that no public method should take a 'session' argument
and instead should call 'get_session'. In addition, all private methods
should take both a 'context' and 'session' argument and they should be
the first and second arguments, respectively. Finally, public methods
should only call private methods. Combined, this ensures we only ever
have one session in play and the transactions are correctly started and
ended as expected.
Change-Id: I5245c402699e179c219039514b626a52fa0028d8
Resolve the following SAWarning warning:
Class DeleteFromSelect will not make use of SQL compilation caching as
it does not set the 'inherit_cache' attribute to ``True``. This can
have significant performance implications including some performance
degradations in comparison to prior SQLAlchemy versions. Set this
attribute to True if this object can make use of the cache key
generated by the superclass. Alternatively, this attribute may be set
to False which will disable this warning.
This uses different arguments so the cache presumably does not make
sense.
Change-Id: I958ba79ca7ac6709e8c16eceb212f329c807aea8
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Resolve the following RemovedIn20Warning warning:
The current statement is being autocommitted using implicit
autocommit, which will be removed in SQLAlchemy 2.0. Use the .begin()
method of Engine or Connection in order to use an explicit transaction
for DML and DDL statements.
This is the trickiest one hence it's left to last. We really need to get
off of the legacy EngineFacade pattern since it's harder to track
transactions when they're not local (the request context-based binding
of sessions allows us to decorate individual methods) but that's a
larger work item that we're not going to tackle here.
Change-Id: Ie16965a8e3e2f316b807068a868cee6fbb2f33ba
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
We were doing this adhoc in tests, which doesn't seem to be possible in
SQLAlchemy 2.0. Do it everywhere instead.
This highlights a small issue that we have to workaround in the tests.
Change-Id: I9dd70c9976f83a00a8cb270422e545ff2c77f12f
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Resolve the following RemovedIn20Warning warning:
The ``aliased`` and ``from_joinpoint`` keyword arguments to
Query.join() are deprecated and will be removed in SQLAlchemy 2.0.
This one is a little trickier to grok, but the SQLAlchemy docs explain
it rather well [1] (Look for "Legacy Features of Query.join()" and
"Joins to a Target with an ON Clause"). Essentially, because we are
joining e.g. the ImageTag table multiple times, we need to create an
alias and use it for both the join and resulting queries.
[1] https://docs.sqlalchemy.org/en/14/orm/query.html#sqlalchemy.orm.Query.join
Change-Id: Ifb818df8ca8500adcf1354eff0671f5762473110
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Resolve the following RemovedIn20Warning warnings:
The "whens" argument to case(), when referring to a sequence of items,
is now passed as a series of positional elements, rather than as a
list.
Change-Id: I4df3a52333d71809d6ff305a786ba6ae03a230ca
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Resolve the following RemovedIn20Warning warnings:
Invoking and_() without arguments is deprecated, and will be
disallowed in a future release. For an empty and_() construct, use
and_(True, *args).
Change-Id: I7f34d8079975e57df904e59e325bee21ca1417ed
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Resolve the following RemovedIn20Warning warnings:
The MetaData.bind argument is deprecated and will be removed in
SQLAlchemy 2.0.
The ``bind`` argument for schema methods that invoke SQL against an
engine or connection will be required in SQLAlchemy 2.0.
We are also able to resolve the following RemovedIn20Warning warning,
since the migration away from the bind parameter requires updating these
calls.
The autoload parameter is deprecated and will be removed in version
2.0. Please use the autoload_with parameter, passing an engine or
connection.
Similarly, we can also resolve the following RemovedIn20Warning warning,
since the calls we're updating here are the only ones affected by this
deprecation:
The insert.values parameter will be removed in SQLAlchemy 2.0. Please
refer to the Insert.values() method.
Change-Id: Ic956980a03e1a0e1b6f90c492d8a03a64ea275cc
Resolve the following RemovedIn20Warning warning:
The legacy calling style of select() is deprecated and will be removed
in SQLAlchemy 2.0. Please use the new calling style described at
select().
Change-Id: Iaebe4bf97b99a0f8f3bb8983ee0a3748f7cd482a
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
These methods were not being tested. Add some basic tests to validate
loading and unloading of our stock definitions, preventing regressions
during e.g. the SQLAlchemy 2.0 preparation series.
Implementation of this test highlights an minor issue in the DB API,
whereby the default values for some methods with 'sort_key' and
'sort_dir' parameters are not valid. This means you can't call these
without overriding the parameters. This is corrected.
Change-Id: I6e76dcacc32f05a70757df28ab2f119b25da440b
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
'session' is a widely used variable name in the glance.db.sqlalchemy.api
module and this is only going to increase over time. This shadowing
prevents easy identification of e.g. missing function parameters. Alias
the import and update the one call to avoid the need to rename all other
instances of the variable name.
Change-Id: I6b2cbf0b97bb3af7354f2c10a8d83909db963d43
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
In 2.6.0, pycodestyle is going to find more E741 issues[1], which is
going to break the gate by making "tox -epep8" fail.
[1] https://www.flake8rules.com/rules/E741.html
Change-Id: I94ee03a00dd01726de15a29424183faabdeba844
When we want to purge all deleted rows
we use "glance-manage db purge" or
"glance-manage purge_images_table"
then we never know how many deleted rows are still in a table.
So we need to launch the command many times until
the command reports that 0 rows were deleted in every table.
It's inconvenient.
The patch introduced new valid value for "max_rows".
If "max_rows" equals -1 all deleted rows are purged.
Closes-Bug: 1702445
Change-Id: Ia9811d93b1c0b67a1b29a9e8ee3e0fc098c57d4a
It was observed that md-tag-create-multiple
(/v2/metadefs/namespaces/{namespace_name}/tags) API overwrites
existing tags for specified namespace rather than creating new one
in addition to the existing tags.
This patch resolves the issue by introducing a header 'X-Openstack-Append'
which on being True will append the new tags to existing ones and
if False will continue to overwrite the tags.
Implements: blueprint append-tags
Closes-Bug: #1939169
Change-Id: I29448746b14c542e5fbf0283011968ae1516642e
In set_property_atomic() we pass an integer query filter for the
deleted field, since most projects use an integer for this column.
However, in glance the column is a boolean, which trips up postgres
since the types are different (mysql and sqlite work fine). This
minor change to use False instead of 0 should fix that for postgres
users.
Change-Id: I5149df76943c1c19f3204b904c0e2d3ef846bdf7
Closes-Bug: #1953063
This is the same as the 'range' keyword in Python 3
Change-Id: If3aa008522c24e870b7bf13de32b8ed1b27cb519
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
We also remove hacking tests for this, along with those for iterkeys and
itervalues (no usage of these).
Change-Id: If5b46580078eb756651ac6118f502eccdc693646
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
This adds an efficient query method for getting a count of the user's
images that are in any state of uploading. This includes images
in the process of saving from upload, staging, importing, or being
copied to another store.
Partially-implements: blueprint glance-unified-quotas
Change-Id: Iebba15771000e86a0701d20ed9620eb578c2cb85
This adds an efficient query method for getting a count of a user's
non-deleted images. It can be used by subsequent patches to enforce
the image count quota.
Partially-implements: blueprint glance-unified-quotas
Change-Id: I4d6e071b8d48862032f36c80ee19632a3fe85009
This adds a method that counts the sizes of images that are in the
process of doing an import, indicating that they are using staging
storage resources. We gather images that are in obvious states of
being created via import, as well as any images that appear to be
active but in the middle of a copy, since those could be using staging
resources as well.
This is somewhat mirrored from user_get_storage_usage(). Since there
was no test for that method, this adds a test case for both that and
the new method.
Partially-implements: blueprint glance-unified-quotas
Change-Id: Ic2b045549ac0d6112b0c565d2e993576a5626799
Occasionally we fail to delete a property from an image and get a
StaleDataError from SQLAlchemy. It looks like maybe we are just racing
with another operation to delete a single property (i.e. a race
between image delete and the end of an import cleanup).
This ignores that "can't delete because it is already deleted" error
and logs to debug about the situation for later auditing and
confirmation.
Change-Id: Ida5bc659b7481fa0a7f3926a54520628a7a4c406
Closes-Bug: #1895173
A slight error in the tasks_get_by_image() DB API method resulted in
our excluding in-progress tasks from the returned list. This is
because those tasks have expires_at=NULL, and we were comparing
the expires_at>=$NOW to find unexpired tasks. This makes us check for
"NULL or not expired" instead.
We did have a test asserting the wrong behavior, but it was done to
increase coverage and thus was asserting the behavior of the code and
not the *desired* behavior. This fixes that as well.
Closes-Bug: #1922928
Change-Id: I1b6971888673b64ef60bed8fbcc97bbcbcf5c2ac
Added new API /v2/images/{id}/tasks to show tasks associated with
image. This API will return list of tasks associated for valid image
else returns 404 not found if image is not present. This API also
initiates task scrubbing before returning tasks to user.
Implements: blueprint messages-api
Change-Id: Ib3cacb4dd4d75de32e539f8a3b48bdaa762e6d8e
Made provision to pass image_id, request_id and user_id information
while creating new task.
Partially-Implements: blueprint messages-api
Change-Id: I299a222eeef81431143db3ba7fc08365c924326b
The image_update() code will clobber, revert, or update lock values
in keys that we use as atomic properties. This adds an exclusion
list of properties that we handle specially and plumbs them down
to image_update() so that they will be excluded from the
add/update/delete logic.
Change-Id: Ib910274472346ce0c336cd1ead8370d5799d0b96
This adds a new DB API method to atomically delete a property on an image
in a way that we can be sure that it is deleted one and only once, and
without affecting the rest of the image. This can be used in conjunction
with image_set_property_atomic() to create and delete properties for locking
without the risk of clobbering other image data in the process.
Change-Id: I0b71a7df04cd330749f35b07f96a120b49b412c7
This adds a new DB API method to atomically create a property on an image
in a way that we can be sure it is created once and only once for the
purposes of exclusion of multiple threads.
Change-Id: Ifdb711cb241ef13eccaa5ae29a234f2fe4a52eb8
Related-Bug: #1884596
First force purging of records that are not soft deleted but are
referencing soft deleted tasks/images records (e.g. task_info records).
Then purge all soft deleted records in glance tables in the right order
to avoid FK constraint violation.
Closes-Bug: #1803643
Change-Id: I1c471adce002545f8965a57ef78a57e1e3031ef0
Co-authored-by: Tee Ngo <tee.ngo@windriver.com>
Signed-off-by: Liang Fang <liang.a.fang@intel.com>
Added new boolean column "os_hidden" in images table. Images where
"os_hidden" = True will be omitted from the image list presented
to the user. This will apply to all image visibilities. However,
the images will continue to be discoverable. User can use
filter "os_hidden=true" in GET v2/images call to see all hidden
images.
Implements: blueprint hidden-images
Change-Id: If8f02ca94fdb8e1ac7a81853cd392988900172d1
Modified the current ``glance-manage db purge`` command to eliminate images
table from purging the records. Added new command
``glance-manage db purge_images_table`` to purge the records from images
table.
DocImpact
SecurityImpact
Change-Id: Ie6641659b54543ed9f96c393d664e52a26bfaf6a
Implements: blueprint mitigate-ossn-0075
Now there is no way to revert the pending-delete images. Once the
admin operators want to revert the delete action, Glance should
give them this kind of ability.
This patch will enhance the glance-scrubber tool to support restoring
the image from pending-delete to active.
Change-Id: I11fe58403e3af102b63d15b3cc702e567e526bad
blueprint: pending-delete-rollback
At this layer, it's a boolean filter; we should let the ORM
make the translation for whatever backend is in use. MySQL
didn't care, but PostgreSQL is a bit more picky.
Co-authored-by: Scott McClymont <scott.mcclymont@verizonwireless.com>
Co-authored-by: Brian Rosmaita <rosmaita.fossdev@gmail.com>
Change-Id: Id1e5d2cec3be48f6a5164582591f2fd2f565a96b
Closes-bug: #1749297
If user passes member id as unicode characters in member create then
HTTP 500 internal server error is raised.
Reason: The unicode format check is not performed in db create member.
This patch fixes the member create by checking member id before
inserting in db. If member id is unicode then proper exception
is raised and same is handled in controller api.
Change-Id: I67be5e990d1269cbb986db7fff21a90a41af06e4
Closes-Bug: #1688189
Adds a RequestDeserializer for the stage and adjusts the image
status transitions so that they can handle the 'uploading' status
of an image with data in the stage.
Closes-bug: #1710958
Change-Id: I6f1cfe44a01542bc93a43cbd518956686adb366d
Since the default value is None when can't get a key from a dict,
So there is no need to use dict.get('key', None).
Change-Id: Ie8008e83f432263e7a97e482e83f240b1056feee
'glance-manage db purge' command fails with DBReferenceError due
to FK constraint failure and exits with stack-trace on the command
prompt.
Made changes to give user-friendly error message to the user as
well as log appropriate error message in glance-manage logs
instead of stack-trace.
Co-author-by: Dinesh Bhor <dinesh.bhor@nttdata.com>
Change-Id: I52e56b69f1b78408018c837d71d75c6df3df9e71
Closes-Bug: #1554412
Glare became a separate project during Newton. The code was copied
out of the Glance tree, but remained in the Glance repository. It
is no longer being maintained, and that has begun to cause some
problems, for example, blocking a recent stevedore upper constraints
change; see I141b17f9dd2acebe2b23f8fc93206e23bc70b568
This patch removes the Glare code from the Glance repository. It
includes Alembic database migration scripts, in both the all-in-one
and E-M-C format. It also includes release notes.
Implements: blueprint glare-ectomy
Change-Id: I3026ca6287a65ab5287bf3843f2a9d756ce15139
Zuul