We already got rid of implicit autocommit. Now get rid of explicit
autocommit. This is far more involved than the prior patch but the basic
strategy here is that no public method should take a 'session' argument
and instead should call 'get_session'. In addition, all private methods
should take both a 'context' and 'session' argument and they should be
the first and second arguments, respectively. Finally, public methods
should only call private methods. Combined, this ensures we only ever
have one session in play and the transactions are correctly started and
ended as expected.
Change-Id: I5245c402699e179c219039514b626a52fa0028d8
Resolve the following RemovedIn20Warning warning:
The current statement is being autocommitted using implicit
autocommit, which will be removed in SQLAlchemy 2.0. Use the .begin()
method of Engine or Connection in order to use an explicit transaction
for DML and DDL statements.
This is the trickiest one hence it's left to last. We really need to get
off of the legacy EngineFacade pattern since it's harder to track
transactions when they're not local (the request context-based binding
of sessions allows us to decorate individual methods) but that's a
larger work item that we're not going to tackle here.
Change-Id: Ie16965a8e3e2f316b807068a868cee6fbb2f33ba
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
We were doing this adhoc in tests, which doesn't seem to be possible in
SQLAlchemy 2.0. Do it everywhere instead.
This highlights a small issue that we have to workaround in the tests.
Change-Id: I9dd70c9976f83a00a8cb270422e545ff2c77f12f
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Resolve the following RemovedIn20Warning warning:
Passing a string to Connection.execute() is deprecated and will be
removed in version 2.0. Use the text() construct, or the
Connection.exec_driver_sql() method to invoke a driver-level SQL
string.
We also resolve two additional RemovedIn20Warning warnings along the
way, since we're already touching this code:
The Engine.execute() method is considered legacy as of the 1.x series
of SQLAlchemy and will be removed in 2.0. All statement execution in
SQLAlchemy 2.0 is performed by the Connection.execute() method of
Connection, or in the ORM by the Session.execute() method of Session.
The Executable.execute() method is considered legacy as of the 1.x
series of SQLAlchemy and will be removed in 2.0. All statement
execution in SQLAlchemy 2.0 is performed by the Connection.execute()
method of Connection, or in the ORM by the Session.execute() method of
Session.
Change-Id: I547bd2f441085fcdc5dc4a0e58d27d11e8a8b223
In 2.6.0, pycodestyle is going to find more E741 issues[1], which is
going to break the gate by making "tox -epep8" fail.
[1] https://www.flake8rules.com/rules/E741.html
Change-Id: I94ee03a00dd01726de15a29424183faabdeba844
This is the same as the 'range' keyword in Python 3
Change-Id: If3aa008522c24e870b7bf13de32b8ed1b27cb519
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
A slight error in the tasks_get_by_image() DB API method resulted in
our excluding in-progress tasks from the returned list. This is
because those tasks have expires_at=NULL, and we were comparing
the expires_at>=$NOW to find unexpired tasks. This makes us check for
"NULL or not expired" instead.
We did have a test asserting the wrong behavior, but it was done to
increase coverage and thus was asserting the behavior of the code and
not the *desired* behavior. This fixes that as well.
Closes-Bug: #1922928
Change-Id: I1b6971888673b64ef60bed8fbcc97bbcbcf5c2ac
Added new API /v2/images/{id}/tasks to show tasks associated with
image. This API will return list of tasks associated for valid image
else returns 404 not found if image is not present. This API also
initiates task scrubbing before returning tasks to user.
Implements: blueprint messages-api
Change-Id: Ib3cacb4dd4d75de32e539f8a3b48bdaa762e6d8e
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib.
Change-Id: I44e7b6f76e2d12f620ec602afc77ce11ba6b9d9a
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
We were capped at a very old version of hacking. Hacking itself caps the
various linters it uses to remain consistent, so our pep8 job was not
checking quite a bit that current versions have added.
This raises that limit to the latest to get up to the level of other
projects and addresses the errors the updated linters uncovered.
Change-Id: I89a9d73fbd59606a649e26077acebc5c42873d67
Co-authored-by: Sean McGinnis <sean.mcginnis@gmail.com>
Some tests compare timestamps (e.g. image created_at/updated_at
fields).
In some cases, subsequent timestamps may be identical, breaking the
assertions made by those tests.
One idea is to just add a 1ms sleep which should be a negligeable.
Mocking the functions used to retrieve timestamps may be undesireable.
For now, we'll do this only for Windows, where time.time has a lower
resolution compared to Linux (1e-7 as opposed to 1e-9).
At the same time, socket timeouts are rather inconsistent on Windows.
When a timeout is hit, recv may either throw ConnectionAbortedError
*or* return nothing. This needs to be taken into account when
expecting socket timeouts.
Change-Id: Ie5a4d8fb4c979c09eda2fdc0fad0baa1d1840c59
In order to run the unit and functional Glance tests on Windows, we
have to:
* avoid monkey patching the os module on Windows (which causes Popen
to fail)
* update sqlite connection URL
* avoid os.fork, not available on Windows.
* we'll use subprocess.Popen when spinning up http servers.
* for the really simple ones defined in the test helpers, we'll just
use threads
* do not attempt to connect to '0.0.0.0', use '127.0.0.1' instead
* some tests aren't properly skipped (xattr ones), so we're covering that
as well
* skip log rotation test, we can't move in-use files. Log rotation can
be performed by the log handler itself.
* expect an exception when hitting connection timeouts
* avoid installing unavailable test requirements (xattr, pysendfile)
* pin the instance creation timestamp. some tests that deal with
markers rely on ordering, which can be flipped if the timestamps are
identical (can happen in case of resources created one after the
other, not sure yet if this happens really fast or the clock isn't
accurate enough).
* add a few seconds to some timeouts (much needed when running the tests
in VMs).
blueprint windows-support
Change-Id: Ife69f56a3f9f4d81e1e2e47fde4778efd490938f
RequestContext.tenant and user fields are deprecated in favor
of project_id and user_id respectively.
This change modifies the glance.context.RequestContext constructor
to transition usage of tenant/user to project_id/user_id until
all tests are moved over to the new attributes. Runtime usage of
the old fiels is updated.
To prevent new code from using the deprecated fields, a warnings
filter is added which will make tests fail if they hit code using
the old fields.
Co-Authored-By: Abhishek Kekane <akekane@redhat.com>
Change-Id: I351380840308a24769ece93abc6d1a9a6d6aa06f
First force purging of records that are not soft deleted but are
referencing soft deleted tasks/images records (e.g. task_info records).
Then purge all soft deleted records in glance tables in the right order
to avoid FK constraint violation.
Closes-Bug: #1803643
Change-Id: I1c471adce002545f8965a57ef78a57e1e3031ef0
Co-authored-by: Tee Ngo <tee.ngo@windriver.com>
Signed-off-by: Liang Fang <liang.a.fang@intel.com>
Modified the current ``glance-manage db purge`` command to eliminate images
table from purging the records. Added new command
``glance-manage db purge_images_table`` to purge the records from images
table.
DocImpact
SecurityImpact
Change-Id: Ie6641659b54543ed9f96c393d664e52a26bfaf6a
Implements: blueprint mitigate-ossn-0075
Implements v2 API image-list filtering on the 'protected' property
following the spec:
https://git.openstack.org/cgit/openstack/glance-specs/tree/specs/pike
/approved/glance/add-protected-filter.rst
The 'protected' query string parameter accepts one of 'true', 'false'.
Any other value raises a 400 with an appropriate message.
See the discussion on https://review.openstack.org/#/c/451560/
for why this implementation strategy was chosen.
Change-Id: Ie90d952edddd3f69b3f489f056ff506e63f96376
Closes-Bug: #1674246
'glance-manage db purge' command fails with DBReferenceError due
to FK constraint failure and exits with stack-trace on the command
prompt.
Made changes to give user-friendly error message to the user as
well as log appropriate error message in glance-manage logs
instead of stack-trace.
Co-author-by: Dinesh Bhor <dinesh.bhor@nttdata.com>
Change-Id: I52e56b69f1b78408018c837d71d75c6df3df9e71
Closes-Bug: #1554412
This change replaces the existing boolean 'is_public' column for
the 'images' table with enum 'visibility' column featuring the
four explicit visibility values - public, private, shared,
and community.
This change also implements and enables all backend code to
utilize the new values.
Co-Authored-By: Timothy Symanczyk <timothy_symanczyk@symantec.com>
Co-Authored-By: Dharini Chandrasekar <dharini.chandrasekar@intel.com>
Implements: blueprint community-level-v2-image-sharing
Closes-Bug: #1394299
Closes-Bug: #1452443
Depends-On: I6e3268f3712cbc0aadb51d204c694023b92d55a5
Change-Id: I94bc7708b291ce37319539e27b3e88c9a17e1a9f
Currently there is no mechanism for deleting tasks on regular basis.
This patch adds a new function that is called in the tasks_get_all
function, so that everytime tasks lists are called, the function
checks if tasks in the database have surpassed the expired_at value.
If that is the case, then it sets the deleted value as 1 for all the
expired tasks.
Co-Authored-By: Mike Fedosin <mfedosin@mirantis.com>
Implements https://review.openstack.org/#/c/324648/
Change-Id: I0bde982de948901f6bfbfab9e57cf84891c22052
Instead of using assertTrue(A * B), developers should
use assertGreater(A, B) or assertLess(A, B) or
assertGreaterEqual(A, B) or assertLessEqual(A, B)
or assertEqual(A, B).
The * operator: >=, <=, ==
Change-Id: I85e0087c530c2f1132061036fd5427368dfa2316
Added support filtering images based on lists using the 'in' operator.
Filters:
*id
*name
*container_format
*disk_format
*status
DocImpact
ApiImpact
Implements bp: in-filtering-operator
Change-Id: I9cac81b9d5cbec979e88cf2dd0e3b710ed45630c
If you replace the membership list of an image with some members
that were previously deleted, we fail to re-add those deleted
members.
This seems to be a consequence of a previous commit
d6800e143d which failed to update
the deleted records.
Closes-Bug: 1527143
Change-Id: Ic3ca9b56712a99652e65cb0a4e3f1e0ba15b8593
Builtin function 'reduce' in Python 2 has been moved to standard
library module in Python 3 [1]. To make code compatible, we should
replace reduce(expr) with six.moves.reduce(expr)
[1] http://python3porting.com/stdlib.html#moved-builtins
Change-Id: Idca3420918471c41548a871d043fafc983aa7ee6
Closes-Bug: #1530249
This change introduces glance.common.timeutils that provides the
timeutils previously consumed from oslo_utils.
Oslo is deprecating some timeutils functionality which of Glance
depends on. Suggested replacement (isoformat) would break glance APIs
so it's cleaner to carry this functionality in Glance rather than
re-invent the wheel.
Co-Authored-By: Erno Kuvaja <jokke@usr.fi>
Co-Authored-By: Sabari Kumar Murugesan <smurugesan@vmware.com>
Change-Id: I91e1cc9a273249fd88749cecf21200f3f5e2bab1
When use 'sort-key=size' for multi-pages searching with postgreSQL
DB, it will raise an error: DBError: (psycopg2.DataError). And return
HTTPInternalServerError (HTTP 500) to users. The reason is that
postgreSQL doesn't support the size's default value to be string.
This patch set the size's default value to zero.
Change-Id: Ib3b5f3f57be3683ba274f0122e6314978a79e75f
Closes-bug:#1481154
Provide support in Glance API for querying the image list for
created_at and updated_at times using guidance from the API Working
Group recommendations for filtering. Filtering is applied at the DB
layer.
DocImpact
ApiImpact
MitakaPriority
Change-Id: Ie94295bb82779ec17ab773928c71ae4a9ee8fbcc
Implements bp: v2-additional-filtering
Utilizing assertIn and assertNotIn in a consistent way rather than
assertTrue( x <not> in y) and assertFalse(x in y).
Change-Id: Ic43d107c40a5a50b852ee476b1167c21e5ad86e8
The default value of 'locations' is [].So there is a wrong check in
db/sqlalchemy/api.py.
When create an image without data, The function '_image_locations_set'
should not be executed.
Change-Id: Ie4cb29ab0d714d3ce67f717d5eaf641424efa5b7
Closes-bug:#1471080
Image params such as checksum, min_ram and min_disk are validated.
Checksum is expected to be not longer than 32 characters,
min_ram/min_disk can't exceed pow(2, 31) - 1.
A method for mysql_int validation has been moved to glance.common.utils.
Change-Id: I78554ef1ba65fc11da3de56467d6d70bb7991787
Closes-Bug: #1454730
Closes-Bug: #1460060
PEP-0274 introduced dict comprehensions to replace dict constructor
with a sequence of key-pairs[1], these are two benefits:
- First, it makes the code look neater.
- Second, it gains a micro-optimization.
Glance dropped python 2.6 support in Kilo, we can leverage this now.
Note: This commit doesn't handle dict constructor with kwargs.
This commit also adds a hacking rule.
[1]http://legacy.python.org/dev/peps/pep-0274/
Co-Authored-By: ChangBo Guo(gcb) <eric.guo@easystack.cn>
Co-Authored-By: Kamil Rykowski <kamil.rykowski@intel.com>
Change-Id: I0ba408f9c616dcdb09618f6256db76b9facc0c1d
Right now if you want to get a single image, two queries need to be
executed:
- Get image + associated locations and properties
- Get associated tags for image
Proposed solution combines these two queries into single one which gets
image and all related data - locations, properties and tags. Similar
pattern has been already implemented when querying for list of images.
Change-Id: Ic4424bb1eb1769d8f621ebe111d95961ecf08479
Closes-Bug: 1434578
This change:
- Adds oslo.policy to the requirements list
- Accounts for changes in Enforcer initialization
- Accounts for changes to config options
- Removes incubated version of oslo.policy
- Updates the in-tree etc/config files
UpgradeImpact
Partially Implements Blueprint: graduate-policy
Change-Id: I5acb1e0f809098991f05ca3b6d78d4d88d98f2db
Extend rest images api v2 with multiple sort directions support.
Example:
/v2/images/detail?sort_key=name&sort_dir=asc&sort_key=size&sort_dir=desc
Changed database api which now can take sort_dir param as a list.
python-glanceclient support will be added in separate commit.
Implements-blueprint: glance-sorting-enhancements
DocImpact
APIImpact
Change-Id: Ib43b53abfba7cb5789d916a014376cf38fc5245b
Mainly to improve consistency, use range() from six.moves
renames across glance.
Behaves consistently like py2 xrange() and py3 range().
Removes unnecessary range() from glace/api/v2/images.py
Change-Id: Id21f923d05600b902f2239e25ef01716c07e74a3