Commit Graph

125 Commits

Author SHA1 Message Date
Takashi Kajinami 9f896ab03d Bump hacking
hacking 3.0.x is too old.

Change-Id: I33875c36dbbdb946841f8c583636ccdf88d3331f
2024-01-27 16:55:11 +00:00
Stephen Finucane d1368745c5 db: Remove explicit autocommit
We already got rid of implicit autocommit. Now get rid of explicit
autocommit. This is far more involved than the prior patch but the basic
strategy here is that no public method should take a 'session' argument
and instead should call 'get_session'. In addition, all private methods
should take both a 'context' and 'session' argument and they should be
the first and second arguments, respectively. Finally, public methods
should only call private methods. Combined, this ensures we only ever
have one session in play and the transactions are correctly started and
ended as expected.

Change-Id: I5245c402699e179c219039514b626a52fa0028d8
2023-07-25 10:08:52 +01:00
Stephen Finucane 8c5091cd48 db: Drop use of implicit auto-commit
Resolve the following RemovedIn20Warning warning:

  The current statement is being autocommitted using implicit
  autocommit, which will be removed in SQLAlchemy 2.0. Use the .begin()
  method of Engine or Connection in order to use an explicit transaction
  for DML and DDL statements.

This is the trickiest one hence it's left to last. We really need to get
off of the legacy EngineFacade pattern since it's harder to track
transactions when they're not local (the request context-based binding
of sessions allows us to decorate individual methods) but that's a
larger work item that we're not going to tackle here.

Change-Id: Ie16965a8e3e2f316b807068a868cee6fbb2f33ba
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
2023-07-25 10:03:11 +01:00
Stephen Finucane 2e716b53f3 db: Enable foreign keys for SQLite backend
We were doing this adhoc in tests, which doesn't seem to be possible in
SQLAlchemy 2.0. Do it everywhere instead.

This highlights a small issue that we have to workaround in the tests.

Change-Id: I9dd70c9976f83a00a8cb270422e545ff2c77f12f
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
2023-07-25 09:56:29 +01:00
Stephen Finucane 25dcacea7c db: Don't pass strings to Connection.execute()
Resolve the following RemovedIn20Warning warning:

  Passing a string to Connection.execute() is deprecated and will be
  removed in version 2.0.  Use the text() construct, or the
  Connection.exec_driver_sql() method to invoke a driver-level SQL
  string.

We also resolve two additional RemovedIn20Warning warnings along the
way, since we're already touching this code:

  The Engine.execute() method is considered legacy as of the 1.x series
  of SQLAlchemy and will be removed in 2.0. All statement execution in
  SQLAlchemy 2.0 is performed by the Connection.execute() method of
  Connection, or in the ORM by the Session.execute() method of Session.

  The Executable.execute() method is considered legacy as of the 1.x
  series of SQLAlchemy and will be removed in 2.0. All statement
  execution in SQLAlchemy 2.0 is performed by the Connection.execute()
  method of Connection, or in the ORM by the Session.execute() method of
  Session.

Change-Id: I547bd2f441085fcdc5dc4a0e58d27d11e8a8b223
2023-07-24 21:34:03 +00:00
Cyril Roelandt 399438cd04 Fix E741 issues
In 2.6.0, pycodestyle is going to find more E741 issues[1], which is
going to break the gate by making "tox -epep8" fail.

[1] https://www.flake8rules.com/rules/E741.html

Change-Id: I94ee03a00dd01726de15a29424183faabdeba844
2023-01-16 12:01:06 +01:00
Zuul c66705069b Merge "Remove unicode literal strings" 2022-07-08 07:55:24 +00:00
Zuul bef3a70aed Merge "Drop support for SQLite < 3.7" 2022-07-07 15:01:32 +00:00
xuanyandong be997b53ab Remove unicode literal strings
Co-Authored-By: Cyril Roelandt <cyril@redhat.com>
Change-Id: Id9e1a5fb9c732c207ee08f0dbf387436a1783174
2022-06-30 19:37:03 +02:00
Stephen Finucane 931809c037 Remove remaining six.moves usage
Change-Id: Ibca3884e1ea3d0fb170bcc9e70a176d144ee24cc
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
2022-01-27 16:37:11 +00:00
Stephen Finucane 39e667a145 Remove six.moves.range usage
This is the same as the 'range' keyword in Python 3

Change-Id: If3aa008522c24e870b7bf13de32b8ed1b27cb519
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
2022-01-27 14:54:05 +00:00
wu.shiming 3d204f5f13 Drop support for SQLite < 3.7
Both Ubuntu 18.04 [1] and CentOS 8 [2] provide sufficiently new versions
of SQLite that make this check unnecessary now.

[1] https://packages.ubuntu.com/bionic/sqlite3
[2] http://mirror.centos.org/centos/7/os/x86_64/Packages/

Change-Id: Ib2e267eb1ac5ef4da711b6f5bccc5eccfb4b6954
2021-09-18 10:16:11 +08:00
Dan Smith 2a0d2303c3 Fix image/tasks API for in-progress tasks
A slight error in the tasks_get_by_image() DB API method resulted in
our excluding in-progress tasks from the returned list. This is
because those tasks have expires_at=NULL, and we were comparing
the expires_at>=$NOW to find unexpired tasks. This makes us check for
"NULL or not expired" instead.

We did have a test asserting the wrong behavior, but it was done to
increase coverage and thus was asserting the behavior of the code and
not the *desired* behavior. This fixes that as well.

Closes-Bug: #1922928
Change-Id: I1b6971888673b64ef60bed8fbcc97bbcbcf5c2ac
2021-04-07 08:10:03 -07:00
Abhishek Kekane 281fadc15c New API /v2/images/{id}/tasks
Added new API /v2/images/{id}/tasks to show tasks associated with
image. This API will return list of tasks associated for valid image
else returns 404 not found if image is not present. This API also
initiates task scrubbing before returning tasks to user.

Implements: blueprint messages-api
Change-Id: Ib3cacb4dd4d75de32e539f8a3b48bdaa762e6d8e
2021-02-24 05:19:43 +00:00
Cyril Roelandt 7839ab0925 Remove unused option "owner_is_tenant"
This option has been deprecated since Rocky.

Change-Id: I8edc957ad50ec28d80a06e76912f4226cea53562
2021-02-11 16:17:10 +01:00
Sean McGinnis 94b0876429 Use unittest.mock instead of third party mock
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib.

Change-Id: I44e7b6f76e2d12f620ec602afc77ce11ba6b9d9a
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2020-04-20 15:07:00 +00:00
Dirk Mueller 595c1b17ff
Raise hacking to latest 2.0.0 release
We were capped at a very old version of hacking. Hacking itself caps the
various linters it uses to remain consistent, so our pep8 job was not
checking quite a bit that current versions have added.

This raises that limit to the latest to get up to the level of other
projects and addresses the errors the updated linters uncovered.

Change-Id: I89a9d73fbd59606a649e26077acebc5c42873d67
Co-authored-by: Sean McGinnis <sean.mcginnis@gmail.com>
2020-03-27 14:11:08 -05:00
Lucian Petrut 8fe78170e4 Windows: fix flaky tests
Some tests compare timestamps (e.g. image created_at/updated_at
fields).

In some cases, subsequent timestamps may be identical, breaking the
assertions made by those tests.

One idea is to just add a 1ms sleep which should be a negligeable.
Mocking the functions used to retrieve timestamps may be undesireable.

For now, we'll do this only for Windows, where time.time has a lower
resolution compared to Linux (1e-7 as opposed to 1e-9).

At the same time, socket timeouts are rather inconsistent on Windows.
When a timeout is hit, recv may either throw ConnectionAbortedError
*or* return nothing. This needs to be taken into account when
expecting socket timeouts.

Change-Id: Ie5a4d8fb4c979c09eda2fdc0fad0baa1d1840c59
2019-03-25 14:23:21 +02:00
Zuul d501799a6a Merge "Allow glance tests to run on Windows" 2019-03-14 15:30:41 +00:00
Lucian Petrut 98b7ef195c Allow glance tests to run on Windows
In order to run the unit and functional Glance tests on Windows, we
have to:

* avoid monkey patching the os module on Windows (which causes Popen
  to fail)
* update sqlite connection URL
* avoid os.fork, not available on Windows.
    * we'll use subprocess.Popen when spinning up http servers.
    * for the really simple ones defined in the test helpers, we'll just
      use threads
* do not attempt to connect to '0.0.0.0', use '127.0.0.1' instead
* some tests aren't properly skipped (xattr ones), so we're covering that
  as well
* skip log rotation test, we can't move in-use files. Log rotation can
  be performed by the log handler itself.
* expect an exception when hitting connection timeouts
* avoid installing unavailable test requirements (xattr, pysendfile)
* pin the instance creation timestamp. some tests that deal with
  markers rely on ordering, which can be flipped if the timestamps are
  identical (can happen in case of resources created one after the
  other, not sure yet if this happens really fast or the clock isn't
  accurate enough).
* add a few seconds to some timeouts (much needed when running the tests
  in VMs).

blueprint windows-support

Change-Id: Ife69f56a3f9f4d81e1e2e47fde4778efd490938f
2019-03-13 16:41:11 +02:00
Sam Morrison cfa9b6e8e8 Add new 'all' visibility filter for listing images
Closes-Bug: #1779251

Change-Id: I01368d1b375d2f538739baaa1e04462f70b43051
2019-02-25 10:34:01 +11:00
Matt Riedemann 88a8ad7823 Fix DeprecationWarnings for RequestContext.tenant/user usage
RequestContext.tenant and user fields are deprecated in favor
of project_id and user_id respectively.

This change modifies the glance.context.RequestContext constructor
to transition usage of tenant/user to project_id/user_id until
all tests are moved over to the new attributes. Runtime usage of
the old fiels is updated.

To prevent new code from using the deprecated fields, a warnings
filter is added which will make tests fail if they hit code using
the old fields.

Co-Authored-By: Abhishek Kekane <akekane@redhat.com>
Change-Id: I351380840308a24769ece93abc6d1a9a6d6aa06f
2019-02-06 09:59:36 -05:00
Liang Fang 72159a4a7b Fix for FK constraint violation
First force purging of records that are not soft deleted but are
referencing soft deleted tasks/images records (e.g. task_info records).
Then purge all soft deleted records in glance tables in the right order
to avoid FK constraint violation.

Closes-Bug: #1803643
Change-Id: I1c471adce002545f8965a57ef78a57e1e3031ef0
Co-authored-by: Tee Ngo <tee.ngo@windriver.com>
Signed-off-by: Liang Fang <liang.a.fang@intel.com>
2018-12-07 11:19:31 +08:00
Abhishek Kekane 5cc9d99935 Mitigate OSSN-0075
Modified the current ``glance-manage db purge`` command to eliminate images
table from purging the records. Added new command
``glance-manage db purge_images_table`` to purge the records from images
table.

DocImpact
SecurityImpact

Change-Id: Ie6641659b54543ed9f96c393d664e52a26bfaf6a
Implements: blueprint mitigate-ossn-0075
2018-07-05 13:33:36 +00:00
space d6068bec44 Add 'protected' filter to image-list call
Implements v2 API image-list filtering on the 'protected' property
following the spec:
https://git.openstack.org/cgit/openstack/glance-specs/tree/specs/pike
/approved/glance/add-protected-filter.rst
The 'protected' query string parameter accepts one of 'true', 'false'.
Any other value raises a 400 with an appropriate message.
See the discussion on https://review.openstack.org/#/c/451560/
for why this implementation strategy was chosen.

Change-Id: Ie90d952edddd3f69b3f489f056ff506e63f96376
Closes-Bug: #1674246
2017-07-06 09:48:17 +08:00
Ravi Shekhar Jethani c9a21f655e Provide user friendly message for FK failure
'glance-manage db purge' command fails with DBReferenceError due
to FK constraint failure and exits with stack-trace on the command
prompt.

Made changes to give user-friendly error message to the user as
well as log appropriate error message in glance-manage logs
instead of stack-trace.

Co-author-by: Dinesh Bhor <dinesh.bhor@nttdata.com>
Change-Id: I52e56b69f1b78408018c837d71d75c6df3df9e71
Closes-Bug: #1554412
2017-04-05 15:56:32 +05:30
Timothy Symanczyk 265659e8c3 Implement and Enable Community Images
This change replaces the existing boolean 'is_public' column for
the 'images' table with enum 'visibility' column featuring the
four explicit visibility values - public, private, shared,
and community.

This change also implements and enables all backend code to
utilize the new values.

Co-Authored-By: Timothy Symanczyk <timothy_symanczyk@symantec.com>
Co-Authored-By: Dharini Chandrasekar <dharini.chandrasekar@intel.com>

Implements: blueprint community-level-v2-image-sharing
Closes-Bug: #1394299
Closes-Bug: #1452443
Depends-On: I6e3268f3712cbc0aadb51d204c694023b92d55a5
Change-Id: I94bc7708b291ce37319539e27b3e88c9a17e1a9f
2017-01-18 17:56:12 +00:00
GeetikaBatra 38563b0555 Add a soft delete functionality for tasks.
Currently there is no mechanism for deleting tasks on regular basis.
This patch adds a new function that is called in the tasks_get_all
function, so that everytime tasks lists are called, the function
checks if tasks in the database have surpassed the expired_at value.
If that is the case, then it sets the deleted value as 1 for all the
expired tasks.

Co-Authored-By: Mike Fedosin <mfedosin@mirantis.com>

Implements https://review.openstack.org/#/c/324648/
Change-Id: I0bde982de948901f6bfbfab9e57cf84891c22052
2016-06-07 11:23:30 +05:30
Jenkins 8198ba42f7 Merge "Use assertGreater/Less/Equal instead of assertTrue(A * B)" 2016-03-06 22:34:02 +00:00
Béla Vancsics f29e9e6138 Use assertGreater/Less/Equal instead of assertTrue(A * B)
Instead of using assertTrue(A * B), developers should
use assertGreater(A, B) or assertLess(A, B) or
assertGreaterEqual(A, B) or assertLessEqual(A, B)
or assertEqual(A, B).
The * operator: >=, <=, ==

Change-Id: I85e0087c530c2f1132061036fd5427368dfa2316
2016-03-02 07:56:45 +00:00
Darja Shakhray 6007061943 Added support new v2 API image filters
Added support filtering images based on lists using the 'in' operator.
Filters:
*id
*name
*container_format
*disk_format
*status

DocImpact
ApiImpact
Implements bp: in-filtering-operator

Change-Id: I9cac81b9d5cbec979e88cf2dd0e3b710ed45630c
2016-02-20 09:41:19 +00:00
Sabari Kumar Murugesan 700b7ef26d Fix re-adding deleted members to an image in v1
If you replace the membership list of an image with some members
that were previously deleted, we fail to re-add those deleted
members.

This seems to be a consequence of a previous commit
d6800e143d which failed to update
the deleted records.

Closes-Bug: 1527143

Change-Id: Ic3ca9b56712a99652e65cb0a4e3f1e0ba15b8593
2016-01-20 11:41:16 +00:00
houming-wang 9ae3d53a8d Use six.moves.reduce instead of builtin reduce
Builtin function 'reduce' in Python 2 has been moved to standard
library module in Python 3 [1]. To make code compatible, we should
replace reduce(expr) with six.moves.reduce(expr)

[1] http://python3porting.com/stdlib.html#moved-builtins

Change-Id: Idca3420918471c41548a871d043fafc983aa7ee6
Closes-Bug: #1530249
2015-12-30 22:02:44 -05:00
Jenkins 585206940a Merge "Add db purge command" 2015-12-19 02:23:20 +00:00
Martin Mágr 9a6823326b Add db purge command
This patch adds "db purge" to glance-manage for deleting soft deleted
images, tasks.

Change-Id: I5b609292aa15f8133d0d785fcf9143825bed8073
Implements: blueprint database-purge
2015-12-11 19:16:58 +01:00
Julien Danjou 4d5330088f Replace oslo_utils.timeutils
This change introduces glance.common.timeutils that provides the
timeutils previously consumed from oslo_utils.

Oslo is deprecating some timeutils functionality which of Glance
depends on. Suggested replacement (isoformat) would break glance APIs
so it's cleaner to carry this functionality in Glance rather than
re-invent the wheel.

Co-Authored-By: Erno Kuvaja <jokke@usr.fi>
Co-Authored-By: Sabari Kumar Murugesan <smurugesan@vmware.com>

Change-Id: I91e1cc9a273249fd88749cecf21200f3f5e2bab1
2015-12-10 10:53:46 +00:00
wangxiyuan d13f088cac Fix default value with postgreSQL
When use 'sort-key=size' for multi-pages searching with postgreSQL
DB, it will raise an error: DBError: (psycopg2.DataError). And return
HTTPInternalServerError (HTTP 500) to users. The reason is that
postgreSQL doesn't support the size's default value to be string.
This patch set the size's default value to zero.

Change-Id: Ib3b5f3f57be3683ba274f0122e6314978a79e75f
Closes-bug:#1481154
2015-12-04 09:33:09 +08:00
Steve Lewis a4c6f12636 Support new v2 API image filters
Provide support in Glance API for querying the image list for
created_at and updated_at times using guidance from the API Working
Group recommendations for filtering. Filtering is applied at the DB
layer.

DocImpact
ApiImpact
MitakaPriority
Change-Id: Ie94295bb82779ec17ab773928c71ae4a9ee8fbcc
Implements bp: v2-additional-filtering
2015-11-16 15:45:17 -08:00
Erno Kuvaja e8e71dba4d Rationalize test asserts
Utilizing assertIn and assertNotIn in a consistent way rather than
assertTrue( x <not> in y) and assertFalse(x in y).

Change-Id: Ic43d107c40a5a50b852ee476b1167c21e5ad86e8
2015-07-09 13:48:49 +00:00
wangxiyuan 356e66c239 Fix wrong check when create image without data.
The default value of 'locations' is [].So there is a wrong check in
db/sqlalchemy/api.py.
When create an image without data, The function '_image_locations_set'
should not be executed.

Change-Id: Ie4cb29ab0d714d3ce67f717d5eaf641424efa5b7
Closes-bug:#1471080
2015-07-06 12:08:21 +08:00
Jenkins a432b1f0f6 Merge "Fix DbError when image params are out of range" 2015-06-12 03:34:52 +00:00
Inessa Vasilevskaya 1badf69b01 Fix DbError when image params are out of range
Image params such as checksum, min_ram and min_disk are validated.
Checksum is expected to be not longer than 32 characters,
min_ram/min_disk can't exceed pow(2, 31) - 1.
A method for mysql_int validation has been moved to glance.common.utils.

Change-Id: I78554ef1ba65fc11da3de56467d6d70bb7991787
Closes-Bug: #1454730
Closes-Bug: #1460060
2015-06-11 19:27:17 +03:00
ChangBo Guo(gcb) 9d4225289b Leverage dict comprehension in PEP-0274
PEP-0274 introduced dict comprehensions to replace dict constructor
with a sequence of key-pairs[1], these are two benefits:
- First, it makes the code look neater.
- Second, it gains a micro-optimization.

Glance dropped python 2.6 support in Kilo, we can leverage this now.
Note: This commit doesn't handle dict constructor with kwargs.
This commit also adds a hacking rule.

[1]http://legacy.python.org/dev/peps/pep-0274/

Co-Authored-By: ChangBo Guo(gcb) <eric.guo@easystack.cn>
Co-Authored-By: Kamil Rykowski <kamil.rykowski@intel.com>
Change-Id: I0ba408f9c616dcdb09618f6256db76b9facc0c1d
2015-05-28 01:46:02 +00:00
Fei Long Wang c37b497dea Revert "Reduce DB calls when getting an image"
This reverts commit 4734d83513.

Change-Id: Ie58c6d272bf55c2ab2243d3fe6d8811ff3dc46ca
2015-04-16 09:02:24 +00:00
Kamil Rykowski 4734d83513 Reduce DB calls when getting an image
Right now if you want to get a single image, two queries need to be
executed:
- Get image + associated locations and properties
- Get associated tags for image

Proposed solution combines these two queries into single one which gets
image and all related data - locations, properties and tags. Similar
pattern has been already implemented when querying for list of images.

Change-Id: Ic4424bb1eb1769d8f621ebe111d95961ecf08479
Closes-Bug: 1434578
2015-04-08 16:30:14 +02:00
Ian Cordasco cb7d5a4795 Use graduated oslo.policy
This change:

- Adds oslo.policy to the requirements list
- Accounts for changes in Enforcer initialization
- Accounts for changes to config options
- Removes incubated version of oslo.policy
- Updates the in-tree etc/config files

UpgradeImpact

Partially Implements Blueprint: graduate-policy
Change-Id: I5acb1e0f809098991f05ca3b6d78d4d88d98f2db
2015-03-17 23:50:02 -05:00
Mike Fedosin fac0cb2f07 Add the ability to specify the sort dir for each key
Extend rest images api v2 with multiple sort directions support.
Example:
/v2/images/detail?sort_key=name&sort_dir=asc&sort_key=size&sort_dir=desc
Changed database api which now can take sort_dir param as a list.

python-glanceclient support will be added in separate commit.

Implements-blueprint: glance-sorting-enhancements
DocImpact
APIImpact
Change-Id: Ib43b53abfba7cb5789d916a014376cf38fc5245b
2015-03-12 12:00:45 +00:00
Erno Kuvaja a4f5bf6ab9 Unify using six.moves.range rename everywhere
Mainly to improve consistency, use range() from six.moves
renames across glance.

Behaves consistently like py2 xrange() and py3 range().

Removes unnecessary range() from glace/api/v2/images.py

Change-Id: Id21f923d05600b902f2239e25ef01716c07e74a3
2015-02-25 12:05:36 +00:00
Louis Taylor 41d45dfb1c Move from oslo.utils to oslo_utils
oslo_utils has been moved out of the oslo namespace.

bp drop-namespace-packages

Change-Id: Ie818e72b31efd2a6ab182444967bdfaca9366f4a
2015-01-09 20:01:42 +00:00
Jenkins c3126c65e7 Merge "Adds the ability to sort images with multiple keys" 2014-12-23 13:48:06 +00:00