This is a rather beefy change due to the number of usages of this
import. The changes are trivial though.
Change-Id: I7badeeaca438b0291f4ed86670e7f217e6372c61
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
This is the same as the 'range' keyword in Python 3
Change-Id: If3aa008522c24e870b7bf13de32b8ed1b27cb519
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
This change uses new policy module to enforce tasks_api_access
policy check and excludes policy enforcement from the policy layer.
NOTE: Since we are excluding policy enforcement at policy layer, removed
below unit tests as those were checking controller methods and we no longer
enforcing policies at policy layer.
test_create_task_unauthorized
test_index_unauthorized
test_get_unauthorized
Partially implements: blueprint policy-refactor
Change-Id: I670c076527b7bce55696b294d2283121dab5c707
Made provision to pass image_id, request_id and user_id information
while creating new task.
Partially-Implements: blueprint messages-api
Change-Id: I299a222eeef81431143db3ba7fc08365c924326b
This teaches glance-api how to do async threading things when it is
running in pure-WSGI mode. In order to do that, a refactoring of things
that currently depend on eventlet is required.
It adds a [wsgi]/task_pool_threads configuration knob, which is used
in the case of pure-WSGI and native threads to constrain the number
of threads in that pool (and thus the task parallelism). This will
allow tuning by the operator, but also lets us default that to just
a single thread in the backport of these fixes so that we can avoid
introducing a new larger footprint in the backport unexpectedly.
Partial-Bug: #1888713
Depends-On: https://review.opendev.org/#/c/742047/
Change-Id: Ie15028b75fb8518ec2b0c0c0386d21782166f759
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib.
Change-Id: I44e7b6f76e2d12f620ec602afc77ce11ba6b9d9a
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
We were capped at a very old version of hacking. Hacking itself caps the
various linters it uses to remain consistent, so our pep8 job was not
checking quite a bit that current versions have added.
This raises that limit to the latest to get up to the level of other
projects and addresses the errors the updated linters uncovered.
Change-Id: I89a9d73fbd59606a649e26077acebc5c42873d67
Co-authored-by: Sean McGinnis <sean.mcginnis@gmail.com>
Registry service is deprecated and due for removal since past
couple of cycles. This patch removes functional and unit tests of
registry.
NOTE:
Skipped 'test_create_with_live_time' test as it was dependent on
test_registry_client and test_registry_api modules. Something is
wrong with the test and will be corrected once entire registry code
is removed from the code base.
Change-Id: I560ab5260bed7e43d83b67a00057ac48f9e366e9
The tasks unit tests that test the RequestDeserializer
are using the real policy engine. They should be using
a mock because we're testing the deserialization, not the
policies.
Closes-bug: #1852354
Change-Id: I35bdacd154b2c8d18ce268a5e6e8aba710e41108
RequestContext.tenant and user fields are deprecated in favor
of project_id and user_id respectively.
This change modifies the glance.context.RequestContext constructor
to transition usage of tenant/user to project_id/user_id until
all tests are moved over to the new attributes. Runtime usage of
the old fiels is updated.
To prevent new code from using the deprecated fields, a warnings
filter is added which will make tests fail if they hit code using
the old fields.
Co-Authored-By: Abhishek Kekane <akekane@redhat.com>
Change-Id: I351380840308a24769ece93abc6d1a9a6d6aa06f
The Tasks API was made admin-only in Mitaka to prevent it from being
exposed directly to end users. The interoperable image import
process introduced in Pike uses the tasks engine to perform the
import. This patch introduces a new policy, 'tasks_api_access',
that determines whether a user can make Tasks API calls.
The currently existing task-related policies are retained so that
operators can have fine-grained control over tasks. With this
new policy, operators can restrict Tasks API access to admins,
while at the same time, admin-level credentials are not required
for glance to perform task-related functions on behalf of users.
Change-Id: I3f66f7efa7c377d999a88457fc6492701a894f34
Closes-bug: #1711468
Since the default value is None when can't get a key from a dict,
So there is no need to use dict.get('key', None).
Change-Id: Ie8008e83f432263e7a97e482e83f240b1056feee
There are several places in the source code where
HTTP response codes are used as numeric values.
These values are used from six.moves and the
numeric values are replaced by constants.
All of the used status codes were replaced with symbolic constants
from six.moves.http_client.
More about six.moves.http_client can be found at [2],
under the table "Supported renames:".
Also, this change improves code readibility.
This patchset does not extract numeric values from
the code itself, but it can be found at [1].
[1]: Ib9e26dcea927e96e65c626c18421621d3a29a64d
[2]: https://pythonhosted.org/six/#module-six.moves
Change-Id: Idfc7b043552f428f01ac3e47b270ee0639a8f5bc
Closes-Bug: #1520159
As we are removing the s3 driver from source tree per change
I032b0fc16400cbd2112687d38e010128be699221 , it seems right to remove the
references of the same in the glance tree.
Updated here are files that referenced s3; these files include:
* documentation files
* documentation of the glance architecture (source and png files)
* config help text
* stores associated with location strategy
* tests
UpgradeImpact
Change-Id: I680ee2baf39b49156d173e2ae108ae5d0221eca7
Depends-On: I032b0fc16400cbd2112687d38e010128be699221
Depends-On: I5d2da0611365b838f41285b64cdcfef9adf50e85
Currently there is no mechanism for deleting tasks on regular basis.
This patch adds a new function that is called in the tasks_get_all
function, so that everytime tasks lists are called, the function
checks if tasks in the database have surpassed the expired_at value.
If that is the case, then it sets the deleted value as 1 for all the
expired tasks.
Co-Authored-By: Mike Fedosin <mfedosin@mirantis.com>
Implements https://review.openstack.org/#/c/324648/
Change-Id: I0bde982de948901f6bfbfab9e57cf84891c22052
exception.Invalid will be raised when use task-create to import
image without properties, as task-create is a asynchronous action,
so we change task status to failure, and save the exception msg to
the task msg, the end users can get the msg when they use "task-show".
Change-Id: I08c26fd073e643d6ae7fef3475343442d24831b5
Closes-bug: #1580848
This change introduces glance.common.timeutils that provides the
timeutils previously consumed from oslo_utils.
Oslo is deprecating some timeutils functionality which of Glance
depends on. Suggested replacement (isoformat) would break glance APIs
so it's cleaner to carry this functionality in Glance rather than
re-invent the wheel.
Co-Authored-By: Erno Kuvaja <jokke@usr.fi>
Co-Authored-By: Sabari Kumar Murugesan <smurugesan@vmware.com>
Change-Id: I91e1cc9a273249fd88749cecf21200f3f5e2bab1
Now, when use wrong parameter to create a import task, the task's
status will be processing forever. Actually the task has been stopped
and returned error.
So in this situation, the task's status should be changed into failure.
Closes-bug: #1495519
Change-Id: Ie4b3b92aaddcdbb839aadf61c75fc0c915e2040e
test_create_with_live_time fails gating constantly. This is due to
the logic in the test which breaks if the second changes between the
two timestamps expires_at and updated_at are created. This change
drops the failure rate to those occasions where we hit that second
change also to minute change.
Closes-Bug: #1505675
Change-Id: I234d80388edd2a659a2374a29179ce23b23bb99a
Some tests used incorrect order assertEqual(observed, expected).
The correct order expected by testtools is
assertEqual(expected, observed).
At some places, corrected argument order for assertNotEqual method as well
and replaced assertEqual with assertTrue/assertFalse wherever required.
Closes-Bug: #1259292
Change-Id: I6fff8a3eedd053d0de8e261245a32f9a9fdab213
CONF.task.task_time_to_live doesn't work now. When change it's value,
the task's properity 'expires_at' and 'updated_at' still have the
difference of 48 hours.
The reason is that taskflow create a new task object and doesn't use
the task_time_to_live which glance api passed before.
Change-Id: Iec77be56585b7fe183d9903515e25fd096a6c0b0
Closes-bug: #1488360
This was never a real part of the mock api, but mock was happily mocking
it out and letting the tests pass. The new mock release made this
behaviour break, causing tests to fail on all patches. This patch
replaces instances of assert_called_once with an assert of the mock's
call_count.
Closes-Bug: #1473369
Co-Authored-By: Ian Cordasco <graffatcolmingov@gmail.com>
Change-Id: I12ff85a9480d8000b0de54939c3be71458162a40
In our codebase for tests we are often using same pattern to initialize
FakeDB instance which by default resets the database state and puts
sample data into it. After that we manually reset it's state, which
makes no sense. Here are the steps we are performing right now:
1. Reset database state
2. Puts data into database
3. Reset database state
It would be cleaner (and slightly faster) if we can skip these redundant
steps (2,3) in case we need an empty database. Provided fix allows
developers who are writting tests to decide if sample data should be
initialized or not.
Change-Id: I9e03b1c26d9ad311b8ff4330d19b0db93e6d1157
The Oslo libraries have moved all of their code out of the 'oslo'
namespace package into per-library packages. The namespace package was
retained during kilo for backwards compatibility, but will be removed by
the liberty-2 milestone. This change removes the use of the namespace
package, replacing it with the new package names.
The patches in the libraries will be put on hold until application
patches have landed, or L2, whichever comes first. At that point, new
versions of the libraries without namespace packages will be released as
a major version update.
Please merge this patch, or an equivalent, before L2 to avoid problems
with those library releases.
Blueprint: remove-namespace-packages
https://blueprints.launchpad.net/oslo-incubator/+spec/remove-namespace-packages
Change-Id: Ifa8baab33cdb3e606cf175a8c29c3a4ef6c44480
This patch adds a first step towards image introspection. The patch adds
a set of tasks to correctly import an image from an external source (old
copy-from) and then introspect the image.
The implementation keeps these 2 implementations separate by considering
the import process a top-level task and the introspection step an
optionally available task that may/may not succeed.
The implementation uses stevedore to load both, import and optional
flows, and chains them using linear and unordered flows where needed.
DocImpact
Co-Authored: Erno Kuvaja <jokke@hp.com>
Partially-implements blueprint: new-upload-workflow
Partially-implements blueprint: introspection-of-images
Change-Id: I9e7505d4e84aabf9d71e03c360514ac824ea84de
Mainly to improve consistency, use range() from six.moves
renames across glance.
Behaves consistently like py2 xrange() and py3 range().
Removes unnecessary range() from glace/api/v2/images.py
Change-Id: Id21f923d05600b902f2239e25ef01716c07e74a3
A sample import script to successfully import image from http or https
location is introduced. This should work on a devstack installtion.
Also, the following changes are introduced:-
1. An interface for implementing any other type of executor.
2. Provides namespace for keeping Tasks scripts.
3. A config for choosing deployer specific executor.
4. An interface for writing Tasks scripts.
5. A module for common script related methods.
6. Logic for limiting number of simultaneous tasks execution on the
Glance API server.
partially implements bp async-glance-workers
DocImpact
Change-Id: I382472fffd0fdad43573e72b2e78a9a6ed1e1f1a
A randomized PYTHONHASHSEED caused the glance API to return certain data
in a nondeterministic order, breaking some tests. This patch sorts the
returned data before testing against a pre-sorted expected values.
This patch also fixes some unit test failures with the new version of
testtools (0.9.36) and directly calling setUp(). This was added to this
patch to avoid blocking more Glance tests.
Co-authored-by: Fei Long Wang <flwang@catalyst.net.nz>
Co-authored-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
Partial-bug: #1348818
Change-Id: Ie92c6bb5663826713ca99e8f2f7d70f3faef6675
The following replacements were done in unit tests to have
clearer messages in case of failure:
- assertTrue(* is None) with assertIsNone
- assertTrue(* is not None) with assertIsNotNone
- assertTrue(* in *) with assertIn
- assertTrue(* not in *) with assertNotIn
- assertFalse(* in *) with assertNotIn
- assertTrue(* == *) with assertEqual
- assertTrue(* != *) with assertNotEqual
Change-Id: I0c47f991c3974e441335e71c9d26fab8a127f2ca
Change I0b406e4662cfd3cb496b71be77ad10a90c178baa introduces a new
class TaskStub. This class is designed to be a subset of what a
task is in the domain layer: it doesn't contain input, message and
result. TaskStub should be used when listing tasks in order to avoid
extra-overhead of transporting these text fields.
As a result, the TaskDetails introduced by change
I0b406e4662cfd3cb496b71be77ad10a90c178baa doesn't need to exist.
With this patch, the domain layer contains Task and TaskStub instead
of Task, TaskStub and TaskDetails.
partially implements bp async-glance-workers
Change-Id: I2a20b0c5033c6920749370355c2d672dec707c28
Partial Task patch, Id I4fbadc9a97e3147128c7c733384c7bb50918806f
removed the result and message vars of the Tasks class. However,
the succeed and fail methods were using those to update them
respectively. This patch introduces a new TaskStub class which
enables the PartialTask functionality without looking result
and message attrs of the Task class.
Fixes bug 1284975
Change-Id: I0b406e4662cfd3cb496b71be77ad10a90c178baa
We currently return 404 NotFound when attempting to delete a task
by hitting the resource /v2/tasks/{task_id}. However, as this
resource exists and we do not allow DELETE on tasks, the correct
response is 405 MethodNotAllowed.
Implementing as per spec on
http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
Fixes bug 1287951
Change-Id: Ia4a969e595b520dbc5f4e9a463d967681465ae6f
- partial-task: Task that includes all the values except the 'Text'
columns(input, result & message).
- This patch is to have the database to return a partial-task list on users
invoking 'GET /v2/tasks'.
- Returning list of partial tasks will help in reducing the response time when the
user queries for the list of tasks.
- This patch also includes changes for making implementation and return values of
display task detail (GET /v2/tasks/{task_id}) and list tasks (GET /v2/tasks)
more explicit.
partially implements bp async-glance-workers
Change-Id: I4fbadc9a97e3147128c7c733384c7bb50918806f
Assign an explicit created_at and updated_at datetime value to task DB
fixtures, it be used to fixes race condition in TasksController test
cases.
Closes-Bug: #1274365
Change-Id: I9f13c7e0e97085faac15713342808722a6692178
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
Fixes bug: 1267148
'expires_at' on a task must be set only after the executor succeeds or
fails on a task. Before the task has finished, there shouldn't be any
'expires_at' set on the task. This change fixes the ResponseSerializer
to not show 'expires_at' if it is not set.
Change-Id: I9880cc03b9d6f0dd98388fcf0d1bdc9c34dd6c0e
Bump hacking dependency to 0.8 to get python 3.x compatibility
Fixes done in order to avoid errors after enabling hacking 0.8
Change-Id: Ic878fe2e1bd3f65f7f95a9b5c7a192dac81b749d
Closes-Bug: #1257282
Every call to json.loads/json.dumps have been changed to
jsonutils.loads/jsonutils.dumps respectively. Import json has been removed
also replaced by import glance.openstack.common.jsonutils
654d80b416dc5f413cb791aa838ec8688bf7da44 Create openstack.common.jsonutils
Change-Id: I8ef580e5eb91526dfaef37050ce1f5c6d88d56b5
Closes-bug: #1257922
Each project should directly use the standard uuid module.
uuidutils will be deprecated/removed in this cycle.
This patch replaces every uuidutils.generate_uuid() with
str(uuid.uuid4()) and uuidutils.is_uuid_like()
with utils.is_uuid_like().
Change-Id: I43642d4f1e137c14134b3d544e367b504b9851ac
Closes-Bug: #1253497
- add REST API layer for async worker(tasks) to
support tasks operations.
- update to include sparse task for tasks.index
- add domain proxies for handling authorization,
policy enforce, notification while processing
task requests.
- add Task domain entity and TaskFactory to create
new Task domain entity objects.
- add integration test for testing tasks api
Co-authored-by: Fei Long Wang <flwang@cn.ibm.com>
Partially implement blueprint async-glance-workers
Change-Id: I072cbf351c06f59a7702733b652bfa63e5abbaa6
Stephen Finucane