Made provision to pass image_id, request_id and user_id information
while creating new task.
Partially-Implements: blueprint messages-api
Change-Id: I299a222eeef81431143db3ba7fc08365c924326b
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib.
Change-Id: I44e7b6f76e2d12f620ec602afc77ce11ba6b9d9a
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
The import image api now supports a list of stores to import data into.
This list can be specified through a new "stores" field that has been
added to the request body.
During import stage, Glance iterates overs this list and send the data
to each store one by one.
If an invalid backend is requested by the user, an exception is raised.
If an errors occurs during verify, already pushed data is removed and
image state is unchanged.
Change-Id: Id3ac19488c0a693d7042be4a3c83f3b9f12313d0
Implements: blueprint import-multi-stores
In Python 3.7, "async" is a keyword. To prevent it from
conflicting, rename the async package to async_.
Change-Id: I1eaf87eedb86679d9ca9323aac05f0770c33efea
Closes-Bug: #1781617
Made provision for multi-store support. Added new config option
'enabled_backends' which will be a comma separated Key:Value pair
of store identifier and store type.
DocImpact
Depends-On: https://review.openstack.org/573648
Implements: blueprint multi-store
Change-Id: I9cfa066bdce51619a78ce86a8b1f1f8d05e5bfb6
Notifier unit tests currently cover image, image.member and
task notification groups. Added equivalent tests to make sure
no notifications are emitted if they are disabled using the
disabled_notifications config option.
Change-Id: Iae2288e6d2c75d29c7126baeb0a80ad2f8b7b428
The usage of oslo_messaging.get_transport is not meant for
notifications; And oslo_messaging.get_notification_transport is
favored for those means. So this change introduces the usage of that
function.
If the settings for the notifications are not set with the
configuration that's under the oslo_messaging_notifications group,
this will fall back to the old settings which are under the DEFAULT
group; just like this used to work.
Closes-Bug: #1567231
Change-Id: I137d0f98533d8121334e8538d321fb6c7495f35f
The virtual_size property was added to the v2 API in Icehouse as a
"core" image property [0], but the field was not added to image
notifications. This patch addresses that oversight.
[0] https://blueprints.launchpad.net/glance/+spec/split-image-size
Change-Id: I423719a88ba1ef17e7475ab5388fb1720a28011e
Closes-bug: #1554206
Image members CRUD doesn't generate notifications which
is impacting searchlight service by not having latest
changes to Image memberships.
If you create an image and later change its members,
the members are not updated via notifications.
You have to run the index sync again to get the updated
member list.
See: https://bugs.launchpad.net/searchlight/+bug/1490697
Membership information is critical for horizon filtering.
Typically, a person is allowed to view an image under the
following conditions:
1) The image is owned by the project I am currently logged into.
2) The image is public
3) The image is owned by another project which has added me
as a member and I have accepted membership to it.
Without current membership information, 3) above is not possible.
See: https://bugs.launchpad.net/searchlight/+bug/1491085
Change-Id: Ia56e42d3d8da36cfa419d5c3c7d69c9ccf8974fd
Closes-Bug: #1441453
This change introduces glance.common.timeutils that provides the
timeutils previously consumed from oslo_utils.
Oslo is deprecating some timeutils functionality which of Glance
depends on. Suggested replacement (isoformat) would break glance APIs
so it's cleaner to carry this functionality in Glance rather than
re-invent the wheel.
Co-Authored-By: Erno Kuvaja <jokke@usr.fi>
Co-Authored-By: Sabari Kumar Murugesan <smurugesan@vmware.com>
Change-Id: I91e1cc9a273249fd88749cecf21200f3f5e2bab1
As we agreed that we'll change the default notification exchange name
from 'openstack' to 'glance' in M release.
The openstack convention is to send out all the notifications to the
exchange with the same name as the project name, i.e. nova sends all its
notifications to the exchange 'nova', cinder sends all its notifications
to the exchange 'cinder', etc. In such way, other projects interested in
those notifications (e.g. ceilometer) can listen on different exchanges
for different notifications.
Change-Id: I658fb0896aecea4fd5db1a9d3a8bbb64a91a7224
Closes-Bug: #1467776
Metadefinition resources - namespaces, objects, properties, tags and
resource types - don't provide any notification events when certain
operations are performed on them. This patch includes following events
that will be triggered when necessary:
* metadef_namespace.create - namespace has been created
* metadef_namespace.update - namespace has been updated
* metadef_namespace.delete - namespace has been deleted
* metadef_namespace.delete_properties - all properties have been removed
from namespace
* metadef_namespace.delete_objects - all objects have been removed from
namespace
* metadef_namespace.delete_tags - all tags have been removed from
namespace
* metadef_object.create - object has been created
* metadef_object.update - object has been updated
* metadef_object.delete - object has been deleted
* metadef_property.create - property has been created
* metadef_property.update - property has been updated
* metadef_property.delete - property has been deleted
* metadef_tag.create - tag has been created
* metadef_tag.update - tag has been updated
* metadef_tag.delete - tag has been deleted
* metadef_resource_type.create - resource type has been added to
namespace
* metadef_resource_type.delete - resource type has been removed from
namespace
Additionally new configuration option has been added to allow for
disabling either individual or group of notifications.
DocImpact
UpgradeImpact
Depends-On: Iaa771ead0114e3941667b1e07ff32472d2f77afd
Change-Id: Ie1635793d80188f8f7a07aea91b9f0842900ffa6
Implements: blueprint metadefs-notifications
oslo_config was moved out of the oslo namespace in oslo.config>=1.6.0.
Related-to: blueprint drop-namespace-packages
Change-Id: I30ecbf8f9de77496fcb643c7ad9738d79ad359f0
This patch adds support for random access to images to the API v2. It's
possible to partially access the image data by passing the offset where
the image should be fetched from and the size of the chunk expected.
Note that not all stores support random access to image data. In such
cases, the API will return an error mentioning the selected store
doesn't support such a feature.
DocImpact:
Content-Range header is now parsed to support random access to image
data.
Implements blueprint: restartable-image-download
Change-Id: Iade692a8197a9a1d9532711a5c572c505e5e7d20
This commits removes the old `store` package from glance and adopts the
usage of the new glance.store library. The library was designed to
preserve backwards compatibility as much as possible. In fact, most of
the changes in this patch are related to function args ordering and not
function renames or workflow changes.
Some changes that are worth mentioning:
1. Glance store doesn't rely on a global config object. All config
options must be explicitly registered.
2. All store operations now accepted an optional context. This is a
fallout from the context not being required in the `Store` constructor
anymore.
3. Store drivers are behind a private package called `_drivers` and
they're not suppose to be accessed directly. Instead, functions like
`get_store_from_scheme` should be used.
4. Stores are disabled by default
5. All the store specific options are under the `glance_store` group.
DocImpact:
The old store related configuration options have been moved under the
`glance_store` section. However, the old options will go through a
deprecation path. That is, they'll still be read from the `DEFAULT`
section to give deployers enough time to update their config files.
In k-2, the deprecated options will be completely obsolete.
Closes-bug: #1291848
Implements-blueprint: create-store-package
Change-Id: Iaacc70993ad5da292b93de42bbecda73d53b19fd
A sample import script to successfully import image from http or https
location is introduced. This should work on a devstack installtion.
Also, the following changes are introduced:-
1. An interface for implementing any other type of executor.
2. Provides namespace for keeping Tasks scripts.
3. A config for choosing deployer specific executor.
4. An interface for writing Tasks scripts.
5. A module for common script related methods.
6. Logic for limiting number of simultaneous tasks execution on the
Glance API server.
partially implements bp async-glance-workers
DocImpact
Change-Id: I382472fffd0fdad43573e72b2e78a9a6ed1e1f1a
If a user is not allowed to download an image because of a policy,
glance v2 API responds with a HTTP 200 status and no data instead
of HTTP 403 for no cache only.
The problem is that get_data implementation for notification
proxy is a generator, this situation delays the other proxies
get_data calls (including the policy proxy) for the first time
data is retrieved.
Hence, there is a delay in enforcing policy, so 200 is sent before
the API gets the chance to catch the policy exception.
DocImpact
Closes-Bug: #1326781
Change-Id: I1e50a069a6b7f9eed7160cd5908a5fa30274e227
The notifier_strategy option is deprecated since we merged commit
I8cd84772bc5867e06b2a50ed7e15b9e86f0b94ad during the Icehouse cycle.
Assuming the usual single cycle deprecation period, this can now be
removed.
DocImpact: the notifier_strategy has been removed in favor of the
notification_driver and related oslo.messaging options.
Change-Id: I3ab0133b697a820536e5da84e30b078915a29f6c
In order to handle the deprecated notifier_strategy configuration option
we just need to translate the notifier_strategy value as follows:
default -> transport_url=None, driver='noop'
noop -> transport_url=None, driver='noop'
logging -> transport_url=None, driver='log'
rabbit -> transport_url=rabbit:///, driver='messaging'
qpid -> transport_url=qpid:///, driver='messaging'
None -> transport_url=None, driver=None
where the last bit simply means that the user hasn't specified any
notifier_strategy and we should just have oslo.messaging load a notifier
based on oslo.messaging's config options.
It really doesn't need to be any more complicated than that, and we
certainly shouldn't need to poke at what the user may have configured
for transport_url in their config file.
The reason this is important is that CONF.transport_url is a private
configuration option which soon will not be visible to users of the
oslo.messaging API. See I9c3bf74f13a820ce295aa77a2bbcd781778b5926.
Also, simplify the testing of this stuff by mocking the oslo.messaging
APIs used and using the test scenarios outlined above. This also means
we stop poking around in oslo.messaging internals like:
self.assertEqual(str(nfier._transport._driver._url),
'qpid:///')
Naughty, naughty!
Note: it's arguable we should just remove notifier_strategy in Juno
since we deprecated it in Icehouse, but this patch will be useful to
backport to stable/icehouse in any case.
Change-Id: I067248974baee50a2a0afb38bf022b65d1bea57a
The following replacements were done in unit tests to have
clearer messages in case of failure:
- assertTrue(* is None) with assertIsNone
- assertTrue(* is not None) with assertIsNotNone
- assertTrue(* in *) with assertIn
- assertTrue(* not in *) with assertNotIn
- assertFalse(* in *) with assertNotIn
- assertTrue(* == *) with assertEqual
- assertTrue(* != *) with assertNotEqual
Change-Id: I0c47f991c3974e441335e71c9d26fab8a127f2ca
Change I0b406e4662cfd3cb496b71be77ad10a90c178baa introduces a new
class TaskStub. This class is designed to be a subset of what a
task is in the domain layer: it doesn't contain input, message and
result. TaskStub should be used when listing tasks in order to avoid
extra-overhead of transporting these text fields.
As a result, the TaskDetails introduced by change
I0b406e4662cfd3cb496b71be77ad10a90c178baa doesn't need to exist.
With this patch, the domain layer contains Task and TaskStub instead
of Task, TaskStub and TaskDetails.
partially implements bp async-glance-workers
Change-Id: I2a20b0c5033c6920749370355c2d672dec707c28
Partial Task patch, Id I4fbadc9a97e3147128c7c733384c7bb50918806f
removed the result and message vars of the Tasks class. However,
the succeed and fail methods were using those to update them
respectively. This patch introduces a new TaskStub class which
enables the PartialTask functionality without looking result
and message attrs of the Task class.
Fixes bug 1284975
Change-Id: I0b406e4662cfd3cb496b71be77ad10a90c178baa
The current implementation always tries to get a transport from
oslo.messaging assuming the transport_url option has been set. This is
done to keep backwards compatibility. However, since the default
`rpc_backend` is rabbit, it'll always try to load such driver. The
problem raises when `kombu` is not installed and the `notifier_strategy`
is set to qpid. This will make glance-api fail because it'll try to load
the rabbit driver *before* loading the qpid one.
This patch puts the first load attempt in a try / except block. The
error caught (if any) will be re-raised *just* if transport_url has been
configured, otherwise it'll move forward and try to load the notifier
driver using the old configuration options.
The patch also removes an obsolete test.
Change-Id: I00e653704463ea8245e2960050e0d5e6839f278a
Closes-bug: #1302661
The set_time_override function in timeutils was written as a
helper function to mock utcnow for unittests before 'mock' was
generally used. Now that we have mock and fixture, we no longer
need to use it.
Change-Id: I203f7369943b948e0462becbddcba65b2d1576f0
Partial-Bug: #1266962
The deprecated notifier_strategy option is always used even if it is set
to its default value and the new notification_driver option is
explicitly set.
This change makes the notification_driver option work.
Change-Id: Icc2a063a828472132d2e2eda7f12e265db990ea1
Closes-Bug: #1271724
- partial-task: Task that includes all the values except the 'Text'
columns(input, result & message).
- This patch is to have the database to return a partial-task list on users
invoking 'GET /v2/tasks'.
- Returning list of partial tasks will help in reducing the response time when the
user queries for the list of tasks.
- This patch also includes changes for making implementation and return values of
display task detail (GET /v2/tasks/{task_id}) and list tasks (GET /v2/tasks)
more explicit.
partially implements bp async-glance-workers
Change-Id: I4fbadc9a97e3147128c7c733384c7bb50918806f
Enable H202 check: assertRaises Exception too broad.
Make changes to glance/tests/unit/test_notifier.py
Change-Id: I258367172e77312ddda06132d2e8371e8fff81e7
The backwards compatibility support turned out being a forward
compatibility blocker. This patch attempts to first load the notifier
transport using the transport_url and then, if the transport_url was not
configured, it loads the transport by using the old strategy config
params.
Closes-bug: #1263945
Change-Id: I4c80c64e2d1afaf518886b51a50300a576ee6317
Some of tests use different method of assertTrue(isinstance(A, B)) or
assertEqual(type(A), B). The correct way is to use assertIsInstance(A, B)
provided by testtools.
Change-Id: Ia8d38f73c159c7ef943a8f6cfe72b945cc493947
Closes-bug: #1268480
This commit makes the glance code base E125 and E126 compliant :
* E125 continuation line does not distinguish itself from next logical line
* E126 continuation line over-indented for hanging indent
Change-Id: I7120149bedb665fb66320498fe98948602a6cd52
Closes-bug: #1263437
- add REST API layer for async worker(tasks) to
support tasks operations.
- update to include sparse task for tasks.index
- add domain proxies for handling authorization,
policy enforce, notification while processing
task requests.
- add Task domain entity and TaskFactory to create
new Task domain entity objects.
- add integration test for testing tasks api
Co-authored-by: Fei Long Wang <flwang@cn.ibm.com>
Partially implement blueprint async-glance-workers
Change-Id: I072cbf351c06f59a7702733b652bfa63e5abbaa6
Glance currently uses a custom notifier and it has been maintaining it
for a long time. In a hope of reducing duplicated code and improving
cross-project contributions, this patch replaces the old notifier with
the one, recently developed, in oslo.messaging.
The oslo.messaging project is a port of the old oslo-rpc code to a
standalone, more stable and improved project. It brings all the benefits
that oslo-rpc would've brought as well as an easier way to integrate
with other projects.
This patch also:
- Reduces the code shipped along with Glance since all the code
copied from oslo-incubator related to the notifier is not needed
anymore.
- Improves the stability of existing, broker based, notifications.
- Brings HA support.
- Keeps backward compatibility by translating the old
`notifier_strategy` into oslo.messaging drivers.
Changes to the code:
- It is now necessary to pass the request context to the
notification call.
- Notifier package is no longer necessary. A notifier module was
added instead.
- New, notifier related, configurations were added.
- A lot of code was removed
Since there's still not an official release, requirements.txt points to
the latest tarball created. A release for oslo.messaging is planned for
Icehouse.
docImpact
Implements bp oslo-messaging
Change-Id: I8cd84772bc5867e06b2a50ed7e15b9e86f0b94ad
Includes:-
- Domain Task class
- Domain TaskFactory class
- DB TaskRepo class
- Necessary proxy classes for all of the above
- Tasks related exceptions
- Unit tests as applicable
Partially implement blueprint async-glance-workers
Change-Id: I619224bcc55b62303f3539454649528f2edc6e9d
Takashi Kajinami