This patch extends the functionality of the API
``GET /v2/info/stores/detail`` to expose store details
of other stores. Currently the ``stores-detail`` API
exposes store details of RBD backend.
Implements: blueprint expanding-stores-detail
Change-Id: I0c7bbc315b56dc0e40397b4ed8a68d1668203c44
This patch adds a new API to glance ``GET /v2/info/stores/detail``
to expose the stores specific details about the store like store
type and other specific store properties.This operation
will be admin only and validated by the new policy rule
``stores_info_detail`` which defaults to admin only
Implements: blueprint expose-store-specific-info
Change-Id: I6882fd2381e6ae245fd8c61bf9f4d52df2b216f5
This patch updates the location URL of the legacy images while
upgrading from single cinder store to multiple stores.
It does that with the help of lazy loading logic i.e. while
GET images call, it checks the location URL and metadata
of the image against the configured store ids and updates
images to respective stores on the basis of volume type (comparing
image-volume's type with the configured cinder_volume_type).
Legacy image URL:
cinder://<volume-id>
New image URL:
cinder://<store-id>/<volume-id>
NOTE: bumping lower-constraints/requirements of glance-store to 2.3.0 as
it includes changes[1] that are a hard requirement for cinder multiple
stores to work with glance
[1] https://review.opendev.org/#/c/746556/
Change-Id: I087a89c20813378fea8ff22ddf81d7a10c220db3
Implements: blueprint multiple-cinder-backend-support
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib.
Change-Id: I44e7b6f76e2d12f620ec602afc77ce11ba6b9d9a
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
This change introduces new 'v2/stores/<store_id>/<image_id>'
endpoint that accepts 'DELETE' method request. Once successful
the request will delete the image <image_id>'s location that
matches the store <store_id>. If the store is not read-only
or return image in use exception the image data will be
deleted. In the case of read-only store, the location will
be removed and if the image in use is raised, the call will
fail.
bp: delete-from-store
Co-authored-by: Brian Rosmaita <rosmaita.fossdev@gmail.com>
Change-Id: I1cb45026489a96a283b82e8e7efc9975c181fceb
Registry service is deprecated and due for removal since past
couple of cycles. This patch removes functional and unit tests of
registry.
NOTE:
Skipped 'test_create_with_live_time' test as it was dependent on
test_registry_client and test_registry_api modules. Something is
wrong with the test and will be corrected once entire registry code
is removed from the code base.
Change-Id: I560ab5260bed7e43d83b67a00057ac48f9e366e9
Around 11 unit tests are failing [1] with a recent patch [2] merged in
glance_store. The reason behind the failure is glance unit tests are
loading rbd store which is not supported in gate. Earlier these tests
were passing because they were not trying to connect to rados while
loading the store, but after merging of recent patch [1] it tries to
connect to rados via configure_add method of rbd driver.
To avoid these kind of failures in near future we should drop using
rbd store in tests as we know it is not supported in gate.
NOTE:
Also corrected the naming conventions of store identifiers, instead
of using file1 etc., used appropriate names.
[1] http://logs.openstack.org/40/663740/1/check/cross-glance-py27/50a9a9c/
[2] https://review.opendev.org/660340
Change-Id: I18add352238e93c5d4ade8b3e8d967b2f639cdad
Closes-Bug: #1831963
Made changes to discovery store to show 'http'
store as 'read-only' if it is configured under
'enabled_backends' cofiguration option.
Change-Id: If5d292bbb4cef1bd21a7e6cd4ef343a767eb91a4
In order to run the unit and functional Glance tests on Windows, we
have to:
* avoid monkey patching the os module on Windows (which causes Popen
to fail)
* update sqlite connection URL
* avoid os.fork, not available on Windows.
* we'll use subprocess.Popen when spinning up http servers.
* for the really simple ones defined in the test helpers, we'll just
use threads
* do not attempt to connect to '0.0.0.0', use '127.0.0.1' instead
* some tests aren't properly skipped (xattr ones), so we're covering that
as well
* skip log rotation test, we can't move in-use files. Log rotation can
be performed by the log handler itself.
* expect an exception when hitting connection timeouts
* avoid installing unavailable test requirements (xattr, pysendfile)
* pin the instance creation timestamp. some tests that deal with
markers rely on ordering, which can be flipped if the timestamps are
identical (can happen in case of resources created one after the
other, not sure yet if this happens really fast or the clock isn't
accurate enough).
* add a few seconds to some timeouts (much needed when running the tests
in VMs).
blueprint windows-support
Change-Id: Ife69f56a3f9f4d81e1e2e47fde4778efd490938f
As of version 3.5.0 moxstub will be deprecated, so remove it where it has been used.
Change-Id: I91861ffd6d3a766e9a8b300de51dc478de8b6ebb
Signed-off-by: Chuck Short <chucks@redhat.com>
We would like to fully remove mox from the test tree. Even for tests
that don't use mox's validation, many of them are using the symbol
patching with self.stubs.Set. We can do the same thing with the
monkeypatch fixture instead.
This introduces self.stub_out to nova/test.py and an example of what a
stubs => stub_out change would look like.
The teardown function in the converted test was removed at the same
time, as those should no longer be used.
Part of the mox community goal for Rocky.
Change-Id: I8f471ff8fee600ebb4e8907bf240007b7b4fe59f
Signed-off-by: Chuck Short <chucks@redhat.com>
Added some unit tests for coverage purpose.
Added functional tests for create and import scenarios.
Note:
For functional tests I have considered file store with two
different image directories.
Related to blueprint multi-store
Change-Id: I59e28ab822fb5f6940f48ddbf6dfba4cb7d4c509
Config option sqlite_db is deprecated in
0a1bae9859079fb21a03716be947c5f1da6db0a2, and deprecate
argument sqlite_db in method set_defaults in
18d26125d02c8a017d8541339737981bf242616b, Glance doesn't
use config option sqlite_db's value, but config option
connection instead. For more details about sqlite connection
string, please see:
http://docs.sqlalchemy.org/en/rel_1_0/core/engines.html#sqlite
Change-Id: I6eadd9d0c3c53d1c8a1afb441d5bdf647f4925e0
verbose option is going to be deleted soon (see
https://review.openstack.org/#/c/206437/ for more info
why it deprecated in oslo.log) because it confuses users
(we already have debug option). So we need to delete it from glance
tests.
Change-Id: Iade46a6097d153f2df0da73554bbfd3e2b1704f5
The new release of glance_store (0.5.0) included
I1d4f95cba47d21f1eb7e580314d01ac8d3481586, which raises errors if
filesystem_store_datadir or filesystem_store_datadirs is not set. Glance
currently is not correctly populating either of those options in certain
unit or functional tests, causing a lot of test failures. This patch
aims to unblock the gate by overriding the CONF option.
Change-Id: I3a9e0c7bfa4a9f38b41e93dce9652c4cc4811a9b
Closes-Bug: #1454384
The Oslo libraries have moved all of their code out of the 'oslo'
namespace package into per-library packages. The namespace package was
retained during kilo for backwards compatibility, but will be removed by
the liberty-2 milestone. This change removes the use of the namespace
package, replacing it with the new package names.
The patches in the libraries will be put on hold until application
patches have landed, or L2, whichever comes first. At that point, new
versions of the libraries without namespace packages will be released as
a major version update.
Please merge this patch, or an equivalent, before L2 to avoid problems
with those library releases.
Blueprint: remove-namespace-packages
https://blueprints.launchpad.net/oslo-incubator/+spec/remove-namespace-packages
Change-Id: Ifa8baab33cdb3e606cf175a8c29c3a4ef6c44480
This change:
- Adds oslo.policy to the requirements list
- Accounts for changes in Enforcer initialization
- Accounts for changes to config options
- Removes incubated version of oslo.policy
- Updates the in-tree etc/config files
UpgradeImpact
Partially Implements Blueprint: graduate-policy
Change-Id: I5acb1e0f809098991f05ca3b6d78d4d88d98f2db
oslo_db was moved out of the oslo namespace in
oslo.db>=1.4.0.
Change-Id: I13620d26ff12c55b2294b3b051669e0e98513a10
Related-to: blueprint drop-namespace-packages
oslo_config was moved out of the oslo namespace in oslo.config>=1.6.0.
Related-to: blueprint drop-namespace-packages
Change-Id: I30ecbf8f9de77496fcb643c7ad9738d79ad359f0
oslo.concurrency-0.3.0 has moved its path to oslo_concurrency,
the old path oslo.concurrency can still work but is deprecated now.
Change-Id: Idbf6ea10ddf6fdab9695c6dfb132bd720394ffdf
Closes-Bug: #1398656
Replace processutils and lockutils modules of oslo-incubator with
oslo.concurrency lib.
Change-Id: Ic1af8753a70f1aada22efe8132e48cbc16e14f3f
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
1. Sync glance.openstack.common.policy up to latest
version from oslo-inc.
2. Clean useless modules which depended by policy
module and pinned gettextutils module there. For
latter one, we are going to use glance.i18n instead.
* jsonutils
* strutils
docImpact
Closes-bug: #1288178
Closes-bug: #1387973
Partial-bug: #1381870
Change-Id: I84511ab1ee600e618985448dfbfbdc26cb130370
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
Fix a bug with storing the request context in thread local where the
storage was not properly cleared between requests.
(This was causing the same request to always be logged.)
Closes-Bug: 1384911
Change-Id: Ie30b93900451b0f9f6fd8158457b0b66903af470
This commits removes the old `store` package from glance and adopts the
usage of the new glance.store library. The library was designed to
preserve backwards compatibility as much as possible. In fact, most of
the changes in this patch are related to function args ordering and not
function renames or workflow changes.
Some changes that are worth mentioning:
1. Glance store doesn't rely on a global config object. All config
options must be explicitly registered.
2. All store operations now accepted an optional context. This is a
fallout from the context not being required in the `Store` constructor
anymore.
3. Store drivers are behind a private package called `_drivers` and
they're not suppose to be accessed directly. Instead, functions like
`get_store_from_scheme` should be used.
4. Stores are disabled by default
5. All the store specific options are under the `glance_store` group.
DocImpact:
The old store related configuration options have been moved under the
`glance_store` section. However, the old options will go through a
deprecation path. That is, they'll still be read from the `DEFAULT`
section to give deployers enough time to update their config files.
In k-2, the deprecated options will be completely obsolete.
Closes-bug: #1291848
Implements-blueprint: create-store-package
Change-Id: Iaacc70993ad5da292b93de42bbecda73d53b19fd
Replace common oslo code glance.openstack.common.db
by usage of oslo.db library and remove common code.
Change-Id: I27216e03763f72e8665121d4440ba2828349eac5
Currently Sheepdog store driver is using configure_add() to handle
those essential options, which are not only be used for image 'add'
handling but also for other key functions e.g. 'get_size', 'get' and
etc. When store driver raises BadStoreConfiguration exception from
configure_add(), Glance will just makes this store be readonly instead
of fully disabled, so the handling for all the essential configurations
should be implemented in configure(), and then if any option of them get
failure the store will be fully disabled instead of being readonly
partially.
Closes-bug: 1302658
Change-Id: Idaa96036515f04ee8414f30990ee638855ede2d2
Signed-off-by: Zhi Yan Liu <zhiyanl@cn.ibm.com>
Customers using a VMware environment with OpenStack should be able to
store their Glance images in VMware datastores. This is a first step to solve
the problem where Nova needs to copy the bits over the network
from Glance to the datastore when spawning an instance.
Also, this give the ability to provide some optimizations for specific
image formats in the future (fast cloning for example).
This patch contains a 'glance/store/vmware/' folder with the code
to manage the connection with vCenter or an ESX(i) host.
This code will go away as soon as it is merged to Olso:
see review https://review.openstack.org/#/c/65075/
The current implementation give this ability to specify the vCenter or
ESX(i) IP. In case of a vCenter IP, there is no optimization to reduce
the datapath (no host selected).
Consequently, it is recommended to specify an ESX IP if the ESX host
API endpoint is accessible from Glance.
docImpact
Implements bp vmware-datastore-storage-backend
Change-Id: I3837912e0d1614b9c31a689f71c2e34d453e2dc3
Every call to json.loads/json.dumps have been changed to
jsonutils.loads/jsonutils.dumps respectively. Import json has been removed
also replaced by import glance.openstack.common.jsonutils
654d80b416dc5f413cb791aa838ec8688bf7da44 Create openstack.common.jsonutils
Change-Id: I8ef580e5eb91526dfaef37050ce1f5c6d88d56b5
Closes-bug: #1257922
Use common oslo database code for work with session, connection, engines
instead of glance implementations.
Work with config options modified due to new common code config.
Removed unused glance code and tests for it.
Details:
glance/cmd/manage.py
- removed load load the glance-registry config. We load sql connection
in common code now.
glance/db/sqlalchemy/api.py
- removed functions _wrap_db_error(), _is_db_connection_error(),
_get_maker(), get_engine(), _ping_listener() and config
`sql_connection_opt`. We have all these thigs in module
`glance.openstack.common.db.sqlalchemy`.
- modified functions setup_db_env(), add_cli_options(), clear_db_env().
- functions _get_session() and get_engine() now call same functions
from module `glance.openstack.common.db.sqlalchemy`.
- replaced sqlalchemy exception `sqlalchemy.exc.IntegrityError` to
database duplicate exception - `db_exc.DBDuplicateEntry`.
Removed redundant tests - we use common code , so a lot of features was
already tested in Oslo.
Global variable _ENGINE was replaced by common function get_engine().
DocImpact
New database config options came from Oslo. See Table below
-------------------------------------------------------------------
Database config options ([group] option)
-----------------------------|-------------------------------------
Glance | Oslo
-----------------------------|-------------------------------------
[DEFAULT] sql_connection | [database] connection
[DEFAULT] sql_idle_timeout | [database] idle_timeout
[DEFAULT] sql_max_retries | [database] max_retries
[DEFAULT] sql_retry_interval | [database] retry_interval
[DEFAULT] db_auto_create | NONE
[DEFAULT] sqlalchemy_debug | [database] connection_debug
[DEFAULT] use_tpool | [database] use_tpool
NONE | [database] slave_connection
NONE | [database] min_pool_size
NONE | [database] max_pool_size
NONE | [database] slave_connection
NONE | [database] min_pool_size
NONE | [database] max_pool_size
NONE | [database] max_overflow
NONE | [database] connection_trace
NONE | [database] pool_timeout
-------------------------------------------------------------------
blueprint db-use-oslo-common-code
Change-Id: I3ff976545b1a82ff8df780e34128fcaf6f892b8c
Clean up imports due to F401 and F403 checks.
F401 'module' imported but unused
F403 unable to detect undefined names with wildcard import
Change-Id: I487edb157de1a6babc7ad8a3fb65f195e476c490
Stub out the sheepdog Store dependency on subprocess communication
with collie. This results in many unnecessary pipes being created
on POSIX systems.
For unit testing purposes, do not create the sheepdog store. (There
are no unit tests for this functionality anyway.)
Change-Id: I54014a502891ae52bb567f3fae32e6ff0c74332a
Fixes time-dependent bug in test case stubs setup that could result
in stubs being unset by GC.
bug 1205091
Change-Id: Ic0d97bd1f96e42631c1ba3e1eaf4cf74b36140b2
This change fixes issues that would arise when testing certain files on
their own. Normal test runs were not affected due to serendipity in the
implied import order.
Change-Id: I9a85bf0112a9796482624e860efc2ec620aaf0f2
The patch implements Registry's v2 Client.
Some notes:
* The implementation uses __getattr__ to map method's calls to remote
RPC methods.
* The client supports both bulk and single calls.
* Client is based on the base HTTPClient
About Exceptions:
* If raise_exc == True, the client will try to re-raise the remote
exception. If the server raised an unknown exception, it will be
re-raised as RPCException in the client side.
Implements blueprint registry-api-v2
Change-Id: I98757c2272c68301373c1a3af96c4d06cb63cc97
The cfg API is now available via the oslo-config library, so switch to
it and remove the copied-and-pasted version.
Add the 2013.1b3 tarball to tools/pip-requires - this will be changed
to 'oslo-config>=2013.1' when oslo-config is published to pypi. This
will happen in time for grizzly final.
Remove the 'deps = pep8==1.3.3' from tox.ini as it means all the other
deps get installed with easy_install which can't install oslo-config
from the URL.
Retain dummy cfg.py file until keystoneclient middleware has been
updated (I18c450174277c8e2d15ed93879da6cd92074c27a).
Change-Id: I4815aeb8a9341a31a250e920157f15ee15cfc5bc
Patchset addresses reviewer's comments, rebase against master,
and follows new guidance from Foundation:
http://wiki.openstack.org/Documentation/Copyright
Change-Id: I94367461505778973528eb8835d991f4fb796dab
testtools.addCleanup is a more resilient way to perform cleanup activities,
as it will continue to clean things up even if there are unforseen problems.
Specifically, replace custom management of tempdirs with fixtures.TempDir
and replace tearDown methods that can be easily replaced with calls to
addCleanup in the setUp method. There are at least two temp dir creations that
did not have a corresponding cleanup in this patch, which is another reason
for using useFixture(fixtures.TempDir) instead of calls to mkdtemp.
Part of blueprint grizzly-testtools.
Change-Id: I4eb548010612bd5a8d30e8e2304fa66d3d5ffb7c
Adds a missing CONF import for the filesystem_store_datadir.
This is required in order to make the policy tests (test_policy.py)
run as a stand alone suite.
Fixes: NoSuchOptError: no such option: filesystem_store_datadir
Fixes LP Bug #1098670.
Change-Id: If8f9db3962b070e7841e4e3ef531dda4645624b7
The call to create_stores was happening in the v1 and v2 API
controllers. This moves that call and the verification that
the default_store exists into the necessary binaries.
Fixes bug 1039727.
Change-Id: I36542fd9b1d536d6266898766317abe110bb71a2
This patch adds a policy, 'download_image', to be enforced when image data is
retrieved. It also does some basic refactoring of how policies are enforced.
Fixes bug 1038086
Change-Id: Idd844b615d362eae3197e106067c29dba8e3eeda
Adds context to the constructor for the Store class. This can be used
to provide Store's access to the RequestContext.
Includes the following changes:
* Updates common store helper methods to create new instance of
Store objects for each request.
* Updates Glance scrubber to create a real context from credentials.
The motivation for this change is that we will need access to the
service catalog and token for the Swift multi-tenant backend.
Partially implements blueprint: swift-tenant-specific-storage
Change-Id: I3def2b7c4085a36dd6c0961e762783ebaf7ffa74
Updates the default value for the 'known_stores' config
variable to include the full list of storage classes
supported in Glance.
Also, removes the known_stores config section from the
example config file since it is arguably an advanced config
that won't get used by most users. Removing this config avoids
the overhead of maintaining internal class names in config files.
Fixes LP Bug #1008698.
Change-Id: I0117376aa4de3103410ecb1a36df6998fcd0d5b5
Mridula Joshi