Added centralized_db driver for image cache which will use
centralized database for storing cache related information.
Related blueprint centralized-cache-db
Change-Id: Iafaaa86666176cc95f77d85a4ab77286f0042bdd
In Xena we have mangaed to move all policy checks to API layer,
now removing the dead code from policy and authorization layer
NOTE: Some of the code is still being used from policy layer,
hence keeping it there only at this moment.
Change-Id: Ibee749cde20687d8c243cf84ae80b4de67d8ef3d
This is the same as the 'range' keyword in Python 3
Change-Id: If3aa008522c24e870b7bf13de32b8ed1b27cb519
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Glance was not supporting uwsgi deployment when we added periodic job
to prefetch images into cache. Later in ussuri uwsgi support was added
but we missed to implement periodic job to pre-cache the image.
This patch add this support for glance + uwsgi. For WSGI, we run the
prefetcher with an external lock, which makes sure that multiple API
workers will not attempt to cache an image at the same time. In this
case, if multiple workers attempt to run at the same time, only one
will grab the lock and do the work. When completed, the other worker
will grab the lock and either find all the work completed, or complete
new work that has been queued since the first one started.
Closes-Bug: #1939307
Co-Authored-By: Dan Smith <dms@danplanet.com>
Change-Id: I2abd1e60f414fbd68ce84e0b280f8b3e4e791a82
This patch enforces policy checks required for caching images
in API layer.
Partially-Implements: blueprint policy-refactor
Depends-On: https://review.opendev.org/c/openstack/nova/+/688802
Change-Id: Ie17b8f5bf308b8f07915ea18ace9b49955b8f0f0
md5 is not an approved algorithm in FIPS mode, and trying to
instantiate a hashlib.md5() will fail when the system is running in
FIPS mode.
md5 is allowed when in a non-security context. There is a plan to
add a keyword parameter (usedforsecurity) to hashlib.md5() to annotate
whether or not the instance is being used in a security context.
In the case where it is not, the instantiation of md5 will be allowed.
See https://bugs.python.org/issue9216 for more details.
Some downstream python versions already support this parameter. To
support these versions, a new encapsulation of md5() has been added to
oslo_utils. See https://review.opendev.org/#/c/750031/
This patch is to replace the instances of hashlib.md5() with this new
encapsulation, adding an annotation indicating whether the usage is
a security context or not.
Reviewers need to pay particular attention as to whether the keyword
parameter (usedforsecurity) is set correctly.
It looks like the usage of md5() here is solely to determine a checksum
of an image.
With this patch and the dependent patch for glance_store, all the
unit and functional tests pass on a FIPS enabled system.
Depends-On: https://review.opendev.org/#/c/756157
Depends-On: https://review.opendev.org/#/c/760160
Change-Id: I3b6d78d9792d4655bf0f4989cf82aced3f27491b
Registry service is deprecated and due for removal since past
couple of cycles. This patch removes functional and unit tests of
registry.
NOTE:
Skipped 'test_create_with_live_time' test as it was dependent on
test_registry_client and test_registry_api modules. Something is
wrong with the test and will be corrected once entire registry code
is removed from the code base.
Change-Id: I560ab5260bed7e43d83b67a00057ac48f9e366e9
Some tests compare timestamps (e.g. image created_at/updated_at
fields).
In some cases, subsequent timestamps may be identical, breaking the
assertions made by those tests.
One idea is to just add a 1ms sleep which should be a negligeable.
Mocking the functions used to retrieve timestamps may be undesireable.
For now, we'll do this only for Windows, where time.time has a lower
resolution compared to Linux (1e-7 as opposed to 1e-9).
At the same time, socket timeouts are rather inconsistent on Windows.
When a timeout is hit, recv may either throw ConnectionAbortedError
*or* return nothing. This needs to be taken into account when
expecting socket timeouts.
Change-Id: Ie5a4d8fb4c979c09eda2fdc0fad0baa1d1840c59
In order to run the unit and functional Glance tests on Windows, we
have to:
* avoid monkey patching the os module on Windows (which causes Popen
to fail)
* update sqlite connection URL
* avoid os.fork, not available on Windows.
* we'll use subprocess.Popen when spinning up http servers.
* for the really simple ones defined in the test helpers, we'll just
use threads
* do not attempt to connect to '0.0.0.0', use '127.0.0.1' instead
* some tests aren't properly skipped (xattr ones), so we're covering that
as well
* skip log rotation test, we can't move in-use files. Log rotation can
be performed by the log handler itself.
* expect an exception when hitting connection timeouts
* avoid installing unavailable test requirements (xattr, pysendfile)
* pin the instance creation timestamp. some tests that deal with
markers rely on ordering, which can be flipped if the timestamps are
identical (can happen in case of resources created one after the
other, not sure yet if this happens really fast or the clock isn't
accurate enough).
* add a few seconds to some timeouts (much needed when running the tests
in VMs).
blueprint windows-support
Change-Id: Ife69f56a3f9f4d81e1e2e47fde4778efd490938f
We would like to fully remove mox from the test tree. Even for tests
that don't use mox's validation, many of them are using the symbol
patching with self.stubs.Set. We can do the same thing with the
monkeypatch fixture instead.
This introduces self.stub_out to nova/test.py and an example of what a
stubs => stub_out change would look like.
The teardown function in the converted test was removed at the same
time, as those should no longer be used.
Part of the mox community goal for Rocky.
Change-Id: I8f471ff8fee600ebb4e8907bf240007b7b4fe59f
Signed-off-by: Chuck Short <chucks@redhat.com>
We have a ton of DeprecationWarning messages in our unit test runs.
Most of these are out of our control from third party libs. This
adds a WarningsFixture to limit warning output to once per test
run. In local py35 unit testing, this went from 14549 warnings to
8913.
Also including ignorning a policy 'is_admin' deprecation warning
that was added before a clear plan or replacement had been put
in place. Other projects have added this rather than fixing it
at the source since it is currently being reworked.
Based on work previously done in Nova and Cinder.
Change-Id: I4d97f74ed37b7b0e9a613ecfe33c4b26216ca768
* Use bytes for image content
* On Python 3, set_xattr() encodes the text value to UTF-8
* Open files in binary mode, not in text mode
* get_caching_iter(): pass a list as the image iterator, not a string.
On Python 3, list(b'abc') returns [97, 98, 99], whereas Python 2 returns
['a', 'b', 'c'].
* tox.ini: add glance.tests.unit.test_image_cache to Python 3.4
Change-Id: I638525d19c42990852cf45dd416318d9a847c303
Sqlite driver doesn't set the last accessed time when an image is
first added to cache. This makes the newly cached images susceptible
to pruning first instead of older images.
Change-Id: I46973a921c7cfb42811c58383e1b7a4004e70f27
Closes-bug: #1438564
Mainly to improve consistency, use range() from six.moves
renames across glance.
Behaves consistently like py2 xrange() and py3 range().
Removes unnecessary range() from glace/api/v2/images.py
Change-Id: Id21f923d05600b902f2239e25ef01716c07e74a3
Args of assertEquals method in glance.tests are arranged
in wrong order. In result when test fails it shows incorrect
information about observed and actual data. It's found more
than 2000 times. Right order of arguments is "expected, actual".
Change-Id: Ifea4809f5a696b608a19e36a9ed9e5eec46c8a21
Co-Authored-By: Li Yingjun <liyingjun1988@gmail.com>
Closes-Bug: 1277104
This commits removes the old `store` package from glance and adopts the
usage of the new glance.store library. The library was designed to
preserve backwards compatibility as much as possible. In fact, most of
the changes in this patch are related to function args ordering and not
function renames or workflow changes.
Some changes that are worth mentioning:
1. Glance store doesn't rely on a global config object. All config
options must be explicitly registered.
2. All store operations now accepted an optional context. This is a
fallout from the context not being required in the `Store` constructor
anymore.
3. Store drivers are behind a private package called `_drivers` and
they're not suppose to be accessed directly. Instead, functions like
`get_store_from_scheme` should be used.
4. Stores are disabled by default
5. All the store specific options are under the `glance_store` group.
DocImpact:
The old store related configuration options have been moved under the
`glance_store` section. However, the old options will go through a
deprecation path. That is, they'll still be read from the `DEFAULT`
section to give deployers enough time to update their config files.
In k-2, the deprecated options will be completely obsolete.
Closes-bug: #1291848
Implements-blueprint: create-store-package
Change-Id: Iaacc70993ad5da292b93de42bbecda73d53b19fd
to keep Python 3.x compatibility, use six.StringIO/BytesIO to
replace StringIO.StringIO
StringIO works for unicode
BytesIO works for bytes
Change-Id: I93e043c633e1de9e4dedcb0a313032403b6a70fb
Closes-Bug: #1280100
Enable hacking H301: one import per line. H304 and H302 check enables too,
so we temporarily disable it since it does not pass.
Fix imports in files below.
Add # noqa to migration scripts.
Change-Id: I6a8d4dcd9c3195d2848f218aafe304b1240ab60c
Enable F841 check: local variable 'name' assigned but never used.
Make appropriate changes to files listed below.
Change-Id: I02837d4abf421dc9d85f3b01587120fd68acfa12
in python3.x, there is no method named xrange(),it
has replaced by range(),which is equal to xrange() in
python2.x. so we must fix this issue.we use six module
to fix this issue.
Change-Id: If4fd7478865e8024025af3e57e42f2a24e1e2bbc
Closes-Bug: #1268439
Some of tests use different method of assertTrue(isinstance(A, B)) or
assertEqual(type(A), B). The correct way is to use assertIsInstance(A, B)
provided by testtools.
Change-Id: Ia8d38f73c159c7ef943a8f6cfe72b945cc493947
Closes-bug: #1268480
Clean up imports due to F401 and F403 checks.
F401 'module' imported but unused
F403 unable to detect undefined names with wildcard import
Change-Id: I487edb157de1a6babc7ad8a3fb65f195e476c490
Replace numeric expressions with olso constants to make code
more readable. This patch won't replace single 1024 if 1024 not
in an expression like number * 1024.
Change-Id: Ic1137774f0f9a2be89a7c2706d6eb52eb906ff1f
Instead of globally ignoring Pyflakes and Hacking
warnings, blacklist explitly only those that trigger
numerously. Fix the rest alongway and start to gate
on those that are now passing.
Change-Id: Ia19dc708cf0ccab2da5b46d1652f16e901499c24
Based on current implement of method delete_stalled_files,
it's just deleting all the images located in /incomplete
instead of using the param 'older_than' to check if it
should be deleted. So this fix will use 'older_than' to
confirm brfore deleting.
Fixes bug 1228256
Change-Id: Ica27915f8b9b098ecae1e38c8e678d6acf4f2f76
Remove the useless arg ("start index" = 0) in files, since its default
value is 0, to make code cleaner.
Change-Id: I9d91ab8a8033b9a6c9b77608dea92c91a994c7fc
Mismatched indentation meant that more chunks than
expected could be yielded.
Make both filesystem ChunkedFile and S3 ChunkedFile
handle the case where the filepointer no longer exists.
Added unit tests which reproduced the problem
using both a filesystem and S3 ChunkedFile.
Fixes bug 1194929.
Change-Id: I0f70155e59f6bc754c88b5b9e96645459bfe770a
testtools.addCleanup is a more resilient way to perform cleanup activities,
as it will continue to clean things up even if there are unforseen problems.
Specifically, replace custom management of tempdirs with fixtures.TempDir
and replace tearDown methods that can be easily replaced with calls to
addCleanup in the setUp method. There are at least two temp dir creations that
did not have a corresponding cleanup in this patch, which is another reason
for using useFixture(fixtures.TempDir) instead of calls to mkdtemp.
Part of blueprint grizzly-testtools.
Change-Id: I4eb548010612bd5a8d30e8e2304fa66d3d5ffb7c
1) caching_iter doesn't handle backend exceptions:
caching_iter assumes any exception that occurs is the result of being
unable to cache. Hence the IOError raised from size_checked_iter, which
indicates a problem with the backend, means the caching_iter will
continuing trying to serve non-existent data. The exception was not
been re-raised in this case, making wsgi keep the connection open and
clients stuck forever waiting for more data.
Raising a GlanceException in size_checked_iter rather than an IOError
allows caching_iter to distinguish between a problem fetching data, and
a problem writing to the cache.
2) Checksum verification happens after cache commit rather than before:
This block was outside the context manager block which meant the
GlanceException was not caught by open_for_write and the rollback didn't
happen. This resulted in an error been logged, but the bad image still
placed in and subsequently served from the cache.
Also:
* Fix test_gate_caching_iter_bad_checksum - the loop to consume the
iterator in was in a subroutine that never got called.
* Move test_gate_caching_iter_(good|bad)_checksum into
ImageCacheTestCase to excercise both the sql and xattr drivers.
* Remove invalid registry_host/registry_port params from
TestImageCacheXattr/TestImageCacheSqlite setup which caused a failure
when testing the file on it's own using nosetests.
Fixes bug 1045792
Change-Id: I8aedec347e7f50566c44c5b6c6db424573c5ebaf
This handles a situation where the pruner would raise an exception when
the image_cache_max_size was set lower than the size of the last image.
Fixes bug 1039854
Change-Id: I4a6c164e9f821f1250314974f829f058fbd02863
On an image GET, recalculate the image checksum as the image
data is streamed to the client. Verify that the checksum matches
the original checksum calculated when the image was added to Glance.
If checksum validation fails, purge the image from the cache.
This type of situation could occur if the backend image store
is malfunctioning.
bug 1028496
Change-Id: I9f38bac8360016bb12b5edaad87c50939a538cc0
Fixes LP bug 1031842
Use a finally clause for the xattr and sqlite
image cache implementations of open_for_write()
such that if the commit() or rollback() aren't
called and the image remains in the incomplete
dir that it is moved to the invalid dir
Change-Id: Id9a0ac56a8ae9fd03af9ccc4989c9b304c95bdde
Abhishek Kekane