oslo.concurrency 6.0.0 has relocated the lock_path that is required
for external locks from [DEFAULT] to the [oslo_concurrency] config
section. Instead of messing with configuration, we can use the
lockutils.set_defaults() method to set this for tests. So add this
to the BaseTestCase so that it's available for all tests.
Change-Id: I2878b4162dddaf935002c168bc544eb1eff71efd
Till now glance was dependent on periodic job to get image
cached locally and also has dependency on ``cachemanage``
middleware for the same.
This patch removes the periodic job and dependency of
``cachemanage`` middleware and initiates immediate caching
of an image via ``PUT /cache/image_id`` API call.
Co-Author: Dan Smith <dms@danplanet.com>
Implements: blueprint instant-caching
Change-Id: I9ab3f1b7595e22dbb03af95168314352a44eb930
This was already fixed in 4889dc1814 but
we did not enforce this rule and reintroduced "bad" string
interpolations. This patch adds a hacking rule to prevent us from doing
this again in the future.
Change-Id: I96d9a157d3887286542859d67138ffdae5a589f1
This is available on all supported Python versions.
Change-Id: I76a696e56e04ec47b107af8e169c04138cf45fbb
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
This is a rather beefy change due to the number of usages of this
import. The changes are trivial though.
Change-Id: I7badeeaca438b0291f4ed86670e7f217e6372c61
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
As per the community goal of migrating the policy file
the format from JSON to YAML[1], we need to do two things:
1. Change the default value of '[oslo_policy] policy_file''
config option from 'policy.json' to 'policy.yaml' with
upgrade checks.
2. Deprecate the JSON formatted policy file on the project side
via warning in doc and releasenotes.
Also convert the ./glance/tests/etc/policy.json to policy.yaml
file. Replace policy.json to policy.yaml ref from doc and tests.
[1]https://governance.openstack.org/tc/goals/selected/wallaby/migrate-policy-format-from-json-to-yaml.html
Depends-On: https://review.opendev.org/c/openstack/nova/+/773192
Change-Id: I17d0374dd4223688e5f95253802a4ae87377953a
This adds a FakeData file-like generator that is able to generate
arbitrary amounts of data when we need to simulate reading from a
file or request pipeline, without having to store all of that in
memory.
Change-Id: Iff1fbe2b55f4be12e69c9fd3dec7e3b3e2593e53
There is something about the process handling in the functional test base
that opens the scenario where we might signal our API master child,
waitpid() on it and never return (see bug 1891190). Specifically the
test_reload() case, where the API master is being sent SIGHUP various times,
which may cause it to temporarily mask out SIGTERM or something, which
means it ignores our request. Either way, a deadlock in the tests can
occur when the cleanup() routine goes to waitpid() on that master,
forever.
This change makes our wait_for_fork() helper call waitpid() with WNOHANG
and progressively ramp up the aggression on asking it to stop. First, it
waits for the original signal to be honored, then it starts sending SIGTERM,
followed by SIGKILLs until the overall timeout expires, after which it
fails the test. The latter should only ever happen if something gets
wedged in uninterruptible sleep.
Related-Bug: #1891190
Change-Id: Ie0e91e2ee86f33bcdb63eed74f2d9948108acfd4
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib.
Change-Id: I44e7b6f76e2d12f620ec602afc77ce11ba6b9d9a
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
Added new import method 'copy-image' which will copy existing image into
specified list of stores. Introduced additional task which will serve
as internal plugin which will allow copying existing image into staging
area and then this data will be uploaded to specified stores via regula
import flow.
NOTE: This new import method 'copy-image' is only supported if multiple
stores are enabled in deployment.
APIImpact
Implements: blueprint copy-existing-image
Change-Id: I13eaab7ab013f44ce18465bdbdbe8052942570ff
Registry service is deprecated and due for removal since past
couple of cycles. This patch removes functional and unit tests of
registry.
NOTE:
Skipped 'test_create_with_live_time' test as it was dependent on
test_registry_client and test_registry_api modules. Something is
wrong with the test and will be corrected once entire registry code
is removed from the code base.
Change-Id: I560ab5260bed7e43d83b67a00057ac48f9e366e9
Some tests compare timestamps (e.g. image created_at/updated_at
fields).
In some cases, subsequent timestamps may be identical, breaking the
assertions made by those tests.
One idea is to just add a 1ms sleep which should be a negligeable.
Mocking the functions used to retrieve timestamps may be undesireable.
For now, we'll do this only for Windows, where time.time has a lower
resolution compared to Linux (1e-7 as opposed to 1e-9).
At the same time, socket timeouts are rather inconsistent on Windows.
When a timeout is hit, recv may either throw ConnectionAbortedError
*or* return nothing. This needs to be taken into account when
expecting socket timeouts.
Change-Id: Ie5a4d8fb4c979c09eda2fdc0fad0baa1d1840c59
In order to run the unit and functional Glance tests on Windows, we
have to:
* avoid monkey patching the os module on Windows (which causes Popen
to fail)
* update sqlite connection URL
* avoid os.fork, not available on Windows.
* we'll use subprocess.Popen when spinning up http servers.
* for the really simple ones defined in the test helpers, we'll just
use threads
* do not attempt to connect to '0.0.0.0', use '127.0.0.1' instead
* some tests aren't properly skipped (xattr ones), so we're covering that
as well
* skip log rotation test, we can't move in-use files. Log rotation can
be performed by the log handler itself.
* expect an exception when hitting connection timeouts
* avoid installing unavailable test requirements (xattr, pysendfile)
* pin the instance creation timestamp. some tests that deal with
markers rely on ordering, which can be flipped if the timestamps are
identical (can happen in case of resources created one after the
other, not sure yet if this happens really fast or the clock isn't
accurate enough).
* add a few seconds to some timeouts (much needed when running the tests
in VMs).
blueprint windows-support
Change-Id: Ife69f56a3f9f4d81e1e2e47fde4778efd490938f
As of version 3.5.0 moxstub will be deprecated, so remove it where it has been used.
Change-Id: I91861ffd6d3a766e9a8b300de51dc478de8b6ebb
Signed-off-by: Chuck Short <chucks@redhat.com>
In web-download import method functional tests we are trying to download
a file from 'https://www.openstack.org/assets/openstack-logo/2016R/OpenStack-Logo-Horizontal.eps.zip'.
Here we are assuming image is downloaded and will be active within 20
seconds else will be marked as failed. As of now these tests never
fails in local environment but, external networking will always be unreliable
from the CI environment which sometimes causes these tests to either
timeout or failure.
Instead of using external URL to download the image, deploying local http
server to dowload local image.
Closes-Bug: #1797571
Change-Id: I781666327f092ad439c23eca57128b8fd0334c89
We would like to fully remove mox from the test tree. Even for tests
that don't use mox's validation, many of them are using the symbol
patching with self.stubs.Set. We can do the same thing with the
monkeypatch fixture instead.
This introduces self.stub_out to nova/test.py and an example of what a
stubs => stub_out change would look like.
The teardown function in the converted test was removed at the same
time, as those should no longer be used.
Part of the mox community goal for Rocky.
Change-Id: I8f471ff8fee600ebb4e8907bf240007b7b4fe59f
Signed-off-by: Chuck Short <chucks@redhat.com>
This follows changes in Nova and Cinder to add a logging fixture that
allows the log level to be raised to limit the amount of junk output
in the unit test logs, while still processing debug level logs to
catch formatting error issues.
Change-Id: If740108db5291ff3fe0963127f9a3efadf3346c3
Modified 'glance-manage db_sync' operation to use
expand, migrate and contract operations.
Added test queens scripts for testing purpose only.
This patch removes use of "monolithic" upgrade scripts
and resolve the issue while upgrading from ocata to pike.
Co-Authored-By: Shilpa Devharakar <Shilpa.Devharakar@nttdata.com>
Closes-Bug: #1723001
Change-Id: I2653560d637a6696f936b49e87f16326fd601dfe
We have a ton of DeprecationWarning messages in our unit test runs.
Most of these are out of our control from third party libs. This
adds a WarningsFixture to limit warning output to once per test
run. In local py35 unit testing, this went from 14549 warnings to
8913.
Also including ignorning a policy 'is_admin' deprecation warning
that was added before a clear plan or replacement had been put
in place. Other projects have added this rather than fixing it
at the source since it is currently being reworked.
Based on work previously done in Nova and Cinder.
Change-Id: I4d97f74ed37b7b0e9a613ecfe33c4b26216ca768
Fixed the tests in the `glance.tests.integration` module. Enabled
them to run for py35 job in tox.ini. All changes are related to
string/bytes object differences in python 2.7 vs 3.5.
Also added testing of tasks location header value due to a change in
how it is handled in py27 vs py35.
Change-Id: I26aae0c518fe84c9967461a3b9fef02bc0c17923
'glance-manage db purge' command fails with DBReferenceError due
to FK constraint failure and exits with stack-trace on the command
prompt.
Made changes to give user-friendly error message to the user as
well as log appropriate error message in glance-manage logs
instead of stack-trace.
Co-author-by: Dinesh Bhor <dinesh.bhor@nttdata.com>
Change-Id: I52e56b69f1b78408018c837d71d75c6df3df9e71
Closes-Bug: #1554412
The parent of this patch introduced the change to Alembic-based
migrations. This commit builds on top of that by adding expand,
migrate and contract commands to the glance-manage tool.
Appropriate documentation is updated and failing tests are adjusted
to accomodate the new database versioning schema.
Data migrations are expected to be run in the background with older
Glance services being active during the upgrade process.
Partially-Implements: blueprint database-strategy-for-rolling-upgrades
Co-Authored-By: Hemanth Makkapati <hemanth.makkapati@rackspace.com>
Change-Id: Ie839e0f240436dce7b151de5b464373516ff5a64
Depends-On: I77921366a05ba6f9841143af89c1f4059d8454c6
* Functional tests now use alembic instead of sqlalchmey-migrate
to build and destroy test database.
* All tests now use a file-based sqlite db as opposed to an in-memory
database.
Partially-Implements: blueprint alembic-migrations
Change-Id: I77921366a05ba6f9841143af89c1f4059d8454c6
Depends-On: Ie8594ff339a13bf190aefa308f54e97ee20ecfa2
This change replaces the existing boolean 'is_public' column for
the 'images' table with enum 'visibility' column featuring the
four explicit visibility values - public, private, shared,
and community.
This change also implements and enables all backend code to
utilize the new values.
Co-Authored-By: Timothy Symanczyk <timothy_symanczyk@symantec.com>
Co-Authored-By: Dharini Chandrasekar <dharini.chandrasekar@intel.com>
Implements: blueprint community-level-v2-image-sharing
Closes-Bug: #1394299
Closes-Bug: #1452443
Depends-On: I6e3268f3712cbc0aadb51d204c694023b92d55a5
Change-Id: I94bc7708b291ce37319539e27b3e88c9a17e1a9f
Fix for a bug 1599123. URL construction is now considering what is
defined as a hostname (IPv6 address or something else). The change
results in a url constructed as http?://IPv4_address:port/, or
http?://hostname:port/, or http?://[IPv6_address]:port/. There
should be no more malformed URLs like http://fd00::f00d:9191/
generated for IPv6 addresses.
It also includes a test in glance/tests/functional/test_images.py,
named TestImagesIPv6. Additional functions which work on IPv6 only
were added since the whole testing suite is hardcoded for IPv4.
Change-Id: I66d6f2c57d1ccd086f941fc9e3764b4cc321241f
Closes-Bug: #1599123
There are several places in the source code where
HTTP response codes are used as numeric values.
These values are used from six.moves and the
numeric values are replaced by constants.
All of the used status codes were replaced with symbolic constants
from six.moves.http_client.
More about six.moves.http_client can be found at [2],
under the table "Supported renames:".
Also, this change improves code readibility.
This patchset does not extract numeric values from
the code itself, but it can be found at [1].
[1]: Ib9e26dcea927e96e65c626c18421621d3a29a64d
[2]: https://pythonhosted.org/six/#module-six.moves
Change-Id: Idfc7b043552f428f01ac3e47b270ee0639a8f5bc
Closes-Bug: #1520159
This change introduces glance.common.timeutils that provides the
timeutils previously consumed from oslo_utils.
Oslo is deprecating some timeutils functionality which of Glance
depends on. Suggested replacement (isoformat) would break glance APIs
so it's cleaner to carry this functionality in Glance rather than
re-invent the wheel.
Co-Authored-By: Erno Kuvaja <jokke@usr.fi>
Co-Authored-By: Sabari Kumar Murugesan <smurugesan@vmware.com>
Change-Id: I91e1cc9a273249fd88749cecf21200f3f5e2bab1
* urlsafe_b64decode(): on Python 3, decode from ASCII to get Unicode
string.
* urlsafe_decrypt(): encode Unicode to UTF-8.
* in some loops, copy dictionary keys or items because the dictionary
is modified in the loop body.
* test_api: use byte string for image content, not Unicode
* test_registry_api: HTTP body type is bytes, use byte strings and
jsonutils.dump_as_bytes(), instead of native strings and
jsonutils.dumps()
* tox.ini: add glance.tests.unit.v1.test_registry_api to Python 3.4
Change-Id: Ib9d18dce6e5728b9adf094b5aae64a86a3fea71a
When unit tests are launched some of these tests are attempting
to access http://example.com. This is not convenient in case
if somebody doesn't have an access to internet.
In order to fix this
- some tests in test_script_utils were deleted
cause they tested urllib instead of glance
- Some responses were replaced with FakeHTTPResponses
- Some requests were redirected to local http server
Change-Id: I66c3d67ae4bff474a4ef5a2d5469cfe04f36cdf9
Closes-bug: #1508868
* rpc: allow also exceptions from the builtins module. On Python 3,
builtin exceptions are part of the builtins module. The exceptions
module was removed in Python 3.
* Fix usage of reraise(): translate_exception() returns an instance
which is the second parameter of the reraise() function, not the
first parameter. The first parameter is the exception type.
* test_rpc: add json_dump_as_bytes() helper to serialize JSON as
bytes.
* JSONRequestDeserializer: don't compare None to int, it raises a
TypeError on Python 3.
* Fix Unicode versus bytes issues: HTTP body type is bytes. Encode
JSON to UTF-8 for example. Replace StringIO with BytesIO.
* Replace filter() with a list-comprehension to get a list on
Python 3.
* test_client, test_image_cache_client: use bytes for HTTP body.
* tox.ini: add glance.tests.unit.common.test_rpc to Python 3.
Change-Id: I9960d6a810375474f89c1788b7016a8fbb0770e0
* Use bytes for image content
* On Python 3, set_xattr() encodes the text value to UTF-8
* Open files in binary mode, not in text mode
* get_caching_iter(): pass a list as the image iterator, not a string.
On Python 3, list(b'abc') returns [97, 98, 99], whereas Python 2 returns
['a', 'b', 'c'].
* tox.ini: add glance.tests.unit.test_image_cache to Python 3.4
Change-Id: I638525d19c42990852cf45dd416318d9a847c303
Update the test configuration management code to use a ConfigFixture so
that configuration changes are rolled back at the end of each test.
Also fix the initialization order in TestSwiftParams to ensure that the
base class is configured before local customizations are applied.
Change-Id: I27620e3d4caf95e2c75fbf2cf2716ae1337030c7
os.dup() now creates a non-inheritable file descriptor since Python 3.4
because of the PEP 446 "Make newly created file descriptors
non-inheritable":
https://www.python.org/dev/peps/pep-0446/
Server.start() uses os.dup() to pass a socket to a child process, so the
file descriptor must be inheritable.
Add pass_fds parameter to fork_exec() of glance.tests.utils: optional
sequence of file descriptors, similar to the pass_fds parameter of
subprocess.Popen. On Python 3.4 and newer, fork_exec() calls
os.set_inheritable() on these file descriptors to make thee inheritable.
Use pass_fds in Server.start().
Change-Id: I7e5a24453cf837391e1fad73b7bb0c0fbd8963e2
* Use six.moves to import BaseHTTPServer, configparser, http_client and
urllib modules.
* Replace UserDict.UserDict with six.moves.UserDict.
The patch was generated by the six_moves and urllib operations of the
sixer tool:
https://pypi.python.org/pypi/sixer
Manual changes:
* Replace UserDict.UserDict with six.moves.UserDict
* Fix indentation
Change-Id: I3130bc82eea28935a310f5e9235f785ccd366654
When running tests with testr oslo.log's config options are always
registered even if you're only running one test. When you wish to debug
into your tests and use python -m testtools.run these options will not
be registered.
Closes-bug: 1433785
Change-Id: Ie3426cee8cd3f6f1222529160d39b281af5ce317
The Oslo libraries have moved all of their code out of the 'oslo'
namespace package into per-library packages. The namespace package was
retained during kilo for backwards compatibility, but will be removed by
the liberty-2 milestone. This change removes the use of the namespace
package, replacing it with the new package names.
The patches in the libraries will be put on hold until application
patches have landed, or L2, whichever comes first. At that point, new
versions of the libraries without namespace packages will be released as
a major version update.
Please merge this patch, or an equivalent, before L2 to avoid problems
with those library releases.
Blueprint: remove-namespace-packages
https://blueprints.launchpad.net/oslo-incubator/+spec/remove-namespace-packages
Change-Id: Ifa8baab33cdb3e606cf175a8c29c3a4ef6c44480
This change:
- Adds oslo.policy to the requirements list
- Accounts for changes in Enforcer initialization
- Accounts for changes to config options
- Removes incubated version of oslo.policy
- Updates the in-tree etc/config files
UpgradeImpact
Partially Implements Blueprint: graduate-policy
Change-Id: I5acb1e0f809098991f05ca3b6d78d4d88d98f2db
Brian Rosmaita