Remove six.PY2 and six.PY3.
Subsequent patches will replace other six usages.
Change-Id: Iccce0ab50eee515e533ab36c8e7adc10cb3f7019
Implements: blueprint six-removal
Signed-off-by: Takashi Natsume <takanattie@gmail.com>
As noted in a TODO by mriedem, we were making two trips to the DB when
retrieving 'Service' objects in order to build a list of availability
zones. Optimize things so that this is no longer the case.
Change-Id: Id0d8c52cba8b548a0e6436d94765b84f040efee5
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Since there were only three non-test locations that
called get_availability_zones, and they can all pass
a HostAPI instance, the hostapi kwarg is made a required
argument to the method. This also means the compute.API
class can create a HostAPI instance to use and provide
its own copy of the RPC ComputeAPI and service group API.
While in here the get_availability_zone docstring is
cleaned up a bit and the context param is documented.
Change-Id: I727a5001f9f75f4bd8c7a7b599d647cf987d1659
Assume number of services can be large (10000 as in the bug description),
this patch removes second service_get_all call.
zone_hosts changed from dict of lists to dict of sets.
The HostAPI instance from the API controller is also passed to the
get_availability_zones method so it does not have to recreate it
per call (this is both for a slight performance gain but mostly also
for test sanity).
On devstack with 10000 services patch decreased response time twice.
openstack availability zone list --long --timing
...
Before:
+-------------------------------------------------------------------------------------------+-----------+
| URL | Seconds |
+-------------------------------------------------------------------------------------------+-----------+
| GET http://192.168.0.45/identity | 0.006816 |
| POST http://192.168.0.45/identity/v3/auth/tokens | 0.456708 |
| POST http://192.168.0.45/identity/v3/auth/tokens | 0.087485 |
| GET http://172.18.237.203/compute/v2.1/os-availability-zone/detail | 95.667192 |
| GET http://172.18.237.203/volume/v2/e2671d37ee2c4374bd1533645261f1d4/os-availability-zone | 0.036528 |
| Total | 96.254729 |
+-------------------------------------------------------------------------------------------+-----------+
After:
+-------------------------------------------------------------------------------------------+-----------+
| URL | Seconds |
+-------------------------------------------------------------------------------------------+-----------+
| GET http://192.168.0.45/identity | 0.020215 |
| POST http://192.168.0.45/identity/v3/auth/tokens | 0.102987 |
| POST http://192.168.0.45/identity/v3/auth/tokens | 0.111899 |
| GET http://172.18.237.203/compute/v2.1/os-availability-zone/detail | 39.346657 |
| GET http://172.18.237.203/volume/v2/e2671d37ee2c4374bd1533645261f1d4/os-availability-zone | 0.026403 |
| Total | 39.608161 |
+-------------------------------------------------------------------------------------------+-----------+
The test_availability_zone_detail unit test is updated to assert that
services are only retrieved twice (once for enabled, once for disabled).
While in there, the expected response dict is formatted for readability
and a duplicate zone/host is added to make sure duplicates are handled
for available services. To ensure service_get_all is called only twice,
the low-level DB API service_get_all stub is replaced with a mock and
the mock is changed to be on the HostAPI.service_get_all method which
is (1) what the API controller code is actually using and (2) allows the
test to only mock the instance of the HostAPI being tested - trying to
mock the DB API service_get_all method causes intermittent failures
in unrelated tests because of the global nature of that mock.
There is another opportunity for optimizing get_availability_zones which
is marked with a TODO but left for a separate patch.
Co-Authored-By: Matt Riedemann <mriedem.os@gmail.com>
Partial-Bug: #1801897
Change-Id: Ib9a9a9a79499272d740a64cc0b909f0299a237d1
When [cinder]cross_az_attach=False (not the default) and doing
boot from volume, the API code validates the BDM by seeing if
the instance and the volume are in the same availability zone.
To get the AZ for the instance, the code is first trying to get
the instance.host value.
In Ocata we stopped creating the instance in the API and moved that
to conductor for cells v2. So the Instance object in this case now
is created in the _provision_instances method and stored in the
BuildRequest object. Since there is no host to set on the instance
yet and the Instance object wasn't populated from DB values, which
before would set the host field on the instance object to None by
default, trying to get instance.host will lazy-load the field and
it blows up with ObjectActionError.
The correct thing to do here is check if the host attribute is set
on the Instance object. There is clear intent to assume host is
not set in the instance since it was using instance.get('host'),
probably from way back in the days when the instance in this case
was a dict. So it's expecting to handle None, but we need to
modernize how that is checked.
Change-Id: I0dccb6a416dfe0eae4f7c52dfc28786a449b17bd
Closes-Bug: #1693600
This removes the direct-to-object listing of Service objects by the
availability zone routines. Instead, it implements a cell-scanning
approach in HostAPI and uses that for consistent behavior.
The modified behavior of service_get_all() in HostAPI was originally
in another patch, but is moved here because of dependency sequencing.
The HostAPI patch later in this series uses this as well.
Related to blueprint cells-aware-api
Change-Id: I65c2b436bef8837c0e10a5e502e9cd598d9aa0c3
* convert_password(): on Python 3, decode password from UTF-8 if it
is a byte string
* convert_password(): don't modify password in the loop body, use an
index increased at each iteration. This is a minor cleanup,
unrelated to Python 3.
* ec2_md_print() now returns byte strings unchanged. On Python 3,
str(bytes) uses "b'...'" format which is not the expected result
(or it raises a BytesWarinng exception when using python3 -bb).
* Replace the base64 module with oslo_serialization.base64 to control
the output type (bytes or text) on Python 2 and Python 3.
* _make_cache_key(): on Python 3, don't encode the host to UTF-8.
Python 3 requires text.
* MetadataRequestHandler: on Python 3, encode HTTP body to UTF-8 if
the body type is Unicode.
* test_metadata:
- encode hmac.new() parameters using encodeutils.to_utf8()
- use response.text rather than response.body to compare with
Unicode string
- use a byte strings for the instance identifier
- fix some Unicode versus bytes issues
* tests-py3.txt: run test_metadata on Python 3
Co-Authored-By: Davanum Srinivas <davanum@gmail.com>
Co-Authored-By: ChangBo Guo(gcb) <eric.guo@easystack.cn>
Partially-Implements: blueprint goal-python35
Change-Id: Ifb43ce164d9fd80f9f85c95c762d7b010e84dfeb
Common memorycache was replaced by analogous tool
from oslo.cache lib. In-memory cache was replaced
by oslo.cache.dict backend. Memcached was replaced
by dogpile.cache.memcached backend.
Implements blueprint oslo-for-mitaka
Closes-Bug: #1483322
Co-Authored-By: Sergey Nikitin <snikitin@mirantis.com>
Co-Authored-By: Pavel Kholkin <pkholkin@mirantis.com>
Change-Id: I371f7a68e6a6c1c4cd101f61b9ad96c15187a80e
This change moves all of the configuration options previously defined
in nova/availability_zones to the new centralized nova/conf directory.
Blueprint centralize-config-options
Change-Id: Ifbe3cc2bf1ee1b07f2a48d41060d5e43342da655
When a new instance is created, the host value is set to None. As a corrolar,
its AZ field is set to the default_availability_zone value.
That's only when the host field is provided (ie. once the instance is started) that
the AZ information is also updated. Unfornately, as the AZ module is using a cache
system, it leaves the old AZ unchanged which can make the instance AZ data wrong.
In order to fix that corner case, we need to compare both cache and instance values
and if different, invalidate the cache key and leave it updated correctly by calling
the AZ related information of the aggregate the host belongs to.
Change-Id: I1b47ae07cf8a4245df00b9dc25e6000428bdc4f2
Closes-Bug: #1390033
This makes getting the availability zone of an instance use the same
code by placing it inside the helper get_instance_availability_zone().
Related-Bug: #1390033
Change-Id: I69b98eacbc8dc91e65611d6bf07272b517fe350d
All tests in nova/tests/unit/objects/test*.py now run with
python3.4 tox target.
* Fix imports in limits.py and urlmap.py
* Add a list() as keys()/values() return iterator in
availability_zones.py, flavors.py and instance_numa_topology.py
* contextlib.nested is not present in python3, so whip up an
alternative using ExitStack(). Directly using ExitStack() wont
work for us.
* Add a few assertJsonEqual in the test cases
* Ensure fingerprinting generates the same exact value in both
python27 and python34
Blueprint nova-python3
Change-Id: I848c48475189c4b4ad8151e14509020ae7d110a4
This change makes the modified code compatible with Python 2 and Python 3.
Add __iter__() and keys() methods to NovaObjectDictCompat. These methods
are added temporarly, the change
I4c52d44c7ba49f98b1bbd123209fce7b70ade98d added these methods to the
base class VersionedObjectDictCompat of oslo.versionedobjects.
sqlalchemy code is unchanged. sqlalchemy objects cannot be converted to
a dictionary using dict(obj) directly yet. It will be possible with the
change I702be362a58155a28482e733e60539d36c039509.
Don't change get_instance_metadata() in nova/compute/api.py, it's
fixed by the change Ifd455e70002eb9636b87f83788384127ba6edeeb.
Blueprint nova-python3
Change-Id: I46e5fcaaef75178fb4db44368e34da8777b4c4b1
set_availability_zones in availability_zone.py is used
at several place, actually, the param of object allows
it to be set inside object function. So switch to
object with param input to clean unnessary code.
Change-Id: I3fd8a661677d6bfee513512d2315e8daeb57d422
The oslo team is recommending everyone to switch to the
non-namespaced versions of libraries. Updating the hacking
rule to include a check to prevent oslo.* import from
creeping back in.
This commit includes:
- using oslo_utils instead of oslo.utils
- using oslo_serialization instead of oslo.serialization
- using oslo_db instead of oslo.db
- using oslo_i18n instead of oslo.i18n
- using oslo_middleware instead of oslo.middleware
- using oslo_config instead of oslo.config
- using oslo_messaging instead of "from oslo import messaging"
- using oslo_vmware instead of oslo.vmware
Change-Id: I3e2eb147b321ce3e928817b62abcb7d023c5f13f
None of the callers of this method use the conductor parameter anymore.
Just remove it and clean up the code that made use of it.
Change-Id: Iba685ea6eea71be17f19e391790eda12dafc7a28
This change converts availability_zones to use the Aggregate and
Service objects instead of direct database access.
Related to blueprint compute-manager-objects-juno
Change-Id: I2ec7373f063ed728dde2af3b7c8259f4391885bc
When updating aggregate metadata or when adding a host into aggregate,
a check is done to make sure that hosts can't be in multiple AZs.
In order to do the check, the routine was looping over each host to look
in the DB which AZs the host is part of.
As it's quite hungry in terms of DB accesses, here the proposal is to
extend the helper method for getting all AZs by also having their hosts,
and so only do memory iterations.
Partial-Bug: #1277230
Change-Id: I7c5bc980acb17f0a0e1f8c1403dac66c305866fb
These were initially noted during import into the manuals at
https://review.openstack.org/67502
Fixes include rework of text, missing or extra spaces, wrong
capitalization and no sentence style capitalization.
A hacking check has been added that tests that all config options
start with a capital letter.
Change-Id: I0a1c333bbb3a812932d3972a7403f94da0bff341
Co-Authored-By: Daniel Berrange <berrange@redhat.com>
Co-Authored-By: Diane Fleming <diane.fleming@rackspace.com>
when add/remove a host to one aggregate or update aggregate
metadata incluing availability_zone, "OS-EXT-AZ:availability_zone"
property of some instances in the host can not show correctly.
The cause is that when getting availability_zone of one instance,
it will try to get the value from cache first, but unfortunately
the cache does not update or reset in time, and it will keep one
hour if we do not change it.
This patch will add update or reset after adding/removing a host
to one aggregate or updating aggregate metadata including
availability_zone.
Change-Id: I5dd07f876471b5faf8fb1016e25a861124b7cb6f
Closes-bug: #1240374
The problem is that when the availability_zones for one host
aggregation was updated, the VM attribute of availability_zones
which belong to the host in the availability_zones was
not updated on time.
If updated values include availability_zones, then the cache
which stored availability_zones and host need to be reset.
Fix bug 1203250
Change-Id: Ib3f08fbe3c74d5aa101da78faed43b0bc03cb988
The memcache library we are using assumes all strings passed to
it are plain ascii, so we need to convert unicode to string.
Fix bug #1212164
Change-Id: Ibf36e6b096893cb9f056371bb43cbb441cae6e32
When using nova boot command with --availability_zone parameter and only
input the zone without host and node, now will check if the zone is
available, if not, then return an error message directly rather than
create an instance with an Error state
Add a get_only_available option to function get_availability_zones
To disable available zone, one should have to disable both the scheduler
filter and the API extension, if the API extension is not disabled, this
patch will still do the validation of available zone
DocImpact
Change-Id: Ibff117d48b60ac4f4ae33e3b3d2840d7313ce0d4
When attempting to find the availability zone of an instance
do so by first trying to locate the availability zone of the
host the instance is active on. If that fails then attempt
to use the value that is in the instances 'availability_zone'
key instead.
For the caching used in the availability_zones module there
was a new method added to get and reset the cache that is
useful for unit tests, since during each run of a test you
do not want to be affected by something in cache from a previous
test.
Fixes bug 1172246
Change-Id: I6f54a44cc87434120656ccc789cebcc08d434418
An instance's availability zone logic may used by many modules,
so move it from extend API into helper module `nova.availability_zones`.
Also add some tests for the new method.
part of bp: different-availability-zone-filter
Change-Id: I5916a0b09d5dddec338e8e36503953720dfd6bcd
Remove a lot of getLogger lines and imports of logging in modules
which never use that functionality.
Change-Id: Icdaee2c540980412b000d02ebf1ec568dcf5b38a
Trying to list availability zones when there is a disabled
services fails with a 500 error
Fixes bug #1160841
Change-Id: Ibdb639f0d6a1a69a6a6331ee053c75449a16fc8f
The cfg API is now available via the oslo-config library, so switch to
it and remove the copied-and-pasted version.
Add the 2013.1b4 tarball to tools/pip-requires - this will be changed
to 'oslo-config>=2013.1' when oslo-config is published to pypi. This
will happen in time for grizzly final.
Add dependency_links to setup.py so that oslo-config can be installed
from the tarball URL specified in pip-requires.
Remove the 'deps = pep8==1.3.3' from tox.ini as it means all the other
deps get installed with easy_install which can't install oslo-config
from the URL.
Make tools/hacking.py include oslo in IMPORT_EXCEPTIONS like it already
does for paste. It turns out imp.find_module() doesn't correct handle
namespace packages.
Retain dummy cfg.py file until keystoneclient middleware has been
updated (I18c450174277c8e2d15ed93879da6cd92074c27a).
Change-Id: I4815aeb8a9341a31a250e920157f15ee15cfc5bc
This patch makes the metadata service use conductor instead of hitting the
database directly.
It adds a get_ec2_ids() method to conductor, which encapsulates a bunch of
very small id translation database queries. It also does the AZ lookup
via conductor.
Fixes bug 1120402
Change-Id: Iceef753b1451ff53e29dfbf97a696b952b47d5a3
Returns a summary or detailed list of availability zone to client.
To keep consistent result with EC2 API and de-duplicate, I have
refactored EC2 cloud controllor code.
Implement one workitem for bp:show-availability-zone
Change-Id: I6c9c673c757c6c319a1cdd94d7420b8983f92bbb
Signed-off-by: zhiyanliu <zhiyanl@cn.ibm.com>
As in nova/availability_zones.py, the method get_host_availability_zone()
expects "context" as a module rather than a RequestContext object.
But for our preference, the argument "context" is better to be a
RequestContext object. So fix this by passing an context object into the
method.
* Fix the usage of argument "context"
* Add unittest for the module "availability_zones"
Fix bug lp:#1100598
Change-Id: I5886ba26131261d97be65f18e9492c40401c5a7f
fix the N402 errors that have slipped in in the last 48 hrs since
starting this patch series.
fix an N401 error that our scanner current can't find because it
only looks for doc strings on classes and defs.
this is the xeno's paradox of patch series, but we're getting close.
Change-Id: I4a763bb4c812335d853eae05c72464f18ab93297
This is the final step in enabling availability_zones using aggregate
metadata. Previously all services had an availability_zone, but the
availability_zone is only used for nova-compute. Services such as
nova-scheduler, nova-network, nova-conductor have always spanned all
availability_zones.
After this change only compute nodes (nova-compute), will have an
availability_zone. In order to preserve current APIs, when running:
* nova host-list (os-hosts)
* euca-describe-availability-zones verbose
* nova-manage service list
Internal services will appear in there own internal availability_zone
(CONF.internal_service_availability_zone)
Internal zone is hidden in euca-describe-availability_zones
(non-verbose)
CONF.node_availability_zone has been renamed to
CONF.default_availability_zone and is only used by the nova-api and
nova-scheduler. CONF.node_availability_zone still works but is
deprecated
DocImpact
Completes blueprint aggregate-based-availability-zones
Change-Id: Ib772df5f9ac2865f20df479f8ddce575a9ce3aff
Takashi Natsume