If the Nova metadata service is unavailable, the requests.request()
function may raise a ConnectionError. This results in the upper code
returning a 500 HTTP status code to the user along with a traceback.
Let's handle this scenario and instead return a 503 HTTP status code
(service unavailable).
If the Nova service is down and is behind another proxy (such as
Nginx), then instead of a ConnectionError, the request may result in
receiving a 502 or 503 HTTP status code. Let's also consider this
situation and add support for an additional 504 code.
Closes-Bug: #2059032
Change-Id: I16be18c46a6796224b0793dc385b0ddec01739c4
This patch is the initial implementation on the suggestion
from this patch[1].
The DHCP agent can query the existing `get_ports` RPC method because
this method is already exposed in the MetadataRpcCallback(server side)
which runs under the same topic(PLUGIN) and namespace(None). The benefit
here is that there is no change needed to the API, however it does
go against how we historically setup the RPC layer between a server and client.
[1] https://review.opendev.org/c/openstack/neutron/+/903572/comments/3d4e0453_4b4d2ab6
Related-Bug: #1982569
Change-Id: Icd7c55d2a5103bdbd90907b1dbfb9ccfe34c020a
After updating pylint, it started emitting additional "W"
warnings in some cases. Fix the missing-timeout ones by
adding a timeout=60 value to the calls done by the
metadata agent code.
Trivialfix
Change-Id: Ica5c2a2d3941813b3518b83738ada96bdf7c3760
Running with a stricter .pylintrc generates a lot of
C0330 warnings (hanging/continued indentation). Fix
the ones in neutron/agent and neutron/privileged/agent.
Trivialfix
Change-Id: Ib94257481e62b99b3d7648ae5137af5411b4867a
Move _sign_instance_id to common utils for distributed metadata.
Partially-Implements: blueprint distributed-metadata-datapath
Change-Id: I0ef9330232e2ed5dbda6e45917c291c7385d1e0d
Both drivers have different approaches when it comes to the metatada
agent, for one the metadata agent for ML2/OVN runs on the compute nodes
(it's distributed) instead of the controller nodes.
The previous default of "<# of CPUs> / 2" did not make sense for ML2/OVN
and if left unchanged could result in scalation problems because of the
number of connections to the OVSDB Southbound database, as seeing in
this email thread for example [0].
This patch puts a placeholder value (None) on the default field of
the "metadata_workers" config by not setting it immediately and then
conditionally set the default value based on each driver:
* ML2/OVS defaults to <# CPUs> // 2, as before.
* ML2/OVN defaults to 2, as suggested in the bug description and also
what's default in TripleO for the OVN driver.
[0]
http://lists.openstack.org/pipermail/openstack-discuss/2020-September/016960.html
Change-Id: I60d5dfef38dc130b47668604c04299b9d23b59b6
Closes-Bug: #1893656
Signed-off-by: Lucas Alvares Gomes <lucasagomes@gmail.com>
In the spec we said:
"""
When the metadata proxy processes a request, it gathers the L2 addresses
of a VM, and the source interface, and passes it to the metadata service.
The Metadata service, instead of using the VM IP, uses the "VM MAC" and
"Gateway MAC" to identify the instance.
"""
But since we switched from the home-grown metadata-ns-proxy to haproxy
we no longer control some of the headers included, like X-Forwarded-For.
haproxy allows us to turn X-Forwarded-For on or off, but it cannot
give us an X-Forwarded-For-MAC header.
Instead it seems we have to rely on the source address being the IPv6
link local address generated from the NIC's MAC address as specified
in RFC 4291:
https://tools.ietf.org/html/rfc4291#section-2.5.6https://tools.ietf.org/html/rfc4291#appendix-A
Note that means you cannot use IPv6 Privacy Extensions:
https://tools.ietf.org/html/rfc4941
Change-Id: Ife592fcfc69e26f61ec1f45c06821cb025cc7cf2
Closes-Bug: #1460177
We push a v6 host route to make the guest send its metadata requests
in the direction of our router. We redirect it to haproxy which
mangles the headers and sends the request along to metadata-agent.
Apparently the supported list of dhcp options for dhcpv6 is quite
short in dnsmasq (cf. dnsmasq --help dhcp6) - not including anything
like classless-static-route for dhcpv4. So we must rely solely on
radvd to push host routes to the guest.
Metadata access over IPv6 is supposed to work both on dual-stack and
v6-only networks.
The following v6 subnet modes are supposed to work:
--ipv6-ra-mode slaac --ipv6-address-mode slaac
--ipv6-ra-mode dhcpv6-stateless --ipv6-address-mode dhcpv6-stateless
--ipv6-ra-mode dhcpv6-stateful --ipv6-address-mode dhcpv6-stateful
Change-Id: I28f2914b1b67659af2db7240eae730ac43daccd2
Partial-Bug: #1460177
Now that we use setproctitle for neutron-server workers (and
neutron-keepalived-state-change), this has the side effect of changing
the process name for agents, impacting some monitoring systems. More
details in launchpad bug.
This patch fixes it by setting the name with setproctitle to:
agent name (original process name).
Also use the newly introduced name constants to replace existing
hardcoded uses.
Change-Id: I74c3a4d3e9f833752571a75f196560cd45529385
Closes-Bug: #1881297
With python 3.x, six.text_type and six.string_type
are just str.
Also removed a six.integer_type since it was the only
one left in a file.
Another step in removing all of six usage from neutron.
Change-Id: I5208dc41bff1983ecd323286f427296b722da62a
The patch fixes issue when the same port is requested for multiple
instances and second one can't get metadata due to cached instance_id.
Closes-Bug: 1868867
Change-Id: If6a5866e4406c9c6c30e989c79ffb4ee1a88cecf
If a user specifies a header in their request for metadata,
it could override what the proxy would have inserted on their
behalf. Make sure to remove any headers we don't want, and
override something that might be present in the request.
If the agent somehow gets a request with both headers it will
silently drop it.
Change-Id: Id6c103b7bcebe441c27c6049d349d84ba7fd15a6
Closes-bug: #1865036
All of the externally consumed variables from neutron.common.constants
now live in neutron-lib. This patch removes neutron.common.constants
and switches all uses over to lib.
NeutronLibImpact
Depends-On: https://review.openstack.org/#/c/647836/
Change-Id: I3c2f28ecd18996a1cee1ae3af399166defe9da87
Reduces E128 warnings by ~260 to just ~900,
no way we're getting rid of all of them at once (or ever).
Files under neutron/tests still have a ton of E128 warnings.
Change-Id: I9137150ccf129bf443e33428267cd4bc9c323b54
Co-Authored-By: Akihiro Motoki <amotoki@gmail.com>
The neutron.common.rpc module has been in neutron-lib for awhile now and
neutron is shimmed to use neutron-lib already.
This patch removes neutron.common.rpc and switches the code over to use
neutron-lib's implementation where needed.
NeutronLibImpact
Change-Id: I733f07a8c4a2af071b3467bd710290eee11a4f4c
It fixes raising exception for response with not recognized
status code.
Co-Authored-By: Brian Haley <haleyb.dev@gmail.com>
Change-Id: I174ff62cb6599e4c7bdc86cb2d0786f9f2499b00
Related-Bug: 1790598
Current logic didn't check if the nova_metadata_host is an IPv6 address
causing the proxy request to fail with an exception because the url is
not valid.
This patchs check if the nova_metadata_host is an IPv6 address and
create a valid url enclosing the IPv6 address with brackets
Closes-Bug: #1796593
Change-Id: Ibfebffcec2c8860237a1f151084de978a7863bd8
Signed-off-by: aojeagarcia <aojeagarcia@suse.com>
httplib2 makes use of the ssl module provided by Python; under Python 2,
the ssl module does not support IP addresses as subject alternate names
(SAN's) which although an optional part of the associated RFC, is awkward
to work with in environments where certificate management approaches
rely on use of IP addresses in SAN's.
The requests module is more than happy to deal with this scenario; switch
to requests in preference of httplib2 for metadata proxy calls.
httplib2 is retained as its used elsewhere in the codebase.
Change-Id: Ife4adf09ddbf7116da2f8596c80aed53fb6790df
If the metadata agent fails to send its report_state
to the server, it logs an exception:
Failed reporting state!: MessagingTimeout: Timed out...
If it then tries a second time (on 30 second intervals)
it just goes on happily. It would be nice if it logged
that it had success on the subsequent attempt so someone
looking at the logs know it recovered.
Change-Id: I7a826811a889c618a39a8a91c8157e1805a327eb
Fix W503 (line break before binary operator) pep8 warnings
and no longer ignore new failures.
Trivialfix
Change-Id: I7539f3b7187f2ad40681781f74b6e05a01bac474
The neutron.common.topics module was rehomed into neutron-lib with
commit Ie88b84949cbd55a4e7ad06341aab77b286cdc485
This patch consumes it by removing the rehomed module from neutron
and using the module from neutron-lib instead.
NeutronLibImpact
Change-Id: Ia4a4604c259ce862597de80c6deeb3d408bf0e95
Current metadata agent doesn't output why it returns NotFound(404)
to a server instance and it is very hard to investigate the reason.
This patch adds some debug logs for doing that.
Change-Id: I0041a56edc50c99c0167fe1ef26161acf772ea7a
Changing rpc_api.rst file path from doc/source/devref/rpc_api.rst
to /doc/source/contributor/internals/rpc_api.rst. Because rpc_api.rst
file is located at this path
doc/source/contributor/internals/rpc_api.rst.
Closes-Bug #1722072
Change-Id: Ic243aab9e3428bfec69db61a94b4129cd768e233
Since Pike log messages should not be translated.
This patch removes calls to i18n _LC, _LI, _LE, _LW from
logging logic throughout the code. Translators definition
from neutron._i18n is removed as well.
This patch also removes log translation verification from
ignore directive in tox.ini.
Change-Id: If9aa76fcf121c0e61a7c08088006c5873faee56e
nova_metadata_ip option name is a bit confusing as it accepts an ip but
also a dns name (which is required when https protocol is used).
This change deprecates nova_metadata_ip option in favor of
nova_metadata_host option and updates option help to highlight that we
can use an ip or a dns name.
DocImpact
Closes-Bug: #1585699
Change-Id: Ia6c2471c7b4f3e924941222133edbb90151757a5
This will help root cause the Cirros image complaining
about a lack of keys when they are visibly present in the
Nova API logs.
Related-Bug: #1668958
Change-Id: I606c56f73be147e7f3281dfc73b874e404c39675
Refactoring neutron agent metadata config opts to be in
neutron/conf/agent/metadata so that all the configurations options
reside in a centralized location. This simplifies the process of looking
up the config opts and provides an easy way to import.
Change-Id: I8bae1facc58a4f9e21196f625478532403651545
Partial-Bug: #1563069
We start state reporting thread before forking child processes. This
leads to possibility of fork in the middle of reporting process. This
can cause child processes to start with corrupted global state. In [0]
it represented itself as metadata agent not being able to make RPC call
because of lock acquired in main process before fork and released after
fork. See bug description for details.
[0] https://review.openstack.org/312393
Partial-Bug: 1594439
Change-Id: Id5079a296bc402c6f5b6cdb2df72811eab5bc6ed
Oslo incubator is about to stop for cache module. We could use
oslo.cache instead. The legacy memory backend is replaced by
oslo_cache.dict.
Closes-Bug: #1517883
Change-Id: I108242ca9f27c9ec47959ce7615bc7d84cae014b
Reactor code:
* Reuse oslo_utils.encodeutils.to_utf8() instead of existing
isinstance(text, six.text_type) test
* Replace jsonutils.dumps(obj).encode('utf-8') with
jsonutils.dump_as_bytes(obj).
* Other minor bytes/Unicode changes
Change-Id: I03b8eff0fd70ab65ac66d6f3221e8ced0a56db17
Python 3 deprecated the logger.warn method, see:
https://docs.python.org/3/library/logging.html#logging.warning
so we prefer to use warning to avoid DeprecationWarning.
Closes-Bugs: #1529913
Change-Id: Icc01ce5fbd10880440cf75a2e0833394783464a0
Co-Authored-By: Gary Kotton <gkotton@vmware.com>
- This does NOT break other projects that rely on neutron.i18n,
as this change includes a debtcollector shim to maintain those
older entry points, until they can migrate.
- Also updates _i18n.py to the latest pattern defined by oslo_i18n
- Guidance and template are from the reference:
http://docs.openstack.org/developer/oslo.i18n/usage.html
Partially-Closes-Bug: #1519493
Change-Id: I1aa3a5fd837d9156da4643a367013c869ed8bf9d
The new RPC interface has proved itself for two cycles, I don't
recollect any serious issues with it, so let's just clean up the
obsolete neutronclient based fallback mechanism.
The metadata agent configuration documentation should be updated
to not require API configuration values for the agent to talk to
neutron-server.
DocImpact
Change-Id: I254c575c66214f50fb93a94c46c4c9caebfc2937
Closes-Bug: #1502947
This optimization is needed for big clusters with hundreds
of agents where the spike of activity may trigger a burst
of RPC requests that would prevent neutron-server from processing
agent heart beats in time, triggering resource rescheduling.
This will be further optimized by running dedicated RPC workers
for state reports processing.
Related-Bug: #1496410
Change-Id: Id86a1f962aaa4f64011d57ae55d240f890cca4f7
Allow operators to set the endpoint url for neutron in the config
overriding the url that comes from the keystone catalog.
Change-Id: I93f81ef1be2de1038d9a899b0c4becdb5a8e8775
DocImpact
Closes-Bug: #1466258
oslo.service has graduated, so neutron should consume it.
Closes-Bug: #1466851
Depends-On: Ie0fd63f969f954029c3c3cf31337fbe38f59331a
Depends-On: I2093b37d411df9a26958fa50ff523c258bbe06ec
Depends-On: I4823d344878fc97e66ddd8fdae25c13a34dede40
Change-Id: I0155b3d8b72f6d031bf6f855488f80acebfc25d4
When troubleshooting problems with cluster it would be
very convenient to have information about agent heartbeats
logged with some searchable identifier which could create
1-to-1 mapping between events in agent's logs and server's logs.
Currently agent's heartbeats are not logged at all on server side.
Since on a large cluster that could create too much logging
(even for troubleshooting cases), it might make sense to make
this configurable both on neutron-server side and on agent-side.
DocImpact
Change-Id: I0a127ef274a84bba5de47395d47b62f48bd4be16
Closes-Bug: #1452582
In Python 3, 'unicode' does not exist; 'six.text_type' should be used instead.
Change-Id: I71011b4beee9817a61278eb473804cfb798de74a
Blueprint: neutron-python3
Currently metadata proxy cannot run with nobody user/group as metadata
proxy requires to connect to metadata_proxy_socket when queried.
This change allows to run metadata proxy with nobody user/group by
allowing to choose the metadata_proxy_socket mode with the new option
metadata_proxy_socket_mode (4 choices) in order to adapt socket
permissions to metadata proxy user/group.
This change refactors also where options are defined to enable
metadata_proxy_user/group options in the metadata agent.
In practice:
* if metadata_proxy_user is agent effective user or root, then:
* metadata proxy is allowed to use rootwrap (unsecure)
* set metadata_proxy_socket_mode = user (0o644)
* else if metadata_proxy_group is agent effective group, then:
* metadata proxy is not allowed to use rootwrap (secure)
* set metadata_proxy_socket_mode = group (0o664)
* set metadata_proxy_log_watch = false
* else:
* metadata proxy has lowest permissions (securest) but metadata proxy
socket can be opened by everyone
* set metadata_proxy_socket_mode = all (0o666)
* set metadata_proxy_log_watch = false
An alternative is to set metadata_proxy_socket_mode = deduce, in such
case metadata agent uses previous rules to choose the correct mode.
DocImpact
Closes-Bug: #1427228
Change-Id: I235a0cc4f0cbd55ae4ec1570daf2ebbb6a72441d
As part of the all consuming report-ha-router-master, a new
per router neutron-keepalived-state-change daemon will alert
the L3 agent on every keepalived state change. Since it will
use the Unix domain socket helpers, and they're currently
located in metadata related places, this patch moves them
to a common location.
Also, the UnixDomainHTTPConnection connection string
may now be overridden.
Partially-Implements: blueprint report-ha-router-master
Change-Id: Ib2cde90059f4e089064b2def2838e9bcf9af30de
Anton Kurbatov