This commit disables writing in binary mode for the execution plan.
The contents being written comes direct from json.dumps, which is
always returned as a string.
Change-Id: I60cab27f29bf1b538dcf348b100d200048d9fd25
The use of the anyjson module causes issues with Python 3.10 with
the following error:
`error in anyjson setup command: use_2to3 is invalid.`
The module itself hasn't been updated since 2012 and the source [0]
now errors with a 404, and there doesn't seem to be any real benefit
over the standard python json module.
[0] https://bitbucket.org/runeh/anyjson/
Change-Id: I00ae5e9545136c1b930f8cf4036b902a326e1c54
This is to address another usage of base64 that was missed from
commit 794c47f3b8823e97a0f019064e4b74a5ee204d04
Change-Id: I9bbff8084d82928deeafa6a55c487ed0f665b470
Python 3 now handles str and bytes differently to Python 2, so when
using b64encode/decode, the input and output must specifically by
bytes.
This commit adds some base64 helpers to coerce the input to become
bytes if the input is a str and decodes the output back to str after
the encode or decode.
It also adds a test case to replicate the issue and cover the new
functions.
Change-Id: I8f64da03c7cfc76e620cb3b98201f6287b752bb3
The mock third party library was needed for mock support in py2
runtimes. Since we now only support py36 and later, we can use the
standard lib unittest.mock module instead.
Change-Id: I9d33976afc6b4d475f11f3f6b751ae7093cf2a04
Monkey patch the original current_thread to use the up-to-date _active
global variable. This solution is based on that documented at:
https://github.com/eventlet/eventlet/issues/592
Change-Id: I0e8f80bb2e7acfb00e5ff47e0aa983afb7ee6e38
Closes-Bug: #1863021
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib.
Change-Id: I24daffca6667325bd7710e02e0b6a057935e9ac9
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
The repo is Python 3 now, so update hacking to version 2.0 which
supports Python 3.
Fix problems found.
Update local hacking checks for new flake8.
Change-Id: Ifa4b61c66561c114f6e35355de51f0e2d525b5d5
Make sure we are passing a string when we are running
the test_script.runner_downloadable otherwise it will fail
when using python3.6 complaining about the following:
TypeError: expected str, bytes or os.PathLike object, not MagicMock
Change-Id: Id2e4f3d6389e1855051f791d95103e7c5d449f49
Signed-off-by: Chuck Short <chucks@redhat.com>
Review https://review.openstack.org/#/c/530865 renamed
storage_opt to opts but missed to rename it in
muranoagent/opts.py. This patch fixes it.
Change-Id: I41f06279147f0d2aafe80efeb1c0e558f2698d5e
Murano Agent uses default folder permissions for the
execution plans and scripts. If the default is too permissive
(which is unusual), other users on that machine can trick
the agent to execute malicious execution plans by putting files
into queue folder and use it to get the root privileges.
In most common sense users won't have write permissions to murano-agent
folders. However, they can hijack execution plans and other data
that might contain sensitive information.
This commit sets 0700 mode to the agent runtime folders so that they
can be accessed only by the user that runs the agent (+ the root,
if it's someone else).
Change-Id: I27f0495a509c4d1435d630e2bc5bfdf3549486d5
Use oslo_utils.strutils to mask any potentially sensitive
information in the execution plan before writing to the log.
Change-Id: I9008dcd68da4ba14bbf1360a672e1a91ae0a8e91
Closes-Bug: #1706059
DeprecationWarning: Using the 'enforce_type' argument is deprecated in
version '4.0' and will be removed in version '5.0': The argument
enforce_type has changed its default value to True and then will be
removed completely."
Change-Id: I3e29f9c443f76fa3acaf0267a0c42618f89961d3
See: https://review.openstack.org/#/c/328692/
Openstack common has a wrapper for generating uuids. We should
use that function to generate uuids for consistency.
Change-Id: I5d0e5523595088d934ee233c48e403f23326cee9
Closes-Bug: #1082248
Some configuration options were accepting both IP addresses
and hostnames. Since there was no specific OSLO opt type to
support this, we were using ``StrOpt``. The change [1] that
added support for ``HostAddressOpt`` type was merged in Ocata
and became available for use with oslo version 3.22.
This patch changes the opt type of configuration options to use
this more relevant opt type - HostAddressOpt.
[1] I77bdb64b7e6e56ce761d76696bc4448a9bd325eb
Change-Id: I0b7b8aebc9cc70f5ea256791cfb2c61ec4357bfd
As this has changed in murano in [0],
this patch change murano-agent.
[0]: I9f41f334e0f32ee31c40c27c977ed801a68830e7
Closes-bug: #1658044
Change-Id: Idc80808dcf54eb2d457cdcf3d42e23236e07a64f
Before this patch tests used to compare json.dumps representation of
dicts. This can lead to false negatives.
This patch fixes to test to compare objects instead.
Closes-Bug: #1610181
Co-Authored-By: Kirill Zaitsev <k.zaitsev@me.com>
Change-Id: I69af1c017362f95b92a81af3212f6520e7b4ceba
The "exec" statement was removed in Python 3. Replace it with
six.exec_() to add compatibility with Python 3.
Blueprint: murano-python-3-support
Change-Id: Idd1c6a6347e22f6a9c7887e7a327d29bb64e46d5
Use absolute import for muranoagent.common.messaging.subscription.
Blueprint: murano-python-3-support
Change-Id: I2b70aef8fb818a841a5034154100f200513dddf7
Run the sixer program to catch most common Python 3 issues and make
the code compatible with Python 2 and Python 3.
* Replace it.next() with next(it)
* Replace dict.keys()[0] with list(dict.keys())[0], since
dict.keys() now returns an dictview on Python 3. Same change
for dict.values()[0].
* Replace urlparse with six.moves.urllib.parse
* Replace __builtin__ with six.moves.builtins
Blueprint: murano-python-3-support
Change-Id: I9b95c3eee14c2de0010e9a3bbdffdda76496a03c
Sometimes network connection between agent and RabbitMQ
becomes corrupted without agent being aware. Usually
it happen due to some networking reconfiguration. For example
changes to iptables. Because consumer connection only reads from
the socket it cannot detect broken connection and will wait
for the message forever.
This commit enables periodic heartbeats that agent sends to the
server. If RabbitMQ doesn't receive heartbeats for some time it
disconnects the client. On the other hand because heartbeat
is a write operation it lets the client to detect broken connection
as well.
Change-Id: Id326f8d3b6dce04d83472837e11342efa8eb32fe
Closes-Bug: #1583007
Partially-Implements: blueprint rabbitmq-heartbeats
There is an issue https://bugs.python.org/issue1652
in Python 2.7 which was resolved in Py3.
By default Python sets SIGPIPE to SIG_IGN on startup, because
it prefers to check every write and raise an IOError exception rather
than taking SIGPIPE. This is all well and good for Python itself.
However, non-Python Unix subprocesses generally expect to have
SIGPIPE set to the default action, since that's what will happen if
they're started from a normal shell.
As a result scripts with piping that is terminated in reverse chain
order are never exit.
See more info here:
http://www.enricozini.org/blog/2009/debian/python-pipes/
Closes-Bug: #1524347
Change-Id: Ia015e7fd46912f6a0be97a6b9c297f8d14fd3f7c
Execution plan can modify iptables rules causing
RabbitMQ connection to break without agent
being notified. Thus it will just not get all subsequent
execution plans.
This commit reestablishes connection upon first
5-second timeout after last executed EP so it
it broke the connection it will be fixed in
5 seconds.
Change-Id: I90c8e062be8528ffd81a295a209c3ec029fc2cd8
Closes-Bug: #1556895
Python’s default arguments are evaluated once when the
function is defined, not each time the function is called.
This means that if you use a mutable default argument
and mutate it, you will and have mutated that object for
all future calls to the function as well.
more details about this wrong usage here:
http://docs.python-guide.org/en/latest/writing/gotchas/#mutable-default-arguments
Change-Id: Ic8eb1328852eb8a3180a08466bf18b72fd1f7956
The Unicode type is 'unicode' in Python 2 and 'str' on Python 3.
This patch replaces unicode with six.text_type to make Murano
Agent more compatible with Python 3.
Blueprint murano-python-3-support
Change-Id: I03788fdfe69c16831406244fd5720a82720775cf
Add Berkshelf support in murano-agent. If Berkshelf support is enabled
(Execution plan with "Type": "Chef" and "useBerkshelf": True), then the
image is expected to contain Berkshelf.
Change-Id: I036b5b9b7e1202d26fa2fd16591b680755cbfbc1
Partial-Implements: blueprint support-chef-berkshelf
When cookbooks are transfered from murano to
murano-agent, they are stored in another folder
which chef/puppet expect.
Closes-Bug: #1508492
Change-Id: I1363e46d655c8a641a1fa404faadc5f652f9fce4
The clear() method is calling rmtree() to delete cache folder
but the call is being made twice. The second call is unnecessary.
Change-Id: I78b0ed2417b3775604816bf078abedce79406edd
Closes-Bug: #1513010
This patch adds a dynamic result routing for static Agent.
This feature is designed to support Murano Agent which is
installed on external out of OpenStack system.
It adds _ReplyTo entry in the message body as well as supports
message.reply_to property.
Implements blueprint static-agent
Co-Authored-by: Alexander Tivelkov <ativelkov@mirantis.com>
Change-Id: Ief4c25a7fb63fc092f68f08b9665bde03fbe7752
Murano agent can download files in the VM when it is executing the
execution plan. However, just git client is allowed for download
files. Svn client should be provided to add files in a svn repository
Closes Bug: #1486055
Change-Id: Ib1fd13fa86d012fb95d19aabda0dff7c28f99620
Use oslo.log instead of oslo-incubator.
Use oslo.log instead of system logging.
Add oslo.log to the requirements.txt.
Remove unused oslo-incubator modules.
Implements: blueprint port-oslo-incubator-to-oslo-log
Partial-Bug: 1481370
Change-Id: I7047497f2022b0c56be0f3811f1bb59ffe470877
Manually updating the requirements, since this project isn't in
the list getting requirements syncs from global requirements. Updating
hacking required some format changes to docstrings, and I included those
in this patch.
Change-Id: Ib33f6308ed121fa2350eb32846e476f2a9dd28f4