Opportunity to hide sensitive data from http action logs, such as:
* Request headers
* Request body
* Response body
Change-Id: I6d1b1844898343b8fa30f704761096e3d2936c4d
Implements: blueprint mistral-hide-sensitive-data-from-http-actions-logs
Signed-off-by: Oleg Ovcharuk <vgvoleg@gmail.com>
* This patch refactors Mistral with the action provider concept
that is responsible for delivering actions to the system. So
it takes all the burden of managing action definitions w/o
having to spread that across multiple subsystems like Engine
and API and w/o having to assume that action definitions are
always stored in DB.
* Added LegacyActionProvider that represents the old way of
delivering action definitions to the system. It pretty much just
analyses what entries are configured in the entry point
"mistral.actions" in setup.cfg and build a collection of
corresponding Python action classes in memory accessible by names.
* The module mistral/services/actions.py is now renamed to
adhoc_actions.py because it's effectively responsible only for
ad-hoc actions (those defined in YAML).
* Added the new entry point in setup.cfg "mistral.action.providers"
to register action provider classes
* Added the module mistral/services/actions.py that will be a facade
for action providers. Engine and other subsystems will need to
work with it.
* Other small code changes.
Depends-On: I13033253d5098655a001135c8702d1b1d13e76d4
Depends-On: Ic9108c9293731b3576081c75f2786e1156ba0ccd
Change-Id: I8e826657acb12bbd705668180f7a3305e1e597e2
Remove six.moves Replace the following items with Python 3 style code.
- six.moves.urllib
- six.moves.queue
- six.moves.range
- six.moves.http_client
Subsequent patches will replace other six usages.
Change-Id: I80c713546fcc97391c64e95ef708830632e1ef32
* When YAQL output data conversion is disabled there's still
an issue caused by presence of not JSON-compatible types within
a YAQL result. The internal Mistral code is already able to deal
with that (due to the previous changes) by checking that and
converting them to what's needed. However, JSON serialization
may still not work if it's done via the standard "json" library.
The library simply doesn't handle those non-standard types and
raises an exception. We have a sanitizing function that all YAQL
results go through, however, it doesn't make sense to do recursive
sanitizing for performance reasons. It does make sense to convert
data as late as possible to avoid redundant data manipulations. So
the sanitizing function handles only the root object in the object
graph. The solution for this problem is to use our own utility
function based on the "oslo_serialization.jsonutils" that is able
to deal with at least part of the mentioned types, specifically
FrozenDict and iterators. Generators are still a problem and this
new function takes care of that separately, assuming that any
generator is just a special iterator and hence represents a
collection, i.e. a list in JSON terms. It works for all the cases
we've encountered so far working with YAQL.
* Used the new function "utils.to_json_str()" everywhere for JSON
serialization, including the action "std.http".
* Added necessary unit tests.
Closes-Bug: #1869168
Depends-On: I1081a44a6f305eb1dfe68a5bad30110385130725
Change-Id: I9e73ea7cbba215c3e1d174b5189be27c640c4d42
To simplify work with jsons and to avoid errors with json arrays
we should add support of request's json param alongside the data
param.
Change-Id: Id866ed13764b1d4db75cf1a819b53a7e8955b34a
Signed-off-by: Oleg Ovcharuk <vgvoleg@gmail.com>
Reply to address is useful when sending email from an unmonitored email
address and to give user a place to respond in order to contact us.
Add Reply-to as described in section 3.6.2 of RFC5222
https://tools.ietf.org/html/rfc5322#section-3.6.2
Change-Id: Ib6b2bdc130e4f9e5170eb88760d69c3e08d2a1c7
This method allows to specify a private key and avoids its storage
in the filesystem of the executors. This can be used later in
combination of a secrets_retrieve to use keys stored in barbican.
Change-Id: Ide438a7f6d24c8bdc9eb2c82e935fd39a6acc2c6
Closes-Bug: #1806703
The ssh error message can lead to information leak.
Removing the extra ssh message effects only the CLI call,
the full message is still being logged
Change-Id: I0b28e1cb17d4ce3ae711a25b6eaffb4ebf00ccd6
Closes-Bug: 1783708
Adds support for cc and bcc addresses to send mails as copy to
administrators and also html formatting. If the html body is specified
the mail will be sent as multipart.
Closes-Bug: #1783349
Change-Id: I2b90354c33052c4b7ae3a98a08e7df1055524a25
remove invalid todo comment in std_actions.py, since there is no
need to implement this feature.
Change-Id: I500312bb039260853a4d96a54c3395992947b9d5
Related-Bug:#1676411
If the HTTP request fails, we need to fail the task. Returning the error
from the parent class will do this. While this means we also return the
success result it will be ignored by the Mistral engine.
Credit to @lijianying10 on GitHub for sending this fix via a pull
request. Tests were then added to verify the change.
Closes-Bug: #1779973
Change-Id: Ib8754c8de2d6677d71383b3793d0fa168be575f5
Fixes the bug where calling std.ssh action without "password" input was resulting in error.
Closes-bug: #1756272
Change-Id: I5f7af13326933658014a9c42c495d779bec7f14f
* Base action class now has an initializer so all subclusses
must call it in their own initializers.
Change-Id: I2923c1af1ec2dfcd6a671cf52c9db08c58839c20
This updates Mistral to reflect the change in mistral-lib, which was
done to clarify the property name. Tasks don't have an ID, but the
execution of a task does. This makes it consistent with
workflow_execution_id and action_execution_id
Change-Id: I80feeaeff42b87c6ea832b92697603a9e6965086
Related-Bug: #1718353
Depends-On: I049cf7312c58d83e9405aaa36a6961cca006cb16
This may be useful for a testing purpose if you want to emulate a
action error with data for timeout, retry and the rest of task policies
Change-Id: I679b8338dade284817000670e96ab108481372ae
Signed-off-by: Vitalii Solodilov <mcdkr@yandex.ru>
Alphabetically order your imports by the full module path.
Organize your imports according to the Import order template and
Real-world Import Order Examples below.
{{stdlib imports in human alphabetical order}}
\n
{{third-party lib imports in human alphabetical order}}
\n
{{project imports in human alphabetical order}}
\n
{{begin your code}}
Change-Id: I16e97549a7ae6ddedc07e76dba72236460060650
This move allows us to remove the previous method for injecting
context information into actions. It was only used by the
MistralHTTPAction.
Related-Bug: #1718353
Co-Authored-By: Renat Akhmerov <renat.akhmerov@gmail.com>
Change-Id: Ie5c39d35e5848accb24f1cd15e88a4a2dd4b69c0
During some cases resp.content may have a context not in json format
and with 'None' in the resp.encoding. Need to properly handle this
situation and don't try to pass 'None' as an argument to decode.
Change-Id: Id87d650996f16b5ffab79d72413134a4c7fe9ca9
Closes-Bug: #1700608
If resp.content is empty and resp.encoding is not utf-8
then it can't be properly decoded, that's why additional
check was added.
Change-Id: Ie1bab1eff209e88552d4c9979ecec233ba041be4
Closes-Bug: #1700608
This is a middle step to move serveral parts into mistral-lib
Fixes a typo on the deprecation message
Changes the last actions to use mistral-lib actions
Convert the serialization to use mistral-lib serialization
Use mistral-lib results as the standard. (the serialization mixin change
is required for the results to work)
Next steps are:
change mistral-lib serialization to take care of all serialization
change dependent libraries to use mistral-lib directly
Change-Id: I4eacf5ce2e72916b700e8bc77ac9d95859131931
Sometimes action result is present not in "utf-8", it cases
to "UnicodeDecodeError: 'utf8' codec can't decode byte" so
need to pass proper encoding format in serialiser and deserialiser
if it is different from default, but it should be resolved in
general as a fix for another bug (#1676411).
As a hotfix for linked bug, encoding was changed in the
run Http action method directly.
Also default encoding value in the FakeHTTPResponse was changed
from 'utf8' to 'utf-8' like in official python request lib docs.
Change-Id: I0ab9ec84c0973f362b57ab82248f27768a984678
Closes-Bug: #1667415
According to SMTP protocol both the subject and body of the email
can be blank. Hence, making the body of std.email as optional.
Change-Id: I1a2551467247a46c4e3fa5425a367d0f83756b9c
Closes-Bug: #1649366
When attempting to run a workbook with an email action (std.email)
you have to specify a value for smtp_password.This patch attempts
to fix this bug.
Change-Id: I60f30e306ba673a1015e4bfb156ce282ed98406a
Closes-Bug: #1639682
In some part in the code we import objects.
In the Openstack style guidelines they recommend to import only
modules.
http://docs.openstack.org/developer/hacking/#imports
Change-Id: I8813c92c23f50f3b46f6cde2f609a384d7afc3c6
Unit tests should not rely on internet connection. This commit
adds new 'std.sleep' action that can symulate long running action.
Change-Id: Ib7bede239942d429d1bb1ed6f5a2611f64520848
Closes-bug: 1538996
* Callback URL is now provided by std.mistral_http
action to easy understand where the result should be sent
Closes-Bug: #1500833
Change-Id: I347edc5848bde4262a2d57f1787db5f16e218f0a
In MIMEtext by default encodes in 'us-ascii'
for py27 and 'utf-8' in py34. So 'utf-8'
encoding is specified in argument.
Also in py27 constant string is already in unicode
type but in py34 it is byte type. So
string is decoded for py27 in unit test.
Partially-Implements: blueprint mistral-py3
Change-Id: I7004bae1d46d20630a30cac1abb65efe68765401
* SSH action uses private key names instead of
private keys themself. All private keys now
should be in <user-home>/.ssh/, e.g. searching
of key 'my_key.pem' will be at <user-home>/.ssh/my_key.pem
* Fixed functional tests
* Remove sudo while running tests
Closes-Bug: #1507600
Depends-On: Idc4340cda80d02f4ee88c91600b72d0f914c4084
Change-Id: I503d78ff541183ce850476229842fa6f20061a00
* New SSH action uses gateway for accessing to guest network VMs
* Connection allowed only using private key
* Keys on gateway VM and target VM must be identical
Change-Id: I3906415eac30fae5bd6fc9e2cab70bedbb377ba9
* Password argument is optional now
* Ability to pass private_key to use on executors
* Modified std.ssh docs
Closes-Bug: #1502076
Change-Id: I1e92adabafada5e45355a0e3236e8965ec921f2f
* The problem comes from JSON serializing. Cookies and
datetime is not JSON serializable, so we need to
preprocess them manually.
Closes-Bug: #1477048
Change-Id: I2dbd1463f1b3549f661c0dba46ab6f949c4fffb4
* It now returns wf.Result() instance in case of error that
carries all needed data.
* Refactored std.http related tests.
* Added a simple test for std.http error result
Change-Id: Icae0b7eccaed9185a521b762a41dd52f6be0e6a7
* use oslo.log for logging functionality
* get rid of the file /mistral/openstack/common/log.py
Implements: blueprint mistral-log-enhancement
Partial-Bug: #1459188
Change-Id: I6b62cffda0a20b6caf59cc30bcc33c21ae1e5898
Oleg Ovcharuk