After this patch, user can update logging format to include root_execution_id in logs, which will be helpful to find and debug logs related to specific workflow execution.
- Logs about creation and status changes of Mistral entities(execution,
task, action execution, etc) are changed to INFO log level.
- User can update logging_context_format_string to include root_execution_id in logs.
Implements: Implements: blueprint improve-mistral-loggers
Change-Id: I54fe058e5451abba6ea7f69d03d498d78a90993e
When building mistral context (using oslo.context) from environment, the
tenant extra key was given, which is not accepted by oslo.context.
This issue was detected when debugging a mistral-client test:
mistralclient.tests.functional.cli.v2.test_cli_v2.NegativeCLITests.test_target_action_execution
Signed-off-by: Arnaud Morin <arnaud.morin@ovhcloud.com>
Change-Id: I72ce8851ea5b379a8af75b32fb600691638836af
For many various support reasons, Mistral should have a special
endpoint to store all necessary info data. This endpoint will read
json from created by admin info file. To configure this you should
use mistral configuration:
[api]
enable_info_endpoint = True
info_json_file_path = info.json
Change-Id: I6f344dc15a4ca5c69a6b21841544a31f95eb393f
Implements: blueprint mistral-info-endpoint
Signed-off-by: Oleg Ovcharuk <vgvoleg@gmail.com>
The tenant argument of RequestContext is no longer available since
oslo.context >= 4.0.0 . This change fixes the compatibility issue
caused by that removal.
Note that this still keeps reference to 'tenant' argument to make
the code compatible with older oslo.context, but that can be removed
once oslo.context >= 4.0.0 becomes available in upper-constraints.
Change-Id: Ie671f50e5ff5a7c746f9e95691eaf4dd19937b52
This sets up the HTTPProxyToWSGI middleware in front of Mistral API. The
purpose of this middleware is to set up the request URL correctly in
the case there is a proxy (for instance, a loadbalancer such as HAProxy)
in front of the Mistral API.
The HTTPProxyToWSGI is off by default and needs to be enabled via a
configuration value.
It can be enabled with the option in mistral.conf:
[oslo_middleware]
enable_proxy_headers_parsing=True
Closes-Bug: #1590608
Closes-Bug: #1816364
Change-Id: I04ba85488b27cb05c3b81ad8c973c3cc3fe56d36
keystone_authtoken/auth_uri is deprecated [1]. Use www_authenticate_uri
instead.
keystonemiddleware in requirements and lower constraints should be increased
because www_authenticate_uri was introduced in keystonemiddleware 4.18.0.
[1] https://review.openstack.org/#/c/508522/
Change-Id: I99b0ee941d702a28fb4f392d9747d0e2257a42c8
Closes-Bug: #1788174
We will now only accept the string values "False" or "True". Previously
any given value was interpreted as a string and thus True.
Closes-Bug: #1666565
Change-Id: Ibd105c881dbe16cd4516bfb775c8f5f43c961b45
This patch delivers the first working version of a distributed
scheduler implementation based on local and persistent job
queues. The idea is inspired by the parallel computing pattern
known as "Work stealing" although it doesn't fully repeat it
due to a nature of Mistral.
See https://en.wikipedia.org/wiki/Work_stealing for details.
Advantages of this scheduler implementation:
* It doesn't have job processing delays when a cluster topology'
is stable caused by DB polling intervals. A job gets scheduled
in memory and also saved into the persistent storage for
reliability. A persistent job can be picked up only after a
configured allowed period of time so that it happens effectively
after a node responsible for local processing crashed.
* Low DB load. DB polling still exists but it's not a primary
scheduling mechamisn now but rather a protection from node crash
situations. That means that a polling interval can now be made
large like 30 seconds, instead of 1-2 seconds. Less DB load
leads to less DB deadlocks between scheduler instances and less
retries on MySQL.
* Since DB load is now less it gives better scalability properties.
A bigger number of engines won't now lead to much bigger
contention because of a big DB polling intervals.
* Protection from having jobs forever hanging in processing state.
In the existing implementation, if a scheduler captured a job
for processing (set its "processing" flag to True) and then
crashed then a job will be in processing state forever in the DB.
Instead of a boolean "processing" flag, the new implementation
uses a timestamp showing when a job was captured. That gives us
the opportunity to make such jobs eligible for recapturing and
further processing after a certain configured timeout.
TODO:
* More testing
* DB migration for the new scheduled jobs table
* Benchmarks and testing under load
* Standardize the scheduler interface and write an adapter for the
existing scheduler so that we could choose between scheduler
implementations. It's highly desired to make transition to the
new scheduler smooth in production: we always need to be able
to roll back to the existing scheduler.
Partial blueprint: mistral-redesign-scheduler
Partial blueprint: mistral-eliminate-scheduler-delays
Change-Id: If7d06b64ac14d01e80d31242e1640cb93f2aa6fe
We previously ported the code to mistral-lib, but Mistral has been using
the original copy.
Closes-Bug: #1782765
Change-Id: Ifb518d821097fdf2ec76161ae00f312ced19c272
Sometimes when mistral requests are failing with "401 Unauthorized"
against keycloak, the reason are not mentioned in the logs.
In case keycloack return 401 it must provide the www-Authenticate
response header with the reason:
https://www.w3.org/Protocols/HTTP/1.0/spec.html#WWW-Authenticate
This code take care of it by adding the WWW-Authenticate value to
mistral api-log.
Change-Id: I7ae221aaeb2233184bd4818490e72ff662dca5cb
Closes-Bug: #1737500
These were removed in Ife653558bfcda794e7f37086832f70b0ad7c28a4
but that breaks any actions which require this data, such as
some TripleO actions.
Closes-Bug: 1740891
Change-Id: I777b1f7c7012b735805e8585938b5ce5dec31d26
get_logging_values has been changed recently to not pass the token
anymore, call to_dict as expected.
Change-Id: I3a7f1293a4d0082274af270f86b5c732d898f8bc
Closes-Bug: #1733345
Also, using oslo.context coordinated with oslo.log will provide more
information in the log.
See more information here:
http://openstack.markmail.org/thread/kuvzwhtblwhoz6o5
Co-Authored-By: Lingxian Kong <anlin.kong@gmail.com>
Change-Id: I1f3f4c1546a85129af1bd58ead9132780909f0d3
The puppet module puppet-mistral is moving to use a proper keystone
authtoken module. This supports that transition. A follow on patch
will remove the transition code.
Change-Id: Ief32ae01372c8c8d32fc5e2c89a2927510983a5b
* The initial driver of this effort was that, as it turned out,
it was impossible to have custom serialization for a custom
type. For example, we had class Result and even had
ResultSerializer but it wasn't used. So after taking a closer
look it became clear that our serialization subsystem did not
allow to register any alternative serializers by design.
Because we had to use one serializer implementation registered
in RPC. The solution is to make it more flexible by adding
a special router-like serializer that knows to which specific
serializer it needs to switch depending on an object type.
* Refactored serialization subsystem so that it's more flexible
and manageble
* Added polymorphic entity serializer that can work as a router
and switch between different serializers depending on their type
* Used Result and ResultSerializer in RPC instead of decomposing
result object into primitive fields explicitly
Co-Authored-By: Dawid Deja <dawid.deja@intel.com>
Change-Id: I29d40a0b1b68a5410f3db2f7280c9c6244d55a84
* This class is not needed in Mistral, we can use the exact same
class from oslo.messaging
* Changed RpcContextSerializer to use JsonPayloadSerializer by
default so that we don't have to pass it as a parameter
every time
Change-Id: Ic4ad92cb732c33e6d1971b50076afde219147536
New Insecure flag inroduced for openstack actions. With that mistral
is able to connect to https clouds without verifying server certificate.
Change-Id: If7839ac586ff0b50f3f323a6bd42349eb0c25ca8
We already supported role based api access control, this series patches
will implement resource access control for mistral, so that
administrator could define the rules of resource accessibility, e.g.
admin user could get/delete/update the workflows of other tenants
according to the policy.
TODO:
- Implement update workflow by admin
- Implement delete workflow by admin
- Implement for other resources(workfbook/execution/task/action, etc.)
Partially implements: blueprint mistral-rbac
Change-Id: I8b00e8a260a74457ad037ee7322a7cba9ae34fab
User now could define the region for the openstack actions.
It could be done via API in X-Region-Name and X-Target-Region-Name
in case of multi-vim feature is used.
*API change*
X-Region-Name: Header added to execution create
X-Target-Region-Name: Header added to execution create
Change-Id: Icbf63962a481c1282b95359894fa6245e0e97bac
Related-Bug: #1633345
When using Mistral in multi openstack deployments, user can pass
'X-Target-Auth-Uri' in the header to let Mistral run openstack service
actions in different openstack deployment. 'X-Target-Service-Catalog'
can also be provided but it's optional.
This patch adds 'is_target' attribute to Mistral context, if it's true,
Mistral will talk to another openstack deployment, 'service_catalog'
in the context can be empty or contain target service catalog provided
by user, Mistral will get service catalog dynamically if it's empty;
if it's false, the 'service_catalog' in context can also be empty(when
auth_enable=False) or the content that get from keystone authentication
response.
This patch also fix the tempest failure introduced by:
https://review.openstack.org/#/c/387883/
Related-Bug: #1634090
Change-Id: Iec3ed0333cd08831f0a15f77e3880f07dd89e1e8
Updates the Mistral server to accept the service catalog
from the client request. This enable the server to cooperate
with Keystone Identity V2 and V3 at the same time.
Change-Id: I7ca2aace4d5095828e5053af6965b833109d338a
Closes-Bug: #1612705
Depends-On: I86fa58de00d01c89e4bbc21dbe128f1306e2a1bf
Signed-off-by: Andras Kovi <akovi@nokia.com>
Abstract authentication function so plugins for other authentication
backends can be implemented in cases where keystone is not used. Currently,
mistral is hard coded to support keystone and keycloak. The domain/project
related trust that is specific to keystone is not addressed.
Change-Id: I21994ab20af519b2ba85efd7cbe043547988e5b3
Implements: blueprint mistral-abstract-auth
Only Mistral Cron execution failing to get endpoint
for the OpenStack Services.
Change-Id: I985a8d21fe48b488eb7e452d31b016e8239a5752
Closes-Bug: 1607788
This change adds caching for all the actions. When an action request
is made, the cache is checked to see if a client has already been
created. If an existing client is found, the keystone token expiration
is verified to still be current within the configurable window. Once a
client's token becomes invalid a new client is created and the cache
is refreshed.
The new configuration option for setting the token expiration window
is expiration_token_duration present in the default section.
Change-Id: I854f0251d9ec3623700d8a4025df8f1bc632a3e9
With this change executor will fail actions that are redelivered
and have flag safe_rerun set to false
Implements blueprint mistral-task-delivery-model
Change-Id: Ie0e728cf59af9fe44c8fd1d243439a82d9478ff4
* Changed AuthHook for Pecan that implements token validation
* Added another config option to disable SSL verification for
KeyCloak access tokens
* Added unit tests for successful and failed KeyCloak
authentication that use request_mock library
* Minor style changes
Change-Id: I87f8d54fc58f82952a4c68831547e6dab320230e
Credentials received in the request are used to retrieve endpoint
list from keystone. This avoids the usage of the admin creds
and opens the gate towards connecting to any clouds without
previous configuration.
Implements: blueprint mistral-multi-vim-support
Change-Id: Ib5ae5911f2535f4f340af8f4bcb4817818747029
This patch correlates with [1] in client side, when authentication
failed, the error message should be delivered to client side.
[1]: https://review.openstack.org/232395
Change-Id: I6a6ced466c05849fd9ff3dcd8377a57c9e9b595f
Closes-Bug: #1502840
* use oslo graduated modules, delete openstack/common package since there
is no dependency on oslo-incubator modules now.
* delete openstack-common.conf for the reason above.
* update project requirements automatically.
Change-Id: I80610cbfe7fd54263c8a2d9178ec9a2498c91899
Closes-Bug: #1459188
The Oslo libraries have moved all of their code out of the 'oslo'
namespace package into per-library packages. The namespace package was
retained during kilo for backwards compatibility, but will be removed by
the liberty-2 milestone. This change removes the use of the namespace
package, replacing it with the new package names.
The patches in the libraries will be put on hold until application
patches have landed, or L2, whichever comes first. At that point, new
versions of the libraries without namespace packages will be released as
a major version update.
Please merge this patch, or an equivalent, before L2 to avoid problems
with those library releases.
Blueprint: remove-namespace-packages
https://blueprints.launchpad.net/oslo-incubator/+spec/remove-namespace-packages
Change-Id: I73addc2c144c76c60f046e83c97e3b6ffe09d879
* Fix work with trust-scoped token
* Fix NovaAction and KeystoneAction
* Added new field to MistralContext - is_trust_scoped
Change-Id: Id1a8c959ffd07032f9ab03e5fdbafbdc66374ce3
Zuul