This commit is part of a series to retire the Packaging Deb
project. Step 2 is to remove all content from the project
repos, replacing it with a README notification where to find
ongoing work, and how to recover the repo if needed at some
future point (as in
https://docs.openstack.org/infra/manual/drivers.html#retiring-a-project).
Change-Id: I6f54f00b686a16e9b3ea96642c78842db12af9d3
Let's dive into what the problem is here.
First a description of what happens to a task that
is to be executed in a external (but local) process
via the process executor mechanism.
When a task is about to be sent to execute in the
external (but local) process its first cloned, this
is mainly done so that its notification callbacks can
be altered in a safe manner (ie not altering the
original task object to do this) and that clone has
its notifier emptied out.
What replaces the clone's notifier callbacks though
is a new object (that has a __call__ method so it
looks like just another callback) that will send
messages to the parent process (the one that has
the engine in it) over a secure(ish) channel whenever
the local task triggers its notifier notify() method.
This allows for callbacks in the parent process to
get triggered because once the messages recieved the
original tasks notifier object has its notify() method
called (therefore those callbacks do not really know
the task they are getting messages from is executing out
of process).
The issue though is that if the ANY(*) event type is registered
due to how it works in the notifier is that if the child/cloned
notifier has the ANY event type registered and the cloned task
calls notify() with a specific event this will cause the ANY
callback (in the clone) to transmit a message *and* it will
cause the *specific* event callback to also transmit a message
back to the parent process.
On the engine process side it will get 2 messages and trigger
the callbacks 3 times (twice for the specific event callback
because how the local notifier has the ANY callback registered
and one more time when the local process also sends the same
event based on its registration of the ANY event in the child
process).
This is not what is expected (the message rcved on the engine
process should only trigger one callback to get triggered
if the engine process task has no ANY callback registered or two
engine process callbacks to get triggered if the engine process
task has the ANY callback registered).
Closes-Bug: #1537948
Change-Id: I271bf1f23ad73df6c177cf00fd902c4881ba44ae
html_last_updated_fmt option is interpreted as a
byte string in python3, causing Sphinx build to break.
This patch makes it utf-8 string.
Changing Popen to .check_output because of 3 reasons:
1. check_output() will raise CalledProcessError if
the called process returns a non-zero return code.
2. For consistency with keystone [1] and cinder [2]
3. It makes the code look much better.
[1] https://review.openstack.org/#/c/457142/
[2] https://review.openstack.org/#/c/433081
Change-Id: I6e9c2009f224abc282acbf957d7e999c7f87e9eb
assertRaisesRegexp was renamed to assertRaisesRegex in Py3.2
For more details, please check:
https://docs.python.org/3/library/
unittest.html#unittest.TestCase.assertRaisesRegex
Change-Id: I89cce19e80b04074aab9f49a76c7652acace78b3
Closes-Bug: #1436957
Running a nonblocking conductor or two conductors on the same host will re-execute the same job multiple times with the current implementation of 'claim' for the redis jobboard backend. This is different from the ZooKeeper jobboard backend, there the same owner of a job is not allowed to reclaim the job again (https://github.com/openstack/taskflow/blob/master/taskflow/jobs/backends/impl_zookeeper.py#L554). If the same owner is allowed to reclaim the job again there can be no concurrent execution on the same owner because all jobs will be re-claimed and re-executed by the same owner every pass as long as it's on the jobboard.
To reproduce this behavior:
- Use the redis jobboard backend
- Create a flow with a task that sleeps 10 seconds in the execute method
- Post that flow as a job
- Run a nonblocking conductor
It will claim and execute the same job multiple times in a loop until the first worker is finished and consumes the job. After this change it will not re-execute the same job multiple times.
Change-Id: I4f6c364211500e510fc496f23b03ce056771417d
dict.keys() returns object dict_keys on Python 3, can't work well
with kombu recently, convert it to list before futhur processing.
Closes-Bug: #1687855
Change-Id: I9dcfadd8f16eccce06f4cde3b2a6669f28a3f8bc
Add simple script to setup mysql and postgresql databases, this script
can be run by users during testing and will be run by CI systems for
specific setup before running unit tests. This is exactly what is
currently done by OpenStack CI in project-config.
This allows to change in project-config the python-db jobs to
python-jobs since python-jobs will call this script initially.
See also
http://lists.openstack.org/pipermail/openstack-dev/2016-November/107784.html
Needed-By: I1c6c23b39a93991d2dcb5e753fd851d244326581
Change-Id: Ibbe9910b0bae398542bf5fb84f2c7151bc898770
Lock down the various state machine action handling
functions so that they are ensured correct (and consistent)
access to the storage layer when they are modifiying and/or
reading it.
Change-Id: Ie893a44aa963ab515f19e77f9904f49c843cb4e5
This moves out the engine next to run (or revert)
selection process to a single class that just does
this and moves out the common functions the analyzer
class provided to the runtime object (which all
components can access).
This makes it easier to adjust the selection algorithm
in different ways.
Change-Id: I091c69297a7bff60729791d3ca6c3fae14d6eea5
Adding constraints support to libraries is slightly more complex than
services as the libraries themselves are listed in upper-constraints.txt
which leads to errors that you can't install a specific version and a
constrained version.
This change adds constraints support by also adding a helper script to
edit the constraints to remove taskflow.
Change-Id: I89a14733295748fff2a30f038084d75596204083
Tony Breeds
OpenStack Release Bot