Add periodic task to clean up workflow failure
Change-Id: I9d7fb601903307b54a71def2aeceb00170313317 Implements: bp add-periodic-tasks
This commit is contained in:
parent
2c10be4ec4
commit
4e746cb5a3
|
@ -0,0 +1,122 @@
|
|||
==================
|
||||
Masakari Spec Lite
|
||||
==================
|
||||
|
||||
Please keep this template section in place and add your own copy of it between
|
||||
the markers. Please fill only one Spec Lite per commit.
|
||||
|
||||
<Title of your Spec Lite>
|
||||
-------------------------
|
||||
|
||||
:link: <Link to the blueprint.>
|
||||
|
||||
:problem: <What is the driver to make the change.>
|
||||
|
||||
:solution: <High level description what needs to get done. For example:
|
||||
"We need to add client function X.Y.Z to interact with new server
|
||||
functionality Z".>
|
||||
|
||||
:impacts: <All possible \*Impact flags (same as in commit messages) or 'None'.>
|
||||
|
||||
Optionals (please remove this line and fill or remove the rest until End):
|
||||
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
|
||||
|
||||
:how: <More technical details than the high level overview of `solution`
|
||||
if needed.>
|
||||
|
||||
:alternatives: <Any alternative approaches that might be worth of bringing
|
||||
to discussion.>
|
||||
|
||||
:timeline: <Estimation of the time needed to complete the work.>
|
||||
|
||||
:reviewers: <If reviewers has been agreed for the functionality, list them
|
||||
here.>
|
||||
|
||||
:assignee: <If known, list who is going to work on the feature implementation
|
||||
here>
|
||||
|
||||
End of Template
|
||||
+++++++++++++++
|
||||
|
||||
Add periodic task to clean up workflow failure
|
||||
----------------------------------------------
|
||||
|
||||
:link: https://blueprints.launchpad.net/masakari/+spec/add-periodic-tasks
|
||||
|
||||
:problem: Due to some unknown circumstances, there is a possibility few of the
|
||||
notifications might get into ‘error’ status or it might remain in
|
||||
‘new’ status forever. There should be some way to retrieve such
|
||||
notifications and process them to completion.
|
||||
|
||||
:solution: Add a periodic task “process_unfinished_notifications’ which will
|
||||
execute at regular interval as defined by the new config option
|
||||
“process_unfinished_notifications_interval” in seconds. Default
|
||||
value for this option will be 120 seconds, however operator can set
|
||||
this interval value as per the requirement. Inside this
|
||||
periodic task, it will retrieve all notifications which are in
|
||||
‘error’ or ‘new’ status and then execute recovery action workflow to
|
||||
process all of them. The notifications which are in ‘new’ status
|
||||
will be picked up based on a new config option
|
||||
‘retry_notification_new_status_interval’. Default value for this
|
||||
option will be 60 seconds, however operator can set this interval
|
||||
value as per the requirement. Each notification has ‘generated_time’
|
||||
field, if this time + retry_notification_new_status_interval value
|
||||
(in seconds) is greater than or equal to the current system time,
|
||||
then all such notifications in ‘new’ status will be picked up by
|
||||
this periodic task. Also, the notifications in ‘error’ status will
|
||||
be picked up too.
|
||||
|
||||
Let’s understand the transition state of notification for different
|
||||
statuses for success case:
|
||||
notification current status error -> running -> finished
|
||||
notification current status new-> running -> finished
|
||||
If the workflow execution fails, then the transition state of
|
||||
notification would be:
|
||||
notification current status error -> running -> failed
|
||||
notification current status new-> running -> failed
|
||||
|
||||
Note: One important point to take note of is if the original
|
||||
notification status is ‘new’ then it won’t be retried again if the
|
||||
workflow fails to process it in the periodic task. It’s status will
|
||||
be directly set to ‘failed’. The operator needs to take corrective
|
||||
action for all notifications which are in ‘failed‘ state manually.
|
||||
|
||||
:alternatives: Add two periodic tasks ‘process_error_notifications’ and
|
||||
‘process_queued_notifications’ to process notifications which
|
||||
are in ‘error’ and ‘new’ status respectively. These periodic
|
||||
tasks will be called at regular intervals as defined by two
|
||||
new config options “process_error_notification_interval’ and
|
||||
‘process_queued_notifications_interval’. The logic for retrieval
|
||||
of notifications which are in ‘error’ and ‘new’ statuses will be
|
||||
exactly same as above solution. The only difference would be
|
||||
in the notification status upon its completion as explained
|
||||
below.
|
||||
|
||||
Transition state of notification for different statuses for
|
||||
success case.
|
||||
notification current status
|
||||
error (process_error_notifications) -> running -> finished
|
||||
notification current status
|
||||
new (process_queued_notifications) -> running -> finished
|
||||
|
||||
If the workflow execution fails, then the transition state of
|
||||
notification would be:
|
||||
notification current status
|
||||
error (process_error_notifications) -> running -> failed
|
||||
notification current status
|
||||
new (process_queued_notifications) -> running -> error
|
||||
This means that the notification will be again eligible for
|
||||
reprocessing during the next cycle of
|
||||
‘process_error_notifications’ periodic task.
|
||||
|
||||
:impacts: None
|
||||
|
||||
:timeline: Expected to be merged within the Ocata time frame.
|
||||
|
||||
:reviewers: sam47priya@gmail.com, kajinamit@nttdata.co.jp,
|
||||
tushar.vitthal.patil@gmail.com
|
||||
|
||||
:assignee: Abhishek Kekane
|
||||
|
||||
End of Add periodic task to clean up workflow failure
|
||||
+++++++++++++++++++++++++++++++++++++++++++++++++++++
|
Loading…
Reference in New Issue