Commit Graph

105 Commits

Author SHA1 Message Date
OpenDev Sysadmins e6565769b5 OpenDev Migration Patch
This commit was bulk generated and pushed by the OpenDev sysadmins
as a part of the Git hosting and code review systems migration
detailed in these mailing list posts:

http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.html
http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004920.html

Attempts have been made to correct repository namespaces and
hostnames based on simple pattern matching, but it's possible some
were updated incorrectly or missed entirely. Please reach out to us
via the contact information listed at https://opendev.org/ with any
questions you may have.
2019-04-19 19:51:20 +00:00
Robert Collins 33c4b02e76 Next version is 0.2
pbr pre-versioning causes current sdist to have versions leading up to 0.1, but
that is already released - we should be leading up to 0.2.

Change-Id: Ia7860bf4588daeb19fc0a75c7f914e6b11c4ccae
2014-08-20 13:57:47 +12:00
Pranesh Pandurangan e2cebf411e Add unittests for entropy
Added run_tests.sh, taken shamelessly from taskflow.

Change-Id: I39b9cfa106c06296b917077cc3ef93dbf5a9a03b
2014-07-19 11:50:19 -07:00
Pranesh Pandurangan c79d12c645 Finish changes to stop an engine
Added a call to stop all react scripts, by passing all known routing
keys to the react_killer function. Then stop the threadpool executor.
Finally, Raise a known exception to stop the watchdog thread. This
will throw an ugly traceback, but will shutdown the engine
gracefully.

Also made minor changes to the example react.json, to change the log
format. Knowing the time a log was printed is useful.

Change-Id: Ibed06f79547312d188feb499f937eb5390d60c3e
2014-07-14 16:47:43 -07:00
Pranesh Pandurangan 857fb52cf1 Add callbacks for futures
Added callbacks for the known futures (react scripts, scheduler,
serializer). This will help us debug known long-lasting futures
easily. Define one function called future_callback, and use
functools to pass arguments to the callbacks to determine type
of future, etc

Change-Id: I579a41e92736568535ce3f64c5507ac5224d5a62
2014-07-14 16:39:03 -07:00
Pranesh Pandurangan 5b647e5f79 Add logic to stop repair scripts
When watchdog detects that repair script(s) have been killed, get
a list of scripts to nuke and pass to stop_repair_scripts. Then,
get its routing key(s), and send a message from a special user to
any queue listening on those keys.

Modified an example repair script to show how it could be killed,
but need a more concrete way that that. For now, messages from
'react_killer' will raise the RepairStoppedException, which will
stop react scripts

Modified the example engine cfg to have some details about the
kombu connection to use.

Implements blueprint kill-repair-scripts
Change-Id: I67e15e9b9ebb5d36c5cb0e01995bc95f7a73b3dd
2014-06-20 11:41:23 -07:00
Pranesh Pandurangan 0173ef0b53 Store routing_keys in a dict instead of set
This will let us keep track of which react script uses what routing
key. We will lose the set property, but that is immaterial because
the system will be in trouble anyway, if there are duplicate keys

Change-Id: I89299fcb7dfbe7ab4d7b2d1df9c1cb001489fd5f
2014-06-19 17:20:35 -07:00
Pranesh Pandurangan d0e9ace151 Separate out adding and deleting repair scripts
Currently, any change in the repair cfg is considered as a new
react script being registered. To change that add a _get_repairs
function, and form a list of audits to create/nuke.

Change-Id: I5c039f0fee3ef79e864493ace9cde366dc87a381
2014-06-19 17:09:17 -07:00
Pranesh Pandurangan 7a6999c9eb Add an unregister repair option
Added a parser and function to unregister repair scripts. Remove
the repair script from the backend repair cfg, and watchdog will
catch it in the engine.

Change-Id: I7b93ca7e5eb4430b7c9502c8dd84af75b2a9fae3
2014-06-19 01:18:59 -07:00
Pranesh Pandurangan c867e3e4bd Add some exceptions and logging around serializer
Added a serializer exception to raise if the serializer fails, to
replace the generic Exception we raise now.

Added a logging statement to another exit point to provide some
more information

Change-Id: I20d58ce8d9d943f957f491fdf2c01516d3975870
2014-06-19 00:17:44 -07:00
Pranesh Pandurangan b6a1d3d1e1 Fix errors caused by wrong rebasing
The conversion of self.ENABLED ot states.ENABLED got mixed up in
a following commit. Just a commit to repair that stuff.

Change-Id: I4ab4c6b35dc04d281f52184f035e86c15b269044
2014-06-18 23:29:19 -07:00
Jenkins e9cb9ab547 Merge "Add more steps to stop an engine" 2014-06-18 22:30:21 +00:00
Jenkins 991550b1f6 Merge "Introduce engine states" 2014-06-18 22:30:16 +00:00
Jenkins b98e3e1467 Merge "Refactor and add a check around a variable" 2014-06-18 22:30:09 +00:00
Pranesh Pandurangan 80d0194d86 Cleanup exceptions a bit
There is no need to override the constructor in every exception.

Change-Id: I551f551611ea4e8902337901512f8498afe16210
2014-06-17 18:39:04 -07:00
Pranesh Pandurangan 119595c6c7 Add more steps to stop an engine
Add elements to audit run queue only if engine is enabled.
Use engine state as the conditional for running serializer, audits.
On stopping the engine, clear the current run queue.

Work on blueprint kill-repair-scripts

Change-Id: Iba8c94caec9e75f4b6fda8d4bd0bf07b25e596f8
2014-06-17 21:40:27 +00:00
Pranesh Pandurangan 68654b7b86 Introduce engine states
Introduce two states, ENABLED and DISABLED, in a new states.py file,
and set engine to disabled when stopped, and enabled when running

Change-Id: I1d4c89b453ac04cd5dce09c5faafe3241cd2dc43
2014-06-17 14:40:20 -07:00
Pranesh Pandurangan 1a860ac47b Refactor and add a check around a variable
Changed serializer to self._serializer, to be able to access it
from other places in the engine class. Added a check to make sure
it doesn't get created twice.

Change-Id: Ia4007a5f98af04677b2006a88b66cbf8ceec2982
2014-06-17 14:02:36 -07:00
Jenkins e2f7101b0b Merge "When creating an engine, add an enabled field" 2014-06-17 00:53:32 +00:00
Jenkins 9ad24284fe Merge "Kill watchdog thread when engine is stopped" 2014-06-17 00:53:27 +00:00
Jenkins 3e19f33b40 Merge "Add engine_cfg to watchdog" 2014-06-16 23:43:37 +00:00
Jenkins fe595af499 Merge "Modify watchdog related functionality" 2014-06-16 23:42:47 +00:00
Jenkins 9c06eb8b49 Merge "Use CLI input as react script name" 2014-06-16 23:41:48 +00:00
Jenkins 99a921150b Merge "Track known repair scripts and their routing keys" 2014-06-16 23:40:36 +00:00
Jenkins 7891dd8eb9 Merge "Replace some generic Exceptions with specific ones" 2014-06-16 23:40:10 +00:00
Jenkins 29047192a5 Merge "Add some exception classes" 2014-06-16 23:39:40 +00:00
Pranesh Pandurangan f3c51ab67d When creating an engine, add an enabled field
We use this field when disabling an engine. Stay clear of possible
KeyErrors by adding this field when registering an engine.

Change-Id: Iacca4f99be018b5147b6a00f82e7c772ec88a8f3
2014-06-12 23:08:41 -07:00
Pranesh Pandurangan 4e4533796b Kill watchdog thread when engine is stopped
Step 1 of stopping the engine via api. Stop the watchdog thread,
so we don't do anymore monitoring.

Change-Id: I3d78834ab3cf81643260a47efec7c738f7a2f63f
2014-06-12 23:07:25 -07:00
Pranesh Pandurangan b41ea8a81f Add engine_cfg to watchdog
Watch changes to engine config via watchdog. When the file is
modified, watchdog picks it up, verifies if the current engine
is disabled, and if yes, take steps in stop_engine(), to stop
the engine. That function is currently empty, will be filled in
in future commits

Change-Id: If3e2e952009f2f8a610459645c4403c77581e717
2014-06-12 23:06:39 -07:00
Pranesh Pandurangan a7479d66e8 Modify watchdog related functionality
Specifically, make the code around watchdog accept a list of dirs
to watch, instead of a single one. That way, we can watch multiple
files.

Change-Id: If685428f7e7bbd251a61ac4fb57e3a2f63edecbb
2014-06-13 01:50:54 +00:00
Pranesh Pandurangan 8967185608 Use CLI input as react script name
Like in audit scripts, it doesn't make sense to keep a name field
in react.json anymore. Use the name passed in CLI instead.

Change-Id: Ic6d25505b62b640a7ade2990a927a9ea18bd90eb
2014-06-13 01:48:24 +00:00
Pranesh Pandurangan dcf82d6490 Track known repair scripts and their routing keys
When we setup a react script, store the future and the routing key.

Work on blueprint kill-repair-scripts

Change-Id: I5718913b3642f5107204b97e5d10480a3ffa3421
2014-06-12 18:47:58 -07:00
Pranesh Pandurangan 473e2febc0 Add an audit unregister option
Allow users to remove audit scripts through a simple CLI call. The
unregister-audit call will remove an audit script from audit.cfg,
thus stopping the execution of all but the currently scheduled audit
scripts.

partially implements blueprint simple api
Change-Id: I1d5328d87b607c2f5cfdaebb7448a11673f38d48
2014-06-11 03:07:58 +00:00
Pranesh Pandurangan b1ca5a4c91 Refactor some parser code
Rename scheduler_parser to start_engine parser to make more sense,
also add the -p (purge) option that an earlier commit needs.

Change-Id: I5058e6f60804166e71fd5fefb2e298d27630b3d8
2014-06-11 03:07:23 +00:00
Pranesh Pandurangan 47e35d1d2c Add a stop engine call
Add a call to stop engine, this function just sets the enabled field
to false, and psutil terminates the process (equivalent to what we
do now). Also add psutil to requirements

Change-Id: Idc3edb2bf1c9ed55d7d77973c59e3d3562e2ad8b
2014-06-10 20:06:05 -07:00
Pranesh Pandurangan 308cd375b9 Replace some generic Exceptions with specific ones
Replaced occurences of Exception with some specific exceptions

Change-Id: Ib6d71172ad607b0cdcc99147920dc6b032b5c34c
2014-06-10 19:50:18 -07:00
Pranesh Pandurangan a1fdc66e4e Add some exception classes
Added NoSuchEngineException and NoEnginesException classes, which
will allow better exception handling in the code, instead of just
raising generic Exception.

Change-Id: Ic9104dda3b1e5414ae11e8576a4ad29961be5e95
2014-06-10 19:28:47 -07:00
pran1990 81a9042ff1 Add some more checks to engine creation
Add a field called enabled in the engine cfg file that keeps track of
files. This can be used to keep track of stopped engines, and
monitored using watchdog/similar. Also added some more checks at
engine creation time.

Partially Closes-bug:1309406
Change-Id: I1c365c2c438e6ed0a44413e1d09c69d3fab7ab7b
2014-06-10 19:20:55 -07:00
Pranesh Pandurangan 1d739dea9c Add some abstract methods to backend
Added some methods in base.py that should probably be defined in
every backend

Change-Id: I6f99994b82c2f17083d83b4723f3b4d61fc80160
2014-06-10 08:31:04 -07:00
Pranesh Pandurangan 0be7808705 Move register-audit/repair code to backend
Abstract out cfg file operations in the register workflow. Changed
the get_driver code in engine to be static, so we can call it from
main too.

Added a new function in file_backend to return the config file given
the script type. Eg. return audit_cfg for audit.

Added a new function in file_backend to replace check_duplicate, that
returns True if a script already is already registered.

Added a couple of string variables in base.py

The function get_cfg_file, when using a db, will actually return a
table. So this belongs in the backend, the code refactor here ensures
this function is not called in the main() code.

Raise errors instead of returning None in the some backend functions

Completes blueprint backend-abstraction
Change-Id: I20d6bd46caf56c750e4b1193a6f5d00ce4e930f6
2014-06-09 19:50:56 -07:00
Jenkins e4a6dc64de Merge "move repair-related code to backend" 2014-06-05 10:21:49 +00:00
Jenkins aa01b73c14 Merge "move audit-related code to backend" 2014-06-05 09:38:18 +00:00
pran1990 e9f7137ceb move repair-related code to backend
Add two functions in filebackend, to retrieve all repairs, and to
retrieve an repair script's config given the name.

Replace occurences of utils.load_yaml in engine.py with calls to
above functions

Change-Id: If7ed5c44b771aa76b94697cca16b251a167e4956
2014-06-05 09:26:32 +00:00
pran1990 e7c8fee0a8 move audit-related code to backend
Add two functions in filebackend, to retrieve all audits, and to
retrieve an audit script's config given the name.

Replace occurences of utils.load_yaml in engine.py with calls to
above functions

Change-Id: Ia204d0db8a078c28ce27a2ae2ed57eb8b7fdf23b
2014-06-05 02:24:19 -07:00
Jenkins af40dcba1b Merge "Move cfg file creation to driver" 2014-06-03 23:42:40 +00:00
Jenkins 568d51bca2 Merge "Move some file creation code from main to utils" 2014-06-03 09:57:05 +00:00
Jenkins 286d0e18f0 Merge "Watchdog cleanup" 2014-06-03 09:56:59 +00:00
pran1990 d4fcac48f1 Move cfg file creation to driver
Added code to load the file backend as a driver in engine __init__,
and a function in FileBackend to create cfg files.

Also added extra field in engine.cfg to specify what kind of backend
to use.

Change-Id: I6d3f24d4f676c72c94afff2c4c7f54a35cf1d4b1
2014-06-02 20:31:14 +00:00
pran1990 0f5954359c Move some file creation code from main to utils
As part of engine creation, we create files for audit and repair cfg.
Move this to utils.

Change-Id: I9d5075cb854ab5585fddcb58013ffa2530add970
2014-06-02 13:29:19 -07:00
pran1990 81cd48dacc Watchdog cleanup
Removing a function we don't use, audit_modified.
Make the file->function mapping a part of the engine.

Change-Id: Id656efb176340ccf9def59f1547e3f6f4ddb79b3
2014-05-30 18:54:35 -07:00