Commit Graph

39 Commits

Author SHA1 Message Date
Monsyne Dragon ec95dd9790 Add JSON format for AtomPub Handler
Add JSON content format to AtomPub pipeline handler.
Also add optional args to specify title and categories
of atom post.

Change-Id: I67c133fb61b2a5248090ff562aad07a436285a8f
2015-08-26 21:45:33 +00:00
Monsyne Dragon b5191bb105 Add AtomPub handler
Adds AtomPub handler to send events to an atom server as a feed.

Also, fixed Usage driver to save new events directly from pipeline.
Fixed minor (but annoyingly hard to diagnose) bug in db layer
    where it didn't recognize 'long' values.

Change-Id: I792e9f77accfea4583fd75805a9ff0d946827df8
2015-08-19 06:52:04 +00:00
Monsyne Dragon 665c9ad328 Add optional database trimming for old events.
Alows you to trim events older than a configurable time from
the events database.

(Remerging to fix timex parse error)

Change-Id: Iaa290705815d1c3ac23c2ca7370a5d705f1f834c
2015-08-17 17:51:05 +00:00
Monsyne Dragon a15b95dbf3 Catch exceptions and reconnect if MySQL goes away.
Catch exception from the db layer and reconnect if
MySQL connection goes away.

Change-Id: Ib931100f4ee6ed78f0839d7ad0ed82bd277c23c0
2015-06-25 17:26:11 +00:00
Jenkins b6dc37aad3 Merge "Add or Delete Trigger Definitions to an existing TriggerManager and PipelineManager" 2015-05-13 20:15:20 +00:00
Sandy Walsh f63e71dba8 Use compute.instance.finish_resize.end for usage comparison.
Vs. compute.instance.prep.resize.end as before.

Change-Id: I837329014a4f7e7bfee8f1943bbdb4597c1da855
2015-05-06 08:24:14 -07:00
cindy oneill 0420542cfd Add or Delete Trigger Definitions to an existing TriggerManager and PipelineManager
Change-Id: I0dd97b140d1e175492de507eb1ebaf99d18c5eae
2015-05-04 18:32:55 -06:00
Levi Blackstone 8892801205 Add PEP8 check and fix related issues
- Add PEP8 section to tox.ini
- Add hacking to requirements to enforce OpenStack style requirements
- Fix large number of formatting issues flagged by flake8 check
- Add copyright notices to all remaining files
- Fix bug in trigger_manager related to logging calls
- Add .gitignore file

Change-Id: I755ab9c8bcc436836f9006fcd671408cc77214c4
2015-05-04 10:33:25 -05:00
Sandy Walsh a2710a98d9 Only consider the last interesting event when checking traits.
Was previously looking at the last .start event when
comparing important traits against the .exists event.
Now it will look at the last important .end event it found (if any).

For delete operations, it will be the last event in the chain.
This may need to change to a specific event as well, like
compute.instance.delete.end or something.

Change-Id: Ie85e69716cd730c3546d06bae28e52e0c301940f
2015-04-17 07:39:44 -07:00
Monsyne Dragon 775256d6b1 Fix default values for usage verified event.
Fix the default values in the compute.instance.verified notification
generated by the UsageHandler. Bandwidth values should be int (0),
not empty string.

Options value should also be '0', not empty string.

Change-Id: I50dc660a46adcd8354e379821e0077cc475e647b
2015-04-08 20:34:30 +00:00
Monsyne Dragon edca8bd8b1 Out darned typoes.
Out I say!

(Fix typo in pipeline_worker daemon)

Change-Id: I9c04044ce2583649306d7ccf42b954ddd01e3672
2015-04-03 20:54:02 +00:00
Monsyne Dragon 08d43dcdd7 Fix typo in pipeline worker daemon
log_file, not config_file.

Change-Id: I1fb6f6c8ac4decbd20816d60a365f7d9a66c8f4c
2015-04-03 19:59:47 +00:00
Monsyne Dragon 0a2bf4471e Improve logging.
Allow pipeline workers to log to different logfiles
(the python logging system is not entirely multiprocess safe)
Allow users to set a logging name for each worker process
to differentiate logs.

Change-Id: Ic93e88270d58495cda918dab503f65d9f550e2b6
2015-04-03 17:50:39 +00:00
Sandy Walsh 4cb16327c7 NotabeneHandler no longer formats notifications.
It just passes it through verbatim. Only required key is 'event_type'.

Change-Id: Idcd911c9f09d405d57ecaf5b14b72bfebe2a179a
2015-03-25 12:45:04 -07:00
Eddie Sheffield e08924f1f8 Add functionality for counting events
Similar to the existing streams functionality, this patch adds
the ability to get just a count of filtered events rather than
the full list of events.

There are corresponding changes in quince, quincy, and klugman.

Change-Id: Ic14845db48170a8481c764f5f2468f137758bd19
2015-03-20 12:42:00 -04:00
Jenkins 8b653622a7 Merge "Stringify .verified dates before json conversion." 2015-03-12 15:51:53 +00:00
Sandy Walsh 20a2458e24 Stringify .verified dates before json conversion.
Kind of a hack right now, need to decide where json
conversion should ultimately occur ... here, or in
notabene. Until then, we'll just stringify the
offending datatypes here.

Change-Id: Ia1e9e867d0da78b376a66005b1180fad307e42d2
2015-03-07 20:54:19 -04:00
Sandy Walsh 7846f6a20f Add support for find_events()
For use with API. You can find events based on traits, event_type or dates.

Doesn't support wildcarding or anything just yet.

Change-Id: Ide6e13086d50bacf356d8e36b9de257db9d543ca
2015-03-04 09:22:11 -08:00
Sandy Walsh 4875e419a6 Fixes the .verified notification format.
Now it matches the notification format expected by the yagi
CUF handler. Requires expanded event_definition.yaml file
to include extra traits from .exists

Change-Id: I83ba05d73890a578699566648b37697d46b58040
2015-02-27 06:05:41 -08:00
Jenkins 516c2b3d75 Merge "Notabene pipeline handler" 2015-02-10 22:29:26 +00:00
Monsyne Dragon 8044f3e7e2 Add mark/limit paging to find call.
Adds Mark/limit to find_streams for api usage.

The 'mark' is an opaque parameter (called '_mark') added into
each item in the list of returned items (currently a mildly obscured
ID for streams, that could be changed.)

Returned items are sorted in descending order (by ID for streams),
allowing paging from the 'top' to the bottom.
By prefacing the 'mark' param with a '+', you can page in ascending
order as well.

This same paging scheme can be used for a future events api, as well.

Change-Id: Ie358c0d1d8644dac24cad3992188ec22d66c5328
2015-02-10 19:29:07 +00:00
Sandy Walsh ce91d56d51 Notabene pipeline handler
For publishing new events into an exchange.

PipelineHandlers can return new Events for subsequent processing.
However, sometimes we need to publish Notifications. Remember that
Notifications are less restrictive than Events. They can be larger.
They can be nested. They can contain lists.

We store these pending notifications in the handler env variable
so downstream handlers can access them.

The NotabeneHandler takes a configuration variable 'env_keys' which
specifies the env keys to look for. Lists of notifications in these
variables are published to the queue using the connection parameters
supplied.

Errors in transmission are logged and ignored since exceptions during
the commit() phase do not flag the steam as in error.

A sample pipeline definition might look like this:

test_expire_pipeline:
    - logger
    - usage
    - name: notabene
      params:⋅
        host: localhost
        user: guest
        password: guest
        port: 5672
        vhost: /
        library: librabbitmq
        exchange: nova
        exchange_type: topic
        queue_name: monitor.info
        env_keys:
            - usage_notifications

Change-Id: If1958135ad6fbed88e2c18b9fac7efde51ee3113
2015-02-10 07:29:35 -08:00
Sandy Walsh 460f8231ee Add count() support to find_streams
Returns [{'count': num_streams}]

Change-Id: I37e7fc462a14dda793ac5c231439ed7d5e5ee523
2015-01-27 12:02:44 -08:00
Sandy Walsh c5f0eca828 Generate .verified per .exists
Slight change on the old way, which was a single .verified on the
one EOD .exists. Now it generates a .verified for every .exists
seen.

Change-Id: I7bdced936f203cbec4bf8b7ee2b956c388e66ff1
2015-01-19 20:01:42 -08:00
Sandy Walsh 10b21a4e74 Non-EOD .exists usage checks.
Also adds support for warnings. Will generate a
compute.instance.exists.warnings event when there
is something suspicious but not an actual error.

Change-Id: I00f135b20917231a8cf3f603f0954f061de15410
2014-12-12 11:19:50 -08:00
Jenkins 78e790c266 Merge "Added support for basic stream API." 2014-12-03 19:15:58 +00:00
Monsyne Dragon c5ae51e5ed Added support for basic stream API.
added find methods for basic stream get/search and reset operation.
also, added gerrit .gitreview

Change-Id: I79e11730336f6200be2143f1943a9bf2dbabda7a
2014-12-02 16:11:56 +00:00
Sandy Walsh 657e358e0e Clean up some verifier messages.
To make things a little more clear.
Also unifies the flavor_id to instance_flavor_id

Change-Id: I43a98965eb6b73f2333fc1f143d8f4f0dac702aa
2014-12-01 07:44:01 -08:00
Sandy Walsh e0256a3fdb Support for playback of old events.
Introduces the time_sync object which can sync
time with an external time service (like the time_sync
service in notigen).

This is used for playback of old events. It prevents
premature expiry triggers.

Also some tweaks to the UsageHandler, work in progress
and used for testing.

Change-Id: I45033fbd9c12d98f96816a4f90cf7dc8c915ef51
2014-10-17 12:40:44 -07:00
Sandy Walsh ac49955386 Adding debugger from oahu
Allows for per-trigger-definition debugging. Simply add
debug_level=1 to your trigger definition.

debug_level=2 gives detailed explainations of why your
stream didn't fire or trigger.

This way, you can leave the overall debug level at INFO
and still get details on the stream you're working on.

Yagi worker now uses idle() callback for debug updates.

and minor cleanups for pep8/HACKING

Change-Id: Id0af7a0adbcc47335ad250130958932c708b5c18
2014-10-02 20:50:31 +00:00
Monsyne Dragon 5e2a29610e Fix config bug.
Fixes issue with the trigger manager loaded by the pipeline worker
not getting the proper default value for distiller_trait_plugins.
2014-09-16 16:15:36 +00:00
Monsyne Dragon db434c24c4 Add ability to purge streams on completion.
default to deleting streams once finished with them (if no error)
2014-09-10 21:43:45 +00:00
Monsyne Dragon 2d47fa6f6e Add reset stream method. 2014-09-08 23:02:52 +00:00
Monsyne Dragon ca0d09f7bc Save newly generated events from pipeline
Save newly created events from pipeline run if pipeline commits.
Refactor trigger manager api wart, move save_event call into add_event
    to make add_event and add_notification symmetric.
2014-09-08 19:57:24 +00:00
Monsyne Dragon 0c619c133d Add database admin command.
Add admin command for db schema upgrade/downgrade/etc.
Move alembic migrations so above can find them when installed
    as a package.
Fix up packaging to use setup.cfg and pbr.
Flesh out README.
2014-09-07 04:07:20 +00:00
Monsyne Dragon a6f84d1603 Fixed a few bugs, added more logging.
Fixed timestamp bug, and streamstate issue missed in unittests.
Added more logging for pipeline manager.
2014-09-04 20:43:41 +00:00
Monsyne Dragon c2aa498beb Move yagi handler into winchester codebase. 2014-09-04 18:05:19 +00:00
Monsyne Dragon a8f373e4bf Added full stream processing, pipeline workers, etc.
Full trigger logic now works.
Added pipeline workers, and test handler.
Added example configs
Lots of unittests.
2014-09-04 01:49:19 +00:00
Monsyne Dragon aa8fb55e87 Initial commit of DB schema.
Initial commit of the event schema for the database.
This includes models and alembic migration.
2014-06-26 01:55:26 +00:00