diff --git a/doc/source/admin/index.rst b/doc/source/admin/index.rst index a35dba34d..970d20048 100644 --- a/doc/source/admin/index.rst +++ b/doc/source/admin/index.rst @@ -6,3 +6,4 @@ Congress Administrator Documentation :maxdepth: 2 deployment + jgress diff --git a/doc/source/admin/jgress.rst b/doc/source/admin/jgress.rst new file mode 100644 index 000000000..e8c9ccf61 --- /dev/null +++ b/doc/source/admin/jgress.rst @@ -0,0 +1,47 @@ +=================================== +JGress: policy over JSON data model +=================================== + + +Introduction +============ +JGress (JSON data model for Congress) is an experimental framework that allows +an operator to query, monitor, and automate clouds using JSON queries/views. +In short, Congress capabilities over a JSON data model. + +JGress is designed to capture the state of a cloud from controllers (Nova, +Neutron, EC2, etc) and alarms (Monasca, Vitrage, Zabbix, etc.) in the form of +a JSON database. The framework enables an operator to perform policy-based +monitoring using queries and views. The framework also allows an operator to +define remedial actions and workflows (Mistral, Heat, etc.) in response to +declarative conditions. + +JGress can be thought of as a kind of declarative "glue" layer that helps +different services work together while avoiding the N-squared problem with +1:1 integrations. + +For documentation on writing policy in this model, refer to +:ref:`JGress policy` +in user documentation. + + +Data sources +============ + +One key advantage offered by JGress is the ability for an administrator or +deployer to integrate arbitrary JSON API as a data source, with nothing more +than a simple YAML config file. Many sample config files are provided here: + +https://github.com/openstack/congress/tree/master/etc/sample_json_ingesters + + +Additional configuration +======================== + +The following config options are used by JGress: + +.. code-block:: text + + [json ingester] + + diff --git a/doc/source/install/index.rst b/doc/source/install/index.rst index 08e30efbc..f484eb91e 100644 --- a/doc/source/install/index.rst +++ b/doc/source/install/index.rst @@ -11,7 +11,9 @@ There are 2 ways to install Congress. * Separate install. Get Congress running alongside an existing OpenStack deployment -Devstack-install +.. _devstack_install: + +Devstack install -------------------- For integrating Congress with DevStack: @@ -34,6 +36,7 @@ For integrating Congress with DevStack: enable_plugin heat https://git.openstack.org/openstack/heat enable_plugin aodh https://git.openstack.org/openstack/aodh enable_service s-proxy s-object s-container s-account + # ENABLE_CONGRESS_JSON=True # uncomment to enable the jgress feature 3. Run ``stack.sh``. The default configuration expects the passwords to be 'password' without the quotes @@ -42,6 +45,12 @@ For integrating Congress with DevStack: $ ./stack.sh +Enablin Optional features +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Config validator datasource ++++++++++++++++++++++++++++ + If you want to use the config datasource in a multi-node environment, you must configure the Congress agent and only the agent on the other nodes. Here is the relevant part @@ -59,7 +68,16 @@ The ``ENABLE_CONGRESS_AGENT`` variable in ``local.conf`` controls the availability of the config datasource and its agent in devstack. Set it to ``False`` to disable it (default value is ``True``). -To enable the use of Z3 as an alternate Datalog engine, the +JGress - Congress JSON Ingester (experimental) +++++++++++++++++++++++++++++++++++++++++++++++ + +To enable the experimental JSON Ingester feature, set the +``ENABLE_CONGRESS_JSON`` variable to ``True`` in ``local.conf``. + +Z3 Engine (experimental) +++++++++++++++++++++++++ + +To enable the use of the experimental Z3 as an alternate Datalog engine, the ``ENABLE_CONGRESS_Z3`` variable must be set to ``True`` in ``local.conf``. You can use a pre-compiled release if your OS supports it (Ubuntu, Debian) by setting ``USE_Z3_RELEASE`` to the number of an existing release diff --git a/doc/source/user/index.rst b/doc/source/user/index.rst index 588b2c276..dfee45ced 100644 --- a/doc/source/user/index.rst +++ b/doc/source/user/index.rst @@ -11,6 +11,7 @@ Congress User Documentation enforcement policy-library api + jgress tutorial-tenant-sharing troubleshooting diff --git a/doc/source/user/jgress.rst b/doc/source/user/jgress.rst new file mode 100644 index 000000000..f5624b73f --- /dev/null +++ b/doc/source/user/jgress.rst @@ -0,0 +1,348 @@ +=================================== +JGress: policy over JSON data model +=================================== + + +Introduction +============ +JGress (JSON data model for Congress) is an experimental framework that allows +an operator to query, monitor, and automate clouds using JSON queries/views. +In short, Congress capabilities over a JSON data model. + +JGress is designed to capture the state of a cloud from controllers (Nova, +Neutron, EC2, etc) and alarms (Monasca, Vitrage, Zabbix, etc.) in the form of +a JSON database. The framework enables an operator to perform policy-based +monitoring using queries and views. The framework also allows an operator to +define remedial actions and workflows (Mistral, Heat, etc.) in response to +declarative conditions. + +JGress can be thought of as a kind of declarative "glue" layer that helps +different services work together while avoiding the N-squared problem with +1:1 integrations. + +.. _jgress_policy: + +Examples +======== +This brief introduction demonstrates the capabilities through a series of +simple examples. For reference on the data used in the examples, +please see the `data representation`_ section. + +Example 1: ad-hoc query +----------------------- +Suppose there has been some significant outage and many hosts are down. +Auto evacuation either was not configured or partially failed. An admin wants +to find out which hosts have the most affected VMs. + +The admin can use the following query to list the host by number of VMs +affected. + +.. code-block:: sql + + SELECT d ->> 'hostId' AS host_id, + Count(*) AS down + FROM _compute.servers + WHERE d ->> 'host_status' = 'DOWN' + GROUP BY host_id + ORDER BY down DESC + LIMIT 5; + +Note: +The ``->>`` operator accesses the content of a JSON object field and returns +the result as text. + +Example result: + +:: + + host_id | down + ---------+------ + host-13 | 11 + host-57 | 10 + host-08 | 10 + host-74 | 10 + host-22 | 9 + +Example 2: ad-hoc query +----------------------- +Continuing from the above example, perhaps the admin is interested only in the +production workloads. Then a simple addition to the filtering condition finds +the results. (Production workloads are indicated by tag: ``production`` in this +example, but can be replaced by filtering on any available metadata). + +.. code-block:: sql + + SELECT d->>'hostId' AS host_id, + Count(*) AS down + FROM _compute.servers + WHERE d->>'host_status' = 'DOWN' + AND d->'tags' ? 'production' + GROUP BY host_id + + ORDER BY down DESC + LIMIT 5; + +Note: +The ``->`` operator accesses the content of a JSON object field and returns the +result as JSON. In this example, ``d->'tags'`` returns the array of tags +associated with each server. +The ``?`` operator checks that a string is a top-level key/element in a JSON +structure. In this example, the ``d->'tags' ? 'production'`` condition checks +that the string ``'production'`` is in the array of tags retrieved by +``d->'tags'``. + +Example result: + +:: + + host_id | down + ---------+------ + host-75 | 4 + host-19 | 4 + host-39 | 4 + host-63 | 3 + host-27 | 3 + +Example 3: Monitoring +--------------------- +The system can also be used for ongoing monitoring to identify problematic +situations. For example, we may expect all critical workloads VMs are protected +by Masakari instance HA. We can monitor for any exceptions by defining the +following view for JGress or another tool to monitor. + +.. code-block:: sql + + CREATE VIEW critical_vm_ha.problem AS + SELECT d->>'id' AS server_id + FROM _compute.servers + WHERE d->'tags' ? 'critical' + AND NOT d->'metadata' @> '{"HA_Enabled": true}'; + +Note: +The ``@>`` operator checks that the left JSON value contain the right JSON +value. In this example, the ``d->'metadata' @> '{"HA_Enabled": true}'`` +condition checks that the metadata contains the ``'HA_Enabled'`` field and the +field is set to true. + +Example result: + +:: + + server_id + ------------ + server-536 + server-556 + server-939 + server-517 + server-811 + +Example 4: Remediation +---------------------- + +Going one step further, we can create a view that defines which APIs to call +(for example a Mistral workflow) to rectify the problem of the non-HA critical +VM. JGress monitors the view and makes the REST API calls as defined by the +view. + +.. code-block:: sql + + CREATE VIEW critical_vm_ha._exec_api AS + SELECT '_mistral' AS endpoint, + '/executions' AS path, + 'POST' AS method, + Format('{ + "workflow_name": "make_server_ha", + "params": {"server_id": "%s"}}', server_id) + AS body, + NULL AS parameters, + NULL AS headers, + FROM critical_vm_ha.problem; + +Note: +``_mistral`` is the name of the endpoint configured to accept requests for +API executions to OpenStack Mistral service. + +Example 5: combining multiple sources of data +--------------------------------------------- +Here’s a slightly more complex example to demonstrate that the queries can span +multiple cloud services. In this case, we want to identify the problematic +situation where production workloads use images considered unstable. The +following view accomplishes the goal by combining server information from Nova +and image information from Glance, then filtering on the tag information. + +.. code-block:: sql + + CREATE SCHEMA production_stable; + CREATE VIEW production_stable.problem AS + SELECT server.d->>'id' AS server_id, + image.d->>'id' AS image_id + FROM _compute.servers server + JOIN _image.images image + ON server.d->'image'->'id' = image.d->'id' + WHERE (server.d->'tags' ? 'production') + AND (image.d->'tags' ? 'unstable'); + +Note: see image document format in the `Glance API documentation +`_. + +Example result: + +:: + + server_id | image_id + --------------+----------- + server-386 | image-6 + server-508 | image-0 + server-972 | image-3 + server-746 | image-3 + server-999 | image-0 + +Example 6: using helper views +----------------------------- + +It’s not always convenient to write queries directly on the source data. For +example, a query to determine which servers have internet connectivity is +rather complex and would be cumbersome to repeat in every query requiring that +information. This is where helper views are useful. Suppose we have defined the +view internet_access.servers which is the subset of _compute.servers with +security group configuration that allows internet connectivity. Then we can use +the view to define the following view which identifies internet connected +servers running on an image not tagged as approved by the security team. + +.. code-block:: sql + + CREATE SCHEMA internet_security; + CREATE VIEW internet_security.problem AS + SELECT server.d->>'id' AS server_id, + image.d->>'id' AS image_id + FROM internet_access.servers server + JOIN _image.images image + ON server.d->'image'->'id' = image.d->'id' + WHERE NOT image.d->'tags' ? 'security-team-approved'; + +Note: see image document format in the `Glance API documentation +`_. + +Example result: + +:: + + server_id | image_id + --------------+----------- + server-705 | image-1 + server-264 | image-0 + server-811 | image-0 + server-224 | image-4 + server-508 | image-0 + +Example 7: using webhook alarm notification +------------------------------------------- +Some cloud services supports sending webhook notifications. For example, the +Monasca monitoring service can be configured to send webhook notifications of +alarm updates. JGress can be configured to received these webhook notifications +and insert the new data into the database (replacing old entries as needed). +Here is an example of using such webhook notification data from Monasca to flag +critical workloads running on hypervisors where the CPU load is too high for +comfort. + +.. code-block:: sql + + CREATE VIEW host_cpu_high.problem AS + SELECT server.d->>'id' AS server_id, + server.d->>'hostId' AS host_id + FROM _compute.servers server + JOIN _monasca.webhook_alarms alarm + ON server.d->'OS-EXT-SRV-ATTR:hypervisor_hostname' IN + (SELECT value->'dimensions'->'hostname' + FROM Jsonb_array_elements(alarm.d->'metrics')) + WHERE alarm.d->>'alarm_name' = 'high_cpu_load' + AND alarm.d->>'state' = 'ALARM'; + + +Data representation +=================== +The examples above use OpenStack Nova (compute service) +`API response data `_ on +servers, stored in a PostgreSQL +`JSONB `_ column. + +Each server is represented as a JSON document as provided by the Nova API. + +Sample server data (simplified): + +.. code-block:: json + + { + "id":"server-134", + "name":"server 134", + "status":"ACTIVE", + "tags":[ + "production", + "critical" + ], + "hostId":"host-05", + "host_status":"ACTIVE", + "metadata":{ + "HA_Enabled":false + }, + "tenant_id":"tenant-52", + "user_id":"user-830" + } + +The ``_compute.servers`` table is the collection of all the JSON documents +representing servers, each document in a row with a column ``d`` containing the +document. + +The content of each JSON document is accessible using the JSON operators +provided by PostgreSQL. + +Additional data sources are available, each in the original format of the +source JSON API representation. To see all data sources available, use the +following ``\dn`` command in a ``psql`` console to list the schemas. + + +Sample policies +=============== + +Additional sample policies can be found here: + +https://github.com/openstack/congress/tree/master/doc/source/user/jgress_sample_policies + +Each policy can be imported using: + +.. code-block:: console + + $ psql -f + + +Connecting to PostgreSQL +======================== +To interact with JGress data, connect to the JGress PostgreSQL database using +any compatible client. Examples include the command-line client ``psql`` +and the browser-based client `pgweb `_. + +For example, here is how to connect using psql: + +.. code-block:: console + + $ psql postgresql://:@/ + +If on the controller node of a :ref:`devstack installation `, +the default values are as follows: + +.. code-block:: console + + $ psql postgresql://jgress_user:@127.0.0.1/congress_json + + +Language reference +================== + +* `PostgreSQL JSON syntax cheat sheet + `_ + +* `PostgreSQL JSON operators and functions + `_ + +* `PostgreSQL language reference + `_ diff --git a/doc/source/user/jgress_sample_policies/forever_password.sql b/doc/source/user/jgress_sample_policies/forever_password.sql new file mode 100644 index 000000000..16eb1cec7 --- /dev/null +++ b/doc/source/user/jgress_sample_policies/forever_password.sql @@ -0,0 +1,13 @@ +-- policy for monitoring users with forever passwords + +CREATE SCHEMA forever_password; + +-- `warning` view consisting of enabled users with forever passwords +CREATE VIEW forever_password.warning AS +SELECT d->>'id' AS user_id, + d->>'name' AS user_name, + d->>'domain_id' AS domain +FROM _identity.users +WHERE (d @> '{"password_expires_at": null}'::jsonb + OR NOT d ? 'password_expires_at') +AND d @> '{"enabled": true}'::jsonb; diff --git a/doc/source/user/jgress_sample_policies/unencrypted_volume.sql b/doc/source/user/jgress_sample_policies/unencrypted_volume.sql new file mode 100644 index 000000000..ac80ead42 --- /dev/null +++ b/doc/source/user/jgress_sample_policies/unencrypted_volume.sql @@ -0,0 +1,10 @@ +-- policy for warning unencryption volumes +CREATE SCHEMA unencrypted_volume; + +-- `warning` view consisting of unencrypted volumes in use (attached) +CREATE VIEW unencrypted_volume.warning AS +SELECT d->>'id' AS volume_id, + d->'attachements' AS attachements +FROM _volume.volumes +WHERE NOT d @> '{"encrypted": true}'::jsonb +AND jsonb_array_length(d->'attachments') != 0; diff --git a/doc/source/user/jgress_sample_policies/vm_ha.sql b/doc/source/user/jgress_sample_policies/vm_ha.sql new file mode 100644 index 000000000..e939ea72a --- /dev/null +++ b/doc/source/user/jgress_sample_policies/vm_ha.sql @@ -0,0 +1,17 @@ +-- policy for monitoring which VMs do not have instance HA configured +CREATE SCHEMA vm_ha; + +-- ignore a VM if it has one of these tags +CREATE TABLE vm_ha._except_tag AS +SELECT * FROM (VALUES ('transient'), ('dev'), ('test')) AS t (tag); + +-- wrap the table in a view to allow easier extension if the list +-- of tags should be a query rather than a base table. +CREATE VIEW vm_ha.except_tag AS SELECT * FROM vm_ha._except_tag; + +-- `warning` view consisting of VMs which do not have HA configured +CREATE VIEW vm_ha.warning AS +SELECT d->>'id' AS server_id +FROM _compute.servers +WHERE NOT d->'tags' ?| (SELECT array_agg(tag) FROM vm_ha.except_tag) +AND NOT d->'metadata' @> '{"HA_Enabled": true}'; diff --git a/releasenotes/notes/jgress-0df6bb2a661b5870.yaml b/releasenotes/notes/jgress-0df6bb2a661b5870.yaml new file mode 100644 index 000000000..1dfb092bf --- /dev/null +++ b/releasenotes/notes/jgress-0df6bb2a661b5870.yaml @@ -0,0 +1,9 @@ +--- +prelude: > +features: + - | + New experimental feature JGress provides a framework for policy over the + JSON API data state of the cloud. This framework enables a deployer to + integrate new data sources without any code or translation driver. + For more details see the relevant documentation: + https://docs.openstack.org/congress/stein/user/jgress.html diff --git a/releasenotes/notes/add-tacker-datasource-driver-43054caac433c800.yaml b/releasenotes/notes/tacker-datasource-driver-43054caac433c800.yaml similarity index 100% rename from releasenotes/notes/add-tacker-datasource-driver-43054caac433c800.yaml rename to releasenotes/notes/tacker-datasource-driver-43054caac433c800.yaml diff --git a/releasenotes/notes/add-upgrade-checkers-178e98db21109e4f.yaml b/releasenotes/notes/upgrade-checkers-178e98db21109e4f.yaml similarity index 88% rename from releasenotes/notes/add-upgrade-checkers-178e98db21109e4f.yaml rename to releasenotes/notes/upgrade-checkers-178e98db21109e4f.yaml index 54186ef68..62792f07e 100644 --- a/releasenotes/notes/add-upgrade-checkers-178e98db21109e4f.yaml +++ b/releasenotes/notes/upgrade-checkers-178e98db21109e4f.yaml @@ -1,6 +1,5 @@ --- prelude: > - Added new tool ``congress-status upgrade check``. features: - | New framework for ``congress-status upgrade check`` command is added.