Commit Graph

1445 Commits

Author SHA1 Message Date
Nate Johnston bba805af02 Retire the Congress project
Recently the TC has worked on determining the criteria for when an
OpenStack project should be retired.  When there was not a PTL nominee
for the Congress project, that triggered the TC to review the project
health per [1], and the TC has determined [2] that development work on
the project has ceased.  This decision was announced in the
openstack-discuss mailing list in April 2020 [3].

This commit retires the repository per the process for governance
removal in the Victoria cycle as specified in the Mandatory Repository
Retirement resolution [4] and detailed in the infra manual [5].

Should interest in developing Congress as part of OpenStack revive,
please revert this commit to have the project rejoin the list of active
projects.

The community wishes to express our thanks and appreciation to all of
those who have contributed to the Congress project over the years.

[1] https://governance.openstack.org/tc/reference/dropping-projects.html
[2] http://eavesdrop.openstack.org/irclogs/%23openstack-tc/latest.log.html#t2020-04-20T15:36:59
[3] http://lists.openstack.org/pipermail/openstack-discuss/2020-April/014292.html
[4] https://governance.openstack.org/tc/resolutions/20190711-mandatory-repository-retirement.html
[5] https://docs.opendev.org/opendev/infra-manual/latest/drivers.html#retiring-a-project

Change-Id: I21c9ab9820f78cf76adf11c5f0591c60f76372a8
2020-05-28 18:12:16 +02:00
Eric K 5d8d0e7c7e update devstack plugin for ipv6
Set listen address to devstack variable SERVICE_LISTEN_ADDRESS
to override the default 0.0.0.0 (which does not listen to ipv6 addresses).

Change-Id: I1bbc3539a751f7c0c0f54ec046f88a847b4647f9
Story: #2005477
2019-08-07 16:37:15 -07:00
Zuul 19a6df32e6 Merge "Changes for url transition to OpenDev" 2019-05-17 05:49:29 +00:00
Zuul 7ca9a2be1e Merge "Resolve instability in json_ingester test_execute_exec_api_rows" 2019-05-13 05:55:52 +00:00
Eric K 593174028c Lower psycopg2 dependency to >=2.5.1
Because 2.5.1 is the version available on RHEL/CentOS 7.
This is done by replacing the use of psycopg2.sql module.

Change-Id: Ie83ca2b3cd439649878517fa0fbc2a4381ad018d
Closes-Bug: 1823095
2019-05-08 22:03:18 +00:00
akhiljain23 ea82c5d0f7 Changes for url transition to OpenDev
This commit changes openstack.org URL to opendev.org URL

Change-Id: Ic05ddf275a9f71cd3382ee3b4580a6004a1e66fe
2019-04-22 12:30:30 +05:30
Eric K 31e7e04294 Resolve instability in json_ingester test_execute_exec_api_rows
Occassionally the unit test congress.tests.datasources.json_ingester.
test_exec_api.TestExecApiManager.test_execute_exec_api_rows
fails inside eventlet code even though the eventlet call is mocked
out.

This patch adds a new mock path because eventlet.spawn_n
somehow redirects there. Hopefully it resolves the occassional
instability gets.

Sample error:
  File "/home/zuul/src/git.openstack.org/openstack/congress/.tox/py27/local/lib/python2.7/site-packages/eventlet/greenpool.py", line 115, in spawn_n
    function, args, kwargs, True)
  File "/home/zuul/src/git.openstack.org/openstack/congress/.tox/py27/local/lib/python2.7/site-packages/mock/mock.py", line 1062, in __call__
    return _mock_self._mock_call(*args, **kwargs)
  File "/home/zuul/src/git.openstack.org/openstack/congress/.tox/py27/local/lib/python2.7/site-packages/mock/mock.py", line 1128, in _mock_call
    ret_val = effect(*args, **kwargs)
  File "congress/tests/datasources/json_ingester/test_exec_api.py", line 27, in mock_spawn_execute
    return func(*args, **kwargs)
  File "/home/zuul/src/git.openstack.org/openstack/congress/.tox/py27/local/lib/python2.7/site-packages/eventlet/greenpool.py", line 99, in _spawn_n_impl
    self._spawn_done(coro)
  File "/home/zuul/src/git.openstack.org/openstack/congress/.tox/py27/local/lib/python2.7/site-packages/eventlet/greenpool.py", line 131, in _spawn_done
    self.coroutines_running.remove(coro)
KeyError: <greenlet.greenlet object at 0x7fd975681cd0>

Change-Id: I72e11a67cf1cd3ff3fbc574f2d6e91df17d63d9c
2019-04-05 17:51:59 -07:00
Eric K 18241b3548 fix incorrect sql compose for index create
Change-Id: Ibe166cad630f5eb586a8e8756981fded10eecf33
Closes-Bug: 1823128
2019-04-04 00:38:08 -07:00
Zuul f5abdf741c Merge "Completes upgrade check for monasca webhook data source" 2019-03-21 18:43:35 +00:00
Zuul 9fa116e1e6 Merge "Return 503 rather than 500 error when _ds_manager unreachable" 2019-03-20 21:44:11 +00:00
Zuul c1a9d2a095 Merge "Replace fixed endpoint config by dynamic in devstack plugin" 2019-03-19 19:30:20 +00:00
Zuul 2836812e82 Merge "Fix redundant index, unindexable, wrong isolation in JSON ingester" 2019-03-19 19:21:05 +00:00
Zuul f00f31cb2c Merge "Fix inconsistent OptionJsonEncoder behavior on unknown type" 2019-03-19 19:21:04 +00:00
Eric K ea1fb9c3ed Replace fixed endpoint config by dynamic in devstack plugin
Some of the fixed endpoint configs caused errors.
Magnum and Zun explicitly call for grabbing the API endpoint from
catalog. Switch all the JSON ingester configs to use API endpoint
from service catalog to avoid incorrect config.

Change-Id: I50d2755317f68928f41d3770e25dfe67ba088969
Closes-bug: 1819556
2019-03-15 13:22:39 -07:00
Eric K a1dfbdec00 Fix redundant index, unindexable, wrong isolation in JSON ingester
Fixes a bug where json ingester repeatedly creates redundant indexes
each time Congress restarts.

Mitigates a problem where indexing fails when a field exceeds the
postgres indexable limit of 2712 bytes. Indexing is made optional.

Fixes a bug where unexpected behavior occurs when the backend DB
default isolation level is not the expected read_committed. The
isolation level is now set explicitly per connection.

Change-Id: I514aa9b96e4efbffe8880cce775dc2259eca4648
Closes-bug: 1819987
Closes-bug: 1819988
Closes-bug: 1819985
2019-03-14 17:18:58 -07:00
Zuul 1f7fa14032 Merge "Use YAML safe_load_all" 2019-03-15 00:03:52 +00:00
Eric K c15f2dd04f Completes upgrade check for monasca webhook data source
Results in warning (schema change) if a monasca webhook data source
is configured.

Change-Id: I7ccb2f2804f3957bbeb84a700583ae666f2dd0af
2019-03-13 19:15:40 -07:00
Eric K 794e5f3719 Use YAML safe_load
The arbitrarily powerful YAML load is not necessary.

Change-Id: I8bc27a2b0416191638f3ece4dd0d1b3aae8c63cc
Partial-Bug: 1819211
2019-03-13 10:55:10 -07:00
Eric K 4752a6fdcf Use YAML safe_load_all
The arbitrarily powerful YAML load_all is not necessary.

Change-Id: I3db0e10d0317ab3bd96a4480308b5661da4691c9
Closes-Bug: 1819211
2019-03-13 17:52:35 +00:00
Eric K 18e3060af5 Return 503 rather than 500 error when _ds_manager unreachable
It can happen that the _ds_manager is temporarily unreachable
due service not ready or network issues. In these cases,
the patch makes sure the API returns 503 http status code
instead of 500, so that the client knows the issue is
temporary.

Change-Id: I8a03a2e0ac84c2395b5ea2015647c5c7d2fed03e
Closes-bug: 1811754
2019-03-12 15:14:43 -07:00
Eric Kao 9abf99939b Fix inconsistent OptionJsonEncoder behavior on unknown type
Update OptionJsonEncoder to encode an unknown type as the expected
{'type': 'type_name'} rather than as 'type_name', fixing an error
encountered by related code that expects to index into the generated
dict construct.

Also updated congress agent tolerate unknown (custom) types instead
of exiting completely.

This reverts commit 69848285c8.

Change-Id: Ie18967b1b4191c43e00817d7661cc4b5efd247be
Closes-Bug: 1819248
2019-03-12 21:07:07 +00:00
Zuul fa524f86bd Merge "Allow port in JSON ingester config" 2019-03-07 13:26:25 +00:00
Zuul 2a1848b930 Merge "Add created_at field for nova servers table" 2019-03-07 09:05:48 +00:00
Zuul b1da280cb0 Merge "Support making unauthenticated APIs requests by JSON ingester" 2019-03-07 08:01:18 +00:00
Eric Kao 7676be3cb6 Allow port in JSON ingester config
Allow config to specify a port to use in making the API calls to
ingest JSON.

Also added additional sample configs: neutron, keystone, monasca.

Change-Id: I7d852742d0943c3857ee4e04d5a0d77b6bafeed7
partially-implements: bp json-data-model
2019-03-06 23:15:29 -08:00
Eric Kao eb8520d684 Support making unauthenticated APIs requests by JSON ingester
Change-Id: I90ee25aea04559af720f573ff065da0c2b4b20e0
partially-implements: bp json-data-model
2019-03-07 05:11:01 +00:00
Zuul b0163a8d9d Merge "Add API header config to JSON ingester" 2019-03-07 03:27:09 +00:00
Zuul 28cd60ff62 Merge "JSON ingester deployment enhancements" 2019-03-07 03:22:08 +00:00
akhiljain23 55ce6f195e Add created_at field for nova servers table
Depends-On: https://review.openstack.org/#/c/640248/
Change-Id: Ic402a13c9f864a124bba194b9985cd4b69da6bee
2019-03-07 00:09:20 +00:00
Eric Kao 86a778ca59 Add API header config to JSON ingester
Add the 'api_default_headers' field which specifies a hash/dict
of headers to pass with each API call w.r.t. that JSON ingester.
Main usage is to specify API microversion. For example:

api_default_headers:
  X-OpenStack-Nova-API-Version: "2.26"

Change-Id: Ie859e45ea03135aa21196fe19767f28627e76c3b
partially-implements: bp json-data-model
2019-03-06 11:33:17 -08:00
Zuul ec39b5f7a8 Merge "Mock rather than modify DataSourceDriver.TRANSLATORS in unit tests" 2019-03-06 19:20:07 +00:00
Eric Kao 5da6e463ff JSON ingester deployment enhancements
Allows json ingester config YAMLs to use the !ref tag to reference
previously defined reusable structures, which makes deployment much
more convenient.

Allows devstack plugin to optionally enable JSON ingester feature.

Renamed and reorganized the JSON ingester config options.

Adds CI job which sets up JSON ingester.

partially-implements: bp json-data-model
Change-Id: I6391224c99249d16fe943b8f00fe12d1e6b7d8e6
2019-03-06 08:35:23 -08:00
Eric Kao 52e5c23485 JSON ingester API execution feature
Operator can define an _exec_api view/table in any schema, including
the columns: endpoint, path, method, body, parameters, headers
(body, parameters, headers may be null).

Each time JSON ingester updates state, Congress will make all the
API requests as specified in these _exec_api views/tables.

The existing action execution semantics is used: execute any new
actions not present as of the last time the execution tables are
checked.

Each JsonIngester instance now keeps a reference to the shared
ExecApiManager instance. Each time a JsonIngester instance updates
state, it tells the ExecApiManager instance to check and execute API
requests.

partially-implements: bp json-data-model
Change-Id: I548b4d097452b2f001efc054780be29eb3a73398
2019-03-04 10:25:15 -08:00
Zuul 099245d369 Merge "JSON ingester experimental feature flag" 2019-02-28 20:26:29 +00:00
Zuul 2df45fffbd Merge "Add tacker datasource driver" 2019-02-28 04:56:34 +00:00
Zuul 5ea52088ed Merge "Generic webhook processer for JsonIngester" 2019-02-28 04:01:19 +00:00
Zuul 40eb118430 Merge "PollingJsonIngester differential update to DB" 2019-02-28 04:01:18 +00:00
Zuul 6d30a16f13 Merge "basic polling JSON ingester" 2019-02-28 04:01:17 +00:00
akhiljain23 fd495c6e2f Add tacker datasource driver
This commit adds Tacker driver and unit tests. Which creates
corresponding datasource on service startup and allows congress
to maintain Tacker's VNF and VNFD data via populating data to tables
vnfs and vnfds respectively. Based on which rules
can be created and actions can be executed on tacker service.

Change-Id: I281e8ff508dd066514694419c5aae6a991ab307c
Partially-Implements: blueprint add-tacker-datasource-driver
2019-02-28 07:27:06 +05:30
Eric Kao 3a704427fd JSON ingester experimental feature flag
Add an experimental feature flag to make sure the experimental
JSON ingester code do not affect normal Congress operations
unless BOTH the experimental flag is set in congress config
AND the JSON ingester YAML config files are present.

Change-Id: I8e3822461fa41e62511b485b807a0df27981609d
partially-implements: bp json-data-model
2019-02-19 18:31:30 -08:00
Eric K 6f01dfafc0 Generic webhook processer for JsonIngester
Each table accepting webhook is configured with a jsonpath expression
to grab the unique ID from the webhook payload.

Each time a webhook comes in, the new data is written to the Postgres
database in jsonb format, replacing any previous webhook data with
the same ID.

The _key column type is changed from bigint to text in order to
accommodate custom keys in generic webhook processing.

PollingJsonIngester is renamed to JsonIngester because it is no
longer specialized to polling.

config format is changed slightly to clearly define which tables
are configured for polling and/or webhook.

A table that is updated through both poll and webhook is not
supported by this patch, but may be supported in the future.

Change-Id: I23c3d5c2c63fd8471868c2b37fd154eb2d1f9783
partially-implements: bp json-data-model
2019-02-14 14:58:50 -08:00
Zuul bd56734f83 Merge "move config init from congress_servers.py to bin/congress-server" 2019-02-08 05:45:02 +00:00
Zuul 3dc1c8a8db Merge "Add required checks in all cases of policy create API" 2019-02-07 23:59:07 +00:00
Eric K 3b5b6b92a8 move config init from congress_servers.py to bin/congress-server
The need to execute config init before importing harness in
congress_servers.py causes a style violation in hacking 1.1.0.

Here we move the initialization in bin/congress-server because it's
the first place that sees the command line options.

It also allows the work on upping hacking to 1.1.0 to go ahead.

Change-Id: I7fbc12195f5d79d7df8da3bb835f3f89c1244efc
2019-02-04 17:35:35 -08:00
Zuul e5d979a3c7 Merge "Add note clarifying a confusing method call" 2019-01-30 23:39:35 +00:00
Eric K 6419ef18d0 Mock rather than modify DataSourceDriver.TRANSLATORS in unit tests
Direct modification of DataSourceDriver.TRANSLATORS constant for unit testing
causes some unit tests to affect one another. So we instead use contextual
mocking.

Change-Id: I1bea6fd68921410724c8427167966752dbb6e947
2019-01-28 17:22:53 -08:00
Eric K 3ce621e711 PollingJsonIngester differential update to DB
Instead of rewriting the entire DB table on each poll, update the
table by adding new records and deleting records no longer
needed.
Because the change is usually much smaller than the entire table,
ingest performance should be much improved, allowing the ingested
state to be more current.
A _key column is added to each table of json records to facilitate
performant deletion of specific records without linear scan.

Also includes small improvements and variable renames.

Change-Id: I3ea4f0325c34d9eef38d7ee64f0bca60e9241620
partially-implements: bp json-data-model
2019-01-28 14:58:23 -08:00
Eric Kao 28b804a48c basic polling JSON ingester
New PollingJsonIngester class which polls API end points as defined
in YAML configs and populates configured postgreSQL database with
JSON data.

Change-Id: I8d22cf8d2da1b1c2e99b8e0cd2d60f3ef71e2def
partially-implements: bp json-data-model
2019-01-09 18:54:38 -08:00
Zuul 21604e9834 Merge "Avoid coreference between current state and _last_published_data" 2019-01-07 20:12:27 +00:00
Eric K 5b9ae41024 Avoid coreference between current state and _last_published_data
Previously, all data source drivers replace the entire state on each
poll or push update (doctor). The DSE publish code relied on this
situation to optimize performance by skipping a copy when setting
_last_published_data. The coreference introduced did not matter
because the original state was never edited, only replaced.

But beginning with the webhook drivers
(monasca webhook and vitrage), the drivers edit the current state.
This a bug was introduced where sometimes an update would not be
published to messaging bus because of an incorrect _last_published_data
due to coreference.

This patch fixes the bug. In addition, the _last_published_data
elements are stored as set rather than list to save the need to
convert to set each time a diff is needed. The times when
conversion back to list is needed to publish on the messaging bus
(used for snapshot publish only)
is expected to be far fewer than the times the conversion to set is
needed.

A few additionalogging lines are also introbuced for improved
debuggability.

Change-Id: I332a844beaf8dec4450bb1ade6ce7e41cc3d4d91
2018-12-18 12:07:01 -08:00