Set listen address to devstack variable SERVICE_LISTEN_ADDRESS
to override the default 0.0.0.0 (which does not listen to ipv6 addresses).
Change-Id: I1bbc3539a751f7c0c0f54ec046f88a847b4647f9
Story: #2005477
Because 2.5.1 is the version available on RHEL/CentOS 7.
This is done by replacing the use of psycopg2.sql module.
Change-Id: Ie83ca2b3cd439649878517fa0fbc2a4381ad018d
Closes-Bug: 1823095
Occassionally the unit test congress.tests.datasources.json_ingester.
test_exec_api.TestExecApiManager.test_execute_exec_api_rows
fails inside eventlet code even though the eventlet call is mocked
out.
This patch adds a new mock path because eventlet.spawn_n
somehow redirects there. Hopefully it resolves the occassional
instability gets.
Sample error:
File "/home/zuul/src/git.openstack.org/openstack/congress/.tox/py27/local/lib/python2.7/site-packages/eventlet/greenpool.py", line 115, in spawn_n
function, args, kwargs, True)
File "/home/zuul/src/git.openstack.org/openstack/congress/.tox/py27/local/lib/python2.7/site-packages/mock/mock.py", line 1062, in __call__
return _mock_self._mock_call(*args, **kwargs)
File "/home/zuul/src/git.openstack.org/openstack/congress/.tox/py27/local/lib/python2.7/site-packages/mock/mock.py", line 1128, in _mock_call
ret_val = effect(*args, **kwargs)
File "congress/tests/datasources/json_ingester/test_exec_api.py", line 27, in mock_spawn_execute
return func(*args, **kwargs)
File "/home/zuul/src/git.openstack.org/openstack/congress/.tox/py27/local/lib/python2.7/site-packages/eventlet/greenpool.py", line 99, in _spawn_n_impl
self._spawn_done(coro)
File "/home/zuul/src/git.openstack.org/openstack/congress/.tox/py27/local/lib/python2.7/site-packages/eventlet/greenpool.py", line 131, in _spawn_done
self.coroutines_running.remove(coro)
KeyError: <greenlet.greenlet object at 0x7fd975681cd0>
Change-Id: I72e11a67cf1cd3ff3fbc574f2d6e91df17d63d9c
Some of the fixed endpoint configs caused errors.
Magnum and Zun explicitly call for grabbing the API endpoint from
catalog. Switch all the JSON ingester configs to use API endpoint
from service catalog to avoid incorrect config.
Change-Id: I50d2755317f68928f41d3770e25dfe67ba088969
Closes-bug: 1819556
Fixes a bug where json ingester repeatedly creates redundant indexes
each time Congress restarts.
Mitigates a problem where indexing fails when a field exceeds the
postgres indexable limit of 2712 bytes. Indexing is made optional.
Fixes a bug where unexpected behavior occurs when the backend DB
default isolation level is not the expected read_committed. The
isolation level is now set explicitly per connection.
Change-Id: I514aa9b96e4efbffe8880cce775dc2259eca4648
Closes-bug: 1819987
Closes-bug: 1819988
Closes-bug: 1819985
It can happen that the _ds_manager is temporarily unreachable
due service not ready or network issues. In these cases,
the patch makes sure the API returns 503 http status code
instead of 500, so that the client knows the issue is
temporary.
Change-Id: I8a03a2e0ac84c2395b5ea2015647c5c7d2fed03e
Closes-bug: 1811754
Update OptionJsonEncoder to encode an unknown type as the expected
{'type': 'type_name'} rather than as 'type_name', fixing an error
encountered by related code that expects to index into the generated
dict construct.
Also updated congress agent tolerate unknown (custom) types instead
of exiting completely.
This reverts commit 69848285c8.
Change-Id: Ie18967b1b4191c43e00817d7661cc4b5efd247be
Closes-Bug: 1819248
Allow config to specify a port to use in making the API calls to
ingest JSON.
Also added additional sample configs: neutron, keystone, monasca.
Change-Id: I7d852742d0943c3857ee4e04d5a0d77b6bafeed7
partially-implements: bp json-data-model
Add the 'api_default_headers' field which specifies a hash/dict
of headers to pass with each API call w.r.t. that JSON ingester.
Main usage is to specify API microversion. For example:
api_default_headers:
X-OpenStack-Nova-API-Version: "2.26"
Change-Id: Ie859e45ea03135aa21196fe19767f28627e76c3b
partially-implements: bp json-data-model
Allows json ingester config YAMLs to use the !ref tag to reference
previously defined reusable structures, which makes deployment much
more convenient.
Allows devstack plugin to optionally enable JSON ingester feature.
Renamed and reorganized the JSON ingester config options.
Adds CI job which sets up JSON ingester.
partially-implements: bp json-data-model
Change-Id: I6391224c99249d16fe943b8f00fe12d1e6b7d8e6
Operator can define an _exec_api view/table in any schema, including
the columns: endpoint, path, method, body, parameters, headers
(body, parameters, headers may be null).
Each time JSON ingester updates state, Congress will make all the
API requests as specified in these _exec_api views/tables.
The existing action execution semantics is used: execute any new
actions not present as of the last time the execution tables are
checked.
Each JsonIngester instance now keeps a reference to the shared
ExecApiManager instance. Each time a JsonIngester instance updates
state, it tells the ExecApiManager instance to check and execute API
requests.
partially-implements: bp json-data-model
Change-Id: I548b4d097452b2f001efc054780be29eb3a73398
This commit adds Tacker driver and unit tests. Which creates
corresponding datasource on service startup and allows congress
to maintain Tacker's VNF and VNFD data via populating data to tables
vnfs and vnfds respectively. Based on which rules
can be created and actions can be executed on tacker service.
Change-Id: I281e8ff508dd066514694419c5aae6a991ab307c
Partially-Implements: blueprint add-tacker-datasource-driver
Add an experimental feature flag to make sure the experimental
JSON ingester code do not affect normal Congress operations
unless BOTH the experimental flag is set in congress config
AND the JSON ingester YAML config files are present.
Change-Id: I8e3822461fa41e62511b485b807a0df27981609d
partially-implements: bp json-data-model
Each table accepting webhook is configured with a jsonpath expression
to grab the unique ID from the webhook payload.
Each time a webhook comes in, the new data is written to the Postgres
database in jsonb format, replacing any previous webhook data with
the same ID.
The _key column type is changed from bigint to text in order to
accommodate custom keys in generic webhook processing.
PollingJsonIngester is renamed to JsonIngester because it is no
longer specialized to polling.
config format is changed slightly to clearly define which tables
are configured for polling and/or webhook.
A table that is updated through both poll and webhook is not
supported by this patch, but may be supported in the future.
Change-Id: I23c3d5c2c63fd8471868c2b37fd154eb2d1f9783
partially-implements: bp json-data-model
The need to execute config init before importing harness in
congress_servers.py causes a style violation in hacking 1.1.0.
Here we move the initialization in bin/congress-server because it's
the first place that sees the command line options.
It also allows the work on upping hacking to 1.1.0 to go ahead.
Change-Id: I7fbc12195f5d79d7df8da3bb835f3f89c1244efc
Direct modification of DataSourceDriver.TRANSLATORS constant for unit testing
causes some unit tests to affect one another. So we instead use contextual
mocking.
Change-Id: I1bea6fd68921410724c8427167966752dbb6e947
Instead of rewriting the entire DB table on each poll, update the
table by adding new records and deleting records no longer
needed.
Because the change is usually much smaller than the entire table,
ingest performance should be much improved, allowing the ingested
state to be more current.
A _key column is added to each table of json records to facilitate
performant deletion of specific records without linear scan.
Also includes small improvements and variable renames.
Change-Id: I3ea4f0325c34d9eef38d7ee64f0bca60e9241620
partially-implements: bp json-data-model
New PollingJsonIngester class which polls API end points as defined
in YAML configs and populates configured postgreSQL database with
JSON data.
Change-Id: I8d22cf8d2da1b1c2e99b8e0cd2d60f3ef71e2def
partially-implements: bp json-data-model
Previously, all data source drivers replace the entire state on each
poll or push update (doctor). The DSE publish code relied on this
situation to optimize performance by skipping a copy when setting
_last_published_data. The coreference introduced did not matter
because the original state was never edited, only replaced.
But beginning with the webhook drivers
(monasca webhook and vitrage), the drivers edit the current state.
This a bug was introduced where sometimes an update would not be
published to messaging bus because of an incorrect _last_published_data
due to coreference.
This patch fixes the bug. In addition, the _last_published_data
elements are stored as set rather than list to save the need to
convert to set each time a diff is needed. The times when
conversion back to list is needed to publish on the messaging bus
(used for snapshot publish only)
is expected to be far fewer than the times the conversion to set is
needed.
A few additionalogging lines are also introbuced for improved
debuggability.
Change-Id: I332a844beaf8dec4450bb1ade6ce7e41cc3d4d91