The pinned versions of webob and hacking weren't functioning properly
with modern Python, so they are unpinned.
Unpinning hacking brought a number of extra rules into use, so this
commit also contains a number of code style fixes.
There was also an incompatibility in the test_setup.sh script, which
meant it doesn't work with MySQL 8.0 (which is in Ubuntu 20.04). This
is also fixed in this commit.
Change-Id: I0889bc35f8babfeec42f6f577d302bb5fbce1c95
Python 2.7 is deprecated now, and some dependencies have dropped their
support for it. We should also just drop support for it rather than
pinning the dependency versions.
This commit also switches to a generic Python 3 tox environment,
adds additional test jobs for more Python 3 versions, and cleans up
some Python 2 compatibility cruft.
Change-Id: I3e7cc1713dec1bada4a85f75362c4fa4f89ceae6
To simplify the launchpadlib dependency installation for the
storyboard-migrate utility, use a separate tox testenv specifically
for this purpose and put that as its only additional runtime
requirement. Name the testenv "lpimport" with the expectation that
in time we may also rename the storyboard-migrate utility similarly
(so as to disambiguate it from the Alembic migrations we rely on
for database schema upgrades).
Change-Id: I8bd36bf69c235c8957ae4a161fd22cbe80028cf4
This implements the storage backend interface added in the previous
commit using Swift. It also adds some example configuration to the
sample config file, and some setup to initialise a storage backend
if one is configured.
Change-Id: I8467486ed42f8674e2b1db635789e88bf4113850
In order to be able to upload to PyPI we need to avoid the name
clash with another existing "storyboard" project which is still
active there, in use and is understandably disinclined consider
renaming (they were there first). We can eventually consider
renaming our project fully to storyboard-api along with its Git
repository and all, but for now simply publishing under an
alternative distribution name will suffice.
While we're in there, add some additional Python package metadata,
correct some more, and remove an invalid trove classifier which
would have prevented uploading. Also mark the resulting wheel as
"universal" (supporting both Python 2 and 3). Bump the minimum PBR
version to accommodate the project_urls and description-content-type
options.
Change-Id: I365ad340ec875f8603e088fa114e7de8aff191c9
A. Fixes depedencies of Python 3:
Following the documentation about Python 3 requirements.
It was needed to change in the file ./requirements.txt
replacing "python-openid" with "python3-openid".
B. Fixing Python 3 compatibility in the migration script:
In order to fix the migration script to run on Python 3
I changed the line #24 on ./migrate/launchpad/reader.py,
changing the iterator.
Also, I added in the file "./requirements.txt" a comment
to use the library "launchpadlib" in order to run
the migrate script.
Change-Id: Iedcd283ade1ea62fdc5777dfe7b1ae5916db3480
This limit pins PyMySQL to a very old version, which lacks support
for recent MySQL versions. Removing this limit allows StoryBoard to
work with MySQL 8.
Change-Id: I4851e25b14913233fd7183756e5f4fe6bc47d28a
The recent oauthlib 3.0.0 release seems to have regressed
authentication, raising "InvalidRedirectURIError: (invalid_request)
Invalid redirect URI." in
oauthlib.oauth2.rfc6749.grant_types.base._handle_redirects() on
returning from the OpenID provider. This is an emergency pin to <3
until we're able to suss out whether this is a bug in storyboard's
authentication client implementation or a legitimate regression in
oauthlib.
Change-Id: Iad4e035d88a0cd335c6ef68263bf7e044a192513
The upper cap on oslo.db and sqlalchemy cause very old versions of both
to be installed. The oslo.db one is particularly bad as it includes a
'def async' which does not work with python3 (fascinated how the tox
py35 tests are working)
eventlet and sqlalchemy-migrate were added a while ago to solve
build-time dependency issues in transitive deps. They are no longer
needed.
Change-Id: I964d87408573008c7b7cca6ce0110a910429b9b8
This reverts commit d84e80ec99 because
it depends on change I5235ae4368f8bff13c767b87ced83173d52b5155 which
is being reverted.
Change-Id: Icfda239538ff9d1ef663e32e6bc9695bb99b2dcb
This commit adds an alternate notification publisher driver for mqtt.
The intent here is to use this driver with firehose.openstack.org, which
is infra's unified event bus. [1] To use this driver you first need to
set it as the notification driver with the driver option in the
notifications config group. Then set the required config options in the
mqtt_notifications section.
[1] https://docs.openstack.org/infra/system-config/firehose.html
Change-Id: Ie82e625fcce7d5b5b794a46f6456ccff1dc5ec3e
Pecan depends on webob. We don't currently specify a version,
so Pecan pulls in the latest version of webob. (1.8.0rc1)
There is some incompatibility between our codebase and
the latest version of webob, causing our tests to fail.
This patch pins webob to the last known compatible version,
so that development is not blocked on investigating the
incompatibility.
In the future, we should investigate further so that we
can use up to date python dependencies
Change-Id: I8d4f14c992c18b53888657580a292a21b565de85
Replace the incubated copy of gettextutils with oslo.i18n, following the
best-practices guidelines in the library documentation for handling
imports, module names, and exceptions to the hacking rules.
Change-Id: Ie366afd8bda2a72c964d9ddf7dd53718002fb4d0
Story: 2000776
Signed-off-by: Doug Hellmann <doug@doughellmann.com>
Removing openstack/common from all OpenStack code in favor of the
Oslo libraries is a project wide goal for the Ocata release.
Change-Id: I548afbdf8a66044a710908468a9de0aa6502cc6f
The tests are failing with weird foreign key constraint failures
on patches which don't affect the relevant bits of code.
This commit limits the version of PyMySQL to less than the recent
release (at the time of writing) which appears to correlate with
the test failures appearing.
Change-Id: I2924feed790786fb029f9cbbd89e186cb8f02967
Version 4.8.0 of oslo.db appears to have broken our scheduler. We
should fix this in the long term, but for now this patch limits the
version to < 4.8.0 so that it is possible to continue work.
Change-Id: I60861981ebdbcdc292b116310afa386c583f1d47
We can use newer setuptools and pbr to use environment markers in our
requirements files which allows us to select deps if the environment
matches. With this we restrict the email dep to python < 3.0 which is
the only dep we have that doesn't work across python 2 and 3.
Note that this also updates the pbr machinery to pull in this new
feature support. Part of this update requires updating hacking to avoid
conflicting pbr requirements. This in turn has created some churn in the
format of the code but should largely be a noop.
Change-Id: I0be5dd8a6b33a51329077b5a5f4c7f5576829956
Searching for tasks is broken because we attempt to filter a query
with multiple columns called `title` ambiguously (on `title` instead
of `tasks.title`).
This is due to a bug in the version of sqlalchemy_fulltext currently
in use on storyboard.openstack.org (version 0.2). This bug was fixed by
version 0.2.2, but we may as well require 0.2.3 (the current latest
release).
Story: 2000579
Change-Id: Ia8c8c6a35a52188ac8f596fb7fed0eb7adbbea33
A deployed instance that had apscheduler==3.1.0 was failing
with the following error:
mod_wsgi (pid=11256): Target WSGI script '/var/lib/storyboard/storyboard.wsgi' cannot be loaded as Python module.
mod_wsgi (pid=11256): Exception occurred processing WSGI script '/var/lib/storyboard/storyboard.wsgi'.
Traceback (most recent call last):
File "/var/lib/storyboard/storyboard.wsgi", line 16, in <module>
from storyboard.api import app
File "/usr/local/lib/python2.7/dist-packages/storyboard/api/app.py", line 34, in <module>
from storyboard.plugin.scheduler import initialize_scheduler
File "/usr/local/lib/python2.7/dist-packages/storyboard/plugin/scheduler/__init__.py", line 22, in <module>
from apscheduler.executors.pool import ThreadPoolExecutor
File "/usr/local/lib/python2.7/dist-packages/apscheduler/__init__.py", line 3, in <module>
version_info = tuple(int(x) if x.isdigit() else x for x in parsed_version.public.split('.'))
AttributeError: 'tuple' object has no attribute 'public'
Installing a previous version fixed the issue.
Change-Id: I0ddd236c2601f10f7440bfdd1b74a80a540732ac
argparse was external in python 2.6 but not anymore, remove it from
requirements.
This should help with pip 8.0 that gets confused in this situation.
Installation of the external argparse is not needed.
Change-Id: Ib7e74912b36c1b5ccb514e31fac35efeff57378d
WSME 0.8 is more restrictive than previous versions, and so we
should avoid it until we've ensured it works properly with
the webclient.
Change-Id: Iedad308dddb7733ed7238aa7643953ee4eefdc45
This updates oslo.config to be in-line with global-requirements,
and renames all the old references of 'oslo.config' to the newer
oslo_config.
Change-Id: I68d8f67d37bca918ce1193c2a19e00f7ed4f6caa
Updated oslo.db to most recent version, which includes a patch by
@greghaynes that fixes pymysql error handling. Also updated all uses
of the deprecated oslo.db namespace to use oslo_db.
Change-Id: I7de41aef6db0f45a638eb2aa22cfae45f99c8195
Inside WSME's parse_isodatetime there's a switch that checks for the
existence of python_dateutil. If this does not exist, it falls back
on its own ISO parsing, which does not support timezones, and causes
havoc in our API. This adds the python_datetime dependency
(BSD License) that triggers this switch.
An unrelated change has been submitted to the WSME project to handle
timezone parsing: https://review.openstack.org/#/c/161013/
Change-Id: Ib1bbe1118cc04f7622d1d0ef3993594d5bd102f7
python-crontab is GPL'd, so we can't use it. This replaces all
related functionality with apscheduler. Notable API changes: Since
the APScheduler guarantees single execution environments, we no
longer have to provide execution time windows for our plugins.
Note, pymysql 0.6.4 seems to have issues with some unicode characters.
Change-Id: Ie8b3473ac316c8c661f7ffe1cdf069e7e822f23b
This version of pyMySQL appears to have some unicode conversion
problems that are preventing use of apscheduler.
Change-Id: I48d1d255777e6f7ef25468fcdbbe7b7b333838d9
mysql-python undesireable over PyMySQL for a few reasons. Mainly, it
does not support Py3* and does not work well with eventlet.
There is also a bug in using oslo.db with PyMySQL where reference errors
are not properly detected. This fixed in oslo.db
ab20754db71e55b79b9e71e36ad86d9befc89a92 but we should still work around
this unless global-requirements are bumped for oslo.db.
Also adding pymysql to requirements since it is a pure python library
and having a codebase that can 'just work' is awesome.
Change-Id: I0058193c7cbd329731ade37108614fa6eb19d0f7
This commit adds an email templating engine which permits batch
generation of emails. By default, it requires a text template,
however other text template types may be added to the engine and will
be appended in order. It is contained inside the email
plugin directory, as it is one component of the emailing system.
Change-Id: I90ee44425807e96d2fb3ddc0adf122a54636a266
This patch adds sanity checking for response_type 'code'. It does
this by switching errors within the OAuth controller to use
exception handling rather than explicit response creation, so that we
can more easily raise errors encountered during our validation process.
Change-Id: I4f3821f8f63f4102def95e407c59c81571c0bb55
Oslo.utils was added to requirements and
openstack.common.excutils,
openstack.common.importutils,
openstack.common.timeutils were deleted.
Change-Id: I3bc76190989c4bc20406dc1b58c9fc2b6bf5436e
Oslo.log was added to requirements and
openstack.common.log was deleted.
All imports openstack.common.log chanhed to
imports from oslo.
Change-Id: Iad5b90255c963b3cbd2e3901ebe2ea2437736fa3
Version of SQLAlchemy changed to 0.9.
Version of sqlalchemy-migrate changed to 0.9.
Function cast in models was fixed.
Change-Id: I4f4c5631d792ff921a3f1a52b7faa5bb4aa462db
This adds a crontab plugin hook to StoryBoard, allowing a plugin
developer to run periodic events. Example use cases include:
- Summary emails.
- Periodic report generation.
- Synchronization check points.
Plugins are expected to provide their own execution interval and
configuration indicator. The management of cron workers is
implemented as its own cron plugin as a sample, and unit tests
for all components are provided.
Change-Id: I3aa466e183f1faede9493123510ee11feb55e7aa
The deferred processor now loads plugins via stevedore, and passes
all event messages to each plugin found. This provides the first
extensibility mechanism for storyboard, by allowing anyone to write
tasks that should be handled asynchronously when the system changes.
Custom workers must implement two methods: 'enabled' and 'handle'.
The first checks to see whether this plugin is enabled, and will be
run only once during initialization. The second handles the event.
The intent is to drive more advanced functionality off of this
mechanism, such as emails, search indexes, and third party tool
integration.
Change-Id: I32b40eab9355c18db1e4ec132b09dc77561a3475
For each action, except for creation of a new project, two messages will
be sent to rabbitmq. One with the main resource; resource_id; user_id;
method and the other with the event_id; user_id; a faked method POST and
a faked resource TIMELINE_EVENT
Publisher creates an exchange called 'storyboard' and publishes the
messages to it. Subscriber creates the queues with different binding_keys
which bind themself to the storyboard exchange and start consuming the
messages from the exchange.
The consumed messages are used by CR 113016.
Co-Authored-By: Nikita Konovalov <nkonovalov@mirantis.com>
Change-Id: Ia573437302dc2d0b1a68d2343e83f9dd397fac04
This reverts commit 9a5a5b59d6.
As agreed on #openstack-infra, this change is being made so
proper credit may be given to the actual authors of this patch.
Change-Id: I07fda79ca0bdd2cfd2a1a45d25fd0a1fc692b6f4
For each action, except for creation of a new project, two messages will
be sent to rabbitmq. One with the main resource; resource_id; user_id;
method and the other with the event_id; user_id; a faked method POST and
a faked resource TIMELINE_EVENT
Publisher creates an exchange called 'storyboard' and publishes the
messages to it. Subscriber creates the queues with different binding_keys
which bind themself to the storyboard exchange and start consuming the
messages from the exchange.
The consumed messages will be printed on the console for now which will
be modified later.
Do not merge until storyboard has Rabbitmq running.
Change-Id: Ic4697f79aaab82dadf1fb1ae66f414a90ae28dac
Search endpoints added for Projects, Stories, Tasks, Comments and Users.
SqlAlchemy plugin for fulltext search added.
The migration for full-text indexes added.
The migration checks the MySQL server version before creating indexes.
The default search engine via sqlalchemy added.
Change-Id: Ie3e4c4f317338d68e82c9c21652d49220c6e4a7d
WSME version 0.6 includes a new type: StringType, which is required
for new storyboard functionality.
Change-Id: I70231464c2a92014c2573bfc05c5adf2f2246e5a
Prod is broken, because I forgot about libraries. With these two
I was able to get the API running with no test dependencies installed.
Change-Id: I178ddf464495b20f9a94f963cf58ab9f28a3af73
Adam Coldrick