Commit Graph

65 Commits

Author SHA1 Message Date
David Moreau Simard 4a8711e73c
opendev: Remove content and leave an URL to the GitHub repository
Change-Id: I82a3238b6a8c21e6bb8056aca22ef71af9ea2538
2021-02-25 15:16:53 -05:00
David Moreau Simard ccad5e992f
tests: Add a playbook to test ara with mitogen
This way we can test it if need be.

Change-Id: Ia21785ea38e2ccbc7220bad354431ca4cf7d184f
2021-02-24 23:18:08 -05:00
David Moreau Simard 52b201dae2
Add support for recording controller fqdn
The controller fqdn that ran the playbook is now recorded and can be
searched for both in the UI and the CLI.

Fixes: https://github.com/ansible-community/ara/issues/193
Change-Id: I53e8d158fc3b6ba7a16582234aaa2542eab5fcdc
2020-12-18 16:35:40 -05:00
David Moreau Simard a4f21d6e3f
tests: miscellaneous improvements to container build tests
- add debug logging and recover the log files
- discard database and settings file between container tests

Change-Id: Ifd8a8d544c4c550873a9eb586de038710a60ba05
2020-12-18 16:31:11 -05:00
David Moreau Simard 4fe18c6492
callback: disable threading by default
Threading is prone to database lock exceptions when using the default
database backend (sqlite) and so disable it by default.

Users can enable threading when using mysql or postgresql by specifying
the ARA_CALLBACK_THREADS environment variable or callback_threads in
ansible.cfg.

Fixes: https://github.com/ansible-community/ara/issues/195
Change-Id: I80edfd6268684722c2783e01323355d791c19b9d
2020-12-18 10:25:01 -05:00
David Moreau Simard e1d470f289
container images: fixes and update to f33
- Install which (because it's sometimes not installed by default)
- Fedora 33 images are out now, build and publish them
- Remove PyYAML workaround that is no longer necessary

Fixes: https://github.com/ansible-community/ara/issues/192
Change-Id: Ia64455655cd37299d84c657a3dc87e079f92f790
2020-12-04 22:08:47 -05:00
David Moreau Simard c511ff94e7
tests: include the max task count in benchmark
This way we can tell the progress.

Change-Id: Ic7e622636ba58228f78cdafe639b013781766cb0
2020-11-01 09:46:14 -05:00
David Moreau Simard 0493ce7015
zuul: fix image publishing with quay.io
Variable precedence is higher in the playbook than from the job vars
which are provided through the inventory.
Use another variable and default properly instead.

Change-Id: I8c5cf72c78ea01ba1bc0a6fef2f9499837e9b262
2020-10-27 10:34:10 -04:00
David Moreau Simard a2fa087f3c
zuul: add post-job to publish container images to quay.io
With the coming rate limiting on docker hub, users might want to have an
alternative to pull from.

Change-Id: I35714fdd069165c9766371c83393aa431237ccec
2020-10-26 23:09:47 -04:00
David Moreau Simard f6df57d25c
tests: Add a simple benchmarking playbook
The playbook creates a specified number of fake hosts (as localhost)
and runs a task file against them a specified number of times.

By default it will create 25 hosts and run 50 tasks against them,
providing 1250 results.

Change-Id: I88642041d8cee3c11f9d993d6dca245d7eb33f8e
2020-10-14 20:58:00 -04:00
David Moreau Simard 40894c7027
callback: record handler tasks too
It turns out that we had been missing out on handler tasks because we
need to use the hook "v2_playbook_on_handler_task_start" for those.

Fixes: https://github.com/ansible-community/ara/issues/178
Change-Id: Id05ba581fb22eee26270c7f4bccc3445e87b5d3b
2020-09-30 22:30:09 -04:00
David Moreau Simard 43061bdd44
zuul: bump fedora nodesets from f31 to f32
Change-Id: I75ae58a026c3aacdeaedbb3dd9fb96a7a377f763
2020-09-15 11:17:54 -04:00
David Moreau Simard 74defc2273
API: Stop returning nested children resources
When querying the API for a playbook's detail, it would return all of
it's children (hosts, files, tasks, results) which could be very slow
when dealing with larger playbooks.

We no longer do that for playbooks as well as plays and tasks.
Instead, we can easily find a playbook's resources by searching for them
with the playbook id like so:

- /api/v1/plays?playbook=<id>
- /api/v1/tasks?playbook=<id>
- /api/v1/results?playbook=<id>
... and so on.

This commit adapts the built-in UI because it would've otherwise been
broken by the change.

Fixes: https://github.com/ansible-community/ara/issues/158
Change-Id: I442bff657e5da9d6a3916ebdbc5e66c0e670b00f
2020-09-06 11:57:32 -04:00
David Moreau Simard 7ef2bb340a
docs: Refresh structure, Ansible plugins and configuration
- Merge documentation about plugins and their use cases in a single page
- Improve documentation about plugins and their use cases
- Streamline docs on setup helpers
- Expand toctree on the index from 2 to 3 to include deeper headers
- Rename and re-order sections
- Generally decapitalize ara
- Generally remove personal pronouns (you, your, etc.)
- Allow line lengths up to 120 up from 80

Change-Id: If79a47ff66fc90415ed74f8507c3fe82d4f3488d
2020-08-22 11:32:57 -04:00
David Moreau Simard 560f8c5d2f
container-images: incorporate, test and publish centos8
This leverages the contribution from Fabian [1] for building an image
based on CentOS8.
The image will be built and tested for every new patch with Zuul and
it'll be pushed to DockerHub [2] after every merge.

The tag nomenclature had to be changed in order to fit the distribution
as an attribute, before:
- latest (tagged from source-latest)
- pypi-latest
- distribution-latest
- source-latest

to:
- latest (tagged from fedora32-source-latest)
- fedora32-source-latest
- fedora32-distribution-latest
- fedora32-pypi-latest
- centos8-pypi-latest

[1]: 8491cc5a72
[2]: https://hub.docker.com/r/recordsansible/ara-api

Change-Id: I971abd97039a28e825b6b7cd43a7559abe709f30
2020-07-23 21:18:48 -04:00
David Moreau Simard a2831d7a6c
zuul: temporarily disable ara-report in post logs
The role no longer works since a change that now prohibits local
execution on the executor [1] and this makes the jobs hit POST_FAILURE.

It is not critical so disable it while we figure it out.

[1]: https://review.opendev.org/#/c/742229/

Change-Id: I127463ae0fa6661e72d77d5af960062f7abdf134
2020-07-21 22:42:25 -04:00
David Moreau Simard 3fa950f5e6
container-images: Run SQL migrations automatically at boot
This removes a necessary step from users when running the container
image.

Change-Id: Ia4d796b77d7db3df93d2c3751533d46f1cb6df13
2020-05-05 13:34:59 -04:00
David Moreau Simard 673b45911f
zuul: fix typo in dockerhub post job
Change-Id: I9d05c45b20e621bc3cdc41bfdf624ace05952877
2020-04-30 20:39:51 -04:00
Zuul 17964427ba Merge "zuul: fix dockerhub post job syntax" 2020-05-01 00:12:21 +00:00
Zuul 72c672a0bb Merge "zuul: adjust static report build location" 2020-05-01 00:12:15 +00:00
David Moreau Simard 25a61cd260
zuul: fix dockerhub post job syntax
The mustache brackets and the semicolon were confusing yaml and ansible:

   template error while templating string:
     expected token 'end of print statement', got ':'.

Change-Id: I34cbfc8132ce698da9dcef23f54ae3869b294a1d
2020-04-30 19:29:25 -04:00
David Moreau Simard 5fa345dc92
zuul: adjust static report build location
At the end of CI jobs that involve running Ansible playbooks which take
care of integration tests, we generate a static report that is linked in
the Zuul build artifacts.

Putting it in the server directory makes sense and it's easier to align
everything with that.

Change-Id: I3070efd025ed63591b888e721e8556c3bac1d2b7
2020-04-30 19:00:59 -04:00
David Moreau Simard 968ce10f8d zuul: Add job to automatically publish images on dockerhub
This inherits from the "ara-container-images" job that already builds
and test container images, this additional job just publishes the
resulting images.

Change-Id: I456eb2ae942bcb8dbd4206c2858b2e408995ae30
2020-04-30 21:40:36 +00:00
David Moreau Simard e5ae8a167b
Add scripts to build and run ARA API containers
This contributes scripts to build and run ARA API server container images.

Both use a Fedora base image but one installs ARA from source while
the the other installs ARA from fedora packages.

The images are set up to use the gunicorn application server and include
the necessary libraries for users that wish to use the postgresql or
mysql backends.

The scripts are tested with the included playbooks and zuul job.

Change-Id: If85210395dd3d93c80da83fd69b86eecfa0185ef
2020-04-30 11:04:28 -04:00
David Moreau Simard 3e62a52b47
zuul: bump jobs from fedora-30 to fedora-31
The fedora-31 image is available, we should use it.

Change-Id: I6e430b55992d3d5aca3f9ac61ff9144334fa1650
2020-04-29 22:24:53 -04:00
David Moreau Simard dba12facd8
zuul: allow retries when lauching database containers
On the offchance that the download of the container image fails, we
should retry to avoid failing for a momentary network error.

Change-Id: I60f7486f3fee996e71804372ddc0faff18294c64
2020-01-30 16:20:45 -05:00
David Moreau Simard b1ae8ed477
zuul: template ansible.cfg file in server directory
This will allow the ansible.cfg file to get picked up by log collection
in order to see what the templated file looks like.

Also fix stray double quotes for labels.

Change-Id: I311cbf99c070ef4b34ec112a134af9b015e64493
2020-01-30 10:00:27 -05:00
David Moreau Simard 4d800d1b35
zuul: don't declare default_labels if there aren't any
Change-Id: I5d46ca569c6f0d9efaedda8dea93ef288b26218c
2020-01-29 19:31:07 -05:00
David Moreau Simard 0b37aab023
zuul: Use no_log when setting up authentication
Otherwise the secret gets picked up by ARA.

Change-Id: I980b856d1bfc2eaf77c3fcaba662c2222f8f3360
2020-01-29 19:26:55 -05:00
David Moreau Simard c5df7f2cd6
Integration tests: differentiate labels for different pipelines
zuul.change isn't available in the post and periodic pipelines so we
shouldn't try to include it.

We're splitting this into a different file so we can add more things
later.

Change-Id: I53a257aafba395e07afc0a1755e26cab28c1b2f0
2020-01-29 13:12:37 -05:00
David Moreau Simard ad8803318f
Add a job to run playbooks and record them on the demo
This adds a secret for authenticating with the live demo instance,
and a job which runs basic integration tests. The results of these tests
are sent to the live demo instance.

The job is set to run in the "post" Zuul pipeline so the secret cannot be
exposed by speculatively testing a job change.

Change-Id: I8a7926ed547b1acb4b7021ca7c63abb9d68c0ac9
2020-01-29 09:40:18 -05:00
David Moreau Simard 8e5dcd976a
Add new Ansible plugins: ara_playbook and ara_api
These new action and lookup plugins makes it easier to query ARA from
inside a playbook.

In terms of practical use, this first iteration allows us to have an
integration test playbook that asserts some of ARA's features such as
playbook names and labels.

Change-Id: I38bea1062f0002886ecde70827f70d27248a1868
2020-01-29 08:46:28 -05:00
David Moreau Simard a8b8ff5c59 callback: Add support for specifying default playbook labels
This allows users to set default playbooks labels without needing to
specify the 'ara_playbook_label' variable.

In practice, it can be used to set labels on playbooks without needing
to modify playbooks or the playbook vars.

Fixes: https://github.com/ansible-community/ara/issues/95
Change-Id: Ib1d3b33d1766905061e5534677764aa9c42a99b0
2020-01-28 21:34:41 +00:00
David Moreau Simard b551db3289
Remove ara-web integration test jobs for the time being
These might come back in the future but for the time being they are
not being maintained.

Change-Id: I7ade139ca7c97415ae4794ae267991dbfca01817
2020-01-27 19:21:29 -05:00
David Moreau Simard 9ff1f9f0c4
Refactor basic integration jobs for Ansible versions
The latest supported version of Ansible (currently 2.9) is used in
integration test jobs unless specified otherwise.

The basic integration test jobs ensure that we have coverage for all
currently supported versions of Ansible as well as devel to let us know
if there are any upcoming breaking changes.

Change-Id: I1745f0cf22f802aba5bf7f3d120a6b03ac88a1dd
2020-01-27 16:42:05 -05:00
David Moreau Simard 18210bde2c
Refactor and improve distributed sqlite integration test coverage
This removes the job "ansible-role-ara-api-fedora-distributed-sqlite" and replaces
it by "ara-api-distributed-sqlite".

The job tests the support for the distributed sqlite database backend
Fedora and Ubuntu simultaneously, allowing for the addition of CentOS8
in the future.

Change-Id: I5851b4364fd567799e92feda5c41b34e5a22dc49
2020-01-27 15:10:25 -05:00
David Moreau Simard 25fbc069e3
Refactor and improve PostgreSQL integration test coverage
This removes the job "ansible-role-ara-api-ubuntu-postgresql" and replaces
it by "ara-api-postgresql".

The job is designed to have a single database server with multiple
API servers on different Linux distributions in order to get test
coverage for Ubuntu, Fedora and soon, CentOS8.

Change-Id: I2ffa6b7e1d97359e3924a7faf842340e6eb39bbe
2020-01-27 13:44:34 -05:00
David Moreau Simard b5b60f502b
Refactor and improve MySQL integration test coverage
This removes the job "ansible-role-ara-api-ubuntu-mysql" and replaces
it by "ara-api-mysql".

The job is designed to have a single database server with multiple
API servers on different Linux distributions in order to get test
coverage for Ubuntu, Fedora and soon, CentOS8.

Change-Id: Ia4916938039cf5566b1e71e4518b9c5e958ca872
2020-01-27 13:37:17 -05:00
David Moreau Simard b1f23c766a
ara_api role: add support for installing from fedora packages
This patch is self tested by the addition of a new job that exercises
the role with the packages but it does not test the current source yet.

In order to test the current source, we'll need to set up a job that
builds a package from the current source and then configure the job to
install from these built packages.

Change-Id: Ic326a5243b55b415cd31b8a5702fe92d5dfe0651
2019-12-17 12:16:41 -05:00
David Moreau Simard 2feed6d7d1
ara_api role: provide sensible PATH defaults without venv
When deploying without a virtualenv, we need to supply a PATH where the
python and ara-manage binaries are likely to be found.

Change-Id: Ie9b498f988a026a2568cc445ef444fe2f90c837b
2019-12-12 21:12:13 -05:00
David Moreau Simard 84cb73eae3
api: Use basename instead of base_name in URL mapping
It turns out base_name was deprecated in 3.9 [1] and was just removed in
3.11 released today.

This is squashed with an integration test fix which adds another error
case handling.

[1]: 7095021db7

Change-Id: Icfbd94e40b9efd7c750cfa72a49bdbf930ad8fc6
2019-12-12 13:30:03 -05:00
David Moreau Simard 5bd2447257
api: Fix UnicodeEncodeError when parsing and rendering results
This is an attempt to prevent encoding errors when posting results or
when rendering them in the API and built-in interface.

Setting UNICODE_JSON [1] to false allows the callback to post results'
content and allows the API browsing interface to render the result.

The built-in UI had a similar problem but isn't fixed by the
UNICODE_JSON because we are using a Response object so use
"surrogateescape" instead.

Related: https://github.com/ansible-community/ara/issues/48

[1]: https://www.django-rest-framework.org/api-guide/settings/#unicode_json

Change-Id: I48bcef440a1ee9c8574fdd24d5c05ef0f13ca666
2019-11-19 16:46:19 -05:00
David Moreau Simard 7bb0b12693 Clean up basic integration tests and add a job for them
These are basic tests but they need to be tested to make sure they
keep working.

Change-Id: I57bd3931655ca70a07b703b204933a70692f1671
2019-11-19 16:06:32 +00:00
David Moreau Simard 802342c01d
Add an integration test for binary result output
Change-Id: I46687b82cd828a25a3897dce020dc39c096872f0
Related: https://github.com/ansible-community/ara/issues/48
2019-11-15 16:28:28 -05:00
David Moreau Simard 5ceeddaf11
Change default for ARA_DISTRIBUTED_SQLITE_PREFIX to /ara-report
Considering the (re-)implementation of a built-in web report interface,
it makes sense to move the default back to ara-report instead of
ara-api.

This results in:
  - /ara-report/: the web report interface
  - /ara-report/api/v1/: the API

Change-Id: I06cdc07f26a39dc356284a2a549f0ea3a4935e1a
2019-07-22 17:39:11 -04:00
David Moreau Simard 16c65d9f11 Add an "ara-manage generate" command to export to static HTML
This new command renders the templates of the built-in web UI
to static html files.

Change-Id: I66a346bcee188a194a7a2dd300a41d23906bbe5f
2019-07-22 19:40:14 +00:00
David Moreau Simard 9ecf5db227
distributed_sqlite tests: query /api/v1/ instead of the root
The root will be moved with the implementation of the upcoming web
interface.

Change-Id: Ie94ceeb3a2e6f66301aa0142092c54241399c68d
2019-07-22 12:15:01 -04:00
Florian Apolloner 9b0c08f77a
Add distributed_sqlite db backend and WSGI app
This new feature is an implementation equivalent to the sqlite
middleware from 0.x but in Django instead of Flask.

It uses a WSGI middleware to map an URL to a location on
the filesystem which is then loaded by the database backend.

Fixes: https://github.com/ansible-community/ara/issues/57

Co-Authored-By: Florian Apolloner <florian@apolloner.eu>
Co-Authored-By: David Moreau-Simard <dmsimard@redhat.com>
Change-Id: I338dae2fabda2af7fd0a75c795c7b4fd6b179940
2019-07-02 11:53:10 -04:00
David Moreau Simard ca5927fc1c
Use include_role instead of import_role for integration tests
Setting "public: yes" makes the variables available after running
the role which is what we need for the purpose of testing.

The problem with import_role is that the variables are parsed only
once at the beginning [1].

[1]: https://docs.ansible.com/ansible/latest/porting_guides/porting_guide_2.7.html#include-role-and-import-role-variable-exposure

Change-Id: I2cc51eb8020bd8928031e75c85bcae22722e33e1
2019-07-02 11:23:31 -04:00
David Moreau Simard b3b0cb1c1f Run actual integration tests for the ara-api role jobs
We ran the role but not the integration tests, run tests too.

Change-Id: I64d4befa4e7acd172078bc2d5de54c0233229bdd
2019-06-25 20:28:30 +00:00