Commit Graph

51 Commits

Author SHA1 Message Date
Sergiy Markin ac4edb0c64 [focal] Deckhand project updates
- adjusted .gitignore to keep fresh egg-info and omit build artifacts
- fresh egg-info data is needed for promenade that depends on Deckhand
- restored deckhand-functional-uwsgi-py38 gate
- restored deckhand-integration-uwsgi-py38 gate
- made deckhand-airskiff-deployment gate voting ( treasuremap project
  has been updated)
- removed bionic gates
- updated focal dockerfile
- added more binary deps into bindep.txt
- updated deckhand chart values to latest images - focal and wallaby
- fixed python code to compy with CVE's found by fresh version of bandit
- implemented pip freeze approach
- added tox -e freeze profile to manage it
- requirements-frozen.txt is now main file with requirements
- requirements-direct.txt is the file to control deps
- updated setup.cfg to adjust to newer version of setuptools
- fixed airskiff-deploy gate
- fixed docker-image-build playbook to restore Quay repo image publish
- updated other playbooks to include roles from zuul/base-jobs in order
  to setup build hosts properly
- removed workaround with hardcoded dns resolver ip 10.96.0.10 as it
  became obsolette due to recent fix in openstack-helm-infra
- adjusted tools/whitespace-linter.sh script
- tox.ini has been brought to compliance with tox4 requirements
- replaced str() calls with six.text_type() according to D325 Deckhand specific
  commandment from Hacking.rst
- locked python-barbicanclient version with 5.2.0 because of breaking
  changes in the upper versions

Change-Id: I1cd3c97e83569c4db7e958b3400bdd4b7ea5e668
2023-04-20 19:39:43 +00:00
Wahlstedt, Walter (ww229g) 70aa35a396 update to focal and python 3.8
update dockerfile for python deckhand install
add deckhand version to chart 1.0
add chart version 0.2.0
update all packages to latest in requirements.txt
update zuul jobs for focal and python 3.8
remove zuul job functional-uwsgi-py38 in favor of functional-docker-py38
update tox config
typecast to string in re.sub() function
add stestr to test-requirements.txt
add SQLAlchemy jsonpickle sphinx-rtd-theme stestr to requirements.txt
deprecated function: BarbicanException -> BarbicanClientException
fix mock import using unittest
fix import collections to collections.abc
fix for collections modules for older than python 3.10 versions.
deprecated function: json -> to_json
deprecated function:  werkzeug.contrib.profiler ->
    werkzeug.middleware.profiler
deprecated function: falcon.AIP -> falcon.App
deprecation warning: switch from resp.body to resp.text
rename fixtures to dh_fixtures because there is an imported module
    fixtures
switch from stream.read to bounded_stream.read
deprecated function: falcon process_response needed additional parameter
deprecated function: falcon default_exception_handler changed parameter
    order
move from MagicMock object to falcon test generated object to fix
    incompatability with upgraded Falcon module.
Adjust gabbi tests to fix incompatability with upgraded DeepDiff module
update Makefile to execute ubuntu_focal
update HTK (helmtoolkit)
unpin barbican to pass integration tests
Use helm 3 in chart build.
    `helm serve` is removed in helm 3 so this moves
    to using local `file://` dependencies [0] instead.

Change-Id: I180416f480edea1b8968d80c993b3e1fcc95c08d
2023-02-24 10:51:57 -05:00
Phil Sphicas d7d3cbde2b Revert jsonschema to 3.2.0
Change-Id: I5dbd56e40592cd7d3da71889ee17fff44cfee6ef
2021-09-08 05:09:32 +00:00
anthony.bellino 1328f5b568 Deckhand gate fix
Unpin python3-six for Opensuse image build.

Update helm-toolkit stable commit to merge of this change:
https://review.opendev.org/#/c/803654/

Update the helm installation script to download and install v2.17.0

Fix integration tests by pinning Barbican to stable commit.

Pinn jsonschema to 4.0.0a2 to fix tox tests

Change-Id: I2badd0e2f6c934098f0c9f5ef7e52354756c12e0
2021-08-30 07:53:24 -07:00
Rick Bartra 1f0c011a17 Update pip package versions in preparation of pip 20.3
When pip is upgraded to 20.3, the pip dependency resolver is much more
strict and will no longer install a combination of packages that is mutually
inconsistent[0].

These changes account for the fact that Shipyard imports Armada, Drydock,
Promenade, and Deckhand. Having said that, with pip 20.3, the pip
packages amongst those projects cannot conflict. A follow-up change may
be needed if more conflicts are found.

[0] https://pip.pypa.io/en/latest/user_guide/#changes-to-the-pip-dependency-resolver-in-20-2-2020

Change-Id: Id75acea82ddf5d915a8b8805e076dac49cab800f
2020-09-28 12:21:20 -04:00
Phil Sphicas 42fe1b85cb Accelerate YAML operations with LibYAML
Patch PyYAML (via the pylibyaml library) to automatically enable the
LibYAML parser and emitter, which are faster than the Python versions.

https://pypi.org/project/pylibyaml/

Change-Id: Iebcc50b5db87518b3b7e0fac124c712afd06da2b
2020-09-25 04:54:48 +00:00
Andrii Ostapenko 9d07671866
Fix pep8 gate running on py3.8
* Bumps hacking to 2.0.0 and addresses corresponding issues
* Bumps bandit to 1.6.2

Change-Id: I9d540acbd40700874b574687bc1fee4a9b5c0cb2
Signed-off-by: Andrii Ostapenko <andrii.ostapenko@att.com>
2020-09-24 22:35:22 -05:00
Ahmad Mahmoudi 0545625da9 Scaling deckhand uwsgi workers
Updated obsolete uwsgi default configuration parameters for better
performance.
Increased number of worker threads to increase performance.
Uplifted uwsgi to the latest for bug fixes since 2018.

For more info please see:
https://uwsgi-docs.readthedocs.io/en/latest/ThingsToKnow.html

Change-Id: Ifedb9c6279e64be86deb6ec375810c5ecf97958a
2020-08-05 22:05:57 +00:00
Ahmad Mahmoudi e7ba6828a0 (fix) Address uwsgi and other gating issues
- With Ubuntu bionic based deckhand docker image, uswsi crashes with
  segmentation fault when it tries to load psycopg2 library, causing
  the deckhand service become unavailable.
  The root cause of this problem seems to be that uwsgi and psycopg2
  binary wheels are built with different ssl libraries.
  To address this issue, upgrading psycopg2 to the latest release to
  the latest binary release.

- For opensuse 15.1 image build, python3-six was updated to address
  package not found issue.

- Updated gating ansbile playbooks to address non-voting gate failures.

Change-Id: I7be920e16e6114eb2bdbc052a6761f29008baf81
2020-02-28 21:28:56 +00:00
Ahmad Mahmoudi e203c15e94 Add support for Ubuntu bionic base image
- Added ability to build deckhand docker image, using either the
  Ubuntu xenial or the Ubuntu bionic base image.
- Made the bionic base image the default base image for deckhand
  docker image build.

Change-Id: I26657de34a233ee3223a7f93fc667e734ac9140b
2020-02-06 17:42:33 -06:00
Roman Gorshunov 416df18353 Remove Python 2.x support
Drop support of Python 2.x as it is being deprecated.

Updated documentation, tox config, requirements.txt.

Job templates added:
- openstack-python35-jobs: openstack-tox-py35
- openstack-python3-ussuri-jobs: openstack-tox-py36, openstack-tox-py37

Job added:
- openstack-tox-pep8

Job templates removed:
- openstack-python-jobs: openstack-tox-pep8, openstack-tox-py27

Jobs removed:
- deckhand-tox-py27-postgresql

Change-Id: I7783ba841258a913f5ecd9d6f1130d378345ab5a
2019-10-08 12:41:12 +00:00
Mahmoudi, Ahmad (am495p) 7e5d81f50f Let the Werkzeug package version float
Apache-airflow release 1.10.3 and later use Flask 1.10.1,
which needs at least Werkzeug 0.15.1.
https://issues.apache.org/jira/browse/AIRFLOW-4900

Airflow depends on deckhand for document validation. But,
deckhand locks the Werkzeug package version to 0.14.1,
which breaks airflow.
This PS lets deckhand use the latest Werkzeug available release.

Change-Id: I4385a69c41571010c426dfaee1082109914c7d2a
2019-09-11 04:27:17 -05:00
Ian H. Pittwood ecc7b834ed Upgrade six to 1.12
Upgrades the six package from 1.11.0 to 1.12.0 to match requirement in
Pegleg

Adds system six 1.12.0 installation in OpenSUSE dockerfile.

Change-Id: Ie03235157c4b30e8e94866ea5bd8f634b61eb0d8
2019-08-19 20:41:49 +00:00
HUGHES, ALEXANDER (ah8742) a4850f8c4f Update packages related to requests
Pegleg Promenade Deckhand and Shipyard should all use same versions
of packages when able.  Requests currently is giving a warning
in Pegleg:
ERROR: deckhand 0.0.1.dev657 has requirement urllib3==1.24, but
you'll have urllib3 1.24.3 which is incompatible.

Change-Id: I271da8c5e3bcf635b24f88c69d966cd39e7ec7f6
2019-07-26 11:38:53 -05:00
Crank, Daniel (dc6350) 2c831c1fd1 [ad-hoc] Update oslo.utils ver to 3.40.2
oslo.utils version increased to 3.40.2 to resolve
requirements conflict with tox.

Change-Id: I43217a4c9e4f6e6547adb1d4ffaf1d4dadd376a7
2019-02-28 14:18:54 -06:00
Samuel Pilla 48818aa01f Update oslo.util version in requirements
Change-Id: I11b44544825c51c9e3a294625776bb6aa3babfd9
2019-01-23 20:29:57 +00:00
Drew Walters f745e4b6e7 CI: Fix integration job
Deckhand's integration job relies on OpenStack-Helm-Infra to deploy
Kubernetes using Kubeadm. Since [0] was merged, Calico requires AppArmor
to be deployed. This commit adds the same role to deploy AppArmor that
exists in OpenStack-Helm-Infra to prevent inadvertent gate failures.

[0] https://review.openstack.org/614805/

This also includes gate fix in https://review.openstack.org/#/c/627906/
because of a circular dependency; that change in question fixes
failing CI jobs for all non-integration jobs.

Change-Id: Ie3fb04ea0fbe0487ae743033315e2a4211248d32
2019-01-18 14:11:04 -06:00
Drew Walters 236e098478 requirements: Update pinned requirements
Version 4.17.0 of the `keystonemiddleware` package is incompatible
Python 3 and will cause Deckhand operations to fail when using
Keystone authentication. This change uplifts the versions in
`requirements.txt` using pur (see [0]).

[0] https://pypi.org/project/pur/

Change-Id: Ie63add37828e4b859751b9c45d05164be31d4a33
2018-10-26 20:53:43 +00:00
Felipe Monteiro b80df59d11 fix: Address small issues with revision rollback controller
1. There is no exception called `InvalidRollback` in Deckhand (it
was removed a while back). Instead, the only exception that
db_api.revision_rollback raises is RevisionNotFound from
the revision_get call internally.

So catch that instead from the controller.

2. The default value of parameters is `str` so when revision_id
of '0' is passed to the db module for processing, it skips over
the check for `if revision_id == 0` as revision_id is a str,
not int. So this leverages builtin int converter logic in
falcon [0] but requires uplifting the version of falcon to
at least 1.3.0 to make use of it [1].

[0] https://falcon.readthedocs.io/en/stable/api/routing.html#field-converters
[1] https://falcon.readthedocs.io/en/1.3.0/api/routing.html#field-converters

Change-Id: I068cd9e9b6818a5d51501f2718ee2d40d556c094
2018-10-18 09:45:16 -04:00
Felipe Monteiro 38c1d7a1a3 fix: Pin down Deckhand package requirements
This patch set pins down package requirements for the following
reasons:

* Inconsistencies between ranges and direct pins
* Deckhand isn't gated against requirements repo so changes to
  some packages isn't actively tested
* Other Airship components use pins, so Deckhand might break them
  when package requirements go out of sync between components
* Deckhand as of yet has no stable release so Airship relies on
  master for stability so it is better to ensure stability over
  anything else

Change-Id: I20ef4be3e01cd267771098d33447ccb61aee20b4
2018-10-18 02:36:35 +01:00
Felipe Monteiro 3f39e639ea [Gate Fix] Fix failing functional/integration tests
This patch set pins oslo.messaging to the last stable version
to fix the gate [0]. 9.0.0 is incompatible with current
requirements.

[0] https://pypi.org/project/oslo.messaging/#history

Change-Id: I6cf3e9f33ac3efcbf48675273899af38fef86e91
2018-10-05 18:18:55 +01:00
pallav 9345035522 Adding api for revisions deep diffing
GET /revisions/{{revision_id}}/deepdiff/{{comparison_revision_id}}

 - Added deepdiff api for generating diff between
   two rendered documents.
 - Deep diffing for data and metadata
 - Refactor diff functions
 - Client update
 - Added unit testcases
 - Added funtional testcases
 - Doc update

Change-Id: Ib60fa60a3b33e9125a1595a999272ca595721b38
2018-09-17 17:01:34 +05:30
Matt McEuen a886daf2d0 Add cryptography to Deckhand
Use of memcached, which is enabled by default in the Deckhand Helm
chart, requires the cryptography package.

Change-Id: I13970b43bf5616d4e0ece933d438a55245eea4e0
2018-07-30 12:21:16 -05:00
Felipe Monteiro d55ee9fb6e Add better caching to jsonpath-ng wrapper functions
This patchset uses beaker (used by Promenade and Drydock) to
achieve better caching around jsonpath-ng wrapper functions
(jsonpath_replace and jsonpath_parse).

Change-Id: Ifae24775b4741ade7673dc91776c35d2de5b9065
2018-07-03 02:07:27 +00:00
Felipe Monteiro 3353833dba Update Deckhand test-/requirements.txt
This simply updates Deckhand's package versions in
test-requirements.txt and requirements.txt using `pur -r` command
line tool.

Change-Id: If4bbd85190502c52a7f10adf6af750172b54c234
2018-06-16 18:18:28 -04:00
Tin Lam 0143b2d727 fix(gate): make the functional gate to pass
This patch set allows the functional gate of airship-deckhand to pass
once again.

Change-Id: I72406464ee490319dc6fa31851a235f0b5850467
Signed-off-by: Tin Lam <tin@irrational.io>
2018-06-16 00:25:38 -05:00
Bryan Strassner 5f1fbbee3c [396582] Add alembic support to Deckhand
Updates Deckhand to use alembic to manage database upgrades.
Moves from creating tables at startup of Deckhand to the
db-sync job.

Change-Id: I6f4cb237fadc46fbee81d1c33096f48a720f589f
2018-04-06 23:30:16 -04:00
Scott Hussey 9f2a0fb347 Collect profile data on DH requests
- Seeing issues with a lot of Drydock
  requests timing out and it seems to be a
  downstream issue with pulling Deckhand
  docs
- Add jsonpath cacheing as the jsonpath-ng
  parser was consuming 54s of the total 56s
  runtime of a rendered-documents GET call.
  With caching, the call is taking closer to 2s.
- All add a .dockerignore file to make image
  building a little faster

Change-Id: I6ef84ffd946dcf2713b4f7570b985156deb1d697
2018-02-15 13:09:16 -05:00
Felipe Monteiro ff5c66046d Bump up package requirements versions
Change-Id: I6922a4dc8c542a3168de471b2abf529b7a11cb12
2018-02-13 22:54:44 -05:00
Felipe Monteiro e4abca1cd7 Use DAG to resolve substitution dependency chain
Currently, Deckhand fails to perform substitution given
a substitution dependency chain for a group of documents.
That is, if A depends on B and C depends on B for substitution
then substitution will fail. Deckhand, at present, can only
perform substitution if A depends on B and C depends on B
and D depends on B, for example: In this case, the dependency
chain is no bigger than 2.

However, for cases where the substitution dependency chain
is larger than 2, then the dependency is no longer trivial. This
is because the substitution dependencies form a DAG that must
be topologically sorted in order to derive the correct order of
substitution. Once the documents are correctly sorted according
to this scheme, then substitution can be carried out as usual.

This PS makes the aforementioned changes to Deckhand's layering
module to make substitution work for non-cyclical dependencies:
A DAG is used to topologically sort the documents according to
their substitution dependency chain. A unit and functional
test has been added to verify the code.

If a cycle is detected, a critical error is thrown. Unit tests
have been added to validate this case as well.

Change-Id: Iaca3963f44aec6c897ad9fd690ce314a3a4d97a2
2018-02-12 20:54:54 +00:00
Alan Meadows 0fae4bec58 The field returned by barbican is secret_ref, not secret_href
Also, this accounts for the the resp.to_dict() call not absorbing
the secret_ref key.

This commit also adds type mapping for missing secret types to
their barbican equivalents.

Change-Id: Idd4895fd441443a3dc41a3358edf6bd3648be5c1
2018-02-07 21:45:04 -05:00
Felipe Monteiro b47f421abf DECKHAND-87: Deckhand API client library
This PS implements the Deckhand API client library
which is based off the python-novaclient code base.
The client library includes managers for all the
Deckhand APIs.

The following features have been implemented:
  * Framework for API client library
  * Manager for each Deckhand API (buckets, revisions, etc.)
  * API client library documentation

Tests will be added in a follow-up (once Deckhand functional
tests use Keystone).

Change-Id: I829a030738f42dc7ddec623d881a99ed97d04520
2017-12-13 20:56:23 +00:00
Felipe Monteiro 90226c2ae1 Integrate Deckhand with keystone auth
This PS integrates Deckhand with keystone auth so
that Deckhand can check whether a keystone token is
authenticated (by way of keystonemiddleware)
before proceeding with any requests.

The architecture for this PS is borrowed from [0]
which successfully integrates keystone authentication
with the falcon web application framework. However,
additional Deckhand-specific changes were made for
tests to pass.

The following changes have been made:

  - add paste deploy configuration file which adds
    keystonemiddleware integration to Deckhand; this
    makes it trivial for keystonemiddleware to determine
    whether a token in the X-Auth-Token header is authenticated
  - use paste.deploy to create a web app
  - update unit tests for testing controllers
  - update functional test script to ignore keystone authentication
    because functional tests don't currently support keystone
    integration

[0] https://github.com/stannum-l/nautilus

Change-Id: I6eeeb4a4d9ab1f1cc8fb338e5cc21136ab4d5684
2017-10-16 19:54:46 +01:00
Felipe Monteiro 3e62ace8ed Support filtering revision (documents) by any legal filter
This PS adds support for filtering revisions and
revision documents documents by any legal filter
(those enumerated in the design document).

Deckhand now supports the following filter arguments:
  * schema
  * metadata.name
  * metadata.label
  * metadata.layeringDefinition.abstract
  * metadata.layeringDefinition.layer
  * status.bucket

Deckhand now supports the following filter arguments for filtering
revisions:
  * tag

Deckhand now supports multiple filters, e.g.:
  * ?metdata.layeringDefinition.layer=site&metadata.name=foo

Deckhand now supports repeated filters, e.g.:
  * ?metadata.label=foo=bar&metadata.label=baz=qux

The following has yet to be implemented will be done in a future
follow-up PS:
  - support sorting by specific keywords as well
  - support query limit and offset filters

Change-Id: I8558481e075715fe7fab98140094d37782a986d9
2017-10-06 16:48:45 -04:00
Felipe Monteiro b30aaf2fec Add requirements for memcached
We need memcached as a requirement in order for Deckhand to make
use of memcached

Change-Id: I1f47eab3db7282612837e177b8f52d430c05f7a8
2017-10-06 15:07:31 -04:00
Felipe Monteiro af0bfd813d Deckhand postgresql compatibility.
Currently, Deckhand is not fully compatible with postgresql as
it uses sqlite for all of its testing, including functional testing.
Since postgresql will be used in prod, Deckhand obviously must
support it, in addition to sqlite, needed for unit testing.

This commit alters the functional testing script to use postgresql
as well as makes necessary back-end changes to support postgresql.

Included in this commit:

  - alter tools/functional-tests.sh so that it uses postgresql
    as the db connection
  - modifies primary key for Bucket DB model to be an Integer rather
    than a String
  - updates foreign key to point to new primary key
  - updates necessary integration logic so that the bucket name
    is still known by the Document DB model and returned in
    appropriate response bodies

Change-Id: I7bc806fb18f7b47c13978dcd806d422a573a06b3
2017-09-22 19:28:47 +01:00
Felipe Monteiro 8715fe3e02 [flake8] Enable extra, optional hacking checks
Update test-requirements.txt to use latest version of:
    * hacking

Enable the following off-by-default checks:
    * [H203] Use assertIs(Not)None to check for None.
    * [H204] Use assert(Not)Equal to check for equality.
    * [H205] Use assert(Greater|Less)(Equal) for comparison.
    * [H210] Require ‘autospec’, ‘spec’, or ‘spec_set’ in
             mock.patch/mock.patch.object calls
    * [H904] Delay string interpolations at logging calls.

Made minimal code changes to comply with changes.

Change-Id: I3559ead76b5476650d7193e7023d349175234922
2017-09-21 10:43:21 -04:00
Felipe Monteiro ad2e232cb9 [TrivialFix] Remove redundant requirements.
Change-Id: I4798d5b88c80e0525cc529116b379e5c77193a78
2017-09-17 02:52:29 -04:00
Anthony Lin c99c180303 DeckHand Dockerfile
1) Add Dockerfile
2) Add entrypoint.sh
3) Add uwsgi in requirements.txt and remove it from
   tox.ini

Change-Id: Ie9086335b5e6403e5b1e46981db110894606b9d1
2017-09-14 16:50:06 +00:00
Felipe Monteiro e62e9e9f40 DECKHAND-33: Add oslo.config options for keystone auth
This commit adds oslo.config options for keystone auth and
updates Deckhand's request context to use oslo_context for
facilitating integration with keystone auth options.

Change-Id: Ifd170e1a192402a970f8538f0c06bf017fe77f88
2017-08-24 20:12:05 +01:00
Felipe Monteiro 6e2238c3d8 Fix flake8 errors
This commit fixes flake8 errors and fixes a minor bug related to
a schema version being v1 rather than v1.0.

OpenStack hacking rules are used to pin down flake8 to sane
standards using [0].

[0] 06e676c461/test-requirements.txt (L5)

Change-Id: Ib236df6f5ec9505c0e635f0faa9877d3397a2e55
2017-08-14 20:48:14 +01:00
Felipe Monteiro 2fd01fd9b4 Add currently necessary oslo namespaces to oslo-config-generator conf file.
This commit adds the following namespaces to deckhand's
config-generator.conf file used by oslo-config-generator
to generate deckhand's conf file automatically:

* oslo.db
* oslo.db.concurrency
* oslo.log

This will automatically populate the generate conf file with
needed oslo config options.
2017-07-21 02:20:13 +01:00
Felipe Monteiro 3630898da9 Added oslo_context-based context for oslo_db compatibility. 2017-07-18 22:41:15 +01:00
Felipe Monteiro f6a58708a4 Temporary change - do not commit 2017-07-17 21:39:35 +01:00
Felipe Monteiro 03cc29116e Initial DB API models implementation. 2017-07-17 20:47:56 +01:00
Felipe Monteiro ac4fc210ee Add jsonschema validation to Deckhand
Previously Deckhand was using manual validation of YAML files.
However, schema validation is much cleaner, robust and thorough.
So, this commit adds jsonschema as a dependence in Deckhand.
Tempest already uses it as a dependence as well -- so a precedence
already exists.

This commit also updates unit tests as needed. Documentation
changes will be made in a follow-up patch.
2017-07-17 20:47:36 +01:00
Felipe Monteiro bc36541f52 Merge branch 'master' into oslo-log-integration 2017-07-06 18:48:42 -04:00
Felipe Monteiro 6a25e6f06b Add oslo.log integration
This commit adds oslo.log integration to Deckhand.
2017-07-06 23:45:09 +01:00
Felipe Monteiro c825e77418 DECKHAND-10: Add Barbican integration to Deckhand
This commit adds Barbican integration to Deckhand.
2017-06-29 22:25:11 +01:00
Felipe Monteiro 00bb92561d DECKHAND-2: Design core Deckhand API framework
This commit implements the core Deckhand API framework.
It does not implement any real API routes. The core
framework is modeled after Drydock's [0].

This commit specifically:

  - implements the core API framework which uses falcon
  - implements errors.py for preliminary errors
  - implements base resource class from which other API
    resources will inherit to build out the API itself
  - implements base API router
  - implements entry-point for kicking off deckhand
  - updates base README.rst with instructions on
  - running and installing -- similar to Drydock's
  - implements dummy API resource for secrets, to
    be fleshed out further in a follow-up commit

[0] https://github.com/att-comdev/drydock
2017-06-27 19:26:51 +01:00