Commit Graph

106 Commits

Author SHA1 Message Date
Ritchie, Frank (fr801x) 1767178234 Update helm toolkit reference
Update helm toolkit ref so that ingress will be created with a pathType
of Prefix.

https: //review.opendev.org/c/openstack/openstack-helm-infra/+/905757
Change-Id: I261603e3c68337347955996f77b86bd60a2f2af5
2024-01-25 14:57:38 +00:00
Sergiy Markin dc3445030a Sync requirements with shipyard
This PS adjusts list of dependences needed to get syncronized with Shipyard project.

Change-Id: I9235702aa34b9f0bf6634229a24e5f1847ac9a7d
2023-04-25 13:11:36 +00:00
Sergiy Markin ac4edb0c64 [focal] Deckhand project updates
- adjusted .gitignore to keep fresh egg-info and omit build artifacts
- fresh egg-info data is needed for promenade that depends on Deckhand
- restored deckhand-functional-uwsgi-py38 gate
- restored deckhand-integration-uwsgi-py38 gate
- made deckhand-airskiff-deployment gate voting ( treasuremap project
  has been updated)
- removed bionic gates
- updated focal dockerfile
- added more binary deps into bindep.txt
- updated deckhand chart values to latest images - focal and wallaby
- fixed python code to compy with CVE's found by fresh version of bandit
- implemented pip freeze approach
- added tox -e freeze profile to manage it
- requirements-frozen.txt is now main file with requirements
- requirements-direct.txt is the file to control deps
- updated setup.cfg to adjust to newer version of setuptools
- fixed airskiff-deploy gate
- fixed docker-image-build playbook to restore Quay repo image publish
- updated other playbooks to include roles from zuul/base-jobs in order
  to setup build hosts properly
- removed workaround with hardcoded dns resolver ip 10.96.0.10 as it
  became obsolette due to recent fix in openstack-helm-infra
- adjusted tools/whitespace-linter.sh script
- tox.ini has been brought to compliance with tox4 requirements
- replaced str() calls with six.text_type() according to D325 Deckhand specific
  commandment from Hacking.rst
- locked python-barbicanclient version with 5.2.0 because of breaking
  changes in the upper versions

Change-Id: I1cd3c97e83569c4db7e958b3400bdd4b7ea5e668
2023-04-20 19:39:43 +00:00
Wahlstedt, Walter (ww229g) 70aa35a396 update to focal and python 3.8
update dockerfile for python deckhand install
add deckhand version to chart 1.0
add chart version 0.2.0
update all packages to latest in requirements.txt
update zuul jobs for focal and python 3.8
remove zuul job functional-uwsgi-py38 in favor of functional-docker-py38
update tox config
typecast to string in re.sub() function
add stestr to test-requirements.txt
add SQLAlchemy jsonpickle sphinx-rtd-theme stestr to requirements.txt
deprecated function: BarbicanException -> BarbicanClientException
fix mock import using unittest
fix import collections to collections.abc
fix for collections modules for older than python 3.10 versions.
deprecated function: json -> to_json
deprecated function:  werkzeug.contrib.profiler ->
    werkzeug.middleware.profiler
deprecated function: falcon.AIP -> falcon.App
deprecation warning: switch from resp.body to resp.text
rename fixtures to dh_fixtures because there is an imported module
    fixtures
switch from stream.read to bounded_stream.read
deprecated function: falcon process_response needed additional parameter
deprecated function: falcon default_exception_handler changed parameter
    order
move from MagicMock object to falcon test generated object to fix
    incompatability with upgraded Falcon module.
Adjust gabbi tests to fix incompatability with upgraded DeepDiff module
update Makefile to execute ubuntu_focal
update HTK (helmtoolkit)
unpin barbican to pass integration tests
Use helm 3 in chart build.
    `helm serve` is removed in helm 3 so this moves
    to using local `file://` dependencies [0] instead.

Change-Id: I180416f480edea1b8968d80c993b3e1fcc95c08d
2023-02-24 10:51:57 -05:00
Phil Sphicas 5cd799cc5d Allow source substring extraction
When performing substitutions, there are occasions when the source value
does not exactly match the format required by the destination document
(e.g. the values.yaml structure of an Armada chart).

This change provides the ability extract a substring of the source
value, and substitute that into the destination document.

Two optional fields are added to `src` under `metadata.substitutions`:

  * `pattern`: a regular expression, with optional capture groups
  * `match_group`: the number of the desired capture group

The canonical use case is a chart that requires an image with the repo
name and tag in separate fields, while the substitution source has the
full image path as a single value.

For example, assuming that the source document "software-versions" has:

    data:
      images:
        hello: docker.io/library/hello-world:latest

Then the following set of substitutions would put the repo and tag in
the applicable values in the destination document:

    metadata:
      substitutions:
        - src:
            schema: pegleg/SoftwareVersions/v1
            name: software-versions
            path: .images.hello
            pattern: '^(.*):(.*)'
            match_group: 1
          dest:
            path: .values.images.hello.repo
        - src:
            schema: pegleg/SoftwareVersions/v1
            name: software-versions
            path: .images.hello
            pattern: '^(.*):(.*)'
            match_group: 2
          dest:
            path: .values.images.hello.tag
    data:
      values:
        images:
          hello:
            repo:  # docker.io/library/hello-world
            tag:   # latest

Change-Id: I2fcb0d2b8e2fe3d85479ac2bad0b7b90f434eb77
2022-01-18 13:04:25 -08:00
Phil Sphicas 7be8e1e6ac Drop Python 3.5, make xenial/opensuse non-voting
Update tox configuration and Zuul config to use Python 3.6, and make the
xenial and opensuse gates non-voting.

Change-Id: Iab98ad6cf269d57acf0598ae431b2f661673b234
2021-10-19 15:20:38 -07:00
Phil Sphicas d7d3cbde2b Revert jsonschema to 3.2.0
Change-Id: I5dbd56e40592cd7d3da71889ee17fff44cfee6ef
2021-09-08 05:09:32 +00:00
anthony.bellino 1328f5b568 Deckhand gate fix
Unpin python3-six for Opensuse image build.

Update helm-toolkit stable commit to merge of this change:
https://review.opendev.org/#/c/803654/

Update the helm installation script to download and install v2.17.0

Fix integration tests by pinning Barbican to stable commit.

Pinn jsonschema to 4.0.0a2 to fix tox tests

Change-Id: I2badd0e2f6c934098f0c9f5ef7e52354756c12e0
2021-08-30 07:53:24 -07:00
Rick Bartra 1f0c011a17 Update pip package versions in preparation of pip 20.3
When pip is upgraded to 20.3, the pip dependency resolver is much more
strict and will no longer install a combination of packages that is mutually
inconsistent[0].

These changes account for the fact that Shipyard imports Armada, Drydock,
Promenade, and Deckhand. Having said that, with pip 20.3, the pip
packages amongst those projects cannot conflict. A follow-up change may
be needed if more conflicts are found.

[0] https://pip.pypa.io/en/latest/user_guide/#changes-to-the-pip-dependency-resolver-in-20-2-2020

Change-Id: Id75acea82ddf5d915a8b8805e076dac49cab800f
2020-09-28 12:21:20 -04:00
Kumar, Nishant (nk613n) a2606e75b1 Remove unused code for policy validation as feature not implemented
Policy validation in Deckhand was not implemented completely. Refer link
below:
https://airshipit.readthedocs.io/projects/deckhand/en/latest/users/validation.html#policy-validations

This PS removes some of the code related to the feature which was being
used in a code path when a set of documents are uploaded to Deckhand.
In standard Airship deployments the number of documents could be quite
high and this leads to significant delay (more than 300seconds in some
cases). As there are no plans to implement the policy validation feature,
it makes sense to remove it from code path which could cause delay and
sometimes timeouts while uploading documents.

This has been tested on a Baremetal lab: GF and BF.

Change-Id: I2ff3f40a7fe37bed5a589fab00d829db726604fe
2020-05-14 00:34:42 +00:00
Ahmad Mahmoudi e203c15e94 Add support for Ubuntu bionic base image
- Added ability to build deckhand docker image, using either the
  Ubuntu xenial or the Ubuntu bionic base image.
- Made the bionic base image the default base image for deckhand
  docker image build.

Change-Id: I26657de34a233ee3223a7f93fc667e734ac9140b
2020-02-06 17:42:33 -06:00
Roman Gorshunov 416df18353 Remove Python 2.x support
Drop support of Python 2.x as it is being deprecated.

Updated documentation, tox config, requirements.txt.

Job templates added:
- openstack-python35-jobs: openstack-tox-py35
- openstack-python3-ussuri-jobs: openstack-tox-py36, openstack-tox-py37

Job added:
- openstack-tox-pep8

Job templates removed:
- openstack-python-jobs: openstack-tox-pep8, openstack-tox-py27

Jobs removed:
- deckhand-tox-py27-postgresql

Change-Id: I7783ba841258a913f5ecd9d6f1130d378345ab5a
2019-10-08 12:41:12 +00:00
Mahmoudi, Ahmad (am495p) 7e5d81f50f Let the Werkzeug package version float
Apache-airflow release 1.10.3 and later use Flask 1.10.1,
which needs at least Werkzeug 0.15.1.
https://issues.apache.org/jira/browse/AIRFLOW-4900

Airflow depends on deckhand for document validation. But,
deckhand locks the Werkzeug package version to 0.14.1,
which breaks airflow.
This PS lets deckhand use the latest Werkzeug available release.

Change-Id: I4385a69c41571010c426dfaee1082109914c7d2a
2019-09-11 04:27:17 -05:00
Rajeshwari Dharwadkar 971c901c4d Update base image from leap15.0 to leap15.1
Changes made in opensuse_15 dockerfile to support opensuse leap15.1

Change-Id: Ifc959e9e8aa5da4ed6703ef8157f1c5beedc8ecd
2019-07-11 17:21:43 -07:00
Arun Kant 8eb74b0509 Adding opensuse image build for deckhand
Adding DISTRO parameter for makefile to invoke distribution specific
Dockerfile and build image accordingly.

Updated logic for existing jobs to have distro specific logic
for building and publishing images.

Added multiple distro specific document in operator section.

Change-Id: I415ab28b06ea17b21e76d28ccb3e284041c8072d
2019-05-30 17:22:57 +00:00
Roman Gorshunov a4e4b77676 Docs build fix (#4)
Change-Id: I99bb346b9a40c27e3e355186086c2ef121063204
2019-05-15 11:23:24 +02:00
Roman Gorshunov 239a6464d2 Docs build fix (#3)
Change-Id: Iad67bbc125a7753d932e511f54f2cf5d9b4f1a2f
2019-05-14 21:52:50 +02:00
Roman Gorshunov def7116b67 Docs build fix (#2)
Change-Id: I156998a219e49be05cc2ad5e4193dcd0142f3c83
2019-05-14 20:42:15 +02:00
Roman Gorshunov 256805d4f0 Docs build fix (#1)
Change-Id: I3995688cb3fb1f34d99264c450f3349b3a2762ad
2019-05-14 19:09:34 +02:00
Zuul 49ad9f3884 Merge "docs: Add use cases for each of the mutation operations" 2019-02-22 23:20:28 +00:00
Roman Gorshunov c81a88b963 Embed UML generated diagrams into docs, fix docs build
Remove manual generation of UML diagrams; they will be built by Sphinx
itself.

Require to install PlantUML and Graphviz via bindep in
openstack-tox-docs job; ReadTheDocs service already has both PlantUML
and Graphviz installed for documentation builds.

Change-Id: I21ab124a80e7768bc9edd891b975b4e0f8a6e50d
Story: 2004747
2019-02-14 13:55:44 +01:00
Dustin Specker 815306ddf8 docs(substitution): mention that all occurrences are replaced
Before it was unclear if all occurrences of a pattern were replaced
in a given target.

Change-Id: Ie02a54fef98a912b36b66891be8fed346ba5d9e0
2018-11-30 13:39:22 -06:00
Felipe Monteiro 82ceafa7bc docs: Add use cases for each of the mutation operations
This patch set adds use cases for each of the mutation
operations used by Deckhand:

* substitution
* layering
* replacement

To help document authors decide which operations they should use.

Change-Id: I8bc470c7ddc2f3c0aa9d4372a2323fa84632e369
2018-11-06 09:37:35 -05:00
Zuul 947810ada5 Merge "docs: Add config documentation to operator's section" 2018-11-05 18:05:49 +00:00
Rick Bartra 60e82b7bd6 Validate additional 'metadata.replacement' scenarios
This patch set adds additional documentation and unit tests
to validate further replacement scenarios.

In particular this commit adds an additional document check that
looks for documents exisitng in different layers that contain the
same name and same schema without any of them having `replacement: true`

Change-Id: I7c033d32a6755f36e609789a748cbc6d4af06bc2
2018-10-30 10:23:14 -04:00
Felipe Monteiro b34156ae7b docs: Add config documentation to operator's section
This PS adds configuration documentation that includes
a literalinclude of the config file as well as some
information on each of the cache config options as
these are important on performance.

Change-Id: I3b06012b8843b7bfbd46307f81397172a41d3675
2018-10-29 18:21:38 +00:00
Zuul 27aeeb8fea Merge "docs: Add documentation on data redaction" 2018-10-29 17:26:37 +00:00
Felipe Monteiro 9d91a072cd docs: Use sphinx-apidoc library for autodoc compatibility
This package is used for generation autodoc documentation
automatically which can be linked to by Deckhand
documentation from other places. This is to make autodoc
generation work in RTD.

More info: https://pypi.org/project/sphinxcontrib-apidoc/

Change-Id: I43aac82728e5935a5a2626f2fd29d7a7188d19f9
2018-10-27 22:52:39 +01:00
Felipe Monteiro 018162f1ef rtd: Fix warnings in RTD causing autodoc to fail
This patch set fixes warnings [0] in RTD which are causing autodoc
to fail to build, resulting in missing documentation as in
here: https://airship-deckhand.readthedocs.io/en/latest/operators/exceptions.html

Example warnings:

WARNING: autodoc: failed to import module 'deckhand.policy'; the following exception was raised:
No module named 'falcon'

[0] https://readthedocs.org/api/v2/build/8001684.txt

Change-Id: I872b90e03a8f9e3de3fd717ed340b53f095f01a5
2018-10-27 15:21:54 +00:00
Zuul 1d4cc81dfa Merge "docs: Elaborate on document layering in documentation" 2018-10-24 14:03:35 +00:00
Zuul b4cd48cde0 Merge "trivial: Add missing alembic upgrade head to manual install" 2018-10-24 02:27:40 +00:00
Felipe Monteiro 0d5189e226 trivial: Add missing alembic upgrade head to manual install
This patch set adds `./entrypoint.sh alembic upgrade head` command
to manual install section in Getting Started guide, without which
Deckhand server cannot start correctly.

Change-Id: Id486cc98b8fd93271a8571364adfb9fb30bf4bcc
2018-10-23 22:09:22 -04:00
Felipe Monteiro f711a83ee7 docs: Add documentation on data redaction
This patch set adds documentation related to data redaction in
order to capture the purpose behind it and how it adds an
added layer of security to Deckhand.

Change-Id: Icb28970684a9026fda200273a14d9ba421f627d7
2018-10-20 14:48:19 -04:00
Aaron Sheffield 349e5600df Redacts Raw Documents
- If a document has a storage policy of encrypted
 - Redacts (sha256) the data section.
 - Redacts (sha256) the substition paths.
- Uses the same /documents endpoint, adds a new query parameter
  ?cleartext-secrets=true to show the non-redacted values.

Change-Id: I42808901b97c667a1148c00fbb7717a0847c9981
2018-10-19 23:56:12 -05:00
Felipe Monteiro 38c1d7a1a3 fix: Pin down Deckhand package requirements
This patch set pins down package requirements for the following
reasons:

* Inconsistencies between ranges and direct pins
* Deckhand isn't gated against requirements repo so changes to
  some packages isn't actively tested
* Other Airship components use pins, so Deckhand might break them
  when package requirements go out of sync between components
* Deckhand as of yet has no stable release so Airship relies on
  master for stability so it is better to ensure stability over
  anything else

Change-Id: I20ef4be3e01cd267771098d33447ccb61aee20b4
2018-10-18 02:36:35 +01:00
Felipe Monteiro 1ac9abb555 docs: Elaborate on document layering in documentation
This patchset elaborates on document layering in the documentation
to provide much greater clarity into what layering is and its
associated concepts, including: layer, layer order, layering policy,
layering definition, document abstraction, parent selection,
layering actions, etc.

Change-Id: I584e67b7984fa4035cef481a116ae3b8a3eb2906
2018-10-10 19:06:38 +00:00
Felipe Monteiro a8660a7e53 docs: Reorganize documentation structure
This patch set reorganizes Deckhand's documentation structure
for better organization into 3 distinct categories:

* developer's guide
* operator's guide
* user's guide

This means that the RTD navigation menu on the left-hand side
will have fewer links (see list above) making navigation much
easier. This is similar to how Armada organizes its documentation
too.

This patch set also updates README section with a better
overview and trims some fat from it (remove testing
documentation as it doesn't really belong there -- there
is a dedicated page for that already).

Finally, this patch set changes the exceptions page to
render as a basic list of autoexception classes because
the current tabularized view is not rendering correctly
on RTD [0].

[0] https://airship-deckhand.readthedocs.io/en/latest/exceptions.html
Change-Id: I162383bf8e3bbd5004603c979ac7b0d760a210c4
2018-09-26 20:29:02 -04:00
Roman Gorshunov d41e5a44ca Fix: various documentation and URL fixes
1) UCP -> Airship
2) readthedocs.org -> readthedocs.io (there is redirect)
3) http -> https
4) attcomdev -> airshipit (repo on quay.io)
5) att-comdev -> openstack/airship-* (repo on github/openstack git)
6) many URLs have been verified and adjusted to be current
7) no need for 'en/latest/' path in URL of the RTD
8) added more info to some setup.cfg and setup.py files
9) ucp-integration docs are now in airship-in-a-bottle
10) various other minor fixes

Change-Id: I12b2fa8fbec37a483a0ad50382e08f51ed97533a
2018-09-25 11:44:57 +02:00
Zuul 7dddbd56d4 Merge "substitution: Recursive pattern replacement" 2018-09-19 21:38:08 +00:00
pallav 9345035522 Adding api for revisions deep diffing
GET /revisions/{{revision_id}}/deepdiff/{{comparison_revision_id}}

 - Added deepdiff api for generating diff between
   two rendered documents.
 - Deep diffing for data and metadata
 - Refactor diff functions
 - Client update
 - Added unit testcases
 - Added funtional testcases
 - Doc update

Change-Id: Ib60fa60a3b33e9125a1595a999272ca595721b38
2018-09-17 17:01:34 +05:30
Felipe Monteiro c9d71a6509 substitution: Recursive pattern replacement
Patterns may now be replaced recursively. This can be
achieved by using specifying a ``pattern`` value and
``recurse`` (with a required ``depth`` argument).

Example:

    substitutions:
      - dest:
          path: .
          pattern: REGEX
          recurse:
            depth: -1
        src:
          schema: deckhand/Passphrase/v1
          name: example-password
          path: .

NOTE:

  Recursive selection of patterns will only consider
  matching patterns. Non-matching patterns will be ignored.
  Thus, even if recursion can "pass over" non-matching patterns,
  they will be silently ignored.

This is useful for reducing the number of substitution
stanzas that are required for the purposes of performing
all the string pattern replacements that are required.
Best practice is to limit the scope of the recursion
as much as possible: e.g. avoid passing in "$" as the
``jsonpath``, but rather a JSON path that lives closer
to the nested strings in question.

Included in this patch set:

* recursive implementation for pattern replacement
* unit tests for most scenarios and edge cases
* docstring updates
* documentation updates
* schema updates for validation

Change-Id: I85048349097ed696667fae80f1180808d264bbcf
2018-09-12 09:24:46 -06:00
Felipe Monteiro 5325b46412 trivial: Update deprecated Airship links in docs
This updates att-comdev to openstack/airship-* namespace as well
as old RTD links to new ones.

Change-Id: Ic7d7fae2ac2f64e4b59eb4f350d26784d2d1ac40
2018-08-30 15:32:18 +00:00
melissaml 7290c91889 Remove the duplicated word
Change-Id: Ifdd74e418c7387e39fd0d8e667e86ccf2ed889fe
2018-08-22 10:57:02 +08:00
Zuul 64975c820a Merge "doc(typo): Correct spelling" 2018-08-08 22:05:17 +00:00
Zuul 9b6eb81c82 Merge "docs: Update document types documentation" 2018-08-06 16:15:09 +00:00
Tin Lam 02e7fc5793 doc(typo): Correct spelling
Trivial fix. Correct a minor spelling error in doc.

Change-Id: Ie111acdf7dbbf884cc12acf9136d324ad332a823
Signed-off-by: Tin Lam <tin@irrational.io>
2018-08-04 21:17:28 -05:00
Felipe Monteiro f1e47e80c1 docs: Update document types documentation
This patch set updates document types documentation
which includes expounding on control documents, layering policy
documents, and dataschema documents.

Change-Id: Id31fcb6b68ca30fdf681dec8348c2fc4237cd48d
2018-07-31 21:27:28 +00:00
Zuul 9077e27589 Merge "Rename some instances of ucp to airship" 2018-07-24 16:27:33 +00:00
Felipe Monteiro c95ec45307 Rename some instances of ucp to airship
This renames some instances of ucp to airship in the documentation
and makes some trivial documentation fixes.

Change-Id: I9a4a81d15bfc13b4fe089b7d65f0df43eeade9fb
2018-07-23 20:56:00 +01:00
Zuul 109b78df59 Merge "Simplify schema validation" 2018-07-10 13:42:57 +00:00