diff --git a/REVIEWING.rst b/REVIEWING.rst new file mode 100644 index 00000000..db2280b5 --- /dev/null +++ b/REVIEWING.rst @@ -0,0 +1,38 @@ +Reviewing Deckhand Code +======================= +To start read the `OpenStack Common Review Checklist +`_ + + +Unit Tests +---------- +For any change that adds new functionality to either common functionality or +fixes a bug unit tests are required. This is to ensure we don't introduce +future regressions and to test conditions which we may not hit in the gate +runs. + + +Functional Tests +---------------- +For any change that adds major new functionality functional tests are required. +This is to ensure that the Deckhand API follows the contract it promises. +In addition, functional tests are run against the Deckhand container, which +uses an image built from the latest source code to validate the integrity +of the image. + + +Deprecated Code +--------------- +Deprecated code should go through a deprecation cycle -- long enough for other +Airship projects to modify their code base to reference new code. Features, +APIs or configuration options are marked deprecated in the code. Appropriate +warnings will be sent to the end user, operator or library user. + + +When to approve +--------------- +* Every patch needs two +2s before being approved. +* Its OK to hold off on an approval until a subject matter expert reviews it. +* If a patch has already been approved but requires a trivial rebase to merge, + you do not have to wait for a second +2, since the patch has already had + two +2s. diff --git a/doc/requirements.txt b/doc/requirements.txt index c79deda9..280af85b 100644 --- a/doc/requirements.txt +++ b/doc/requirements.txt @@ -4,6 +4,7 @@ sphinx>=1.6.2 # BSD sphinx_rtd_theme==0.2.4 reno>=2.5.0 # Apache-2.0 +plantuml # NOTE(fmontei): The requirement below is only included because readthedocs # depends on it to work, since it only takes in one requirements file. diff --git a/doc/source/HACKING.rst b/doc/source/HACKING.rst index 5d9c0f16..53d3411c 100644 --- a/doc/source/HACKING.rst +++ b/doc/source/HACKING.rst @@ -13,7 +13,10 @@ See the License for the specific language governing permissions and limitations under the License. -======= -Hacking -======= +.. _coding-guide: + +===================== +Deckhand Coding Guide +===================== + .. include:: ../../HACKING.rst diff --git a/doc/source/REVIEWING.rst b/doc/source/REVIEWING.rst new file mode 100644 index 00000000..6d84d41e --- /dev/null +++ b/doc/source/REVIEWING.rst @@ -0,0 +1,5 @@ +======================= +Reviewing Deckhand Code +======================= + +.. include:: ../../REVIEWING.rst diff --git a/doc/source/developer-overview.rst b/doc/source/developer-overview.rst new file mode 100644 index 00000000..9ab67c92 --- /dev/null +++ b/doc/source/developer-overview.rst @@ -0,0 +1,141 @@ +.. + Copyright 2018 AT&T Intellectual Property. + All Rights Reserved. + + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + +============================== +Developer Overview of Deckhand +============================== + +The core objective of Deckhand is to provide storage, rendering, validation and +version control for declarative YAML documents. Deckhand ingests raw, +Airship-formatted documents and outputs fully rendered documents to other +Airship components. + +Architecture +============ + +.. image:: images/architecture.png + :alt: High level architecture of Deckhand + +From a high-level perspective, Deckhand consists of a RESTful API, a document +rendering engine, and a PostgreSQL relational database for document storage. +Deckhand ingests Airship-formatted documents, validates them, and stores them +in its database for future processing. On demand, Deckhand will fully render +the documents, after which they can be consumed by the other Airship +components. + +Deckhand uses Barbican to securely storage sensitive document data. + +`Pegleg `_ in effect provides +Deckhand with a CLI, which facilitates communication with Deckhand. + +.. image:: images/architecture-pegleg.png + :alt: High level architecture of Deckhand + +Components +========== + +control +------- + +The ``control`` module is simply the RESTful API. It is based on the +`Falcon Framework `_ and utilizes +`oslo.policy `_ +for RBAC enforcement of the API endpoints. The normal deployment of Deckhand +uses `uWSGI `_ and PasteDeploy +to build a pipeline that includes Keystone Middleware for authentication +and role decoration of the request. + +The ``control`` module is also responsible for communicating with +`Barbican `_, which it uses to +store and retrieve document :ref:`secrets `, which it passes to the +``engine`` module for :ref:`rendering`. + +engine +------ + +The ``engine`` module is the interface responsible for all +:ref:`rendering`. Rendering consists of applying a series of algorithms to the +documents, including: topological sorting, :ref:`layering`, +:ref:`substitution`, and :ref:`replacement`. + +db +-- + +The ``db`` module is responsible for implementing the database tables needed +to store all Airship documents. The module also realizes version control and +:ref:`revision-diffing` functionality. + +client +------ + +The API client library provides an interface for other services to communicate +with Deckhand's API. Requires +`Keystone `_ authentication to +use. + +Developer Workflow +================== + +Because Airship is a container-centric platform, the developer workflow heavily +utilizes containers for testing and publishing. It also requires Deckhand to +produce multiple artifacts that are related, but separate: the Python package, +the Docker image and the Helm chart. The code is published via the +Docker image artifact. + +Deckhand strives to conform to the +`Airship coding conventions `_. + +Python +------ + +The Deckhand code base lives under ``/deckhand``. Deckhand supports py27 +and py35 interpreters. Once OpenStack deprecates usage of py27, Deckhand +will as well. + +See :ref:`coding-guide` for more information on contribution guidelines. + +Docker +------ + +The Deckhand Dockerfile is located in ``/images/deckhand`` along with any +artifacts built specifically to enable the container image. Make targets are +used for generating and testing the artifacts. + +* ``make images`` - Build the Deckhand Docker image. + +Helm +---- + +The Deckhand Helm chart is located in ``/charts/deckhand``. Local testing +currently only supports linting and previewing the rendered artifacts. +Richer functional chart testing is a TODO. + +* ``make charts`` - Pull down dependencies for the Deckhand charts and package + everything into a ``.tgz`` file. +* ``make helm_lint`` - Lint the Helm charts. +* ``make dry-run`` - Render the chart and output the Kubernetes manifest YAML + documents. + +Testing +======= + +All Deckhand tests are nested under ``/deckhand/tests``. + +Deckhand comes equipped with a number of +`tox `_ targets for running unit and +functional tests. See :ref:`development-utilities` for a list of commands. + +See :ref:`testing` for more information on testing guidelines. diff --git a/doc/source/diagrams/architecture-pegleg.uml b/doc/source/diagrams/architecture-pegleg.uml new file mode 100644 index 00000000..0d4017f7 --- /dev/null +++ b/doc/source/diagrams/architecture-pegleg.uml @@ -0,0 +1,39 @@ +' PlantUML file to generate the architecture component diagram +@startuml + +actor "DE" +entity "HTTP" +entity "SQL" +entity "WSGI" + +frame "Deckhand" { + [Control] ..> [Engine] +} + +frame "Pegleg" { + [Pegleg CLI] ..> [Pegleg Engine] +} + +frame "OpenStack Components" { + [Barbican] ..> [Secret Store Back-end] + [Keystone] +} + +database "PostgreSQL" { + SQL - [deckhand_db] +} + +HTTP - [uWSGI] +[uWSGI] --> [Keystone Middleware] +[Keystone Middleware] --> WSGI +WSGI - [Control] +[Engine] --> [SQL] +[Control] --> [Barbican] +[Pegleg Engine] --> [Engine] +DE -> [Pegleg CLI] + +legend right + DE: Deployment Engineer +endlegend + +@enduml diff --git a/doc/source/diagrams/architecture.uml b/doc/source/diagrams/architecture.uml new file mode 100644 index 00000000..9b44c32a --- /dev/null +++ b/doc/source/diagrams/architecture.uml @@ -0,0 +1,28 @@ +' PlantUML file to generate the architecture component diagram +@startuml + +entity "HTTP" +entity "SQL" +entity "WSGI" + +frame "Deckhand" { + [Control] ..> [Engine] +} + +frame "OpenStack Components" { + [Barbican] ..> [Secret Store Back-end] + [Keystone] +} + +database "PostgreSQL" { + SQL - [deckhand_db] +} + +HTTP - [uWSGI] +[uWSGI] --> [Keystone Middleware] +[Keystone Middleware] --> WSGI +WSGI - [Control] +[Engine] --> [SQL] +[Control] --> [Barbican] + +@enduml diff --git a/doc/source/getting-started.rst b/doc/source/getting-started.rst index 271ab34b..fcb59452 100644 --- a/doc/source/getting-started.rst +++ b/doc/source/getting-started.rst @@ -206,6 +206,8 @@ After, from the command line, execute: -v $CONF_DIR:/etc/deckhand \ quay.io/attcomdev/deckhand:latest server +.. _development-utilities: + Development Utilities --------------------- @@ -244,6 +246,9 @@ deployment, execute (respectively):: .. _Bandit: https://github.com/openstack/bandit +For additional commands, reference the ``tox.ini`` file for a list of all +the jobs. + Database Model Updates ---------------------- diff --git a/doc/source/images/architecture-pegleg.png b/doc/source/images/architecture-pegleg.png new file mode 100644 index 00000000..ca821a8f Binary files /dev/null and b/doc/source/images/architecture-pegleg.png differ diff --git a/doc/source/images/architecture.png b/doc/source/images/architecture.png new file mode 100644 index 00000000..d82baa0b Binary files /dev/null and b/doc/source/images/architecture.png differ diff --git a/doc/source/index.rst b/doc/source/index.rst index e43515c8..a56cf909 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -60,6 +60,8 @@ Developer's Guide :maxdepth: 2 HACKING + REVIEWING + developer-overview policy-enforcement testing diff --git a/doc/source/revision-history.rst b/doc/source/revision-history.rst index abd4f829..20188868 100644 --- a/doc/source/revision-history.rst +++ b/doc/source/revision-history.rst @@ -29,6 +29,8 @@ documents. Revisions can be thought of as commits in a linear git history, thus looking at a revision includes all content from previous revisions. +.. _revision-diffing: + Revision Diffing ---------------- diff --git a/doc/source/testing.rst b/doc/source/testing.rst index 03047a0c..f576291b 100644 --- a/doc/source/testing.rst +++ b/doc/source/testing.rst @@ -13,11 +13,13 @@ See the License for the specific language governing permissions and limitations under the License. +.. _testing: + ======= Testing ======= -.. warning:: +.. note:: Deckhand has only been tested against a Ubuntu 16.04 environment. The guide below assumes the user is using Ubuntu. @@ -100,11 +102,11 @@ Overview Deckhand uses `gabbi `_ as its functional testing framework. Functional tests can be executed via:: - $ tox -e functional + $ tox -e functional-dev You can also run a subset of tests via a regex:: - $ tox -e functional -- gabbi.suitemaker.test_gabbi_document-crud-success-multi-bucket + $ tox -e functional-dev -- gabbi.suitemaker.test_gabbi_document-crud-success-multi-bucket The command executes ``tools/functional-tests.sh`` which: @@ -142,7 +144,7 @@ testing. To test Deckhand against a containerized image, run, for example: :: export DECKHAND_IMAGE=quay.io/attcomdev/deckhand:latest - tox -e functional + tox -e functional-dev Which will result in the following script output: diff --git a/tools/build-docs.sh b/tools/build-docs.sh new file mode 100755 index 00000000..8cd81e74 --- /dev/null +++ b/tools/build-docs.sh @@ -0,0 +1,11 @@ +#!/usr/bin/env bash + +# Builds documentation and generates documentation diagrams from .uml +# files. Must be run from root project directory. + +set -ex +rm -rf doc/build +rm -rf releasenotes/build +sphinx-build -W -b html doc/source doc/build/html +python -m plantuml doc/source/diagrams/*.uml +mv doc/source/diagrams/*.png doc/source/images diff --git a/tools/whitespace-linter.sh b/tools/whitespace-linter.sh index 62ad0b60..a023399c 100755 --- a/tools/whitespace-linter.sh +++ b/tools/whitespace-linter.sh @@ -6,6 +6,7 @@ RES=$(find . \ -not -path "*/*.egg-info/*" \ -not -path "*/releasenotes/build/*" \ -not -path "*/doc/build/*" \ + -not -path "*/doc/source/images/*" \ -not -name "*.tgz" \ -not -name "*.html" \ -not -name "*.pyc" \ diff --git a/tox.ini b/tox.ini index 50a07df8..6dd700b3 100644 --- a/tox.ini +++ b/tox.ini @@ -118,13 +118,10 @@ ignore = H405 exclude = .venv,.git,.tox,dist,*lib/python*,*egg,build,releasenotes,doc,alembic/versions [testenv:docs] -deps = -r{toxinidir}/doc/requirements.txt +deps = + -r{toxinidir}/doc/requirements.txt commands = - rm -rf doc/build - rm -rf releasenotes/build - sphinx-build -W -b html doc/source doc/build/html -whitelist_externals = - rm + {toxinidir}/tools/build-docs.sh [testenv:releasenotes] deps = -r{toxinidir}/doc/requirements.txt