Merge "docs: Add developer overview documentation"

This commit is contained in:
Zuul 2018-06-29 18:00:27 +00:00 committed by Gerrit Code Review
commit 59182380e3
16 changed files with 288 additions and 13 deletions

38
REVIEWING.rst Normal file
View File

@ -0,0 +1,38 @@
Reviewing Deckhand Code
=======================
To start read the `OpenStack Common Review Checklist
<https://docs.openstack.org/infra/manual/developers.html#peer-review>`_
Unit Tests
----------
For any change that adds new functionality to either common functionality or
fixes a bug unit tests are required. This is to ensure we don't introduce
future regressions and to test conditions which we may not hit in the gate
runs.
Functional Tests
----------------
For any change that adds major new functionality functional tests are required.
This is to ensure that the Deckhand API follows the contract it promises.
In addition, functional tests are run against the Deckhand container, which
uses an image built from the latest source code to validate the integrity
of the image.
Deprecated Code
---------------
Deprecated code should go through a deprecation cycle -- long enough for other
Airship projects to modify their code base to reference new code. Features,
APIs or configuration options are marked deprecated in the code. Appropriate
warnings will be sent to the end user, operator or library user.
When to approve
---------------
* Every patch needs two +2s before being approved.
* Its OK to hold off on an approval until a subject matter expert reviews it.
* If a patch has already been approved but requires a trivial rebase to merge,
you do not have to wait for a second +2, since the patch has already had
two +2s.

View File

@ -4,6 +4,7 @@
sphinx>=1.6.2 # BSD
sphinx_rtd_theme==0.2.4
reno>=2.5.0 # Apache-2.0
plantuml
# NOTE(fmontei): The requirement below is only included because readthedocs
# depends on it to work, since it only takes in one requirements file.

View File

@ -13,7 +13,10 @@
See the License for the specific language governing permissions and
limitations under the License.
=======
Hacking
=======
.. _coding-guide:
=====================
Deckhand Coding Guide
=====================
.. include:: ../../HACKING.rst

5
doc/source/REVIEWING.rst Normal file
View File

@ -0,0 +1,5 @@
=======================
Reviewing Deckhand Code
=======================
.. include:: ../../REVIEWING.rst

View File

@ -0,0 +1,141 @@
..
Copyright 2018 AT&T Intellectual Property.
All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.
==============================
Developer Overview of Deckhand
==============================
The core objective of Deckhand is to provide storage, rendering, validation and
version control for declarative YAML documents. Deckhand ingests raw,
Airship-formatted documents and outputs fully rendered documents to other
Airship components.
Architecture
============
.. image:: images/architecture.png
:alt: High level architecture of Deckhand
From a high-level perspective, Deckhand consists of a RESTful API, a document
rendering engine, and a PostgreSQL relational database for document storage.
Deckhand ingests Airship-formatted documents, validates them, and stores them
in its database for future processing. On demand, Deckhand will fully render
the documents, after which they can be consumed by the other Airship
components.
Deckhand uses Barbican to securely storage sensitive document data.
`Pegleg <http://airship-pegleg.readthedocs.io/en/latest/>`_ in effect provides
Deckhand with a CLI, which facilitates communication with Deckhand.
.. image:: images/architecture-pegleg.png
:alt: High level architecture of Deckhand
Components
==========
control
-------
The ``control`` module is simply the RESTful API. It is based on the
`Falcon Framework <https://falconframework.org/>`_ and utilizes
`oslo.policy <https://docs.openstack.org/oslo.policy/latest/>`_
for RBAC enforcement of the API endpoints. The normal deployment of Deckhand
uses `uWSGI <http://uwsgi-docs.readthedocs.io/en/latest/>`_ and PasteDeploy
to build a pipeline that includes Keystone Middleware for authentication
and role decoration of the request.
The ``control`` module is also responsible for communicating with
`Barbican <https://docs.openstack.org/barbican/latest/>`_, which it uses to
store and retrieve document :ref:`secrets <encryption>`, which it passes to the
``engine`` module for :ref:`rendering`.
engine
------
The ``engine`` module is the interface responsible for all
:ref:`rendering`. Rendering consists of applying a series of algorithms to the
documents, including: topological sorting, :ref:`layering`,
:ref:`substitution`, and :ref:`replacement`.
db
--
The ``db`` module is responsible for implementing the database tables needed
to store all Airship documents. The module also realizes version control and
:ref:`revision-diffing` functionality.
client
------
The API client library provides an interface for other services to communicate
with Deckhand's API. Requires
`Keystone <https://docs.openstack.org/keystone/latest/>`_ authentication to
use.
Developer Workflow
==================
Because Airship is a container-centric platform, the developer workflow heavily
utilizes containers for testing and publishing. It also requires Deckhand to
produce multiple artifacts that are related, but separate: the Python package,
the Docker image and the Helm chart. The code is published via the
Docker image artifact.
Deckhand strives to conform to the
`Airship coding conventions <http://airshipit.readthedocs.io/en/latest/conventions.html>`_.
Python
------
The Deckhand code base lives under ``/deckhand``. Deckhand supports py27
and py35 interpreters. Once OpenStack deprecates usage of py27, Deckhand
will as well.
See :ref:`coding-guide` for more information on contribution guidelines.
Docker
------
The Deckhand Dockerfile is located in ``/images/deckhand`` along with any
artifacts built specifically to enable the container image. Make targets are
used for generating and testing the artifacts.
* ``make images`` - Build the Deckhand Docker image.
Helm
----
The Deckhand Helm chart is located in ``/charts/deckhand``. Local testing
currently only supports linting and previewing the rendered artifacts.
Richer functional chart testing is a TODO.
* ``make charts`` - Pull down dependencies for the Deckhand charts and package
everything into a ``.tgz`` file.
* ``make helm_lint`` - Lint the Helm charts.
* ``make dry-run`` - Render the chart and output the Kubernetes manifest YAML
documents.
Testing
=======
All Deckhand tests are nested under ``/deckhand/tests``.
Deckhand comes equipped with a number of
`tox <https://tox.readthedocs.io/en/latest/>`_ targets for running unit and
functional tests. See :ref:`development-utilities` for a list of commands.
See :ref:`testing` for more information on testing guidelines.

View File

@ -0,0 +1,39 @@
' PlantUML file to generate the architecture component diagram
@startuml
actor "DE"
entity "HTTP"
entity "SQL"
entity "WSGI"
frame "Deckhand" {
[Control] ..> [Engine]
}
frame "Pegleg" {
[Pegleg CLI] ..> [Pegleg Engine]
}
frame "OpenStack Components" {
[Barbican] ..> [Secret Store Back-end]
[Keystone]
}
database "PostgreSQL" {
SQL - [deckhand_db]
}
HTTP - [uWSGI]
[uWSGI] --> [Keystone Middleware]
[Keystone Middleware] --> WSGI
WSGI - [Control]
[Engine] --> [SQL]
[Control] --> [Barbican]
[Pegleg Engine] --> [Engine]
DE -> [Pegleg CLI]
legend right
DE: Deployment Engineer
endlegend
@enduml

View File

@ -0,0 +1,28 @@
' PlantUML file to generate the architecture component diagram
@startuml
entity "HTTP"
entity "SQL"
entity "WSGI"
frame "Deckhand" {
[Control] ..> [Engine]
}
frame "OpenStack Components" {
[Barbican] ..> [Secret Store Back-end]
[Keystone]
}
database "PostgreSQL" {
SQL - [deckhand_db]
}
HTTP - [uWSGI]
[uWSGI] --> [Keystone Middleware]
[Keystone Middleware] --> WSGI
WSGI - [Control]
[Engine] --> [SQL]
[Control] --> [Barbican]
@enduml

View File

@ -206,6 +206,8 @@ After, from the command line, execute:
-v $CONF_DIR:/etc/deckhand \
quay.io/attcomdev/deckhand:latest server
.. _development-utilities:
Development Utilities
---------------------
@ -244,6 +246,9 @@ deployment, execute (respectively)::
.. _Bandit: https://github.com/openstack/bandit
For additional commands, reference the ``tox.ini`` file for a list of all
the jobs.
Database Model Updates
----------------------

Binary file not shown.

After

Width:  |  Height:  |  Size: 37 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 23 KiB

View File

@ -60,6 +60,8 @@ Developer's Guide
:maxdepth: 2
HACKING
REVIEWING
developer-overview
policy-enforcement
testing

View File

@ -29,6 +29,8 @@ documents.
Revisions can be thought of as commits in a linear git history, thus looking
at a revision includes all content from previous revisions.
.. _revision-diffing:
Revision Diffing
----------------

View File

@ -13,11 +13,13 @@
See the License for the specific language governing permissions and
limitations under the License.
.. _testing:
=======
Testing
=======
.. warning::
.. note::
Deckhand has only been tested against a Ubuntu 16.04 environment. The guide
below assumes the user is using Ubuntu.
@ -100,11 +102,11 @@ Overview
Deckhand uses `gabbi <https://github.com/cdent/gabbi>`_ as its functional
testing framework. Functional tests can be executed via::
$ tox -e functional
$ tox -e functional-dev
You can also run a subset of tests via a regex::
$ tox -e functional -- gabbi.suitemaker.test_gabbi_document-crud-success-multi-bucket
$ tox -e functional-dev -- gabbi.suitemaker.test_gabbi_document-crud-success-multi-bucket
The command executes ``tools/functional-tests.sh`` which:
@ -142,7 +144,7 @@ testing. To test Deckhand against a containerized image, run, for example:
::
export DECKHAND_IMAGE=quay.io/attcomdev/deckhand:latest
tox -e functional
tox -e functional-dev
Which will result in the following script output:

11
tools/build-docs.sh Executable file
View File

@ -0,0 +1,11 @@
#!/usr/bin/env bash
# Builds documentation and generates documentation diagrams from .uml
# files. Must be run from root project directory.
set -ex
rm -rf doc/build
rm -rf releasenotes/build
sphinx-build -W -b html doc/source doc/build/html
python -m plantuml doc/source/diagrams/*.uml
mv doc/source/diagrams/*.png doc/source/images

View File

@ -6,6 +6,7 @@ RES=$(find . \
-not -path "*/*.egg-info/*" \
-not -path "*/releasenotes/build/*" \
-not -path "*/doc/build/*" \
-not -path "*/doc/source/images/*" \
-not -name "*.tgz" \
-not -name "*.html" \
-not -name "*.pyc" \

View File

@ -118,13 +118,10 @@ ignore = H405
exclude = .venv,.git,.tox,dist,*lib/python*,*egg,build,releasenotes,doc,alembic/versions
[testenv:docs]
deps = -r{toxinidir}/doc/requirements.txt
deps =
-r{toxinidir}/doc/requirements.txt
commands =
rm -rf doc/build
rm -rf releasenotes/build
sphinx-build -W -b html doc/source doc/build/html
whitelist_externals =
rm
{toxinidir}/tools/build-docs.sh
[testenv:releasenotes]
deps = -r{toxinidir}/doc/requirements.txt