Merge "Add spec on creating Docker images from Monasca repos"

This commit is contained in:
Zuul 2018-06-18 14:50:14 +00:00 committed by Gerrit Code Review
commit 2d9f0805a5
1 changed files with 342 additions and 0 deletions

View File

@ -0,0 +1,342 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================
Monasca services in Docker
==========================
The main task of this story is to move building and publishing Docker images
of all Monasca components from https://github.com/monasca/monasca-docker
to their respective OpenStack repositories.
Problem description
===================
At the moment Docker images for Monasca components are built from Dockerfiles
provided in `monasca-docker repository <https://github.com/monasca/monasca-docker>`_.
The process of building and publishing the images has to be explicitly
triggered and is described
`here <https://github.com/monasca/monasca-docker/blob/master/CONTRIBUTING.md#releasing-changes>`_.
Due to the separation of image definitions and the actual upstream code they
can easily diverge.
Use Cases
---------
Have supported and standardized option to deploy Monasca services with Docker.
From the development point of view very little will change.
A developer is writing code and putting it into Gerrit, then automatic process
test if Docker image will be built properly.
The images can be used in integration tests running in OpenStack CI.
End User should see no change.
Deployer has one more supported way of deploying Monasca.
Proposed change
===============
Every repository with each component would have additional `docker` folder
that would contain all necessary files for building Docker image.
Example files:
* `Dockerfile`
* `README`
* Starting script that will template needed configuration files, wait for other
needed components to be available (like Keystone) and start service
on Docker image run.
* Templates for configuration files, bootstrap scripts for configuring database
schemas etc..
The process should be started after commit land in git repository (so all
tests passing).
The goal is a fully automatic process without the need for human intervention.
Every image should have an easy and standardized way of getting information
from what version of Monasca component it was built (definitely needed for
`master` and `latest` tags).
Every Dockerfile will be in a separate repository so there is a need
for standardization of these files so that they look mostly the same.
There are linters for Dockerfiles, one need to be chosen and added to
the testing process (will help with standardization of the Dockerfiles,
check Kolla jobs).
Images will be build with Python 3.5. If any Monasca service does not support
Python 3, we will wait for support before creating container for it.
Images will be published to https://hub.docker.com/u/monasca/
Each image should have README file in RST format with information about running
specific component.
We will be supporting Docker actual stable version and 2 previous versions.
All stable branches will have specific versions of Docker that they are build
with and this versions will be frozen at the moment of branching them from
master. Master branch will be build with 3 newest Docker versions.
All images will be based on custom `monasca-base` image that is using official
Python on Alpine Linux to conserve disc space. We will be using ideas for
minimal configuration from other base image examples (like based on Ubuntu
https://github.com/phusion/baseimage-docker or based on Alpine
https://github.com/blacklabelops/baseimages).
All images should be created in a way that will try to make them as small
as possible without compromising on functionality. Possible ways of making them
smaller could be found in articles like the following
https://blog.codeship.com/alpine-based-docker-images-make-difference-real-world-apps/
https://docs.docker.com/develop/develop-images/dockerfile_best-practices/
All images standardize on Jinja2 for templating configuration files with the
following tool:
* https://github.com/Aisbergg/python-templer - Python3 command-line tool for
templating in shell-scripts, leveraging the Jinja2 library
* Allows to use environment variables
* YAML data sources supported
* LGPL license but we are not linking against it, only using CLI
Alternatives
------------
As an example of the CI configuration, we could use Kolla process
of creating Docker images:
https://github.com/openstack/kolla
https://hub.docker.com/u/kolla/
It is though much too complex for what we need, so should be used just
as a reference.
For templating:
* https://github.com/wrouesnel/p2cli - command line tool for rendering pongo2
(Django-syntax like, similar to Jinja2) templates (Go binary, 4x smaller
size if the image does not already have Python)
* Allows to use environment variables
* YAML, JSON data sources supported
* https://github.com/jwilder/dockerize - Utility to simplify running
applications in docker containers (Go binary)
* Generate application configuration files at container startup time from
templates (Go text/template) and container environment variables
* Tail multiple log files to stdout and/or stderr
* Wait for other services to be available using TCP, HTTP(S), unix before
starting the main process.
Data model impact
-----------------
None
REST API impact
---------------
None
Security impact
---------------
All services will be closed in Docker container.
Other end user impact
---------------------
None
Performance Impact
------------------
Because of additional Docker layer services could run slower.
Other deployer impact
---------------------
Deployment should be easier as deployer would need to create configuration
and services with all dependencies will be enclosed in Docker images.
Developer impact
----------------
Features adding new dependencies or changing the way Monasca components
are installed would have to be reflected in Docker image definitions.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
dobroslaw-zybort <dobroslaw.zybort@ts.fujitsu.com>
Work Items
----------
What is needed:
* Building test images for every change in Gerrit.
* Automated process of building images on every git commit and pushing them
to the Docker Hub.
* Automated process of building images on every tag and pushing them
to the Docker Hub.
* Automated process of building images on OpenStack releases and pushing
them to the Docker Hub (could be the second step).
* Adding labels with source code base information of the image (e.g.
build-date, commit-sha1).
* Running integration tests on each code commit to verify the Docker binary.
Single node deployment setup with Docker Compose.
* Tempest tests.
* Smoke tests.
Five types of tags on Docker Hub:
* On every new release/tag.
* `latest` pointing to the last tag.
* `master` pointing to the last git commit - useful for testing last.
changes/fixes before release.
* Tags for the last commit on stable OpenStack versions (`queens`, `rocky` etc.) -
useful for testing last changes/fixes before release.
Optional:
* Build all images with Python 3.6 and test stability.
* Run multi node deployment integration tests with Kubernetes Helm.
Needs to remember (to make maintenance easier):
* Using proper init process.
* Cleaning after the installation process.
* Same name of starting script in all repositories.
* Use same config templating mechanism in all repositories.
* All components should be downloaded from Git repository and have a possibility
to change branch for building Docker image (useful for testing changes
proposed on Gerrit).
Non objectives of this story (could be next steps):
* Automated process of building images on specific past commits and pushing
them to the Docker Hub.
* Migrate Devstack to Docker images of Monasca.
Dependencies
============
What should be moved to where:
* https://github.com/monasca/monasca-docker/tree/master/monasca-agent-base =>
https://github.com/openstack/monasca-agent
* https://github.com/monasca/monasca-docker/tree/master/monasca-agent-collector =>
https://github.com/openstack/monasca-agent
* https://github.com/monasca/monasca-docker/tree/master/monasca-agent-forwarder =>
https://github.com/openstack/monasca-agent
* https://github.com/monasca/monasca-docker/tree/master/monasca-api-python =>
https://github.com/openstack/monasca-api
* https://github.com/monasca/monasca-docker/tree/master/monasca-client =>
https://github.com/openstack/python-monascaclient
* https://github.com/monasca/monasca-docker/tree/master/monasca-log-api =>
https://github.com/openstack/monasca-log-api
* https://github.com/monasca-docker/monasca-notification =>
https://github.com/openstack/monasca-notification
* https://github.com/monasca/monasca-docker/tree/master/monasca-persister-python =>
https://github.com/openstack/monasca-persister
* https://github.com/monasca/monasca-docker/tree/master/monasca-python =>
https://github.com/openstack/monasca-common
* https://github.com/monasca/monasca-docker/tree/master/monasca-thresh =>
https://github.com/openstack/monasca-thresh
* will need also https://github.com/monasca/monasca-docker/tree/master/storm =>
https://github.com/openstack/monasca-thresh
* https://github.com/monasca/monasca-docker/tree/master/tempest-tests =>
https://github.com/openstack/monasca-tempest-plugin
Not in scope:
* https://github.com/monasca/monasca-docker/monasca-alarms - This image
contains a container that can be used to create Monasca Notifications
and Alarm Definitions.
* https://github.com/monasca/monasca-docker/monasca-log-agent - Logstash
output monasca_log_api plugin.
* https://github.com/monasca/monasca-docker/monasca-log-metrics - Contains
Logstash configuration to transform logs into metrics based on log's severity.
* https://github.com/monasca/monasca-docker/monasca-log-persister - Contains
Logstash configuration to save logs inside **log-db** (i.e. ElasticSearch).
* https://github.com/monasca/monasca-docker/monasca-log-transformer - Image
contains Logstash configuration to detect log's severity.
Testing
=======
At this moment CI for https://github.com/monasca/monasca-docker run two types
of tests on each change:
* tempest-tests https://github.com/monasca/monasca-docker/tree/master/tempest-tests
* smoke-tests https://github.com/monasca/monasca-docker/tree/master/smoke-tests
* https://github.com/monasca/smoke-test
Both of them, at the moment, are running on metrics part of the Monasca stack.
Tests should consider if tested service started and is behaving correctly
in built and running Docker container. We need to decide if we want to run
all tests on the whole Monasca stack on every change or if we should create
some smaller tests for each separate service.
Documentation Impact
====================
Basic installation instructions should be added here [1] and published
to https://docs.openstack.org
[1] https://git.openstack.org/cgit/openstack/monasca-api/tree/doc/source/install
Now high level documentation is stored in:
https://github.com/monasca/monasca-docker/tree/master/docs
Separate images also have `README.md` files that give lower level information.
Documentation should contain all necessary information how to configure and run
all services.
References
==========
* https://github.com/monasca/monasca-docker
* http://eavesdrop.openstack.org/meetings/monasca/2018/monasca.2018-01-10-15.00.log.html
History
=======
Optional section intended to be used each time the spec is updated to describe
the new design, API or any database schema updated. Useful to let reader
understand what's happened at the time.
.. list-table:: Revisions
:header-rows: 1
* - Release Name
- Description
* - Rocky
- Introduced