Import all implemented specs from magnum repo

* Add all implemented specs to specs/implemented.
* Create ocata directory to store the corresponding specs.

Change-Id: I36da3a26ece3b6f7d27b7af80460303f4a6c4226
This commit is contained in:
Spyros Trigazis 2016-11-09 16:28:28 +01:00
parent 74da0acf5d
commit 8b0a99e44b
11 changed files with 3167 additions and 0 deletions

1
doc/source/specs Symbolic link
View File

@ -0,0 +1 @@
../../specs

View File

@ -0,0 +1,452 @@
=================================
Asynchronous Container Operations
=================================
Launchpad blueprint:
https://blueprints.launchpad.net/magnum/+spec/async-container-operations
At present, container operations are done in a synchronous way, end-to-end.
This model does not scale well, and incurs a penalty on the client to be
stuck till the end of completion of the operation.
Problem Description
-------------------
At present Magnum-Conductor executes the container operation as part of
processing the request forwarded from Magnum-API. For
container-create, if the image needs to be pulled down, it may take
a while depending on the responsiveness of the registry, which can be a
substantial delay. At the same time, experiments suggest that even for
pre-pulled image, the time taken by each operations, namely
create/start/delete, are in the same order, as it involves complete turn
around between the magnum-client and the COE-API, via Magnum-API and
Magnum-Conductor[1].
Use Cases
---------
For wider enterprise adoption of Magnum, we need it to scale better.
For that we need to replace some of these synchronous behaviors with
suitable alternative of asynchronous implementation.
To understand the use-case better, we can have a look at the average
time spent during container operations, as noted at[1].
Proposed Changes
----------------
The design has been discussed over the ML[6]. The conclusions have been kept
on the 'whiteboard' of the Blueprint.
The amount of code change is expected to be significant. To ease the
process of adoption, code review, functional tests, an approach of phased
implementation may be required. We can define the scope of the three phases of
the implementation as follows -
* Phase-0 will bring in the basic feature of asynchronous mode of operation in
Magnum - (A) from API to Conductor and (B) from Conductor to COE-API. During
phase-0, this mode will be optional through configuration.
Both the communications of (A) and (B) are proposed to be made asynchronous
to achieve the best of it. If we do (A) alone, it does not gain us much, as
(B) takes up the higher cycles of the operation. If we do (B) alone, it does
not make sense, as (A) will synchronously wait for no meaningful data.
* Phase-1 will concentrate on making the feature persistent to address various
scenarios of conductor restart, worker failure etc. We will support this
feature for multiple Conductor-workers in this phase.
* Phase-2 will select asynchronous mode of operation as the default mode. At
the same time, we can evaluate to drop the code for synchronous mode, too.
Phase-0 is required as a meaningful temporary step, to establish the
importance and tangible benefits of phase-1. This is also to serve as a
proof-of-concept at a lower cost of code changes with a configurable option.
This will enable developers and operators to have a taste of the feature,
before bringing in the heavier dependencies and changes proposed in phase-1.
A reference implementation for the phase-0 items, has been put for review[2].
Following is the summary of the design -
1. Configurable mode of operation - async
-----------------------------------------
For ease of adoption, the async_mode of communication between API-conductor,
conductor-COE in magnum, can be controlled using a configuration option. So
the code-path for sync mode and async mode would co-exist for now. To achieve
this with minimal/no code duplication and cleaner interface, we are using
openstack/futurist[4]. Futurist interface hides the details of type of executor
being used. In case of async configuration, a greenthreadpool of configured
poolsize gets created. Here is a sample of how the config would look
like: ::
[DEFAULT]
async_enable = False
[conductor]
async_threadpool_max_workers = 64
Futurist library is used in oslo.messaging. Thus, it is used by almost all
OpenStack projects, in effect. Futurist is very useful to run same code
under different execution model and hence saving potential duplication of
code.
2. Type of operations
---------------------
There are two classes of container operations - one that can be made async,
namely create/delete/start/stop/pause/unpause/reboot, which do not need data
about the container in return. The other type requires data, namely
container-logs. For async-type container-operations, magnum-API will be
using 'cast' instead of 'call' from oslo_messaging[5].
'cast' from oslo.messaging.rpcclient is used to invoke a method and return
immediately, whereas 'call' invokes a method and waits for a reply. While
operating in asynchronous mode, it is intuitive to use cast method, as the
result of the response may not be available immediately.
Magnum-api first fetches the details of a container, by doing
'get_rpc_resource'. This function uses magnum objects. Hence, this function
uses a 'call' method underneath. Once, magnum-api gets back the details,
it issues the container operation next, using another 'call' method.
The above proposal is to replace the second 'call' with 'cast'.
If user issues a container operation, when there is no listening
conductor (because of process failure), there will be a RPC timeout at the
first 'call' method. In this case, user will observe the request to
get blocked at client and finally fail with HTTP 500 ERROR, after the RPC
timeout, which is 60 seconds by default. This behavior is independent of the
usage of 'cast' or 'call' for the second message, mentioned above. This
behavior does not influence our design, but it is documented here for clarity
of understanding.
3. Ensuring the order of execution - Phase-0
--------------------------------------------
Magnum-conductor needs to ensure that for a given bay and given container,
the operations are executed in sequence. In phase-0, we want to demonstrate
how asynchronous behavior helps scaling. Asynchronous mode of container
operations would be supported for single magnum-conductor scenario, in
phase-0. If magnum-conductor crashes, there will be no recovery for the
operations accepted earlier - which means no persistence in phase-0, for
operations accepted by magnum-conductor. Multiple conductor scenario and
persistence will be addressed in phase-1 [please refer to the next section
for further details]. If COE crashes or does not respond, the error will be
detected, as it happens in sync mode, and reflected on the container-status.
Magnum-conductor will maintain a job-queue. Job-queue is indexed by bay-id and
container-id. A job-queue entry would contain the sequence of operations
requested for a given bay-id and container-id, in temporal order. A
greenthread will execute the tasks/operations in order for a given job-queue
entry, till the queue empties. Using a greenthread in this fashion saves us
from the cost and complexity of locking, along with functional correctness.
When request for new operation comes in, it gets appended to the corresponding
queue entry.
For a sequence of container operations, if an intermediate operation fails,
we will stop continuing the sequence. The community feels more confident to
start with this strictly defensive policy[17]. The failure will be logged
and saved into the container-object, which will help an operator be informed
better about the result of the sequence of container operations. We may revisit
this policy later, if we think it is too restrictive.
4. Ensuring the order of execution - phase-1
--------------------------------------------
The goal is to execute requests for a given bay and a given container in
sequence. In phase-1, we want to address persistence and capability of
supporting multiple magnum-conductor processes. To achieve this, we will
reuse the concepts laid out in phase-0 and use a standard library.
We propose to use taskflow[7] for this implementation. Magnum-conductors
will consume the AMQP message and post a task[8] on a taskflow jobboard[9].
Greenthreads from magnum-conductors would subscribe to the taskflow
jobboard as taskflow-conductors[10]. Taskflow jobboard is maintained with
a choice of persistent backend[11]. This will help address the concern of
persistence for accepted operations, when a conductor crashes. Taskflow
will ensure that tasks, namely container operations, in a job, namely a
sequence of operations for a given bay and container, would execute in
sequence. We can easily notice that some of the concepts used in phase-0
are reused as it is. For example, job-queue maps to jobboard here, use of
greenthread maps to the conductor concept of taskflow. Hence, we expect easier
migration from phase-0 to phase-1, with the choice of taskflow.
For taskflow jobboard[11], the available choices of backend are Zookeeper and
Redis. But, we plan to use MySQL as default choice of backend, for magnum
conductor jobboard use-case. This support will be added to taskflow. Later,
we may choose to support the flexibility of other backends like ZK/Redis via
configuration. But, phase-1 will keep the implementation simple with MySQL
backend and revisit this, if required.
Let's consider the scenarios of Conductor crashing -
- If a task is added to jobboard, and conductor crashes after that,
taskflow can assign a particular job to any available greenthread agents
from other conductor instances. If the system was running with single
magnum-conductor, it will wait for the conductor to come back and join.
- A task is picked up and magnum-conductor crashes. In this case, the task
is not complete from jobboard point-of-view. As taskflow detects the
conductor going away, it assigns another available conductor.
- When conductor picks up a message from AMQP, it will acknowledge AMQP,
only after persisting it to jobboard. This will prevent losing the message,
if conductor crashes after picking up the message from AMQP. Explicit
acknowledgement from application may use NotificationResult.HANDLED[12]
to AMQP. We may use the at-least-one-guarantee[13] feature in
oslo.messaging[14], as it becomes available.
To summarize some of the important outcomes of this proposal -
- A taskflow job represents the sequence of container operations on a given
bay and given container. At a given point of time, the sequence may contain
a single or multiple operations.
- There will be a single jobboard for all conductors.
- Task-flow conductors are multiple greenthreads from a given
magnum-conductor.
- Taskflow-conductor will run in 'blocking' mode[15], as those greenthreads
have no other job than claiming and executing the jobs from jobboard.
- Individual jobs are supposed to maintain a temporal sequence. So the
taskflow-engine would be 'serial'[16].
- The proposed model for a 'job' is to consist of a temporal sequence of
'tasks' - operations on a given bay and a given container. Henceforth,
it is expected that when a given operation, namely container-create is in
progress, a request for container-start may come in. Adding the task to
the existing job is intuitive to maintain the sequence of operations.
To fit taskflow exactly into our use-case, we may need to do two enhancements
in taskflow -
- Supporting mysql plugin as a DB backend for jobboard. Support for redis
exists, so it will be similar.
We do not see any technical roadblock for adding mysql support for taskflow
jobboard. If the proposal does not get approved by taskflow team, we may have
to use redis, as an alternative option.
- Support for dynamically adding tasks to a job on jobboard. This also looks
feasible, as discussed over the #openstack-state-management [Unfortunately,
this channel is not logged, but if we agree in this direction, we can initiate
discussion over ML, too]
If taskflow team does not allow adding this feature, even though they have
agreed now, we will use the dependency feature in taskflow. We will explore
and elaborate this further, if it requires.
5. Status of progress
---------------------
The progress of execution of a container operation is reflected on the status
of a container as - 'create-in-progress', 'delete-in-progress' etc.
Alternatives
------------
Without an asynchronous implementation, Magnum will suffer from complaints
about poor scalability and slowness.
In this design, stack-lock[3] has been considered as an alternative to
taskflow. Following are the reasons for preferring taskflow over
stack-lock, as of now,
- Stack-lock used in Heat is not a library, so it will require making a copy
for Magnum, which is not desirable.
- Taskflow is relatively mature, well supported, feature-rich library.
- Taskflow has in-built capacity to scale out[in] as multiple conductors
can join in[out] the cluster.
- Taskflow has a failure detection and recovery mechanism. If a process
crashes, then worker threads from other conductor may continue the execution.
In this design, we describe futurist[4] as a choice of implementation. The
choice was to prevent duplication of code for async and sync mode. For this
purpose, we could not find any other solution to compare.
Data model impact
-----------------
Phase-0 has no data model impact. But phase-1 may introduce an additional
table into the Magnum database. As per the present proposal for using taskflow
in phase-1, we have to introduce a new table for jobboard under magnum db.
This table will be exposed to taskflow library as a persistent db plugin.
Alternatively, an implementation with stack-lock will also require an
introduction of a new table for stack-lock objects.
REST API impact
---------------
None.
Security impact
---------------
None.
Notifications impact
--------------------
None
Other end user impact
---------------------
None
Performance impact
------------------
Asynchronous mode of operation helps in scalability. Hence, it improves
responsiveness and reduces the turn around time in a significant
proportion. A small test on devstack, comparing both the modes,
demonstrate this with numbers.[1]
Other deployer impact
---------------------
None.
Developer impact
----------------
None
Implementation
--------------
Assignee(s)
-----------
Primary assignee
suro-patz(Surojit Pathak)
Work Items
----------
For phase-0
* Introduce config knob for asynchronous mode of container operations.
* Changes for Magnum-API to use CAST instead of CALL for operations eligible
for asynchronous mode.
* Implement the in-memory job-queue in Magnum conductor, and integrate futurist
library.
* Unit tests and functional tests for async mode.
* Documentation changes.
For phase-1
* Get the dependencies on taskflow being resolved.
* Introduce jobboard table into Magnum DB.
* Integrate taskflow in Magnum conductor to replace the in-memory job-queue
with taskflow jobboard. Also, we need conductor greenthreads to subscribe
as workers to the taskflow jobboard.
* Add unit tests and functional tests for persistence and multiple conductor
scenario.
* Documentation changes.
For phase-2
* We will promote asynchronous mode of operation as the default mode of
operation.
* We may decide to drop the code for synchronous mode and corresponding config.
* Documentation changes.
Dependencies
------------
For phase-1, if we choose to implement using taskflow, we need to get
following two features added to taskflow first -
* Ability to add new task to an existing job on jobboard.
* mysql plugin support as persistent DB.
Testing
-------
All the existing test cases are run to ensure async mode does not break them.
Additionally more functional tests and unit tests will be added specific to
async mode.
Documentation Impact
--------------------
Magnum documentation will include a description of the option for asynchronous
mode of container operations and its benefits. We will also add to
developer documentation on guideline for implementing a container operation in
both the modes - sync and async. We will add a section on 'how to debug
container operations in async mode'. The phase-0 and phase-1 implementation
and their support for single or multiple conductors will be clearly documented
for the operators.
References
----------
[1] - Execution time comparison between sync and async modes:
https://gist.github.com/surojit-pathak/2cbdad5b8bf5b569e755
[2] - Proposed change under review:
https://review.openstack.org/#/c/267134/
[3] - Heat's use of stacklock
http://docs.openstack.org/developer/heat/_modules/heat/engine/stack_lock.html
[4] - openstack/futurist
http://docs.openstack.org/developer/futurist/
[5] - openstack/oslo.messaging
http://docs.openstack.org/developer/oslo.messaging/rpcclient.html
[6] - ML discussion on the design
http://lists.openstack.org/pipermail/openstack-dev/2015-December/082524.html
[7] - Taskflow library
http://docs.openstack.org/developer/taskflow/
[8] - task in taskflow
http://docs.openstack.org/developer/taskflow/atoms.html#task
[9] - job and jobboard in taskflow
http://docs.openstack.org/developer/taskflow/jobs.html
[10] - conductor in taskflow
http://docs.openstack.org/developer/taskflow/conductors.html
[11] - persistent backend support in taskflow
http://docs.openstack.org/developer/taskflow/persistence.html
[12] - oslo.messaging notification handler
http://docs.openstack.org/developer/oslo.messaging/notification_listener.html
[13] - Blueprint for at-least-once-guarantee, oslo.messaging
https://blueprints.launchpad.net/oslo.messaging/+spec/at-least-once-guarantee
[14] - Patchset under review for at-least-once-guarantee, oslo.messaging
https://review.openstack.org/#/c/229186/
[15] - Taskflow blocking mode for conductor
http://docs.openstack.org/developer/taskflow/conductors.html#taskflow.conductors.backends.impl_executor.ExecutorConductor
[16] - Taskflow serial engine
http://docs.openstack.org/developer/taskflow/engines.html
[17] - Community feedback on policy to handle failure within a sequence
http://eavesdrop.openstack.org/irclogs/%23openstack-containers/%23openstack-containers.2016-03-08.log.html#t2016-03-08T20:41:17

View File

@ -0,0 +1,344 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
======================================
Container Orchestration Engine drivers
======================================
Launchpad blueprint:
https://blueprints.launchpad.net/magnum/+spec/bay-drivers
Container Orchestration Engines (COEs) are different systems for managing
containerized applications in a clustered environment, each having their own
conventions and ecosystems. Three of the most common, which also happen to be
supported in Magnum, are: Docker Swarm, Kubernetes, and Mesos. In order to
successfully serve developers, Magnum needs to be able to provision and manage
access to the latest COEs through its API in an effective and scalable way.
Problem description
===================
Magnum currently supports the three most popular COEs, but as more emerge and
existing ones change, it needs an effective and scalable way of managing
them over time.
One of the problems with the current implementation is that COE-specific logic,
such as Kubernetes replication controllers and services, is situated in the
core Magnum library and made available to users through the main API. Placing
COE-specific logic in a core API introduces tight coupling and forces
operators to work with an inflexible design.
By formalising a more modular and extensible architecture, Magnum will be
in a much better position to help operators and consumers satisfy custom
use-cases.
Use cases
---------
1. Extensibility. Contributors and maintainers need a suitable architecture to
house current and future COE implementations. Moving to a more extensible
architecture, where core classes delegate to drivers, provides a more
effective and elegant model for handling COE differences without the need
for tightly coupled and monkey-patched logic.
One of the key use cases is allowing operators to customise their
orchestration logic, such as modifying Heat templates or even using their
own tooling like Ansible. Moreover, operators will often expect to use a
custom distro image with lots of software pre-installed and many special
security requirements that is extremely difficult or impossible to do with
the current upstream templates. COE drivers solves these problems.
2. Maintainability. Moving to a modular architecture will be easier to manage
in the long-run because the responsibility of maintaining non-standard
implementations is shifted into the operator's domain. Maintaining the
default drivers which are packaged with Magnum will also be easier and
cleaner since logic is now demarcated from core codebase directories.
3. COE & Distro choice. In the community there has been a lot of discussion
about which distro and COE combination to support with the templates.
Having COE drivers allows for people or organizations to maintain
distro-specific implementations (e.g CentOS+Kubernetes).
4. Addresses dependency concerns. One of the direct results of
introducing a driver model is the ability to give operators more freedom
about choosing how Magnum integrates with the rest of their OpenStack
platform. For example, drivers would remove the necessity for users to
adopt Barbican for secret management.
5. Driver versioning. The new driver model allows operators to modify existing
drivers or creating custom ones, release new bay types based on the newer
version, and subsequently launch news bays running the updated
functionality. Existing bays which are based on older driver versions would
be unaffected in this process and would still be able to have lifecycle
operations performed on them. If one were to list their details from the
API, it would reference the old driver version. An operator can see which
driver version a bay type is based on through its ``driver`` value,
which is exposed through the API.
Proposed change
===============
1. The creation of new directory at the project root: ``./magnum/drivers``.
Each driver will house its own logic inside its own directory. Each distro
will house its own logic inside that driver directory. For example, the
Fedora Atomic distro using Swarm will have the following directory
structure:
::
drivers/
swarm_atomic_v1/
image/
...
templates/
...
api.py
driver.py
monitor.py
scale.py
template_def.py
version.py
The directory name should be a string which uniquely identifies the driver
and provides a descriptive reference. The driver version number and name are
provided in the manifest file and will be included in the bay metadata at
cluster build time.
There are two workflows for rolling out driver updates:
- if the change is relatively minor, they modify the files in the
existing driver directory and update the version number in the manifest
file.
- if the change is significant, they create a new directory
(either from scratch or by forking).
Further explanation of the three top-level files:
- an ``image`` directory is *optional* and should contain documentation
which tells users how to build the image and register it to glance. This
directory can also hold artifacts for building the image, for instance
diskimagebuilder elements, scripts, etc.
- a ``templates`` directory is *required* and will (for the forseeable
future) store Heat template YAML files. In the future drivers will allow
operators to use their own orchestration tools like Ansible.
- ``api.py`` is *optional*, and should contain the API controller which
handles custom API operations like Kubernetes RCs or Pods. It will be
this class which accepts HTTP requests and delegates to the Conductor. It
should contain a uniquely named class, such as ``SwarmAtomicXYZ``, which
extends from the core controller class. The COE class would have the
opportunity of overriding base methods if necessary.
- ``driver.py`` is *required*, and should contain the logic which maps
controller actions to COE interfaces. These include: ``bay_create``,
``bay_update``, ``bay_delete``, ``bay_rebuild``, ``bay_soft_reboot`` and
``bay_hard_reboot``.
- ``version.py`` is *required*, and should contain the version number of
the bay driver. This is defined by a ``version`` attribute and is
represented in the ``1.0.0`` format. It should also include a ``Driver``
attribute and should be a descriptive name such as ``swarm_atomic``.
Due to the varying nature of COEs, it is up to the bay
maintainer to implement this in their own way. Since a bay is a
combination of a COE and an image, ``driver.py`` will also contain
information about the ``os_distro`` property which is expected to be
attributed to Glance image.
- ``monitor.py`` is *optional*, and should contain the logic which monitors
the resource utilization of bays.
- ``template_def.py`` is *required* and should contain the COE's
implementation of how orchestration templates are loaded and matched to
Magnum objects. It would probably contain multiple classes, such as
``class SwarmAtomicXYZTemplateDef(BaseTemplateDefinition)``.
- ``scale.py`` is *optional* per bay specification and should contain the
logic for scaling operations.
2. Renaming the ``coe`` attribute of BayModel to ``driver``. Because this
value would determine which driver classes and orchestration templates to
load, it would need to correspond to the name of the driver as it is
registered with stevedore_ and setuptools entry points.
During the lifecycle of an API operation, top-level Magnum classes (such as
a Bay conductor) would then delegate to the driver classes which have been
dynamically loaded. Validation will need to ensure that whichever value
is provided by the user is correct.
By default, drivers are located under the main project directory and their
namespaces are accessible via ``magnum.drivers.foo``. But a use case that
needs to be looked at and, if possible, provided for is drivers which are
situated outside the project directory, for example in
``/usr/share/magnum``. This will suit operators who want greater separation
between customised code and Python libraries.
3. The driver implementations for the 3 current COE and Image combinations:
Docker Swarm Fedora, Kubernetes Fedora, Kubernetes CoreOS, and Mesos
Ubuntu. Any templates would need to be moved from
``magnum/templates/{coe_name}`` to
``magnum/drivers/{driver_name}/templates``.
4. Removal of the following files:
::
magnum/magnum/conductor/handlers/
docker_conductor.py
k8s_conducter.py
Design Principles
-----------------
- Minimal, clean API without a high cognitive burden
- Ensure Magnum's priority is to do one thing well, but allow extensibility
by external contributors
- Do not force ineffective abstractions that introduce feature divergence
- Formalise a modular and loosely coupled driver architecture that removes
COE logic from the core codebase
Alternatives
------------
This alternative relates to #5 of Proposed Change. Instead of having a
drivers registered using stevedore_ and setuptools entry points, an alternative
is to use the Magnum config instead.
Data model impact
-----------------
Since drivers would be implemented for the existing COEs, there would be
no loss of functionality for end-users.
REST API impact
---------------
Attribute change when creating and updating a BayModel (``coe`` to
``driver``). This would occur before v1 of the API is frozen.
COE-specific endpoints would be removed from the core API.
Security impact
---------------
None
Notifications impact
--------------------
None
Other end user impact
---------------------
There will be deployer impacts because deployers will need to select
which drivers they want to activate.
Performance Impact
------------------
None
Other deployer impact
---------------------
In order to utilize new functionality and bay drivers, operators will need
to update their installation and configure bay models to use a driver.
Developer impact
----------------
Due to the significant impact on the current codebase, a phased implementation
approach will be necessary. This is defined in the Work Items section.
Code will be contributed for COE-specific functionality in a new way, and will
need to abide by the new architecture. Documentation and a good first
implementation will play an important role in helping developers contribute
new functionality.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
murali-allada
Other contributors:
jamiehannaford
strigazi
Work Items
----------
1. New ``drivers`` directory
2. Change ``coe`` attribute to ``driver``
3. COE drivers implementation (swarm-fedora, k8s-fedora, k8s-coreos,
mesos-ubuntu). Templates should remain in directory tree until their
accompanying driver has been implemented.
4. Delete old conductor files
5. Update client
6. Add documentation
7. Improve user experience for operators of forking/creating new
drivers. One way we could do this is by creating new client commands or
scripts. This is orthogonal to this spec, and will be considered after
its core implementation.
Dependencies
============
None
Testing
=======
Each commit will be accompanied with unit tests, and Tempest functional tests.
Documentation Impact
====================
A set of documentation for this architecture will be required. We should also
provide a developer guide for creating a new bay driver and updating existing
ones.
References
==========
`Using Stevedore in your Application
<http://docs.openstack.org/developer/stevedore/tutorial/index.html/>`_.
.. _stevedore: http://docs.openstack.org/developer/stevedore/

View File

@ -0,0 +1,458 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
=================================
Magnum Container Networking Model
=================================
Launchpad Blueprint:
https://blueprints.launchpad.net/magnum/+spec/extensible-network-model
For Magnum to prosper, the project must support a range of networking tools
and techniques, while maintaining a simple, developer-focused user
experience. The first step in achieving this goal is to standardize the
process of allocating networking to containers, while providing an
abstraction for supporting various networking capabilities through
pluggable back-end implementations. This document recommends using Docker's
libnetwork library to implement container networking abstractions and
plugins. Since libnetwork is not a standard and the container ecosystem
is rapidly evolving, the Magnum community should continue evaluating
container networking options on a frequent basis.
Problem Description
===================
The container networking ecosystem is undergoing rapid changes. The
networking tools and techniques used in today's container deployments are
different than twelve months ago and will continue to evolve. For example,
Flannel [6]_, Kubernetes preferred networking implementation, was initially
released in July of 2014 and was not considered preferred until early 2015.
Furthermore, the various container orchestration engines have not
standardized on a container networking implementation and may never. For
example, Flannel is the preferred container networking implementation for
Kubernetes but not for Docker Swarm. Each container networking implementation
comes with its own API abstractions, data model, tooling, etc.. Natively
supporting each container networking implementation can be a burden on the
Magnum community and codebase. By supporting only a subset of container
networking implementations, the project may not be widely adopted or may
provide a suboptimal user experience.
Lastly, Magnum has limited support for advanced container networking
functionality. Magnum instantiates container networks behind the scenes
through Heat templates, exposing little-to-no user configurability. Some
users require the ability to customize their container environments,
including networking details. However, networking needs to "just work" for
users that require no networking customizations.
Roles
-----
The following are roles that the Magnum Container Networking Model takes
into consideration. Roles are an important reference point when creating
user stories. This is because each role provides different functions and
has different requirements.
1. Cloud Provider (CP): Provides standard OpenStack cloud infrastructure
services, including the Magnum service.
2. Container Service Provider (CSP): Uses Magnum to deliver
Containers-as-a-Service (CaaS) to users. CSPs are a consumer of CP
services and a CaaS provider to users.
3. Users: Consume Magnum services to provision and manage clustered
container environments and deploy apps within the container clusters.
The container ecosystem focuses on the developer user type. It is imperative
that the Magnum Container Networking Model meets the need of this user type.
These roles are not mutually exclusive. For example:
1. A CP can also be a CSP. In this case, the CP/CSP provisions and manages
standard OpenStack services, the Magnum service, and provides CaaS
services to users.
2. A User can also be a CSP. In this case, the user provisions their own
baymodels, bays, etc. from the CP.
Definitions
-----------
COE
Container Orchestration Engine
Baymodel
An object that stores template information about the bay which is
used to create new bays consistently.
Bay
A Magnum resource that includes at least one host to run containers on,
and a COE to manage containers created on hosts within the bay.
Pod
Is the smallest deployable unit that can be created, scheduled, and
managed within Kubernetes.
Additional Magnum definitions can be found in the Magnum Developer
documentation [2]_.
Use Cases
----------
This document does not intend to address each use case. The use cases are
provided as reference for the long-term development of the Magnum Container
Networking Model.
As a User:
1. I need to easily deploy containerized apps in an OpenStack cloud.
My user experience should be similar to how I deploy containerized apps
outside of an OpenStack cloud.
2. I need to have containers communicate with vm-based apps that use
OpenStack networking.
3. I need the option to preserve the container's IP address so I can
manage containers by IP's, not just ports.
4. I need to block unwanted traffic to/from my containerized apps.
5. I need the ability for my containerized apps to be highly available.
6. I need confidence that my traffic is secure from other tenants traffic.
As a CSP:
1. I need to easily deploy a bay for consumption by users. The bay must
support the following:
A. One or more hosts to run containers.
B. The ability to choose between virtual or physical hosts to run
containers.
C. The ability to automatically provision networking to containers.
2. I need to provide clustering options that support different
container/image, formats and technologies.
3. After deploying my initial cluster, I need the ability to provide ongoing
management, including:
A. The ability to add/change/remove networks that containers connect to.
B. The ability to add/change/remove nodes within the cluster.
4. I need to deploy a Bay without admin rights to OpenStack services.
5. I need the freedom to choose different container networking tools and
techniques offered by the container ecosystem beyond OpenStack.
As a CP:
1. I need to easily and reliably add the Magnum service to my existing
OpenStack cloud environment.
2. I need to easily manage (monitor, troubleshoot, etc..) the Magnum
service. Including the ability to mirror ports to capture traffic
for analysis.
3. I need to make the Magnum services highly-available.
4. I need to make Magnum services highly performant.
5. I need to easily scale-out Magnum services as needed.
6. I need Magnum to be robust regardless of failures within the container
orchestration engine.
Proposed Changes
================
1. Currently, Magnum supports Flannel [6]_ as the only multi-host container
networking implementation. Although Flannel has become widely accepted
for providing networking capabilities to Kubernetes-based container
clusters, other networking tools exist and future tools may develop.
This document proposes extending Magnum to support specifying a
container networking implementation through a combination of user-facing
baymodel configuration flags. Configuration parameters that are common
across Magnum or all networking implementations will be exposed as unique
flags. For example, a flag named network-driver can be used to instruct
Magnum which network driver to use for implementing a baymodel
container/pod network. network driver examples may include:
flannel, weave, calico, midonet, netplugin, etc..
Here is an example of creating a baymodel that uses Flannel as the
network driver: ::
magnum baymodel-create --name k8sbaymodel \
--image-id fedora-21-atomic-5 \
--keypair-id testkey \
--external-network-id 1hsdhs88sddds889 \
--dns-nameserver 8.8.8.8 \
--flavor-id m1.small \
--docker-volume-size 5 \
--coe kubernetes \
--network-driver flannel
If no network-driver parameter is supplied by the user, the baymodel is
created using the default network driver of the specified Magnum COE.
Each COE must support a default network driver and each driver must
provide reasonable default configurations that allow users to instantiate
a COE without supplying labels. The default network driver for each COE
should be consistent with existing Magnum default settings. Where current
defaults do not exist, the defaults should be consistent with upstream
network driver projects.
2. Each network driver supports a range of configuration parameters that
should be observed by Magnum. This document suggests using an attribute
named "labels" for supplying driver-specific configuration parameters.
Labels consist of one or more arbitrary key/value pairs. Here is an
example of using labels to change default settings of the Flannel
network driver: ::
magnum baymodel-create --name k8sbaymodel \
--image-id fedora-21-atomic-5 \
--keypair-id testkey \
--external-network-id ${NIC_ID} \
--dns-nameserver 8.8.8.8 \
--flavor-id m1.small \
--docker-volume-size 5 \
--coe kubernetes \
--network-driver flannel \
--labels flannel_network_cidr=10.0.0.0/8,\
flannel_network_subnetlen=22,\
flannel_backend=vxlan
With Magnum's current implementation, this document would support
labels for the Kubernetes COE type. However, labels are applicable
beyond Kubernetes, as the Docker daemon, images and containers now
support labels as a mechanism for providing custom metadata. The labels
attribute within Magnum should be extended beyond Kubernetes pods, so a
single mechanism can be used to pass arbitrary metadata throughout the
entire system. A blueprint [2]_ has been registered to expand the scope
of labels for Magnum. This document intends on adhering to the
expand-labels-scope blueprint.
Note: Support for daemon-labels was added in Docker 1.4.1. Labels for
containers and images were introduced in Docker 1.6.0
If the --network-driver flag is specified without any labels, default
configuration values of the driver will be used by the baymodel. These
defaults are set within the Heat template of the associated COE. Magnum
should ignore label keys and/or values not understood by any of the
templates during the baymodel operation.
Magnum will continue to CRUD bays in the same way:
magnum bay-create --name k8sbay --baymodel k8sbaymodel --node-count 1
3. Update python-magnumclient to understand the new Container Networking
Model attributes. The client should also be updated to support passing
the --labels flag according to the expand-labels-scope blueprint [2]_.
4. Update the conductor template definitions to support the new Container
Networking Model attributes.
5. Refactor Heat templates to support the Magnum Container Networking Model.
Currently, Heat templates embed Flannel-specific configuration within
top-level templates. For example, the top-level Kubernetes Heat
template [8]_ contains the flannel_network_subnetlen parameter. Network
driver specific configurations should be removed from all top-level
templates and instead be implemented in one or more template fragments.
As it relates to container networking, top-level templates should only
expose the labels and generalized parameters such as network-driver.
Heat templates, template definitions and definition entry points should
be suited for composition, allowing for a range of supported labels. This
document intends to follow the refactor-heat-templates blueprint [3]_ to
achieve this goal.
6. Update unit and functional tests to support the new attributes of the
Magnum Container Networking Model.
7. The spec will not add support for natively managing container networks.
Due to each network driver supporting different API operations, this
document suggests that Magnum not natively manage container networks at
this time and instead leave this job to native tools. References [4]_ [5]_
[6]_ [7]_.
provide additional details to common labels operations.
8. Since implementing the expand-labels-scope blueprint [2]_ may take a while,
exposing network functionality through baymodel configuration parameters
should be considered as an interim solution.
Alternatives
------------
1. Observe all networking configuration parameters, including labels
within a configuration file instead of exposing the labels attribute to
the user.
2. Only support a single networking implementation such as Flannel. Flannel
is currently supported for the Kubernetes COE type. It can be ported to
support the swarm COE type.
3. Add support for managing container networks. This will require adding
abstractions for each supported network driver or creating an
abstraction layer that covers all possible network drivers.
4. Use the Kuryr project [10]_ to provide networking to Magnum containers.
Kuryr currently contains no documentation or code, so this alternative
is highly unlikely if the Magnum community requires a pluggable
container networking implementation in the near future. However, Kuryr
could become the long-term solution for container networking within
OpenStack. A decision should be made by the Magnum community whether
to move forward with Magnum's own container networking model or to wait
for Kuryr to mature. In the meantime, this document suggests the Magnum
community become involved in the Kuryr project.
Data Model Impact
-----------------
This document adds the labels and network-driver attribute to the baymodel
database table. A migration script will be provided to support the attribute
being added. ::
+-------------------+-----------------+---------------------------------------------+
| Attribute | Type | Description |
+===================+=================+=============================================+
| labels | JSONEncodedDict | One or more arbitrary key/value pairs |
+-------------------+-----------------+---------------------------------------------+
| network-driver | string | Container networking backend implementation |
+-------------------+-----------------+---------------------------------------------+
REST API Impact
---------------
This document adds the labels and network-driver attribute to the BayModel
API class. ::
+-------------------+-----------------+---------------------------------------------+
| Attribute | Type | Description |
+===================+=================+=============================================+
| labels | JSONEncodedDict | One or more arbitrary key/value pairs |
+-------------------+-----------------+---------------------------------------------+
| network-driver | string | Container networking backend implementation |
+-------------------+-----------------+---------------------------------------------+
Security Impact
---------------
Supporting more than one network driver increases the attack
footprint of Magnum.
Notifications Impact
--------------------
None
Other End User Impact
---------------------
Most end users will never use the labels configuration flag
and simply use the default network driver and associated
configuration options. For those that wish to customize their
container networking environment, it will be important to understand
what network-driver and labels are supported, along with their
associated configuration options, capabilities, etc..
Performance Impact
------------------
Performance will depend upon the chosen network driver and its
associated configuration. For example, when creating a baymodel with
"--network-driver flannel" flag, Flannel's default configuration
will be used. If the default for Flannel is an overlay networking technique
(i.e. VXLAN), then networking performance will be less than if Flannel used
the host-gw configuration that does not perform additional packet
encapsulation to/from containers. If additional performance is required
when using this driver, Flannel's host-gw configuration option could be
exposed by the associated Heat template and instantiated through the labels
attribute.
Other Deployer Impact
---------------------
Currently, container networking and OpenStack networking are different
entities. Since no integration exists between the two, deployers/operators
will be required to manage each networking environment individually.
However, Magnum users will continue to deploy baymodels, bays, containers,
etc. without having to specify any networking parameters. This will be
accomplished by setting reasonable default parameters within the Heat
templates.
Developer impact
----------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
Daneyon Hansen (danehans)
Other contributors:
Ton Ngo (Tango)
Hongbin Lu (hongbin)
Work Items
----------
1. Extend the Magnum API to support new baymodel attributes.
2. Extend the Client API to support new baymodel attributes.
3. Extend baymodel objects to support new baymodel attributes. Provide a
database migration script for adding attributes.
4. Refactor Heat templates to support the Magnum Container Networking Model.
5. Update Conductor template definitions and definition entry points to
support Heat template refactoring.
6. Extend unit and functional tests to support new baymodel attributes.
Dependencies
============
Although adding support for these new attributes does not depend on the
following blueprints, it's highly recommended that the Magnum Container
Networking Model be developed in concert with the blueprints to maintain
development continuity within the project.
1. Common Plugin Framework Blueprint [1]_.
2. Expand the Scope of Labels Blueprint [9]_.
3. Refactor Heat Templates, Definitions and Entry Points Blueprint [3]_.
Testing
=======
Each commit will be accompanied with unit tests. There will also be
functional tests which will be used as part of a cross-functional gate
test for Magnum.
Documentation Impact
====================
The Magnum Developer Quickstart document will be updated to support the
configuration flags introduced by this document. Additionally, background
information on how to use these flags will be included.
References
==========
.. [1] https://blueprints.launchpad.net/magnum/+spec/common-plugin-framework
.. [2] http://docs.openstack.org/developer/magnum/
.. [3] https://blueprints.launchpad.net/magnum/+spec/refactor-heat-templates
.. [4] https://github.com/docker/libnetwork/blob/master/docs/design.md
.. [5] https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/design/networking.md
.. [6] https://github.com/coreos/flannel
.. [7] https://github.com/coreos/rkt/blob/master/Documentation/networking.md
.. [8] https://github.com/openstack/magnum/blob/master/magnum/templates/kubernetes/kubecluster.yaml
.. [9] https://blueprints.launchpad.net/magnum/+spec/expand-labels-scope
.. [10] https://github.com/openstack/kuryr

View File

@ -0,0 +1,500 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
=========================================
Magnum Container Volume Integration Model
=========================================
Launchpad Blueprint:
https://blueprints.launchpad.net/magnum/+spec/magnum-integrate-with-cinder
Storage is a key part of any computing system. Containers in particular have
the interesting characteristic that local storage by default is ephemeral:
any changes to the file system disappear when the container is deleted. This
introduces the need for persistent storage to retain and share data between
containers, and this is currently an active area of development in all
container orchestration engines (COE).
As the component in OpenStack for managing COE's, Magnum must fully enable the
features for persistent storage in the COE's. To achieve this goal, we propose
in this specification to generalize the process for utilizing persistent
storage with containers so that it is applicable for different bay types.
Despite the complexity, we aim to maintain a good user experience by a simple
abstraction for working with various volume capabilities. For the rest of this
specification, we will use the term Volume to refer to persistent storage, and
Volume Driver as the plugin in a COE to support the particular persistent
storage.
Problem Description
===================
Containers requires full life cycle management such as create, run, stop,
delete,... and a key operation is to manage the data - making the data
persistent, reusing the data, sharing data between containers, etc.
In this area, the support for container volume is undergoing rapid change
to bring more integration with open source software and third party
storage solutions.
A clear evidence of this growth is the many plugin volume drivers [1]_ [4]_
such as NFS, GlusterFS, EBS, etc. They provide different functionality, use
different storage backend and have different requirements. The COE's are
naturally motivated to be flexible and allow as many choices as possible for
the users with respect to the storage backend. Since Magnum's role is to
support the COE's within OpenStack, the goal is to be transparent and enable
these same storage backends for the COE's through the COE's lifecycle
operation.
Currently, Magnum provides limited support for managing container volume
. The only option available is to specify the docker-volume-size for a
pre-allocated block storage in the COE to host the containers. Magnum
instantiates container volumes through Heat templates, exposing no other
mechanism to configure and operate on volumes. In practice, some users
require the ability to manage volumes easily in the COEs .
Note that we are not proposing to create a new volume management interface
in Magnum. After the users create the baymodel and bays, we assume that the
users would manage the volumes through existing techniques:
1. Log in to the COE, use COE specific CLI or GUI to manage volumes.
2. Use native tools to manage volumes.
The initial implementation will focus on OpenStack Cinder integration; as
other alternatives become available, contributors are welcome through
3rd-party maintained projects.
Definitions
-----------
COE
Container Orchestration Engine
Baymodel
An object that stores template information about the bay which is
used to create new bays consistently.
Bay
A Magnum resource that includes at least one host to run containers on,
and a COE to manage containers created on hosts within the bay.
Pod
Is the smallest deployable unit that can be created, scheduled, and
managed within Kubernetes.
Volume
storage that is persistent
Volume plugin
COE specific code that supports the functionality of a type of volume.
Additional Magnum definitions can be found in the Magnum Developer
documentation[7]_ .
Use Cases
----------
This document does not intend to address all use cases. We list below a number
of use cases for 3 different roles; they should be useful as reference for the
long-term development of the Magnum Container Volume Integration.
As a User:
As mentioned above, our goal is to preserve the user experience specific to
the COE in managing the volumes. Therefore, we expect the use cases for the
users will be fulfilled by the COE's themselves; Magnum will simply ensure
that the necessary supports are in place.
1. I need to easily create volume for containers to use as persistent
data store.
2. I need the ability to create and mount a data volume container for cross
container sharing.
3. I need to mount a host directory as a data volume.
4. I need to easily attach a known volume to container to use the
existing data.
5. I need the ability to delete the volume.
6. I need to list and view the details of the volume
7. I need to modify the volume.
As a CSP:
1. I need to easily deploy a bay for consumption by users. The bay must
support the following:
A. One or more hosts to run containers.
B. The ability to choose between virtual or physical hosts to
run containers.
C. The ability to automatically enable volume plugins to containers.
2. I need to provide clustering options that support different volume plugins
per COE.
3. After deploying my initial cluster, I need the ability to provide lifecycle
management, including:
A. The ability to add/remove volumes that containers used.
B. The ability to add/remove nodes within the cluster with the necessary
adjustment to the volumes
As a CP:
1. I need to easily and reliably add the Magnum service to my existing
OpenStack cloud environment.
2. I need to make the Magnum services highly-available.
3. I need to make Magnum services highly performant.
4. I need to easily scale-out Magnum services as needed.
Proposed Changes
================
We propose extending Magnum as follows.
1. The new attribute volume-driver for a baymodel specifies the volume backend
driver to use when deploying a bay.
Volume drivers may include:
rexray, flocker, nfs, glusterfs, etc..
Here is an example of creating a Docker Swarm baymodel that uses rexray [5]_
[6]_ as the volume driver: ::
magnum baymodel-create --name swarmbaymodel \
--image-id fedora-21-atomic-5 \
--keypair-id testkey \
--external-network-id 1hsdhs88sddds889 \
--dns-nameserver 8.8.8.8 \
--flavor-id m1.small \
--docker-volume-size 5 \
--coe swarm\
--network-driver flannel \
--volume-driver rexray
When a Swarm bay is created with this bay model, the REX-Ray storage
subsystem will be installed, configured and started on the Swarm nodes,
then the REX-Ray volume plugin will be registered in Docker. When a container
is created with rexray as the volume driver, the container will have full
access to the REX-Ray capabilities such as creating, mounting, deleting
volumes [6]_. REX-Ray in turn will interface with Cinder to manage the
volumes in OpenStack.
Here is an example of creating a Kubernetes baymodel that uses Cinder [2]_
[3]_ as the volume driver: ::
magnum baymodel-create --name k8sbaymodel \
--image-id fedora-21-atomic-5 \
--keypair-id testkey \
--external-network-id 1hsdhs88sddds889 \
--dns-nameserver 8.8.8.8 \
--flavor-id m1.small \
--docker-volume-size 5 \
--coe kubernetes\
--network-driver flannel \
--volume-driver cinder
When the Kubernetes bay is created using this bay model, the kubelet will be
configured so that an existing Cinder volume can be mounted in a pod by
specifying the volume ID in the pod manifest as follows: ::
volumes:
- name: mysql-persistent-storage
cinder:
volumeID: bd82f7e2-wece-4c01-a505-4acf60b07f4a
fsType: ext4
Here is an example of creating a mesos baymodel that uses rexray as the
volume driver: ::
magnum baymodel-create --name mesosbaymodel \
--image-id ubuntu-mesos\
--keypair-id testkey \
--external-network-id 1hsdhs88sddds889 \
--dns-nameserver 8.8.8.8 \
--flavor-id m1.small \
--coe mesos\
--network-driver docker \
--volume-driver rexray
When the mesos bay is created using this bay model, the mesos bay will be
configured so that an existing Cinder volume can be mounted in a container
by configuring the parameters to mount the cinder volume in the json file. ::
"parameters": [
{ "key": "volume-driver", "value": "rexray" },
{ "key": "volume", "value": "redisdata:/data" }
]
If no volume-driver parameter is supplied by the user, the baymodel is
created using the default volume driver of the particular COE.
Magnum will provide a default volume driver for each COE as well as the
reasonable default configuration for each driver so that
users can instantiate a COE without supplying a volume driver and
associated labels. Generally the defaults should be consistent with upstream
volume driver projects.
2. Each volume driver supports a range of configuration parameters that are
handled by the "labels" attribute.
Labels consist of one or more arbitrary key/value pairs.
Here is an example of using labels to choose ¡°storage-provider¡± for
rexray driver.
Volume driver: ::
magnum baymodel-create --name k8sbaymodel \
--image-id fedora-21-atomic-5 \
--keypair-id testkey \
--external-network-id ${NIC_ID} \
--dns-nameserver 8.8.8.8 \
--flavor-id m1.small \
--docker-volume-size 5 \
--coe kubernetes \
--volume-driver rexray \
--labels storage-provider=openstack \
[, key2=value2...]
If the --volume-driver flag is specified without any labels, default
configuration values of the driver will be used by the baymodel.
Magnum will validate the labels together with the driver specified before
creating the bay and will return an error if the validation fails.
Magnum will continue to CRUD bays in the same way:
magnum bay-create --name k8sbay --baymodel k8sbaymodel --node-count 1
3. Update python-magnumclient to handle the new container volume-
driver attributes.
4. Update the conductor template definitions to support the new container
volume-driver model attributes.
5. Refactor Heat templates to support the Magnum volume driver plugin.
Configurations specific to volume drivers should be
implemented in one or more template fragments.
Top-level templates should only
expose the labels and generalized parameters such as volume-driver.
Heat templates, template definitions and definition entry points should
be designed for composition, allowing for a range of supported labels.
6. Update unit and functional tests to support the new attributes of the
Magnum container volume driver.
7. Preserve the user experience by ensuring that any operation on volume will
be identical between a COE deployed by Magnum and a COE deployed by other
methods.
Alternatives
------------
1. Without the support proposed, the user will need to manually enable and
configure the volume plugin. This will require the user to log into the
nodes in the cluster and understand the low level infrastructure of the
cluster as deployed by the heat templates.
2. We can add full support for managing container volume in Magnum user
interface itself. This will require adding abstractions for each supported
COE volume plugins driver or creating an abstraction layer that covers all
possible COE volume drivers.
Data Model Impact
-----------------
This document adds the volume-driver attribute to the baymodel
database table. A migration script will be provided to support the attribute
being added. ::
+-------------------+-----------------+---------------------------------------------+
| Attribute | Type | Description |
+===================+=================+=============================================+
+-------------------+-----------------+---------------------------------------------+
| volume-driver | string | Container volume backend implementation |
+-------------------+-----------------+---------------------------------------------+
REST API Impact
---------------
This document adds volume-driver attribute to the BayModel
API class. ::
+-------------------+-----------------+---------------------------------------------+
| Attribute | Type | Description |
+===================+=================+=============================================+
+-------------------+-----------------+---------------------------------------------+
| volume-driver | string | Container volume backend implementation |
+-------------------+-----------------+---------------------------------------------+
Security Impact
---------------
Supporting volume drivers can potentially increase the attack surface
on containers.
Notifications Impact
--------------------
None
Other End User Impact
---------------------
There is no impact if the user does not use a volume driver.
We anticipate that most users would not use the labels for volume
and would simply use the default volume driver and associated
configuration options. For those who wish to customize their
container volume driver environment, it will be important to understand
what volume-driver and labels are supported, along with their
associated configuration options, capabilities, etc..
Performance Impact
------------------
There is no impact if the user does not use a volume driver.
When a volume driver is used, the performance will depend upon the specific
volume driver and its associated storage backends. For example, Kubernetes
supports Cinder and awsEBS; the two types of volumes can have different
performance.
An example of the second case is a docker swarm bay with
"--volume-driver rexray" where the rexray driver's storage provider is
OpenStack cinder. The resulting performance for container may vary depending
on the storage backends. As listed in [8]_ , Cinder supports many storage
drivers. Besides this, different container volume driver can also cause
performance variance.
High-Availability Impact
------------------------------
+-----------------+--------------------+--------------------------+
| COE | Master HA | Pod/Container/App HA |
+=================+====================+==========================+
| Kubernetes | No | Yes |
+-----------------+--------------------+--------------------------+
| Docker Swarm | No | Yes |
+-----------------+--------------------+--------------------------+
| Mesos | No | No |
+-----------------+--------------------+--------------------------+
"No" means that the volume doesn't affect the high-availability.
"Yes" means that the volume affect the high-availability.
Kubernetes does support pod high-availability through the replication
controller, however, this doesn't work when a pod with volume attached
fails. Refer the link [11]_ for details.
Docker swarm doesn't support the containers rescheduling when a node fails, so
volume can not be automatically detached by volume driver. Refer the
link [12]_ for details.
Mesos supports the application high-availability when a node fails, which
means application would be started on new node, and volumes can be
automatically attached to the new node by the volume driver.
Other Deployer Impact
---------------------
Currently, both Kubernetes and Docker community have supported some volume
plugins. The changes proposed will enable these volume plugins in Magnum.
However, Magnum users will be able to continue to deploy baymodels, bays,
containers, etc. without having to specify any parameters for volume.
This will be accomplished by setting reasonable default parameters within
the Heat templates.
Developer impact
----------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
- Kai Qiang Wu (Kennan)
Other contributors:
- Qun Wang (wangqun)
- Ton Ngo (Tango)
Work Items
----------
1. Extend the Magnum API to support new baymodel attributes.
2. Extend the Client API to support new baymodel attributes.
3. Extend baymodel objects to support new baymodel attributes. Provide a
database migration script for adding attributes.
4. Refactor Heat templates to support the Magnum container volume driver.
5. Update Conductor template definitions and definition entry points to
support Heat template refactoring.
6. Extend unit and functional tests to support new baymodel attributes.
7. Document how to use the volume drivers with examples.
Dependencies
============
Although adding support for these new attributes does not depend on the
following blueprints, it's highly recommended that the Magnum Container
Networking Model be developed in concert with the blueprints to maintain
development continuity within the project.
https://blueprints.launchpad.net/magnum/+spec/ubuntu-image-build
Kubernetes with cinder support need Kubernetes version >= 1.1.1
Swarm need version >= 1.8.3, as Kubernetes 1.1.1 upgraded to that version
Testing
=======
Each commit will be accompanied with unit tests. There will also be
functional tests which will be used as part of a cross-functional gate
test for Magnum.
Documentation Impact
====================
The Magnum Developer Quickstart document will be updated to support the
configuration flags introduced by this document. Additionally, background
information on how to use these flags will be included.
References
==========
.. [1] http://kubernetes.io/v1.1/docs/user-guide/volumes.html
.. [2] http://kubernetes.io/v1.1/examples/mysql-cinder-pd/
.. [3] https://github.com/kubernetes/kubernetes/tree/master/pkg/volume/cinder
.. [4] http://docs.docker.com/engine/extend/plugins/
.. [5] https://github.com/emccode/rexray
.. [6] http://rexray.readthedocs.org/en/stable/user-guide/storage-providers/openstack
.. [7] http://docs.openstack.org/developer/magnum/
.. [8] http://docs.openstack.org/liberty/config-reference/content/section_volume-drivers.html
.. [9] http://docs.openstack.org/admin-guide-cloud/blockstorage_multi_backend.html#
.. [10] http://docs.openstack.org/user-guide-admin/dashboard_manage_volumes.html
.. [11] https://github.com/kubernetes/kubernetes/issues/14642
.. [12] https://github.com/docker/swarm/issues/1488

View File

@ -0,0 +1,400 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==================
Containers Service
==================
Launchpad blueprint:
https://blueprints.launchpad.net/nova/+spec/containers-service
Containers share many features in common with Nova instances. For the common
features, virt drivers for Nova can be used to surface basic instance
functionality. For features that go beyond what can be naturally fit within
a virt driver, we propose a new API service that allows for advanced features
to be added without conflating the worlds of instances and containers.
Some examples of containers specific features are setting of shell environment
variables, and accepting a shell command to execute at runtime. Capturing the
STDIO of the process(es) within a container, and tracking the return status
of processes are all beyond the scope of what was contemplated for Nova. All
of these features will be implemented in the Containers Service.
Problem description
===================
Container technology is rapidly gaining popularity as a way to bundle and
deploy applications. Recognizing and adapting to this trend will position
OpenStack to be useful not only to clouds that employ bare metal and virtual
machine instances, but can remain competitive in offering container services
as well.
Nova's concepts of an instance, and the actions that may be taken on it do not
match completely with containers.
Use cases
---------
1. App Consolidation. End-user wants to run multiple small applications in
separate operating system environments, but wants to optimize for efficiency
to control hosting costs. Each application belongs to the same tenant, so
security isolation between applications is nice-to-have but not critical.
Isolation is desired primarily for simplified management of the execution
environment for each application.
2. App Portability. End-user wants to create a single container image, and
deploy the same image to multiple hosting environments, including OpenStack.
Other environments may include local servers, dedicated servers, private
clouds, and public clouds. Switching environments requires passing database
connection strings by environment variables at the time a container starts
to allow the application to use the services available in each environment
without changing the container image.
3. Docker Compatibility. End-user has a Dockerfile used to build an application
and its runtime environment and dependencies in a Docker container image.
They want an easy way to run the Docker resulting image on an OpenStack
cloud.
4. LXC Compatibility. End-user wants an easy way to remotely create multiple
LXC containers within a single Nova instance.
5. OpenVZ Compatibility. End-user wants an easy way to remotely create multiple
OpenVZ containers within a single Nova instance.
6. Containers-Centric World View. End-user wants to communicate with a single
OpenStack API, and request the addition of containers, without the need to
be concerned with keeping track of how many containers are already running
on a given Nova instance, and when more need to be created. They want to
simply create and remove containers, and allow the appropriate resource
scheduling to happen automatically.
7. Platform Integration. Cloud operator already has an OpenStack cloud, and
wants to add a service/application centric management system on top.
Examples of such systems are Cloud Foundry, Kubernetes, Apache Mesos, etc.
The selected system is already Docker compatible. Allow this cloud operator
easy integration with OpenStack to run applications in containers. The
Cloud Operator now harnesses the power of both the management system, and
OpenStack, and does not need to manage a second infrastructure for his/her
application hosting needs. All details involving the integration of
containers with Nova instances is managed by OpenStack.
8. Container network. End-user wants to define a custom overlay network for
containers, and wants to have admin privilege to manage the network
topology. Building a container network can decouple application deployment
and management from the underlying network infrastructure, and enable
additional usage scenario, such as (i) software-defined networking, and
(ii) extending the container network (i.e. connecting various resources from
multiple hosting environments). End-users want a single service that could
help them build the container network, and dynamically modify the network
topology by adding or removing containers to or from the network.
9. Permit secure use of native REST APIs. Provide two models of operation with
Magnum. The first model allows Magnum to manage the lifecycle of Pods,
ReplicationControllers, and Services. The second model allows end-users to
manage the lifecycle of Pods, ReplicationControllers, and Services by
providing direct secure access to the native ReST APIs in Kubernetes and
possibly Docker.
Long Term Use Cases
-------------------
These use cases have been identified by the community as important, but
unlikely to be tackled in short term (especially prior to incubation). We wish
to adapt to these use cases in long term, but this is not a firm project
commitment.
1. Multi-region/multi-cloud support. End-user wants to deploy applications to
multiple regions/clouds, and dynamically relocate deployed applications
across different regions/clouds. In particular, they want a single service
that could help them (i) provision nodes from multiple regions/clouds, thus
running containers on top of them, and (ii) dynamically relocate containers
(e.g. through container migration) between nodes regardless of the
underlying infrastructure.
Proposed change
===============
Add a new API service for CRUD and advanced management of containers.
If cloud operators only want to offer basic instance features for their
containers, they may use nova with an alternate virt-driver, such as
libvirt/lxc or nova-docker. For those wanting a full-featured container
experience, they may offer the Containers Service API as well, in combination
with Nova instances that contain an OpenStack agent that connects to the
containers service through a security controlled agent (daemon) that allows
the OpenStack control plane to provision and control containers running on
Compute Hosts.
The Containers Service will call the Nova API to create one or more Nova
instances inside which containers will be created. The Nova instances may
be of any type, depending on the virt driver(s) chosen by the cloud operator.
This includes bare-metal, virtual machines, containers, and potentially other
instance types.
This allows the following configurations of containers in OpenStack.
* Containers in Virtual Machine Instances
* Containers in Bare Metal Instances
* Containers in Container Instances (nested)
The concept of nesting containers is currently possible if the parent container
runs in privileged mode. Patches to the linux kernel are being developed to
allow nesting of non-privileged containers as well, which provides a higher
level of security.
The spirit of this plan aims to duplicate as little as possible between Nova
and the Containers Service. Common components like the scheduler are expected
to be abstracted into modules, such as Gantt that can be shared by multiple
projects. Until Gantt is ready for use by the Containers Service, we will
implement only two provisioning schemes for containers:
1. Create a container on a specified instance by using a nova instance guid.
2. Auto-create instances (applies only until the Gantt scheduler is used)
2.1. Fill them sequentially until full.
2.2. Remove them automatically when they become empty.
The above orchestration will be implemented using Heat. This requires some
kind of hypervisor painting (such as host aggregates) for security reasons.
The diagram below offers an overview of the system architecture. The OSC box
indicates an OpenStack client, which will communicate with the Containers
Service through a REST API. The containers service may silently create Nova
instances if one with enough capacity to host the requested container is not
already known to the Containers service. The containers service will maintain
a database "Map" of containers, and what Nova instance each belongs to. Nova
creates instances. Instances are created in Nova, and containers belong only
to the Containers Service, and run within a Nova instance. If the instance
includes the agent software "A", then it may be included in the inventory of
the Containers service. Instances that do not contain an agent may not interact
with the Containers Service, and can be controlled only by a Nova virt driver.
::
+---------+
|OSC|
+----+----+
|
+----+----+
+--------Nova-------++-+REST+--Containers-+
|||+---------+Service|
||||
|+-------++--++-----+|
||Gantt||||Map||
|+-------+||+-----+|
||||
+-----------+---------++---------------+-----------+
||
+-----------+----+ComputeHost---------|-----------+
|+---+---+|
|+----+Relay+---+|
||+-------+||
||||
|+--Instance--++--Instance|-++--Instance|-+|
||||||||||
||||+---+||+---+||
||||||||||||
|||||A||||A|||
||||||||||||
||||+---+||+---+||
||||||||
||||+---++---+||+---++---+||
||||||||||||||||
|||||C||C||||C||C|||
||||||||||||||||
||||+---++---+||+---++---+||
||||||||
|+--------------++--------------++--------------+|
||
+----------------------------------------------------+
+---+
| |
| A | = Agent
| |
+---+
+---+
| |
| C | = Container
| |
+---+
Design Principles
-----------------
1. Leverage existing OpenStack projects for what they are good at. Do not
duplicate functionality, or copy code that can be otherwise accessed through
API calls.
2. Keep modifications to Nova to a minimum.
3. Make the user experience for end users simple and familiar.
4. Allow for implementation of all features containers are intended to offer.
Alternatives
------------
1. Extending Nova's existing feature set to offer container features
1.1. Container features don't fit into Nova's idea of compute (VM/Server)
2. A completely separate containers service forked from Nova.
2.1. Would result in large overlap and duplication in features and code
Data model impact
-----------------
For Nova, None. All new data planned will be in the Containers Service.
REST API impact
---------------
For Nova, none. All new API calls will be implemented in the Containers
Service. The OpenStack Containers Service API will be a superset of
functionality offered by the, The `Docker Remote API:
<https://docs.docker.com/reference/api/docker_remote_api/>`_
with additionals to make is suitable for general use regardless of the backend
container technology used, and to be compatible with OpenStack multi-tenancy
and Keystone authentication.
Specific Additions:
1. Support for the X-Auth-Project-Id HTTP request header to allow for
multi-tenant use.
2. Support for the X-Auth-Token HTTP request header to allow for authentication
with keystone.
If either of the above headers are missing, a 401 Unauthorized response will
be generated.
Docker CLI clients may communicate with a Swarmd instance that is configured
to use the OpenStack Containers API as the backend for libswarm. This will
allow for tool compatibility with the Docker ecosystem using the officially
supported means for integration of a distributed system.
The scope of the full API will cause this spec to be too long to review, so
the intent is to deal with the specific API design as a series of Gerrit
reviews that submit API code as Not Implemented stubs with docstrings that
clearly document the design, so allow for approval, and further implementation.
Security impact
---------------
Because Nova will not be changed, there should be no security impacts to Nova.
The Containers Service implementation, will have the following security related
issues:
* Need to authenticate against keystone using python-keystoneclient.
* A trust token from Nova will be needed in order for the Containers Service
to call the Nova API on behalf of a user.
* Limits must be implemented to control resource consumption in accordance with
quotas.
* Providing STDIO access may generate a considerable amount of network chatter
between containers and clients through the relay. This could lead to
bandwidth congestion at the relays, or API nodes. An approach similar to
how we handle serial console access today will need to be considered to
mitigate this concern.
Using containers implies a range of security considerations for cloud
operators. These include:
* Containers in the same instance share an operating system. If the kernel is
exploited using a security vulnerability, processes in once container may
escape the constraints of the container and potentially access other
resources on the host, including contents of other containers.
* Output of processes may be persisted by the containers service in order to
allow asynchronous collection of exit status, and terminal output. Such
content may include sensitive information. Features may be added to mitigate
the risk of this data being replicated in log messages, including errors.
* Creating containers usually requires root access. This means that the Agent
may need to be run with special privileges, or be given a method to
escalate privileges using techniques such as sudo.
* User provided data is passed through the API. This will require sensible
data input validation.
Notifications impact
--------------------
Contemplated features (in subsequent release cycles):
* Notify the end user each time a Nova instance is created or deleted by
the Containers service, if (s)he has registered for such notifications.
* Notify the user each on CRUD of containers containing start and end
notifications. (compute.container.create/delete/etc)
* Notify user periodically of existence of container service managed
containers (ex compute.container.exists)
Other end user impact
---------------------
The user interface will be a REST API. On top of that API will be an
implementation of the libswarm API to allow for tools designed to use Docker
to treat OpenStack as an upstream system.
Performance Impact
------------------
The Nova API will be used to create instances as needed. If the Container to
Instance ratio is 10, then the Nova API will be called at least once for every
10 calls to the Containers Service. Instances that are left empty will be
automatically deleted, so in the example of a 10:1 ratio, the Nova API will be
called to perform a delete for every 10 deletes in the Container Service.
Depending on the configuration, the ratio may be as low as 1:1.
The Containers Service will only access Nova through its API, not by accessing
its database.
Other deployer impact
---------------------
Deployers may want to adjust the default flavor used for Nova Instances created
by the Containers Service.
There should be no impact on users of prior releases, as this introduces a new
API.
Developer impact
----------------
Minimal. There will be minimal changes required in Nova, if any.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
aotto
Other contributors:
andrew-melton
ewindisch
Work Items
----------
1. Agent
2. Relay
3. API Service
4. IO Relays
Dependencies
============
1. <Links to Agent Blueprint and Spec here, once ready>
2. Early implementations may use libswarm, or a python port of libswarm to
implement Docker API compatibility.
Testing
=======
Each commit will be accompanied with unit tests, and Tempest functional tests.
Documentation Impact
====================
A set of documentation for this new service will be required.
References
==========
* Link to high level draft proposal from the Nova Midcycle Meetup for Juno:
`PDF <https://wiki.openstack.org/w/images/5/51/Containers_Proposal.pdf>`_
* `Libswarm Source <https://github.com/docker/libswarm>`_

View File

@ -0,0 +1,186 @@
==================================
Create a trustee user for each bay
==================================
https://blueprints.launchpad.net/magnum/+spec/create-trustee-user-for-each-bay
Some services which are running in a bay need to access OpenStack services.
For example, Kubernetes load balancer [1]_ needs to access Neutron. Docker
registry [2]_ needs to access Swift. In order to access OpenStack services,
we can create a trustee for each bay and delegate a limited set of rights to
the trustee. [3]_ and [4]_ give a brief introduction to Keystone's trusts
mechanism.
Problem description
===================
Some services which are running in a bay need to access OpenStack services,
so we need to pass user credentials into the vms.
Use Cases
---------
1. Kubernetes load balancer needs to access Neutron [1]_.
2. For persistent storage, Cloud Provider needs to access Cinder to
mount/unmount block storage to the node as volume [5]_.
3. TLS cert is generated in the vms and need to be uploaded to Magnum [6]_ and
[7]_.
4. Docker registry needs to access Swift [2]_.
Project Priority
----------------
High
Proposed change
===============
When a user (the "trustor") wants to create a bay, steps for trust are as
follows.
1. Create a new service account (the "trustee") without any role in a domain
which is dedicated for trust. Without any role, the service account can do
nothing in Openstack.
2. Define a trust relationship between the trustor and the trustee. The trustor
can delegate a limited set of roles to the trustee. We can add an option
named trust_roles in baymodel. Users can add roles which they want to
delegate into trust_roles. If trust_roles is not provided, we delegate all
the roles to the trustee.
3. Services in the bay can access OpenStack services with the trustee
credentials and the trust.
The roles which are delegated to the trustee should be limited. If the services
in the bay only need access to Neutron, we should not allow the services to
access to other OpenStack services. But there is a limitation that a trustor
must have the role which is delegated to a trustee [4]_.
Magnum now only allows the user who create the bay to get the certificate to
avoid the security risk introduced by Docker [8]_. For example, if other users
in the same tenant can get the certificate, then they can use Docker API to
access the host file system of a bay node and get anything they want::
docker run --rm -v /:/hostroot ubuntu /bin/bash \
-c "cat /hostroot/etc/passwd"
If Keystone doesn't allow to create new service accounts when LDAP is used as
the backend for Keystone, we can use a pre-create service account for all
bays. In this situation, all the bays use the same service account and
different trust. We should add an config option to choose this method.
Alternatives
------------
Magnum can create a user for each bay with roles to access OpenStack Services
in a dedicated domain. The method has one disadvantage. The user which is
created by magnum may get the access to OpenStack services which this user can
not access before. For example, a user can not access Swift service and create
a bay. Then Magnum create a service account for this bay with roles to access
Swift. If the user logins into the vms and get the credentials, the user can
use these credentials to access Swift.
Or Magnum doesn't prepare credentials and the user who create a bay needs to
login into the nodes to manully add credentials in config files for services.
Data model impact
-----------------
Trustee id, trustee password and trust id are added to Bay table in Magnum
database.
REST API impact
---------------
Only the user who create a bay can get the certificate of this bay. Other
users in the same tenant can not get the certificate now.
Security impact
---------------
Trustee id and trustee password are encrypted in magnum database. When Magnum
passes these parameters to heat to create a stack, the transmission is
encrypted by tls, so we don't need to encrypt these credentials. These
credentials are hidden in heat, users can not query them in stack parameters.
Trustee id, trustee password and trust id can be obtained in the vms. Anyone
who can login into the vms can get them and use these credentials to access
OpenStack services. In a production environment, these vms must be secured
properly to prevent unauthorized access.
Only the user who create the bay can get the certificate to access the COE
api, so it is not a security risk even if the COE api is not safe.
Notifications impact
--------------------
None
Other end user impact
---------------------
None
Performance impact
------------------
None
Other deployer impact
---------------------
None
Developer impact
----------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
humble00 (wanghua.humble@gmail.com)
Other contributors:
None
Work Items
----------
1. Create an trustee for each bay.
2. Change the policy so that only the user who create a bay can get the
certificate of the bay.
Dependencies
============
None
Testing
=======
Unit test and functional test for service accounts and the policy change.
Documentation Impact
====================
The user guide and troubleshooting guide will be updated with details
regarding the service accounts.
References
==========
.. [1] http://docs.openstack.org/developer/magnum/dev/dev-kubernetes-load-balancer.html
.. [2] https://blueprints.launchpad.net/magnum/+spec/registryv2-in-master
.. [3] http://blogs.rdoproject.org/5858/role-delegation-in-keystone-trusts
.. [4] https://wiki.openstack.org/wiki/Keystone/Trusts
.. [5] https://github.com/kubernetes/kubernetes/blob/release-1.1/examples/mysql-cinder-pd/README.md
.. [6] https://bugs.launchpad.net/magnum/+bug/1503863
.. [7] https://review.openstack.org/#/c/232152/
.. [8] https://docs.docker.com/engine/articles/security/#docker-daemon-attack-surface
History
=======
None

View File

@ -0,0 +1,171 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
===================================
Web Interface for Magnum in Horizon
===================================
Launchpad blueprint:
https://blueprints.launchpad.net/magnum/+spec/magnum-horizon-plugin
Currently there is no way for a user to interact with Magnum through a web
based user interface, as they are used to doing with other OpenStack
components. This implementation aims to introduce this interface as an
extension of Horizon (the OpenStack Dashboard) and expose all the features of
Magnum in a way familiar to users.
Problem description
===================
In order to increase adoption and usability of Magnum we need to introduce a UI
component for users and administrators to interact with Magnum without the need
to use the command line. The UI proposed to be built will model all of the
features currently available in the Magnum REST API and built using the Horizon
plugin architecture to remain in line with other OpenStack UI projects and
minimise the amount of new code that needs to be added.
Use Cases
----------
1. An end user wanting to use Magnum with OpenStack who is not comfortable in
issuing commands with the python client will use the web user interface to
interact with Magnum.
2. An administrator may use the user interface to provide a quick overview of
what Magnum has deployed in their OpenStack environment.
Proposed change
===============
The first step will be to extend the Horizon API to include CRUD operations
that are needed to interact with Magnum. Assuming that there are no issues here
and API changes/additions are not required to Magnum, we can begin to
design/implement the interface. We will aim to minimize the amount of Magnum
specific UI code that will need to be maintained by reusing components from
Horizon. This will also speed up the development significantly.
It is suggested the initial implementation of Magnum UI will include basic CRUD
operations on BayModel and Bay resources. This will be the starting point for
development and upon completion this will represent version 1.
Future direction includes adding CRUD operations for other Magnum features
(Pod, Container, Service, ReplicationController) and will be tracked by new
blueprints as they represent significant additional effort. The ultimate goal,
a user should be able to perform all normal interactions with Magnum through
the UI with no need for interaction with the python client.
Suggestions for further improvement include visualising Magnum resources to
provide a quick overview of how resources are deployed.
Bugs/Blueprints relating specifically to the Magnum UI will be tracked here:
https://launchpad.net/magnum-ui
Mockups/Designs will be shared using the OpenStack Invision account located
here:
https://openstack.invisionapp.com
Alternatives
------------
One alternative to this approach is to develop an entirely separate UI
specifically for Magnum. We will not use this approach as it does not fall in
line with how other projects are managing their user interfaces and this
approach would ultimately result in a significantly larger effort with much
duplication with Horizon.
Data model impact
-----------------
None
REST API impact
---------------
For Magnum, none. The Horizon API will need to be extended to include Create,
Read, Update, Delete operations for all features available in the Magnum REST
API. However, this extension to the Horizon API will live in the Magnum UI tree
not the upstream Horizon tree.
Security impact
---------------
None
Notifications impact
--------------------
None
Other end user impact
---------------------
None
Performance Impact
------------------
The Magnum API will be called from the user interface to return information to
the user about the current state of Magnum objects and perform new interactions
with Magnum. For every action a user performs from the user interface at least
one API call to Magnum will need to be made.
Other deployer impact
---------------------
As the Magnum user interface will be managed and stored outside of the Horizon
project deployers will need to pull down the Magnum UI code and add this to
their Horizon install.
In order to add the Magnum UI to Horizon the deployer will have to copy an
enable file to openstack_dashboard/local/enabled/ in their Horizon directory
and then run Horizon as they would normally.
Developer impact
----------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
bradjones
Work Items
----------
1. Extend Horizon API in include Magnum calls
2. CRUD operations on BayModel and Bay resources
3. CRUD operations on other Magnum features (Pod, Container, Service, etc.)
4. Refine the user experience
Dependencies
============
None
Testing
=======
Each commit will be accompanied with unit tests. There will also be functional
tests which will be used as part of a cross-functional gate test for Magnum.
This additional gate test will be non-voting as failures will not indicate
issues with Magnum but instead serves as advanced warning of any changes that
could potentially break the UI.
Documentation Impact
====================
An installation guide will be required.
References
==========
None

View File

@ -0,0 +1,177 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
=================================
Magnum and Open DC/OS Integration
=================================
Launchpad Blueprint:
https://blueprints.launchpad.net/magnum/+spec/mesos-dcos
Open DC/OS [1]_ is a distributed operating system based on the Apache Mesos
distributed systems kernel. It enables the management of multiple machines as
if they were a single computer. It automates resource management, schedules
process placement, facilitates inter-process communication, and simplifies
the installation and management of distributed services. Its included web
interface and available command-line interface (CLI) facilitate remote
management and monitoring of the cluster and its services.
Open DC/OS now supports both docker containerizer and mesos containerizer.
The mesos containerizer support both docker and AppC image spec, the mesos
containerizer can manage docker containers well even if docker daemon is not
running.
End user can install Open DC/OS with different ways, such as vagrant, cloud,
local etc. For cloud, the Open DC/OS only supports AWS now, end user can
deploy a DC/OS cluster quickly with a template. For local install, there
are many steps to install a Open DC/OS cluster.
Problem Description
===================
COEs (Container Orchestration Engines) are the first class citizen in Magnum,
there are different COEs in Magnum now including Kubernetes, Swarm and Mesos.
All of those COEs are focusing docker container management, the problem is
that the concept of container is not only limited in docker container, but
also others, such as AppC, linux container etc, Open DC/OS is planning to
support different containers by leveraging Mesos unified container feature
and the Open DC/OS has a better management console for container orchestration.
Currently, Magnum provides limited support for Mesos Bay as there is only one
framework named as Marathon running on top of Mesos. Compared with Open DC/OS,
the current Mesos Bay lack the following features:
1. App Store for application management. The Open DC/OS has a universe to
provide app store functions.
2. Different container technology support. The Open DC/OS support different
container technologies, such as docker, AppC etc, and may introduce OCI
support in future. Introducing Open DC/OS Bay can enable Magnum to support
more container technologies.
3. Better external storage integration. The Open DC/OS is planning to introduce
docker volume isolator support in next release, the docker volume isolator
is leveraging docker volume driver API to integrate with 3rd party
distributed storage platforms, such as OpenStack Cinder, GlusterFS, Ceph
etc.
4. Better network management. The Open DC/OS is planning to introduce CNI
network isolator in next release, the CNI network isolator is leveraging CNI
technologies to manage network for containers.
5. Loosely coupled with docker daemon. The Open DC/OS can work well for docker
container even if docker daemon is not running. The docker daemon now have
some issues in large scale cluster, so this approach avoids the limitation
of the docker daemon but still can enable end user get some docker features
in large scale cluster.
Proposed Changes
================
We propose extending Magnum as follows.
1. Leverage bay driver work and structure this new COE as a bay driver.
2. Leverage mesos-slave-flags [3]_ to customize Open DC/OS.
Here is an example of creating an Open DC/OS baymodel that uses
docker/volume as isolator, linux as launcher and docker as image
provider: ::
magnum baymodel-create --name dcosbaymodel \
--image-id dcos-centos-7.2 \
--keypair-id testkey \
--external-network-id 1hsdhs88sddds889 \
--dns-nameserver 8.8.8.8 \
--flavor-id m1.small \
--docker-volume-size 5 \
--coe dcos \
--labels isolation=docker/volume,\
launcher=linux, \
image_providers=docker
Magnum will validate the labels together with the driver specified before
creating the bay and will return an error if the validation fails.
Magnum will continue to CRUD bays in the same way:
magnum bay-create --name dcosbay --baymodel dcosbaymodel --node-count 1
3. Keep the old Mesos Bay and add a new Open DC/OS Bay. Once the Open DC/OS Bay
is stable, deprecate the Mesos Bay.
4. Update unit and functional tests to support Open DC/OS Bay, it is also an
option to verify the Open DC/OS Bay in gate.
5. Preserve the user experience by ensuring that any operation on Open DC/OS
Bay will be identical between a COE deployed by Magnum and a COE deployed
by other methods.
REST API Impact
---------------
There will be no REST API exposed from Magnum for end user to operate Open
DC/OS, end user can logon to Open DC/OS dashboard or call Open DC/OS REST
API directly to manage the containers or the applications.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
- Guang Ya Liu (jay-lau-513)
Other contributors:
- Qun Wang (wangqun)
- Gao Jin Cao
Work Items
----------
1. Build VM image for Open DC/OS Bay.
2. Add Open DC/OS Bay driver.
3. Add Heat template for Open DC/OS Bay.
4. Add Open DC/OS Bay monitor.
5. Document how to use the Open DC/OS Bay.
Dependencies
============
1. This blueprint will focus on running on Open DC/OS in CentOS 7.2.
2. Depend on blueprint
https://blueprints.launchpad.net/magnum/+spec/mesos-slave-flags
Testing
=======
Each commit will be accompanied with unit tests. There will also be
functional tests which will be used as part of a cross-functional gate
test for Magnum.
Documentation Impact
====================
The Magnum Developer Quickstart document will be updated to support the Open
DC/OS Bay introduced by including a short example and a full documentation
with all the explanation for the labels in the user guide. Additionally,
background information on how to use the Open DC/OS Bay will be included.
References
==========
.. [1] https://dcos.io/docs/1.7/overview/what-is-dcos/
.. [2] https://dcos.io/install/
.. [3] https://blueprints.launchpad.net/magnum/+spec/mesos-slave-flags

View File

@ -0,0 +1,252 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================
Quota for Magnum Resources
==========================
Launchpad blueprint:
https://blueprints.launchpad.net/magnum/+spec/resource-quota
There are multiple ways to slice an OpenStack cloud. Imposing quota on these
various slices puts a limitation on the amount of resources that can be
consumed which helps to guarantee "fairness" or fair distribution of resource
at the creation time. If a particular project needs more resources, the
concept of quota, gives the ability to increase the resource count on-demand,
given that the system constraints are not exceeded.
Problem description
===================
At present in Magnum we don't have the concept of Quota on Magnum resources as
a result of which, as long as the underlying Infrastructure as a Service(IaaS)
layer has resources, any user can consume as many resources as they want, with
the hardlimit associated with the tenant/project being the upper bound for the
resources to be consumed. Quotas are tied closely to physical resources and are
billable entity and hence from Magnum's perspective it makes sense to limit the
creation and consumption of a particular kind of resource to a certain value.
Use cases
---------
Alice is the admin. She would like to have the feature which will give her
details of Magnum resource consumption so that she can manage her resource
appropriately.
a. Ability to know current resource consumption.
b. Ability to prohibit overuse by a project.
c. Prevent situation where users in the project get starved because users in
other project consume all the resource. Alice feels something like
"Quota Management" would help to guarantee "fairness".
d. Prevent DOS kind of attack, abuse or error by users where an excessive
amount of resources are created.
Proposed change
===============
Proposed change is to introduce a Quota Table which will primarily store the
quota assigned to each resource in a project. For Mitaka, we will restrict
the scope to a Bay, which are Magnum resources. Primarily, as a first step we
will start of by imposing quota on number of bays to be created in a project.
The change also plans to introduce REST API's to GET/PUT/POST/DELETE. CLIs to
get information of Quota for a particular project will also be provided.
For Mitaka, we will restrict the scope of the resources explicit created and
managed by Magnum. Specifically for Mitaka we will focus on number of
Bays only. Going ahead we might add Quota for containers, etc. The resources
of which a Bay is constructed out of is inherently not only Magnum resource
but involve resource from Nova, Cinder, Neutron etc. Limiting those resource
consumption is out of the scope of this spec and needs a close collaboration
with the quota management framework of the orchestration layer, since the
orchestration layer can invoke the respective IaaS projects API's and get the
consumption details before provisioning. As of now the orchestration layer
used by Magnum, Heat, does not have the concept of Quota, so we will start with
imposing Quota on resources which Magnum manages, Bay, more specifically for
Mitaka.
When a project is created and if the Magnum service is running, the default
quota for Magnum resources will be set by the values configured in magnum.conf.
Other Openstack projects like Nova [2]_, Cinder [3]_ follow a similar pattern
and we will also do so and hence won't have a separate CLI for quota-create.
Later if the user wants to change the Quota of the resource option will be
provided to do so using magnum quota-update. In situation where all of the
quota for a specific Magnum resource (Bay) have been consumed and is
under use, admin will be allowed to set the quota to a any value lower than
the usage or hardlimit to prohibit users from the project to create new
Bays. This gives more flexibility to the admin to have a better control
on resource consumption. Till the time the resource is not explicitly deleted
the quota associated with the project, for a particular resource, won't be
decreased. In short quota-update support will take into consideration the
new hardlimit for a resource, specified by the admin, and will set the new
value for this resource.
Before the resource is created, Magnum will check for current count of the
resource(Bays) created for a project. If the resource(Bay) count is less
than the hardlimit set for the Bay, new Bay creation will be allowed. Since
Bay creation is a long running operation, special care will be taken while
computing the available quota. For example, 'in-progress' field in the Quota
usages table will be updated when the resource(Bay) creation is initiated and
is in progress. Lets say the quota hardlimit is 5 and 3 Bay's have already been
created and two new requests come in to create new Bays. Since we have 3 Bays
already created the 'used' field will be set to 3. Now the 'in-progress'
field will be set to 2 till the time the Bay creation is successful. Once
the Bay creation is done this field will be reset to 0, and the 'used'
count will be updated from 3 to 5. So at this moment, hardlimit is 5, used
is 5 and in-progress is 0. So lets say one more request comes in to create
new Bay this request will be prohibited since there is not enough quota
available.
For Bays,
available = hard_limit - [in_progress + used]
In general,
Resource quota available = Resource hard_limit - [
(Resource creation in progress + Resources already created for project)]
Alternatives
------------
At present there is not quota infrastructure in Magnum.
Adding Quota Management layer at the Orchestration layer, Heat, could be an
alternative. Doing so will give a finer view of resource consumption at the
IaaS layer which can be used while provisioning Magnum resources which
depend on the IaaS layer [1]_.
Data model impact
-----------------
New Quota and Quota usages table will be introduced to Magnum database to
store quota consumption for each resource in a project.
Quota Table :
+------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+------------+--------------+------+-----+---------+----------------+
| id | int(11) | NO | PRI | NULL | auto_increment |
| created_at | datetime | YES | | NULL | |
| updated_at | datetime | YES | | NULL | |
| project_id | varchar(255) | YES | MUL | NULL | |
| resource | varchar(255) | NO | | NULL | |
| hard_limit | int(11) | YES | | NULL | |
+------------+--------------+------+-----+---------+----------------+
Quota usages table :
+---------------+--------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+---------------+--------------+------+-----+---------+----------------+
| created_at | datetime | YES | | NULL | |
| updated_at | datetime | YES | | NULL | |
| id | int(11) | NO | PRI | NULL | auto_increment |
| project_id | varchar(255) | YES | MUL | NULL | |
| resource | varchar(255) | NO | | NULL | |
| in_progress | int(11) | NO | | NULL | |
| used | int(11) | NO | | NULL | |
+---------------+--------------+------+-----+---------+----------------+
REST API impact
---------------
REST API will be added for :
1. quota-defaults List all default quotas for all tenants.
2. quota-show List the currently set quota values for a tenant.
3. quota-update Updates quotas for a tenant.
4. quota-usage Lists quota usage for a tenant.
5. quota-list List quota for all the tenants.
A user with "admin" role will be able to do all the above operations but a user
with "non-admin" role will be restricted to only get/list quota associated to
his/her tenant. User with "non-admin" role can be a Member of the tenant less
"admin" role.
REST API for resources which will have quota imposed will be enhanced :
1. Bay create
Will check if there is quota available for Bay creation, if so proceed
ahead with the request otherwise throw exception that not enough quota is
available.
Security impact
---------------
None
Notifications impact
--------------------
None
Other end user impact
---------------------
End user will have the option to look at the quota set on the resources, quota
usage by a particular project.
Performance Impact
------------------
None
Other deployer impact
---------------------
None
Developer impact
----------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
vilobhmm
Other contributors:
None
Work Items
----------
1. Introduce Quota and Quota usages table in Magnum database.
2. Introduce API to set/update Quota for a resource, specifically
bay, for Mitaka release.
3. Introduce API to create Quota entry, by default, for a resource.
4. Provide config options that will allow users/admins to set Quota.
5. Make sure that if the resource is deleted the used count from the
quota_usages table will be decremented by the number of resources
deleted. For example, if resource, bay, is deleted then the entries
for it in the Quota usages table should be decremented by the
number of Bays deleted.
6. Provide CLI options to view the quota details :
a. magnum quota-show <project-id>
b. magnum quota-update <project-id> <resource> <hard-limit>
c. magnum quota-defaults <project-id>
d. magnum quota-usage <project-id>
e. magnum quota-list
7. Add conf setting for bays default quota since we will focus
on Bays for Mitaka.
Dependencies
============
None
Testing
=======
1. Each commit will be accompanied with unit tests.
2. Gate functional tests will also be covered.
Documentation Impact
====================
None
References
==========
.. [1] http://lists.openstack.org/pipermail/openstack-dev/2015-December/082266.html
.. [2] https://github.com/openstack/nova/blob/master/nova/quota.py
.. [3] https://github.com/openstack/nova/blob/master/cinder/quota.py

View File

@ -0,0 +1,226 @@
=====================
TLS support in Magnum
=====================
Launchpad blueprint:
https://blueprints.launchpad.net/magnum/+spec/secure-kubernetes
Currently there is no authentication in Magnum to provide access control to
limit communication between the Magnum service and the Kubernetes service so
that Kubernetes can not be controlled by a third party. This implementation
closes this security loophole by using TLS as an access control mechanism.
Only the Magnum server will have the key to communicate with any given
Kubernetes API service under its control. An additional benefit of this
approach is that communication over the network will be encrypted, reducing
the chance of eavesdropping on the communication stream.
Problem Description
-------------------
Magnum currently controls Kubernetes API services using unauthenticated HTTP.
If an attacker knows the api_address of a Kubernetes Bay, (s)he can control
the cluster without any access control.
Use Cases
---------
1. Operators expect system level control to be protected by access control that
is consistent with industry best practices. Lack of this feature may result in
rejection of Magnum as an option for hosting containerized workloads.
Proposed Changes
----------------
The complete implementation of TLS support in Magnum can be further decomposed
into below smaller implementations.
1. TLS support in Kubernetes Client Code.
-----------------------------------------
The current implementation of Kubernetes Client code doesn't have any
authentication. So this implementation will change the client code to
provide authentication using TLS.
Launchpad blueprint:
https://blueprints.launchpad.net/magnum/+spec/tls-pythonk8sclient
2. Generating certificates
----------------------------
This task is mainly on how certificates for both client(magnum-conductor)
and server(kube-apiserver) will be generated and who will be the certificate
authority(CA).
These files can be generated in two ways:
2.1. Magnum script
-------------------
This implementation will use standard tool to generate certificates and
keys. This script will be registered on Kubernetes master node while creating
bay. This script will generate certificates, start the secure kube-apiserver
and then register the client certificates at Magnum.
2.2. Using Barbican
-------------------
Barbican can also be used as a CA using Dogtag. This implementation will use
Barbican to generate certificates.
3. TLS Support in Magnum code
------------------------------
This work mainly involves deploying a secure bay and supporting the use of
certificates in Magnum to call Kubernetes API. This implementation can be
decomposed into smaller tasks.
3.1. Create secure bay
----------------------
This implementation will deploy a secure kube-apiserver running on Kubernetes
master node. To do so following things needs to be done:
* Generate certificates
* Copy certificates
* Start a secure kube-apiserver
3.1.1. Generate certificates
----------------------------
The certificates will be generated using any of the above implementation in
section 2.
3.1.2. Copy certificates
------------------------
This depends on how cert and key is generated, the implementation will differ
with each case.
3.1.2.1. Using Magnum script
----------------------------
This script will generate both server and client certificates on Kubernetes
master node. Hence only client certificates needs to be copied to magnum host
node. To copy these files, the script will make a call to magnum-api to store
files.
3.1.2.2. Using Barbican
-----------------------
When using Barbican, the cert and key will be generated and stored in Barbican
itself. Either magnum-conductor can fetch the certificates from Barbican and
copy on Kubernetes master node or it can be fetched from Kubernetes master node
also.
3.1.3. Start a secure kube-apiserver
------------------------------------
Above generated certificates will be used to start a secure kube-apiserver
running on Kubernetes master node.
Now that we have a secure Kubernetes cluster running, any API call to
Kubernetes will be secure.
3.2. Support https
------------------
While running any Kubernetes resource related APIs, magnum-conductor will
fetch certificate from magnum database or Barbican and use it to make secure
API call.
4. Barbican support to store certificates securely
----------------------------------------------------
Barbican is a REST API designed for the secure storage, provisioning and
management of secrets. The client cert and key must be stored securely. This
implementation will support Barbican in Magnum to store the sensitive data.
Data model impact
-----------------
New table 'cert' will be introduced to store the certificates.
REST API impact
---------------
New API /certs will be introduced to store the certificates.
Security impact
---------------
After this support, Magnum will be secure to be used in actual production
environment. Now all the communication to Kubernetes master node will be
secure.
The certificates will be generated by Barbican or standard tool signed by
trusted CAs.
The certificates will be stored safely in Barbican when the Barbican cert
storage option is selected by the administrator.
Notifications impact
--------------------
None
Other end user impact
---------------------
None
Performance impact
------------------
None
Other deployer impact
---------------------
Deployer will need to install Barbican to store certificates.
Developer impact
----------------
None
Implementation
--------------
Assignee(s)
-----------
Primary assignee
madhuri(Madhuri Kumari)
yuanying(Motohiro Otsuka)
Work Items
----------
1. TLS Support in Kubernetes Client code
2. Support for generating keys in Magnum
3. Support creating secure Kubernetes cluster
4. Support Barbican in Magnum to store certificates
Dependencies
------------
Barbican(optional)
Testing
-------
Each commit will be accompanied with unit tests. There will also be functional
test to test both good and bad certificates.
Documentation Impact
--------------------
Add a document explaining how TLS cert and keys can be generated and guide
updated with how to use the secure model of bays.
References
----------
None