Import all implemented specs from magnum repo
* Add all implemented specs to specs/implemented. * Create ocata directory to store the corresponding specs. Change-Id: I36da3a26ece3b6f7d27b7af80460303f4a6c4226
This commit is contained in:
parent
74da0acf5d
commit
8b0a99e44b
|
@ -0,0 +1 @@
|
|||
../../specs
|
|
@ -0,0 +1,452 @@
|
|||
=================================
|
||||
Asynchronous Container Operations
|
||||
=================================
|
||||
|
||||
Launchpad blueprint:
|
||||
|
||||
https://blueprints.launchpad.net/magnum/+spec/async-container-operations
|
||||
|
||||
At present, container operations are done in a synchronous way, end-to-end.
|
||||
This model does not scale well, and incurs a penalty on the client to be
|
||||
stuck till the end of completion of the operation.
|
||||
|
||||
Problem Description
|
||||
-------------------
|
||||
|
||||
At present Magnum-Conductor executes the container operation as part of
|
||||
processing the request forwarded from Magnum-API. For
|
||||
container-create, if the image needs to be pulled down, it may take
|
||||
a while depending on the responsiveness of the registry, which can be a
|
||||
substantial delay. At the same time, experiments suggest that even for
|
||||
pre-pulled image, the time taken by each operations, namely
|
||||
create/start/delete, are in the same order, as it involves complete turn
|
||||
around between the magnum-client and the COE-API, via Magnum-API and
|
||||
Magnum-Conductor[1].
|
||||
|
||||
Use Cases
|
||||
---------
|
||||
|
||||
For wider enterprise adoption of Magnum, we need it to scale better.
|
||||
For that we need to replace some of these synchronous behaviors with
|
||||
suitable alternative of asynchronous implementation.
|
||||
|
||||
To understand the use-case better, we can have a look at the average
|
||||
time spent during container operations, as noted at[1].
|
||||
|
||||
Proposed Changes
|
||||
----------------
|
||||
|
||||
The design has been discussed over the ML[6]. The conclusions have been kept
|
||||
on the 'whiteboard' of the Blueprint.
|
||||
|
||||
The amount of code change is expected to be significant. To ease the
|
||||
process of adoption, code review, functional tests, an approach of phased
|
||||
implementation may be required. We can define the scope of the three phases of
|
||||
the implementation as follows -
|
||||
|
||||
* Phase-0 will bring in the basic feature of asynchronous mode of operation in
|
||||
Magnum - (A) from API to Conductor and (B) from Conductor to COE-API. During
|
||||
phase-0, this mode will be optional through configuration.
|
||||
|
||||
Both the communications of (A) and (B) are proposed to be made asynchronous
|
||||
to achieve the best of it. If we do (A) alone, it does not gain us much, as
|
||||
(B) takes up the higher cycles of the operation. If we do (B) alone, it does
|
||||
not make sense, as (A) will synchronously wait for no meaningful data.
|
||||
|
||||
* Phase-1 will concentrate on making the feature persistent to address various
|
||||
scenarios of conductor restart, worker failure etc. We will support this
|
||||
feature for multiple Conductor-workers in this phase.
|
||||
|
||||
* Phase-2 will select asynchronous mode of operation as the default mode. At
|
||||
the same time, we can evaluate to drop the code for synchronous mode, too.
|
||||
|
||||
|
||||
Phase-0 is required as a meaningful temporary step, to establish the
|
||||
importance and tangible benefits of phase-1. This is also to serve as a
|
||||
proof-of-concept at a lower cost of code changes with a configurable option.
|
||||
This will enable developers and operators to have a taste of the feature,
|
||||
before bringing in the heavier dependencies and changes proposed in phase-1.
|
||||
|
||||
A reference implementation for the phase-0 items, has been put for review[2].
|
||||
|
||||
Following is the summary of the design -
|
||||
|
||||
1. Configurable mode of operation - async
|
||||
-----------------------------------------
|
||||
|
||||
For ease of adoption, the async_mode of communication between API-conductor,
|
||||
conductor-COE in magnum, can be controlled using a configuration option. So
|
||||
the code-path for sync mode and async mode would co-exist for now. To achieve
|
||||
this with minimal/no code duplication and cleaner interface, we are using
|
||||
openstack/futurist[4]. Futurist interface hides the details of type of executor
|
||||
being used. In case of async configuration, a greenthreadpool of configured
|
||||
poolsize gets created. Here is a sample of how the config would look
|
||||
like: ::
|
||||
|
||||
[DEFAULT]
|
||||
async_enable = False
|
||||
|
||||
[conductor]
|
||||
async_threadpool_max_workers = 64
|
||||
|
||||
Futurist library is used in oslo.messaging. Thus, it is used by almost all
|
||||
OpenStack projects, in effect. Futurist is very useful to run same code
|
||||
under different execution model and hence saving potential duplication of
|
||||
code.
|
||||
|
||||
|
||||
2. Type of operations
|
||||
---------------------
|
||||
|
||||
There are two classes of container operations - one that can be made async,
|
||||
namely create/delete/start/stop/pause/unpause/reboot, which do not need data
|
||||
about the container in return. The other type requires data, namely
|
||||
container-logs. For async-type container-operations, magnum-API will be
|
||||
using 'cast' instead of 'call' from oslo_messaging[5].
|
||||
|
||||
'cast' from oslo.messaging.rpcclient is used to invoke a method and return
|
||||
immediately, whereas 'call' invokes a method and waits for a reply. While
|
||||
operating in asynchronous mode, it is intuitive to use cast method, as the
|
||||
result of the response may not be available immediately.
|
||||
|
||||
Magnum-api first fetches the details of a container, by doing
|
||||
'get_rpc_resource'. This function uses magnum objects. Hence, this function
|
||||
uses a 'call' method underneath. Once, magnum-api gets back the details,
|
||||
it issues the container operation next, using another 'call' method.
|
||||
The above proposal is to replace the second 'call' with 'cast'.
|
||||
|
||||
If user issues a container operation, when there is no listening
|
||||
conductor (because of process failure), there will be a RPC timeout at the
|
||||
first 'call' method. In this case, user will observe the request to
|
||||
get blocked at client and finally fail with HTTP 500 ERROR, after the RPC
|
||||
timeout, which is 60 seconds by default. This behavior is independent of the
|
||||
usage of 'cast' or 'call' for the second message, mentioned above. This
|
||||
behavior does not influence our design, but it is documented here for clarity
|
||||
of understanding.
|
||||
|
||||
|
||||
3. Ensuring the order of execution - Phase-0
|
||||
--------------------------------------------
|
||||
|
||||
Magnum-conductor needs to ensure that for a given bay and given container,
|
||||
the operations are executed in sequence. In phase-0, we want to demonstrate
|
||||
how asynchronous behavior helps scaling. Asynchronous mode of container
|
||||
operations would be supported for single magnum-conductor scenario, in
|
||||
phase-0. If magnum-conductor crashes, there will be no recovery for the
|
||||
operations accepted earlier - which means no persistence in phase-0, for
|
||||
operations accepted by magnum-conductor. Multiple conductor scenario and
|
||||
persistence will be addressed in phase-1 [please refer to the next section
|
||||
for further details]. If COE crashes or does not respond, the error will be
|
||||
detected, as it happens in sync mode, and reflected on the container-status.
|
||||
|
||||
Magnum-conductor will maintain a job-queue. Job-queue is indexed by bay-id and
|
||||
container-id. A job-queue entry would contain the sequence of operations
|
||||
requested for a given bay-id and container-id, in temporal order. A
|
||||
greenthread will execute the tasks/operations in order for a given job-queue
|
||||
entry, till the queue empties. Using a greenthread in this fashion saves us
|
||||
from the cost and complexity of locking, along with functional correctness.
|
||||
When request for new operation comes in, it gets appended to the corresponding
|
||||
queue entry.
|
||||
|
||||
For a sequence of container operations, if an intermediate operation fails,
|
||||
we will stop continuing the sequence. The community feels more confident to
|
||||
start with this strictly defensive policy[17]. The failure will be logged
|
||||
and saved into the container-object, which will help an operator be informed
|
||||
better about the result of the sequence of container operations. We may revisit
|
||||
this policy later, if we think it is too restrictive.
|
||||
|
||||
4. Ensuring the order of execution - phase-1
|
||||
--------------------------------------------
|
||||
|
||||
The goal is to execute requests for a given bay and a given container in
|
||||
sequence. In phase-1, we want to address persistence and capability of
|
||||
supporting multiple magnum-conductor processes. To achieve this, we will
|
||||
reuse the concepts laid out in phase-0 and use a standard library.
|
||||
|
||||
We propose to use taskflow[7] for this implementation. Magnum-conductors
|
||||
will consume the AMQP message and post a task[8] on a taskflow jobboard[9].
|
||||
Greenthreads from magnum-conductors would subscribe to the taskflow
|
||||
jobboard as taskflow-conductors[10]. Taskflow jobboard is maintained with
|
||||
a choice of persistent backend[11]. This will help address the concern of
|
||||
persistence for accepted operations, when a conductor crashes. Taskflow
|
||||
will ensure that tasks, namely container operations, in a job, namely a
|
||||
sequence of operations for a given bay and container, would execute in
|
||||
sequence. We can easily notice that some of the concepts used in phase-0
|
||||
are reused as it is. For example, job-queue maps to jobboard here, use of
|
||||
greenthread maps to the conductor concept of taskflow. Hence, we expect easier
|
||||
migration from phase-0 to phase-1, with the choice of taskflow.
|
||||
|
||||
For taskflow jobboard[11], the available choices of backend are Zookeeper and
|
||||
Redis. But, we plan to use MySQL as default choice of backend, for magnum
|
||||
conductor jobboard use-case. This support will be added to taskflow. Later,
|
||||
we may choose to support the flexibility of other backends like ZK/Redis via
|
||||
configuration. But, phase-1 will keep the implementation simple with MySQL
|
||||
backend and revisit this, if required.
|
||||
|
||||
Let's consider the scenarios of Conductor crashing -
|
||||
- If a task is added to jobboard, and conductor crashes after that,
|
||||
taskflow can assign a particular job to any available greenthread agents
|
||||
from other conductor instances. If the system was running with single
|
||||
magnum-conductor, it will wait for the conductor to come back and join.
|
||||
- A task is picked up and magnum-conductor crashes. In this case, the task
|
||||
is not complete from jobboard point-of-view. As taskflow detects the
|
||||
conductor going away, it assigns another available conductor.
|
||||
- When conductor picks up a message from AMQP, it will acknowledge AMQP,
|
||||
only after persisting it to jobboard. This will prevent losing the message,
|
||||
if conductor crashes after picking up the message from AMQP. Explicit
|
||||
acknowledgement from application may use NotificationResult.HANDLED[12]
|
||||
to AMQP. We may use the at-least-one-guarantee[13] feature in
|
||||
oslo.messaging[14], as it becomes available.
|
||||
|
||||
To summarize some of the important outcomes of this proposal -
|
||||
- A taskflow job represents the sequence of container operations on a given
|
||||
bay and given container. At a given point of time, the sequence may contain
|
||||
a single or multiple operations.
|
||||
- There will be a single jobboard for all conductors.
|
||||
- Task-flow conductors are multiple greenthreads from a given
|
||||
magnum-conductor.
|
||||
- Taskflow-conductor will run in 'blocking' mode[15], as those greenthreads
|
||||
have no other job than claiming and executing the jobs from jobboard.
|
||||
- Individual jobs are supposed to maintain a temporal sequence. So the
|
||||
taskflow-engine would be 'serial'[16].
|
||||
- The proposed model for a 'job' is to consist of a temporal sequence of
|
||||
'tasks' - operations on a given bay and a given container. Henceforth,
|
||||
it is expected that when a given operation, namely container-create is in
|
||||
progress, a request for container-start may come in. Adding the task to
|
||||
the existing job is intuitive to maintain the sequence of operations.
|
||||
|
||||
To fit taskflow exactly into our use-case, we may need to do two enhancements
|
||||
in taskflow -
|
||||
- Supporting mysql plugin as a DB backend for jobboard. Support for redis
|
||||
exists, so it will be similar.
|
||||
We do not see any technical roadblock for adding mysql support for taskflow
|
||||
jobboard. If the proposal does not get approved by taskflow team, we may have
|
||||
to use redis, as an alternative option.
|
||||
- Support for dynamically adding tasks to a job on jobboard. This also looks
|
||||
feasible, as discussed over the #openstack-state-management [Unfortunately,
|
||||
this channel is not logged, but if we agree in this direction, we can initiate
|
||||
discussion over ML, too]
|
||||
If taskflow team does not allow adding this feature, even though they have
|
||||
agreed now, we will use the dependency feature in taskflow. We will explore
|
||||
and elaborate this further, if it requires.
|
||||
|
||||
|
||||
5. Status of progress
|
||||
---------------------
|
||||
|
||||
The progress of execution of a container operation is reflected on the status
|
||||
of a container as - 'create-in-progress', 'delete-in-progress' etc.
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
Without an asynchronous implementation, Magnum will suffer from complaints
|
||||
about poor scalability and slowness.
|
||||
|
||||
In this design, stack-lock[3] has been considered as an alternative to
|
||||
taskflow. Following are the reasons for preferring taskflow over
|
||||
stack-lock, as of now,
|
||||
- Stack-lock used in Heat is not a library, so it will require making a copy
|
||||
for Magnum, which is not desirable.
|
||||
- Taskflow is relatively mature, well supported, feature-rich library.
|
||||
- Taskflow has in-built capacity to scale out[in] as multiple conductors
|
||||
can join in[out] the cluster.
|
||||
- Taskflow has a failure detection and recovery mechanism. If a process
|
||||
crashes, then worker threads from other conductor may continue the execution.
|
||||
|
||||
In this design, we describe futurist[4] as a choice of implementation. The
|
||||
choice was to prevent duplication of code for async and sync mode. For this
|
||||
purpose, we could not find any other solution to compare.
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
|
||||
Phase-0 has no data model impact. But phase-1 may introduce an additional
|
||||
table into the Magnum database. As per the present proposal for using taskflow
|
||||
in phase-1, we have to introduce a new table for jobboard under magnum db.
|
||||
This table will be exposed to taskflow library as a persistent db plugin.
|
||||
Alternatively, an implementation with stack-lock will also require an
|
||||
introduction of a new table for stack-lock objects.
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
|
||||
None.
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
|
||||
None.
|
||||
|
||||
Notifications impact
|
||||
--------------------
|
||||
|
||||
None
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
|
||||
None
|
||||
|
||||
Performance impact
|
||||
------------------
|
||||
|
||||
Asynchronous mode of operation helps in scalability. Hence, it improves
|
||||
responsiveness and reduces the turn around time in a significant
|
||||
proportion. A small test on devstack, comparing both the modes,
|
||||
demonstrate this with numbers.[1]
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
|
||||
None.
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
None
|
||||
|
||||
Implementation
|
||||
--------------
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee
|
||||
suro-patz(Surojit Pathak)
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
For phase-0
|
||||
* Introduce config knob for asynchronous mode of container operations.
|
||||
|
||||
* Changes for Magnum-API to use CAST instead of CALL for operations eligible
|
||||
for asynchronous mode.
|
||||
|
||||
* Implement the in-memory job-queue in Magnum conductor, and integrate futurist
|
||||
library.
|
||||
|
||||
* Unit tests and functional tests for async mode.
|
||||
|
||||
* Documentation changes.
|
||||
|
||||
For phase-1
|
||||
* Get the dependencies on taskflow being resolved.
|
||||
|
||||
* Introduce jobboard table into Magnum DB.
|
||||
|
||||
* Integrate taskflow in Magnum conductor to replace the in-memory job-queue
|
||||
with taskflow jobboard. Also, we need conductor greenthreads to subscribe
|
||||
as workers to the taskflow jobboard.
|
||||
|
||||
* Add unit tests and functional tests for persistence and multiple conductor
|
||||
scenario.
|
||||
|
||||
* Documentation changes.
|
||||
|
||||
For phase-2
|
||||
* We will promote asynchronous mode of operation as the default mode of
|
||||
operation.
|
||||
|
||||
* We may decide to drop the code for synchronous mode and corresponding config.
|
||||
|
||||
* Documentation changes.
|
||||
|
||||
|
||||
Dependencies
|
||||
------------
|
||||
|
||||
For phase-1, if we choose to implement using taskflow, we need to get
|
||||
following two features added to taskflow first -
|
||||
* Ability to add new task to an existing job on jobboard.
|
||||
* mysql plugin support as persistent DB.
|
||||
|
||||
Testing
|
||||
-------
|
||||
|
||||
All the existing test cases are run to ensure async mode does not break them.
|
||||
Additionally more functional tests and unit tests will be added specific to
|
||||
async mode.
|
||||
|
||||
Documentation Impact
|
||||
--------------------
|
||||
|
||||
Magnum documentation will include a description of the option for asynchronous
|
||||
mode of container operations and its benefits. We will also add to
|
||||
developer documentation on guideline for implementing a container operation in
|
||||
both the modes - sync and async. We will add a section on 'how to debug
|
||||
container operations in async mode'. The phase-0 and phase-1 implementation
|
||||
and their support for single or multiple conductors will be clearly documented
|
||||
for the operators.
|
||||
|
||||
References
|
||||
----------
|
||||
|
||||
[1] - Execution time comparison between sync and async modes:
|
||||
|
||||
https://gist.github.com/surojit-pathak/2cbdad5b8bf5b569e755
|
||||
|
||||
[2] - Proposed change under review:
|
||||
|
||||
https://review.openstack.org/#/c/267134/
|
||||
|
||||
[3] - Heat's use of stacklock
|
||||
|
||||
http://docs.openstack.org/developer/heat/_modules/heat/engine/stack_lock.html
|
||||
|
||||
[4] - openstack/futurist
|
||||
|
||||
http://docs.openstack.org/developer/futurist/
|
||||
|
||||
[5] - openstack/oslo.messaging
|
||||
|
||||
http://docs.openstack.org/developer/oslo.messaging/rpcclient.html
|
||||
|
||||
[6] - ML discussion on the design
|
||||
|
||||
http://lists.openstack.org/pipermail/openstack-dev/2015-December/082524.html
|
||||
|
||||
[7] - Taskflow library
|
||||
|
||||
http://docs.openstack.org/developer/taskflow/
|
||||
|
||||
[8] - task in taskflow
|
||||
|
||||
http://docs.openstack.org/developer/taskflow/atoms.html#task
|
||||
|
||||
[9] - job and jobboard in taskflow
|
||||
|
||||
http://docs.openstack.org/developer/taskflow/jobs.html
|
||||
|
||||
[10] - conductor in taskflow
|
||||
|
||||
http://docs.openstack.org/developer/taskflow/conductors.html
|
||||
|
||||
[11] - persistent backend support in taskflow
|
||||
|
||||
http://docs.openstack.org/developer/taskflow/persistence.html
|
||||
|
||||
[12] - oslo.messaging notification handler
|
||||
|
||||
http://docs.openstack.org/developer/oslo.messaging/notification_listener.html
|
||||
|
||||
[13] - Blueprint for at-least-once-guarantee, oslo.messaging
|
||||
|
||||
https://blueprints.launchpad.net/oslo.messaging/+spec/at-least-once-guarantee
|
||||
|
||||
[14] - Patchset under review for at-least-once-guarantee, oslo.messaging
|
||||
|
||||
https://review.openstack.org/#/c/229186/
|
||||
|
||||
[15] - Taskflow blocking mode for conductor
|
||||
|
||||
http://docs.openstack.org/developer/taskflow/conductors.html#taskflow.conductors.backends.impl_executor.ExecutorConductor
|
||||
|
||||
[16] - Taskflow serial engine
|
||||
|
||||
http://docs.openstack.org/developer/taskflow/engines.html
|
||||
|
||||
[17] - Community feedback on policy to handle failure within a sequence
|
||||
|
||||
http://eavesdrop.openstack.org/irclogs/%23openstack-containers/%23openstack-containers.2016-03-08.log.html#t2016-03-08T20:41:17
|
|
@ -0,0 +1,344 @@
|
|||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
======================================
|
||||
Container Orchestration Engine drivers
|
||||
======================================
|
||||
|
||||
Launchpad blueprint:
|
||||
|
||||
https://blueprints.launchpad.net/magnum/+spec/bay-drivers
|
||||
|
||||
Container Orchestration Engines (COEs) are different systems for managing
|
||||
containerized applications in a clustered environment, each having their own
|
||||
conventions and ecosystems. Three of the most common, which also happen to be
|
||||
supported in Magnum, are: Docker Swarm, Kubernetes, and Mesos. In order to
|
||||
successfully serve developers, Magnum needs to be able to provision and manage
|
||||
access to the latest COEs through its API in an effective and scalable way.
|
||||
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
Magnum currently supports the three most popular COEs, but as more emerge and
|
||||
existing ones change, it needs an effective and scalable way of managing
|
||||
them over time.
|
||||
|
||||
One of the problems with the current implementation is that COE-specific logic,
|
||||
such as Kubernetes replication controllers and services, is situated in the
|
||||
core Magnum library and made available to users through the main API. Placing
|
||||
COE-specific logic in a core API introduces tight coupling and forces
|
||||
operators to work with an inflexible design.
|
||||
|
||||
By formalising a more modular and extensible architecture, Magnum will be
|
||||
in a much better position to help operators and consumers satisfy custom
|
||||
use-cases.
|
||||
|
||||
Use cases
|
||||
---------
|
||||
|
||||
1. Extensibility. Contributors and maintainers need a suitable architecture to
|
||||
house current and future COE implementations. Moving to a more extensible
|
||||
architecture, where core classes delegate to drivers, provides a more
|
||||
effective and elegant model for handling COE differences without the need
|
||||
for tightly coupled and monkey-patched logic.
|
||||
|
||||
One of the key use cases is allowing operators to customise their
|
||||
orchestration logic, such as modifying Heat templates or even using their
|
||||
own tooling like Ansible. Moreover, operators will often expect to use a
|
||||
custom distro image with lots of software pre-installed and many special
|
||||
security requirements that is extremely difficult or impossible to do with
|
||||
the current upstream templates. COE drivers solves these problems.
|
||||
|
||||
2. Maintainability. Moving to a modular architecture will be easier to manage
|
||||
in the long-run because the responsibility of maintaining non-standard
|
||||
implementations is shifted into the operator's domain. Maintaining the
|
||||
default drivers which are packaged with Magnum will also be easier and
|
||||
cleaner since logic is now demarcated from core codebase directories.
|
||||
|
||||
3. COE & Distro choice. In the community there has been a lot of discussion
|
||||
about which distro and COE combination to support with the templates.
|
||||
Having COE drivers allows for people or organizations to maintain
|
||||
distro-specific implementations (e.g CentOS+Kubernetes).
|
||||
|
||||
4. Addresses dependency concerns. One of the direct results of
|
||||
introducing a driver model is the ability to give operators more freedom
|
||||
about choosing how Magnum integrates with the rest of their OpenStack
|
||||
platform. For example, drivers would remove the necessity for users to
|
||||
adopt Barbican for secret management.
|
||||
|
||||
5. Driver versioning. The new driver model allows operators to modify existing
|
||||
drivers or creating custom ones, release new bay types based on the newer
|
||||
version, and subsequently launch news bays running the updated
|
||||
functionality. Existing bays which are based on older driver versions would
|
||||
be unaffected in this process and would still be able to have lifecycle
|
||||
operations performed on them. If one were to list their details from the
|
||||
API, it would reference the old driver version. An operator can see which
|
||||
driver version a bay type is based on through its ``driver`` value,
|
||||
which is exposed through the API.
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
1. The creation of new directory at the project root: ``./magnum/drivers``.
|
||||
Each driver will house its own logic inside its own directory. Each distro
|
||||
will house its own logic inside that driver directory. For example, the
|
||||
Fedora Atomic distro using Swarm will have the following directory
|
||||
structure:
|
||||
|
||||
::
|
||||
|
||||
drivers/
|
||||
swarm_atomic_v1/
|
||||
image/
|
||||
...
|
||||
templates/
|
||||
...
|
||||
api.py
|
||||
driver.py
|
||||
monitor.py
|
||||
scale.py
|
||||
template_def.py
|
||||
version.py
|
||||
|
||||
|
||||
The directory name should be a string which uniquely identifies the driver
|
||||
and provides a descriptive reference. The driver version number and name are
|
||||
provided in the manifest file and will be included in the bay metadata at
|
||||
cluster build time.
|
||||
|
||||
There are two workflows for rolling out driver updates:
|
||||
|
||||
- if the change is relatively minor, they modify the files in the
|
||||
existing driver directory and update the version number in the manifest
|
||||
file.
|
||||
|
||||
- if the change is significant, they create a new directory
|
||||
(either from scratch or by forking).
|
||||
|
||||
Further explanation of the three top-level files:
|
||||
|
||||
- an ``image`` directory is *optional* and should contain documentation
|
||||
which tells users how to build the image and register it to glance. This
|
||||
directory can also hold artifacts for building the image, for instance
|
||||
diskimagebuilder elements, scripts, etc.
|
||||
|
||||
- a ``templates`` directory is *required* and will (for the forseeable
|
||||
future) store Heat template YAML files. In the future drivers will allow
|
||||
operators to use their own orchestration tools like Ansible.
|
||||
|
||||
- ``api.py`` is *optional*, and should contain the API controller which
|
||||
handles custom API operations like Kubernetes RCs or Pods. It will be
|
||||
this class which accepts HTTP requests and delegates to the Conductor. It
|
||||
should contain a uniquely named class, such as ``SwarmAtomicXYZ``, which
|
||||
extends from the core controller class. The COE class would have the
|
||||
opportunity of overriding base methods if necessary.
|
||||
|
||||
- ``driver.py`` is *required*, and should contain the logic which maps
|
||||
controller actions to COE interfaces. These include: ``bay_create``,
|
||||
``bay_update``, ``bay_delete``, ``bay_rebuild``, ``bay_soft_reboot`` and
|
||||
``bay_hard_reboot``.
|
||||
|
||||
- ``version.py`` is *required*, and should contain the version number of
|
||||
the bay driver. This is defined by a ``version`` attribute and is
|
||||
represented in the ``1.0.0`` format. It should also include a ``Driver``
|
||||
attribute and should be a descriptive name such as ``swarm_atomic``.
|
||||
|
||||
Due to the varying nature of COEs, it is up to the bay
|
||||
maintainer to implement this in their own way. Since a bay is a
|
||||
combination of a COE and an image, ``driver.py`` will also contain
|
||||
information about the ``os_distro`` property which is expected to be
|
||||
attributed to Glance image.
|
||||
|
||||
- ``monitor.py`` is *optional*, and should contain the logic which monitors
|
||||
the resource utilization of bays.
|
||||
|
||||
- ``template_def.py`` is *required* and should contain the COE's
|
||||
implementation of how orchestration templates are loaded and matched to
|
||||
Magnum objects. It would probably contain multiple classes, such as
|
||||
``class SwarmAtomicXYZTemplateDef(BaseTemplateDefinition)``.
|
||||
|
||||
- ``scale.py`` is *optional* per bay specification and should contain the
|
||||
logic for scaling operations.
|
||||
|
||||
2. Renaming the ``coe`` attribute of BayModel to ``driver``. Because this
|
||||
value would determine which driver classes and orchestration templates to
|
||||
load, it would need to correspond to the name of the driver as it is
|
||||
registered with stevedore_ and setuptools entry points.
|
||||
|
||||
During the lifecycle of an API operation, top-level Magnum classes (such as
|
||||
a Bay conductor) would then delegate to the driver classes which have been
|
||||
dynamically loaded. Validation will need to ensure that whichever value
|
||||
is provided by the user is correct.
|
||||
|
||||
By default, drivers are located under the main project directory and their
|
||||
namespaces are accessible via ``magnum.drivers.foo``. But a use case that
|
||||
needs to be looked at and, if possible, provided for is drivers which are
|
||||
situated outside the project directory, for example in
|
||||
``/usr/share/magnum``. This will suit operators who want greater separation
|
||||
between customised code and Python libraries.
|
||||
|
||||
3. The driver implementations for the 3 current COE and Image combinations:
|
||||
Docker Swarm Fedora, Kubernetes Fedora, Kubernetes CoreOS, and Mesos
|
||||
Ubuntu. Any templates would need to be moved from
|
||||
``magnum/templates/{coe_name}`` to
|
||||
``magnum/drivers/{driver_name}/templates``.
|
||||
|
||||
4. Removal of the following files:
|
||||
|
||||
::
|
||||
|
||||
magnum/magnum/conductor/handlers/
|
||||
docker_conductor.py
|
||||
k8s_conducter.py
|
||||
|
||||
Design Principles
|
||||
-----------------
|
||||
|
||||
- Minimal, clean API without a high cognitive burden
|
||||
|
||||
- Ensure Magnum's priority is to do one thing well, but allow extensibility
|
||||
by external contributors
|
||||
|
||||
- Do not force ineffective abstractions that introduce feature divergence
|
||||
|
||||
- Formalise a modular and loosely coupled driver architecture that removes
|
||||
COE logic from the core codebase
|
||||
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
This alternative relates to #5 of Proposed Change. Instead of having a
|
||||
drivers registered using stevedore_ and setuptools entry points, an alternative
|
||||
is to use the Magnum config instead.
|
||||
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
|
||||
Since drivers would be implemented for the existing COEs, there would be
|
||||
no loss of functionality for end-users.
|
||||
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
|
||||
Attribute change when creating and updating a BayModel (``coe`` to
|
||||
``driver``). This would occur before v1 of the API is frozen.
|
||||
|
||||
COE-specific endpoints would be removed from the core API.
|
||||
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
|
||||
None
|
||||
|
||||
|
||||
Notifications impact
|
||||
--------------------
|
||||
|
||||
None
|
||||
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
|
||||
There will be deployer impacts because deployers will need to select
|
||||
which drivers they want to activate.
|
||||
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
None
|
||||
|
||||
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
|
||||
In order to utilize new functionality and bay drivers, operators will need
|
||||
to update their installation and configure bay models to use a driver.
|
||||
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
Due to the significant impact on the current codebase, a phased implementation
|
||||
approach will be necessary. This is defined in the Work Items section.
|
||||
|
||||
Code will be contributed for COE-specific functionality in a new way, and will
|
||||
need to abide by the new architecture. Documentation and a good first
|
||||
implementation will play an important role in helping developers contribute
|
||||
new functionality.
|
||||
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
murali-allada
|
||||
|
||||
Other contributors:
|
||||
jamiehannaford
|
||||
strigazi
|
||||
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
1. New ``drivers`` directory
|
||||
|
||||
2. Change ``coe`` attribute to ``driver``
|
||||
|
||||
3. COE drivers implementation (swarm-fedora, k8s-fedora, k8s-coreos,
|
||||
mesos-ubuntu). Templates should remain in directory tree until their
|
||||
accompanying driver has been implemented.
|
||||
|
||||
4. Delete old conductor files
|
||||
|
||||
5. Update client
|
||||
|
||||
6. Add documentation
|
||||
|
||||
7. Improve user experience for operators of forking/creating new
|
||||
drivers. One way we could do this is by creating new client commands or
|
||||
scripts. This is orthogonal to this spec, and will be considered after
|
||||
its core implementation.
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
None
|
||||
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
Each commit will be accompanied with unit tests, and Tempest functional tests.
|
||||
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
A set of documentation for this architecture will be required. We should also
|
||||
provide a developer guide for creating a new bay driver and updating existing
|
||||
ones.
|
||||
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
`Using Stevedore in your Application
|
||||
<http://docs.openstack.org/developer/stevedore/tutorial/index.html/>`_.
|
||||
|
||||
.. _stevedore: http://docs.openstack.org/developer/stevedore/
|
|
@ -0,0 +1,458 @@
|
|||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
=================================
|
||||
Magnum Container Networking Model
|
||||
=================================
|
||||
|
||||
Launchpad Blueprint:
|
||||
|
||||
https://blueprints.launchpad.net/magnum/+spec/extensible-network-model
|
||||
|
||||
For Magnum to prosper, the project must support a range of networking tools
|
||||
and techniques, while maintaining a simple, developer-focused user
|
||||
experience. The first step in achieving this goal is to standardize the
|
||||
process of allocating networking to containers, while providing an
|
||||
abstraction for supporting various networking capabilities through
|
||||
pluggable back-end implementations. This document recommends using Docker's
|
||||
libnetwork library to implement container networking abstractions and
|
||||
plugins. Since libnetwork is not a standard and the container ecosystem
|
||||
is rapidly evolving, the Magnum community should continue evaluating
|
||||
container networking options on a frequent basis.
|
||||
|
||||
Problem Description
|
||||
===================
|
||||
|
||||
The container networking ecosystem is undergoing rapid changes. The
|
||||
networking tools and techniques used in today's container deployments are
|
||||
different than twelve months ago and will continue to evolve. For example,
|
||||
Flannel [6]_, Kubernetes preferred networking implementation, was initially
|
||||
released in July of 2014 and was not considered preferred until early 2015.
|
||||
|
||||
Furthermore, the various container orchestration engines have not
|
||||
standardized on a container networking implementation and may never. For
|
||||
example, Flannel is the preferred container networking implementation for
|
||||
Kubernetes but not for Docker Swarm. Each container networking implementation
|
||||
comes with its own API abstractions, data model, tooling, etc.. Natively
|
||||
supporting each container networking implementation can be a burden on the
|
||||
Magnum community and codebase. By supporting only a subset of container
|
||||
networking implementations, the project may not be widely adopted or may
|
||||
provide a suboptimal user experience.
|
||||
|
||||
Lastly, Magnum has limited support for advanced container networking
|
||||
functionality. Magnum instantiates container networks behind the scenes
|
||||
through Heat templates, exposing little-to-no user configurability. Some
|
||||
users require the ability to customize their container environments,
|
||||
including networking details. However, networking needs to "just work" for
|
||||
users that require no networking customizations.
|
||||
|
||||
Roles
|
||||
-----
|
||||
|
||||
The following are roles that the Magnum Container Networking Model takes
|
||||
into consideration. Roles are an important reference point when creating
|
||||
user stories. This is because each role provides different functions and
|
||||
has different requirements.
|
||||
|
||||
1. Cloud Provider (CP): Provides standard OpenStack cloud infrastructure
|
||||
services, including the Magnum service.
|
||||
|
||||
2. Container Service Provider (CSP): Uses Magnum to deliver
|
||||
Containers-as-a-Service (CaaS) to users. CSPs are a consumer of CP
|
||||
services and a CaaS provider to users.
|
||||
|
||||
3. Users: Consume Magnum services to provision and manage clustered
|
||||
container environments and deploy apps within the container clusters.
|
||||
|
||||
The container ecosystem focuses on the developer user type. It is imperative
|
||||
that the Magnum Container Networking Model meets the need of this user type.
|
||||
|
||||
These roles are not mutually exclusive. For example:
|
||||
|
||||
1. A CP can also be a CSP. In this case, the CP/CSP provisions and manages
|
||||
standard OpenStack services, the Magnum service, and provides CaaS
|
||||
services to users.
|
||||
|
||||
2. A User can also be a CSP. In this case, the user provisions their own
|
||||
baymodels, bays, etc. from the CP.
|
||||
|
||||
Definitions
|
||||
-----------
|
||||
|
||||
COE
|
||||
Container Orchestration Engine
|
||||
|
||||
Baymodel
|
||||
An object that stores template information about the bay which is
|
||||
used to create new bays consistently.
|
||||
|
||||
Bay
|
||||
A Magnum resource that includes at least one host to run containers on,
|
||||
and a COE to manage containers created on hosts within the bay.
|
||||
|
||||
Pod
|
||||
Is the smallest deployable unit that can be created, scheduled, and
|
||||
managed within Kubernetes.
|
||||
|
||||
Additional Magnum definitions can be found in the Magnum Developer
|
||||
documentation [2]_.
|
||||
|
||||
Use Cases
|
||||
----------
|
||||
|
||||
This document does not intend to address each use case. The use cases are
|
||||
provided as reference for the long-term development of the Magnum Container
|
||||
Networking Model.
|
||||
|
||||
As a User:
|
||||
|
||||
1. I need to easily deploy containerized apps in an OpenStack cloud.
|
||||
My user experience should be similar to how I deploy containerized apps
|
||||
outside of an OpenStack cloud.
|
||||
|
||||
2. I need to have containers communicate with vm-based apps that use
|
||||
OpenStack networking.
|
||||
|
||||
3. I need the option to preserve the container's IP address so I can
|
||||
manage containers by IP's, not just ports.
|
||||
|
||||
4. I need to block unwanted traffic to/from my containerized apps.
|
||||
|
||||
5. I need the ability for my containerized apps to be highly available.
|
||||
|
||||
6. I need confidence that my traffic is secure from other tenants traffic.
|
||||
|
||||
As a CSP:
|
||||
|
||||
1. I need to easily deploy a bay for consumption by users. The bay must
|
||||
support the following:
|
||||
|
||||
A. One or more hosts to run containers.
|
||||
B. The ability to choose between virtual or physical hosts to run
|
||||
containers.
|
||||
C. The ability to automatically provision networking to containers.
|
||||
|
||||
2. I need to provide clustering options that support different
|
||||
container/image, formats and technologies.
|
||||
|
||||
3. After deploying my initial cluster, I need the ability to provide ongoing
|
||||
management, including:
|
||||
|
||||
A. The ability to add/change/remove networks that containers connect to.
|
||||
B. The ability to add/change/remove nodes within the cluster.
|
||||
|
||||
4. I need to deploy a Bay without admin rights to OpenStack services.
|
||||
|
||||
5. I need the freedom to choose different container networking tools and
|
||||
techniques offered by the container ecosystem beyond OpenStack.
|
||||
|
||||
As a CP:
|
||||
|
||||
1. I need to easily and reliably add the Magnum service to my existing
|
||||
OpenStack cloud environment.
|
||||
|
||||
2. I need to easily manage (monitor, troubleshoot, etc..) the Magnum
|
||||
service. Including the ability to mirror ports to capture traffic
|
||||
for analysis.
|
||||
|
||||
3. I need to make the Magnum services highly-available.
|
||||
|
||||
4. I need to make Magnum services highly performant.
|
||||
|
||||
5. I need to easily scale-out Magnum services as needed.
|
||||
|
||||
6. I need Magnum to be robust regardless of failures within the container
|
||||
orchestration engine.
|
||||
|
||||
Proposed Changes
|
||||
================
|
||||
|
||||
1. Currently, Magnum supports Flannel [6]_ as the only multi-host container
|
||||
networking implementation. Although Flannel has become widely accepted
|
||||
for providing networking capabilities to Kubernetes-based container
|
||||
clusters, other networking tools exist and future tools may develop.
|
||||
|
||||
This document proposes extending Magnum to support specifying a
|
||||
container networking implementation through a combination of user-facing
|
||||
baymodel configuration flags. Configuration parameters that are common
|
||||
across Magnum or all networking implementations will be exposed as unique
|
||||
flags. For example, a flag named network-driver can be used to instruct
|
||||
Magnum which network driver to use for implementing a baymodel
|
||||
container/pod network. network driver examples may include:
|
||||
|
||||
flannel, weave, calico, midonet, netplugin, etc..
|
||||
|
||||
Here is an example of creating a baymodel that uses Flannel as the
|
||||
network driver: ::
|
||||
|
||||
magnum baymodel-create --name k8sbaymodel \
|
||||
--image-id fedora-21-atomic-5 \
|
||||
--keypair-id testkey \
|
||||
--external-network-id 1hsdhs88sddds889 \
|
||||
--dns-nameserver 8.8.8.8 \
|
||||
--flavor-id m1.small \
|
||||
--docker-volume-size 5 \
|
||||
--coe kubernetes \
|
||||
--network-driver flannel
|
||||
|
||||
If no network-driver parameter is supplied by the user, the baymodel is
|
||||
created using the default network driver of the specified Magnum COE.
|
||||
Each COE must support a default network driver and each driver must
|
||||
provide reasonable default configurations that allow users to instantiate
|
||||
a COE without supplying labels. The default network driver for each COE
|
||||
should be consistent with existing Magnum default settings. Where current
|
||||
defaults do not exist, the defaults should be consistent with upstream
|
||||
network driver projects.
|
||||
|
||||
2. Each network driver supports a range of configuration parameters that
|
||||
should be observed by Magnum. This document suggests using an attribute
|
||||
named "labels" for supplying driver-specific configuration parameters.
|
||||
Labels consist of one or more arbitrary key/value pairs. Here is an
|
||||
example of using labels to change default settings of the Flannel
|
||||
network driver: ::
|
||||
|
||||
magnum baymodel-create --name k8sbaymodel \
|
||||
--image-id fedora-21-atomic-5 \
|
||||
--keypair-id testkey \
|
||||
--external-network-id ${NIC_ID} \
|
||||
--dns-nameserver 8.8.8.8 \
|
||||
--flavor-id m1.small \
|
||||
--docker-volume-size 5 \
|
||||
--coe kubernetes \
|
||||
--network-driver flannel \
|
||||
--labels flannel_network_cidr=10.0.0.0/8,\
|
||||
flannel_network_subnetlen=22,\
|
||||
flannel_backend=vxlan
|
||||
|
||||
With Magnum's current implementation, this document would support
|
||||
labels for the Kubernetes COE type. However, labels are applicable
|
||||
beyond Kubernetes, as the Docker daemon, images and containers now
|
||||
support labels as a mechanism for providing custom metadata. The labels
|
||||
attribute within Magnum should be extended beyond Kubernetes pods, so a
|
||||
single mechanism can be used to pass arbitrary metadata throughout the
|
||||
entire system. A blueprint [2]_ has been registered to expand the scope
|
||||
of labels for Magnum. This document intends on adhering to the
|
||||
expand-labels-scope blueprint.
|
||||
|
||||
Note: Support for daemon-labels was added in Docker 1.4.1. Labels for
|
||||
containers and images were introduced in Docker 1.6.0
|
||||
|
||||
If the --network-driver flag is specified without any labels, default
|
||||
configuration values of the driver will be used by the baymodel. These
|
||||
defaults are set within the Heat template of the associated COE. Magnum
|
||||
should ignore label keys and/or values not understood by any of the
|
||||
templates during the baymodel operation.
|
||||
|
||||
Magnum will continue to CRUD bays in the same way:
|
||||
|
||||
magnum bay-create --name k8sbay --baymodel k8sbaymodel --node-count 1
|
||||
|
||||
3. Update python-magnumclient to understand the new Container Networking
|
||||
Model attributes. The client should also be updated to support passing
|
||||
the --labels flag according to the expand-labels-scope blueprint [2]_.
|
||||
|
||||
4. Update the conductor template definitions to support the new Container
|
||||
Networking Model attributes.
|
||||
|
||||
5. Refactor Heat templates to support the Magnum Container Networking Model.
|
||||
Currently, Heat templates embed Flannel-specific configuration within
|
||||
top-level templates. For example, the top-level Kubernetes Heat
|
||||
template [8]_ contains the flannel_network_subnetlen parameter. Network
|
||||
driver specific configurations should be removed from all top-level
|
||||
templates and instead be implemented in one or more template fragments.
|
||||
As it relates to container networking, top-level templates should only
|
||||
expose the labels and generalized parameters such as network-driver.
|
||||
Heat templates, template definitions and definition entry points should
|
||||
be suited for composition, allowing for a range of supported labels. This
|
||||
document intends to follow the refactor-heat-templates blueprint [3]_ to
|
||||
achieve this goal.
|
||||
|
||||
6. Update unit and functional tests to support the new attributes of the
|
||||
Magnum Container Networking Model.
|
||||
|
||||
7. The spec will not add support for natively managing container networks.
|
||||
Due to each network driver supporting different API operations, this
|
||||
document suggests that Magnum not natively manage container networks at
|
||||
this time and instead leave this job to native tools. References [4]_ [5]_
|
||||
[6]_ [7]_.
|
||||
provide additional details to common labels operations.
|
||||
|
||||
8. Since implementing the expand-labels-scope blueprint [2]_ may take a while,
|
||||
exposing network functionality through baymodel configuration parameters
|
||||
should be considered as an interim solution.
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
|
||||
1. Observe all networking configuration parameters, including labels
|
||||
within a configuration file instead of exposing the labels attribute to
|
||||
the user.
|
||||
|
||||
2. Only support a single networking implementation such as Flannel. Flannel
|
||||
is currently supported for the Kubernetes COE type. It can be ported to
|
||||
support the swarm COE type.
|
||||
|
||||
3. Add support for managing container networks. This will require adding
|
||||
abstractions for each supported network driver or creating an
|
||||
abstraction layer that covers all possible network drivers.
|
||||
|
||||
4. Use the Kuryr project [10]_ to provide networking to Magnum containers.
|
||||
Kuryr currently contains no documentation or code, so this alternative
|
||||
is highly unlikely if the Magnum community requires a pluggable
|
||||
container networking implementation in the near future. However, Kuryr
|
||||
could become the long-term solution for container networking within
|
||||
OpenStack. A decision should be made by the Magnum community whether
|
||||
to move forward with Magnum's own container networking model or to wait
|
||||
for Kuryr to mature. In the meantime, this document suggests the Magnum
|
||||
community become involved in the Kuryr project.
|
||||
|
||||
Data Model Impact
|
||||
-----------------
|
||||
|
||||
This document adds the labels and network-driver attribute to the baymodel
|
||||
database table. A migration script will be provided to support the attribute
|
||||
being added. ::
|
||||
|
||||
+-------------------+-----------------+---------------------------------------------+
|
||||
| Attribute | Type | Description |
|
||||
+===================+=================+=============================================+
|
||||
| labels | JSONEncodedDict | One or more arbitrary key/value pairs |
|
||||
+-------------------+-----------------+---------------------------------------------+
|
||||
| network-driver | string | Container networking backend implementation |
|
||||
+-------------------+-----------------+---------------------------------------------+
|
||||
|
||||
REST API Impact
|
||||
---------------
|
||||
|
||||
This document adds the labels and network-driver attribute to the BayModel
|
||||
API class. ::
|
||||
|
||||
+-------------------+-----------------+---------------------------------------------+
|
||||
| Attribute | Type | Description |
|
||||
+===================+=================+=============================================+
|
||||
| labels | JSONEncodedDict | One or more arbitrary key/value pairs |
|
||||
+-------------------+-----------------+---------------------------------------------+
|
||||
| network-driver | string | Container networking backend implementation |
|
||||
+-------------------+-----------------+---------------------------------------------+
|
||||
|
||||
Security Impact
|
||||
---------------
|
||||
|
||||
Supporting more than one network driver increases the attack
|
||||
footprint of Magnum.
|
||||
|
||||
Notifications Impact
|
||||
--------------------
|
||||
|
||||
None
|
||||
|
||||
Other End User Impact
|
||||
---------------------
|
||||
|
||||
Most end users will never use the labels configuration flag
|
||||
and simply use the default network driver and associated
|
||||
configuration options. For those that wish to customize their
|
||||
container networking environment, it will be important to understand
|
||||
what network-driver and labels are supported, along with their
|
||||
associated configuration options, capabilities, etc..
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
Performance will depend upon the chosen network driver and its
|
||||
associated configuration. For example, when creating a baymodel with
|
||||
"--network-driver flannel" flag, Flannel's default configuration
|
||||
will be used. If the default for Flannel is an overlay networking technique
|
||||
(i.e. VXLAN), then networking performance will be less than if Flannel used
|
||||
the host-gw configuration that does not perform additional packet
|
||||
encapsulation to/from containers. If additional performance is required
|
||||
when using this driver, Flannel's host-gw configuration option could be
|
||||
exposed by the associated Heat template and instantiated through the labels
|
||||
attribute.
|
||||
|
||||
Other Deployer Impact
|
||||
---------------------
|
||||
|
||||
Currently, container networking and OpenStack networking are different
|
||||
entities. Since no integration exists between the two, deployers/operators
|
||||
will be required to manage each networking environment individually.
|
||||
However, Magnum users will continue to deploy baymodels, bays, containers,
|
||||
etc. without having to specify any networking parameters. This will be
|
||||
accomplished by setting reasonable default parameters within the Heat
|
||||
templates.
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
None
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
Daneyon Hansen (danehans)
|
||||
|
||||
Other contributors:
|
||||
Ton Ngo (Tango)
|
||||
Hongbin Lu (hongbin)
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
1. Extend the Magnum API to support new baymodel attributes.
|
||||
2. Extend the Client API to support new baymodel attributes.
|
||||
3. Extend baymodel objects to support new baymodel attributes. Provide a
|
||||
database migration script for adding attributes.
|
||||
4. Refactor Heat templates to support the Magnum Container Networking Model.
|
||||
5. Update Conductor template definitions and definition entry points to
|
||||
support Heat template refactoring.
|
||||
6. Extend unit and functional tests to support new baymodel attributes.
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
Although adding support for these new attributes does not depend on the
|
||||
following blueprints, it's highly recommended that the Magnum Container
|
||||
Networking Model be developed in concert with the blueprints to maintain
|
||||
development continuity within the project.
|
||||
|
||||
1. Common Plugin Framework Blueprint [1]_.
|
||||
|
||||
2. Expand the Scope of Labels Blueprint [9]_.
|
||||
|
||||
3. Refactor Heat Templates, Definitions and Entry Points Blueprint [3]_.
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
Each commit will be accompanied with unit tests. There will also be
|
||||
functional tests which will be used as part of a cross-functional gate
|
||||
test for Magnum.
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
The Magnum Developer Quickstart document will be updated to support the
|
||||
configuration flags introduced by this document. Additionally, background
|
||||
information on how to use these flags will be included.
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
.. [1] https://blueprints.launchpad.net/magnum/+spec/common-plugin-framework
|
||||
.. [2] http://docs.openstack.org/developer/magnum/
|
||||
.. [3] https://blueprints.launchpad.net/magnum/+spec/refactor-heat-templates
|
||||
.. [4] https://github.com/docker/libnetwork/blob/master/docs/design.md
|
||||
.. [5] https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/design/networking.md
|
||||
.. [6] https://github.com/coreos/flannel
|
||||
.. [7] https://github.com/coreos/rkt/blob/master/Documentation/networking.md
|
||||
.. [8] https://github.com/openstack/magnum/blob/master/magnum/templates/kubernetes/kubecluster.yaml
|
||||
.. [9] https://blueprints.launchpad.net/magnum/+spec/expand-labels-scope
|
||||
.. [10] https://github.com/openstack/kuryr
|
|
@ -0,0 +1,500 @@
|
|||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
=========================================
|
||||
Magnum Container Volume Integration Model
|
||||
=========================================
|
||||
|
||||
Launchpad Blueprint:
|
||||
|
||||
https://blueprints.launchpad.net/magnum/+spec/magnum-integrate-with-cinder
|
||||
|
||||
Storage is a key part of any computing system. Containers in particular have
|
||||
the interesting characteristic that local storage by default is ephemeral:
|
||||
any changes to the file system disappear when the container is deleted. This
|
||||
introduces the need for persistent storage to retain and share data between
|
||||
containers, and this is currently an active area of development in all
|
||||
container orchestration engines (COE).
|
||||
|
||||
As the component in OpenStack for managing COE's, Magnum must fully enable the
|
||||
features for persistent storage in the COE's. To achieve this goal, we propose
|
||||
in this specification to generalize the process for utilizing persistent
|
||||
storage with containers so that it is applicable for different bay types.
|
||||
Despite the complexity, we aim to maintain a good user experience by a simple
|
||||
abstraction for working with various volume capabilities. For the rest of this
|
||||
specification, we will use the term Volume to refer to persistent storage, and
|
||||
Volume Driver as the plugin in a COE to support the particular persistent
|
||||
storage.
|
||||
|
||||
Problem Description
|
||||
===================
|
||||
|
||||
Containers requires full life cycle management such as create, run, stop,
|
||||
delete,... and a key operation is to manage the data - making the data
|
||||
persistent, reusing the data, sharing data between containers, etc.
|
||||
In this area, the support for container volume is undergoing rapid change
|
||||
to bring more integration with open source software and third party
|
||||
storage solutions.
|
||||
|
||||
A clear evidence of this growth is the many plugin volume drivers [1]_ [4]_
|
||||
such as NFS, GlusterFS, EBS, etc. They provide different functionality, use
|
||||
different storage backend and have different requirements. The COE's are
|
||||
naturally motivated to be flexible and allow as many choices as possible for
|
||||
the users with respect to the storage backend. Since Magnum's role is to
|
||||
support the COE's within OpenStack, the goal is to be transparent and enable
|
||||
these same storage backends for the COE's through the COE's lifecycle
|
||||
operation.
|
||||
|
||||
Currently, Magnum provides limited support for managing container volume
|
||||
. The only option available is to specify the docker-volume-size for a
|
||||
pre-allocated block storage in the COE to host the containers. Magnum
|
||||
instantiates container volumes through Heat templates, exposing no other
|
||||
mechanism to configure and operate on volumes. In practice, some users
|
||||
require the ability to manage volumes easily in the COEs .
|
||||
|
||||
Note that we are not proposing to create a new volume management interface
|
||||
in Magnum. After the users create the baymodel and bays, we assume that the
|
||||
users would manage the volumes through existing techniques:
|
||||
|
||||
1. Log in to the COE, use COE specific CLI or GUI to manage volumes.
|
||||
|
||||
2. Use native tools to manage volumes.
|
||||
|
||||
The initial implementation will focus on OpenStack Cinder integration; as
|
||||
other alternatives become available, contributors are welcome through
|
||||
3rd-party maintained projects.
|
||||
|
||||
|
||||
Definitions
|
||||
-----------
|
||||
|
||||
COE
|
||||
Container Orchestration Engine
|
||||
|
||||
Baymodel
|
||||
An object that stores template information about the bay which is
|
||||
used to create new bays consistently.
|
||||
|
||||
Bay
|
||||
A Magnum resource that includes at least one host to run containers on,
|
||||
and a COE to manage containers created on hosts within the bay.
|
||||
|
||||
Pod
|
||||
Is the smallest deployable unit that can be created, scheduled, and
|
||||
managed within Kubernetes.
|
||||
|
||||
Volume
|
||||
storage that is persistent
|
||||
|
||||
Volume plugin
|
||||
COE specific code that supports the functionality of a type of volume.
|
||||
|
||||
Additional Magnum definitions can be found in the Magnum Developer
|
||||
documentation[7]_ .
|
||||
|
||||
Use Cases
|
||||
----------
|
||||
|
||||
This document does not intend to address all use cases. We list below a number
|
||||
of use cases for 3 different roles; they should be useful as reference for the
|
||||
long-term development of the Magnum Container Volume Integration.
|
||||
|
||||
As a User:
|
||||
|
||||
As mentioned above, our goal is to preserve the user experience specific to
|
||||
the COE in managing the volumes. Therefore, we expect the use cases for the
|
||||
users will be fulfilled by the COE's themselves; Magnum will simply ensure
|
||||
that the necessary supports are in place.
|
||||
|
||||
1. I need to easily create volume for containers to use as persistent
|
||||
data store.
|
||||
|
||||
2. I need the ability to create and mount a data volume container for cross
|
||||
container sharing.
|
||||
|
||||
3. I need to mount a host directory as a data volume.
|
||||
|
||||
4. I need to easily attach a known volume to container to use the
|
||||
existing data.
|
||||
|
||||
5. I need the ability to delete the volume.
|
||||
|
||||
6. I need to list and view the details of the volume
|
||||
|
||||
7. I need to modify the volume.
|
||||
|
||||
|
||||
As a CSP:
|
||||
|
||||
1. I need to easily deploy a bay for consumption by users. The bay must
|
||||
support the following:
|
||||
|
||||
A. One or more hosts to run containers.
|
||||
B. The ability to choose between virtual or physical hosts to
|
||||
run containers.
|
||||
C. The ability to automatically enable volume plugins to containers.
|
||||
|
||||
2. I need to provide clustering options that support different volume plugins
|
||||
per COE.
|
||||
|
||||
3. After deploying my initial cluster, I need the ability to provide lifecycle
|
||||
management, including:
|
||||
|
||||
A. The ability to add/remove volumes that containers used.
|
||||
B. The ability to add/remove nodes within the cluster with the necessary
|
||||
adjustment to the volumes
|
||||
|
||||
As a CP:
|
||||
|
||||
1. I need to easily and reliably add the Magnum service to my existing
|
||||
OpenStack cloud environment.
|
||||
|
||||
2. I need to make the Magnum services highly-available.
|
||||
|
||||
3. I need to make Magnum services highly performant.
|
||||
|
||||
4. I need to easily scale-out Magnum services as needed.
|
||||
|
||||
|
||||
Proposed Changes
|
||||
================
|
||||
|
||||
We propose extending Magnum as follows.
|
||||
|
||||
|
||||
|
||||
1. The new attribute volume-driver for a baymodel specifies the volume backend
|
||||
driver to use when deploying a bay.
|
||||
|
||||
Volume drivers may include:
|
||||
|
||||
rexray, flocker, nfs, glusterfs, etc..
|
||||
|
||||
Here is an example of creating a Docker Swarm baymodel that uses rexray [5]_
|
||||
[6]_ as the volume driver: ::
|
||||
|
||||
|
||||
magnum baymodel-create --name swarmbaymodel \
|
||||
--image-id fedora-21-atomic-5 \
|
||||
--keypair-id testkey \
|
||||
--external-network-id 1hsdhs88sddds889 \
|
||||
--dns-nameserver 8.8.8.8 \
|
||||
--flavor-id m1.small \
|
||||
--docker-volume-size 5 \
|
||||
--coe swarm\
|
||||
--network-driver flannel \
|
||||
--volume-driver rexray
|
||||
|
||||
When a Swarm bay is created with this bay model, the REX-Ray storage
|
||||
subsystem will be installed, configured and started on the Swarm nodes,
|
||||
then the REX-Ray volume plugin will be registered in Docker. When a container
|
||||
is created with rexray as the volume driver, the container will have full
|
||||
access to the REX-Ray capabilities such as creating, mounting, deleting
|
||||
volumes [6]_. REX-Ray in turn will interface with Cinder to manage the
|
||||
volumes in OpenStack.
|
||||
|
||||
Here is an example of creating a Kubernetes baymodel that uses Cinder [2]_
|
||||
[3]_ as the volume driver: ::
|
||||
|
||||
magnum baymodel-create --name k8sbaymodel \
|
||||
--image-id fedora-21-atomic-5 \
|
||||
--keypair-id testkey \
|
||||
--external-network-id 1hsdhs88sddds889 \
|
||||
--dns-nameserver 8.8.8.8 \
|
||||
--flavor-id m1.small \
|
||||
--docker-volume-size 5 \
|
||||
--coe kubernetes\
|
||||
--network-driver flannel \
|
||||
--volume-driver cinder
|
||||
|
||||
When the Kubernetes bay is created using this bay model, the kubelet will be
|
||||
configured so that an existing Cinder volume can be mounted in a pod by
|
||||
specifying the volume ID in the pod manifest as follows: ::
|
||||
|
||||
volumes:
|
||||
- name: mysql-persistent-storage
|
||||
cinder:
|
||||
volumeID: bd82f7e2-wece-4c01-a505-4acf60b07f4a
|
||||
fsType: ext4
|
||||
|
||||
|
||||
|
||||
Here is an example of creating a mesos baymodel that uses rexray as the
|
||||
volume driver: ::
|
||||
|
||||
magnum baymodel-create --name mesosbaymodel \
|
||||
--image-id ubuntu-mesos\
|
||||
--keypair-id testkey \
|
||||
--external-network-id 1hsdhs88sddds889 \
|
||||
--dns-nameserver 8.8.8.8 \
|
||||
--flavor-id m1.small \
|
||||
--coe mesos\
|
||||
--network-driver docker \
|
||||
--volume-driver rexray
|
||||
|
||||
When the mesos bay is created using this bay model, the mesos bay will be
|
||||
configured so that an existing Cinder volume can be mounted in a container
|
||||
by configuring the parameters to mount the cinder volume in the json file. ::
|
||||
|
||||
"parameters": [
|
||||
{ "key": "volume-driver", "value": "rexray" },
|
||||
{ "key": "volume", "value": "redisdata:/data" }
|
||||
]
|
||||
|
||||
If no volume-driver parameter is supplied by the user, the baymodel is
|
||||
created using the default volume driver of the particular COE.
|
||||
Magnum will provide a default volume driver for each COE as well as the
|
||||
reasonable default configuration for each driver so that
|
||||
users can instantiate a COE without supplying a volume driver and
|
||||
associated labels. Generally the defaults should be consistent with upstream
|
||||
volume driver projects.
|
||||
|
||||
2. Each volume driver supports a range of configuration parameters that are
|
||||
handled by the "labels" attribute.
|
||||
|
||||
Labels consist of one or more arbitrary key/value pairs.
|
||||
Here is an example of using labels to choose ¡°storage-provider¡± for
|
||||
rexray driver.
|
||||
Volume driver: ::
|
||||
|
||||
magnum baymodel-create --name k8sbaymodel \
|
||||
--image-id fedora-21-atomic-5 \
|
||||
--keypair-id testkey \
|
||||
--external-network-id ${NIC_ID} \
|
||||
--dns-nameserver 8.8.8.8 \
|
||||
--flavor-id m1.small \
|
||||
--docker-volume-size 5 \
|
||||
--coe kubernetes \
|
||||
--volume-driver rexray \
|
||||
--labels storage-provider=openstack \
|
||||
[, key2=value2...]
|
||||
|
||||
|
||||
If the --volume-driver flag is specified without any labels, default
|
||||
configuration values of the driver will be used by the baymodel.
|
||||
|
||||
Magnum will validate the labels together with the driver specified before
|
||||
creating the bay and will return an error if the validation fails.
|
||||
|
||||
Magnum will continue to CRUD bays in the same way:
|
||||
|
||||
magnum bay-create --name k8sbay --baymodel k8sbaymodel --node-count 1
|
||||
|
||||
3. Update python-magnumclient to handle the new container volume-
|
||||
driver attributes.
|
||||
|
||||
4. Update the conductor template definitions to support the new container
|
||||
volume-driver model attributes.
|
||||
|
||||
5. Refactor Heat templates to support the Magnum volume driver plugin.
|
||||
Configurations specific to volume drivers should be
|
||||
implemented in one or more template fragments.
|
||||
Top-level templates should only
|
||||
expose the labels and generalized parameters such as volume-driver.
|
||||
Heat templates, template definitions and definition entry points should
|
||||
be designed for composition, allowing for a range of supported labels.
|
||||
|
||||
6. Update unit and functional tests to support the new attributes of the
|
||||
Magnum container volume driver.
|
||||
|
||||
7. Preserve the user experience by ensuring that any operation on volume will
|
||||
be identical between a COE deployed by Magnum and a COE deployed by other
|
||||
methods.
|
||||
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
1. Without the support proposed, the user will need to manually enable and
|
||||
configure the volume plugin. This will require the user to log into the
|
||||
nodes in the cluster and understand the low level infrastructure of the
|
||||
cluster as deployed by the heat templates.
|
||||
2. We can add full support for managing container volume in Magnum user
|
||||
interface itself. This will require adding abstractions for each supported
|
||||
COE volume plugins driver or creating an abstraction layer that covers all
|
||||
possible COE volume drivers.
|
||||
|
||||
Data Model Impact
|
||||
-----------------
|
||||
|
||||
This document adds the volume-driver attribute to the baymodel
|
||||
database table. A migration script will be provided to support the attribute
|
||||
being added. ::
|
||||
|
||||
+-------------------+-----------------+---------------------------------------------+
|
||||
| Attribute | Type | Description |
|
||||
+===================+=================+=============================================+
|
||||
+-------------------+-----------------+---------------------------------------------+
|
||||
| volume-driver | string | Container volume backend implementation |
|
||||
+-------------------+-----------------+---------------------------------------------+
|
||||
|
||||
REST API Impact
|
||||
---------------
|
||||
|
||||
This document adds volume-driver attribute to the BayModel
|
||||
API class. ::
|
||||
|
||||
+-------------------+-----------------+---------------------------------------------+
|
||||
| Attribute | Type | Description |
|
||||
+===================+=================+=============================================+
|
||||
+-------------------+-----------------+---------------------------------------------+
|
||||
| volume-driver | string | Container volume backend implementation |
|
||||
+-------------------+-----------------+---------------------------------------------+
|
||||
|
||||
Security Impact
|
||||
---------------
|
||||
|
||||
Supporting volume drivers can potentially increase the attack surface
|
||||
on containers.
|
||||
|
||||
Notifications Impact
|
||||
--------------------
|
||||
|
||||
None
|
||||
|
||||
Other End User Impact
|
||||
---------------------
|
||||
|
||||
There is no impact if the user does not use a volume driver.
|
||||
We anticipate that most users would not use the labels for volume
|
||||
and would simply use the default volume driver and associated
|
||||
configuration options. For those who wish to customize their
|
||||
container volume driver environment, it will be important to understand
|
||||
what volume-driver and labels are supported, along with their
|
||||
associated configuration options, capabilities, etc..
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
There is no impact if the user does not use a volume driver.
|
||||
When a volume driver is used, the performance will depend upon the specific
|
||||
volume driver and its associated storage backends. For example, Kubernetes
|
||||
supports Cinder and awsEBS; the two types of volumes can have different
|
||||
performance.
|
||||
|
||||
An example of the second case is a docker swarm bay with
|
||||
"--volume-driver rexray" where the rexray driver's storage provider is
|
||||
OpenStack cinder. The resulting performance for container may vary depending
|
||||
on the storage backends. As listed in [8]_ , Cinder supports many storage
|
||||
drivers. Besides this, different container volume driver can also cause
|
||||
performance variance.
|
||||
|
||||
|
||||
High-Availability Impact
|
||||
------------------------------
|
||||
|
||||
|
||||
|
||||
+-----------------+--------------------+--------------------------+
|
||||
| COE | Master HA | Pod/Container/App HA |
|
||||
+=================+====================+==========================+
|
||||
| Kubernetes | No | Yes |
|
||||
+-----------------+--------------------+--------------------------+
|
||||
| Docker Swarm | No | Yes |
|
||||
+-----------------+--------------------+--------------------------+
|
||||
| Mesos | No | No |
|
||||
+-----------------+--------------------+--------------------------+
|
||||
|
||||
"No" means that the volume doesn't affect the high-availability.
|
||||
"Yes" means that the volume affect the high-availability.
|
||||
|
||||
Kubernetes does support pod high-availability through the replication
|
||||
controller, however, this doesn't work when a pod with volume attached
|
||||
fails. Refer the link [11]_ for details.
|
||||
|
||||
Docker swarm doesn't support the containers rescheduling when a node fails, so
|
||||
volume can not be automatically detached by volume driver. Refer the
|
||||
link [12]_ for details.
|
||||
|
||||
Mesos supports the application high-availability when a node fails, which
|
||||
means application would be started on new node, and volumes can be
|
||||
automatically attached to the new node by the volume driver.
|
||||
|
||||
Other Deployer Impact
|
||||
---------------------
|
||||
|
||||
Currently, both Kubernetes and Docker community have supported some volume
|
||||
plugins. The changes proposed will enable these volume plugins in Magnum.
|
||||
However, Magnum users will be able to continue to deploy baymodels, bays,
|
||||
containers, etc. without having to specify any parameters for volume.
|
||||
This will be accomplished by setting reasonable default parameters within
|
||||
the Heat templates.
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
None
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
|
||||
- Kai Qiang Wu (Kennan)
|
||||
|
||||
Other contributors:
|
||||
|
||||
- Qun Wang (wangqun)
|
||||
- Ton Ngo (Tango)
|
||||
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
1. Extend the Magnum API to support new baymodel attributes.
|
||||
2. Extend the Client API to support new baymodel attributes.
|
||||
3. Extend baymodel objects to support new baymodel attributes. Provide a
|
||||
database migration script for adding attributes.
|
||||
4. Refactor Heat templates to support the Magnum container volume driver.
|
||||
5. Update Conductor template definitions and definition entry points to
|
||||
support Heat template refactoring.
|
||||
6. Extend unit and functional tests to support new baymodel attributes.
|
||||
7. Document how to use the volume drivers with examples.
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
Although adding support for these new attributes does not depend on the
|
||||
following blueprints, it's highly recommended that the Magnum Container
|
||||
Networking Model be developed in concert with the blueprints to maintain
|
||||
development continuity within the project.
|
||||
https://blueprints.launchpad.net/magnum/+spec/ubuntu-image-build
|
||||
|
||||
Kubernetes with cinder support need Kubernetes version >= 1.1.1
|
||||
Swarm need version >= 1.8.3, as Kubernetes 1.1.1 upgraded to that version
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
Each commit will be accompanied with unit tests. There will also be
|
||||
functional tests which will be used as part of a cross-functional gate
|
||||
test for Magnum.
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
The Magnum Developer Quickstart document will be updated to support the
|
||||
configuration flags introduced by this document. Additionally, background
|
||||
information on how to use these flags will be included.
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
.. [1] http://kubernetes.io/v1.1/docs/user-guide/volumes.html
|
||||
.. [2] http://kubernetes.io/v1.1/examples/mysql-cinder-pd/
|
||||
.. [3] https://github.com/kubernetes/kubernetes/tree/master/pkg/volume/cinder
|
||||
.. [4] http://docs.docker.com/engine/extend/plugins/
|
||||
.. [5] https://github.com/emccode/rexray
|
||||
.. [6] http://rexray.readthedocs.org/en/stable/user-guide/storage-providers/openstack
|
||||
.. [7] http://docs.openstack.org/developer/magnum/
|
||||
.. [8] http://docs.openstack.org/liberty/config-reference/content/section_volume-drivers.html
|
||||
.. [9] http://docs.openstack.org/admin-guide-cloud/blockstorage_multi_backend.html#
|
||||
.. [10] http://docs.openstack.org/user-guide-admin/dashboard_manage_volumes.html
|
||||
.. [11] https://github.com/kubernetes/kubernetes/issues/14642
|
||||
.. [12] https://github.com/docker/swarm/issues/1488
|
|
@ -0,0 +1,400 @@
|
|||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
==================
|
||||
Containers Service
|
||||
==================
|
||||
|
||||
Launchpad blueprint:
|
||||
|
||||
https://blueprints.launchpad.net/nova/+spec/containers-service
|
||||
|
||||
Containers share many features in common with Nova instances. For the common
|
||||
features, virt drivers for Nova can be used to surface basic instance
|
||||
functionality. For features that go beyond what can be naturally fit within
|
||||
a virt driver, we propose a new API service that allows for advanced features
|
||||
to be added without conflating the worlds of instances and containers.
|
||||
|
||||
Some examples of containers specific features are setting of shell environment
|
||||
variables, and accepting a shell command to execute at runtime. Capturing the
|
||||
STDIO of the process(es) within a container, and tracking the return status
|
||||
of processes are all beyond the scope of what was contemplated for Nova. All
|
||||
of these features will be implemented in the Containers Service.
|
||||
|
||||
|
||||
Problem description
|
||||
===================
|
||||
Container technology is rapidly gaining popularity as a way to bundle and
|
||||
deploy applications. Recognizing and adapting to this trend will position
|
||||
OpenStack to be useful not only to clouds that employ bare metal and virtual
|
||||
machine instances, but can remain competitive in offering container services
|
||||
as well.
|
||||
|
||||
Nova's concepts of an instance, and the actions that may be taken on it do not
|
||||
match completely with containers.
|
||||
|
||||
Use cases
|
||||
---------
|
||||
1. App Consolidation. End-user wants to run multiple small applications in
|
||||
separate operating system environments, but wants to optimize for efficiency
|
||||
to control hosting costs. Each application belongs to the same tenant, so
|
||||
security isolation between applications is nice-to-have but not critical.
|
||||
Isolation is desired primarily for simplified management of the execution
|
||||
environment for each application.
|
||||
2. App Portability. End-user wants to create a single container image, and
|
||||
deploy the same image to multiple hosting environments, including OpenStack.
|
||||
Other environments may include local servers, dedicated servers, private
|
||||
clouds, and public clouds. Switching environments requires passing database
|
||||
connection strings by environment variables at the time a container starts
|
||||
to allow the application to use the services available in each environment
|
||||
without changing the container image.
|
||||
3. Docker Compatibility. End-user has a Dockerfile used to build an application
|
||||
and its runtime environment and dependencies in a Docker container image.
|
||||
They want an easy way to run the Docker resulting image on an OpenStack
|
||||
cloud.
|
||||
4. LXC Compatibility. End-user wants an easy way to remotely create multiple
|
||||
LXC containers within a single Nova instance.
|
||||
5. OpenVZ Compatibility. End-user wants an easy way to remotely create multiple
|
||||
OpenVZ containers within a single Nova instance.
|
||||
6. Containers-Centric World View. End-user wants to communicate with a single
|
||||
OpenStack API, and request the addition of containers, without the need to
|
||||
be concerned with keeping track of how many containers are already running
|
||||
on a given Nova instance, and when more need to be created. They want to
|
||||
simply create and remove containers, and allow the appropriate resource
|
||||
scheduling to happen automatically.
|
||||
7. Platform Integration. Cloud operator already has an OpenStack cloud, and
|
||||
wants to add a service/application centric management system on top.
|
||||
Examples of such systems are Cloud Foundry, Kubernetes, Apache Mesos, etc.
|
||||
The selected system is already Docker compatible. Allow this cloud operator
|
||||
easy integration with OpenStack to run applications in containers. The
|
||||
Cloud Operator now harnesses the power of both the management system, and
|
||||
OpenStack, and does not need to manage a second infrastructure for his/her
|
||||
application hosting needs. All details involving the integration of
|
||||
containers with Nova instances is managed by OpenStack.
|
||||
8. Container network. End-user wants to define a custom overlay network for
|
||||
containers, and wants to have admin privilege to manage the network
|
||||
topology. Building a container network can decouple application deployment
|
||||
and management from the underlying network infrastructure, and enable
|
||||
additional usage scenario, such as (i) software-defined networking, and
|
||||
(ii) extending the container network (i.e. connecting various resources from
|
||||
multiple hosting environments). End-users want a single service that could
|
||||
help them build the container network, and dynamically modify the network
|
||||
topology by adding or removing containers to or from the network.
|
||||
9. Permit secure use of native REST APIs. Provide two models of operation with
|
||||
Magnum. The first model allows Magnum to manage the lifecycle of Pods,
|
||||
ReplicationControllers, and Services. The second model allows end-users to
|
||||
manage the lifecycle of Pods, ReplicationControllers, and Services by
|
||||
providing direct secure access to the native ReST APIs in Kubernetes and
|
||||
possibly Docker.
|
||||
|
||||
Long Term Use Cases
|
||||
-------------------
|
||||
These use cases have been identified by the community as important, but
|
||||
unlikely to be tackled in short term (especially prior to incubation). We wish
|
||||
to adapt to these use cases in long term, but this is not a firm project
|
||||
commitment.
|
||||
|
||||
1. Multi-region/multi-cloud support. End-user wants to deploy applications to
|
||||
multiple regions/clouds, and dynamically relocate deployed applications
|
||||
across different regions/clouds. In particular, they want a single service
|
||||
that could help them (i) provision nodes from multiple regions/clouds, thus
|
||||
running containers on top of them, and (ii) dynamically relocate containers
|
||||
(e.g. through container migration) between nodes regardless of the
|
||||
underlying infrastructure.
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
Add a new API service for CRUD and advanced management of containers.
|
||||
If cloud operators only want to offer basic instance features for their
|
||||
containers, they may use nova with an alternate virt-driver, such as
|
||||
libvirt/lxc or nova-docker. For those wanting a full-featured container
|
||||
experience, they may offer the Containers Service API as well, in combination
|
||||
with Nova instances that contain an OpenStack agent that connects to the
|
||||
containers service through a security controlled agent (daemon) that allows
|
||||
the OpenStack control plane to provision and control containers running on
|
||||
Compute Hosts.
|
||||
|
||||
The Containers Service will call the Nova API to create one or more Nova
|
||||
instances inside which containers will be created. The Nova instances may
|
||||
be of any type, depending on the virt driver(s) chosen by the cloud operator.
|
||||
This includes bare-metal, virtual machines, containers, and potentially other
|
||||
instance types.
|
||||
|
||||
This allows the following configurations of containers in OpenStack.
|
||||
|
||||
* Containers in Virtual Machine Instances
|
||||
* Containers in Bare Metal Instances
|
||||
* Containers in Container Instances (nested)
|
||||
|
||||
The concept of nesting containers is currently possible if the parent container
|
||||
runs in privileged mode. Patches to the linux kernel are being developed to
|
||||
allow nesting of non-privileged containers as well, which provides a higher
|
||||
level of security.
|
||||
|
||||
The spirit of this plan aims to duplicate as little as possible between Nova
|
||||
and the Containers Service. Common components like the scheduler are expected
|
||||
to be abstracted into modules, such as Gantt that can be shared by multiple
|
||||
projects. Until Gantt is ready for use by the Containers Service, we will
|
||||
implement only two provisioning schemes for containers:
|
||||
|
||||
1. Create a container on a specified instance by using a nova instance guid.
|
||||
2. Auto-create instances (applies only until the Gantt scheduler is used)
|
||||
2.1. Fill them sequentially until full.
|
||||
2.2. Remove them automatically when they become empty.
|
||||
|
||||
The above orchestration will be implemented using Heat. This requires some
|
||||
kind of hypervisor painting (such as host aggregates) for security reasons.
|
||||
|
||||
The diagram below offers an overview of the system architecture. The OSC box
|
||||
indicates an OpenStack client, which will communicate with the Containers
|
||||
Service through a REST API. The containers service may silently create Nova
|
||||
instances if one with enough capacity to host the requested container is not
|
||||
already known to the Containers service. The containers service will maintain
|
||||
a database "Map" of containers, and what Nova instance each belongs to. Nova
|
||||
creates instances. Instances are created in Nova, and containers belong only
|
||||
to the Containers Service, and run within a Nova instance. If the instance
|
||||
includes the agent software "A", then it may be included in the inventory of
|
||||
the Containers service. Instances that do not contain an agent may not interact
|
||||
with the Containers Service, and can be controlled only by a Nova virt driver.
|
||||
|
||||
::
|
||||
|
||||
+---------+
|
||||
| OSC |
|
||||
+----+----+
|
||||
|
|
||||
+----+----+
|
||||
+-------- Nova -------+ +-+ REST +-- Containers -+
|
||||
| | | +---------+ Service |
|
||||
| | | |
|
||||
| +-------+ +--+ +-----+ |
|
||||
| | Gantt | | | | Map | |
|
||||
| +-------+ | | +-----+ |
|
||||
| | | |
|
||||
+-----------+---------+ +---------------+-----------+
|
||||
| |
|
||||
+-----------+----+ Compute Host ---------|-----------+
|
||||
| +---+---+ |
|
||||
| +----+ Relay +---+ |
|
||||
| | +-------+ | |
|
||||
| | | |
|
||||
| +-- Instance --+ +-- Instance |-+ +-- Instance |-+ |
|
||||
| | | | | | | | | |
|
||||
| | | | +---+ | | +---+ | |
|
||||
| | | | | | | | | | | |
|
||||
| | | | | A | | | | A | | |
|
||||
| | | | | | | | | | | |
|
||||
| | | | +---+ | | +---+ | |
|
||||
| | | | | | | |
|
||||
| | | | +---+ +---+ | | +---+ +---+ | |
|
||||
| | | | | | | | | | | | | | | |
|
||||
| | | | | C | | C | | | | C | | C | | |
|
||||
| | | | | | | | | | | | | | | |
|
||||
| | | | +---+ +---+ | | +---+ +---+ | |
|
||||
| | | | | | | |
|
||||
| +--------------+ +--------------+ +--------------+ |
|
||||
| |
|
||||
+----------------------------------------------------+
|
||||
+---+
|
||||
| |
|
||||
| A | = Agent
|
||||
| |
|
||||
+---+
|
||||
+---+
|
||||
| |
|
||||
| C | = Container
|
||||
| |
|
||||
+---+
|
||||
|
||||
|
||||
Design Principles
|
||||
-----------------
|
||||
1. Leverage existing OpenStack projects for what they are good at. Do not
|
||||
duplicate functionality, or copy code that can be otherwise accessed through
|
||||
API calls.
|
||||
2. Keep modifications to Nova to a minimum.
|
||||
3. Make the user experience for end users simple and familiar.
|
||||
4. Allow for implementation of all features containers are intended to offer.
|
||||
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
1. Extending Nova's existing feature set to offer container features
|
||||
1.1. Container features don't fit into Nova's idea of compute (VM/Server)
|
||||
2. A completely separate containers service forked from Nova.
|
||||
2.1. Would result in large overlap and duplication in features and code
|
||||
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
For Nova, None. All new data planned will be in the Containers Service.
|
||||
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
For Nova, none. All new API calls will be implemented in the Containers
|
||||
Service. The OpenStack Containers Service API will be a superset of
|
||||
functionality offered by the, The `Docker Remote API:
|
||||
<https://docs.docker.com/reference/api/docker_remote_api/>`_
|
||||
with additionals to make is suitable for general use regardless of the backend
|
||||
container technology used, and to be compatible with OpenStack multi-tenancy
|
||||
and Keystone authentication.
|
||||
|
||||
Specific Additions:
|
||||
|
||||
1. Support for the X-Auth-Project-Id HTTP request header to allow for
|
||||
multi-tenant use.
|
||||
2. Support for the X-Auth-Token HTTP request header to allow for authentication
|
||||
with keystone.
|
||||
|
||||
If either of the above headers are missing, a 401 Unauthorized response will
|
||||
be generated.
|
||||
|
||||
Docker CLI clients may communicate with a Swarmd instance that is configured
|
||||
to use the OpenStack Containers API as the backend for libswarm. This will
|
||||
allow for tool compatibility with the Docker ecosystem using the officially
|
||||
supported means for integration of a distributed system.
|
||||
|
||||
The scope of the full API will cause this spec to be too long to review, so
|
||||
the intent is to deal with the specific API design as a series of Gerrit
|
||||
reviews that submit API code as Not Implemented stubs with docstrings that
|
||||
clearly document the design, so allow for approval, and further implementation.
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
Because Nova will not be changed, there should be no security impacts to Nova.
|
||||
The Containers Service implementation, will have the following security related
|
||||
issues:
|
||||
|
||||
* Need to authenticate against keystone using python-keystoneclient.
|
||||
* A trust token from Nova will be needed in order for the Containers Service
|
||||
to call the Nova API on behalf of a user.
|
||||
* Limits must be implemented to control resource consumption in accordance with
|
||||
quotas.
|
||||
* Providing STDIO access may generate a considerable amount of network chatter
|
||||
between containers and clients through the relay. This could lead to
|
||||
bandwidth congestion at the relays, or API nodes. An approach similar to
|
||||
how we handle serial console access today will need to be considered to
|
||||
mitigate this concern.
|
||||
|
||||
Using containers implies a range of security considerations for cloud
|
||||
operators. These include:
|
||||
|
||||
* Containers in the same instance share an operating system. If the kernel is
|
||||
exploited using a security vulnerability, processes in once container may
|
||||
escape the constraints of the container and potentially access other
|
||||
resources on the host, including contents of other containers.
|
||||
* Output of processes may be persisted by the containers service in order to
|
||||
allow asynchronous collection of exit status, and terminal output. Such
|
||||
content may include sensitive information. Features may be added to mitigate
|
||||
the risk of this data being replicated in log messages, including errors.
|
||||
* Creating containers usually requires root access. This means that the Agent
|
||||
may need to be run with special privileges, or be given a method to
|
||||
escalate privileges using techniques such as sudo.
|
||||
* User provided data is passed through the API. This will require sensible
|
||||
data input validation.
|
||||
|
||||
|
||||
Notifications impact
|
||||
--------------------
|
||||
|
||||
Contemplated features (in subsequent release cycles):
|
||||
|
||||
* Notify the end user each time a Nova instance is created or deleted by
|
||||
the Containers service, if (s)he has registered for such notifications.
|
||||
* Notify the user each on CRUD of containers containing start and end
|
||||
notifications. (compute.container.create/delete/etc)
|
||||
* Notify user periodically of existence of container service managed
|
||||
containers (ex compute.container.exists)
|
||||
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
|
||||
The user interface will be a REST API. On top of that API will be an
|
||||
implementation of the libswarm API to allow for tools designed to use Docker
|
||||
to treat OpenStack as an upstream system.
|
||||
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
The Nova API will be used to create instances as needed. If the Container to
|
||||
Instance ratio is 10, then the Nova API will be called at least once for every
|
||||
10 calls to the Containers Service. Instances that are left empty will be
|
||||
automatically deleted, so in the example of a 10:1 ratio, the Nova API will be
|
||||
called to perform a delete for every 10 deletes in the Container Service.
|
||||
Depending on the configuration, the ratio may be as low as 1:1.
|
||||
The Containers Service will only access Nova through its API, not by accessing
|
||||
its database.
|
||||
|
||||
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
|
||||
Deployers may want to adjust the default flavor used for Nova Instances created
|
||||
by the Containers Service.
|
||||
|
||||
There should be no impact on users of prior releases, as this introduces a new
|
||||
API.
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
Minimal. There will be minimal changes required in Nova, if any.
|
||||
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
aotto
|
||||
|
||||
Other contributors:
|
||||
andrew-melton
|
||||
ewindisch
|
||||
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
1. Agent
|
||||
2. Relay
|
||||
3. API Service
|
||||
4. IO Relays
|
||||
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
1. <Links to Agent Blueprint and Spec here, once ready>
|
||||
2. Early implementations may use libswarm, or a python port of libswarm to
|
||||
implement Docker API compatibility.
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
Each commit will be accompanied with unit tests, and Tempest functional tests.
|
||||
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
A set of documentation for this new service will be required.
|
||||
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
* Link to high level draft proposal from the Nova Midcycle Meetup for Juno:
|
||||
`PDF <https://wiki.openstack.org/w/images/5/51/Containers_Proposal.pdf>`_
|
||||
* `Libswarm Source <https://github.com/docker/libswarm>`_
|
|
@ -0,0 +1,186 @@
|
|||
==================================
|
||||
Create a trustee user for each bay
|
||||
==================================
|
||||
|
||||
https://blueprints.launchpad.net/magnum/+spec/create-trustee-user-for-each-bay
|
||||
|
||||
Some services which are running in a bay need to access OpenStack services.
|
||||
For example, Kubernetes load balancer [1]_ needs to access Neutron. Docker
|
||||
registry [2]_ needs to access Swift. In order to access OpenStack services,
|
||||
we can create a trustee for each bay and delegate a limited set of rights to
|
||||
the trustee. [3]_ and [4]_ give a brief introduction to Keystone's trusts
|
||||
mechanism.
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
Some services which are running in a bay need to access OpenStack services,
|
||||
so we need to pass user credentials into the vms.
|
||||
|
||||
Use Cases
|
||||
---------
|
||||
|
||||
1. Kubernetes load balancer needs to access Neutron [1]_.
|
||||
2. For persistent storage, Cloud Provider needs to access Cinder to
|
||||
mount/unmount block storage to the node as volume [5]_.
|
||||
3. TLS cert is generated in the vms and need to be uploaded to Magnum [6]_ and
|
||||
[7]_.
|
||||
4. Docker registry needs to access Swift [2]_.
|
||||
|
||||
Project Priority
|
||||
----------------
|
||||
|
||||
High
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
When a user (the "trustor") wants to create a bay, steps for trust are as
|
||||
follows.
|
||||
|
||||
1. Create a new service account (the "trustee") without any role in a domain
|
||||
which is dedicated for trust. Without any role, the service account can do
|
||||
nothing in Openstack.
|
||||
|
||||
2. Define a trust relationship between the trustor and the trustee. The trustor
|
||||
can delegate a limited set of roles to the trustee. We can add an option
|
||||
named trust_roles in baymodel. Users can add roles which they want to
|
||||
delegate into trust_roles. If trust_roles is not provided, we delegate all
|
||||
the roles to the trustee.
|
||||
|
||||
3. Services in the bay can access OpenStack services with the trustee
|
||||
credentials and the trust.
|
||||
|
||||
The roles which are delegated to the trustee should be limited. If the services
|
||||
in the bay only need access to Neutron, we should not allow the services to
|
||||
access to other OpenStack services. But there is a limitation that a trustor
|
||||
must have the role which is delegated to a trustee [4]_.
|
||||
|
||||
Magnum now only allows the user who create the bay to get the certificate to
|
||||
avoid the security risk introduced by Docker [8]_. For example, if other users
|
||||
in the same tenant can get the certificate, then they can use Docker API to
|
||||
access the host file system of a bay node and get anything they want::
|
||||
|
||||
docker run --rm -v /:/hostroot ubuntu /bin/bash \
|
||||
-c "cat /hostroot/etc/passwd"
|
||||
|
||||
If Keystone doesn't allow to create new service accounts when LDAP is used as
|
||||
the backend for Keystone, we can use a pre-create service account for all
|
||||
bays. In this situation, all the bays use the same service account and
|
||||
different trust. We should add an config option to choose this method.
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
Magnum can create a user for each bay with roles to access OpenStack Services
|
||||
in a dedicated domain. The method has one disadvantage. The user which is
|
||||
created by magnum may get the access to OpenStack services which this user can
|
||||
not access before. For example, a user can not access Swift service and create
|
||||
a bay. Then Magnum create a service account for this bay with roles to access
|
||||
Swift. If the user logins into the vms and get the credentials, the user can
|
||||
use these credentials to access Swift.
|
||||
|
||||
Or Magnum doesn't prepare credentials and the user who create a bay needs to
|
||||
login into the nodes to manully add credentials in config files for services.
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
|
||||
Trustee id, trustee password and trust id are added to Bay table in Magnum
|
||||
database.
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
|
||||
Only the user who create a bay can get the certificate of this bay. Other
|
||||
users in the same tenant can not get the certificate now.
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
|
||||
Trustee id and trustee password are encrypted in magnum database. When Magnum
|
||||
passes these parameters to heat to create a stack, the transmission is
|
||||
encrypted by tls, so we don't need to encrypt these credentials. These
|
||||
credentials are hidden in heat, users can not query them in stack parameters.
|
||||
|
||||
Trustee id, trustee password and trust id can be obtained in the vms. Anyone
|
||||
who can login into the vms can get them and use these credentials to access
|
||||
OpenStack services. In a production environment, these vms must be secured
|
||||
properly to prevent unauthorized access.
|
||||
|
||||
Only the user who create the bay can get the certificate to access the COE
|
||||
api, so it is not a security risk even if the COE api is not safe.
|
||||
|
||||
Notifications impact
|
||||
--------------------
|
||||
|
||||
None
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
|
||||
None
|
||||
|
||||
Performance impact
|
||||
------------------
|
||||
|
||||
None
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
|
||||
None
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
None
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
humble00 (wanghua.humble@gmail.com)
|
||||
Other contributors:
|
||||
None
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
1. Create an trustee for each bay.
|
||||
2. Change the policy so that only the user who create a bay can get the
|
||||
certificate of the bay.
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
None
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
Unit test and functional test for service accounts and the policy change.
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
The user guide and troubleshooting guide will be updated with details
|
||||
regarding the service accounts.
|
||||
|
||||
References
|
||||
==========
|
||||
.. [1] http://docs.openstack.org/developer/magnum/dev/dev-kubernetes-load-balancer.html
|
||||
.. [2] https://blueprints.launchpad.net/magnum/+spec/registryv2-in-master
|
||||
.. [3] http://blogs.rdoproject.org/5858/role-delegation-in-keystone-trusts
|
||||
.. [4] https://wiki.openstack.org/wiki/Keystone/Trusts
|
||||
.. [5] https://github.com/kubernetes/kubernetes/blob/release-1.1/examples/mysql-cinder-pd/README.md
|
||||
.. [6] https://bugs.launchpad.net/magnum/+bug/1503863
|
||||
.. [7] https://review.openstack.org/#/c/232152/
|
||||
.. [8] https://docs.docker.com/engine/articles/security/#docker-daemon-attack-surface
|
||||
|
||||
History
|
||||
=======
|
||||
|
||||
None
|
|
@ -0,0 +1,171 @@
|
|||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
===================================
|
||||
Web Interface for Magnum in Horizon
|
||||
===================================
|
||||
|
||||
Launchpad blueprint:
|
||||
|
||||
https://blueprints.launchpad.net/magnum/+spec/magnum-horizon-plugin
|
||||
|
||||
Currently there is no way for a user to interact with Magnum through a web
|
||||
based user interface, as they are used to doing with other OpenStack
|
||||
components. This implementation aims to introduce this interface as an
|
||||
extension of Horizon (the OpenStack Dashboard) and expose all the features of
|
||||
Magnum in a way familiar to users.
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
In order to increase adoption and usability of Magnum we need to introduce a UI
|
||||
component for users and administrators to interact with Magnum without the need
|
||||
to use the command line. The UI proposed to be built will model all of the
|
||||
features currently available in the Magnum REST API and built using the Horizon
|
||||
plugin architecture to remain in line with other OpenStack UI projects and
|
||||
minimise the amount of new code that needs to be added.
|
||||
|
||||
Use Cases
|
||||
----------
|
||||
1. An end user wanting to use Magnum with OpenStack who is not comfortable in
|
||||
issuing commands with the python client will use the web user interface to
|
||||
interact with Magnum.
|
||||
2. An administrator may use the user interface to provide a quick overview of
|
||||
what Magnum has deployed in their OpenStack environment.
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
The first step will be to extend the Horizon API to include CRUD operations
|
||||
that are needed to interact with Magnum. Assuming that there are no issues here
|
||||
and API changes/additions are not required to Magnum, we can begin to
|
||||
design/implement the interface. We will aim to minimize the amount of Magnum
|
||||
specific UI code that will need to be maintained by reusing components from
|
||||
Horizon. This will also speed up the development significantly.
|
||||
|
||||
It is suggested the initial implementation of Magnum UI will include basic CRUD
|
||||
operations on BayModel and Bay resources. This will be the starting point for
|
||||
development and upon completion this will represent version 1.
|
||||
|
||||
Future direction includes adding CRUD operations for other Magnum features
|
||||
(Pod, Container, Service, ReplicationController) and will be tracked by new
|
||||
blueprints as they represent significant additional effort. The ultimate goal,
|
||||
a user should be able to perform all normal interactions with Magnum through
|
||||
the UI with no need for interaction with the python client.
|
||||
|
||||
Suggestions for further improvement include visualising Magnum resources to
|
||||
provide a quick overview of how resources are deployed.
|
||||
|
||||
Bugs/Blueprints relating specifically to the Magnum UI will be tracked here:
|
||||
|
||||
https://launchpad.net/magnum-ui
|
||||
|
||||
Mockups/Designs will be shared using the OpenStack Invision account located
|
||||
here:
|
||||
|
||||
https://openstack.invisionapp.com
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
One alternative to this approach is to develop an entirely separate UI
|
||||
specifically for Magnum. We will not use this approach as it does not fall in
|
||||
line with how other projects are managing their user interfaces and this
|
||||
approach would ultimately result in a significantly larger effort with much
|
||||
duplication with Horizon.
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
|
||||
None
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
|
||||
For Magnum, none. The Horizon API will need to be extended to include Create,
|
||||
Read, Update, Delete operations for all features available in the Magnum REST
|
||||
API. However, this extension to the Horizon API will live in the Magnum UI tree
|
||||
not the upstream Horizon tree.
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
|
||||
None
|
||||
|
||||
Notifications impact
|
||||
--------------------
|
||||
|
||||
None
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
|
||||
None
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
The Magnum API will be called from the user interface to return information to
|
||||
the user about the current state of Magnum objects and perform new interactions
|
||||
with Magnum. For every action a user performs from the user interface at least
|
||||
one API call to Magnum will need to be made.
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
|
||||
As the Magnum user interface will be managed and stored outside of the Horizon
|
||||
project deployers will need to pull down the Magnum UI code and add this to
|
||||
their Horizon install.
|
||||
|
||||
In order to add the Magnum UI to Horizon the deployer will have to copy an
|
||||
enable file to openstack_dashboard/local/enabled/ in their Horizon directory
|
||||
and then run Horizon as they would normally.
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
None
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
bradjones
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
1. Extend Horizon API in include Magnum calls
|
||||
2. CRUD operations on BayModel and Bay resources
|
||||
3. CRUD operations on other Magnum features (Pod, Container, Service, etc.)
|
||||
4. Refine the user experience
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
None
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
Each commit will be accompanied with unit tests. There will also be functional
|
||||
tests which will be used as part of a cross-functional gate test for Magnum.
|
||||
This additional gate test will be non-voting as failures will not indicate
|
||||
issues with Magnum but instead serves as advanced warning of any changes that
|
||||
could potentially break the UI.
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
An installation guide will be required.
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
None
|
|
@ -0,0 +1,177 @@
|
|||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
=================================
|
||||
Magnum and Open DC/OS Integration
|
||||
=================================
|
||||
|
||||
Launchpad Blueprint:
|
||||
|
||||
https://blueprints.launchpad.net/magnum/+spec/mesos-dcos
|
||||
|
||||
Open DC/OS [1]_ is a distributed operating system based on the Apache Mesos
|
||||
distributed systems kernel. It enables the management of multiple machines as
|
||||
if they were a single computer. It automates resource management, schedules
|
||||
process placement, facilitates inter-process communication, and simplifies
|
||||
the installation and management of distributed services. Its included web
|
||||
interface and available command-line interface (CLI) facilitate remote
|
||||
management and monitoring of the cluster and its services.
|
||||
|
||||
Open DC/OS now supports both docker containerizer and mesos containerizer.
|
||||
The mesos containerizer support both docker and AppC image spec, the mesos
|
||||
containerizer can manage docker containers well even if docker daemon is not
|
||||
running.
|
||||
|
||||
End user can install Open DC/OS with different ways, such as vagrant, cloud,
|
||||
local etc. For cloud, the Open DC/OS only supports AWS now, end user can
|
||||
deploy a DC/OS cluster quickly with a template. For local install, there
|
||||
are many steps to install a Open DC/OS cluster.
|
||||
|
||||
Problem Description
|
||||
===================
|
||||
|
||||
COEs (Container Orchestration Engines) are the first class citizen in Magnum,
|
||||
there are different COEs in Magnum now including Kubernetes, Swarm and Mesos.
|
||||
All of those COEs are focusing docker container management, the problem is
|
||||
that the concept of container is not only limited in docker container, but
|
||||
also others, such as AppC, linux container etc, Open DC/OS is planning to
|
||||
support different containers by leveraging Mesos unified container feature
|
||||
and the Open DC/OS has a better management console for container orchestration.
|
||||
|
||||
Currently, Magnum provides limited support for Mesos Bay as there is only one
|
||||
framework named as Marathon running on top of Mesos. Compared with Open DC/OS,
|
||||
the current Mesos Bay lack the following features:
|
||||
|
||||
1. App Store for application management. The Open DC/OS has a universe to
|
||||
provide app store functions.
|
||||
|
||||
2. Different container technology support. The Open DC/OS support different
|
||||
container technologies, such as docker, AppC etc, and may introduce OCI
|
||||
support in future. Introducing Open DC/OS Bay can enable Magnum to support
|
||||
more container technologies.
|
||||
|
||||
3. Better external storage integration. The Open DC/OS is planning to introduce
|
||||
docker volume isolator support in next release, the docker volume isolator
|
||||
is leveraging docker volume driver API to integrate with 3rd party
|
||||
distributed storage platforms, such as OpenStack Cinder, GlusterFS, Ceph
|
||||
etc.
|
||||
|
||||
4. Better network management. The Open DC/OS is planning to introduce CNI
|
||||
network isolator in next release, the CNI network isolator is leveraging CNI
|
||||
technologies to manage network for containers.
|
||||
|
||||
5. Loosely coupled with docker daemon. The Open DC/OS can work well for docker
|
||||
container even if docker daemon is not running. The docker daemon now have
|
||||
some issues in large scale cluster, so this approach avoids the limitation
|
||||
of the docker daemon but still can enable end user get some docker features
|
||||
in large scale cluster.
|
||||
|
||||
|
||||
Proposed Changes
|
||||
================
|
||||
|
||||
We propose extending Magnum as follows.
|
||||
|
||||
1. Leverage bay driver work and structure this new COE as a bay driver.
|
||||
|
||||
2. Leverage mesos-slave-flags [3]_ to customize Open DC/OS.
|
||||
|
||||
Here is an example of creating an Open DC/OS baymodel that uses
|
||||
docker/volume as isolator, linux as launcher and docker as image
|
||||
provider: ::
|
||||
|
||||
magnum baymodel-create --name dcosbaymodel \
|
||||
--image-id dcos-centos-7.2 \
|
||||
--keypair-id testkey \
|
||||
--external-network-id 1hsdhs88sddds889 \
|
||||
--dns-nameserver 8.8.8.8 \
|
||||
--flavor-id m1.small \
|
||||
--docker-volume-size 5 \
|
||||
--coe dcos \
|
||||
--labels isolation=docker/volume,\
|
||||
launcher=linux, \
|
||||
image_providers=docker
|
||||
|
||||
Magnum will validate the labels together with the driver specified before
|
||||
creating the bay and will return an error if the validation fails.
|
||||
|
||||
Magnum will continue to CRUD bays in the same way:
|
||||
|
||||
magnum bay-create --name dcosbay --baymodel dcosbaymodel --node-count 1
|
||||
|
||||
3. Keep the old Mesos Bay and add a new Open DC/OS Bay. Once the Open DC/OS Bay
|
||||
is stable, deprecate the Mesos Bay.
|
||||
|
||||
4. Update unit and functional tests to support Open DC/OS Bay, it is also an
|
||||
option to verify the Open DC/OS Bay in gate.
|
||||
|
||||
5. Preserve the user experience by ensuring that any operation on Open DC/OS
|
||||
Bay will be identical between a COE deployed by Magnum and a COE deployed
|
||||
by other methods.
|
||||
|
||||
|
||||
REST API Impact
|
||||
---------------
|
||||
|
||||
There will be no REST API exposed from Magnum for end user to operate Open
|
||||
DC/OS, end user can logon to Open DC/OS dashboard or call Open DC/OS REST
|
||||
API directly to manage the containers or the applications.
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
|
||||
- Guang Ya Liu (jay-lau-513)
|
||||
|
||||
Other contributors:
|
||||
|
||||
- Qun Wang (wangqun)
|
||||
- Gao Jin Cao
|
||||
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
1. Build VM image for Open DC/OS Bay.
|
||||
2. Add Open DC/OS Bay driver.
|
||||
3. Add Heat template for Open DC/OS Bay.
|
||||
4. Add Open DC/OS Bay monitor.
|
||||
5. Document how to use the Open DC/OS Bay.
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
1. This blueprint will focus on running on Open DC/OS in CentOS 7.2.
|
||||
|
||||
2. Depend on blueprint
|
||||
|
||||
https://blueprints.launchpad.net/magnum/+spec/mesos-slave-flags
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
Each commit will be accompanied with unit tests. There will also be
|
||||
functional tests which will be used as part of a cross-functional gate
|
||||
test for Magnum.
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
The Magnum Developer Quickstart document will be updated to support the Open
|
||||
DC/OS Bay introduced by including a short example and a full documentation
|
||||
with all the explanation for the labels in the user guide. Additionally,
|
||||
background information on how to use the Open DC/OS Bay will be included.
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
.. [1] https://dcos.io/docs/1.7/overview/what-is-dcos/
|
||||
.. [2] https://dcos.io/install/
|
||||
.. [3] https://blueprints.launchpad.net/magnum/+spec/mesos-slave-flags
|
|
@ -0,0 +1,252 @@
|
|||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
==========================
|
||||
Quota for Magnum Resources
|
||||
==========================
|
||||
|
||||
Launchpad blueprint:
|
||||
|
||||
https://blueprints.launchpad.net/magnum/+spec/resource-quota
|
||||
|
||||
There are multiple ways to slice an OpenStack cloud. Imposing quota on these
|
||||
various slices puts a limitation on the amount of resources that can be
|
||||
consumed which helps to guarantee "fairness" or fair distribution of resource
|
||||
at the creation time. If a particular project needs more resources, the
|
||||
concept of quota, gives the ability to increase the resource count on-demand,
|
||||
given that the system constraints are not exceeded.
|
||||
|
||||
|
||||
Problem description
|
||||
===================
|
||||
At present in Magnum we don't have the concept of Quota on Magnum resources as
|
||||
a result of which, as long as the underlying Infrastructure as a Service(IaaS)
|
||||
layer has resources, any user can consume as many resources as they want, with
|
||||
the hardlimit associated with the tenant/project being the upper bound for the
|
||||
resources to be consumed. Quotas are tied closely to physical resources and are
|
||||
billable entity and hence from Magnum's perspective it makes sense to limit the
|
||||
creation and consumption of a particular kind of resource to a certain value.
|
||||
|
||||
Use cases
|
||||
---------
|
||||
Alice is the admin. She would like to have the feature which will give her
|
||||
details of Magnum resource consumption so that she can manage her resource
|
||||
appropriately.
|
||||
|
||||
a. Ability to know current resource consumption.
|
||||
b. Ability to prohibit overuse by a project.
|
||||
c. Prevent situation where users in the project get starved because users in
|
||||
other project consume all the resource. Alice feels something like
|
||||
"Quota Management" would help to guarantee "fairness".
|
||||
d. Prevent DOS kind of attack, abuse or error by users where an excessive
|
||||
amount of resources are created.
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
Proposed change is to introduce a Quota Table which will primarily store the
|
||||
quota assigned to each resource in a project. For Mitaka, we will restrict
|
||||
the scope to a Bay, which are Magnum resources. Primarily, as a first step we
|
||||
will start of by imposing quota on number of bays to be created in a project.
|
||||
The change also plans to introduce REST API's to GET/PUT/POST/DELETE. CLIs to
|
||||
get information of Quota for a particular project will also be provided.
|
||||
|
||||
For Mitaka, we will restrict the scope of the resources explicit created and
|
||||
managed by Magnum. Specifically for Mitaka we will focus on number of
|
||||
Bays only. Going ahead we might add Quota for containers, etc. The resources
|
||||
of which a Bay is constructed out of is inherently not only Magnum resource
|
||||
but involve resource from Nova, Cinder, Neutron etc. Limiting those resource
|
||||
consumption is out of the scope of this spec and needs a close collaboration
|
||||
with the quota management framework of the orchestration layer, since the
|
||||
orchestration layer can invoke the respective IaaS projects API's and get the
|
||||
consumption details before provisioning. As of now the orchestration layer
|
||||
used by Magnum, Heat, does not have the concept of Quota, so we will start with
|
||||
imposing Quota on resources which Magnum manages, Bay, more specifically for
|
||||
Mitaka.
|
||||
|
||||
When a project is created and if the Magnum service is running, the default
|
||||
quota for Magnum resources will be set by the values configured in magnum.conf.
|
||||
Other Openstack projects like Nova [2]_, Cinder [3]_ follow a similar pattern
|
||||
and we will also do so and hence won't have a separate CLI for quota-create.
|
||||
Later if the user wants to change the Quota of the resource option will be
|
||||
provided to do so using magnum quota-update. In situation where all of the
|
||||
quota for a specific Magnum resource (Bay) have been consumed and is
|
||||
under use, admin will be allowed to set the quota to a any value lower than
|
||||
the usage or hardlimit to prohibit users from the project to create new
|
||||
Bays. This gives more flexibility to the admin to have a better control
|
||||
on resource consumption. Till the time the resource is not explicitly deleted
|
||||
the quota associated with the project, for a particular resource, won't be
|
||||
decreased. In short quota-update support will take into consideration the
|
||||
new hardlimit for a resource, specified by the admin, and will set the new
|
||||
value for this resource.
|
||||
|
||||
Before the resource is created, Magnum will check for current count of the
|
||||
resource(Bays) created for a project. If the resource(Bay) count is less
|
||||
than the hardlimit set for the Bay, new Bay creation will be allowed. Since
|
||||
Bay creation is a long running operation, special care will be taken while
|
||||
computing the available quota. For example, 'in-progress' field in the Quota
|
||||
usages table will be updated when the resource(Bay) creation is initiated and
|
||||
is in progress. Lets say the quota hardlimit is 5 and 3 Bay's have already been
|
||||
created and two new requests come in to create new Bays. Since we have 3 Bays
|
||||
already created the 'used' field will be set to 3. Now the 'in-progress'
|
||||
field will be set to 2 till the time the Bay creation is successful. Once
|
||||
the Bay creation is done this field will be reset to 0, and the 'used'
|
||||
count will be updated from 3 to 5. So at this moment, hardlimit is 5, used
|
||||
is 5 and in-progress is 0. So lets say one more request comes in to create
|
||||
new Bay this request will be prohibited since there is not enough quota
|
||||
available.
|
||||
|
||||
For Bays,
|
||||
|
||||
available = hard_limit - [in_progress + used]
|
||||
|
||||
In general,
|
||||
|
||||
Resource quota available = Resource hard_limit - [
|
||||
(Resource creation in progress + Resources already created for project)]
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
At present there is not quota infrastructure in Magnum.
|
||||
|
||||
Adding Quota Management layer at the Orchestration layer, Heat, could be an
|
||||
alternative. Doing so will give a finer view of resource consumption at the
|
||||
IaaS layer which can be used while provisioning Magnum resources which
|
||||
depend on the IaaS layer [1]_.
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
New Quota and Quota usages table will be introduced to Magnum database to
|
||||
store quota consumption for each resource in a project.
|
||||
|
||||
Quota Table :
|
||||
|
||||
+------------+--------------+------+-----+---------+----------------+
|
||||
| Field | Type | Null | Key | Default | Extra |
|
||||
+------------+--------------+------+-----+---------+----------------+
|
||||
| id | int(11) | NO | PRI | NULL | auto_increment |
|
||||
| created_at | datetime | YES | | NULL | |
|
||||
| updated_at | datetime | YES | | NULL | |
|
||||
| project_id | varchar(255) | YES | MUL | NULL | |
|
||||
| resource | varchar(255) | NO | | NULL | |
|
||||
| hard_limit | int(11) | YES | | NULL | |
|
||||
+------------+--------------+------+-----+---------+----------------+
|
||||
|
||||
Quota usages table :
|
||||
|
||||
+---------------+--------------+------+-----+---------+----------------+
|
||||
| Field | Type | Null | Key | Default | Extra |
|
||||
+---------------+--------------+------+-----+---------+----------------+
|
||||
| created_at | datetime | YES | | NULL | |
|
||||
| updated_at | datetime | YES | | NULL | |
|
||||
| id | int(11) | NO | PRI | NULL | auto_increment |
|
||||
| project_id | varchar(255) | YES | MUL | NULL | |
|
||||
| resource | varchar(255) | NO | | NULL | |
|
||||
| in_progress | int(11) | NO | | NULL | |
|
||||
| used | int(11) | NO | | NULL | |
|
||||
+---------------+--------------+------+-----+---------+----------------+
|
||||
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
REST API will be added for :
|
||||
|
||||
1. quota-defaults List all default quotas for all tenants.
|
||||
2. quota-show List the currently set quota values for a tenant.
|
||||
3. quota-update Updates quotas for a tenant.
|
||||
4. quota-usage Lists quota usage for a tenant.
|
||||
5. quota-list List quota for all the tenants.
|
||||
|
||||
A user with "admin" role will be able to do all the above operations but a user
|
||||
with "non-admin" role will be restricted to only get/list quota associated to
|
||||
his/her tenant. User with "non-admin" role can be a Member of the tenant less
|
||||
"admin" role.
|
||||
|
||||
REST API for resources which will have quota imposed will be enhanced :
|
||||
|
||||
1. Bay create
|
||||
Will check if there is quota available for Bay creation, if so proceed
|
||||
ahead with the request otherwise throw exception that not enough quota is
|
||||
available.
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
None
|
||||
|
||||
Notifications impact
|
||||
--------------------
|
||||
None
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
End user will have the option to look at the quota set on the resources, quota
|
||||
usage by a particular project.
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
None
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
None
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
None
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
vilobhmm
|
||||
|
||||
Other contributors:
|
||||
None
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
1. Introduce Quota and Quota usages table in Magnum database.
|
||||
2. Introduce API to set/update Quota for a resource, specifically
|
||||
bay, for Mitaka release.
|
||||
3. Introduce API to create Quota entry, by default, for a resource.
|
||||
4. Provide config options that will allow users/admins to set Quota.
|
||||
5. Make sure that if the resource is deleted the used count from the
|
||||
quota_usages table will be decremented by the number of resources
|
||||
deleted. For example, if resource, bay, is deleted then the entries
|
||||
for it in the Quota usages table should be decremented by the
|
||||
number of Bays deleted.
|
||||
6. Provide CLI options to view the quota details :
|
||||
a. magnum quota-show <project-id>
|
||||
b. magnum quota-update <project-id> <resource> <hard-limit>
|
||||
c. magnum quota-defaults <project-id>
|
||||
d. magnum quota-usage <project-id>
|
||||
e. magnum quota-list
|
||||
7. Add conf setting for bays default quota since we will focus
|
||||
on Bays for Mitaka.
|
||||
|
||||
Dependencies
|
||||
============
|
||||
None
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
1. Each commit will be accompanied with unit tests.
|
||||
2. Gate functional tests will also be covered.
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
None
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
.. [1] http://lists.openstack.org/pipermail/openstack-dev/2015-December/082266.html
|
||||
.. [2] https://github.com/openstack/nova/blob/master/nova/quota.py
|
||||
.. [3] https://github.com/openstack/nova/blob/master/cinder/quota.py
|
|
@ -0,0 +1,226 @@
|
|||
=====================
|
||||
TLS support in Magnum
|
||||
=====================
|
||||
|
||||
Launchpad blueprint:
|
||||
|
||||
https://blueprints.launchpad.net/magnum/+spec/secure-kubernetes
|
||||
|
||||
Currently there is no authentication in Magnum to provide access control to
|
||||
limit communication between the Magnum service and the Kubernetes service so
|
||||
that Kubernetes can not be controlled by a third party. This implementation
|
||||
closes this security loophole by using TLS as an access control mechanism.
|
||||
Only the Magnum server will have the key to communicate with any given
|
||||
Kubernetes API service under its control. An additional benefit of this
|
||||
approach is that communication over the network will be encrypted, reducing
|
||||
the chance of eavesdropping on the communication stream.
|
||||
|
||||
Problem Description
|
||||
-------------------
|
||||
|
||||
Magnum currently controls Kubernetes API services using unauthenticated HTTP.
|
||||
If an attacker knows the api_address of a Kubernetes Bay, (s)he can control
|
||||
the cluster without any access control.
|
||||
|
||||
Use Cases
|
||||
---------
|
||||
|
||||
1. Operators expect system level control to be protected by access control that
|
||||
is consistent with industry best practices. Lack of this feature may result in
|
||||
rejection of Magnum as an option for hosting containerized workloads.
|
||||
|
||||
Proposed Changes
|
||||
----------------
|
||||
|
||||
The complete implementation of TLS support in Magnum can be further decomposed
|
||||
into below smaller implementations.
|
||||
|
||||
1. TLS support in Kubernetes Client Code.
|
||||
-----------------------------------------
|
||||
|
||||
The current implementation of Kubernetes Client code doesn't have any
|
||||
authentication. So this implementation will change the client code to
|
||||
provide authentication using TLS.
|
||||
|
||||
Launchpad blueprint:
|
||||
|
||||
https://blueprints.launchpad.net/magnum/+spec/tls-pythonk8sclient
|
||||
|
||||
2. Generating certificates
|
||||
----------------------------
|
||||
|
||||
This task is mainly on how certificates for both client(magnum-conductor)
|
||||
and server(kube-apiserver) will be generated and who will be the certificate
|
||||
authority(CA).
|
||||
|
||||
These files can be generated in two ways:
|
||||
|
||||
2.1. Magnum script
|
||||
-------------------
|
||||
|
||||
This implementation will use standard tool to generate certificates and
|
||||
keys. This script will be registered on Kubernetes master node while creating
|
||||
bay. This script will generate certificates, start the secure kube-apiserver
|
||||
and then register the client certificates at Magnum.
|
||||
|
||||
2.2. Using Barbican
|
||||
-------------------
|
||||
|
||||
Barbican can also be used as a CA using Dogtag. This implementation will use
|
||||
Barbican to generate certificates.
|
||||
|
||||
3. TLS Support in Magnum code
|
||||
------------------------------
|
||||
|
||||
This work mainly involves deploying a secure bay and supporting the use of
|
||||
certificates in Magnum to call Kubernetes API. This implementation can be
|
||||
decomposed into smaller tasks.
|
||||
|
||||
3.1. Create secure bay
|
||||
----------------------
|
||||
|
||||
This implementation will deploy a secure kube-apiserver running on Kubernetes
|
||||
master node. To do so following things needs to be done:
|
||||
|
||||
* Generate certificates
|
||||
* Copy certificates
|
||||
* Start a secure kube-apiserver
|
||||
|
||||
3.1.1. Generate certificates
|
||||
----------------------------
|
||||
|
||||
The certificates will be generated using any of the above implementation in
|
||||
section 2.
|
||||
|
||||
3.1.2. Copy certificates
|
||||
------------------------
|
||||
|
||||
This depends on how cert and key is generated, the implementation will differ
|
||||
with each case.
|
||||
|
||||
3.1.2.1. Using Magnum script
|
||||
----------------------------
|
||||
|
||||
This script will generate both server and client certificates on Kubernetes
|
||||
master node. Hence only client certificates needs to be copied to magnum host
|
||||
node. To copy these files, the script will make a call to magnum-api to store
|
||||
files.
|
||||
|
||||
3.1.2.2. Using Barbican
|
||||
-----------------------
|
||||
|
||||
When using Barbican, the cert and key will be generated and stored in Barbican
|
||||
itself. Either magnum-conductor can fetch the certificates from Barbican and
|
||||
copy on Kubernetes master node or it can be fetched from Kubernetes master node
|
||||
also.
|
||||
|
||||
3.1.3. Start a secure kube-apiserver
|
||||
------------------------------------
|
||||
|
||||
Above generated certificates will be used to start a secure kube-apiserver
|
||||
running on Kubernetes master node.
|
||||
|
||||
Now that we have a secure Kubernetes cluster running, any API call to
|
||||
Kubernetes will be secure.
|
||||
|
||||
|
||||
3.2. Support https
|
||||
------------------
|
||||
|
||||
While running any Kubernetes resource related APIs, magnum-conductor will
|
||||
fetch certificate from magnum database or Barbican and use it to make secure
|
||||
API call.
|
||||
|
||||
4. Barbican support to store certificates securely
|
||||
----------------------------------------------------
|
||||
|
||||
Barbican is a REST API designed for the secure storage, provisioning and
|
||||
management of secrets. The client cert and key must be stored securely. This
|
||||
implementation will support Barbican in Magnum to store the sensitive data.
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
|
||||
New table 'cert' will be introduced to store the certificates.
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
|
||||
New API /certs will be introduced to store the certificates.
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
|
||||
After this support, Magnum will be secure to be used in actual production
|
||||
environment. Now all the communication to Kubernetes master node will be
|
||||
secure.
|
||||
The certificates will be generated by Barbican or standard tool signed by
|
||||
trusted CAs.
|
||||
The certificates will be stored safely in Barbican when the Barbican cert
|
||||
storage option is selected by the administrator.
|
||||
|
||||
Notifications impact
|
||||
--------------------
|
||||
|
||||
None
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
|
||||
None
|
||||
|
||||
Performance impact
|
||||
------------------
|
||||
|
||||
None
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
|
||||
Deployer will need to install Barbican to store certificates.
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
None
|
||||
|
||||
Implementation
|
||||
--------------
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee
|
||||
madhuri(Madhuri Kumari)
|
||||
yuanying(Motohiro Otsuka)
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
1. TLS Support in Kubernetes Client code
|
||||
2. Support for generating keys in Magnum
|
||||
3. Support creating secure Kubernetes cluster
|
||||
4. Support Barbican in Magnum to store certificates
|
||||
|
||||
Dependencies
|
||||
------------
|
||||
|
||||
Barbican(optional)
|
||||
|
||||
Testing
|
||||
-------
|
||||
|
||||
Each commit will be accompanied with unit tests. There will also be functional
|
||||
test to test both good and bad certificates.
|
||||
|
||||
Documentation Impact
|
||||
--------------------
|
||||
|
||||
Add a document explaining how TLS cert and keys can be generated and guide
|
||||
updated with how to use the secure model of bays.
|
||||
|
||||
|
||||
References
|
||||
----------
|
||||
|
||||
None
|
Loading…
Reference in New Issue