Custom language packs
Change-Id: Ic98b61df0e7f58b7382646e5a502adad39d0ef70
This commit is contained in:
parent
3070d48960
commit
96c8ff5fe5
|
@ -0,0 +1,258 @@
|
||||||
|
Problem description
|
||||||
|
===================
|
||||||
|
|
||||||
|
This spec considers requirements, implementation, and changes to Solum
|
||||||
|
CLI and API to support custom language packs.
|
||||||
|
|
||||||
|
|
||||||
|
Problem Details:
|
||||||
|
------------------
|
||||||
|
A language pack in Solum is essentially a base image (VM) with appropriate
|
||||||
|
libraries, compilers, runtimes installed on it.
|
||||||
|
|
||||||
|
Custom language pack is a base image that has libraries and packages
|
||||||
|
specified by the user (cloud operator and/or an application developer)
|
||||||
|
installed on it.
|
||||||
|
|
||||||
|
Towards building such custom language packs Solum needs:
|
||||||
|
(a) the ability to specify the required base image type and version
|
||||||
|
(b) the ability to specify the required libraries and packages to be installed
|
||||||
|
on the base image
|
||||||
|
(c) the ability to register the language pack with Glance
|
||||||
|
|
||||||
|
One way to implement this feature is to require that a user provide two things
|
||||||
|
to Solum:
|
||||||
|
(a) base image name and type
|
||||||
|
(b) link to github repository with a script with a predefined name
|
||||||
|
(such as 'prepare') in it, which contains installation instructions to install
|
||||||
|
the required libraries and packages.
|
||||||
|
|
||||||
|
Solum would provide a 'language pack builder' API to build the language pack
|
||||||
|
and register it in glance.
|
||||||
|
|
||||||
|
At a high-level the builder API will do following actions in a secure
|
||||||
|
environment:
|
||||||
|
(a) mount the specified base image
|
||||||
|
(b) clone the repository
|
||||||
|
(c) run the 'prepare' script
|
||||||
|
(d) snapshot the new image
|
||||||
|
(e) upload the image to glance
|
||||||
|
|
||||||
|
|
||||||
|
Proposed implementation:
|
||||||
|
-------------------------
|
||||||
|
|
||||||
|
User provides a Dockerfile in their language
|
||||||
|
pack repository. Solum builds the language pack via 'docker build'.
|
||||||
|
|
||||||
|
Note that the language pack created using this approach is going to be a
|
||||||
|
Docker container and not a VM image.
|
||||||
|
|
||||||
|
One of the main advantage of this approach is that it uses Dockerfiles as
|
||||||
|
standard format which became mainstream in several systems requiring
|
||||||
|
containers.
|
||||||
|
|
||||||
|
Towards using such a language pack in creating a DU (deployment unit), we have
|
||||||
|
to consider the following.
|
||||||
|
|
||||||
|
If the operator has configured Solum to use 'docker' as the SOLUM_IMAGE_FORMAT
|
||||||
|
then the Docker based LP will work without any issues.
|
||||||
|
|
||||||
|
However, if SOLUM_IMAGE_FORMAT is set to 'vm' then we will have to provide
|
||||||
|
a VM image that has Docker installed. CoreOS is a possible approach
|
||||||
|
(see this WIP https://review.openstack.org/#/c/102646/).
|
||||||
|
Another approach would be to use the heat docker plugin.
|
||||||
|
|
||||||
|
|
||||||
|
Additional considerations:
|
||||||
|
----------------------------
|
||||||
|
The question of why to separate out language pack creation step and not have
|
||||||
|
the language pack Dockerfile listed in users' application code repository
|
||||||
|
can be asked. While this is certainly possible there are several advantages of
|
||||||
|
separating language pack creation from application building. First, language
|
||||||
|
pack creation is a one time process. Clubbing it with application
|
||||||
|
building will lead to build performance overhead. Second, by requiring
|
||||||
|
application developers to define Dockerfiles in their application
|
||||||
|
repositories Solum will be binding them
|
||||||
|
in a contract that their code would work only when the operator has configured
|
||||||
|
Solum to use Docker as the image format.
|
||||||
|
|
||||||
|
|
||||||
|
Proposed change
|
||||||
|
===============
|
||||||
|
1) CLI changes:
|
||||||
|
- Create a new command 'languagepack build' which will be used as follows:
|
||||||
|
|
||||||
|
solum languagepack build <github url of custom lp repository>
|
||||||
|
|
||||||
|
This will use the 'builder' API (which runs on port 9778) which Solum already provides.
|
||||||
|
|
||||||
|
|
||||||
|
2) API changes:
|
||||||
|
(a) Start with our builder API. Modify it if required.
|
||||||
|
We can start with POST /v1/solum/builder/
|
||||||
|
The data sent in will be
|
||||||
|
github repository URL containing the Dockerfile.
|
||||||
|
This will lead to following steps:
|
||||||
|
|
||||||
|
- clone the git repo
|
||||||
|
- do 'docker build'
|
||||||
|
- do 'docker push' to Solum's docker registry to upload the LP and make it
|
||||||
|
available in Glance
|
||||||
|
|
||||||
|
|
||||||
|
Alternatives
|
||||||
|
------------
|
||||||
|
|
||||||
|
Option 1:
|
||||||
|
---------
|
||||||
|
Use disk-image-builder (dib) as the mechanism to build the language pack
|
||||||
|
The issues with this option are:
|
||||||
|
|
||||||
|
(1) Figuring out how do the contents of 'prepare' translate into dib element
|
||||||
|
(pre-install.d, install.d, post-install.d, finalize.d, etc.)
|
||||||
|
|
||||||
|
(2) Figuring out how to run 'prepare' in a sandboxed environment.
|
||||||
|
We need this as the contents of 'prepare' can be anything.
|
||||||
|
|
||||||
|
Advantages:
|
||||||
|
-----------
|
||||||
|
The images created are compatible with glance.
|
||||||
|
|
||||||
|
Disadvantages:
|
||||||
|
----------------
|
||||||
|
One of the disadvantages is that Solum would need to
|
||||||
|
build a translator to translate contents of 'prepare' in the dib's dsl.
|
||||||
|
|
||||||
|
Option 2:
|
||||||
|
---------
|
||||||
|
Don't use disk-image-builder, but do similar to what we are currently
|
||||||
|
doing as part of 'build-app' (i.e. through Solum code perform the steps
|
||||||
|
of mounting fs, executing installation steps, creating image snapshot).
|
||||||
|
|
||||||
|
Advantages:
|
||||||
|
-----------
|
||||||
|
One of the advantages of Option 2 is that no translation is required of
|
||||||
|
the contents of the 'prepare' script.
|
||||||
|
|
||||||
|
Disadvantages:
|
||||||
|
--------------
|
||||||
|
The disadvantage is that Solum would need to build mechanisms to create a
|
||||||
|
Glance compatible image.
|
||||||
|
|
||||||
|
One approach to achieve advantages of both options without incurring their
|
||||||
|
disadvantages is to use dibs but avoid converting 'prepare' into various
|
||||||
|
elements. Just convert it into install.d element.
|
||||||
|
Overtime Solum can support 'hints' which can be used to indicate how should
|
||||||
|
'prepare' be broken up to be used within different elements. It is possible
|
||||||
|
that such hints are provided as separate scripts that are supported by Solum.
|
||||||
|
For example, Solum may start supporting 'pre_install, prepare, post_install'
|
||||||
|
scripts.
|
||||||
|
|
||||||
|
|
||||||
|
Data model impact
|
||||||
|
-----------------
|
||||||
|
|
||||||
|
Would need to be changed to support LP creation actions. This includes
|
||||||
|
changes to the data model to persist build and test status.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
REST API impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
Discussed above
|
||||||
|
|
||||||
|
|
||||||
|
Security impact
|
||||||
|
---------------
|
||||||
|
Language pack builds would need to be done in isolated environments so
|
||||||
|
as to not affect other builds. Need to investigate in detail if Docker-based
|
||||||
|
approach would provide good enough contained environment.
|
||||||
|
|
||||||
|
|
||||||
|
Notifications impact
|
||||||
|
--------------------
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Other end user impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
Performance Impact
|
||||||
|
------------------
|
||||||
|
Building a languagepack is expensive operation as it involves building an image
|
||||||
|
by downloading and installing the specified packages in the Dockerfile.
|
||||||
|
Depending on the available CPU, network, and memory resources the performance
|
||||||
|
of this operation will get affected.
|
||||||
|
|
||||||
|
|
||||||
|
Other deployer impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
|
||||||
|
Developer impact
|
||||||
|
----------------
|
||||||
|
|
||||||
|
|
||||||
|
Implementation
|
||||||
|
==============
|
||||||
|
|
||||||
|
Assignee(s)
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Devdatta Kulkarni (devdatta-kulkarni) will implement this custom language
|
||||||
|
pack proposal:
|
||||||
|
|
||||||
|
Work Items
|
||||||
|
----------
|
||||||
|
- Create an example custom language pack
|
||||||
|
(https://review.openstack.org/#/c/103671/)
|
||||||
|
- Make the required REST API changes (identified in proposed changes section)
|
||||||
|
- Make the required CLI change (identified in proposed changes section)
|
||||||
|
(Arati Mahimane has started on this)
|
||||||
|
- Enchance the data model to store data about image build actions and the
|
||||||
|
test results.
|
||||||
|
|
||||||
|
|
||||||
|
Dependencies
|
||||||
|
============
|
||||||
|
We will start with the solum-builder api (and code). Once the build-farm
|
||||||
|
stuff is merged we can revisit this to see if that would be something
|
||||||
|
that can be used for this.
|
||||||
|
|
||||||
|
|
||||||
|
Testing
|
||||||
|
=======
|
||||||
|
Test that the created language pack was installed with all the packages and
|
||||||
|
libraries listed in the Dockerfile.
|
||||||
|
One way to achieve this would be to delegate the testing responsibility
|
||||||
|
itself to the language pack author. For instance, lp author could use
|
||||||
|
RUN statements within their Dockerfile that does the necessary checks.
|
||||||
|
We would require that after checks are completed a test result file be
|
||||||
|
created (say, /solum.lp.test) which contains the language pack creation
|
||||||
|
status. We can then look for that to determine if the testing passed or not,
|
||||||
|
without worrying about figuring out what package management style was used.
|
||||||
|
|
||||||
|
If language pack creation fails then Solum will take following actions:
|
||||||
|
- Save the details of the build and test results in Solum's internal database
|
||||||
|
- Log the build failure status in log stream for the user's actions
|
||||||
|
- If a user's email address is available, send a notification of build failure
|
||||||
|
to that address. Solum would first need to be enhanced with a
|
||||||
|
mail server functionality to support this.
|
||||||
|
|
||||||
|
|
||||||
|
Documentation Impact
|
||||||
|
====================
|
||||||
|
Documentation would need to be updated to reflect the addition of new CLI
|
||||||
|
commands.
|
||||||
|
|
||||||
|
|
||||||
|
References
|
||||||
|
==========
|
||||||
|
|
||||||
|
https://blueprints.launchpad.net/solum/+spec/custom-language-packs
|
||||||
|
|
||||||
|
https://etherpad.openstack.org/p/custom-language-packs
|
Loading…
Reference in New Issue