From c6862ec53715de628a96af1caf075b3e02a7190d Mon Sep 17 00:00:00 2001 From: Michal Rostecki Date: Wed, 25 May 2016 15:20:45 +0200 Subject: [PATCH] Repositories split Change-Id: Ie10f20c04aa2d0bc889184df4f4ed6a1d57764bf --- specs/repositories-split.rst | 324 +++++++++++++++++++++++++++++++++++ 1 file changed, 324 insertions(+) create mode 100644 specs/repositories-split.rst diff --git a/specs/repositories-split.rst b/specs/repositories-split.rst new file mode 100644 index 0000000..127d3b6 --- /dev/null +++ b/specs/repositories-split.rst @@ -0,0 +1,324 @@ +========================================== +Repositories split +========================================== + +MCP project is going to use splitted repositories for Dockerfiles and +config files per each OpenStack service and infra service. + + +Problem description +=================== + +MCP project wants to move out from using Kolla images and it's aiming to +provide its own repositories for Dockerfiles and configuration files + +The layout of Kolla repository, which contains all the Dockerfiles and config +files is not satysfying NextGen needs. + +Pros for splitting repositories: + +* Possibility to engage engineers from teams to take some responsibility on + containerizing each component. +* Similarity to the Puppet, Chef and Ansible repositories in upstream. + +Cons: + +* CI will have to handle cross-repository dependencies. That may be hard + especially in case of fixing the issues which have to be fixed in the + configuration of multiple OpenStack components at once, because CI will become + green after accepting a stream of patches in many repos. +* Release management and stable branch maintenance will be harder, because + the whole Microservices project will contain ~38 repositories. + +Use Cases +--------- + +Proposed change +=============== + +The "nextgen" organization should be created on Gerrit with following git +repositories: + +* nextgen/microservices - the builder of Docker images and CLI tool for pushing + configuration in etcd and deploying OpenStack on Kubernetes +* nextgen/ms-ext-config - start script for containers which fetches config + from etcd/ZooKeeper +* nextgen/ms-debian-base - the layer for base Linux packages +* nextgen/ms-openstack-base - the base layer for OpenStack services, containing + common Python modules +* nextgen/ms-aodh-api +* nextgen/ms-aodh-base +* nextgen/ms-aodh-evaluator +* nextgen/ms-aodh-expirer +* nextgen/ms-aodh-listener +* nextgen/ms-aodh-notifier +* nextgen/ms-ceilometer-api +* nextgen/ms-ceilometer-base +* nextgen/ms-ceilometer-central +* nextgen/ms-ceilometer-collector +* nextgen/ms-ceilometer-compute +* nextgen/ms-ceilometer-notification +* nextgen/ms-ceph-base +* nextgen/ms-ceph-mon +* nextgen/ms-ceph-osd +* nextgen/ms-ceph-rgw +* nextgen/ms-cinder-api +* nextgen/ms-cinder-backup +* nextgen/ms-cinder-base +* nextgen/ms-cinder-rpcbind +* nextgen/ms-cinder-scheduler +* nextgen/ms-cinder-volume +* nextgen/ms-designate-api +* nextgen/ms-designate-backend-bind9 +* nextgen/ms-designate-base +* nextgen/ms-designate-central +* nextgen/ms-designate-mdns +* nextgen/ms-designate-poolmanager +* nextgen/ms-designate-sink +* nextgen/ms-elasticsearch +* nextgen/ms-glance-api +* nextgen/ms-glance-base +* nextgen/ms-glance-registry +* nextgen/ms-heat-api +* nextgen/ms-heat-api-cfn +* nextgen/ms-heat-base +* nextgen/ms-heat-engine +* nextgen/ms-heka +* nextgen/ms-horizon +* nextgen/ms-ironic-api +* nextgen/ms-ironic-base +* nextgen/ms-ironic-conductor +* nextgen/ms-ironic-inspector +* nextgen/ms-ironic-pxe +* nextgen/ms-keystone +* nextgen/ms-kibana +* nextgen/ms-toolbox +* nextgen/ms-magnum-api +* nextgen/ms-magnum-base +* nextgen/ms-magnum-conductor +* nextgen/ms-manila-api +* nextgen/ms-manila-base +* nextgen/ms-manila-scheduler +* nextgen/ms-manila-share +* nextgen/ms-mariadb +* nextgen/ms-memcached +* nextgen/ms-mistral-api +* nextgen/ms-mistral-base +* nextgen/ms-mistral-engine +* nextgen/ms-mistral-executor +* nextgen/ms-mongodb +* nextgen/ms-murano-api +* nextgen/ms-murano-base +* nextgen/ms-murano-engine +* nextgen/ms-neutron-base +* nextgen/ms-neutron-dhcp-agent +* nextgen/ms-neutron-l3-agent +* nextgen/ms-neutron-linuxbridge-agent +* nextgen/ms-neutron-metadata-agent +* nextgen/ms-neutron-openvswitch-agent +* nextgen/ms-neutron-server +* nextgen/ms-nova-api +* nextgen/ms-nova-base +* nextgen/ms-nova-compute-ironic +* nextgen/ms-nova-compute +* nextgen/ms-nova-conductor +* nextgen/ms-nova-consoleauth +* nextgen/ms-nova-libvirt +* nextgen/ms-nova-network +* nextgen/ms-nova-novncproxy +* nextgen/ms-nova-scheduler +* nextgen/ms-nova-spicehtml5proxy +* nextgen/ms-openvswitch-base +* nextgen/ms-openvswitch-db-server +* nextgen/ms-openvswitch-vswitchd +* nextgen/ms-rabbitmq +* nextgen/ms-sahara-api +* nextgen/ms-sahara-base +* nextgen/ms-sahara-engine +* nextgen/ms-swift-account +* nextgen/ms-swift-base +* nextgen/ms-swift-container +* nextgen/ms-swift-object +* nextgen/ms-swift-proxy-server +* nextgen/ms-swift-rsyncd +* nextgen/ms-tempest +* nextgen/ms-trove-api +* nextgen/ms-trove-base +* nextgen/ms-trove-conductor +* nextgen/ms-trove-guestagent +* nextgen/ms-trove-taskmanager +* nextgen/ms-zaqar +* nextgen/dev-env - Vagrantfile for creating the development environment +* nextgen/puppet-deployment - Puppet mianifests for deploying Kubernetes and all + infra aroud it +* nextgen/project-config - config of NextGen/Microservices repositories + +Repositories structure +---------------------- + +nextgen/microservices +~~~~~~~~~~~~~~~ + +This repository will be an usual Python package, made from the OpenStack +cookiecutter template[1]. + +nextgen/ms-ext-config +~~~~~~~~~~~~~~~~~~~~~ + +This repository will be an usual Python package as well. + +nextgen/ms-* +~~~~~~~~~~~~ + +The structure of this repository will look like: + +* *docker/* + * *image.json.j2 / Dockerfile.j2* - template of appc definition or Dockerfile + (TBD), rendered and consumed by microservices CLI +* *config/* + * *.conf.j2* - template of config file + * *-some-other-config-file.conf.j2* - template of config file +* *service/* + * *.yaml.j2* - service definition for Kubernetes + +nextgen/dev-env +~~~~~~~~~~~~~~~ + +The structure of this repository will look like: + +* *Vagrantfile* - main Vagrantfile +* *Vagrantfile.custom.example* - example config file for Vagrant +* *provision.sh* - provisioning script + +Alternatives +------------ + +Kolla community is working on spec about running the kolla-kubernetes project.[2] +The only difference between this project and our planned Microservices project is +a single git repository for all Dockerfiles. + +Data model impact +----------------- + +There is no data model impact. + +REST API impact +--------------- + +There is no REST API impact. + +Security impact +--------------- + +There is no security impact. + +Notifications impact +-------------------- + +There is no notification impact. + +Other end user impact +--------------------- + +There shoudn't be impact on the end user. The experience which currently is +comming from kolla-build and kolla-mesos utilities shouldn't be changed from +user's point of view and we can even achieve a compatibility of configuration +and usage if we want. + +However, in addition to the backwards compatibility, we are going to provide +all needed tooling to be able to easy deploy and plug-in any set of services +just by using nextgen/microservies tool. + +Performance Impact +------------------ + +There will be two options regarding the service repositories. + +First option is to clone all needed repositories. That will decrease performance, +but not in critical way. + +Second option is to specify already cloned repositories. That will not change +anything in performance. + +Other deployer impact +--------------------- + +Deployer will have to be aware of using the multiple git repositories when +deploying NG OpenStack and Kubernetes infra. + +Developer impact +---------------- + +There will be an impact on developer's experience: + +* CI will have to use many repositories and check whether the deployment from + the current Code Review + the trunk of the other NextGen repositories is + working. +* In case of cross-repository failures, developers should care about making + appropriate dependencies by Depends-On clause in commits as well as handling + and contributing the changes in the other repositories if needed. + + +Implementation +============== + +Assignee(s) +----------- + +Primary assignee: + Michal Rostecki + +Work Items +---------- + +* Create the repositories for all OpenStack big tent services, infra services + around them and base layers in fuel-infra Gerrit. The name of these + repositories should follow the following pattern: "ms-". +* Move kolla-build tool from openstack/kolla repository to ng-builder repository. +* Move kolla-mesos tool from openstack/kolla-mesos repository to nextgen/microservices + repository. +* Make changes to build and mesos deployment tools to fetch the external git + repositories. +* Move the Dockerfiles from openstack/kolla repository to the appropriate + NextGen repositories. +* Move the configuration files from openstack/kolla and/or openstack/kolla-mesos + to the appropriate NextGen repositories. + + +Dependencies +============ + +* Specs about Docker build architecture [3] and layers [4]. + + +Testing +======= + +The testing scenarios for NextGen will be similar to what was planned +during our work on kolla-mesos. But of course it should include change +from Mesos to Kubernetes and repositories splitup. + +Testing scenarios for kolla-mesos were described on Mirantis wiki[5]. +We need to re-write them when implementing this bleuprint. + +However, there will be some infrastructure impact due to running tests +separately for each repo. + + +Documentation Impact +==================== + +Only parts of documentation describing the kolla-build and kolla-mesos CLI-s +can be preserved with the small changes. Every other part of documentation +should be rewritten from scratch. + + +References +========== + +* https://docs.google.com/document/d/1rtuINpfTvUFc1lJLuMK5LIzGKWwdh8uRUTvOLKnXuPM/edit +* [1] https://github.com/openstack-dev/cookiecutter +* [2] https://review.openstack.org/#/c/304182/ +* [3] https://review.fuel-infra.org/#/c/18867/ +* [4] https://review.fuel-infra.org/#/c/19028/ +* [5] https://mirantis.jira.com/wiki/display/KOLLA/CI+jobs