Distributed serialization is implemented with python distributed
library. We have scheduler for jobs management and workers for
jobs processing. Scheduler is started on the master node as well
as set of workers on it. Also workers are started on all nodes.
In the cluster settings we can select the type of serialization
and nodes statuses that allows serialization on it. By default
nodes with status 'ready' are excluded from the workers list.
For data serialization we are using only nodes from the cluster
where serialization is performing.
Before the computation fresh nailgun code is sent to the workers
as zip file and it will be imported for job execution. So we always
have fresh nailgun code on the workers.
In one job we are processing chunks of tasks on the workers. This
approach significantly boosts performance. The tasks chunk size
is defined as settings.LCM_DS_TASKS_PER_JOB parameter.
For limiting memory consumption on the master node we use parameter
settings.LCM_DS_NODE_LOAD_COEFF for calculation max number of jobs
in the processing queue.
Synthetic tests of distributed serialization for 500 nodes with
nubmer of ifaces >= 5 performed on 40 cores (4 different machines)
took 6-7 minutes on average.
Change-Id: Id8ff8fada2f1ab036775fc01c78d91befdda9ea2
Implements: blueprint distributed-serialization
Changed all JSON types fields of Release object on corresponding
Mutable(Dict/List) types.
Changed default value of 'vmware_attributes_metadata' field.
Change-Id: I024e199518526c1e60058c0ad25150ed4a543d26
Partial-Bug: 1482658
Update requirements in requirements.txt file.
Require the following modules only if python2.6:
* argparse
* ordereddict
Is NOT compatible with CentOS6 master node.
Use new oslo import notation.
Change-Id: Ib6a5be6ad784339ba4ee2f69a94a400ab8d79dfe
Blueprint: master-on-centos7
python2.7, which is default in CentOS7, provides
the following modules out of box:
* argparse
* orderecdict
There is no such packages on CentOS7 so we don't need to
declare them as requirements.
Blueprint: master-on-centos7
Compatible with CentOS6 master node
Change-Id: Ia1659a5c423698747929465f6f85074c5894bcf5
The thing is that this requirement setuptools==18.5
becomes a part of fuel-nailgun RPM package.
The highest available version in mirror.fuel-infra.org
at the moment is python-setuptools-0.7.2 which
leads to the error pkg_resources.VersionConflict when
trying to load nailgun entry points using for example
load_entry_point('nailgun==8.0.0', 'console_scripts', 'nailgun_syncdb')
Console script entry points are used during the Fuel
master node deployment.
Change-Id: I1e9b326912ef2e60da87f89af89d3a7968818276
Related-Bug: #1519727
Closes-Bug: #1519990
Yesterday the new versions of setuptools (18.6 and 18.6.1) were released
where some functionality such as installation in the editable mode was
broken. It will be fixed in the 18.6.2 release.
Change-Id: Id19879c788efbba6c6e72fe6a4a97418df124525
Partial-Bug: #1519727
The libraries we use from oslo-incubator project has been evolved into
separate libraries long time ago. Let's drop incubator, and use oslo.*
projects instead. It reduces our tech-debt, and introduce possibility
to fit upstream criteria.
Change-Id: Ib0bffc1d3e7c46f1028e06cda9bc0900acc26ee4
Closes-Bug: #1511427
Few hours ago the new version of oslo.utils (1.4.1) has been released [1].
It has incompatible requirements for oslo.i18n, so we have to pin it
temporary until we up oslo.i18n version.
[1]: http://git.openstack.org/cgit/openstack/oslo.utils/commit/?id=1.4.1
Closes-Bug: #1496751
Change-Id: Ic2f13d0aa4a8e3c9d1744401113db8238c9d01a2
Signed-off-by: Igor Kalnitsky <igor@kalnitsky.org>
Updated version of upper packages bounds to be more in line
with those from Kilo's global requirements.
This solved some of the problems with constantly breaking
releases of new OSt & oslo packages.
Unfortunately not all of them are in line still so we need pinning
for example inc ase of stevedore.
Change-Id: Iefa3628af5ae86414082b7100a671544b354e355
Partial-Bug: #1481212
Released a newest version of stevedore 1.7.0 (used by oslo.config)
requires pbr<2.0,>=1.3 that break requirements of pinned oslo.config.
The same thing happened with oslo.context.
Change-Id: I2113d929bd5556b13cf77cdc89976ad703977ac5
Closes-Bug: #1481212
Since 'oslo.config' and 'oslo.serialization' namespaces are deprecated, they
are swapped with 'oslo_config' and 'oslo_serialization' correspondingly.
Lower bounds of supported versions are updated as well.
Change-Id: I133c7676da504933d03e147af3a239d6559d5aac
Closes-Bug: #1478900
To resolve requirements conflicts in most painless way before 6.1 GA
let's ping keystonemiddleware to max version 2.0.0 and in 7.0 we
can remove upper bounds as well update packages in our mirros.
Change-Id: I98055060480a2ef80b9d56a211812e98e27fa2e1
Closes-Bug: #1466733
python-cinderclient > 1.2.1 require python-keystoneclinet 1.6 when we install
1.4 due to fix problem with pbr version caused by keystonemiddleware
We could require higher versiona of keystonemiddleware (>=1.5) but it
would be much higher than the version we have in repos (1.2) for 6.1
Change-Id: I449d23cf5e563b9de386beb249437bbf98db47f3
Closes-Bug: #1464132
Because some of the packages (python-cinderclient, keystonemiddleware)
do not release new versions with pbr requirements aligned with
OpenStack global requirements, we have a conflict in requirements
since new packages install pbr 1.0.1 (newest one) and "old" packages
want pbr<=1.0.
Because of that we limit higher version of explicitly installed packages
so we avoid this issue.
Change-Id: If90e0795cf8cebdeb381b17ad0a694b7b35f3d88
Closes-Bug: #1461980
OrderedDict will preserve order in which tasks are inserted
and given that fact that json array is structured, and
tasks found always in same order from directory -
we will guarantee that order of execution will not be changed,
even for tasks with same requirements
Change-Id: I5bc60b9754bb6a8e0d1318848d810603ec7b201e
Closes-Bug: 1434132
* added handler which allow to get a DOT representation of
deployment graph (full or selected tasks):
/api/v1/clusters/<id>/deploy_tasks/graph.gv
Colors for nodes in graph:
- groups - blue
- stages - red
- task that will not be executed - gray
- normal task - green
Available params:
- tasks - list of tasks that will be selected to make a deployment
- parents_for - show only predecessors/parents for given task
* added pydot-ng as a requirement
* in tox env always install newest package
Depends on: I706e18815f35904bf1f148edba818d281f2005b0
Change-Id: Id6fe85efe2549a63737ad50e5e55a70a480c83ab
Implements: blueprint granular-deployment-based-on-tasks
jsonutils is now in separate library, and the oslo.serialization will be
used from now on.
Change-Id: I28b6b70e437f154cf1b9ce0b15b0c3e716d170ee
Depends-On: https://review.openstack.org/#/c/158674/
Closes-Bug: #1400661
OSWL utilities updated. Code that supports collecting volumes info
added. This patch is dependent on [1].
[1]: https://review.openstack.org/#/c/156643/
Change-Id: Ifcb44bf990d46ce79cdcc67d533f598146710ec2
Implements: blueprint openstack-workload-statistics
The requirements list should be the same in both requirements.txt and
nailgun.spec. So change urllib3==1.7 to urllib3>=1.7.
Change-Id: I9e2da92c08f138da5ce9fcda362b2e2bfbe34fb9
Closes-Bug: #1412711
Historically dependencies between roles was hardcoded in nailgun
orchestration logic. To support granular roles and plugins
we need to be able to define it with configuration.
The easiest and most essential way to resolve dependencies
between roles, tasks and stages is to use graph model, which is
added in this patch.
RFC of API:
https://etherpad.openstack.org/p/rfc_granular_api
Required change in astute:
Change-Id: I28928e86ea4017288478703c6075b315b120349a
implements blueprint granular-deployment-based-on-tasks
Change-Id: Ie627cbe54cd88627ece3d5b559daf0475f460abb
For tests it's important to use `build_app` from app.py because it doesn't
use the `HTTPLoggerMiddleware`. This makes us possible to run tests without
write permissions in `/var/log/nailgun`.
The commit also removes the obsolete `paste` package from dependencies.
Change-Id: I401ef74abc95961d5687d12835752b881ef0e7ef
Closes-Bug: #1396685
Should be merged at once with relevant patches to:
* fuel-ostf
* fuel-astute
* fuel-main
Change-Id: Ic68983a8fb91c32d73408cd1f54439062175ee75
Closes-Bug: #1395279
urllib3.exceptions.ProxyError and urllib3.exceptions.DecodeError are
the urllib3 exceptions which are not handled by requests (old version)
so it is required to catch urllib3 exceptions separately.
This can be removed when requests==2.4.3 will be included to Fuel
(see https://bugs.launchpad.net/fuel/+bug/1394182).
This patch requires changes to fuel-main:
https://review.openstack.org/137343
Closes-Bug: #1390457
Change-Id: I30891796ea85289f81231c4cad3c802e33d21b03
POC for logic that implements collector of OpenStack installalation parameters
added. Now only number of instances inside of installation is gathered.
Collected information extends installation structure.
Is dependent on change in fuel-main repo [1].
[1]: https://review.openstack.org/#/c/135653/
Change-Id: I42f7021ccf8ef9fa22b9d5fdedf0445a26a7cfbe
Implements: blueprint send-anon-usage
There is also a patch to fuel-main which adds
python-requests to nailgun package spec.
fuel-main patch: I280de416af0079da1976ab37112dbf8764ef2839
Change-Id: I73db9cd7724291dc57904834d38dbe9afb371462
Closes-Bug: 1393859
keystonemiddleware package will be used instead of deprecated
keystoneclient.middleware module to authorize requests with
keystone auth_token
python-keystoneclient was left as a requirement to allow
usage of 'keystone' command and updated to new version
Change-Id: I525a4914dba31f7f0309900936ace01f9752866d
Implements: blueprint access-control-master-node-improvments
Related-Bug: #1383616
Depends: I90005ac4054cc585fd23c0e91cfcd51ef1ce6a9d
- Task serialization logic for pre/post hooks
- PluginSerializer object added
- API for interaction with Plugins db schema
General API requests supported
GET/POST /plugins
GET/PUT/DELETE /plugins/<plugin_id>
- Plugins and ClustersPlugins db models added
ClusterPlugins used to identify enabled plugins for a
given cluster
- Configuration uploading logic stored in Plugin wrapper
Co-authored-by: Dima Shulyak <dshulyak@mirantis.com>
Co-authored-by: Evgeniy L <eli@mirantis.com>
implements: blueprint cinder-neutron-plugins-in-fuel
Change-Id: I986f5be9ed3c3adaf7583d1bfc546cbe705db9ec
Due to the missing entry for 'six' in requirements.txt for nailgun
in some cases old version of six was used (from other packages) which
was causes issues with missing features in older version of six.
Now this dependency has been added.
Change-Id: Ib44dbd02e6d775ca269460d836d9a2c354513e3b
Closes-Bug: #1366728
Required to ensure that no system level dependencies will be
added to keystoneclient, which can break our ci
Related-Bug: #1347583
Change-Id: I1baed0598ae730f5a2a14b8dc9cd331aea37d34c
Andreas Jaeger