This patch adds support to deploy Ironic with separate Neutron network
for provisioning baremetal instances.
* Add NetworkDeploymentSerializer100 as Ironic multitenancy is supported
from Newton, and we will backport this till stable/newton.
* Update network scheme generation to create 'vlan' baremetal network,
assign IPs to Ironic conductors from this network, make them
accessible from baremetal servers.
* Add new checkbox at 'Openstack Settings/Additional components' tab
which allows to define if separate provisioning network should be used
during deployment. This is a trigger to switch ironic deployment to
multitenancy case. If not selected old behaviour is kept, 'flat'
network is used. The checkbox is shown only when Ironic component is
enabled.
Change-Id: I861a8b3b046202526d6a856c9f2dca2cfaddc887
Related-Bug: #1588380
Distributed serialization is implemented with python distributed
library. We have scheduler for jobs management and workers for
jobs processing. Scheduler is started on the master node as well
as set of workers on it. Also workers are started on all nodes.
In the cluster settings we can select the type of serialization
and nodes statuses that allows serialization on it. By default
nodes with status 'ready' are excluded from the workers list.
For data serialization we are using only nodes from the cluster
where serialization is performing.
Before the computation fresh nailgun code is sent to the workers
as zip file and it will be imported for job execution. So we always
have fresh nailgun code on the workers.
In one job we are processing chunks of tasks on the workers. This
approach significantly boosts performance. The tasks chunk size
is defined as settings.LCM_DS_TASKS_PER_JOB parameter.
For limiting memory consumption on the master node we use parameter
settings.LCM_DS_NODE_LOAD_COEFF for calculation max number of jobs
in the processing queue.
Synthetic tests of distributed serialization for 500 nodes with
nubmer of ifaces >= 5 performed on 40 cores (4 different machines)
took 6-7 minutes on average.
Change-Id: Id8ff8fada2f1ab036775fc01c78d91befdda9ea2
Implements: blueprint distributed-serialization
This is the first step for handling arbitrary data structures
in OpenstackConfig. At this moment we are handling explicit
serialization of the configuration_options.
The future steps are:
- changing configuration data format in the Fuel CLI
- data migration in the Fuel DB
- changing configuration data serialization in Fuel
The future steps are planned to be done for the Fuel 10 release
Change-Id: I6961e3d7a2bc5ae8b9c668c67a0ab5c23792f5b1
Co-Authored-By: Alexander Kislitsky <akislitsky@mirantis.com>
Add ability to override generated values for 'ovs_core_mask'
and 'ovs_pmd_core_mask'.
Change-Id: Ifdc3b45bf4ec5bc3da31966c17be6fd18fc97b39
Closes-bug: #1648371
As a part of cluster serialization primary tags on nodes are updated.
This results in updated nodes record to be locked until the transaction
where serialization happens ends. When there is a lot of nodes this
transaction can be very long and, as a result, a lot of node updates by
nailgun agent can be blocked until it ends. At some points this results
in all workers being busy blocking any REST call.
Change-Id: Ie5341290097bddce299bf5726e2607d150c23768
Closes-Bug: #1643024
In order to conform to library changes switch
bridge provider from ovs to dpdkovs.
Related-Bug: #1638036
Depends-On: I249b15b799da3b649c5843f71d8eabf8b460f510
Change-Id: I8296cfafa7322babd3831724b88aa253bdb13fec
If there is no specified hugepages or CPU pinning
distributors shouldn't be called. Also changed
initialization of custom_hugepages type.
Change-Id: Iedb819b1da7dcb3877a6a94b9e7cfb93aa949a9e
Closes-Bug: #1594443
(cherry picked from commit 661ce479a6)
This commit switching tasks resolution approach to the tags based one.
Tag - minimal unit what's necessary only for task resolution and can be
mapped to the node through the role interface only. Each role provides set
of tags in its 'tags' field and may be modified via role API. Tag may be
created separately via tag API, but, this tag can not be used unless it's
stuck to the role.
Change-Id: Icd78fd124997c8aafb07964eeb8e0f7dbb1b1cd2
Implements: blueprint role-decomposition
Provide ability for plugins to add additional attributes for nodes.
* Added new NodeAttributesDefaultsHandler handler for getting
default node attributes.
* Added code stuff for getting and updating additional plugin
node attributes.
* Added code stuff to serialize plugin node attributes for orchestrator.
Change-Id: I5c6d6f3c815a034683e624b98fa1f360ab4b491c
Implements: blueprint nics-and-nodes-attributes-via-plugin
A 'tags' attribute has been added to each role in 'roles_metadata'.
Initially all non-controller roles will only have a tag of their own
role name. This will allow existing tasks which do not have tags
associated with them to work correctly. In the abscence of tags a
task's roles will be used to determine which nodes it will run on.
Implements: blueprint role-decomposition
Change-Id: I390580146048b6e00ec5c42d0adf995a4cff9167
Provision info already contains list of packages as
string, but fa_builder expects list and conversion from
string to list was done by using yaql expression.
This patch moves this conversion to nailgun side.
Change-Id: I1ad44b7f2e69edbb8b8e28fd7840010af2594cd7
Closes-Bug: 1630275
Added option skip_extensions for core-serializers, to prevent
loop when core-serializer is called from extension.
Also method process_'deployment_for_node' does not have access
to cluster_info and it is not possible to call 'serialize_node'
from extension because cluster_info was mandatory argument.
Because 'cluster_info' was needed only for provision serialzer,
which is called from LCM serializer, this argument was removed
from serialze_node method.
Change-Id: Ifbf9e4a0b9487605e6268fde0499aecc1e20cb36
Closes-Bug: 1621851
The size of deployment_info grows as n^2 depending on
nodes number. That's because common_attrs, which is
merged into each node's contains info about all nodes.
For example for 600 nodes we store about 1Gb of data in
the database. So as first step let's store common_attrs
separately in deployment_info structure inside python
code and in the database.
Also removed old test for migrations, which are not related
to actual database state.
Change-Id: I431062b3f9c8dedd407570729166072b780dc59a
Partial-Bug: #1596987
Reduced cluster and release attributes which are added to
deployment_info on serialization
Change-Id: I98a960ad4b4c1ac12f25059e9b429fdca97cffb4
Closes-Bug: 1618002
Now it is possible:
* To bring release as plugin
* To define multiple graphs types for the plugin
Change-Id: Idcac14e6f4055ca0e488efc2cc23945036a493d9
Implements-Blueprint: release-as-a-plugin
Because there is no difference between provision graph
and deployment graph from nailgun side,
the graph execution context should contain both
provison info and deployment info. This patch
extends LCM deployment info serializer to include
provison info in serializer output.
Change-Id: I672e8fde1a662bd190dcbda0869c290ff110e2d2
Blueprint: graph-concept-extension
Lack of testing was found, it's impossible to specify hugepages if
host supports only 2M huge pages due to incorrect validation.
This reverts commit 9bd61c68dc.
Change-Id: I7945d165764608ca7ee8ac5f46463267718678f5
If there is no specified hugepages or CPU pinning
distributors shouldn't be called. Also changed
initialization of custom_hugepages type.
Change-Id: I3835c9d163ba2692adf34193b57510060158b8e3
Closes-Bug: #1594443
Instead of wide callbacks process_deployment or process_provision
implemented methods to patch deployemnt or provision info per
cluster or node.
The code of extensions and tests was updated accordingly.
Also added helper to mark methods of extension as deprecated.
the extension load behaviour was modified, instead of fail operation
when extension cannot be loaded, the nailgun only write error in log
that extension is not loaded and continue operation.
Partial-Bug: 1596987
Change-Id: I577c8ffc105734e12646ca7c6a4fe4927e70b119
DocImpact
CPU distribution mechanism should be changed due
to incorect requirements to nova and dpdk CPUs allocation
Changes:
* Change CPU distribution
* Add function for recognizing DPDK NICs for node
* Remove requirement of enabled hugepages for
DPDK NICs (it's checked before deployment)
* Change HugePages distribution. Now it take into
account Nova CPUs placement
Requirements Before:
DPDK's CPUs should be located on the same NUMAs as
Nova CPUs
Requirements Now:
1. DPDK component CPU pinning has two parts:
* OVS pmd core CPUs - These CPUs must be placed on the
NUMAs where DPDK NIC is located. Since DPDK NIC can
handle about 12 Mpps/s and 1 CPU can handle about
3 Mpps/s there is no necessity to place more than
4 CPUs per NIC. Let's name all remained CPUs as
additional CPUs.
* OVS Core CPUs - 1 CPU is enough and that CPU should
be taken from any NUMA where at least 1 OVS pmd core
CPU is located
2. To improve Nova and DPDK performance, all additional CPUs
should be distributed along with Nova's CPUs as
OVS pmd core CPUs.
Change-Id: Ib2adf39c36b2e1536bb02b07fd8b5af50e3744b2
Closes-Bug: #1584006
This patch:
* extends current DB model with new entities and provides
related migrations.
* extends plugin sync method to support storing new plugins
attributes.
* provides cosmetic fix for ClusterPlugin model. Lets write table
names in the plural but model names in the singular.
Change-Id: I3edbde1d48461ce3fab7c93f17e2db5332b1f7fb
Implements: blueprint nics-and-nodes-attributes-via-plugin
'nodes' data is deprecated so must be removed from 10.0 version
serializer output. The change was moved to separate serializer class in
order to maintain compatibility with previously deployed 9.0 clusters
after upgrade to 10.0.
Change-Id: I8fef591908cd8b1e4ace5502bb1e80cb37786c6e
Related-Bug: #1531128
Roles that are passed via parameter to function
inject_configs retrieved from `objects.Node.all_roles`
function. This function adds prefix `primary-` to
generic role names.
In this case configuration, that was created for role
`controller`, won't be executed on `primary-controller`
node, since role `controller` was replaced with
`primary-controller`
Change-Id: Ibba3a1d25079bd7b356e8e65da11c19a8017d4ba
Closes-Bug: #1576127
Calls to network manager from object methods have been
moved into callbacks implemented by the network manager
extension.
This commit makes the following new callbacks available
to extensions:
* on_cluster_create: called when a new cluster is created
* on_cluster_patch_attributes: called when a cluster's attributes
are updated
* on_nodegroup_create: called when a node group is created
* on_before_deployment_serialization: called before deployment
serialization begins
* on_before_provisioning_serialization: called before provisioning
serialization begins
* on_remove_node_from_cluster: called when a node is removed from
a cluster
Blueprint: network-manager-extension
Change-Id: I9a3413f54c881edd098e623ea204d12a86695f87
Now serialized deployment info for environments >= 9.0 does not contain
'nodes' field; it is removed from the output of the serializer.
Unit and integration tests are updated. Deployment tasks fixture
changed: 'nodes' references in yaql expressions changed to
'network_metadata.nodes'
Change-Id: Ibdb2cf85732c0293314298e6a342b071ece18cc3
Closes-Bug: #1531128
Since node can be customized with LCM
deployment info, it's not needed to
convert old serialization style to
new.
Change-Id: I4822745849925076cbcedbd028c9e37d91d1aa1c
Closes-Bug: #1572965
This change updates the deployment serializer for the test vm data to
pass the glance properties as a hash that can be used by the
glance_image provider rather than using the glance_properties string
that is currently in place. The glance_properties string should be
considered deprecated and anything that uses it should switch to the
properties hash.
DocImpact: glance_properties string provided as part of the
test_vm_image hiera data is deprecated in favor of the properties hash
provided by this change
Change-Id: I79a9b20d89ae00a7ceaa24c4ce655cbd16972c30
Partial-Bug: #1566434
* remove node_extension_call from everywhere Nailgun
core source code
* remove volume_manager Node property (models)
* moved volume and disk releated data manipulations to
volume manager extension pipeline
* removed no longer valid tests
* added new extension callback for pre deployment check
* fix some tests
* moved volume_manger specific tests to volume_manger module
* marked 'skip test' to some tests which are no longer
valid in current places but they valuable and should be moved
to volume_manager module in next patches
implements: blueprint data-pipeline
Change-Id: I8edd25166e5eccf914eb92882b6b4a7b3fff6a89
The following legacy tasks were reworked to use ClusterTransaction:
- OpenstackConfigTaskManager
- SpawnVMsTaskManager
Change-Id: I4a6f5f37161e4290050ec4926cf029cd7af566e4
Closes-Bug: 1565885
Closes-Bug: 1561994
Closes-Bug: 1565760
Using permanent identifier instead of hostname based identifier fixes
problems which might be caused by renaming a host
Change-Id: I17d84da97296144acbf5983ea18139ffdee0db8a
Closes-bug: #1548332
Added information about plugins scripts and plugins repositories.
This information will be used by puppet to create repositories and
synchronise plugins scripts.
Change-Id: Ie2df470a4f05b73162144f9c87d1023760b282c7
Closes-Bug: 1565061
Andreas Jaeger