This patch adds support to deploy Ironic with separate Neutron network
for provisioning baremetal instances.
* Add NetworkDeploymentSerializer100 as Ironic multitenancy is supported
from Newton, and we will backport this till stable/newton.
* Update network scheme generation to create 'vlan' baremetal network,
assign IPs to Ironic conductors from this network, make them
accessible from baremetal servers.
* Add new checkbox at 'Openstack Settings/Additional components' tab
which allows to define if separate provisioning network should be used
during deployment. This is a trigger to switch ironic deployment to
multitenancy case. If not selected old behaviour is kept, 'flat'
network is used. The checkbox is shown only when Ironic component is
enabled.
Change-Id: I861a8b3b046202526d6a856c9f2dca2cfaddc887
Related-Bug: #1588380
Distributed serialization is implemented with python distributed
library. We have scheduler for jobs management and workers for
jobs processing. Scheduler is started on the master node as well
as set of workers on it. Also workers are started on all nodes.
In the cluster settings we can select the type of serialization
and nodes statuses that allows serialization on it. By default
nodes with status 'ready' are excluded from the workers list.
For data serialization we are using only nodes from the cluster
where serialization is performing.
Before the computation fresh nailgun code is sent to the workers
as zip file and it will be imported for job execution. So we always
have fresh nailgun code on the workers.
In one job we are processing chunks of tasks on the workers. This
approach significantly boosts performance. The tasks chunk size
is defined as settings.LCM_DS_TASKS_PER_JOB parameter.
For limiting memory consumption on the master node we use parameter
settings.LCM_DS_NODE_LOAD_COEFF for calculation max number of jobs
in the processing queue.
Synthetic tests of distributed serialization for 500 nodes with
nubmer of ifaces >= 5 performed on 40 cores (4 different machines)
took 6-7 minutes on average.
Change-Id: Id8ff8fada2f1ab036775fc01c78d91befdda9ea2
Implements: blueprint distributed-serialization
On NIC: XL710 with driver 'i40e' MTU does not take into
account 4 bytes VLAN tag, so we should increase it manually
DocImpact
Change-Id: I3d95db9ec6fae4d8cd397c429d785dbdf1502b21
Partial-Bug: #1587310
Co-Authored-By: Fedor Zhadaev <fzhadaev@mirantis.com>
* min value was set in consts
* appropriate validator was added
* tests for validator were changed
* test for serializer were changed
Change-Id: Ib8ccb0658bd401ce492257f855013d1d7e0f2dac
Closes-Bug: #1653081
* reset_environment supertask contains 3 subtasks:
base_reset_environment, remove_keys_task,
remove_ironic_bootstrap_task
* names for tasks were changed
* response methods for remove_keys_task and
remove_ironic_bootstrap_task were added to receiver
* _restore_pending_changes method was add only for
reset_environment_resp
* migration for adding new transaction names and appropriate
test were added
* test for check task message was added
Change-Id: Ib8a215174431486316bca533797932e02969c037
Closes-Bug: #1541868
Remove restrictions on set of BOND properties. Now BOND can have
any attributes.
Remove redundant `bond_properties`.
Change-Id: I60d1a0628d84c5ba49bb5b45824d660297dacccc
Implements: blueprint nics-and-nodes-attributes-via-plugin
There are 3 new orchestration tasks:
* master_shell
Run task on master node with a context of other node. If 'roles'
selects N nodes, the task will be executed N times.
* erase_node
Erase node. It's necessary task if we want to remove nodes by means
of graphs and not pre-hardcoded actions in Astute.
* move_to_bootstrap
Change node's PXE config to boot via LAN (into bootstrap). As a
previous one, it's necessary task for deletion graph.
Change-Id: Ie8f852762b837a68e0e0b49e11653a8f2e56a014
Blueprint: graph-concept-extension
dkms packages non needed because Xenial provides newest drivers.
nonfree package was removed from Xenial.
Change-Id: Ieff11865074865a28e6f78ac927200aea40c3438
blueprint: mos-xenial
NFV features (DPDK, SR-IOV, NUMA/CPU pinning, HugePages)
can't be checked for old clusters, due to old nailgun-agent.
Old nailgun-agent doesn't send NFV specific information.
So all NFV related checks and functional should be disabled for old
environments.
Change-Id: Ib589d67658f45414b8049398316af5c7298d459e
Closes-Bug: #1594443
CPU distribution mechanism should be changed due
to incorect requirements to nova and dpdk CPUs allocation
Changes:
* Change CPU distribution
* Add function for recognizing DPDK NICs for node
* Remove requirement of enabled hugepages for
DPDK NICs (it's checked before deployment)
* Change HugePages distribution. Now it take into
account Nova CPUs placement
Requirements Before:
DPDK's CPUs should be located on the same NUMAs as
Nova CPUs
Requirements Now:
1. DPDK component CPU pinning has two parts:
* OVS pmd core CPUs - These CPUs must be placed on the
NUMAs where DPDK NIC is located. Since DPDK NIC can
handle about 12 Mpps/s and 1 CPU can handle about
3 Mpps/s there is no necessity to place more than
4 CPUs per NIC. Let's name all remained CPUs as
additional CPUs.
* OVS Core CPUs - 1 CPU is enough and that CPU should
be taken from any NUMA where at least 1 OVS pmd core
CPU is located
2. To improve Nova and DPDK performance, all additional CPUs
should be distributed along with Nova's CPUs as
OVS pmd core CPUs.
Change-Id: Ib2adf39c36b2e1536bb02b07fd8b5af50e3744b2
Closes-Bug: #1584006
Now following handlers:
/clusters/:cluster_id/changes/
/clusters/:cluster_id/changes/redeploy/
/clusters/:cluster_id/deploy/
/clusters/:cluster_id/deploy_tasks/
?dry_run=1 that is telling Astute not to run cluster executionat all.
Dry run assumes that it does not actually affect
cluster status regardless of their result.
Also, remove redundant update of nodes statuses to 'deploying'
within OpenStackConfigManager and DeploymentTaskManager as it should be
done by receiever.
Do not set cluster status to 'deployment' for these nodes in order to
retain its real status
Modify stop deployment tests to move failing stop deployment for already
deployed clusters to another test class. Since 9.0 we can run stop
deployment for new clusters.
Change-Id: I374fc86b63af64411d4a5ca45ff6c3680cb44897
Partial-bug: #1569839
to allow clean separation between OVS and DPDK bonds so that
both can be used in Fuel 9.0 and later without confusion.
Change-Id: If1a9eae087517d841c20f1e425734a44547ea2c5
Closes-Bug: 1572944
We need separate task serializers for LCM,
because LCM uses context per node for serialize tasks.
It also allows to isolate LCM related code for backward
compatibility with existing environments, that is not ready for LCM.
Change-Id: Ie95a58c8cf86eac1a5c3dbd956fafc401e40fed6
Implements: blueprint computable-task-fields-yaql
API url: `/api/transactions/<id>/deployment_history`
This change proposes to store all the information about
particular deployment tasks ever executed for each
particular cluster.
Change-Id: I73010a713ab8592418eb59bb133a427ac4c4a665
Implements: blueprint store-deployment-tasks-history
* Add ovs bonding modes into openstack.yaml with appropriate
parameters. It's needed for configuring DPDK bonds on UI.
* Add new bonding mode and 'lacp' parameter for generating correct
transformations for OVS bonds.
Implements blueprint support-dpdk
Change-Id: I54a9a413ef9ab410765884ff3d74d8a205023fb3
* Add list of supported DPDK drivers to openstack.yaml
* Add correct transformations for enabling DPDK
* Add handling of DPDK data received from nailgun-agent
Implements blueprint support-dpdk
Change-Id: I13c3b45fbc4fc43efbb9fca0ee994a8720138b38
Validator checks that node have enough memory to allocate all required
hugepages
Change-Id: I6b4e963548a673d3a86d3b4cc06529701c8d0a85
Implements: blueprint support-hugepages
Depends-On: Id5898f0b3e193fbd6267975d0ec70ef72cbce789
Constants is removed to avoid difference in 9.0 migrations and models work
if constants values will be changed in the next updates.
Change-Id: Ibc8d846fef787e4d93364b22c4f626b01a6d7d71
Implements: blueprint custom-graph-execution
This models store deployment tasks sets that is grouped under
DeploymentGraph record with unique id.
DeploymentGraph is linked via m2m to the:
- releases
- plugins
- clusters
This transitive tables containing 'type' field, that should be used as
graphs merge key in business logic.
DocImpact
Implements: blueprint custom-graph-execution
Change-Id: If53cb58e8ae4cf8fe4caa470bf207ff1683b1cda
Introduce new node status: stopped, which used for nodes,
where orchestrator was successfully gracefully stopped task
deployment. This status will be able node redeploy.
Also introduced new type of node error_type: stop_deployment
It will be used if gracefully stop on node will be failed
for example, because of timeout. It will be apply automatically
by recevier for nodes in deploying status
if orchestrator return error for stop deployment task
Change-Id: I30d606d7d11d670d1a68ee90b01b932c1543fccc
Implements: blueprint graceful-stop-restart-deployment
Added new status for cluster:
* partially_deployed - the nodes of cluster has different statuses.
Change-Id: I5be8c4ed5f58e2c6545a7f06b8eab1ae37c8575f
Closes-Bug: 1542201
In Fuel 5.1 we had an experimental feature - 'patching openstack env'.
The idea was to update and to rollback OpenStack environments between
minor releases. However, we have encounter a lot of problems with
restoring OpenStack databases andresolving dependency hell in packages,
so we buried it and never release it.
This patch removes legacy code from the source tree. We can do it
without fear, since it wasn't released in public.
Related-Bug: #1511499
Change-Id: I58b3fedd239eb7fe4226e51c2d6386efab14395d
* Redesigned a role list view on Add Nodes/Edit Roles screens in Fuel UI
to take up less space on the screen
* Also the role list is grouped by a new 'group' role attribute
Implements: blueprint redesign-of-node-roles-panel
Change-Id: Ie99e2b911439ae050a5212febe0fac7502550ea9
Checking repositories on availability is long task.
Http request can timeouts, while the task running.
Decided to move checking from http request context to
astute task, which running asynchronously.
Depends-On: I76ed1f0b6413cc94c539d795bbba92f8e9dad50b
Depends-On: I0ccf42fd7ff05d68aa4a4aee426c1c8bfa4daea4
Closes-Bug: #1540346
Change-Id: Iff76af0b0a6573f0f8f779fda4333a3658cccc04
* Add 'is_user_defined' field to 'ip_addrs' table.
* Rename 'vip_type' field to 'vip_name' of table 'ip_addrs'.
* Add 'vip_namespace' field to 'ip_addrs' table.
* Copy vip namespaces from plugin table network roles to 'vip_namespace' field
according to the unique vip name.
* Add database migrations.
* Add unit test for migrations.
Change-Id: Ia3e1d7f6e08dbebcb182de75eeaf58ddf6be4a8d
Partial-bug: #1482399
Plugins v5.0.0 use task-based deployment directives using tasks
v2.0.0 with cross-dependencies and per-task strategy.
V4.0.0 allows both experimental and granular deployment directives
(tasks v1.0.0 and tasks v2.0.0) and using them according to `task_deploy`
environment flag setting.
Change-Id: I697dc22a58a9ff2ad8bd9d955d2b2c5351c746ab
Closes-Bug: #1533771
- The plugin deployment tasks are proceeded with the core-tasks.
- The plugin pre-deployment tasks will be added as
chain between pre_deployment_end and deploy_start.
- The plugin post-deployment tasks will be added as
chain after post_deployment_end.
Closes-Bug: #1527325
Change-Id: I5b416d9507b2ae1503c356a9828a0b7be85c1eb1
- Moved all Task Role related constants to Enum TASK_ROLES
- Renamed consts.MASTER_ROLE to consts.MASTER_NODE_UID
Change-Id: I292f0f5471e295564a318530f5397e4c438cb40a
Closed-Bug: #1526688
NullResolver the fake resolver
PatternBasedRoleResolver allows to use pattern in name of role
implements blueprint: task-based-deployment-astute
Change-Id: I5bfb135fe95ed8faee6df81e31748e0143c568e6
This is required to apply changes that could be made to the networking
configuration of environment.
The major case is: add/remove node network group(s).
The configuration changes are not tracked as nailgun does not have
proper functionality for that.
New key is introduced in task description: reexecute_on (list).
Deployment tasks which are to be re-executed on deploy changes must have
reexecute_on:
- deploy_changes
DocImpact
Change-Id: I668fcb426826cecf69a5c7fc1e92b8ae8dfeef9f
Blueprint: redeploy-network-on-changes
Get rid of component object.
Store info with compatibility and uncompatiblity
lists of components in plugin and release.
Change-Id: I47ed959f9079c346d9d968e7ebd9bf1b7b150e2f
Implements: blueprint component-registry
This change adds:
* new optional baremetal network
* baremetal sections in Neutron L3 configuration
Ironic requires separate baremetal network on infra nodes and neutron flat
network mapped to this physical network.
New baremetal network is optional and depends on Ironic component enabled in
settings.
Implements: blueprint fuel-integrate-ironic (partially)
Change-Id: I742bf69d3f3ac5e4b4ecd39a12e3171c566563e8
At the moment feature_groups are defined during
ISO build in version.yaml file. We need to avoid
using build time defined parameters because
we are moving to the package based delivery approach.
Depends-On: I05e841d4279bc61645cefb68f411a93b67e57189
Related-Bug: #1513737
Related-Bug: #1515517
Change-Id: I4f96fec73f991094bca65de5f1d1014662fbb2b4
Andreas Jaeger