This commit switching tasks resolution approach to the tags based one.
Tag - minimal unit what's necessary only for task resolution and can be
mapped to the node through the role interface only. Each role provides set
of tags in its 'tags' field and may be modified via role API. Tag may be
created separately via tag API, but, this tag can not be used unless it's
stuck to the role.
Change-Id: Icd78fd124997c8aafb07964eeb8e0f7dbb1b1cd2
Implements: blueprint role-decomposition
A 'tags' attribute has been added to each role in 'roles_metadata'.
Initially all non-controller roles will only have a tag of their own
role name. This will allow existing tasks which do not have tags
associated with them to work correctly. In the abscence of tags a
task's roles will be used to determine which nodes it will run on.
Implements: blueprint role-decomposition
Change-Id: I390580146048b6e00ec5c42d0adf995a4cff9167
Added option skip_extensions for core-serializers, to prevent
loop when core-serializer is called from extension.
Also method process_'deployment_for_node' does not have access
to cluster_info and it is not possible to call 'serialize_node'
from extension because cluster_info was mandatory argument.
Because 'cluster_info' was needed only for provision serialzer,
which is called from LCM serializer, this argument was removed
from serialze_node method.
Change-Id: Ifbf9e4a0b9487605e6268fde0499aecc1e20cb36
Closes-Bug: 1621851
The size of deployment_info grows as n^2 depending on
nodes number. That's because common_attrs, which is
merged into each node's contains info about all nodes.
For example for 600 nodes we store about 1Gb of data in
the database. So as first step let's store common_attrs
separately in deployment_info structure inside python
code and in the database.
Also removed old test for migrations, which are not related
to actual database state.
Change-Id: I431062b3f9c8dedd407570729166072b780dc59a
Partial-Bug: #1596987
Instead of wide callbacks process_deployment or process_provision
implemented methods to patch deployemnt or provision info per
cluster or node.
The code of extensions and tests was updated accordingly.
Also added helper to mark methods of extension as deprecated.
the extension load behaviour was modified, instead of fail operation
when extension cannot be loaded, the nailgun only write error in log
that extension is not loaded and continue operation.
Partial-Bug: 1596987
Change-Id: I577c8ffc105734e12646ca7c6a4fe4927e70b119
DocImpact
Since self.env.create always return db object now,
we can use this returned value instead of
self.env.clusters list.
It's a refactoring, so no bug or blueprint.
Change-Id: If7c84cb7124bcf08ef5ff110542012564190fae1
In default setup we should not assign public ip to all nodes.
Just to ones with controller role. So we filter out public ip
network for nodes that should not have it.
Change-Id: I2a9ea4d06cc1ba15bad20b817659b7539827472a
Closes-Bug: 1415552
This moves the files for NetworkManager and its sub-classes into
a new extension. All import paths have been updated.
Blueprint: network-manager-extension
Change-Id: Icc2410fd9c411a47a3dee4573d4ef6f1a039c303
Avoid reading `tasks.yaml` every time
Fix missing of `environment.yaml` file sync for plugins V(1,2)
Change-Id: Id3aed7b943b6942c223797a40d3d0214834ba31f
Closes-bug: #1527320
'netgroups[ngname]' is replaced with 'netgroups.get(ngname, {})' as
in other places in serializer.
Problem was the 'br-prv' endpoint that didn't have IP in case of VLAN
segmentation. The difference between 'netgroup_mapping' and 'netgroups'
exists only in case of using network template. Otherwise, absence of IP
for 'br-prv' did not lead to this issue.
Also, there is a small refactoring of serialization tests was made here.
Change-Id: I67616409017631023e707e533cb8e2faab943b34
Closes-Bug: #1556917
To avoid confusion with objest.DeploymentGraph and
orchestrator.DeploymentGraph second one was renamed
to the GraphSolver because this entity is used to
find subgraphs, loops, and solve graph for the
old orchestrators.
Secondary changes:
DeploymentGraphValidator -> GraphSolverValidator
DeploymentGraphTasksValidator -> GraphSolverTasksValidator
GraphVisualizationValidator -> GraphSolverVisualizationValidator
validators/graph.py -> validators/orchestrator_graph.py
orchestrator/deployment_graph.py -> orchestrator/orchestrator_graph.py
Change-Id: Ic42a17115723b8145b9d94cc034bf2217d1d634b
Implements: blueprint custom-graph-execution
It contains a gateway for described network.
Closes-bug: #1549034
Co-Author: Vitalii Myhal <vm198r@att.com>
Change-Id: I1550e126b16447f83f743601dab23677a5eacd52
Now VIP info is build for VIPs that are not allocated, that is not
present in data base. For such VIPs 'ipaddr' field will contain None
value.
'is_user_defined' flag is included in the serialized info.
Now logically VIPs may be divided into auto allocated and set manually
by user by combinations of 'ipaddr' value and 'is_user_defined' flag.
Change-Id: I36bb90dcc0d1e14da113ba512008d9b9eb30ba4c
Blueprint: allow-any-vip
Currently, astute supports 'retries' parameter, but this
parameter is not processed by nailgun, so, we always get
default pupet retries value from astute component.
Change-Id: I68615b053ebc1c07bb8ca85084e48a9c93f5e495
Related-Bug: #1457794
For the time being we used custom non-cached SQLAlchemy query. That
means the following drawbacks:
* each time we access to relationship attribute the SQL query is spawn;
* each time we trying to get instance by pk (.get) the SQL query is
spawn;
Besides the way we get non-cached query is by patching internal
SQLAlchemy's Query attribute - '_populate_existing' - which is designed
for another purpose. It cause us a lot of troubles because the
SQLAlchemy behaves in slightley different way that it's pointed in
documentation (for instance, there're much more reasons for implicitly
calling .flush()).
This commit finally drops this unconvenient NoCacheQuery in favor of
standard Query.
Change-Id: Ic1b11684dddf970268eedd297e8577c86e750f1e
Closes-Bug: #1498382
The ID was sent only when it is possible, without any guarantees.
Because some of tasks do not have ID by its nature. for example
plugins pre/post deployment tasks.
Change-Id: Ib0e0e334e99140eccedb63114bb1377c5f07a9fb
Closes-Bug: #1527596
- Moved all Task Role related constants to Enum TASK_ROLES
- Renamed consts.MASTER_ROLE to consts.MASTER_NODE_UID
Change-Id: I292f0f5471e295564a318530f5397e4c438cb40a
Closed-Bug: #1526688
This is required to apply changes that could be made to the networking
configuration of environment.
The major case is: add/remove node network group(s).
The configuration changes are not tracked as nailgun does not have
proper functionality for that.
New key is introduced in task description: reexecute_on (list).
Deployment tasks which are to be re-executed on deploy changes must have
reexecute_on:
- deploy_changes
DocImpact
Change-Id: I668fcb426826cecf69a5c7fc1e92b8ae8dfeef9f
Blueprint: redeploy-network-on-changes
This change introduces:
* NovaNetworkManager80
* ironic/api network role
* ironic/baremetal network role with VIP and iptables rules
Added separate fixture for network template.
Tests updated.
Partially Implements: blueprint fuel-integrate-ironic
Change-Id: Ib867f843951f7b86be31a9ffe4cf54c56127157b
The patching mostly done for tests for nailgun components adhering to
the release version. E.g. network serializers. Some tests are forced to
create environments of 8.0 version. The change is needed to resolve
possible issues with the tests when network roles metadata is changed
drastically and is not compatible with mentioned components.
Partial-Bug: #1517874
Change-Id: I55607157ae7767ffdfd1d855b872630832706e8e
The network related function shall be in NetworkManager,
this makes easier further refactoring.
Change-Id: I8eb5404509c9fe0a8c367ea119bfb18233941fde
Partial-Bug: #1354803
In PostgreSQL there are special types for storing IPv4, IPv6 address and
subnetworks and also MAC addresses. Storing data in these types is more
efficient than in strings because it requires less space. In addition,
it provides additional operations for columns of these types.
This patch changes types of some columns for models such as:
IPAddr
ip_addr: inet
IPAddrRange
first: inet
last: inet
Node
mac: macaddr
ip: inet
NodeNICInterface
mac: macaddr
ip_addr: inet
netmask: inet
NodeNICInterface
mac: macaddr
NetworkGroup
cidr: cidr
gateway: inet
NeutronConfig
base_mac: macaddr
internal_cidr: cidr
internal_gateway: inet
baremetal_gateway: inet
NovaNetworkConfig
fixed_networks_cidr: cidr
Co-Authored-By: Ilya Kharin <ikharin@mirantis.com>
Change-Id: Id8431eb7d2e3ca1a1cc29fc865d57359b29d0e35
Partial-Bug: #1484973
VIPs from network roles with mapping to non-existing network should be skipped
from assignment.
Added NeutronManager80 and AllocateVIPs80Mixin
Change-Id: Id8dd1d26872edd1e8055de5ebcaa2d7dd9a955f7
Implements: blueprint fuel-integrate-ironic (partially)
This change adds:
* new optional baremetal network
* baremetal sections in Neutron L3 configuration
Ironic requires separate baremetal network on infra nodes and neutron flat
network mapped to this physical network.
New baremetal network is optional and depends on Ironic component enabled in
settings.
Implements: blueprint fuel-integrate-ironic (partially)
Change-Id: I742bf69d3f3ac5e4b4ecd39a12e3171c566563e8
This is redesign of plugins architecture in order to store
the plugin's attributes in a separate table, not in cluster
attributes, so it will be possible to remove connection between
plugin and cluster when a plugin gets deleted.
Added ability to work with different versions of a plugin.
User can choose the preferred version in UI.
The test "test_plugin_generator" was removed because no longer
relevant.
Closes-Bug: #1440046
Implements: blueprint store-plugins-attributes
Change-Id: I52115f130bf1c7c80c66e18d0bf9f7acb16dd56c
When a node is added to a cluster with a network template
networks will be mapped to NICs based on transformations
in the template. Any bonds will be created in the database.
Any time the cluster's network template is changed the NIC
mapping will be recreated.
Change-Id: I2b52444f3019241362e5a957bf191645b94ebaa7
Closes-bug: #1493391
test_vlan_range
The test has been failing because JSON schema validates restriction in
order that depends on PYTHOHASHSEED. So the fix is to use proper
values with only one incorrect value.
test_non_default_bridge_mapping
The test has been failing because we have a suffix generation for
bridges implemented via counter. We generate a bridge name with a
suffix iterating over networks, and PYTHONHASHSEED affects order of
iteration. So the fix is to do not test bridge names strictly, but
just most important limits.
test_update_networks_meta
The test has been failing because dictionarie's .update() method do
not perform recursive update, and removes some needed keys
(such as 'name') so next network lookup may fail.
TestDeleteIBPImagesTask
The test has been failing because CMD line has files to remove, and
their order depend on PYTHONHASHSEED (since they are extracted as
.keys() from some dictionary structure). So the fix is to perform
smart check for the command line, not strict one.
Closes-Bug: #1496448
Change-Id: I882bbf53f48c842070cb336b1185c40a3dfacc6b
Signed-off-by: Igor Kalnitsky <igor@kalnitsky.org>
Proper transformation is not added to network scheme after recent
refactoring. This patch fixes the bug and adds corresponding test.
Closes-Bug: #1492138
Change-Id: I5d1336a10f855e17dede93d156cf586ca5f708f3
When neutron DVR is enabled compute nodes should have public access but
don't require IP address set on br-ex. In this case no gateway should be
set on br-ex as well.
Closes-Bug: #1491023
Co-Authored-By: Alexander Kislitsky <akislitsky@mirantis.com>
Change-Id: I31de7dc93f22f275d8b7b2dacbd97f6c98902604
* Use 'network_assignments' field from processed network_template
with applied nic mappings for each node instead of using this field
from raw template
Change-Id: I0f789386a6de56d04476a7ced1ba389677a98388
Closes-bug: #1489508
By design since version 7.0 all information about
VIP should be stored in network_metadata["vips"].
VIP duplications in root space of astute.yaml must
be removed.
Change-Id: Ide082d3ba9e063c2601837a772d21b0efa6e65a2
Closes-bug: #1488454
This change adds generate_network_scheme method for 7.0 instead of inherit it.
Now generate_network_scheme for 7.0 supports non-default network groups.
Change-Id: Ib89594867c57b3aeb1b6503fed87c291ca3fd308
Closes-bug: #1483307
IPs were allocated for every network on every node
regardless of network to nodes mapping which is set in template.
It led to excessive IP usage.
Now it is fixed so that IPs are allocated only for nodes where they are required.
And IP allocation is optimized for the case with network template and for other
cases partially.
A number of small fixes were made on the base of comments in earlier RCs.
IP allocation comparison in DB queries count (only for replaced part):
old version: 1 query per IP to allocate + 1 query per occupied IP in that range
new version: 1 queries per network (for all IPs) if (occupied IPs)+(IPs to allocate) <= MIN_IPS_PER_DB_QUERY
2 queries per network otherwise (typically).
New version is much better for large numbers of nodes.
nodes | queries count |
count | old | new |
|min| max |min| max |
1 | 1| 1+N| 1|1+ceil(N/5)**|
10 | 10| 10+N| 1|1+ceil(N/10) |
100 |100|100+N| 1|1+ceil(N/100)|
where N is a number of occupied IPs.
** 5 here is value of MIN_IPS_PER_DB_QUERY.
Change-Id: I4479bcae9e80f7b96750f9fdb2868d6df164baf6
Closes-Bug: #1480345
remove "mgmt/api" network role
fix tests for "ceph/public" network role
it has "management" as default_mapping
Change-Id: Id6c87de91d3bb63b2906d4df64ecea364a2c519b
Closes-Bug: #1484560
Network template serialiser should check that endpoint for
"neutron/floating" role is not present in corresponding section
before define it with empty ip address.
Change-Id: I76eb4e1df70811d664579d8b74107e3c638d0b83
Closes-Bug: 1483115
Andreas Jaeger