Alexander Kislitsky 4fa861fa8a Distributed serialization implementation ... Distributed serialization is implemented with python distributed library. We have scheduler for jobs management and workers for jobs processing. Scheduler is started on the master node as well as set of workers on it. Also workers are started on all nodes. In the cluster settings we can select the type of serialization and nodes statuses that allows serialization on it. By default nodes with status 'ready' are excluded from the workers list. For data serialization we are using only nodes from the cluster where serialization is performing. Before the computation fresh nailgun code is sent to the workers as zip file and it will be imported for job execution. So we always have fresh nailgun code on the workers. In one job we are processing chunks of tasks on the workers. This approach significantly boosts performance. The tasks chunk size is defined as settings.LCM_DS_TASKS_PER_JOB parameter. For limiting memory consumption on the master node we use parameter settings.LCM_DS_NODE_LOAD_COEFF for calculation max number of jobs in the processing queue. Synthetic tests of distributed serialization for 500 nodes with nubmer of ifaces >= 5 performed on 40 cores (4 different machines) took 6-7 minutes on average. Change-Id: Id8ff8fada2f1ab036775fc01c78d91befdda9ea2 Implements: blueprint distributed-serialization		2017-03-24 15:15:30 +00:00
..
__init__.py	Files with no code must be left completely empty	2016-09-30 16:13:45 +07:00
test_action_logs.py	Do not allow to run 2 or more tasks simultaneously	2016-08-30 14:45:56 +03:00
test_api_doc_generation.py	Move volume manager specific urls into extension	2015-06-29 12:08:48 +00:00
test_assassin.py	Add logging setup for receiverd and assassind	2016-01-28 18:54:34 +02:00
test_attributes.py	remove vmware	2017-02-14 15:52:27 +00:00
test_auth_token.py	Limit cookie usage to download routes	2016-08-01 12:29:50 +03:00
test_capacity_handler.py	Modify use of assertTrue(A in B)	2016-10-13 14:55:58 +08:00
test_changes_model.py	remove vmware	2017-02-14 15:52:27 +00:00
test_charset_issues.py	Do not allow to run 2 or more tasks simultaneously	2016-08-30 14:45:56 +03:00
test_cluster.py	Get rid of fake tasks where they are not required	2016-05-11 16:35:53 +03:00
test_cluster_changes_handler.py	Distributed serialization implementation	2017-03-24 15:15:30 +00:00
test_cluster_collection_handlers.py	remove vmware	2017-02-14 15:52:27 +00:00
test_cluster_handler.py	Use assertIsNone instead of assertEqual(None, obj)	2017-01-23 13:04:32 +07:00
test_cluster_plugin_links_collection_handler.py	Plugin links handlers now returning 409 on existing url	2016-05-16 13:45:42 +03:00
test_cluster_plugin_links_handler.py	Plugin links handlers now returning 409 on existing url	2016-05-16 13:45:42 +03:00
test_cluster_roles_handler.py	Introduce new validation rules for tags and roles	2016-11-30 19:56:52 +00:00
test_cluster_scaling.py	Get rid of fake tasks where they are not required	2016-05-11 16:35:53 +03:00
test_component_handler.py	Fixed constructing of a Mutable objects tree	2016-02-01 16:55:48 +03:00
test_deletion_task.py	remove zabbix from nailgun	2016-05-31 06:28:20 +00:00
test_deployment_error_handling.py	Allow errored cluster to have settings unlocked	2016-04-05 23:23:55 +03:00
test_deployment_graph_handlers.py	Message for tasks duplication fixed	2016-12-28 11:56:08 +03:00
test_deployment_history_handlers.py	Merge "Merge property 'custom' of deployment_history with root"	2016-10-06 08:22:24 +00:00
test_deployment_sequences_handler.py	Added default node filter	2016-10-05 21:07:05 +03:00
test_extension_handler.py	Align code for Extension Handler	2016-09-07 13:46:01 +03:00
test_fixture_uploading.py	DeploymentGraph and DeploymentGraphTasks models added	2016-03-03 02:28:56 +03:00
test_graph_execution.py	Added default node filter	2016-10-05 21:07:05 +03:00
test_graph_related_handlers.py	Avoid creating tasks with empty nodes list	2016-11-09 12:37:19 +03:00
test_master_node_settings_handler.py	Remove Mirantis-specific code from fuel-web repo	2016-02-26 13:25:17 +00:00
test_mellanox_orchestrator_serializer.py	fix assert to assertTrue	2016-10-18 21:16:41 +08:00
test_mongo_nodes.py	Don't use self.env.clusters[0] in tests where possible	2016-04-26 17:53:36 +03:00
test_network_verification_manager.py	Sort result of MulticastVerificationTask in integration test	2016-05-06 14:25:34 -07:00
test_node_allocation_stats_handler.py	Snapshot download with authentication	2015-07-03 09:59:35 +00:00
test_node_assignment.py	Allow to update roles for deployed nodes.	2016-11-18 16:38:34 +00:00
test_node_collection_handlers.py	Use assertIsNone instead of assertEqual(None, obj)	2017-01-23 13:04:32 +07:00
test_node_deletion.py	Message format on node(-s) deletion error changed	2017-01-26 20:41:56 +03:00
test_node_handler.py	Implement mechanism for plugin node attributes	2016-10-21 19:47:14 +03:00
test_notification.py	Make more strict notification about deleted environment	2016-06-26 13:33:27 +03:00
test_openstack_config_task_manager.py	Allow a user to run dry-run deployment	2016-05-30 15:46:31 +00:00
test_orchestrator_handlers.py	Message about no tasks to run deploy is fixed	2016-12-21 20:10:52 +00:00
test_orchestrator_serializer.py	remove vmware	2017-02-14 15:52:27 +00:00
test_orchestrator_serializer_70.py	remove vmware	2017-02-14 15:52:27 +00:00
test_orchestrator_serializer_80.py	Conform tests to the 11/ocata release	2016-12-16 16:06:40 +04:00
test_orchestrator_serializer_90.py	Serialize mtu for dpdk interface with 'i40e' driver	2017-03-06 12:01:47 +00:00
test_orchestrator_serializer_110.py	Switch provider from ovs to dpdkovs	2016-12-16 16:22:20 +04:00
test_plugin_links_collection_handler.py	Plugin links handlers now returning 409 on existing url	2016-05-16 13:45:42 +03:00
test_plugin_links_handler.py	Plugin links handlers now returning 409 on existing url	2016-05-16 13:45:42 +03:00
test_plugin_manager.py	Merge "Plugin rename is denied"	2017-01-10 10:40:28 +00:00
test_plugins_api.py	Plugin rename is denied	2016-12-23 17:48:46 +03:00
test_provisioning.py	Get rid of fake tasks where they are not required	2016-05-11 16:35:53 +03:00
test_provisioning_serializer.py	Revert "Limit the minimal RAM amount for OVS+DPDK to 1024MB"	2017-01-19 12:54:31 +04:00
test_public_api.py	Require auth for version api call	2016-06-06 21:39:00 +00:00
test_put_same_json.py	Don't use self.env.clusters[0] in tests where possible	2016-04-26 17:53:36 +03:00
test_release_attributes_metadata_handler.py	Added ReleaseAttributesMetadataHandler	2016-06-17 13:25:52 +00:00
test_release_networks_handler.py	Swap deprecated oslo namespaces	2015-07-23 23:33:58 +03:00
test_removed_handler.py	Fix hacking 0.10 errors in nailgun	2015-10-08 12:01:55 +02:00
test_reset_environment.py	Fix naming of reset tasks and message duplication	2017-01-12 17:24:02 +04:00
test_role_api.py	Introduce new validation rules for tags and roles	2016-11-30 19:56:52 +00:00
test_roles.py	Introduce new validation rules for tags and roles	2016-11-30 19:56:52 +00:00
test_rpc_consumer.py	Fix naming of reset tasks and message duplication	2017-01-12 17:24:02 +04:00
test_spawning_vms.py	Validate node VMs format	2016-05-19 12:05:38 +02:00
test_stats_user_task_manager.py	Don't use self.env.clusters[0] in tests where possible	2016-04-26 17:53:36 +03:00
test_stop_deployment.py	Do not allow to run 2 or more tasks simultaneously	2016-08-30 14:45:56 +03:00
test_tag_api.py	Introduce new validation rules for tags and roles	2016-11-30 19:56:52 +00:00
test_task_deploy.py	Fix DPDK settings get for upgrades	2016-12-26 15:55:13 +00:00
test_task_logging.py	Get rid of fake tasks where they are not required	2016-05-11 16:35:53 +03:00
test_task_managers.py	Merge "Use assertIsNone instead of assertEqual(None, obj)"	2017-03-22 17:23:16 +00:00
test_transactions_manager.py	Resolve tasks by role tags	2016-11-17 18:07:30 +03:00
test_verify_networks_task_manager.py	Bond attributes in interface API	2016-11-30 11:39:35 +02:00
test_version_api.py	Require auth for version api call	2016-06-06 21:39:00 +00:00