openstack
/
fuel-web
Code Issues Proposed changes
fuel-web/nailgun/nailgun/test/unit
History
Alexander Kislitsky 4fa861fa8a Distributed serialization implementation 
Distributed serialization is implemented with python distributed
library. We have scheduler for jobs management and workers for
jobs processing. Scheduler is started on the master node as well
as set of workers on it. Also workers are started on all nodes.
In the cluster settings we can select the type of serialization
and nodes statuses that allows serialization on it. By default
nodes with status 'ready' are excluded from the workers list.

For data serialization we are using only nodes from the cluster
where serialization is performing.

Before the computation fresh nailgun code is sent to the workers
as zip file and it will be imported for job execution. So we always
have fresh nailgun code on the workers.

In one job we are processing chunks of tasks on the workers. This
approach significantly boosts performance. The tasks chunk size
is defined as settings.LCM_DS_TASKS_PER_JOB parameter.

For limiting memory consumption on the master node we use parameter
settings.LCM_DS_NODE_LOAD_COEFF for calculation max number of jobs
in the processing queue.

Synthetic tests of distributed serialization for 500 nodes with
nubmer of ifaces >= 5 performed on 40 cores (4 different machines)
took 6-7 minutes on average.

Change-Id: Id8ff8fada2f1ab036775fc01c78d91befdda9ea2
Implements: blueprint distributed-serialization
 		2017-03-24 15:15:30 +00:00
..
fuel_statistics_tests remove vmware 2017-02-14 15:52:27 +00:00
fuyaql_tests Use assertIsNone instead of assertEqual(None, obj) 2017-01-23 13:04:32 +07:00
__init__.py Files with no code must be left completely empty 2016-09-30 16:13:45 +07:00
test_assignment_validator.py Fix role restriction iteration cycle according to docs 2016-06-03 14:52:51 +00:00
test_attributes_generators.py Improve password quality for generated passwords 2015-10-02 08:41:01 -05:00
test_attributes_validator.py Add nullable flag to NIC's mtu attribute 2016-12-26 10:07:01 +03:00
test_ceph.py Ceph: fix PG count number 2015-08-13 17:36:01 +03:00
test_check_repo_connection_task.py Don't use self.env.clusters[0] in tests where possible 2016-04-26 17:53:36 +03:00
test_cluster_validator.py Support stop deployment for already deployed cluster with LCM 2016-04-20 20:09:04 +03:00
test_cpu_distribution.py Change CPU distribution 2016-06-06 18:36:27 +03:00
test_datadiff.py Introduced datadiff tool 2016-03-24 13:23:32 +03:00
test_db.py Allow to use socket to connect to DB in Nailgun 2014-12-10 07:21:00 +00:00
test_db_migrations.py New VIP-related fields in the database 2016-02-02 18:16:27 +03:00
test_db_models.py Remove offloading modes from data model 2016-11-30 18:35:00 +02:00
test_deadlock_detector.py Fixed dead-locks and race conditions in tasks execution 2016-09-05 17:46:36 +03:00
test_default_ip_addrs.py Load fixture with master node ip for fuelweb_admin 2014-02-24 14:17:34 +02:00
test_deployment_graph_object.py remove vmware 2017-02-14 15:52:27 +00:00
test_deployment_history.py Merge property 'custom' of deployment_history with root 2016-09-26 11:47:28 +03:00
test_deployment_history_object.py Fuel Nailgun: Noop run support 2016-08-25 02:30:45 +03:00
test_deployment_network_serializer.py Move network serialization code to network manager extension 2016-05-12 12:40:35 +03:00
test_deployment_nodes_filtering.py Don't use self.env.clusters[0] in tests where possible 2016-04-26 17:53:36 +03:00
test_deployment_serializer.py Revert "Remove 'nodes' key from serializer for 10.0" 2016-07-19 12:42:12 +00:00
test_downgrade_fuel_9_0_1.py Add downgrade migration tests 2016-06-09 10:58:06 +00:00
test_downgrade_fuel_9_1.py Add migrations for NICs and Bonds 'interface_properties' 2016-11-23 14:22:34 +02:00
test_downgrade_fuel_9_2.py Use assertIsNone instead of assertEqual(None, obj) 2017-01-23 13:04:32 +07:00
test_downgrade_fuel_10_0.py Revert "Add tag API" 2016-11-08 16:58:59 +00:00
test_expression_parser.py Add exceptions hierarchy 2016-04-14 17:41:25 +03:00
test_extensions.py Remove list conversion for cluster extension handler 2016-09-13 18:10:28 +03:00
test_fault_tolerance.py Fault tolerance for provision 2015-03-08 10:53:01 +01:00
test_graph.py Orchestrator DeploymentGraph is renamed to the GraphSolver 2016-03-17 03:25:37 +03:00
test_graph_executor_handler.py Itroduced deployment sequences 2016-09-07 16:28:49 +03:00
test_graph_serializer.py remove zabbix from nailgun 2016-05-31 06:28:20 +00:00
test_graph_solver_tasks_validator.py Allow YAQL expressions in task's dependencies 2016-05-26 18:42:29 +03:00
test_handlers.py add removed handlers 2017-03-23 12:26:38 +03:00
test_hugepages_distribution.py Change CPU distribution 2016-06-06 18:36:27 +03:00
test_lcm_task_serializers.py Add ability to get common or node part of context in lcm 2016-09-09 17:42:48 +03:00
test_lcm_transaction_serializer.py Distributed serialization implementation 2017-03-24 15:15:30 +00:00
test_legacy_tasks_adapter.py Resolve tasks by role tags 2016-11-17 18:07:30 +03:00
test_load_db_driver_handler.py Merge NodeAttributes model into Node model 2016-02-10 12:42:02 +02:00
test_logs_handlers.py Pass auth token to diagnostic snapshot dumper 2016-09-06 14:34:21 +03:00
test_logs_utils.py prepare_for_deployment was moved to network manager 2015-11-30 13:48:14 +03:00
test_middleware_utils.py HTTP requests monitoring middleware added 2014-10-17 18:13:31 +03:00
test_migration_fuel_6_1.py New VIP-related fields in the database 2016-02-02 18:16:27 +03:00
test_migration_fuel_7_0.py Fix minor comments in NetworkGroup API code request 2016-04-22 10:17:50 +00:00
test_migration_fuel_8_0.py Data model changes for Plugin NIC, BOND and Node attributes 2016-05-30 13:16:45 +03:00
test_migration_fuel_9_0.py Data model changes for Plugin NIC, BOND and Node attributes 2016-05-30 13:16:45 +03:00
test_migration_fuel_9_0_1.py Allow long error messages for nodes 2016-06-07 18:56:19 +02:00
test_migration_fuel_9_1.py Adjust db migrations according to stable branch 2016-09-27 11:16:37 +03:00
test_migration_fuel_9_2.py Use assertIsNone instead of assertEqual(None, obj) 2017-01-23 13:04:32 +07:00
test_migration_fuel_10_0.py add tests for alembic migration 2017-03-13 14:02:23 +03:00
test_migration_plugins_cluster_attrs_use_ids_mapping.py Use assertIsNone instead of assertEqual(None, obj) 2017-01-23 13:04:32 +07:00
test_mutable.py Added YAML representers for Mutable objects 2016-03-18 23:18:33 +03:00
test_nailgun_keystone_auth_middleware.py Fix hacking 0.10 errors in nailgun 2015-10-08 12:01:55 +02:00
test_network_verify.py Support for plugin NIC attributes 2016-11-29 10:37:57 +02:00
test_node_assignment_handler.py Use assertIsNone instead of assertEqual(None, obj) 2017-01-23 13:04:32 +07:00
test_node_attributes.py Add method for getting default hugepages 2016-12-06 08:49:20 +00:00
test_node_attributes_validator.py Limit the minimal RAM amount for OVS+DPDK to 1024MB 2017-02-07 16:42:01 +00:00
test_node_groups.py Set nodes' statuses to 'error' when their nodegroup is deleted 2017-02-07 17:44:06 +00:00
test_node_validator.py Validate node VMs format 2016-05-19 12:05:38 +02:00
test_notification_collection_handler.py Swap deprecated oslo namespaces 2015-07-23 23:33:58 +03:00
test_notification_handler.py Handler for changing notifications statuses added 2017-02-07 16:39:05 +00:00
test_object_plugin.py Use dict_merge instead of update 2016-12-14 15:02:02 +02:00
test_objects.py Merge "Use tags instead of roles" 2017-03-24 11:48:45 +00:00
test_openstack_config_handler.py node_ids parameter is interpreted correctly 2016-08-02 19:02:20 +03:00
test_orchestrator_stages.py Fixed executing plugins in pre deployment stage 2016-01-18 12:08:04 +03:00
test_plugin_adapters.py Merge "Tags info calculation is supported on plugins update" 2017-01-12 14:01:19 +00:00
test_plugin_loader.py Set legacy tasks to empty if file is removed or empty 2016-12-07 21:05:10 +00:00
test_plugin_manager.py Fix wrong detection of plugin deletable property 2017-01-10 15:37:04 +00:00
test_plugins_serializers.py Resolve tasks by role tags 2016-11-17 18:07:30 +03:00
test_policies.py Add exceptions hierarchy 2016-04-14 17:41:25 +03:00
test_primary_tags_assignment.py Resolve tasks by role tags 2016-11-17 18:07:30 +03:00
test_rbac_middleware.py Read-only mode for Fuel API was added 2016-05-25 18:39:40 +03:00
test_receiver.py Do not move task status to running after ack 2016-09-13 11:23:18 +00:00
test_redeployment_scenarios.py Don't use self.env.clusters[0] in tests where possible 2016-04-26 17:53:36 +03:00
test_release_collection_handlers.py Fix hacking 0.10 errors in nailgun 2015-10-08 12:01:55 +02:00
test_release_handler.py Introduce new validation rules for tags and roles 2016-11-30 19:56:52 +00:00
test_release_networks_schema.py Add support for release network_metadata serialziation 2014-11-06 15:33:05 +02:00
test_release_validator.py Fix role validation in release PUT handler 2016-10-24 07:05:08 +00:00
test_requirements.py Check if nailgun's requirements do not conflict 2015-06-22 09:17:41 +00:00
test_restriction.py remove vmware 2017-02-14 15:52:27 +00:00
test_reusage_admin_address.py Move network manager to extension 2016-04-11 05:33:29 -07:00
test_role_resolver.py Resolve tasks by role tags 2016-11-17 18:07:30 +03:00
test_role_validator.py Introduce new validation rules for tags and roles 2016-11-30 19:56:52 +00:00
test_rpc_acknowledge.py Message in receiverd should be requeued on deadlock 2016-05-12 10:35:00 +03:00
test_settings_config.py Urgent fix for log page 2014-12-15 17:07:58 +04:00
test_stages_task_serialization.py Resolve tasks by role tags 2016-11-17 18:07:30 +03:00
test_task.py remove vmware 2017-02-14 15:52:27 +00:00
test_task_based_deployment.py Resolve tasks by role tags 2016-11-17 18:07:30 +03:00
test_task_handler.py Added field parent_id to serialized task data 2016-07-25 07:28:21 +00:00
test_task_helpers.py Fix cluster progress calculation if Astute didn't send it 2016-10-20 08:03:14 +00:00
test_task_managers.py Do not allow to run 2 or more tasks simultaneously 2016-08-30 14:45:56 +03:00
test_tasks_templates.py Remove downloading debian-installer components 2016-04-18 10:18:20 +00:00
test_testing_utils.py Fix test test_put_handler_with_one_node 2016-02-03 14:06:02 +01:00
test_transaction_handler.py don't merge common_attrs with node data in deployment_info 2016-08-31 17:28:11 +03:00
test_transaction_object.py don't merge common_attrs with node data in deployment_info 2016-08-31 17:28:11 +03:00
test_transactions_manager.py Fixed switch to deploying state for nodes which will not be deployed 2017-02-08 13:20:22 +00:00
test_uniondict.py Add UnionDict class for lcm 2016-09-05 19:52:51 +03:00
test_urls.py Remove Mirantis-specific code from fuel-web repo 2016-02-26 13:25:17 +00:00
test_utils.py Bond attributes in interface API 2016-11-30 11:39:35 +02:00
test_version_handler.py Deprecate version.yaml 2015-11-17 01:05:39 +03:00
test_yaql_ext.py Stop sharing yaql engine between child processes 2016-07-26 00:18:26 +03:00