Distributed serialization is implemented with python distributed
library. We have scheduler for jobs management and workers for
jobs processing. Scheduler is started on the master node as well
as set of workers on it. Also workers are started on all nodes.
In the cluster settings we can select the type of serialization
and nodes statuses that allows serialization on it. By default
nodes with status 'ready' are excluded from the workers list.
For data serialization we are using only nodes from the cluster
where serialization is performing.
Before the computation fresh nailgun code is sent to the workers
as zip file and it will be imported for job execution. So we always
have fresh nailgun code on the workers.
In one job we are processing chunks of tasks on the workers. This
approach significantly boosts performance. The tasks chunk size
is defined as settings.LCM_DS_TASKS_PER_JOB parameter.
For limiting memory consumption on the master node we use parameter
settings.LCM_DS_NODE_LOAD_COEFF for calculation max number of jobs
in the processing queue.
Synthetic tests of distributed serialization for 500 nodes with
nubmer of ifaces >= 5 performed on 40 cores (4 different machines)
took 6-7 minutes on average.
Change-Id: Id8ff8fada2f1ab036775fc01c78d91befdda9ea2
Implements: blueprint distributed-serialization
At the moment feature_groups are defined during
ISO build in version.yaml file. We need to avoid
using build time defined parameters because
we are moving to the package based delivery approach.
Depends-On: I05e841d4279bc61645cefb68f411a93b67e57189
Related-Bug: #1513737
Related-Bug: #1515517
Change-Id: I4f96fec73f991094bca65de5f1d1014662fbb2b4
This commit adds functionality to control logging from nailgun
by APP_LOGLEVEL value in the '/etc/nailgun/settings.yaml'
Also it reduces log level of notices indicating hardware_info_locked
when trying to fetch metadata to debug because they are much
too frequent.
Change-Id: I9d2ff84df1d663be363d2f1f7dc7d5e141bfdf27
Closes-Bug: #1510151
Added:
- ARTIFACTS variable to store all test artifacts
under single directory
- yaml config generation based on environment variables
- NAILGUN_CONFIG variable in run_tests.sh and settings.py
Commands used for testing:
TEST_NAILGUN_DB=nailgun1 ARTIFACTS=/tmp/test_runs/1 \
UI_SERVER_PORT=8011 FUELCLIENT_SERVER_PORT=8021 ./run_tests.sh
TEST_NAILGUN_DB=nailgun2 ARTIFACTS=/tmp/test_runs/2 \
UI_SERVER_PORT=8012 FUELCLIENT_SERVER_PORT=8022 ./run_tests.sh
Change-Id: If03dddee9a8259a80b526a53fe61a79c544c72a4
Closes-Bug: #1282565
Closes-Bug: #1334180
Now nailgun searches for version.yaml in this order of priority:
/etc/fuel/version.yaml
/etc/fuel/nailgun/version.yaml
/etc/nailgun/version.yaml
This reflects the modifications made to Fuel library for parsing version file in https://review.openstack.org/85296
Closes-Bug: #1313303
Change-Id: I9005909f0852527d4df422ace80764841ae0fb0e
We have several log files for nailgun app.log and api.log.
Also we have several threads. It was very difficult to find
out which part of log was written by rpc thread and
which by keep alive or web.py thread. Now we can use
standard bash utils to merge two log files and find specific
thread.
* move logs configuration in one place.
Use single format of time and log format
for api log and nailgun log.
* add thread logging
* add logging of milliseconds
Implements: blueprint nailgun-improve-logging-for-multithreading-app
Change-Id: I1d8816ad01c39de9602acdba8e99859d0fcd7a57
For testing purposes, set CHEF_CONF_FOLDER to base nailgun dir.
node_id.json is created during tests. To clean unused files use './run_tests.sh -c' since now.
ToDo:
* use IP address to ssh into node, not node.name
* use MAC address for json's names
* mock SshConnect in tests
* create test(s) for deploy_env to ensure that subtasks work fine
Andreas Jaeger