Retire repository
Fuel (from openstack namespace) and fuel-ccp (in x namespace) repositories are unused and ready to retire. This change removes all content from the repository and adds the usual README file to point out that the repository is retired following the process from https://docs.openstack.org/infra/manual/drivers.html#retiring-a-project See also http://lists.openstack.org/pipermail/openstack-discuss/2019-December/011647.html Depends-On: https://review.opendev.org/699362 Change-Id: If0c2011947c9eb63f59093812b5f9f95ce56a6f8
This commit is contained in:
parent
46d1908d60
commit
3e5483d1dc
|
@ -1,23 +0,0 @@
|
|||
.idea
|
||||
*.gem
|
||||
|
||||
# SimpleCov
|
||||
coverage
|
||||
coverage.data
|
||||
|
||||
# Need only on local machine for gem
|
||||
Gemfile.lock
|
||||
docs/_build
|
||||
|
||||
|
||||
#Vim swap files
|
||||
*.swp
|
||||
|
||||
# Local raemon copy
|
||||
raemon/
|
||||
|
||||
*.svg
|
||||
*.png
|
||||
|
||||
*.yaml
|
||||
!examples/example_astute_config.yaml
|
|
@ -1 +0,0 @@
|
|||
2.1
|
3
Gemfile
3
Gemfile
|
@ -1,3 +0,0 @@
|
|||
source 'https://rubygems.org'
|
||||
gem 'raemon', :git => 'https://github.com/pressly/raemon', :ref => 'b78eaae57c8e836b8018386dd96527b8d9971acc'
|
||||
gemspec
|
176
LICENSE
176
LICENSE
|
@ -1,176 +0,0 @@
|
|||
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
67
MAINTAINERS
67
MAINTAINERS
|
@ -1,67 +0,0 @@
|
|||
---
|
||||
description:
|
||||
For Fuel team structure and contribution policy, see [1].
|
||||
|
||||
This is repository level MAINTAINERS file. All contributions to this
|
||||
repository must be approved by one or more Core Reviewers [2].
|
||||
If you are contributing to files (or create new directories) in
|
||||
root folder of this repository, please contact Core Reviewers for
|
||||
review and merge requests.
|
||||
|
||||
If you are contributing to subfolders of this repository, please
|
||||
check 'maintainers' section of this file in order to find maintainers
|
||||
for those specific modules.
|
||||
|
||||
It is mandatory to get +1 from one or more maintainers before asking
|
||||
Core Reviewers for review/merge in order to decrease a load on Core Reviewers [3].
|
||||
Exceptions are when maintainers are actually cores, or when maintainers
|
||||
are not available for some reason (e.g. on vacation).
|
||||
|
||||
[1] https://specs.openstack.org/openstack/fuel-specs/policy/team-structure
|
||||
[2] https://review.openstack.org/#/admin/groups/655,members
|
||||
[3] http://lists.openstack.org/pipermail/openstack-dev/2015-August/072406.html
|
||||
|
||||
Please keep this file in YAML format in order to allow helper scripts
|
||||
to read this as a configuration data.
|
||||
|
||||
maintainers:
|
||||
|
||||
- ./:
|
||||
- name: Vladimir Sharshov
|
||||
email: vsharshov@mirantis.com
|
||||
IRC: warpc
|
||||
|
||||
- name: Vladimir Kozhukalov
|
||||
email: vkozhukalov@mirantis.com
|
||||
IRC: kozhukalov
|
||||
|
||||
- name: Ivan Ponomarev
|
||||
email: iponomarev@mirantis.com
|
||||
IRC: iponomarev
|
||||
|
||||
- debian/: &packaging_team
|
||||
- name: Mikhail Ivanov
|
||||
email: mivanov@mirantis.com
|
||||
IRC: mivanov
|
||||
|
||||
- name: Artem Silenkov
|
||||
email: asilenkov@mirantis.com
|
||||
IRC: asilenkov
|
||||
|
||||
- name: Alexander Tsamutali
|
||||
email: atsamutali@mirantis.com
|
||||
IRC: astsmtl
|
||||
|
||||
- name: Daniil Trishkin
|
||||
email: dtrishkin@mirantis.com
|
||||
IRC: dtrishkin
|
||||
|
||||
- name: Ivan Udovichenko
|
||||
email: iudovichenko@mirantis.com
|
||||
IRC: tlbr
|
||||
|
||||
- name: Igor Yozhikov
|
||||
email: iyozhikov@mirantis.com
|
||||
IRC: IgorYozhikov
|
||||
|
||||
- specs/: *packaging_team
|
75
README.md
75
README.md
|
@ -1,75 +0,0 @@
|
|||
#Astute
|
||||
[![Team and repository tags](http://governance.openstack.org/badges/fuel-astute.svg)](http://governance.openstack.org/reference/tags/index.html)
|
||||
<!-- Change things from this point on -->
|
||||
|
||||
Astute is orchestrator, which is using data about nodes and deployment settings performs two things:
|
||||
- provision
|
||||
- deploy
|
||||
|
||||
Provision
|
||||
-----
|
||||
|
||||
OS installation on selected nodes.
|
||||
|
||||
Provisioning is done using Cobbler. Astute orchestrator collects data about nodes and creates corresponding Cobbler systems using parameters specified in engine section of provision data. After the systems are created, it connects to Cobbler engine and reboots nodes according to the power management parameters of the node.
|
||||
|
||||
Deploy
|
||||
-----
|
||||
|
||||
OpenStack installation in the desired configuration on the selected nodes.
|
||||
|
||||
Astute uses data about nodes and deployment settings and recalculates parameters needed for deployment. Calculated parameters are passed to the nodes being deployed by use of nailyfact MCollective agent that uploads these attributes to `/etc/astute.yaml` file of the node. Then puppet parses this file using Facter plugin and uploads these facts into puppet. These facts are used during catalog compilation phase by puppet. Finally catalog is executed and Astute orchestrator passes to the next node in deployment sequence. Fuel Library provides puppet modules for Astute.
|
||||
|
||||
Using as library
|
||||
-----
|
||||
|
||||
```ruby
|
||||
require 'astute'
|
||||
|
||||
class ConsoleReporter
|
||||
def report(msg)
|
||||
puts msg.inspect
|
||||
end
|
||||
end
|
||||
|
||||
reporter = ConsoleReporter.new
|
||||
|
||||
orchestrator = Astute::Orchestrator.new(log_parsing=false)
|
||||
|
||||
# Add systems to cobbler, reboot and start installation process.
|
||||
orchestrator.provision(reporter, environment['engine'], environment['nodes'])
|
||||
|
||||
# Observation OS installation
|
||||
orchestrator.watch_provision_progress(reporter, environment['task_uuid'], environment['nodes'])
|
||||
|
||||
# Deploy OpenStack
|
||||
orchestrator.deploy(reporter, environment['task_uuid'], environment['nodes'])
|
||||
|
||||
```
|
||||
Example of using Astute as library: lib/astute/server/dispatcher.rb
|
||||
|
||||
|
||||
Using as CLI
|
||||
-----
|
||||
|
||||
CLI interface in Astute no longer supported. Please use new Fuel-CLI. More details you can get by link: https://github.com/openstack/fuel-docs/blob/master/pages/user-guide/cli.rst
|
||||
|
||||
-----
|
||||
|
||||
- ISO, other materials: http://fuel.mirantis.com/
|
||||
- User guide: http://docs.mirantis.com/
|
||||
- Development documentation: http://docs.mirantis.com/fuel-dev/
|
||||
|
||||
|
||||
License
|
||||
------
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
not use this file except in compliance with the License. You may obtain
|
||||
a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
License for the specific language governing permissions and limitations
|
||||
under the License.
|
|
@ -0,0 +1,10 @@
|
|||
This project is no longer maintained.
|
||||
|
||||
The contents of this repository are still available in the Git
|
||||
source code management system. To see the contents of this
|
||||
repository before it reached its end of life, please check out the
|
||||
previous commit with "git checkout HEAD^1".
|
||||
|
||||
For any further questions, please email
|
||||
openstack-discuss@lists.openstack.org or join #openstack-dev on
|
||||
Freenode.
|
12
Rakefile
12
Rakefile
|
@ -1,12 +0,0 @@
|
|||
require 'rspec/core/rake_task'
|
||||
|
||||
namespace :spec do
|
||||
RSpec::Core::RakeTask.new(:unit) do |t|
|
||||
t.rspec_opts = "--color --format documentation"
|
||||
specfile = ENV['S'].to_s.strip.length > 0 ? "*#{ENV['S']}*" : '*'
|
||||
t.pattern = "spec/unit/**/#{specfile}_spec.rb"
|
||||
end
|
||||
end
|
||||
|
||||
task :default => 'spec:unit'
|
||||
task :spec => 'spec:unit'
|
|
@ -1,33 +0,0 @@
|
|||
$:.unshift File.expand_path('lib', File.dirname(__FILE__))
|
||||
require 'astute/version'
|
||||
|
||||
Gem::Specification.new do |s|
|
||||
s.name = 'astute'
|
||||
s.version = Astute::VERSION
|
||||
|
||||
s.summary = 'Orchestrator for OpenStack deployment'
|
||||
s.description = 'Deployment Orchestrator of Puppet via MCollective. Works as a library or from CLI.'
|
||||
s.authors = ['Mike Scherbakov']
|
||||
s.email = ['mscherbakov@mirantis.com']
|
||||
|
||||
s.add_dependency 'activesupport', '~> 4.1'
|
||||
s.add_dependency 'mcollective-client', '>= 2.4.1'
|
||||
s.add_dependency 'symboltable', '>= 1.0.2'
|
||||
s.add_dependency 'rest-client', '>= 1.6.7'
|
||||
|
||||
# Astute as service
|
||||
s.add_dependency 'bunny', '>= 2.0'
|
||||
s.add_dependency 'raemon', '>= 0.3'
|
||||
|
||||
s.add_development_dependency 'facter'
|
||||
s.add_development_dependency 'rake', '10.0.4'
|
||||
s.add_development_dependency 'rspec', '>= 3.4.0'
|
||||
s.add_development_dependency 'mocha', '0.13.3'
|
||||
s.add_development_dependency 'simplecov', '~> 0.7.1'
|
||||
s.add_development_dependency 'simplecov-rcov', '~> 0.2.3'
|
||||
|
||||
s.files = Dir.glob("{bin,lib,spec,examples}/**/*")
|
||||
s.executables = %w(astuted astute-simulator)
|
||||
s.require_path = 'lib'
|
||||
end
|
||||
|
|
@ -1,9 +0,0 @@
|
|||
[Unit]
|
||||
Name=Astute daemon
|
||||
|
||||
[Service]
|
||||
EnvironmentFile=-/etc/sysconfig/astute
|
||||
ExecStart=/usr/bin/astuted $ASTUTE_OPTIONS
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
|
@ -1 +0,0 @@
|
|||
ASTUTE_OPTIONS="--config /etc/astute/astuted.conf --logfile /var/log/astute/astute.log --loglevel debug --workers 7"
|
|
@ -1,20 +0,0 @@
|
|||
#!/usr/bin/env ruby
|
||||
# Copyright 2016 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
require 'fuel_deployment/simulator'
|
||||
require 'astute'
|
||||
|
||||
simulator = Astute::Simulator.new
|
||||
simulator.run
|
88
bin/astuted
88
bin/astuted
|
@ -1,88 +0,0 @@
|
|||
#!/usr/bin/env ruby
|
||||
|
||||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
require 'astute'
|
||||
require 'logger'
|
||||
require 'ostruct'
|
||||
require 'optparse'
|
||||
require 'yaml'
|
||||
require 'raemon'
|
||||
|
||||
options = OpenStruct.new
|
||||
options.daemonize = false
|
||||
options.pidfile = '/var/run/astuted.pid'
|
||||
options.config_path = '/etc/astute/astuted.conf'
|
||||
options.log_path = nil
|
||||
options.log_level = 'debug'
|
||||
options.workers = 1
|
||||
|
||||
OptionParser.new do |opts|
|
||||
opts.banner = 'Usage: astuted [options]'
|
||||
opts.separator "\nOptions:"
|
||||
opts.on('-d', '--[no-]daemonize', 'Daemonize server') do |flag|
|
||||
options.daemonize = flag
|
||||
end
|
||||
opts.on('-P', '--pidfile PATH', 'Path to pidfile') do |path|
|
||||
options.pidfile = path
|
||||
end
|
||||
opts.on('-w', '--workers NUMBER', 'Number of worker processes') do |number|
|
||||
options.workers = number.to_i
|
||||
end
|
||||
opts.on('-c', '--config PATH', 'Use custom config file') do |path|
|
||||
unless File.exists?(path)
|
||||
puts "Error: config file #{path} was not found"
|
||||
exit
|
||||
end
|
||||
options.config_path = path
|
||||
end
|
||||
opts.on('-l', '--logfile PATH' 'Log file path') do |path|
|
||||
options.log_path = path
|
||||
end
|
||||
levels = %w{fatal error warn info debug}
|
||||
opts.on('--loglevel LEVEL', levels, "Logging level (#{levels.join(', ')})") do |level|
|
||||
options.log_level = level
|
||||
end
|
||||
opts.on_tail('-h', '--help', 'Show this message') do
|
||||
puts opts
|
||||
exit
|
||||
end
|
||||
opts.on_tail('-v', '--version', 'Show version') do
|
||||
puts Astute::VERSION
|
||||
exit
|
||||
end
|
||||
end.parse!
|
||||
|
||||
if options.daemonize
|
||||
# After daemonize we can't log to STDOUT, pick a default log file
|
||||
options.log_path ||= "#{Dir.pwd}/astute.log"
|
||||
end
|
||||
|
||||
Astute.config.update(YAML.load(File.read(options.config_path))) if File.exists?(options.config_path)
|
||||
Astute.logger = options.log_path ? Logger.new(options.log_path) : Logger.new(STDOUT)
|
||||
Astute.logger.level = Logger.const_get(options.log_level.upcase)
|
||||
Astute.logger.formatter = proc do |severity, datetime, progname, msg|
|
||||
severity_map = {'DEBUG' => 'DEBUG', 'INFO' => 'INFO', 'WARN' => 'WARNING', 'ERROR' => 'ERROR', 'FATAL' => 'CRITICAL'}
|
||||
"#{datetime.strftime("%Y-%m-%d %H:%M:%S")} #{severity_map[severity]} [#{Process.pid}] #{msg}\n"
|
||||
end
|
||||
|
||||
Astute.logger.debug "Starting with settings\n#{Astute.config.to_yaml}"
|
||||
|
||||
Raemon::Master.start(options.workers, Astute::Server::Worker,
|
||||
:detach => options.daemonize,
|
||||
:name => 'astute',
|
||||
:pid_file => options.pidfile,
|
||||
:logger => Astute.logger
|
||||
)
|
|
@ -1,47 +0,0 @@
|
|||
astute (10.0.0-1) trusty; urgency=low
|
||||
|
||||
* Bump version to 10.0
|
||||
|
||||
-- Sergey Kulanov <skulanov@mirantis.com> Mon, 21 Mar 2016 13:49:10 +0200
|
||||
|
||||
astute (9.0.0-1) trusty; urgency=low
|
||||
|
||||
* Update version to 9.0.0
|
||||
|
||||
-- Sergey Kulanov <skulanov@mirantis.com> Thu, 17 Dec 2015 15:35:14 +0200
|
||||
|
||||
astute (8.0.0-1) trusty; urgency=low
|
||||
|
||||
* Update version to 8.0.0
|
||||
|
||||
-- Vladimir Sharshov <vsharshov@mirantis.com> Fri, 28 Aug 2015 13:30:00 +0300
|
||||
|
||||
astute (7.0.0-1) trusty; urgency=low
|
||||
|
||||
* Update version to 7.0.0
|
||||
|
||||
-- Aleksandra Fedorova <afedorova@mirantis.com> Mon, 08 Jun 2015 19:30:00 +0300
|
||||
|
||||
astute (6.1.0-1) trusty; urgency=low
|
||||
|
||||
* Update version to 6.1.0
|
||||
|
||||
-- Matthew Mosesohn <mmosesohn@mirantis.com> Wed, 22 Apr 2015 14:44:00 +0300
|
||||
|
||||
astute (6.0.0-1) trusty; urgency=low
|
||||
|
||||
* Update code from upstream
|
||||
|
||||
-- Igor Kalnitsky <ikalnitsky@mirantis.com> Wed, 26 Nov 2014 19:49:00 +0200
|
||||
|
||||
astute (0.0.1-ubuntu1) precise; urgency=low
|
||||
|
||||
* Update code from upstream
|
||||
|
||||
-- OSCI Jenkins <dburmistrov@mirantis.com> Wed, 03 Sep 2014 15:20:13 +0400
|
||||
|
||||
astute (0.0.1) unstable; urgency=low
|
||||
|
||||
* Initial release.
|
||||
|
||||
-- Mirantis Product <product@mirantis.com> Tue, 20 Aug 2013 22:20:46 +0400
|
|
@ -1 +0,0 @@
|
|||
7
|
|
@ -1,12 +0,0 @@
|
|||
Source: astute
|
||||
Section: admin
|
||||
Priority: optional
|
||||
Maintainer: Mirantis Product <product@mirantis.com>
|
||||
Build-Depends: debhelper (>= 9), gem2deb
|
||||
Standards-Version: 3.9.2
|
||||
|
||||
Package: nailgun-mcagents
|
||||
Architecture: all
|
||||
Depends: ${misc:Depends}, ${shlibs:Depends}, mcollective
|
||||
Description: NailGun mcagents
|
||||
.
|
|
@ -1 +0,0 @@
|
|||
mcagents/* /usr/share/mcollective/plugins/mcollective/agent/
|
|
@ -1,4 +0,0 @@
|
|||
#!/usr/bin/make -f
|
||||
|
||||
%:
|
||||
dh $@
|
|
@ -1 +0,0 @@
|
|||
3.0 (quilt)
|
|
@ -1,22 +0,0 @@
|
|||
# This is example config file for Astute. Your config file should be placed
|
||||
# to /opt/astute/astute.conf. You can check default values in config.rb file.
|
||||
---
|
||||
# mc_retries is used in mclient.rb file.
|
||||
# MClient tries mc_retries times to call MCagent before failure.
|
||||
mc_retries: 5
|
||||
# puppet_timeout is used in puppetd.rb file.
|
||||
# Maximum time (in seconds) Astute waits for the whole deployment.
|
||||
puppet_timeout: 3600
|
||||
# puppet_deploy_interval is used in puppetd.rb file.
|
||||
# Astute sleeps for puppet_deploy_interval seconds, then check Puppet agents
|
||||
# statuses again.
|
||||
puppet_deploy_interval: 2
|
||||
# puppet_fade_timeout is used in puppetd.rb file.
|
||||
# After Puppet agent has finished real work it spend some time to graceful exit.
|
||||
# puppet_fade_timeout means how long (in seconds) Astute can take for Puppet
|
||||
# to exit after real work has finished.
|
||||
puppet_fade_timeout: 120
|
||||
# puppet_fade_interval is used in puppetd.rb file.
|
||||
# Retry every puppet_fade_interval seconds to check puppet state if it was
|
||||
# in 'running' state.
|
||||
puppet_fade_interval: 10
|
120
lib/astute.rb
120
lib/astute.rb
|
@ -1,120 +0,0 @@
|
|||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
require 'astute/ruby_removed_functions'
|
||||
|
||||
require 'json'
|
||||
require 'yaml'
|
||||
require 'logger'
|
||||
require 'shellwords'
|
||||
require 'active_support/all'
|
||||
require 'pp'
|
||||
require 'bunny'
|
||||
require 'zlib'
|
||||
|
||||
require 'astute/ext/array'
|
||||
require 'astute/ext/exception'
|
||||
require 'astute/ext/deep_copy'
|
||||
require 'astute/ext/hash'
|
||||
require 'astute/exceptions'
|
||||
require 'astute/config'
|
||||
require 'astute/logparser'
|
||||
require 'astute/orchestrator'
|
||||
require 'astute/deployment_engine'
|
||||
require 'astute/network'
|
||||
require 'astute/puppetd'
|
||||
require 'astute/provision'
|
||||
require 'astute/deployment_engine/granular_deployment'
|
||||
require 'astute/cobbler'
|
||||
require 'astute/cobbler_manager'
|
||||
require 'astute/image_provision'
|
||||
require 'astute/dump'
|
||||
require 'astute/deploy_actions'
|
||||
require 'astute/nailgun_hooks'
|
||||
require 'astute/puppet_task'
|
||||
require 'astute/puppet_job'
|
||||
require 'astute/task_manager'
|
||||
require 'astute/pre_delete'
|
||||
require 'astute/version'
|
||||
require 'astute/server/async_logger'
|
||||
require 'astute/reporter'
|
||||
require 'astute/mclient'
|
||||
require 'astute/context'
|
||||
require 'astute/nodes_remover'
|
||||
require 'astute/task'
|
||||
require 'astute/task_deployment'
|
||||
require 'astute/task_node'
|
||||
require 'astute/task_proxy_reporter'
|
||||
require 'astute/task_cluster'
|
||||
require 'astute/common/reboot'
|
||||
require 'astute/time_observer'
|
||||
require 'fuel_deployment'
|
||||
|
||||
['/astute/pre_deployment_actions/*.rb',
|
||||
'/astute/pre_deploy_actions/*.rb',
|
||||
'/astute/pre_node_actions/*.rb',
|
||||
'/astute/post_deploy_actions/*.rb',
|
||||
'/astute/post_deployment_actions/*.rb',
|
||||
'/astute/common_actions/*.rb',
|
||||
'/astute/tasks/*.rb',
|
||||
'/astute/mclients/*.rb'
|
||||
].each do |path|
|
||||
Dir[File.dirname(__FILE__) + path].each{ |f| require f }
|
||||
end
|
||||
|
||||
# Server
|
||||
require 'astute/server/worker'
|
||||
require 'astute/server/server'
|
||||
require 'astute/server/producer'
|
||||
require 'astute/server/dispatcher'
|
||||
require 'astute/server/reporter'
|
||||
|
||||
module Astute
|
||||
# Library
|
||||
autoload 'Node', 'astute/node'
|
||||
autoload 'NodesHash', 'astute/node'
|
||||
autoload 'Rsyslogd', 'astute/rsyslogd'
|
||||
LogParser.autoload :ParseDeployLogs, 'astute/logparser/deployment'
|
||||
LogParser.autoload :ParseProvisionLogs, 'astute/logparser/provision'
|
||||
LogParser.autoload :ParseImageBuildLogs, 'astute/logparser/provision'
|
||||
LogParser.autoload :Patterns, 'astute/logparser/parser_patterns'
|
||||
|
||||
LOG_PATH = '/var/log/astute.log'
|
||||
|
||||
def self.logger
|
||||
unless @logger
|
||||
@logger = Logger.new(LOG_PATH)
|
||||
@logger.formatter = proc do |severity, datetime, progname, msg|
|
||||
severity_map = {
|
||||
'DEBUG' => 'DEBUG',
|
||||
'INFO' => 'INFO',
|
||||
'WARN' => 'WARNING',
|
||||
'ERROR' => 'ERROR',
|
||||
'FATAL' => 'CRITICAL'
|
||||
}
|
||||
|
||||
"#{datetime.strftime("%Y-%m-%d %H:%M:%S")} #{severity_map[severity]} [#{Process.pid}] #{msg}\n"
|
||||
end
|
||||
end
|
||||
@logger
|
||||
end
|
||||
|
||||
def self.logger=(logger)
|
||||
@logger = logger
|
||||
Deployment::Log.logger = @logger
|
||||
end
|
||||
|
||||
config_file = '/opt/astute/astute.conf'
|
||||
Astute.config.update(YAML.load(File.read(config_file))) if File.exists?(config_file)
|
||||
end
|
|
@ -1,344 +0,0 @@
|
|||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
require 'xmlrpc/client'
|
||||
|
||||
module Astute
|
||||
module Provision
|
||||
class CobblerError < RuntimeError; end
|
||||
|
||||
class Cobbler
|
||||
|
||||
attr_reader :remote
|
||||
|
||||
def initialize(o={})
|
||||
Astute.logger.debug("Cobbler options:\n#{o.pretty_inspect}")
|
||||
|
||||
if (match = /^http:\/\/([^:]+?):?(\d+)?(\/.+)/.match(o['url']))
|
||||
host = match[1]
|
||||
port = match[2] || '80'
|
||||
path = match[3]
|
||||
else
|
||||
host = o['host'] || 'localhost'
|
||||
port = o['port'] || '80'
|
||||
path = o['path'] || '/cobbler_api'
|
||||
end
|
||||
@username = o['username'] || 'cobbler'
|
||||
@password = o['password'] || 'cobbler'
|
||||
|
||||
Astute.logger.debug("Connecting to cobbler with: host: #{host} port: #{port} path: #{path}")
|
||||
@remote = XMLRPC::Client.new(host, path, port)
|
||||
@remote.timeout = 120
|
||||
Astute.logger.debug("Cobbler initialize with username: #{@username}, password: #{@password}")
|
||||
end
|
||||
|
||||
def token
|
||||
remote.call('login', @username, @password)
|
||||
end
|
||||
|
||||
def item_from_hash(what, name, data, opts = {})
|
||||
options = {
|
||||
:item_preremove => true,
|
||||
}.merge!(opts)
|
||||
cobsh = Cobsh.new(data.merge({'what' => what, 'name' => name}))
|
||||
cobblerized = cobsh.cobblerized
|
||||
|
||||
Astute.logger.debug("Creating/editing item from hash:\n#{cobsh.pretty_inspect}")
|
||||
remove_item(what, name) if options[:item_preremove]
|
||||
# get existent item id or create new one
|
||||
item_id = get_item_id(what, name)
|
||||
|
||||
# defining all item options
|
||||
cobblerized.each do |opt, value|
|
||||
next if opt == 'interfaces'
|
||||
Astute.logger.debug("Setting #{what} #{name} opt: #{opt}=#{value}")
|
||||
remote.call('modify_item', what, item_id, opt, value, token)
|
||||
end
|
||||
|
||||
# defining system interfaces
|
||||
if what == 'system' && cobblerized.has_key?('interfaces')
|
||||
Astute.logger.debug("Defining system interfaces #{name} #{cobblerized['interfaces']}")
|
||||
remote.call('modify_system', item_id, 'modify_interface',
|
||||
cobblerized['interfaces'], token)
|
||||
end
|
||||
|
||||
# save item into cobbler database
|
||||
Astute.logger.debug("Saving #{what} #{name}")
|
||||
remote.call('save_item', what, item_id, token)
|
||||
end
|
||||
|
||||
def remove_item(what, name, recursive=true)
|
||||
remote.call('remove_item', what, name, token, recursive) if item_exists(what, name)
|
||||
end
|
||||
|
||||
def remove_system(name)
|
||||
remove_item('system', name)
|
||||
end
|
||||
|
||||
def item_exists(what, name)
|
||||
remote.call('has_item', what, name)
|
||||
end
|
||||
|
||||
def items_by_criteria(what, criteria)
|
||||
remote.call('find_items', what, criteria)
|
||||
end
|
||||
|
||||
def system_by_mac(mac)
|
||||
items_by_criteria('system', {"mac_address" => mac})[0]
|
||||
end
|
||||
|
||||
def system_exists?(name)
|
||||
item_exists('system', name)
|
||||
end
|
||||
|
||||
def get_item_id(what, name)
|
||||
if item_exists(what, name)
|
||||
item_id = remote.call('get_item_handle', what, name, token)
|
||||
else
|
||||
item_id = remote.call('new_item', what, token)
|
||||
remote.call('modify_item', what, item_id, 'name', name, token)
|
||||
end
|
||||
item_id
|
||||
end
|
||||
|
||||
def sync
|
||||
remote.call('sync', token)
|
||||
rescue Net::ReadTimeout, XMLRPC::FaultException => e
|
||||
retries ||= 0
|
||||
retries += 1
|
||||
raise e if retries > 2
|
||||
|
||||
Astute.logger.warn("Cobbler problem. Try to repeat: #{retries} attempt")
|
||||
sleep 10
|
||||
retry
|
||||
end
|
||||
|
||||
def power(name, action)
|
||||
options = {"systems" => [name], "power" => action}
|
||||
remote.call('background_power_system', options, token)
|
||||
end
|
||||
|
||||
def power_on(name)
|
||||
power(name, 'on')
|
||||
end
|
||||
|
||||
def power_off(name)
|
||||
power(name, 'off')
|
||||
end
|
||||
|
||||
def power_reboot(name)
|
||||
power(name, 'reboot')
|
||||
end
|
||||
|
||||
def event_status(event_id)
|
||||
remote.call('get_task_status', event_id)
|
||||
end
|
||||
|
||||
def netboot(name, state)
|
||||
state = ['on', 'yes', true, 'true', 1, '1'].include?(state)
|
||||
if system_exists?(name)
|
||||
system_id = get_item_id('system', name)
|
||||
else
|
||||
raise CobblerError, "System #{name} not found."
|
||||
end
|
||||
remote.call('modify_system', system_id, 'netboot_enabled', state, token)
|
||||
remote.call('save_system', system_id, token, 'edit')
|
||||
end
|
||||
|
||||
end
|
||||
|
||||
class Cobsh < ::Hash
|
||||
ALIASES = {
|
||||
'ks_meta' => ['ksmeta'],
|
||||
'mac_address' => ['mac'],
|
||||
'ip_address' => ['ip'],
|
||||
}
|
||||
|
||||
# these fields can be get from the cobbler code
|
||||
# you can just import cobbler.item_distro.FIELDS
|
||||
# or cobbler.item_system.FIELDS
|
||||
FIELDS = {
|
||||
'system' => {
|
||||
'fields' => [
|
||||
'name', 'owners', 'profile', 'image', 'status', 'kernel_options',
|
||||
'kernel_options_post', 'ks_meta', 'enable_gpxe', 'proxy',
|
||||
'netboot_enabled', 'kickstart', 'comment', 'server',
|
||||
'virt_path', 'virt_type', 'virt_cpus', 'virt_file_size',
|
||||
'virt_disk_driver', 'virt_ram', 'virt_auto_boot', 'power_type',
|
||||
'power_address', 'power_user', 'power_pass', 'power_id',
|
||||
'hostname', 'gateway', 'name_servers', 'name_servers_search',
|
||||
'ipv6_default_device', 'ipv6_autoconfiguration', 'mgmt_classes',
|
||||
'mgmt_parameters', 'boot_files', 'fetchable_files',
|
||||
'template_files', 'redhat_management_key', 'redhat_management_server',
|
||||
'repos_enabled', 'ldap_enabled', 'ldap_type', 'monit_enabled',
|
||||
],
|
||||
'interfaces_fields' => [
|
||||
'mac_address', 'mtu', 'ip_address', 'interface_type',
|
||||
'interface_master', 'bonding_opts', 'bridge_opts',
|
||||
'management', 'static', 'netmask', 'dhcp_tag', 'dns_name',
|
||||
'static_routes', 'virt_bridge', 'ipv6_address', 'ipv6_secondaries',
|
||||
'ipv6_mtu', 'ipv6_static_routes', 'ipv6_default_gateway'
|
||||
],
|
||||
'special' => ['interfaces', 'interfaces_extra']
|
||||
},
|
||||
'profile' => {
|
||||
'fields' => [
|
||||
'name', 'owners', 'distro', 'parent', 'enable_gpxe',
|
||||
'enable_menu', 'kickstart', 'kernel_options', 'kernel_options_post',
|
||||
'ks_meta', 'proxy', 'repos', 'comment', 'virt_auto_boot',
|
||||
'virt_cpus', 'virt_file_size', 'virt_disk_driver',
|
||||
'virt_ram', 'virt_type', 'virt_path', 'virt_bridge',
|
||||
'dhcp_tag', 'server', 'name_servers', 'name_servers_search',
|
||||
'mgmt_classes', 'mgmt_parameters', 'boot_files', 'fetchable_files',
|
||||
'template_files', 'redhat_management_key', 'redhat_management_server'
|
||||
]
|
||||
},
|
||||
'distro' => {
|
||||
'fields' => ['name', 'owners', 'kernel', 'initrd', 'kernel_options',
|
||||
'kernel_options_post', 'ks_meta', 'arch', 'breed',
|
||||
'os_version', 'comment', 'mgmt_classes', 'boot_files',
|
||||
'fetchable_files', 'template_files', 'redhat_management_key',
|
||||
'redhat_management_server']
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
def initialize(h)
|
||||
Astute.logger.debug("Cobsh is initialized with:\n#{h.pretty_inspect}")
|
||||
raise CobblerError, "Cobbler hash must have 'name' key" unless h.has_key? 'name'
|
||||
raise CobblerError, "Cobbler hash must have 'what' key" unless h.has_key? 'what'
|
||||
raise CobblerError, "Unsupported 'what' value" unless FIELDS.has_key? h['what']
|
||||
h.each{|k, v| store(k, v)}
|
||||
end
|
||||
|
||||
|
||||
def cobblerized
|
||||
Astute.logger.debug("Cobblerizing hash:\n#{pretty_inspect}")
|
||||
ch = {}
|
||||
ks_meta = ''
|
||||
kernel_options = ''
|
||||
|
||||
each do |k, v|
|
||||
k = aliased(k)
|
||||
if ch.has_key?(k) && ch[k] == v
|
||||
next
|
||||
elsif ch.has_key?(k)
|
||||
raise CobblerError, "Wrong cobbler data: #{k} is duplicated"
|
||||
end
|
||||
|
||||
# skiping not valid item options
|
||||
unless valid_field?(k)
|
||||
Astute.logger.warn("Key #{k} is not valid. Will be skipped.")
|
||||
next
|
||||
end
|
||||
|
||||
ks_meta = serialize_cobbler_parameter(v) if 'ks_meta' == k
|
||||
kernel_options = serialize_cobbler_parameter(v) if 'kernel_options' == k
|
||||
|
||||
# special handling for system interface fields
|
||||
# which are the only objects in cobbler that will ever work this way
|
||||
if k == 'interfaces'
|
||||
ch.store('interfaces', cobblerized_interfaces)
|
||||
next
|
||||
end
|
||||
|
||||
# here we convert interfaces_extra options into ks_meta format
|
||||
if k == 'interfaces_extra'
|
||||
ks_meta << cobblerized_interfaces_extra
|
||||
next
|
||||
end
|
||||
|
||||
ch.store(k, v)
|
||||
end # each do |k, v|
|
||||
ch.store('ks_meta', ks_meta.strip) unless ks_meta.strip.empty?
|
||||
ch.store('kernel_options', kernel_options.strip) unless kernel_options.strip.empty?
|
||||
ch
|
||||
end
|
||||
|
||||
def serialize_cobbler_parameter(param)
|
||||
serialized_param = ''
|
||||
if param.kind_of?(Hash)
|
||||
param.each do |ks_meta_key, ks_meta_value|
|
||||
serialized_param << " #{ks_meta_key}=#{serialize_cobbler_value(ks_meta_value)}"
|
||||
end
|
||||
elsif param.kind_of?(String)
|
||||
param
|
||||
else
|
||||
raise CobblerError, "Wrong param format. It must be Hash or String: '#{param}'"
|
||||
end
|
||||
|
||||
serialized_param
|
||||
end
|
||||
|
||||
def serialize_cobbler_value(value)
|
||||
if value.kind_of?(Hash) || value.kind_of?(Array)
|
||||
return "\"#{value.to_json.gsub('"', '\"')}\""
|
||||
end
|
||||
|
||||
value
|
||||
end
|
||||
|
||||
def aliased(k)
|
||||
# converting 'foo-bar' keys into 'foo_bar' keys
|
||||
k1 = k.gsub(/-/,'_')
|
||||
# converting orig keys into alias keys
|
||||
# example: 'ksmeta' into 'ks_meta'
|
||||
k2 = ALIASES.each_key.select{|ak| ALIASES[ak].include?(k1)}[0] || k1
|
||||
Astute.logger.debug("Key #{k} aliased with #{k2}") if k != k2
|
||||
k2
|
||||
end
|
||||
|
||||
def valid_field?(k)
|
||||
(FIELDS[fetch('what')]['fields'].include?(k) or
|
||||
(FIELDS[fetch('what')]['special'] or []).include?(k))
|
||||
end
|
||||
|
||||
def valid_interface_field?(k)
|
||||
(FIELDS[fetch('what')]['interfaces_fields'] or []).include?(k)
|
||||
end
|
||||
|
||||
def cobblerized_interfaces
|
||||
interfaces = {}
|
||||
fetch('interfaces').each do |iname, ihash|
|
||||
ihash.each do |iopt, ivalue|
|
||||
iopt = aliased(iopt)
|
||||
if interfaces.has_key?("#{iopt}-#{iname}")
|
||||
raise CobblerError, "Wrong interface cobbler data: #{iopt} is duplicated"
|
||||
end
|
||||
unless valid_interface_field?(iopt)
|
||||
Astute.logger.debug("Interface key #{iopt} is not valid. Skipping")
|
||||
next
|
||||
end
|
||||
Astute.logger.debug("Defining interfaces[#{iopt}-#{iname}] = #{ivalue}")
|
||||
interfaces["#{iopt}-#{iname}"] = ivalue
|
||||
end
|
||||
end
|
||||
interfaces
|
||||
end
|
||||
|
||||
def cobblerized_interfaces_extra
|
||||
# here we just want to convert interfaces_extra into ks_meta
|
||||
interfaces_extra_str = ""
|
||||
fetch('interfaces_extra').each do |iname, iextra|
|
||||
iextra.each do |k, v|
|
||||
Astute.logger.debug("Adding into ks_meta interface_extra_#{iname}_#{k}=#{v}")
|
||||
interfaces_extra_str << " interface_extra_#{iname}_#{k}=#{v}"
|
||||
end
|
||||
end
|
||||
interfaces_extra_str
|
||||
end
|
||||
end
|
||||
|
||||
end
|
||||
end
|
|
@ -1,260 +0,0 @@
|
|||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
class CobblerManager
|
||||
def initialize(engine_attrs, reporter)
|
||||
raise "Settings for Cobbler must be set" if engine_attrs.blank?
|
||||
|
||||
begin
|
||||
Astute.logger.debug("Trying to instantiate cobbler engine:"\
|
||||
"\n#{engine_attrs.pretty_inspect}")
|
||||
@engine = Astute::Provision::Cobbler.new(engine_attrs)
|
||||
rescue => e
|
||||
Astute.logger.error("Error occured during cobbler initializing")
|
||||
reporter.report({
|
||||
'status' => 'error',
|
||||
'error' => 'Cobbler can not be initialized',
|
||||
'progress' => 100
|
||||
})
|
||||
raise e
|
||||
end
|
||||
end
|
||||
|
||||
def add_nodes(nodes)
|
||||
nodes.each do |node|
|
||||
cobbler_name = node['slave_name']
|
||||
begin
|
||||
Astute.logger.info("Adding #{cobbler_name} into cobbler")
|
||||
@engine.item_from_hash('system', cobbler_name, node, :item_preremove => true)
|
||||
rescue RuntimeError => e
|
||||
Astute.logger.error("Error occured while adding system #{cobbler_name} to cobbler")
|
||||
raise e
|
||||
end
|
||||
end
|
||||
ensure
|
||||
sync
|
||||
end
|
||||
|
||||
def remove_nodes(nodes, retries=3, interval=2)
|
||||
nodes_to_remove = nodes.map {|node| node['slave_name']}.uniq
|
||||
Astute.logger.info("Total list of nodes to remove: #{nodes_to_remove.pretty_inspect}")
|
||||
retries.times do
|
||||
nodes_to_remove.select! do |name|
|
||||
unless @engine.system_exists?(name)
|
||||
Astute.logger.info("System is not in cobbler: #{name}")
|
||||
next
|
||||
else
|
||||
Astute.logger.info("Trying to remove system from cobbler: #{name}")
|
||||
@engine.remove_system(name)
|
||||
end
|
||||
@engine.system_exists?(name)
|
||||
end
|
||||
return if nodes_to_remove.empty?
|
||||
sleep(interval) if interval > 0
|
||||
end
|
||||
ensure
|
||||
Astute.logger.error("Cannot remove nodes from cobbler: #{nodes_to_remove.pretty_inspect}") if nodes_to_remove.present?
|
||||
sync
|
||||
end
|
||||
|
||||
def reboot_nodes(nodes)
|
||||
splay = calculate_splay_between_nodes(nodes)
|
||||
nodes.inject({}) do |reboot_events, node|
|
||||
cobbler_name = node['slave_name']
|
||||
Astute.logger.debug("Trying to reboot node: #{cobbler_name}")
|
||||
|
||||
#Sleep up to splay seconds before reboot for load balancing
|
||||
sleep splay
|
||||
reboot_events.merge(cobbler_name => @engine.power_reboot(cobbler_name))
|
||||
end
|
||||
end
|
||||
|
||||
def check_reboot_nodes(reboot_events)
|
||||
begin
|
||||
Astute.logger.debug("Waiting for reboot to be complete: nodes: #{reboot_events.keys}")
|
||||
failed_nodes = []
|
||||
Timeout::timeout(Astute.config.reboot_timeout) do
|
||||
while not reboot_events.empty?
|
||||
reboot_events.each do |node_name, event_id|
|
||||
event_status = @engine.event_status(event_id)
|
||||
Astute.logger.debug("Reboot task status: node: #{node_name} status: #{event_status}")
|
||||
if event_status[2] =~ /^failed$/
|
||||
Astute.logger.error("Error occured while trying to reboot: #{node_name}")
|
||||
reboot_events.delete(node_name)
|
||||
failed_nodes << node_name
|
||||
elsif event_status[2] =~ /^complete$/
|
||||
Astute.logger.debug("Successfully rebooted: #{node_name}")
|
||||
reboot_events.delete(node_name)
|
||||
end
|
||||
end
|
||||
sleep(5)
|
||||
end
|
||||
end
|
||||
rescue Timeout::Error => e
|
||||
Astute.logger.debug("Reboot timeout: reboot tasks not completed for nodes #{reboot_events.keys}")
|
||||
raise e
|
||||
end
|
||||
failed_nodes
|
||||
end
|
||||
|
||||
def edit_nodes(nodes, data)
|
||||
nodes.each do |node|
|
||||
cobbler_name = node['slave_name']
|
||||
begin
|
||||
Astute.logger.info("Changing cobbler system #{cobbler_name}")
|
||||
@engine.item_from_hash('system', cobbler_name, data, :item_preremove => false)
|
||||
rescue RuntimeError => e
|
||||
Astute.logger.error("Error occured while changing cobbler system #{cobbler_name}")
|
||||
raise e
|
||||
end
|
||||
end
|
||||
ensure
|
||||
sync
|
||||
end
|
||||
|
||||
def netboot_nodes(nodes, state)
|
||||
nodes.each do |node|
|
||||
cobbler_name = node['slave_name']
|
||||
begin
|
||||
Astute.logger.info("Changing node netboot state #{cobbler_name}")
|
||||
@engine.netboot(cobbler_name, state)
|
||||
rescue RuntimeError => e
|
||||
Astute.logger.error("Error while changing node netboot state #{cobbler_name}")
|
||||
raise e
|
||||
end
|
||||
end
|
||||
ensure
|
||||
sync
|
||||
end
|
||||
|
||||
def get_existent_nodes(nodes)
|
||||
existent_nodes = []
|
||||
nodes.each do |node|
|
||||
cobbler_name = node['slave_name']
|
||||
if @engine.system_exists?(cobbler_name)
|
||||
Astute.logger.info("Update #{cobbler_name}, node already exists in cobbler")
|
||||
existent_nodes << node
|
||||
end
|
||||
end
|
||||
existent_nodes
|
||||
end
|
||||
|
||||
def existent_node?(cobbler_name)
|
||||
return false unless @engine.system_exists?(cobbler_name)
|
||||
Astute.logger.info("Node #{cobbler_name} already exists in cobbler")
|
||||
true
|
||||
end
|
||||
|
||||
def edit_node(cobbler_name, data)
|
||||
begin
|
||||
Astute.logger.info("Changing cobbler system #{cobbler_name}")
|
||||
@engine.item_from_hash('system', cobbler_name, data, :item_preremove => false)
|
||||
rescue RuntimeError => e
|
||||
Astute.logger.error("Error occured while changing cobbler system #{cobbler_name}")
|
||||
raise e
|
||||
end
|
||||
ensure
|
||||
sync
|
||||
end
|
||||
|
||||
def netboot_node(cobbler_name, state)
|
||||
begin
|
||||
Astute.logger.info("Changing node netboot state #{cobbler_name}")
|
||||
@engine.netboot(cobbler_name, state)
|
||||
rescue RuntimeError => e
|
||||
Astute.logger.error("Error while changing node netboot state #{cobbler_name}")
|
||||
raise e
|
||||
end
|
||||
ensure
|
||||
sync
|
||||
end
|
||||
|
||||
def remove_node(cobbler_name, retries=3, interval=2)
|
||||
Astute.logger.info("Node to remove: #{cobbler_name}")
|
||||
retries.times do
|
||||
unless @engine.system_exists?(cobbler_name)
|
||||
Astute.logger.info("System is not in cobbler: #{cobbler_name}")
|
||||
return
|
||||
else
|
||||
Astute.logger.info("Trying to remove system from cobbler: #{cobbler_name}")
|
||||
@engine.remove_system(cobbler_name)
|
||||
end
|
||||
return unless @engine.system_exists?(cobbler_name)
|
||||
sleep(interval) if interval > 0
|
||||
end
|
||||
ensure
|
||||
Astute.logger.error("Cannot remove node #{cobbler_name} from cobbler") if @engine.system_exists?(cobbler_name)
|
||||
sync
|
||||
end
|
||||
|
||||
def add_node(node)
|
||||
cobbler_name = node['slave_name']
|
||||
begin
|
||||
Astute.logger.info("Adding #{cobbler_name} into cobbler")
|
||||
@engine.item_from_hash('system', cobbler_name, node, :item_preremove => true)
|
||||
rescue RuntimeError => e
|
||||
Astute.logger.error("Error occured while adding system #{cobbler_name} to cobbler")
|
||||
raise e
|
||||
end
|
||||
ensure
|
||||
sync
|
||||
end
|
||||
|
||||
def node_mac_duplicate_names(node)
|
||||
mac_duplicate_names = []
|
||||
Astute.logger.info("Trying to find MAC duplicates for node #{node['slave_name']}")
|
||||
if node['interfaces']
|
||||
node['interfaces'].each do |iname, ihash|
|
||||
if ihash['mac_address']
|
||||
Astute.logger.info("Trying to find system with MAC: #{ihash['mac_address']}")
|
||||
found_node = @engine.system_by_mac(ihash['mac_address'])
|
||||
mac_duplicate_names << found_node['name'] if found_node
|
||||
end
|
||||
end
|
||||
end
|
||||
mac_duplicate_names.uniq
|
||||
end
|
||||
|
||||
def get_mac_duplicate_names(nodes)
|
||||
mac_duplicate_names = []
|
||||
nodes.each do |node|
|
||||
Astute.logger.info("Trying to find MAC duplicates for node #{node['slave_name']}")
|
||||
if node['interfaces']
|
||||
node['interfaces'].each do |iname, ihash|
|
||||
if ihash['mac_address']
|
||||
Astute.logger.info("Trying to find system with MAC: #{ihash['mac_address']}")
|
||||
found_node = @engine.system_by_mac(ihash['mac_address'])
|
||||
mac_duplicate_names << found_node['name'] if found_node
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
mac_duplicate_names.uniq
|
||||
end
|
||||
|
||||
def sync
|
||||
Astute.logger.debug("Cobbler syncing")
|
||||
@engine.sync
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def calculate_splay_between_nodes(nodes)
|
||||
# For 20 nodes, 120 iops and 180 splay_factor splay will be 1.5749
|
||||
(nodes.size + 1) / Astute.config.iops.to_f * Astute.config.splay_factor / nodes.size
|
||||
end
|
||||
|
||||
end
|
||||
end
|
|
@ -1,38 +0,0 @@
|
|||
# Copyright 2014 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
|
||||
module Astute
|
||||
module RebootCommand
|
||||
# Reboot immediately if we're in a bootstrap. Wait until system boots
|
||||
# completely in case of provisioned node. We check it by existense
|
||||
# of /run/cloud-init/status.json (it's located on tmpfs, so no stale
|
||||
# file from previous boot can be found). If this file hasn't appeared
|
||||
# after 60 seconds - reboot as is.
|
||||
CMD = <<-REBOOT_COMMAND
|
||||
if [ $(hostname) = bootstrap ]; then
|
||||
reboot;
|
||||
fi;
|
||||
t=0;
|
||||
while true; do
|
||||
if [ -f /run/cloud-init/status.json -o $t -gt 60 ]; then
|
||||
reboot;
|
||||
else
|
||||
sleep 1;
|
||||
t=$((t + 1));
|
||||
fi;
|
||||
done
|
||||
REBOOT_COMMAND
|
||||
end
|
||||
end
|
|
@ -1,90 +0,0 @@
|
|||
# Copyright 2014 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
class Pacemaker
|
||||
|
||||
def self.commands(behavior, deployment_info)
|
||||
return [] if deployment_info.first['deployment_mode'] !~ /ha/i
|
||||
|
||||
controller_nodes = deployment_info.select{ |n| n['role'] =~ /controller/i }.map{ |n| n['uid'] }
|
||||
return [] if controller_nodes.empty?
|
||||
|
||||
ha_size = deployment_info.first['nodes'].count { |n|
|
||||
['controller', 'primary-controller'].include? n['role']
|
||||
}
|
||||
|
||||
action = if ha_size < 3
|
||||
case behavior
|
||||
when 'stop' then 'stop'
|
||||
when 'start' then 'start'
|
||||
end
|
||||
else
|
||||
case behavior
|
||||
when 'stop' then 'ban'
|
||||
when 'start' then 'clear'
|
||||
end
|
||||
end
|
||||
|
||||
cmds = pacemaker_services_list(deployment_info).inject([]) do |cmds, pacemaker_service|
|
||||
if ha_size < 3
|
||||
cmds << "crm resource #{action} #{pacemaker_service} && sleep 3"
|
||||
else
|
||||
cmds << "pcs resource #{action} #{pacemaker_service} `crm_node -n` && sleep 3"
|
||||
end
|
||||
end
|
||||
|
||||
cmds
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def self.pacemaker_services_list(deployment_info)
|
||||
services_list = []
|
||||
#Heat engine service is present everywhere
|
||||
services_list += heat_service_name(deployment_info)
|
||||
|
||||
if deployment_info.first['quantum']
|
||||
services_list << 'p_neutron-openvswitch-agent'
|
||||
services_list << 'p_neutron-metadata-agent'
|
||||
services_list << 'p_neutron-l3-agent'
|
||||
services_list << 'p_neutron-dhcp-agent'
|
||||
end
|
||||
|
||||
if deployment_info.first.fetch('ceilometer', {})['enabled']
|
||||
services_list += ceilometer_service_names(deployment_info)
|
||||
end
|
||||
return services_list
|
||||
end
|
||||
|
||||
def self.ceilometer_service_names(deployment_info)
|
||||
case deployment_info.first['cobbler']['profile']
|
||||
when /centos/i
|
||||
['p_openstack-ceilometer-compute','p_openstack-ceilometer-central']
|
||||
when /ubuntu/i
|
||||
['p_ceilometer-agent-central','p_ceilometer-agent-compute']
|
||||
end
|
||||
end
|
||||
|
||||
def self.heat_service_name(deployment_info)
|
||||
case deployment_info.first['cobbler']['profile']
|
||||
when /centos/i
|
||||
['openstack-heat-engine', 'p_openstack-heat-engine']
|
||||
when /ubuntu/i
|
||||
['heat-engine', 'p_heat-engine']
|
||||
end
|
||||
end
|
||||
|
||||
end #class
|
||||
end
|
|
@ -1,130 +0,0 @@
|
|||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
|
||||
require 'symboltable'
|
||||
require 'singleton'
|
||||
|
||||
module Astute
|
||||
class ConfigError < StandardError; end
|
||||
class UnknownOptionError < ConfigError
|
||||
attr_reader :name
|
||||
|
||||
def initialize(name)
|
||||
super("Unknown config option #{name}")
|
||||
@name = name
|
||||
end
|
||||
end
|
||||
|
||||
class MyConfig
|
||||
include Singleton
|
||||
attr_reader :configtable
|
||||
|
||||
def initialize
|
||||
@configtable = SymbolTable.new
|
||||
end
|
||||
end
|
||||
|
||||
class ParseError < ConfigError
|
||||
attr_reader :line
|
||||
|
||||
def initialize(message, line)
|
||||
super(message)
|
||||
@line = line
|
||||
end
|
||||
end
|
||||
|
||||
def self.config
|
||||
config = MyConfig.instance.configtable
|
||||
config.update(default_config) if config.empty?
|
||||
return config
|
||||
end
|
||||
|
||||
def self.default_config
|
||||
conf = {}
|
||||
|
||||
# Library settings
|
||||
conf[:puppet_timeout] = 90 * 60 # maximum time it waits for single puppet run
|
||||
conf[:puppet_deploy_interval] = 2 # sleep for ## sec, then check puppet status again
|
||||
conf[:puppet_fade_timeout] = 120 # how long it can take for puppet to exit after dumping to last_run_summary
|
||||
conf[:puppet_start_timeout] = 10 # how long it can take for puppet to start
|
||||
conf[:puppet_start_interval] = 2 # interval between attemps to start puppet
|
||||
conf[:puppet_retries] = 2 # how many times astute will try to run puppet
|
||||
conf[:upload_retries] = 3 # how many times astute will try to run upload task
|
||||
conf[:puppet_succeed_retries] = 0 # use this to rerun a puppet task again if it was successful (idempotency)
|
||||
conf[:puppet_undefined_retries] = 3 # how many times astute will try to get actual status of node before fail
|
||||
conf[:puppet_module_path] = '/etc/puppet/modules' # where we should find basic modules for puppet
|
||||
conf[:puppet_noop_run] = false # enable Puppet noop run
|
||||
conf[:mc_retries] = 10 # MClient tries to call mcagent before failure
|
||||
conf[:mc_retry_interval] = 1 # MClient sleeps for ## sec between retries
|
||||
conf[:puppet_fade_interval] = 30 # retry every ## seconds to check puppet state if it was running
|
||||
conf[:provisioning_timeout] = 90 * 60 # timeout for booting target OS in provision
|
||||
conf[:reboot_timeout] = 900 # how long it can take for node to reboot
|
||||
conf[:dump_timeout] = 3600 # maximum time it waits for the dump (meaningles to be larger
|
||||
# than the specified in timeout of execute_shell_command mcagent
|
||||
conf[:shell_retries] = 2 # default retries for shell task
|
||||
conf[:shell_interval] = 2 # default interval for shell task
|
||||
conf[:shell_timeout] = 300 # default timeout for shell task
|
||||
conf[:upload_timeout] = 60 # default timeout for upload task
|
||||
conf[:shell_cwd] = '/' # default cwd for shell task
|
||||
conf[:stop_timeout] = 600 # how long it can take for stop
|
||||
conf[:rsync_options] = '-c -r --delete -l' # default rsync options
|
||||
conf[:keys_src_dir] = '/var/lib/fuel/keys' # path where ssh and openssl keys will be created
|
||||
conf[:puppet_ssh_keys] = [
|
||||
'neutron',
|
||||
'nova',
|
||||
'ceph',
|
||||
'mysql',
|
||||
] # name of ssh keys what will be generated and uploaded to all nodes before deploy
|
||||
conf[:puppet_keys] = [
|
||||
'mongodb'
|
||||
] # name of keys what will be generated and uploaded to all nodes before deploy
|
||||
conf[:keys_dst_dir] = '/var/lib/astute' # folder where keys will be uploaded. Warning!
|
||||
conf[:max_nodes_per_call] = 50 # how many nodes to deploy simultaneously
|
||||
conf[:max_nodes_to_provision] = 50 # how many nodes to provision simultaneously
|
||||
conf[:ssh_retry_timeout] = 30 # SSH sleeps for ## sec between retries
|
||||
|
||||
conf[:max_nodes_per_remove_call] = 10 # how many nodes to remove in one call
|
||||
conf[:nodes_remove_interval] = 10 # sleeps for ## sec between remove calls
|
||||
conf[:max_nodes_net_validation] = 10 # how many nodes will send in parallel test packets
|
||||
# during network verification
|
||||
conf[:dhcp_repeat] = 3 # Dhcp discover will be sended 3 times
|
||||
|
||||
conf[:iops] = 120 # Default IOPS master node IOPS performance
|
||||
conf[:splay_factor] = 180 # Formula: 20(amount of nodes nodes) div 120(iops) = 0.1667
|
||||
# 0.1667 / 180 = 30 sec. Delay between reboot command for first
|
||||
# and last node in group should be 30 sec. Empirical observation.
|
||||
# Please increase if nodes could not provisioning
|
||||
conf[:agent_nodiscover_file] = '/etc/nailgun-agent/nodiscover' # if this file in place, nailgun-agent will do nothing
|
||||
conf[:bootstrap_profile] = 'ubuntu_bootstrap' # use the Ubuntu based bootstrap by default
|
||||
conf[:graph_dot_dir] = "/var/lib/astute/graphs" # default dir patch for debug graph file
|
||||
conf[:enable_graph_file] = true # enable debug graph records to file
|
||||
conf[:puppet_raw_report] = false # enable puppet detailed report
|
||||
conf[:task_poll_delay] = 1 # sleeps for ## sec between task status calls
|
||||
|
||||
# Server settings
|
||||
conf[:broker_host] = 'localhost'
|
||||
conf[:broker_port] = 5672
|
||||
conf[:broker_rest_api_port] = 15672
|
||||
conf[:broker_username] = 'mcollective'
|
||||
conf[:broker_password] = 'mcollective'
|
||||
|
||||
conf[:broker_service_exchange] = 'naily_service'
|
||||
conf[:broker_queue] = 'naily'
|
||||
conf[:broker_publisher_queue] = 'nailgun'
|
||||
conf[:broker_exchange] = 'nailgun'
|
||||
|
||||
conf
|
||||
end
|
||||
end
|
|
@ -1,43 +0,0 @@
|
|||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
|
||||
module Astute
|
||||
class Context
|
||||
attr_accessor :reporter, :deploy_log_parser
|
||||
attr_reader :task_id, :status
|
||||
|
||||
def initialize(task_id, reporter, deploy_log_parser=nil)
|
||||
@task_id = task_id
|
||||
@reporter = reporter
|
||||
@status = {}
|
||||
@deploy_log_parser = deploy_log_parser
|
||||
end
|
||||
|
||||
def report_and_update_status(data)
|
||||
if data['nodes']
|
||||
data['nodes'].each do |node|
|
||||
#TODO(vsharshov): save node role to hash
|
||||
@status.merge! node['uid'] => node['status'] if node['uid'] && node['status']
|
||||
end
|
||||
end
|
||||
reporter.report(data)
|
||||
end
|
||||
|
||||
def report(msg)
|
||||
@reporter.report msg
|
||||
end
|
||||
|
||||
end
|
||||
end
|
|
@ -1,215 +0,0 @@
|
|||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
class DeployActions
|
||||
|
||||
def initialize(deployment_info, context)
|
||||
@deployment_info = deployment_info
|
||||
@context = context
|
||||
@actions = []
|
||||
end
|
||||
|
||||
def process
|
||||
@actions.each { |action| action.process(@deployment_info, @context) }
|
||||
end
|
||||
end
|
||||
|
||||
class PreDeployActions < DeployActions
|
||||
def initialize(deployment_info, context)
|
||||
super
|
||||
@actions = [
|
||||
ConnectFacts.new
|
||||
]
|
||||
end
|
||||
end
|
||||
|
||||
class GranularPreDeployActions < DeployActions
|
||||
def initialize(deployment_info, context)
|
||||
super
|
||||
@actions = [
|
||||
ConnectFacts.new
|
||||
]
|
||||
end
|
||||
end
|
||||
|
||||
class PostDeployActions < DeployActions
|
||||
def initialize(deployment_info, context)
|
||||
super
|
||||
@actions = [
|
||||
PostPatchingHa.new
|
||||
]
|
||||
end
|
||||
end
|
||||
|
||||
class GranularPostDeployActions < DeployActions
|
||||
def initialize(deployment_info, context)
|
||||
super
|
||||
@actions = [
|
||||
PostPatchingHa.new
|
||||
]
|
||||
end
|
||||
end
|
||||
|
||||
class PreNodeActions
|
||||
|
||||
def initialize(context)
|
||||
@node_uids = []
|
||||
@context = context
|
||||
@actions = [
|
||||
PrePatchingHa.new,
|
||||
StopOSTServices.new,
|
||||
PrePatching.new
|
||||
]
|
||||
end
|
||||
|
||||
def process(deployment_info)
|
||||
nodes_to_process = deployment_info.select { |n| !@node_uids.include?(n['uid']) }
|
||||
return if nodes_to_process.empty?
|
||||
|
||||
@actions.each { |action| action.process(nodes_to_process, @context) }
|
||||
@node_uids += nodes_to_process.map { |n| n['uid'] }
|
||||
end
|
||||
end
|
||||
|
||||
class GranularPreNodeActions
|
||||
|
||||
def initialize(context)
|
||||
@node_uids = []
|
||||
@context = context
|
||||
@actions = [
|
||||
PrePatchingHa.new,
|
||||
StopOSTServices.new,
|
||||
PrePatching.new
|
||||
]
|
||||
end
|
||||
|
||||
def process(deployment_info)
|
||||
nodes_to_process = deployment_info.select { |n| !@node_uids.include?(n['uid']) }
|
||||
return if nodes_to_process.empty?
|
||||
|
||||
@actions.each { |action| action.process(nodes_to_process, @context) }
|
||||
@node_uids += nodes_to_process.map { |n| n['uid'] }
|
||||
end
|
||||
end
|
||||
|
||||
class PreDeploymentActions < DeployActions
|
||||
|
||||
def initialize(deployment_info, context)
|
||||
super
|
||||
@actions = [
|
||||
SyncTime.new,
|
||||
GenerateSshKeys.new,
|
||||
GenerateKeys.new,
|
||||
UploadSshKeys.new,
|
||||
UploadKeys.new,
|
||||
UpdateRepoSources.new,
|
||||
SyncPuppetStuff.new,
|
||||
SyncTasks.new,
|
||||
EnablePuppetDeploy.new,
|
||||
UploadFacts.new,
|
||||
InitialConnectFacts.new
|
||||
]
|
||||
end
|
||||
|
||||
end
|
||||
|
||||
class GranularPreDeploymentActions < DeployActions
|
||||
|
||||
def initialize(deployment_info, context)
|
||||
super
|
||||
@actions = [
|
||||
EnablePuppetDeploy.new,
|
||||
UploadFacts.new,
|
||||
InitialConnectFacts.new
|
||||
]
|
||||
end
|
||||
|
||||
end
|
||||
|
||||
class TaskPreDeploymentActions < DeployActions
|
||||
|
||||
def initialize(deployment_info, context)
|
||||
super
|
||||
@actions = [
|
||||
EnablePuppetDeploy.new,
|
||||
UploadFacts.new,
|
||||
InitialConnectFacts.new
|
||||
]
|
||||
end
|
||||
|
||||
end
|
||||
|
||||
class PostDeploymentActions < DeployActions
|
||||
|
||||
def initialize(deployment_info, context)
|
||||
super
|
||||
@actions = [
|
||||
UpdateNoQuorumPolicy.new,
|
||||
UploadCirrosImage.new,
|
||||
RestartRadosgw.new,
|
||||
UpdateClusterHostsInfo.new
|
||||
]
|
||||
|
||||
end
|
||||
end
|
||||
|
||||
class DeployAction
|
||||
|
||||
def process(deployment_info, context)
|
||||
raise "Should be implemented!"
|
||||
end
|
||||
|
||||
def run_shell_command(context, node_uids, cmd, timeout=60)
|
||||
shell = MClient.new(context,
|
||||
'execute_shell_command',
|
||||
node_uids,
|
||||
check_result=true,
|
||||
timeout=timeout,
|
||||
retries=1)
|
||||
|
||||
#TODO: return result for all nodes not only for first
|
||||
response = shell.execute(:cmd => cmd).first
|
||||
Astute.logger.debug("#{context.task_id}: cmd: #{cmd}
|
||||
stdout: #{response[:data][:stdout]}
|
||||
stderr: #{response[:data][:stderr]}
|
||||
exit code: #{response[:data][:exit_code]}")
|
||||
response
|
||||
rescue MClientTimeout, MClientError => e
|
||||
Astute.logger.error("#{context.task_id}: cmd: #{cmd}
|
||||
mcollective error: #{e.message}")
|
||||
{:data => {}}
|
||||
end
|
||||
|
||||
def only_uniq_nodes(nodes)
|
||||
nodes.uniq { |n| n['uid'] }
|
||||
end
|
||||
|
||||
# Prevent high load for tasks
|
||||
def perform_with_limit(nodes, &block)
|
||||
nodes.each_slice(Astute.config[:max_nodes_per_call]) do |part|
|
||||
block.call(part)
|
||||
end
|
||||
end
|
||||
|
||||
end # DeployAction
|
||||
|
||||
class PreDeployAction < DeployAction; end
|
||||
class PostDeployAction < DeployAction; end
|
||||
class PreNodeAction < DeployAction; end
|
||||
class PostNodeAction < DeployAction; end
|
||||
class PreDeploymentAction < DeployAction; end
|
||||
class PostDeploymentAction < DeployAction; end
|
||||
|
||||
end
|
|
@ -1,269 +0,0 @@
|
|||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
class DeploymentEngine
|
||||
|
||||
def initialize(context)
|
||||
if self.class.superclass.name == 'Object'
|
||||
raise "Instantiation of this superclass is not allowed. Please subclass from #{self.class.name}."
|
||||
end
|
||||
@ctx = context
|
||||
end
|
||||
|
||||
def deploy(deployment_info, pre_deployment=[], post_deployment=[])
|
||||
raise "Deployment info are not provided!" if deployment_info.blank?
|
||||
|
||||
deployment_info, pre_deployment, post_deployment = remove_failed_nodes(deployment_info,
|
||||
pre_deployment,
|
||||
post_deployment)
|
||||
|
||||
@ctx.deploy_log_parser.deploy_type = deployment_info.first['deployment_mode']
|
||||
Astute.logger.info "Deployment mode #{@ctx.deploy_log_parser.deploy_type}"
|
||||
|
||||
begin
|
||||
pre_deployment_actions(deployment_info, pre_deployment)
|
||||
rescue => e
|
||||
Astute.logger.error("Unexpected error #{e.message} traceback #{e.format_backtrace}")
|
||||
raise e
|
||||
end
|
||||
|
||||
failed = []
|
||||
# Sort by priority (the lower the number, the higher the priority)
|
||||
# and send groups to deploy
|
||||
deployment_info.sort_by { |f| f['priority'] }.group_by{ |f| f['priority'] }.each do |_, nodes|
|
||||
# Prevent attempts to run several deploy on a single node.
|
||||
# This is possible because one node
|
||||
# can perform multiple roles.
|
||||
group_by_uniq_values(nodes).each do |nodes_group|
|
||||
# Prevent deploy too many nodes at once
|
||||
nodes_group.each_slice(Astute.config[:max_nodes_per_call]) do |part|
|
||||
|
||||
# for each chunk run group deployment pipeline
|
||||
|
||||
# create links to the astute.yaml
|
||||
pre_deploy_actions(part)
|
||||
|
||||
# run group deployment
|
||||
deploy_piece(part)
|
||||
|
||||
failed = critical_failed_nodes(part)
|
||||
|
||||
# if any of the node are critical and failed
|
||||
# raise an error and mark all other nodes as error
|
||||
if failed.any?
|
||||
# TODO(dshulyak) maybe we should print all failed tasks for this nodes
|
||||
# but i am not sure how it will look like
|
||||
raise Astute::DeploymentEngineError, "Deployment failed on nodes #{failed.join(', ')}"
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
# Post deployment hooks
|
||||
post_deployment_actions(deployment_info, post_deployment)
|
||||
end
|
||||
|
||||
protected
|
||||
|
||||
def validate_nodes(nodes)
|
||||
return true unless nodes.empty?
|
||||
|
||||
Astute.logger.info "#{@ctx.task_id}: Nodes to deploy are not provided. Do nothing."
|
||||
false
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
# Transform nodes source array to array of nodes arrays where subarray
|
||||
# contain only uniq elements from source
|
||||
# Source: [
|
||||
# {'uid' => 1, 'role' => 'cinder'},
|
||||
# {'uid' => 2, 'role' => 'cinder'},
|
||||
# {'uid' => 2, 'role' => 'compute'}]
|
||||
# Result: [
|
||||
# [{'uid' =>1, 'role' => 'cinder'},
|
||||
# {'uid' => 2, 'role' => 'cinder'}],
|
||||
# [{'uid' => 2, 'role' => 'compute'}]]
|
||||
def group_by_uniq_values(nodes_array)
|
||||
nodes_array = deep_copy(nodes_array)
|
||||
sub_arrays = []
|
||||
while !nodes_array.empty?
|
||||
sub_arrays << uniq_nodes(nodes_array)
|
||||
uniq_nodes(nodes_array).clone.each {|e| nodes_array.slice!(nodes_array.index(e)) }
|
||||
end
|
||||
sub_arrays
|
||||
end
|
||||
|
||||
def uniq_nodes(nodes_array)
|
||||
nodes_array.inject([]) { |result, node| result << node unless include_node?(result, node); result }
|
||||
end
|
||||
|
||||
def include_node?(nodes_array, node)
|
||||
nodes_array.find { |n| node['uid'] == n['uid'] }
|
||||
end
|
||||
|
||||
def nodes_status(nodes, status, data_to_merge)
|
||||
{
|
||||
'nodes' => nodes.map do |n|
|
||||
{'uid' => n['uid'], 'status' => status, 'role' => n['role']}.merge(data_to_merge)
|
||||
end
|
||||
}
|
||||
end
|
||||
|
||||
def critical_failed_nodes(part)
|
||||
part.select{ |n| n['fail_if_error'] }.map{ |n| n['uid'] } &
|
||||
@ctx.status.select { |k, v| v == 'error' }.keys
|
||||
end
|
||||
|
||||
def pre_deployment_actions(deployment_info, pre_deployment)
|
||||
raise "Should be implemented"
|
||||
end
|
||||
|
||||
def pre_node_actions(part)
|
||||
raise "Should be implemented"
|
||||
end
|
||||
|
||||
def pre_deploy_actions(part)
|
||||
raise "Should be implemented"
|
||||
end
|
||||
|
||||
def post_deploy_actions(part)
|
||||
raise "Should be implemented"
|
||||
end
|
||||
|
||||
def post_deployment_actions(deployment_info, post_deployment)
|
||||
raise "Should be implemented"
|
||||
end
|
||||
|
||||
# Removes nodes which failed to provision
|
||||
def remove_failed_nodes(deployment_info, pre_deployment, post_deployment)
|
||||
uids = get_uids_from_deployment_info deployment_info
|
||||
required_nodes = deployment_info.select { |node| node["fail_if_error"] }
|
||||
required_uids = required_nodes.map { |node| node["uid"]}
|
||||
|
||||
available_uids = detect_available_nodes(uids)
|
||||
offline_uids = uids - available_uids
|
||||
if offline_uids.present?
|
||||
# set status for all failed nodes to error
|
||||
nodes = (uids - available_uids).map do |uid|
|
||||
{'uid' => uid,
|
||||
'status' => 'error',
|
||||
'error_type' => 'provision',
|
||||
# Avoid deployment reporter param validation
|
||||
'role' => 'hook',
|
||||
'error_msg' => 'Node is not ready for deployment: mcollective has not answered'
|
||||
}
|
||||
end
|
||||
|
||||
@ctx.report_and_update_status('nodes' => nodes, 'error' => 'Node is not ready for deployment')
|
||||
|
||||
# check if all required nodes are online
|
||||
# if not, raise error
|
||||
missing_required = required_uids - available_uids
|
||||
if missing_required.present?
|
||||
error_message = "Critical nodes are not available for deployment: #{missing_required}"
|
||||
raise Astute::DeploymentEngineError, error_message
|
||||
end
|
||||
end
|
||||
|
||||
return remove_offline_nodes(
|
||||
uids,
|
||||
available_uids,
|
||||
pre_deployment,
|
||||
deployment_info,
|
||||
post_deployment,
|
||||
offline_uids)
|
||||
end
|
||||
|
||||
def remove_offline_nodes(uids, available_uids, pre_deployment, deployment_info, post_deployment, offline_uids)
|
||||
if offline_uids.blank?
|
||||
return [deployment_info, pre_deployment, post_deployment]
|
||||
end
|
||||
|
||||
Astute.logger.info "Removing nodes which failed to provision: #{offline_uids}"
|
||||
deployment_info = cleanup_nodes_block(deployment_info, offline_uids)
|
||||
deployment_info = deployment_info.select { |node| available_uids.include? node['uid'] }
|
||||
|
||||
available_uids += ["master"]
|
||||
pre_deployment.each do |task|
|
||||
task['uids'] = task['uids'].select { |uid| available_uids.include? uid }
|
||||
end
|
||||
post_deployment.each do |task|
|
||||
task['uids'] = task['uids'].select { |uid| available_uids.include? uid }
|
||||
end
|
||||
|
||||
[pre_deployment, post_deployment].each do |deployment_task|
|
||||
deployment_task.select! do |task|
|
||||
if task['uids'].present?
|
||||
true
|
||||
else
|
||||
Astute.logger.info "Task(hook) was deleted because there is no " \
|
||||
"node where it should be run \n#{task.to_yaml}"
|
||||
false
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
[deployment_info, pre_deployment, post_deployment]
|
||||
end
|
||||
|
||||
def cleanup_nodes_block(deployment_info, offline_uids)
|
||||
return deployment_info if offline_uids.blank?
|
||||
|
||||
nodes = deployment_info.first['nodes']
|
||||
|
||||
# In case of deploy in already existing cluster in nodes block
|
||||
# we will have all cluster nodes. We should remove only missing
|
||||
# nodes instead of stay only available.
|
||||
# Example: deploy 3 nodes, after it deploy 2 nodes.
|
||||
# In 1 of 2 seconds nodes missing, in nodes block we should
|
||||
# contain only 4 nodes.
|
||||
nodes_wthout_missing = nodes.select { |node| !offline_uids.include?(node['uid']) }
|
||||
deployment_info.each { |node| node['nodes'] = nodes_wthout_missing }
|
||||
deployment_info
|
||||
end
|
||||
|
||||
def detect_available_nodes(uids)
|
||||
all_uids = uids.clone
|
||||
available_uids = []
|
||||
|
||||
# In case of big amount of nodes we should do several calls to be sure
|
||||
# about node status
|
||||
Astute.config[:mc_retries].times.each do
|
||||
systemtype = Astute::MClient.new(@ctx, "systemtype", all_uids, check_result=false, 10)
|
||||
available_nodes = systemtype.get_type.select do |node|
|
||||
node.results[:data][:node_type].chomp == "target"
|
||||
end
|
||||
|
||||
available_uids += available_nodes.map { |node| node.results[:sender] }
|
||||
all_uids -= available_uids
|
||||
break if all_uids.empty?
|
||||
|
||||
sleep Astute.config[:mc_retry_interval]
|
||||
end
|
||||
|
||||
available_uids
|
||||
end
|
||||
|
||||
def get_uids_from_deployment_info(deployment_info)
|
||||
top_level_uids = deployment_info.map{ |node| node["uid"] }
|
||||
|
||||
inside_uids = deployment_info.inject([]) do |uids, node|
|
||||
uids += node.fetch('nodes', []).map{ |n| n['uid'] }
|
||||
end
|
||||
top_level_uids | inside_uids
|
||||
end
|
||||
end
|
||||
end
|
|
@ -1,271 +0,0 @@
|
|||
# Copyright 2014 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
class Astute::DeploymentEngine::GranularDeployment < Astute::DeploymentEngine
|
||||
|
||||
NAILGUN_STATUS = ['ready', 'error', 'deploying']
|
||||
|
||||
def deploy_piece(nodes, retries=1)
|
||||
report_ready_for_nodes_without_tasks(nodes)
|
||||
nodes = filter_nodes_with_tasks(nodes)
|
||||
return false unless validate_nodes(nodes)
|
||||
|
||||
@ctx.reporter.report(nodes_status(nodes, 'deploying', {'progress' => 0}))
|
||||
log_preparation(nodes)
|
||||
|
||||
Astute.logger.info "#{@ctx.task_id}: Starting deployment"
|
||||
|
||||
@running_tasks = {}
|
||||
@start_times = {}
|
||||
@nodes_roles = nodes.inject({}) { |h, n| h.merge({n['uid'] => n['role']}) }
|
||||
@nodes_by_uid = nodes.inject({}) { |h, n| h.merge({ n['uid'] => n }) }
|
||||
@puppet_debug = nodes.first.fetch('puppet_debug', true)
|
||||
|
||||
begin
|
||||
@task_manager = Astute::TaskManager.new(nodes)
|
||||
@hook_context = Astute::Context.new(
|
||||
@ctx.task_id,
|
||||
HookReporter.new,
|
||||
Astute::LogParser::NoParsing.new
|
||||
)
|
||||
deploy_nodes(nodes)
|
||||
rescue => e
|
||||
# We should fail all nodes in case of post deployment
|
||||
# process. In other case they will not sending back
|
||||
# for redeploy
|
||||
report_nodes = nodes.uniq{ |n| n['uid'] }.map do |node|
|
||||
{ 'uid' => node['uid'],
|
||||
'status' => 'error',
|
||||
'role' => node['role'],
|
||||
'error_type' => 'deploy'
|
||||
}
|
||||
end
|
||||
|
||||
@ctx.report_and_update_status('nodes' => report_nodes)
|
||||
raise e
|
||||
end
|
||||
|
||||
Astute.logger.info "#{@ctx.task_id}: Finished deployment of nodes" \
|
||||
" => roles: #{@nodes_roles.pretty_inspect}"
|
||||
end
|
||||
|
||||
def puppet_task(node_id, task)
|
||||
# Use fake reporter because of logic. We need to handle report here
|
||||
Astute::PuppetTask.new(
|
||||
@hook_context,
|
||||
@nodes_by_uid[node_id], # Use single node uid instead of task['uids']
|
||||
{
|
||||
:retries => task['parameters']['retries'],
|
||||
:puppet_manifest => task['parameters']['puppet_manifest'],
|
||||
:puppet_modules => task['parameters']['puppet_modules'],
|
||||
:cwd => task['parameters']['cwd'],
|
||||
:timeout => task['parameters']['timeout'],
|
||||
:puppet_debug => @puppet_debug
|
||||
}
|
||||
)
|
||||
end
|
||||
|
||||
def run_task(node_id, task)
|
||||
@start_times[node_id] = {
|
||||
'time_start' => Time.now.to_i,
|
||||
'task_name' => task_name(task)
|
||||
}
|
||||
|
||||
Astute.logger.info "#{@ctx.task_id}: run task '#{task.to_yaml}' on " \
|
||||
"node #{node_id}"
|
||||
@running_tasks[node_id] = puppet_task(node_id, task)
|
||||
@running_tasks[node_id].run
|
||||
end
|
||||
|
||||
def check_status(node_id)
|
||||
status = @running_tasks[node_id].status
|
||||
if NAILGUN_STATUS.include? status
|
||||
status
|
||||
else
|
||||
raise "Internal error. Unknown status '#{status}'"
|
||||
end
|
||||
end
|
||||
|
||||
def deploy_nodes(nodes)
|
||||
@task_manager.node_uids.each { |n| task = @task_manager.next_task(n) and run_task(n, task) }
|
||||
|
||||
while @task_manager.task_in_queue?
|
||||
nodes_to_report = []
|
||||
sleep Astute.config.puppet_deploy_interval
|
||||
@task_manager.node_uids.each do |node_id|
|
||||
if task = @task_manager.current_task(node_id)
|
||||
case status = check_status(node_id)
|
||||
when 'ready'
|
||||
Astute.logger.info "Task '#{task}' on node uid=#{node_id} " \
|
||||
"ended successfully"
|
||||
time_summary(node_id, status)
|
||||
|
||||
new_task = @task_manager.next_task(node_id)
|
||||
if new_task
|
||||
run_task(node_id, new_task)
|
||||
else
|
||||
nodes_to_report << process_success_node(node_id, task)
|
||||
end
|
||||
when 'deploying'
|
||||
progress_report = process_running_node(node_id, task, nodes)
|
||||
nodes_to_report << progress_report if progress_report
|
||||
when 'error'
|
||||
Astute.logger.error "Task '#{task}' failed on node #{node_id}"
|
||||
nodes_to_report << process_fail_node(node_id, task)
|
||||
time_summary(node_id, status)
|
||||
else
|
||||
raise "Internal error. Known status '#{status}', but " \
|
||||
"handler not provided"
|
||||
end
|
||||
else
|
||||
Astute.logger.debug "No more tasks provided for node #{node_id}"
|
||||
end
|
||||
end
|
||||
|
||||
@ctx.report_and_update_status('nodes' => nodes_to_report) if nodes_to_report.present?
|
||||
|
||||
break unless @task_manager.task_in_queue?
|
||||
end
|
||||
end
|
||||
|
||||
def process_success_node(node_id, task)
|
||||
Astute.logger.info "No more tasks provided for node #{node_id}. All node " \
|
||||
"tasks completed successfully"
|
||||
{
|
||||
"uid" => node_id,
|
||||
'status' => 'ready',
|
||||
'role' => @nodes_roles[node_id],
|
||||
"progress" => 100,
|
||||
'task' => task
|
||||
}
|
||||
end
|
||||
|
||||
def process_fail_node(node_id, task)
|
||||
Astute.logger.error "No more tasks will be executed on the node #{node_id}"
|
||||
@task_manager.delete_node(node_id)
|
||||
{
|
||||
'uid' => node_id,
|
||||
'status' => 'error',
|
||||
'error_type' => 'deploy',
|
||||
'role' => @nodes_roles[node_id],
|
||||
'task' => task
|
||||
}
|
||||
end
|
||||
|
||||
def process_running_node(node_id, task, nodes)
|
||||
Astute.logger.debug "Task '#{task}' on node uid=#{node_id} deploying"
|
||||
begin
|
||||
# Pass nodes because logs calculation needs IP address of node, not just uid
|
||||
nodes_progress = @ctx.deploy_log_parser.progress_calculate(Array(node_id), nodes)
|
||||
if nodes_progress.present?
|
||||
nodes_progress.map! { |x| x.merge!(
|
||||
'status' => 'deploying',
|
||||
'role' => @nodes_roles[x['uid']],
|
||||
'task' => task
|
||||
) }
|
||||
nodes_progress.first
|
||||
else
|
||||
nil
|
||||
end
|
||||
rescue => e
|
||||
Astute.logger.warn "Some error occurred when parse logs for nodes progress: #{e.message}, "\
|
||||
"trace: #{e.format_backtrace}"
|
||||
nil
|
||||
end
|
||||
end
|
||||
|
||||
def log_preparation(nodes)
|
||||
@ctx.deploy_log_parser.prepare(nodes)
|
||||
rescue => e
|
||||
Astute.logger.warn "Some error occurred when prepare LogParser: " \
|
||||
"#{e.message}, trace: #{e.format_backtrace}"
|
||||
end
|
||||
|
||||
# If node doesn't have tasks, it means that node
|
||||
# is ready, because it doesn't require deployment
|
||||
def report_ready_for_nodes_without_tasks(nodes)
|
||||
nodes_without_tasks = filter_nodes_without_tasks(nodes)
|
||||
@ctx.reporter.report(nodes_status(nodes_without_tasks, 'ready', {'progress' => 100}))
|
||||
end
|
||||
|
||||
def filter_nodes_with_tasks(nodes)
|
||||
nodes.select { |n| node_with_tasks?(n) }
|
||||
end
|
||||
|
||||
def filter_nodes_without_tasks(nodes)
|
||||
nodes.select { |n| !node_with_tasks?(n) }
|
||||
end
|
||||
|
||||
def node_with_tasks?(node)
|
||||
node['tasks'].present?
|
||||
end
|
||||
|
||||
# Pre/post hooks
|
||||
def pre_deployment_actions(deployment_info, pre_deployment)
|
||||
Astute::GranularPreDeploymentActions.new(deployment_info, @ctx).process
|
||||
Astute::NailgunHooks.new(pre_deployment, @ctx).process
|
||||
end
|
||||
|
||||
def pre_node_actions(part)
|
||||
@action ||= Astute::GranularPreNodeActions.new(@ctx)
|
||||
@action.process(part)
|
||||
end
|
||||
|
||||
def pre_deploy_actions(part)
|
||||
Astute::GranularPreDeployActions.new(part, @ctx).process
|
||||
end
|
||||
|
||||
def post_deploy_actions(part)
|
||||
Astute::GranularPostDeployActions.new(part, @ctx).process
|
||||
end
|
||||
|
||||
def post_deployment_actions(deployment_info, post_deployment)
|
||||
begin
|
||||
Astute::NailgunHooks.new(post_deployment, @ctx).process
|
||||
rescue => e
|
||||
# We should fail all nodes in case of post deployment
|
||||
# process. In other case they will not sending back
|
||||
# for redeploy
|
||||
nodes = deployment_info.uniq {|n| n['uid']}.map do |node|
|
||||
{ 'uid' => node['uid'],
|
||||
'status' => 'error',
|
||||
'role' => 'hook',
|
||||
'error_type' => 'deploy',
|
||||
}
|
||||
end
|
||||
@ctx.report_and_update_status('nodes' => nodes)
|
||||
raise e
|
||||
end
|
||||
end
|
||||
|
||||
def time_summary(node_id, status)
|
||||
return unless @start_times.fetch(node_id, {}).fetch('time_start', nil)
|
||||
amount_time = (Time.now.to_i - @start_times[node_id]['time_start']).to_i
|
||||
wasted_time = Time.at(amount_time).utc.strftime("%H:%M:%S")
|
||||
Astute.logger.debug("Task time summary:" \
|
||||
" #{@start_times[node_id]['task_name']} with status" \
|
||||
" #{status.to_s} on node #{node_id} took #{wasted_time}")
|
||||
end
|
||||
|
||||
def task_name(task)
|
||||
task['id'] || task['diagnostic_name'] || task['type']
|
||||
end
|
||||
|
||||
class HookReporter
|
||||
def report(msg)
|
||||
Astute.logger.debug msg
|
||||
end
|
||||
end
|
||||
|
||||
end
|
|
@ -1,86 +0,0 @@
|
|||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
|
||||
module Astute
|
||||
module Dump
|
||||
def self.dump_environment(ctx, settings)
|
||||
shell = MClient.new(
|
||||
ctx,
|
||||
'execute_shell_command',
|
||||
['master'],
|
||||
check_result=true,
|
||||
settings['timeout'] || Astute.config.dump_timeout,
|
||||
retries=0,
|
||||
enable_result_logging=false
|
||||
)
|
||||
|
||||
begin
|
||||
log_file = "/var/log/timmy.log"
|
||||
snapshot = File.basename(settings['target'])
|
||||
if settings['timestamp']
|
||||
snapshot = DateTime.now.strftime("#{snapshot}-%Y-%m-%d_%H-%M-%S")
|
||||
end
|
||||
base_dir = File.dirname(settings['target'])
|
||||
dest_dir = File.join(base_dir, snapshot)
|
||||
dest_file = File.join(dest_dir, "config.tar.gz")
|
||||
token = settings['auth-token']
|
||||
dump_cmd = "mkdir -p #{dest_dir} && "\
|
||||
"timmy --logs --days 3 --dest-file #{dest_file}"\
|
||||
" --fuel-token #{token} --log-file #{log_file} && "\
|
||||
"tar --directory=#{base_dir} -cf #{dest_dir}.tar #{snapshot} && "\
|
||||
"echo #{dest_dir}.tar > #{settings['lastdump']} && "\
|
||||
"rm -rf #{dest_dir}"
|
||||
Astute.logger.debug("Try to execute command: #{dump_cmd}")
|
||||
result = shell.execute(:cmd => dump_cmd).first.results
|
||||
|
||||
Astute.logger.debug("#{ctx.task_id}: exit code: #{result[:data][:exit_code]}")
|
||||
|
||||
if result[:data][:exit_code] == 0
|
||||
Astute.logger.info("#{ctx.task_id}: Snapshot is done.")
|
||||
report_success(ctx, "#{dest_dir}.tar")
|
||||
elsif result[:data][:exit_code] == 28
|
||||
Astute.logger.error("#{ctx.task_id}: Disk space for creating snapshot exceeded.")
|
||||
report_error(ctx, "Timmy exit code: #{result[:data][:exit_code]}. Disk space for creating snapshot exceeded.")
|
||||
elsif result[:data][:exit_code] == 100
|
||||
Astute.logger.error("#{ctx.task_id}: Not enough free space for logs. Decrease logs coefficient via CLI or config or free up space.")
|
||||
report_error(ctx, "Timmy exit code: #{result[:data][:exit_code]}. Not enough free space for logs.")
|
||||
else
|
||||
Astute.logger.error("#{ctx.task_id}: Dump command returned non zero exit code. For details see #{log_file}")
|
||||
report_error(ctx, "Timmy exit code: #{result[:data][:exit_code]}")
|
||||
end
|
||||
rescue Timeout::Error
|
||||
msg = "Dump is timed out"
|
||||
Astute.logger.error("#{ctx.task_id}: #{msg}")
|
||||
report_error(ctx, msg)
|
||||
rescue => e
|
||||
msg = "Exception occured during dump task: message: #{e.message} \
|
||||
trace:\n#{e.backtrace.pretty_inspect}"
|
||||
Astute.logger.error("#{ctx.task_id}: #{msg}")
|
||||
report_error(ctx, msg)
|
||||
end
|
||||
end
|
||||
|
||||
def self.report_success(ctx, msg=nil)
|
||||
success_msg = {'status' => 'ready', 'progress' => 100}
|
||||
success_msg.merge!({'msg' => msg}) if msg
|
||||
ctx.reporter.report(success_msg)
|
||||
end
|
||||
|
||||
def self.report_error(ctx, msg)
|
||||
ctx.reporter.report({'status' => 'error', 'error' => msg, 'progress' => 100})
|
||||
end
|
||||
|
||||
end
|
||||
end
|
|
@ -1,36 +0,0 @@
|
|||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
require 'timeout'
|
||||
|
||||
module Astute
|
||||
|
||||
# Base class for all errors
|
||||
class AstuteError < StandardError; end
|
||||
|
||||
# Provisioning log errors
|
||||
class ParseProvisionLogsError < AstuteError; end
|
||||
# Image provisioning errors
|
||||
class FailedImageProvisionError < AstuteError; end
|
||||
# Deployment engine error
|
||||
class DeploymentEngineError < AstuteError; end
|
||||
# MClient errors
|
||||
class MClientError < AstuteError; end
|
||||
# MClient timeout error
|
||||
class MClientTimeout < Timeout::Error; end
|
||||
# Task validation error
|
||||
class TaskValidationError < AstuteError; end
|
||||
# Status error
|
||||
class StatusValidationError < AstuteError; end
|
||||
|
||||
end
|
|
@ -1,28 +0,0 @@
|
|||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
|
||||
class Array
|
||||
|
||||
def compact_blank
|
||||
reject do |val|
|
||||
case val
|
||||
when Hash then val.compact_blank.blank?
|
||||
when Array then val.map { |v| v.respond_to?(:compact_blank) ? v.compact_blank : v }.blank?
|
||||
when String then val.blank?
|
||||
else val.blank?
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
|
@ -1,18 +0,0 @@
|
|||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
|
||||
def deep_copy(data)
|
||||
Marshal.load(Marshal.dump(data))
|
||||
end
|
|
@ -1,20 +0,0 @@
|
|||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
|
||||
class Exception
|
||||
def format_backtrace
|
||||
"\n" + backtrace.pretty_inspect
|
||||
end
|
||||
end
|
|
@ -1,32 +0,0 @@
|
|||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
|
||||
class Hash
|
||||
|
||||
def absent_keys(array)
|
||||
array.select { |key| self[key].blank? }
|
||||
end
|
||||
|
||||
def compact_blank
|
||||
delete_if do |_key, val|
|
||||
case val
|
||||
when Hash then val.compact_blank.blank?
|
||||
when Array then val.map { |v| v.respond_to?(:compact_blank) ? v.compact_blank : v }.blank?
|
||||
when String then val.blank?
|
||||
else val.blank?
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
|
@ -1,160 +0,0 @@
|
|||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
|
||||
module Astute
|
||||
module ImageProvision
|
||||
|
||||
def self.provision(ctx, nodes)
|
||||
failed_uids = []
|
||||
uids_to_provision, failed_uids = upload_provision(ctx, nodes)
|
||||
run_provision(ctx, uids_to_provision, failed_uids)
|
||||
rescue => e
|
||||
msg = "Error while provisioning: message: #{e.message}" \
|
||||
" trace\n: #{e.format_backtrace}"
|
||||
Astute.logger.error("#{ctx.task_id}: #{msg}")
|
||||
report_error(ctx, msg)
|
||||
failed_uids
|
||||
end
|
||||
|
||||
def self.upload_provision(ctx, nodes)
|
||||
failed_uids = []
|
||||
nodes.each do |node|
|
||||
succees = upload_provision_data(ctx, node)
|
||||
next if succees
|
||||
|
||||
failed_uids << node['uid']
|
||||
Astute.logger.error("#{ctx.task_id}: Upload provisioning data " \
|
||||
"failed on node #{node['uid']}. Provision on such node will " \
|
||||
"not start")
|
||||
end
|
||||
|
||||
uids_to_provision = nodes.select { |n| !failed_uids.include?(n['uid']) }
|
||||
.map { |n| n['uid'] }
|
||||
[uids_to_provision, failed_uids]
|
||||
end
|
||||
|
||||
def self.upload_provision_data(ctx, node)
|
||||
Astute.logger.debug("#{ctx.task_id}: uploading provision " \
|
||||
"data on node #{node['uid']}: #{node.to_json}")
|
||||
|
||||
upload_task = Astute::UploadFile.new(
|
||||
generate_upload_provision_task(node),
|
||||
ctx
|
||||
)
|
||||
|
||||
upload_task.sync_run
|
||||
end
|
||||
|
||||
def self.generate_upload_provision_task(node)
|
||||
{
|
||||
"id" => 'upload_provision_data',
|
||||
"node_id" => node['uid'],
|
||||
"parameters" => {
|
||||
"path" => '/tmp/provision.json',
|
||||
"data" => node.to_json,
|
||||
"user_owner" => 'root',
|
||||
"group_owner" => 'root',
|
||||
"overwrite" => true,
|
||||
"timeout" => Astute.config.upload_timeout
|
||||
}
|
||||
}
|
||||
end
|
||||
|
||||
def self.run_provision(ctx, uids, failed_uids)
|
||||
Astute.logger.debug("#{ctx.task_id}: running provision script: " \
|
||||
"#{uids.join(', ')}")
|
||||
|
||||
failed_uids |= run_shell_task(
|
||||
ctx,
|
||||
uids,
|
||||
'flock -n /var/lock/provision.lock provision',
|
||||
Astute.config.provisioning_timeout
|
||||
)
|
||||
|
||||
failed_uids
|
||||
end
|
||||
|
||||
def self.report_error(ctx, msg)
|
||||
ctx.reporter.report({
|
||||
'status' => 'error',
|
||||
'error' => msg,
|
||||
'progress' => 100
|
||||
})
|
||||
end
|
||||
|
||||
def self.reboot(ctx, node_ids, task_id="reboot_provisioned_nodes")
|
||||
if node_ids.empty?
|
||||
Astute.logger.warn("No nodes were sent to reboot for " \
|
||||
"task: #{task_id}")
|
||||
return
|
||||
end
|
||||
|
||||
Astute::NailgunHooks.new(
|
||||
[{
|
||||
"priority" => 100,
|
||||
"type" => "reboot",
|
||||
"fail_on_error" => false,
|
||||
"id" => task_id,
|
||||
"uids" => node_ids,
|
||||
"parameters" => {
|
||||
"timeout" => Astute.config.reboot_timeout
|
||||
}
|
||||
}],
|
||||
ctx,
|
||||
'provision'
|
||||
).process
|
||||
end
|
||||
|
||||
def self.run_shell_task(ctx, node_uids, cmd, timeout=3600)
|
||||
shell_tasks = node_uids.inject([]) do |tasks, node_id|
|
||||
tasks << Shell.new(generate_shell_hook(node_id, cmd, timeout), ctx)
|
||||
end
|
||||
|
||||
shell_tasks.each(&:run)
|
||||
|
||||
while shell_tasks.any? { |t| !t.finished? } do
|
||||
shell_tasks.select { |t| !t.finished? }.each(&:status)
|
||||
sleep Astute.config.task_poll_delay
|
||||
end
|
||||
|
||||
failed_uids = shell_tasks.select{ |t| t.failed? }
|
||||
.inject([]) do |failed_nodes, task|
|
||||
Astute.logger.error("#{ctx.task_id}: Provision command returned " \
|
||||
"non zero exit code on node: #{task.node_id}")
|
||||
failed_nodes << task.node_id
|
||||
end
|
||||
|
||||
failed_uids
|
||||
rescue => e
|
||||
Astute.logger.error("#{ctx.task_id}: cmd: #{cmd} " \
|
||||
"error: #{e.message}, trace #{e.backtrace}")
|
||||
node_uids
|
||||
end
|
||||
|
||||
def self.generate_shell_hook(node_id, cmd, timeout)
|
||||
{
|
||||
"node_id" => node_id,
|
||||
"id" => "provision_#{node_id}",
|
||||
"parameters" => {
|
||||
"cmd" => cmd,
|
||||
"cwd" => "/",
|
||||
"timeout" => timeout,
|
||||
"retries" => 0
|
||||
}
|
||||
}
|
||||
end
|
||||
|
||||
end
|
||||
end
|
|
@ -1,261 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
|
||||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
|
||||
require 'erb'
|
||||
|
||||
module Astute
|
||||
module LogParser
|
||||
LOG_PORTION = 10000
|
||||
# Default values. Can be overrided by pattern_spec.
|
||||
# E.g. pattern_spec = {'separator' => 'new_separator', ...}
|
||||
PATH_PREFIX = '/var/log/remote/'
|
||||
SEPARATOR = "SEPARATOR\n"
|
||||
|
||||
class NoParsing
|
||||
def initialize(*args)
|
||||
end
|
||||
|
||||
def method_missing(*args)
|
||||
# We just eat the call if we don't want to deal with logs
|
||||
end
|
||||
|
||||
def progress_calculate(*args)
|
||||
[]
|
||||
end
|
||||
end
|
||||
|
||||
class DirSizeCalculation
|
||||
attr_reader :nodes
|
||||
|
||||
def initialize(nodes)
|
||||
@nodes = nodes.map{|n| n.dup}
|
||||
@nodes.each{|node| node[:path_items] = weight_reassignment(node[:path_items])}
|
||||
end
|
||||
|
||||
def deploy_type=(*args)
|
||||
# Because we mimic the DeploymentParser, we should define all auxiliary method
|
||||
# even they do nothing.
|
||||
end
|
||||
|
||||
def prepare(nodes)
|
||||
# Because we mimic the DeploymentParser, we should define all auxiliary method
|
||||
# even they do nothing.
|
||||
end
|
||||
|
||||
def progress_calculate(uids_to_calc, nodes)
|
||||
uids_to_calc.map do |uid|
|
||||
node = @nodes.find{|n| n[:uid] == uid}
|
||||
node[:path_items] ||= []
|
||||
progress = 0
|
||||
node[:path_items].each do |item|
|
||||
size = recursive_size(item[:path])
|
||||
sub_progress = 100 * size / item[:max_size]
|
||||
sub_progress = 0 if sub_progress < 0
|
||||
sub_progress = 100 if sub_progress > 100
|
||||
progress += sub_progress * item[:weight]
|
||||
end
|
||||
{'uid' => uid, 'progress' => progress.to_i}
|
||||
end
|
||||
end
|
||||
|
||||
private
|
||||
def recursive_size(path, opts={})
|
||||
return File.size?(path).to_i if not File.directory?(path)
|
||||
|
||||
total_size = 0
|
||||
Dir[File.join("#{path}", '**/*')].each do |f|
|
||||
# Option :files_only used when you want to calculate total size of
|
||||
# regular files only. The default :files_only is false, so the function will
|
||||
# include inode size of each dir (4096 bytes in most cases) to total value
|
||||
# as the unix util 'du' does it.
|
||||
total_size += File.size?(f).to_i if File.file?(f) || ! opts[:files_only]
|
||||
end
|
||||
total_size
|
||||
end
|
||||
|
||||
def weight_reassignment(items)
|
||||
# The finction normalizes the weights of each item in order to make sum of
|
||||
# all weights equal to one.
|
||||
# It divides items as wighted and unweighted depending on the existence of
|
||||
# the :weight key in the item.
|
||||
# - Each unweighted item will be weighted as a one N-th part of the total number of items.
|
||||
# - All weights of weighted items are summed up and then each weighted item
|
||||
# gets a new weight as a multiplication of a relative weight among all
|
||||
# weighted items and the ratio of the number of the weighted items to
|
||||
# the total number of items.
|
||||
# E.g. we have four items: one with weight 0.5, another with weight 1.5, and
|
||||
# two others as unweighted. All unweighted items will get the weight 1/4.
|
||||
# Weight's sum of weighted items is 2. So the first item will get the weight:
|
||||
# (relative weight 0.5/2) * (weighted items ratio 2/4) = 1/8.
|
||||
# Finally all items will be normalised with next weights:
|
||||
# 1/8, 3/8, 1/4, and 1/4.
|
||||
|
||||
ret_items = items.reject do |item|
|
||||
weight = item[:weight]
|
||||
# Save an item if it unweighted.
|
||||
next if weight.nil?
|
||||
raise "Weight should be a non-negative number" unless [Fixnum, Float].include?(weight.class) && weight >= 0
|
||||
# Drop an item if it weighted as zero.
|
||||
item[:weight] == 0
|
||||
end
|
||||
return [] if ret_items.empty?
|
||||
ret_items.map!{|n| n.dup}
|
||||
|
||||
partial_weight = 1.0 / ret_items.length
|
||||
weighted_items = ret_items.select{|n| n[:weight]}
|
||||
weighted_sum = 0.0
|
||||
weighted_items.each{|n| weighted_sum += n[:weight]}
|
||||
weighted_sum = weighted_sum * ret_items.length / weighted_items.length if weighted_items.any?
|
||||
raise "Unexpectedly a summary weight of weighted items is a non-positive" if weighted_items.any? && weighted_sum <= 0
|
||||
ret_items.each do |item|
|
||||
weight = item[:weight]
|
||||
item[:weight] = weight ? weight / weighted_sum : partial_weight
|
||||
end
|
||||
|
||||
ret_items
|
||||
end
|
||||
end
|
||||
|
||||
class ParseNodeLogs
|
||||
attr_reader :pattern_spec
|
||||
|
||||
def initialize
|
||||
@pattern_spec = {}
|
||||
@pattern_spec['path_prefix'] ||= PATH_PREFIX.to_s
|
||||
@pattern_spec['separator'] ||= SEPARATOR.to_s
|
||||
@nodes_patterns = {}
|
||||
end
|
||||
|
||||
def progress_calculate(uids_to_calc, nodes)
|
||||
nodes_progress = []
|
||||
|
||||
patterns = patterns_for_nodes(nodes, uids_to_calc)
|
||||
uids_to_calc.each do |uid|
|
||||
node = nodes.find {|n| n['uid'] == uid}
|
||||
@nodes_patterns[uid] ||= patterns[uid]
|
||||
node_pattern_spec = @nodes_patterns[uid]
|
||||
# FIXME(eli): this var is required for binding() below
|
||||
@pattern_spec = @nodes_patterns[uid]
|
||||
|
||||
erb_path = node_pattern_spec['path_format']
|
||||
path = ERB.new(erb_path).result(binding())
|
||||
|
||||
progress = 0
|
||||
begin
|
||||
# Return percent of progress
|
||||
progress = (get_log_progress(path, node_pattern_spec) * 100).to_i
|
||||
rescue => e
|
||||
Astute.logger.warn "Some error occurred when calculate progress " \
|
||||
"for node '#{uid}': #{e.message}, trace: #{e.format_backtrace}"
|
||||
end
|
||||
|
||||
nodes_progress << {
|
||||
'uid' => uid,
|
||||
'progress' => progress
|
||||
}
|
||||
end
|
||||
|
||||
nodes_progress
|
||||
end
|
||||
|
||||
def prepare(nodes)
|
||||
patterns = patterns_for_nodes(nodes)
|
||||
nodes.each do |node|
|
||||
pattern = patterns[node['uid']]
|
||||
path = "#{pattern['path_prefix']}#{node['ip']}/#{pattern['filename']}"
|
||||
File.open(path, 'a') { |fo| fo.write pattern['separator'] } if File.writable?(path)
|
||||
end
|
||||
end
|
||||
|
||||
# Get patterns for selected nodes
|
||||
# if uids_to_calc is nil, then
|
||||
# patterns for all nodes will be returned
|
||||
def patterns_for_nodes(nodes, uids_to_calc=nil)
|
||||
uids_to_calc = nodes.map { |node| node['uid'] } if uids_to_calc.nil?
|
||||
nodes_to_calc = nodes.select { |node| uids_to_calc.include?(node['uid']) }
|
||||
|
||||
patterns = {}
|
||||
nodes_to_calc.map do |node|
|
||||
patterns[node['uid']] = get_pattern_for_node(node)
|
||||
end
|
||||
|
||||
patterns
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def get_log_progress(path, node_pattern_spec)
|
||||
unless File.readable?(path)
|
||||
Astute.logger.debug "Can't read file with logs: #{path}"
|
||||
return 0
|
||||
end
|
||||
if node_pattern_spec.nil?
|
||||
Astute.logger.warn "Can't parse logs. Pattern_spec is empty."
|
||||
return 0
|
||||
end
|
||||
progress = nil
|
||||
File.open(path) do |fo|
|
||||
# Try to find well-known ends of log.
|
||||
endlog = find_endlog_patterns(fo, node_pattern_spec)
|
||||
return endlog if endlog
|
||||
# Start reading from end of file.
|
||||
fo.pos = fo.stat.size
|
||||
|
||||
# Method 'calculate' should be defined at child classes.
|
||||
progress = calculate(fo, node_pattern_spec)
|
||||
node_pattern_spec['file_pos'] = fo.pos
|
||||
end
|
||||
unless progress
|
||||
Astute.logger.warn("Wrong pattern\n#{node_pattern_spec.pretty_inspect}\ndefined for calculating progress via logs.")
|
||||
return 0
|
||||
end
|
||||
progress
|
||||
end
|
||||
|
||||
def find_endlog_patterns(fo, pattern_spec)
|
||||
# Pattern example:
|
||||
# pattern_spec = {...,
|
||||
# 'endlog_patterns' => [{'pattern' => /Finished catalog run in [0-9]+\.[0-9]* seconds\n/, 'progress' => 1.0}],
|
||||
# }
|
||||
endlog_patterns = pattern_spec['endlog_patterns']
|
||||
return nil unless endlog_patterns
|
||||
fo.pos = fo.stat.size
|
||||
chunk = get_chunk(fo, 100)
|
||||
return nil unless chunk
|
||||
endlog_patterns.each do |pattern|
|
||||
return pattern['progress'] if Regexp.new("#{pattern['pattern']}$").match(chunk)
|
||||
end
|
||||
nil
|
||||
end
|
||||
|
||||
def get_chunk(fo, size=nil, pos=nil)
|
||||
if pos
|
||||
fo.pos = pos
|
||||
return fo.read
|
||||
end
|
||||
size = LOG_PORTION unless size
|
||||
return nil if fo.pos == 0
|
||||
size = fo.pos if fo.pos < size
|
||||
next_pos = fo.pos - size
|
||||
fo.pos = next_pos
|
||||
block = fo.read(size)
|
||||
fo.pos = next_pos
|
||||
block
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
|
@ -1,160 +0,0 @@
|
|||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
|
||||
module Astute
|
||||
module LogParser
|
||||
class ParseDeployLogs < ParseNodeLogs
|
||||
attr_reader :deploy_type
|
||||
|
||||
def deploy_type=(deploy_type)
|
||||
@deploy_type = deploy_type
|
||||
@nodes_patterns = {}
|
||||
end
|
||||
|
||||
def get_pattern_for_node(node)
|
||||
role = node['role']
|
||||
node_pattern = Patterns::get_default_pattern(
|
||||
"puppet-log-components-list-#{@deploy_type}-#{role}")
|
||||
node_pattern['path_prefix'] ||= PATH_PREFIX.to_s
|
||||
node_pattern['separator'] ||= SEPARATOR.to_s
|
||||
|
||||
node_pattern
|
||||
end
|
||||
|
||||
private
|
||||
def calculate(fo, node_pattern_spec)
|
||||
case node_pattern_spec['type']
|
||||
when 'count-lines'
|
||||
progress = simple_line_counter(fo, node_pattern_spec)
|
||||
when 'components-list'
|
||||
progress = component_parser(fo, node_pattern_spec)
|
||||
end
|
||||
return progress
|
||||
end
|
||||
|
||||
def simple_line_counter(fo, pattern_spec)
|
||||
# Pattern specification example:
|
||||
# pattern_spec = {'type' => 'count-lines',
|
||||
# 'endlog_patterns' => [{'pattern' => /Finished catalog run in [0-9]+\.[0-9]* seconds\n/, 'progress' => 1.0}],
|
||||
# 'expected_line_number' => 500}
|
||||
# Use custom separator if defined.
|
||||
separator = pattern_spec['separator']
|
||||
counter = 0
|
||||
end_of_scope = false
|
||||
previous_subchunk = ''
|
||||
until end_of_scope
|
||||
chunk = get_chunk(fo, pattern_spec['chunk_size'])
|
||||
break unless chunk
|
||||
# Trying to find separator on border between chunks.
|
||||
subchunk = chunk.slice((1-separator.size)..-1)
|
||||
# End of file reached. Exit from cycle.
|
||||
end_of_scope = true unless subchunk
|
||||
if subchunk and (subchunk + previous_subchunk).include?(separator)
|
||||
# Separator found on border between chunks. Exit from cycle.
|
||||
end_of_scope = true
|
||||
continue
|
||||
end
|
||||
|
||||
pos = chunk.rindex(separator)
|
||||
if pos
|
||||
end_of_scope = true
|
||||
chunk = chunk.slice((pos + separator.size)..-1)
|
||||
end
|
||||
counter += chunk.count("\n")
|
||||
end
|
||||
number = pattern_spec['expected_line_number']
|
||||
unless number
|
||||
Astute.logger.warn("Wrong pattern\n#{pattern_spec.pretty_inspect} defined for calculating progress via log.")
|
||||
return 0
|
||||
end
|
||||
progress = counter.to_f / number
|
||||
progress = 1 if progress > 1
|
||||
return progress
|
||||
end
|
||||
|
||||
def component_parser(fo, pattern_spec)
|
||||
# Pattern specification example:
|
||||
# pattern_spec = {'type' => 'components-list',
|
||||
# 'chunk_size' => 40000,
|
||||
# 'components_list' => [
|
||||
# {'name' => 'Horizon', 'weight' => 10, 'patterns' => [
|
||||
# {'pattern' => '/Stage[main]/Horizon/Package[mod_wsgi]/ensure) created', 'progress' => 0.1},
|
||||
# {'pattern' => '/Stage[main]/Horizon/File_line[horizon_redirect_rule]/ensure) created', 'progress' => 0.3},
|
||||
# {'pattern' => '/Stage[main]/Horizon/File[/etc/openstack-dashboard/local_settings]/group)', 'progress' => 0.7},
|
||||
# {'pattern' => '/Stage[main]/Horizon/Service[$::horizon::params::http_service]/ensure)'\
|
||||
# ' ensure changed \'stopped\' to \'running\'', 'progress' => 1},
|
||||
# ]
|
||||
# },
|
||||
# ]
|
||||
# }
|
||||
# Use custom separator if defined.
|
||||
separator = pattern_spec['separator']
|
||||
components_list = pattern_spec['components_list']
|
||||
unless components_list
|
||||
Astute.logger.warn("Wrong pattern\n#{pattern_spec.pretty_inspect} defined for calculating progress via logs.")
|
||||
return 0
|
||||
end
|
||||
|
||||
chunk = get_chunk(fo, pos=pattern_spec['file_pos'])
|
||||
return 0 unless chunk
|
||||
pos = chunk.rindex(separator)
|
||||
chunk = chunk.slice((pos + separator.size)..-1) if pos
|
||||
block = chunk.split("\n")
|
||||
|
||||
# Update progress of each component.
|
||||
while block.any?
|
||||
string = block.pop
|
||||
components_list.each do |component|
|
||||
matched_pattern = nil
|
||||
component['patterns'].each do |pattern|
|
||||
if pattern['regexp']
|
||||
matched_pattern = pattern if string.match(pattern['pattern'])
|
||||
else
|
||||
matched_pattern = pattern if string.include?(pattern['pattern'])
|
||||
end
|
||||
break if matched_pattern
|
||||
end
|
||||
if matched_pattern and
|
||||
(not component['_progress'] or matched_pattern['progress'] > component['_progress'])
|
||||
component['_progress'] = matched_pattern['progress']
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
# Calculate integral progress.
|
||||
weighted_components = components_list.select{|n| n['weight']}
|
||||
weight_sum = 0.0
|
||||
if weighted_components.any?
|
||||
weighted_components.each{|n| weight_sum += n['weight']}
|
||||
weight_sum = weight_sum * components_list.length / weighted_components.length
|
||||
raise "Total weight of weighted components equal to zero." if weight_sum == 0
|
||||
end
|
||||
nonweighted_delta = 1.0 / components_list.length
|
||||
progress = 0
|
||||
components_list.each do |component|
|
||||
component['_progress'] = 0.0 unless component['_progress']
|
||||
weight = component['weight']
|
||||
if weight
|
||||
progress += component['_progress'] * weight / weight_sum
|
||||
else
|
||||
progress += component['_progress'] * nonweighted_delta
|
||||
end
|
||||
end
|
||||
|
||||
return progress
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
|
@ -1,75 +0,0 @@
|
|||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
|
||||
module Astute
|
||||
module LogParser
|
||||
module Patterns
|
||||
def self.get_default_pattern(key)
|
||||
pattern_key = key
|
||||
pattern_key = 'default' unless @default_patterns.has_key?(key)
|
||||
deep_copy(@default_patterns[pattern_key])
|
||||
end
|
||||
|
||||
def self.list_default_patterns
|
||||
return @default_patterns.keys
|
||||
end
|
||||
|
||||
@default_patterns = {
|
||||
'provisioning-image-building' =>
|
||||
{'type' => 'supposed-time',
|
||||
'chunk_size' => 10000,
|
||||
'date_format' => '%Y-%m-%d %H:%M:%S',
|
||||
'date_regexp' => '^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}',
|
||||
'pattern_list' => [
|
||||
{'pattern' => '--- Building image (do_build_image) ---', 'supposed_time' => 12},
|
||||
{'pattern' => '*** Shipping image content ***', 'supposed_time' => 12},
|
||||
{'pattern' => 'Running deboostrap completed', 'supposed_time' => 270},
|
||||
{'pattern' => 'Running apt-get install completed', 'supposed_time' => 480},
|
||||
{'pattern' => '--- Building image END (do_build_image) ---', 'supposed_time' => 240},
|
||||
{'pattern' => 'All necessary images are available.', 'supposed_time' => 10}
|
||||
].reverse,
|
||||
'filename' => "fuel-agent-env",
|
||||
'path_format' => "<%= @pattern_spec['path_prefix']%><%= @pattern_spec['filename']%>-<%= @pattern_spec['cluster_id']%>.log"
|
||||
},
|
||||
|
||||
'image-based-provisioning' =>
|
||||
{'type' => 'pattern-list',
|
||||
'chunk_size' => 10000,
|
||||
'pattern_list' => [
|
||||
{'pattern' => '--- Provisioning (do_provisioning) ---', 'progress' => 0.81},
|
||||
{'pattern' => '--- Partitioning disks (do_partitioning) ---', 'progress' => 0.82},
|
||||
{'pattern' => '--- Creating configdrive (do_configdrive) ---', 'progress' => 0.92},
|
||||
{'pattern' => 'Next chunk',
|
||||
'number' => 600,
|
||||
'p_min' => 0.92,
|
||||
'p_max' => 0.98},
|
||||
{'pattern' => '--- Installing bootloader (do_bootloader) ---', 'progress' => 0.99},
|
||||
{'pattern' => '--- Provisioning END (do_provisioning) ---', 'progress' => 1}
|
||||
],
|
||||
'filename' => 'bootstrap/fuel-agent.log',
|
||||
'path_format' => "<%= @pattern_spec['path_prefix'] %><%= node['hostname'] %>/<%= @pattern_spec['filename'] %>",
|
||||
},
|
||||
|
||||
'default' => {
|
||||
'type' => 'count-lines',
|
||||
'endlog_patterns' => [{'pattern' => /Finished catalog run in [0-9]+\.[0-9]* seconds\n/, 'progress' => 1.0}],
|
||||
'expected_line_number' => 345,
|
||||
'filename' => 'puppet-apply.log',
|
||||
'path_format' => "<%= @pattern_spec['path_prefix'] %><%= node['fqdn'] %>/<%= @pattern_spec['filename'] %>"
|
||||
},
|
||||
}
|
||||
end
|
||||
end
|
||||
end
|
|
@ -1,262 +0,0 @@
|
|||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
|
||||
require 'date'
|
||||
|
||||
module Astute
|
||||
module LogParser
|
||||
class ParseProvisionLogs < ParseNodeLogs
|
||||
|
||||
def get_pattern_for_node(node)
|
||||
os = node['profile']
|
||||
|
||||
pattern_spec_name = if node.fetch('ks_meta', {}).key?('image_data')
|
||||
'image-based-provisioning'
|
||||
elsif ['centos-x86_64'].include?(os)
|
||||
'centos-anaconda-log-supposed-time-kvm'
|
||||
elsif os == 'ubuntu_1404_x86_64'
|
||||
'ubuntu-provisioning'
|
||||
else
|
||||
raise Astute::ParseProvisionLogsError, "Cannot find profile for os with: #{os}"
|
||||
end
|
||||
|
||||
pattern_spec = deep_copy(Patterns::get_default_pattern(pattern_spec_name))
|
||||
pattern_spec['path_prefix'] ||= PATH_PREFIX.to_s
|
||||
pattern_spec['separator'] ||= SEPARATOR.to_s
|
||||
|
||||
pattern_spec
|
||||
end
|
||||
|
||||
private
|
||||
def calculate(fo, node_pattern_spec)
|
||||
case node_pattern_spec['type']
|
||||
when 'pattern-list'
|
||||
progress = simple_pattern_finder(fo, node_pattern_spec)
|
||||
when 'supposed-time'
|
||||
progress = supposed_time_parser(fo, node_pattern_spec)
|
||||
end
|
||||
|
||||
progress
|
||||
end
|
||||
|
||||
# Pattern specification example:
|
||||
# pattern_spec = {'type' => 'supposed-time',
|
||||
# 'chunk_size' => 10000,
|
||||
# 'date_format' => '%Y-%m-%dT%H:%M:%S',
|
||||
# 'date_regexp' => '^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}',
|
||||
# 'pattern_list' => [
|
||||
# {'pattern' => 'Running anaconda script', 'supposed_time' => 60},
|
||||
# ....
|
||||
# {'pattern' => 'leaving (1) step postscripts', 'supposed_time' => 130},
|
||||
# ].reverse,
|
||||
# 'filename' => 'install/anaconda.log'
|
||||
# }
|
||||
# Use custom separator if defined.
|
||||
def supposed_time_parser(fo, pattern_spec)
|
||||
separator = pattern_spec['separator']
|
||||
log_patterns = pattern_spec['pattern_list']
|
||||
date_format = pattern_spec['date_format']
|
||||
date_regexp = pattern_spec['date_regexp']
|
||||
unless date_regexp and date_format and log_patterns
|
||||
Astute.logger.warn("Wrong pattern_spec\n#{pattern_spec.pretty_inspect} defined for calculating progress via logs.")
|
||||
return 0
|
||||
end
|
||||
|
||||
def self.get_elapsed_time(patterns)
|
||||
elapsed_time = 0
|
||||
patterns.each do |p|
|
||||
if p['_progress']
|
||||
break
|
||||
else
|
||||
elapsed_time += p['supposed_time']
|
||||
end
|
||||
end
|
||||
return elapsed_time
|
||||
end
|
||||
|
||||
def self.get_progress(base_progress, elapsed_time, delta_time, supposed_time=nil)
|
||||
return 1.0 if elapsed_time.zero?
|
||||
k = (1.0 - base_progress) / elapsed_time
|
||||
supposed_time ? surplus = delta_time - supposed_time : surplus = nil
|
||||
if surplus and surplus > 0
|
||||
progress = supposed_time * k + surplus * k/3 + base_progress
|
||||
else
|
||||
progress = delta_time * k + base_progress
|
||||
end
|
||||
progress = 1.0 if progress > 1
|
||||
return progress
|
||||
end
|
||||
|
||||
def self.get_seconds_from_time(date)
|
||||
hours, mins, secs, _frac = Date::day_fraction_to_time(date)
|
||||
return hours*60*60 + mins*60 + secs
|
||||
end
|
||||
|
||||
|
||||
chunk = get_chunk(fo, pattern_spec['chunk_size'])
|
||||
return 0 unless chunk
|
||||
pos = chunk.rindex(separator)
|
||||
chunk = chunk.slice((pos + separator.size)..-1) if pos
|
||||
block = chunk.split("\n")
|
||||
|
||||
now = DateTime.now()
|
||||
prev_time = pattern_spec['_prev_time'] ||= now
|
||||
prev_progress = pattern_spec['_prev_progress'] ||= 0
|
||||
elapsed_time = pattern_spec['_elapsed_time'] ||= get_elapsed_time(log_patterns)
|
||||
seconds_since_prev = get_seconds_from_time(now - prev_time)
|
||||
|
||||
until block.empty?
|
||||
string = block.pop
|
||||
log_patterns.each do |pattern|
|
||||
if string.include?(pattern['pattern'])
|
||||
if pattern['_progress']
|
||||
# We not found any new messages. Calculate progress with old data.
|
||||
progress = get_progress(prev_progress, elapsed_time,
|
||||
seconds_since_prev, pattern['supposed_time'])
|
||||
return progress
|
||||
|
||||
else
|
||||
# We found message that we never find before. We need to:
|
||||
# calculate progress for this message;
|
||||
# recalculate control point and elapsed_time;
|
||||
# calculate progress for current time.
|
||||
|
||||
# Trying to find timestamp of message.
|
||||
date_string = string.match(date_regexp)
|
||||
if date_string
|
||||
# Get relative time when the message realy occured.
|
||||
date = DateTime.strptime(date_string[0], date_format) - prev_time.offset
|
||||
real_time = get_seconds_from_time(date - prev_time)
|
||||
# Update progress of the message.
|
||||
prev_supposed_time = log_patterns.select{|n| n['_progress'] == prev_progress}[0]
|
||||
prev_supposed_time = prev_supposed_time['supposed_time'] if prev_supposed_time
|
||||
progress = get_progress(prev_progress, elapsed_time, real_time, prev_supposed_time)
|
||||
pattern['_progress'] = progress
|
||||
# Recalculate elapsed time.
|
||||
elapsed_time = pattern_spec['_elapsed_time'] = get_elapsed_time(log_patterns)
|
||||
# Update time and progress for control point.
|
||||
prev_time = pattern_spec['_prev_time'] = date
|
||||
prev_progress = pattern_spec['_prev_progress'] = progress
|
||||
seconds_since_prev = get_seconds_from_time(now - date)
|
||||
# Calculate progress for current time.
|
||||
progress = get_progress(prev_progress, elapsed_time,
|
||||
seconds_since_prev, pattern['supposed_time'])
|
||||
return progress
|
||||
else
|
||||
Astute.logger.info("Can't gather date (format: '#{date_regexp}') from string: #{string}")
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
# We found nothing.
|
||||
progress = get_progress(prev_progress, elapsed_time, seconds_since_prev, log_patterns[0]['supposed_time'])
|
||||
return progress
|
||||
end
|
||||
|
||||
def simple_pattern_finder(fo, pattern_spec)
|
||||
# Pattern specification example:
|
||||
# pattern_spec = {'type' => 'pattern-list', 'separator' => "custom separator\n",
|
||||
# 'chunk_size' => 40000,
|
||||
# 'pattern_list' => [
|
||||
# {'pattern' => 'Running kickstart %%pre script', 'progress' => 0.08},
|
||||
# {'pattern' => 'to step enablefilesystems', 'progress' => 0.09},
|
||||
# {'pattern' => 'to step reposetup', 'progress' => 0.13},
|
||||
# {'pattern' => 'to step installpackages', 'progress' => 0.16},
|
||||
# {'pattern' => 'Installing',
|
||||
# 'number' => 210, # Now it install 205 packets. Add 5 packets for growth in future.
|
||||
# 'p_min' => 0.16, # min percent
|
||||
# 'p_max' => 0.87 # max percent
|
||||
# },
|
||||
# {'pattern' => 'to step postinstallconfig', 'progress' => 0.87},
|
||||
# {'pattern' => 'to step dopostaction', 'progress' => 0.92},
|
||||
# ]
|
||||
# }
|
||||
# Use custom separator if defined.
|
||||
separator = pattern_spec['separator']
|
||||
log_patterns = pattern_spec['pattern_list']
|
||||
unless log_patterns
|
||||
Astute.logger.warn("Wrong pattern\n#{pattern_spec.pretty_inspect} defined for calculating progress via logs.")
|
||||
return 0
|
||||
end
|
||||
|
||||
chunk = get_chunk(fo, pattern_spec['chunk_size'])
|
||||
# NOTE(mihgen): Following line fixes "undefined method `rindex' for nil:NilClass" for empty log file
|
||||
return 0 unless chunk
|
||||
pos = chunk.rindex(separator)
|
||||
chunk = chunk.slice((pos + separator.size)..-1) if pos
|
||||
block = chunk.split("\n")
|
||||
return 0 unless block
|
||||
while true
|
||||
string = block.pop
|
||||
return 0 unless string # If we found nothing
|
||||
log_patterns.each do |pattern|
|
||||
if string.include?(pattern['pattern'])
|
||||
return pattern['progress'] if pattern['progress']
|
||||
if pattern['number']
|
||||
string = block.pop
|
||||
counter = 1
|
||||
while string
|
||||
counter += 1 if string.include?(pattern['pattern'])
|
||||
string = block.pop
|
||||
end
|
||||
progress = counter.to_f / pattern['number']
|
||||
progress = 1 if progress > 1
|
||||
progress = pattern['p_min'] + progress * (pattern['p_max'] - pattern['p_min'])
|
||||
return progress
|
||||
end
|
||||
Astute.logger.warn("Wrong pattern\n#{pattern_spec.pretty_inspect} defined for calculating progress via log.")
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
end # ParseProvisionLogs
|
||||
|
||||
class ParseImageBuildLogs < ParseProvisionLogs
|
||||
|
||||
PATH_PREFIX = '/var/log/'
|
||||
attr_accessor :cluster_id
|
||||
|
||||
def get_pattern_for_node(node)
|
||||
os = node['profile']
|
||||
|
||||
pattern_spec_name = 'provisioning-image-building'
|
||||
|
||||
pattern_spec = deep_copy(Patterns::get_default_pattern(pattern_spec_name))
|
||||
pattern_spec['path_prefix'] ||= PATH_PREFIX.to_s
|
||||
pattern_spec['separator'] ||= SEPARATOR.to_s
|
||||
pattern_spec['cluster_id'] = cluster_id
|
||||
|
||||
pattern_spec
|
||||
end
|
||||
|
||||
def prepare(nodes)
|
||||
# This is common file for all nodes
|
||||
pattern_spec = get_pattern_for_node(nodes.first)
|
||||
path = pattern_spec['path_format']
|
||||
File.open(path, 'a') { |fo| fo.write pattern['separator'] } if File.writable?(path)
|
||||
end
|
||||
|
||||
def progress_calculate(uids_to_calc, nodes)
|
||||
result = super
|
||||
# Limit progress for this part to 80% as max
|
||||
result.map { |h| h['progress'] = (h['progress'] * 0.8).to_i }
|
||||
result
|
||||
end
|
||||
|
||||
end # ParseImageProvisionLogs
|
||||
end
|
||||
end
|
|
@ -1,190 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
|
||||
require 'mcollective'
|
||||
require 'timeout'
|
||||
|
||||
module Astute
|
||||
|
||||
class MClient
|
||||
include MCollective::RPC
|
||||
|
||||
attr_accessor :retries
|
||||
|
||||
def initialize(
|
||||
ctx,
|
||||
agent,
|
||||
nodes=nil,
|
||||
check_result=true,
|
||||
timeout=nil,
|
||||
retries=Astute.config.mc_retries,
|
||||
enable_result_logging=true
|
||||
)
|
||||
|
||||
@task_id = ctx.task_id
|
||||
@agent = agent
|
||||
@nodes = nodes.map { |n| n.to_s } if nodes
|
||||
@check_result = check_result
|
||||
# Will be used a minimum of two things: the specified parameter(timeout)
|
||||
# and timeout from DDL (10 sec by default if not explicitly specified in DDL)
|
||||
# If timeout here is nil will be used value from DDL.
|
||||
# Examples:
|
||||
# timeout - 10 sec, DDL - 20 sec. Result — 10 sec.
|
||||
# timeout - 30 sec, DDL - 20 sec. Result — 20 sec.
|
||||
# timeout - 20 sec, DDL - not set. Result — 10 sec.
|
||||
@timeout = timeout
|
||||
@retries = retries
|
||||
@enable_result_logging = enable_result_logging
|
||||
initialize_mclient
|
||||
end
|
||||
|
||||
def on_respond_timeout(&block)
|
||||
@on_respond_timeout = block
|
||||
self
|
||||
end
|
||||
|
||||
def method_missing(method, *args)
|
||||
@mc_res = mc_send(method, *args)
|
||||
|
||||
if method == :discover
|
||||
@nodes = args[0][:nodes]
|
||||
return @mc_res
|
||||
end
|
||||
|
||||
# Enable if needed. In normal case it eats the screen pretty fast
|
||||
log_result(@mc_res, method) if @enable_result_logging
|
||||
|
||||
check_results_with_retries(method, args) if @check_result
|
||||
|
||||
@mc_res
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def check_results_with_retries(method, args)
|
||||
err_msg = ''
|
||||
timeout_nodes_count = 0
|
||||
# Following error might happen because of misconfiguration, ex. direct_addressing = 1 only on client
|
||||
# or.. could be just some hang? Let's retry if @retries is set
|
||||
if @mc_res.length < @nodes.length
|
||||
# some nodes didn't respond
|
||||
retry_index = 1
|
||||
while retry_index <= @retries
|
||||
sleep rand
|
||||
nodes_responded = @mc_res.map { |n| n.results[:sender] }
|
||||
not_responded = @nodes - nodes_responded
|
||||
Astute.logger.debug "Retry ##{retry_index} to run mcollective agent on nodes: '#{not_responded.join(',')}'"
|
||||
mc_send :discover, :nodes => not_responded
|
||||
@new_res = mc_send(method, *args)
|
||||
|
||||
log_result(@new_res, method) if @enable_result_logging
|
||||
# @new_res can have some nodes which finally responded
|
||||
|
||||
@mc_res += @new_res
|
||||
break if @mc_res.length == @nodes.length
|
||||
retry_index += 1
|
||||
end
|
||||
if @mc_res.length < @nodes.length
|
||||
nodes_responded = @mc_res.map { |n| n.results[:sender] }
|
||||
not_responded = @nodes - nodes_responded
|
||||
if @on_respond_timeout
|
||||
@on_respond_timeout.call not_responded
|
||||
else
|
||||
err_msg += "MCollective agents '#{@agent}' " \
|
||||
"'#{not_responded.join(',')}' didn't respond within the " \
|
||||
"allotted time.\n"
|
||||
timeout_nodes_count += not_responded.size
|
||||
end
|
||||
end
|
||||
end
|
||||
failed = @mc_res.select { |x| x.results[:statuscode] != 0 }
|
||||
if failed.any?
|
||||
err_msg += "MCollective call failed in agent '#{@agent}', "\
|
||||
"method '#{method}', failed nodes: \n"
|
||||
failed.each do |n|
|
||||
err_msg += "ID: #{n.results[:sender]} - Reason: #{n.results[:statusmsg]}\n"
|
||||
end
|
||||
end
|
||||
if err_msg.present?
|
||||
Astute.logger.error err_msg
|
||||
expired_size = failed.count { |n| n.results[:statusmsg] == 'execution expired' }
|
||||
# Detect TimeOut: 1 condition - fail because of DDL timeout, 2 - fail because of custom timeout
|
||||
if (failed.present? && failed.size == expired_size) || (timeout_nodes_count > 0 && failed.empty?)
|
||||
raise MClientTimeout, "#{@task_id}: #{err_msg}"
|
||||
else
|
||||
raise MClientError, "#{@task_id}: #{err_msg}"
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def mc_send(*args)
|
||||
retries = 1
|
||||
begin
|
||||
@mc.send(*args)
|
||||
rescue => ex
|
||||
case ex
|
||||
when Stomp::Error::NoCurrentConnection
|
||||
# stupid stomp cannot recover severed connection
|
||||
stomp = MCollective::PluginManager["connector_plugin"]
|
||||
stomp.disconnect rescue nil
|
||||
stomp.instance_variable_set :@connection, nil
|
||||
initialize_mclient
|
||||
end
|
||||
if retries < 3
|
||||
Astute.logger.error "Retrying MCollective call after exception:\n#{ex.pretty_inspect}"
|
||||
sleep rand
|
||||
retries += 1
|
||||
retry
|
||||
else
|
||||
Astute.logger.error "No more retries for MCollective call after exception: " \
|
||||
"#{ex.format_backtrace}"
|
||||
raise MClientError, "#{ex.pretty_inspect}"
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def initialize_mclient
|
||||
retries = 1
|
||||
begin
|
||||
@mc = rpcclient(@agent, :exit_on_failure => false)
|
||||
|
||||
@mc.timeout = @timeout if @timeout
|
||||
@mc.progress = false
|
||||
if @nodes
|
||||
@mc.discover :nodes => @nodes
|
||||
end
|
||||
rescue => ex
|
||||
if retries < 3
|
||||
Astute.logger.error "Retrying RPC client instantiation after exception:\n#{ex.pretty_inspect}"
|
||||
sleep 5
|
||||
retries += 1
|
||||
retry
|
||||
else
|
||||
Astute.logger.error "No more retries for MCollective client instantiation after exception: " \
|
||||
"#{ex.format_backtrace}"
|
||||
raise MClientError, "#{ex.pretty_inspect}"
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def log_result(result, method)
|
||||
result.each do |node|
|
||||
Astute.logger.debug "#{@task_id}: MC agent '#{node.agent}', method '#{method}', "\
|
||||
"results:\n#{node.results.pretty_inspect}"
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
|
@ -1,138 +0,0 @@
|
|||
# Copyright 2016 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
class PuppetMClient
|
||||
|
||||
PUPPET_STATUSES = [
|
||||
'running', 'stopped', 'disabled'
|
||||
]
|
||||
|
||||
attr_reader :summary, :node_id
|
||||
|
||||
def initialize(ctx, node_id, options)
|
||||
@ctx = ctx
|
||||
@node_id = node_id
|
||||
@options = options
|
||||
@summary = {}
|
||||
end
|
||||
|
||||
# Return actual status of puppet using mcollective puppet agent
|
||||
# @return [String] status: succeed, one of PUPPET_STATUSES or undefined
|
||||
def status
|
||||
last_run_summary
|
||||
succeed? ? 'succeed' : @summary.fetch(:status, 'undefined')
|
||||
end
|
||||
|
||||
# Run puppet on node if available
|
||||
# @return [true, false]
|
||||
def run
|
||||
is_succeed, err_msg = runonce
|
||||
return true if is_succeed
|
||||
|
||||
Astute.logger.warn "Fail to start puppet on node #{@node_id}. "\
|
||||
"Reason: #{err_msg}"
|
||||
false
|
||||
end
|
||||
|
||||
# Return path to manifest using by mcollective puppet agent
|
||||
# @return [String] path to manifest
|
||||
def manifest
|
||||
File.join(@options['cwd'], @options['puppet_manifest'])
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
# Create configured puppet mcollective agent
|
||||
# @return [Astute::MClient]
|
||||
def puppetd(timeout=nil, retries=1)
|
||||
puppetd = MClient.new(
|
||||
@ctx,
|
||||
"puppetd",
|
||||
[@node_id],
|
||||
_check_result=true,
|
||||
_timeout=timeout,
|
||||
_retries=retries,
|
||||
_enable_result_logging=false
|
||||
)
|
||||
puppetd.on_respond_timeout do |uids|
|
||||
msg = "Nodes #{uids} reached the response timeout"
|
||||
Astute.logger.error msg
|
||||
raise MClientTimeout, msg
|
||||
end
|
||||
puppetd
|
||||
end
|
||||
|
||||
# Run last_run_summary action using mcollective puppet agent
|
||||
# @return [Hash] return hash with status and resources
|
||||
def last_run_summary
|
||||
@summary = puppetd(_timeout=10, _retries=6).last_run_summary(
|
||||
:puppet_noop_run => @options['puppet_noop_run'],
|
||||
:raw_report => @options['raw_report']
|
||||
).first[:data]
|
||||
validate_status!(@summary[:status])
|
||||
@summary
|
||||
rescue MClientError, MClientTimeout => e
|
||||
Astute.logger.warn "Unable to get actual status of puppet on "\
|
||||
"node #{@node_id}. Reason: #{e.message}"
|
||||
@summary = {}
|
||||
end
|
||||
|
||||
# Run runonce action using mcollective puppet agent
|
||||
# @return [[true, false], String] boolean status of run and error message
|
||||
def runonce
|
||||
result = puppetd.runonce(
|
||||
:puppet_debug => @options['puppet_debug'],
|
||||
:manifest => @options['puppet_manifest'],
|
||||
:modules => @options['puppet_modules'],
|
||||
:cwd => @options['cwd'],
|
||||
:command_prefix => @options['command_prefix'],
|
||||
:puppet_noop_run => @options['puppet_noop_run'],
|
||||
).first
|
||||
return result[:statuscode] == 0, result[:statusmsg]
|
||||
rescue MClientError, MClientTimeout => e
|
||||
return false, e.message
|
||||
end
|
||||
|
||||
# Validate puppet status
|
||||
# @param [String] status The puppet status
|
||||
# @return [void]
|
||||
# @raise [MClientError] Unknown status
|
||||
def validate_status!(status)
|
||||
unless PUPPET_STATUSES.include?(status)
|
||||
raise MClientError, "Unknow status '#{status}' from mcollective agent"
|
||||
end
|
||||
end
|
||||
|
||||
# Detect succeed of puppet run using summary from last_run_summary call
|
||||
# @return [true, false]
|
||||
def succeed?
|
||||
return false if @summary.blank?
|
||||
|
||||
@summary[:status] == 'stopped' &&
|
||||
@summary[:resources] &&
|
||||
@summary[:resources]['failed'].to_i == 0 &&
|
||||
@summary[:resources]['failed_to_restart'].to_i == 0
|
||||
end
|
||||
|
||||
# Generate shell cmd for file deletion
|
||||
# @return [String] shell cmd for deletion
|
||||
def rm_cmd
|
||||
PUPPET_FILES_TO_CLEANUP.inject([]) do
|
||||
|cmd, file| cmd << "rm -f #{file}"
|
||||
end.join(" && ")
|
||||
end
|
||||
|
||||
end # PuppetMclient
|
||||
end
|
|
@ -1,84 +0,0 @@
|
|||
# Copyright 2016 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
class ShellMClient
|
||||
|
||||
def initialize(ctx, node_id)
|
||||
@ctx = ctx
|
||||
@node_id = node_id
|
||||
end
|
||||
|
||||
# Run shell cmd without check using mcollective agent
|
||||
# @param [String] cmd Shell command for run
|
||||
# @param [Integer] timeout Timeout for shell command
|
||||
# @return [Hash] shell result
|
||||
def run_without_check(cmd, timeout=2)
|
||||
Astute.logger.debug("Executing shell command without check: "\
|
||||
"#{details_for_log(cmd, timeout)}")
|
||||
|
||||
results = shell(_check_result=false, timeout).execute(:cmd => cmd)
|
||||
Astute.logger.debug("Mcollective shell #{details_for_log(cmd, timeout)}"\
|
||||
" result: #{results.pretty_inspect}")
|
||||
if results.present?
|
||||
result = results.first
|
||||
log_result(result, cmd, timeout)
|
||||
{
|
||||
:stdout => result.results[:data][:stdout].chomp,
|
||||
:stderr => result.results[:data][:stderr].chomp,
|
||||
:exit_code => result.results[:data][:exit_code]
|
||||
}
|
||||
else
|
||||
Astute.logger.warn("#{@ctx.task_id}: Failed to run shell "\
|
||||
"#{details_for_log(cmd, timeout)}. Error will not raise "\
|
||||
"because shell was run without check")
|
||||
{}
|
||||
end
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
# Create configured shell mcollective agent
|
||||
# @return [Astute::MClient]
|
||||
def shell(check_result=false, timeout=2)
|
||||
MClient.new(
|
||||
@ctx,
|
||||
'execute_shell_command',
|
||||
[@node_id],
|
||||
check_result,
|
||||
timeout
|
||||
)
|
||||
end
|
||||
|
||||
# Return short useful info about node and shell task
|
||||
# @return [String] detail info about cmd
|
||||
def details_for_log(cmd, timeout)
|
||||
"command '#{cmd}' on node #{@node_id} with timeout #{timeout}"
|
||||
end
|
||||
|
||||
# Write to log shell command result including exit code
|
||||
# @param [Hash] result Actual magent shell result
|
||||
# @return [void]
|
||||
def log_result(result, cmd, timeout)
|
||||
return if result.results[:data].blank?
|
||||
|
||||
Astute.logger.debug(
|
||||
"#{@ctx.task_id}: #{details_for_log(cmd, timeout)}\n" \
|
||||
"stdout: #{result.results[:data][:stdout]}\n" \
|
||||
"stderr: #{result.results[:data][:stderr]}\n" \
|
||||
"exit code: #{result.results[:data][:exit_code]}")
|
||||
end
|
||||
|
||||
end
|
||||
end
|
|
@ -1,133 +0,0 @@
|
|||
# Copyright 2016 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
class UploadFileMClient
|
||||
|
||||
attr_reader :ctx, :node_id
|
||||
def initialize(ctx, node_id)
|
||||
@ctx = ctx
|
||||
@node_id = node_id
|
||||
end
|
||||
|
||||
# Run upload without check using mcollective agent
|
||||
# @param [Hash] mco_params Upload file options
|
||||
# @return [true, false] upload result
|
||||
def upload_without_check(mco_params)
|
||||
upload_mclient = upload_mclient(
|
||||
:check_result => false,
|
||||
:timeout => mco_params['timeout']
|
||||
)
|
||||
upload(mco_params, upload_mclient)
|
||||
end
|
||||
|
||||
# Run upload with check using mcollective agent
|
||||
# @param [Hash] mco_params Upload file options
|
||||
# @return [true, false] upload result
|
||||
def upload_with_check(mco_params)
|
||||
upload_mclient = upload_mclient(
|
||||
:check_result => false,
|
||||
:timeout => mco_params['timeout'],
|
||||
:retries => mco_params['retries']
|
||||
)
|
||||
process_with_retries(:retries => mco_params['retries']) do
|
||||
upload(mco_params, upload_mclient)
|
||||
end
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
|
||||
def upload(mco_params, magent)
|
||||
mco_params = setup_default(mco_params)
|
||||
|
||||
results = magent.upload(
|
||||
:path => mco_params['path'],
|
||||
:content => mco_params['content'],
|
||||
:overwrite => mco_params['overwrite'],
|
||||
:parents => mco_params['parents'],
|
||||
:permissions => mco_params['permissions'],
|
||||
:user_owner => mco_params['user_owner'],
|
||||
:group_owner => mco_params['group_owner'],
|
||||
:dir_permissions => mco_params['dir_permissions']
|
||||
)
|
||||
|
||||
if results.present? && results.first[:statuscode] == 0
|
||||
Astute.logger.debug("#{ctx.task_id}: file was uploaded "\
|
||||
"#{details_for_log(mco_params)} successfully")
|
||||
true
|
||||
else
|
||||
Astute.logger.error("#{ctx.task_id}: file was not uploaded "\
|
||||
"#{details_for_log(mco_params)}: "\
|
||||
"#{results.present? ? results.first[:msg] : "node has not answered" }")
|
||||
false
|
||||
end
|
||||
rescue MClientTimeout, MClientError => e
|
||||
Astute.logger.error("#{ctx.task_id}: file was not uploaded "\
|
||||
"#{details_for_log(mco_params)}: #{e.message}")
|
||||
false
|
||||
end
|
||||
|
||||
# Create configured shell mcollective agent
|
||||
# @return [Astute::MClient]
|
||||
def upload_mclient(args={})
|
||||
MClient.new(
|
||||
ctx,
|
||||
"uploadfile",
|
||||
[node_id],
|
||||
args.fetch(:check_result, false),
|
||||
args.fetch(:timeout, 2),
|
||||
args.fetch(:retries, Astute.config.upload_retries)
|
||||
)
|
||||
end
|
||||
|
||||
# Setup default value for upload mcollective agent
|
||||
# @param [Hash] mco_params Upload file options
|
||||
# @return [Hash] mco_params
|
||||
def setup_default(mco_params)
|
||||
mco_params['retries'] ||= Astute.config.upload_retries
|
||||
mco_params['timeout'] ||= Astute.config.upload_timeout
|
||||
mco_params['overwrite'] = true if mco_params['overwrite'].nil?
|
||||
mco_params['parents'] = true if mco_params['parents'].nil?
|
||||
mco_params['permissions'] ||= '0644'
|
||||
mco_params['user_owner'] ||= 'root'
|
||||
mco_params['group_owner'] ||= 'root'
|
||||
mco_params['dir_permissions'] ||= '0755'
|
||||
|
||||
mco_params
|
||||
end
|
||||
|
||||
# Return short useful info about node and shell task
|
||||
# @return [String] detail info about upload task
|
||||
def details_for_log(mco_params)
|
||||
"#{mco_params['path']} on node #{node_id} "\
|
||||
"with timeout #{mco_params['timeout']}"
|
||||
end
|
||||
|
||||
def process_with_retries(args={}, &block)
|
||||
retries = args.fetch(:retries, 1) + 1
|
||||
result = false
|
||||
|
||||
retries.times do |attempt|
|
||||
result = block.call
|
||||
break if result
|
||||
|
||||
Astute.logger.warn("#{ctx.task_id} Upload retry for node "\
|
||||
"#{node_id}: attempt № #{attempt + 1}/#{retries}")
|
||||
end
|
||||
result
|
||||
end
|
||||
|
||||
end
|
||||
end
|
|
@ -1,467 +0,0 @@
|
|||
# Copyright 2014 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
|
||||
module Astute
|
||||
class NailgunHooks
|
||||
|
||||
def initialize(nailgun_hooks, context, type='deploy')
|
||||
@nailgun_hooks = nailgun_hooks
|
||||
@ctx = context
|
||||
@type = type
|
||||
end
|
||||
|
||||
def process
|
||||
@nailgun_hooks.sort_by { |f| f['priority'] }.each do |hook|
|
||||
Astute.logger.debug "Run hook #{hook.to_yaml}"
|
||||
|
||||
time_start = Time.now.to_i
|
||||
|
||||
hook_return = case hook['type']
|
||||
when 'copy_files' then copy_files_hook(hook)
|
||||
when 'upload_files' then upload_files_hook(hook)
|
||||
when 'sync' then sync_hook(hook)
|
||||
when 'shell' then shell_hook(hook)
|
||||
when 'upload_file' then upload_file_hook(hook)
|
||||
when 'puppet' then puppet_hook(hook)
|
||||
when 'reboot' then reboot_hook(hook)
|
||||
when 'cobbler_sync' then cobbler_sync_hook(hook)
|
||||
else raise "Unknown hook type #{hook['type']}"
|
||||
end
|
||||
|
||||
time_summary(time_start, hook, hook_return)
|
||||
hook_name = task_name(hook)
|
||||
|
||||
is_raise_on_error = hook.fetch('fail_on_error', true)
|
||||
|
||||
if hook_return['error'] && is_raise_on_error
|
||||
nodes = hook['uids'].map do |uid|
|
||||
{ 'uid' => uid,
|
||||
'status' => 'error',
|
||||
'error_type' => @type,
|
||||
'role' => 'hook',
|
||||
'hook' => hook_name,
|
||||
'error_msg' => hook_return['error']
|
||||
}
|
||||
end
|
||||
error_message = 'Failed to execute hook'
|
||||
error_message += " '#{hook_name}'" if hook_name
|
||||
error_message += "\n\n#{hook_return['error']}"
|
||||
@ctx.report_and_update_status('nodes' => nodes, 'error' => error_message)
|
||||
error_message += "#{hook.to_yaml}"
|
||||
|
||||
raise Astute::DeploymentEngineError, error_message
|
||||
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def task_name(hook)
|
||||
hook['id'] || hook['diagnostic_name'] || hook['type']
|
||||
end
|
||||
|
||||
def time_summary(time_start, hook, hook_return)
|
||||
status = hook_return && !hook_return['error'] ? 'successful' : 'error'
|
||||
amount_time = (Time.now.to_i - time_start).to_i
|
||||
wasted_time = Time.at(amount_time).utc.strftime("%H:%M:%S")
|
||||
|
||||
hook['uids'].each do |node_id|
|
||||
Astute.logger.debug("Task time summary: #{task_name(hook)} with status" \
|
||||
" #{status} on node #{node_id} took #{wasted_time}")
|
||||
end
|
||||
end
|
||||
|
||||
def copy_files_hook(hook)
|
||||
validate_presence(hook, 'uids')
|
||||
validate_presence(hook['parameters'], 'files')
|
||||
|
||||
ret = {'error' => nil}
|
||||
hook['parameters']['files'].each do |file|
|
||||
if File.file?(file['src']) && File.readable?(file['src'])
|
||||
parameters = {
|
||||
'content' => File.binread(file['src']),
|
||||
'path' => file['dst'],
|
||||
'permissions' => file['permissions'] || hook['parameters']['permissions'],
|
||||
'dir_permissions' => file['dir_permissions'] || hook['parameters']['dir_permissions'],
|
||||
}
|
||||
perform_with_limit(hook['uids']) do |node_uids|
|
||||
status = upload_file(@ctx, node_uids, parameters)
|
||||
if !status
|
||||
ret['error'] = 'Upload not successful'
|
||||
end
|
||||
end
|
||||
else
|
||||
ret['error'] = "File does not exist or is not readable #{file['src']}"
|
||||
Astute.logger.warn(ret['error'])
|
||||
end
|
||||
end
|
||||
ret
|
||||
end #copy_file_hook
|
||||
|
||||
def puppet_hook(hook)
|
||||
validate_presence(hook, 'uids')
|
||||
validate_presence(hook['parameters'], 'puppet_manifest')
|
||||
validate_presence(hook['parameters'], 'puppet_modules')
|
||||
validate_presence(hook['parameters'], 'cwd')
|
||||
|
||||
timeout = hook['parameters']['timeout'] || 300
|
||||
retries = hook['parameters']['retries'] || Astute.config.puppet_retries
|
||||
|
||||
ret = {'error' => nil}
|
||||
perform_with_limit(hook['uids']) do |node_uids|
|
||||
result = run_puppet(
|
||||
@ctx,
|
||||
node_uids,
|
||||
hook['parameters']['puppet_manifest'],
|
||||
hook['parameters']['puppet_modules'],
|
||||
hook['parameters']['cwd'],
|
||||
timeout,
|
||||
retries
|
||||
)
|
||||
unless result
|
||||
ret['error'] = "Puppet run failed. Check puppet logs for details"
|
||||
Astute.logger.warn(ret['error'])
|
||||
end
|
||||
end
|
||||
|
||||
ret
|
||||
end #puppet_hook
|
||||
|
||||
def upload_file_hook(hook)
|
||||
validate_presence(hook, 'uids')
|
||||
validate_presence(hook['parameters'], 'path')
|
||||
validate_presence(hook['parameters'], 'data')
|
||||
|
||||
hook['parameters']['content'] = hook['parameters']['data']
|
||||
|
||||
ret = {'error' => nil}
|
||||
perform_with_limit(hook['uids']) do |node_uids|
|
||||
status = upload_file(@ctx, node_uids, hook['parameters'])
|
||||
if status == false
|
||||
ret['error'] = 'File upload failed'
|
||||
end
|
||||
end
|
||||
|
||||
ret
|
||||
end
|
||||
|
||||
def upload_files_hook(hook)
|
||||
validate_presence(hook['parameters'], 'nodes')
|
||||
ret = {'error' => nil}
|
||||
hook['parameters']['nodes'].each do |node|
|
||||
node['files'].each do |file|
|
||||
parameters = {
|
||||
'content' => file['data'],
|
||||
'path' => file['dst'],
|
||||
'permissions' => file['permissions'] || '0644',
|
||||
'dir_permissions' => file['dir_permissions'] || '0755',
|
||||
}
|
||||
status = upload_file(@ctx, node['uid'], parameters)
|
||||
if !status
|
||||
ret['error'] = 'File upload failed'
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
ret
|
||||
end
|
||||
|
||||
def shell_hook(hook)
|
||||
validate_presence(hook, 'uids')
|
||||
validate_presence(hook['parameters'], 'cmd')
|
||||
|
||||
|
||||
timeout = hook['parameters']['timeout'] || 300
|
||||
cwd = hook['parameters']['cwd'] || "/"
|
||||
retries = hook['parameters']['retries'] || Astute.config.mc_retries
|
||||
interval = hook['parameters']['interval'] || Astute.config.mc_retry_interval
|
||||
shell_command = "cd #{cwd} && #{hook['parameters']['cmd']}"
|
||||
|
||||
ret = {'error' => nil}
|
||||
|
||||
perform_with_limit(hook['uids']) do |node_uids|
|
||||
Timeout::timeout(timeout) do
|
||||
err_msg = run_shell_command(
|
||||
@ctx,
|
||||
node_uids,
|
||||
shell_command,
|
||||
retries,
|
||||
interval,
|
||||
timeout,
|
||||
cwd
|
||||
)
|
||||
|
||||
ret['error'] = "Failed to run command #{shell_command} (#{err_msg})." if err_msg
|
||||
end
|
||||
end
|
||||
|
||||
ret
|
||||
rescue Astute::MClientTimeout, Astute::MClientError, Timeout::Error => e
|
||||
err = case [e.class]
|
||||
when [Astute::MClientTimeout] then 'mcollective client timeout error'
|
||||
when [Astute::MClientError] then 'mcollective client error'
|
||||
when [Timeout::Error] then 'overall timeout error'
|
||||
end
|
||||
|
||||
ret['error'] = "command: #{shell_command}" \
|
||||
"\n\nTask: #{@ctx.task_id}: " \
|
||||
"#{err}: #{e.message}\n" \
|
||||
"Task timeout: #{timeout}, " \
|
||||
"Retries: #{hook['parameters']['retries']}"
|
||||
Astute.logger.error(ret['error'])
|
||||
|
||||
ret
|
||||
end # shell_hook
|
||||
|
||||
def cobbler_sync_hook(hook)
|
||||
validate_presence(hook['parameters'], 'provisioning_info')
|
||||
|
||||
ret = {'error' => nil}
|
||||
cobbler = CobblerManager.new(
|
||||
hook['parameters']['provisioning_info']['engine'],
|
||||
@ctx.reporter
|
||||
)
|
||||
cobbler.sync
|
||||
|
||||
ret
|
||||
end # cobbler_sync_hook
|
||||
|
||||
def sync_hook(hook)
|
||||
validate_presence(hook, 'uids')
|
||||
validate_presence(hook['parameters'], 'dst')
|
||||
validate_presence(hook['parameters'], 'src')
|
||||
|
||||
path = hook['parameters']['dst']
|
||||
source = hook['parameters']['src']
|
||||
|
||||
timeout = hook['parameters']['timeout'] || 300
|
||||
|
||||
rsync_cmd = "mkdir -p #{path} && rsync #{Astute.config.rsync_options} " \
|
||||
"#{source} #{path}"
|
||||
|
||||
ret = {'error' => nil}
|
||||
|
||||
perform_with_limit(hook['uids']) do |node_uids|
|
||||
err_msg = run_shell_command(
|
||||
@ctx,
|
||||
node_uids,
|
||||
rsync_cmd,
|
||||
10,
|
||||
Astute.config.mc_retry_interval,
|
||||
timeout
|
||||
)
|
||||
|
||||
ret = {'error' => "Failed to perform sync from #{source} to #{path} (#{err_msg})"} if err_msg
|
||||
end
|
||||
|
||||
ret
|
||||
end # sync_hook
|
||||
|
||||
def reboot_hook(hook)
|
||||
validate_presence(hook, 'uids')
|
||||
hook_timeout = hook['parameters']['timeout'] || 300
|
||||
|
||||
control_time = {}
|
||||
|
||||
perform_with_limit(hook['uids']) do |node_uids|
|
||||
control_time.merge!(boot_time(node_uids))
|
||||
end
|
||||
|
||||
perform_with_limit(hook['uids']) do |node_uids|
|
||||
run_shell_without_check(@ctx, node_uids, RebootCommand::CMD, timeout=60)
|
||||
end
|
||||
|
||||
already_rebooted = Hash[hook['uids'].collect { |uid| [uid, false] }]
|
||||
|
||||
ret = {'error' => nil}
|
||||
|
||||
begin
|
||||
Timeout::timeout(hook_timeout) do
|
||||
while already_rebooted.values.include?(false)
|
||||
sleep hook_timeout/10
|
||||
|
||||
results = boot_time(already_rebooted.select { |k, v| !v }.keys)
|
||||
results.each do |node_id, time|
|
||||
next if already_rebooted[node_id]
|
||||
already_rebooted[node_id] = (time.to_i != control_time[node_id].to_i)
|
||||
end
|
||||
end
|
||||
end
|
||||
rescue Timeout::Error => e
|
||||
Astute.logger.warn("Time detection (#{hook_timeout} sec) for node reboot has expired")
|
||||
end
|
||||
|
||||
if already_rebooted.values.include?(false)
|
||||
fail_nodes = already_rebooted.select {|k, v| !v }.keys
|
||||
ret['error'] = "Reboot command failed for nodes #{fail_nodes}. Check debug output for details"
|
||||
Astute.logger.warn(ret['error'])
|
||||
end
|
||||
|
||||
update_node_status(already_rebooted.select { |node, rebooted| rebooted }.keys)
|
||||
|
||||
ret
|
||||
end # reboot_hook
|
||||
|
||||
def validate_presence(data, key)
|
||||
raise "Missing a required parameter #{key}" unless data[key].present?
|
||||
end
|
||||
|
||||
def run_puppet(context, node_uids, puppet_manifest, puppet_modules, cwd, timeout, retries)
|
||||
# Prevent send report status to Nailgun
|
||||
hook_context = Context.new(context.task_id, HookReporter.new, LogParser::NoParsing.new)
|
||||
nodes = node_uids.map { |node_id| {'uid' => node_id.to_s, 'role' => 'hook'} }
|
||||
|
||||
Timeout::timeout(timeout) {
|
||||
PuppetdDeployer.deploy(
|
||||
hook_context,
|
||||
nodes,
|
||||
retries=retries,
|
||||
puppet_manifest,
|
||||
puppet_modules,
|
||||
cwd
|
||||
)
|
||||
}
|
||||
|
||||
!hook_context.status.has_value?('error')
|
||||
rescue Astute::MClientTimeout, Astute::MClientError, Timeout::Error => e
|
||||
Astute.logger.error("#{context.task_id}: puppet timeout error: #{e.message}")
|
||||
false
|
||||
end
|
||||
|
||||
def run_shell_command(context, node_uids, cmd, shell_retries, interval, timeout=60, cwd="/tmp")
|
||||
responses = nil
|
||||
(shell_retries + 1).times.each do |retry_number|
|
||||
shell = MClient.new(context,
|
||||
'execute_shell_command',
|
||||
node_uids,
|
||||
check_result=true,
|
||||
timeout=timeout,
|
||||
retries=1)
|
||||
|
||||
begin
|
||||
responses = shell.execute(:cmd => cmd, :cwd => cwd)
|
||||
rescue MClientTimeout, MClientError => e
|
||||
Astute.logger.error "#{context.task_id}: cmd: #{cmd} \n" \
|
||||
"mcollective error: #{e.message}"
|
||||
next
|
||||
end
|
||||
responses.each do |response|
|
||||
Astute.logger.debug(
|
||||
"#{context.task_id}: cmd: #{cmd}\n" \
|
||||
"cwd: #{cwd}\n" \
|
||||
"stdout: #{response[:data][:stdout]}\n" \
|
||||
"stderr: #{response[:data][:stderr]}\n" \
|
||||
"exit code: #{response[:data][:exit_code]}")
|
||||
end
|
||||
|
||||
node_uids -= responses.select { |response| response[:data][:exit_code] == 0 }
|
||||
.map { |response| response[:sender] }
|
||||
return nil if node_uids.empty?
|
||||
Astute.logger.warn "Problem while performing cmd on nodes: #{node_uids}. " \
|
||||
"Retrying... Attempt #{retry_number} of #{shell_retries}"
|
||||
sleep interval
|
||||
end
|
||||
|
||||
if responses
|
||||
responses.select { |response| response[:data][:exit_code] != 0 }
|
||||
.map {|r| "node #{r[:sender]} returned #{r[:data][:exit_code]}"}
|
||||
.join(", ")
|
||||
else
|
||||
"mclient failed to execute command"
|
||||
end
|
||||
end
|
||||
|
||||
def upload_file(context, node_uids, mco_params={})
|
||||
upload_mclient = Astute::MClient.new(context, "uploadfile", Array(node_uids))
|
||||
|
||||
mco_params['overwrite'] = true if mco_params['overwrite'].nil?
|
||||
mco_params['parents'] = true if mco_params['parents'].nil?
|
||||
mco_params['permissions'] ||= '0644'
|
||||
mco_params['user_owner'] ||= 'root'
|
||||
mco_params['group_owner'] ||= 'root'
|
||||
mco_params['dir_permissions'] ||= '0755'
|
||||
|
||||
upload_mclient.upload(
|
||||
:path => mco_params['path'],
|
||||
:content => mco_params['content'],
|
||||
:overwrite => mco_params['overwrite'],
|
||||
:parents => mco_params['parents'],
|
||||
:permissions => mco_params['permissions'],
|
||||
:user_owner => mco_params['user_owner'],
|
||||
:group_owner => mco_params['group_owner'],
|
||||
:dir_permissions => mco_params['dir_permissions']
|
||||
)
|
||||
|
||||
true
|
||||
rescue MClientTimeout, MClientError => e
|
||||
Astute.logger.error("#{context.task_id}: mcollective upload_file agent error: #{e.message}")
|
||||
false
|
||||
end
|
||||
|
||||
def perform_with_limit(nodes, &block)
|
||||
nodes.each_slice(Astute.config[:max_nodes_per_call]) do |part|
|
||||
block.call(part)
|
||||
end
|
||||
end
|
||||
|
||||
def run_shell_without_check(context, node_uids, cmd, timeout=10)
|
||||
shell = MClient.new(
|
||||
context,
|
||||
'execute_shell_command',
|
||||
node_uids,
|
||||
check_result=false,
|
||||
timeout=timeout
|
||||
)
|
||||
results = shell.execute(:cmd => cmd)
|
||||
results.inject({}) do |h, res|
|
||||
Astute.logger.debug(
|
||||
"#{context.task_id}: cmd: #{cmd}\n" \
|
||||
"stdout: #{res.results[:data][:stdout]}\n" \
|
||||
"stderr: #{res.results[:data][:stderr]}\n" \
|
||||
"exit code: #{res.results[:data][:exit_code]}")
|
||||
h.merge({res.results[:sender] => res.results[:data][:stdout].chomp})
|
||||
end
|
||||
end
|
||||
|
||||
def boot_time(uids)
|
||||
run_shell_without_check(
|
||||
@ctx,
|
||||
uids,
|
||||
"stat --printf='%Y' /proc/1",
|
||||
timeout=10
|
||||
)
|
||||
end
|
||||
|
||||
def update_node_status(uids)
|
||||
run_shell_without_check(
|
||||
@ctx,
|
||||
uids,
|
||||
"flock -w 0 -o /var/lock/nailgun-agent.lock -c '/usr/bin/nailgun-agent"\
|
||||
" 2>&1 | tee -a /var/log/nailgun-agent.log | "\
|
||||
"/usr/bin/logger -t nailgun-agent'",
|
||||
_timeout=60 # nailgun-agent start with random (30) delay
|
||||
)
|
||||
end
|
||||
|
||||
end # class
|
||||
|
||||
class HookReporter
|
||||
def report(msg)
|
||||
Astute.logger.debug msg
|
||||
end
|
||||
end
|
||||
|
||||
end # module
|
|
@ -1,280 +0,0 @@
|
|||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
module Network
|
||||
|
||||
def self.check_network(ctx, nodes)
|
||||
if nodes.empty?
|
||||
Astute.logger.info(
|
||||
"#{ctx.task_id}: Network checker: nodes list is empty. Nothing to check.")
|
||||
return {
|
||||
'status' => 'error',
|
||||
'error' => "Network verification requires a minimum of two nodes."
|
||||
}
|
||||
elsif nodes.length == 1
|
||||
Astute.logger.info(
|
||||
"#{ctx.task_id}: Network checker: nodes list contains one node only. Do nothing.")
|
||||
return {'nodes' => [{
|
||||
'uid' => nodes[0]['uid'],
|
||||
'networks' => nodes[0]['networks']
|
||||
}]}
|
||||
end
|
||||
|
||||
uids = nodes.map { |node| node['uid'].to_s }
|
||||
# TODO Everything breakes if agent not found. We have to handle that
|
||||
net_probe = MClient.new(ctx, "net_probe", uids)
|
||||
|
||||
versioning = Versioning.new(ctx)
|
||||
old_version, new_version = versioning.split_on_version(uids, '6.1.0')
|
||||
|
||||
old_uids = old_version.map { |node| node['uid'].to_i }
|
||||
new_uids = new_version.map { |node| node['uid'].to_i }
|
||||
old_nodes = nodes.select { |node| old_uids.include? node['uid'] }
|
||||
new_nodes = nodes.select { |node| new_uids.include? node['uid'] }
|
||||
|
||||
if old_uids.present?
|
||||
start_frame_listeners_60(ctx, net_probe, old_nodes)
|
||||
end
|
||||
|
||||
if new_uids.present?
|
||||
start_frame_listeners(ctx, net_probe, new_nodes)
|
||||
end
|
||||
ctx.reporter.report({'progress' => 30})
|
||||
|
||||
if old_uids.present?
|
||||
send_probing_frames_60(ctx, net_probe, old_nodes)
|
||||
end
|
||||
|
||||
if new_uids.present?
|
||||
send_probing_frames(ctx, net_probe, new_nodes)
|
||||
end
|
||||
ctx.reporter.report({'progress' => 60})
|
||||
|
||||
net_probe.discover(:nodes => uids)
|
||||
stats = net_probe.get_probing_info
|
||||
result = format_result(stats)
|
||||
Astute.logger.debug "#{ctx.task_id}: Network checking is done. Results:\n#{result.pretty_inspect}"
|
||||
|
||||
{'nodes' => result}
|
||||
end
|
||||
|
||||
def self.check_dhcp(ctx, nodes)
|
||||
uids = nodes.map { |node| node['uid'].to_s }
|
||||
net_probe = MClient.new(ctx, "net_probe", uids)
|
||||
|
||||
data_to_send = {}
|
||||
nodes.each do |node|
|
||||
data_to_send[node['uid'].to_s] = make_interfaces_to_send(node['networks'], joined=false).to_json
|
||||
end
|
||||
repeat = Astute.config.dhcp_repeat
|
||||
result = net_probe.dhcp_discover(:interfaces => data_to_send,
|
||||
:timeout => 10, :repeat => repeat).map do |response|
|
||||
format_dhcp_response(response)
|
||||
end
|
||||
|
||||
status = result.any?{|node| node[:status] == 'error'} && 'error' || 'ready'
|
||||
|
||||
{'nodes' => result, 'status'=> status}
|
||||
end
|
||||
|
||||
def self.multicast_verification(ctx, nodes)
|
||||
uids = nodes.map { |node| node['uid'].to_s }
|
||||
net_probe = MClient.new(ctx, "net_probe", uids)
|
||||
data_to_send = {}
|
||||
nodes.each do |node|
|
||||
data_to_send[node['uid']] = node
|
||||
end
|
||||
|
||||
listen_resp = net_probe.multicast_listen(:nodes => data_to_send.to_json)
|
||||
Astute.logger.debug("Mutlicast verification listen:\n#{listen_resp.pretty_inspect}")
|
||||
ctx.reporter.report({'progress' => 30})
|
||||
|
||||
send_resp = net_probe.multicast_send()
|
||||
Astute.logger.debug("Mutlicast verification send:\n#{send_resp.pretty_inspect}")
|
||||
ctx.reporter.report({'progress' => 60})
|
||||
|
||||
results = net_probe.multicast_info()
|
||||
Astute.logger.debug("Mutlicast verification info:\n#{results.pretty_inspect}")
|
||||
response = {}
|
||||
results.each do |node|
|
||||
if node.results[:data][:out].present?
|
||||
response[node.results[:sender].to_i] = JSON.parse(node.results[:data][:out])
|
||||
end
|
||||
end
|
||||
{'nodes' => response}
|
||||
end
|
||||
|
||||
def self.check_urls_access(ctx, nodes, urls)
|
||||
uids = nodes.map { |node| node['uid'].to_s }
|
||||
net_probe = MClient.new(ctx, "net_probe", uids)
|
||||
|
||||
result = net_probe.check_url_retrieval(:urls => urls)
|
||||
|
||||
{'nodes' => flatten_response(result), 'status'=> 'ready'}
|
||||
end
|
||||
|
||||
def self.check_repositories_with_setup(ctx, nodes)
|
||||
uids = nodes.map { |node| node['uid'].to_s }
|
||||
net_probe = MClient.new(ctx, "net_probe", uids, check_result=false)
|
||||
|
||||
data = nodes.inject({}) { |h, node| h.merge({node['uid'].to_s => node}) }
|
||||
|
||||
result = net_probe.check_repositories_with_setup(:data => data)
|
||||
bad_nodes = nodes.map { |n| n['uid'] } - result.map { |n| n.results[:sender] }
|
||||
|
||||
if bad_nodes.present?
|
||||
error_msg = "Astute could not get result from nodes #{bad_nodes}. " \
|
||||
"Please check mcollective log on problem nodes " \
|
||||
"for more details. Hint: try to execute check manually " \
|
||||
"using command from mcollective log on nodes, also " \
|
||||
"check nodes availability using command `mco ping` " \
|
||||
"on master node."
|
||||
raise MClientTimeout, error_msg
|
||||
end
|
||||
|
||||
{'nodes' => flatten_response(result), 'status'=> 'ready'}
|
||||
end
|
||||
|
||||
private
|
||||
def self.start_frame_listeners(ctx, net_probe, nodes)
|
||||
data_to_send = {}
|
||||
nodes.each do |node|
|
||||
data_to_send[node['uid'].to_s] = make_interfaces_to_send(node['networks'])
|
||||
end
|
||||
|
||||
uids = nodes.map { |node| node['uid'].to_s }
|
||||
|
||||
Astute.logger.debug(
|
||||
"#{ctx.task_id}: Network checker listen: nodes: #{uids} data:\n#{data_to_send.pretty_inspect}")
|
||||
|
||||
net_probe.discover(:nodes => uids)
|
||||
net_probe.start_frame_listeners(:interfaces => data_to_send.to_json)
|
||||
end
|
||||
|
||||
def self.send_probing_frames(ctx, net_probe, nodes)
|
||||
nodes.each_slice(Astute.config[:max_nodes_net_validation]) do |nodes_part|
|
||||
data_to_send = {}
|
||||
nodes_part.each do |node|
|
||||
data_to_send[node['uid'].to_s] = make_interfaces_to_send(node['networks'])
|
||||
end
|
||||
|
||||
uids = nodes_part.map { |node| node['uid'].to_s }
|
||||
|
||||
Astute.logger.debug(
|
||||
"#{ctx.task_id}: Network checker send: nodes: #{uids} data:\n#{data_to_send.pretty_inspect}")
|
||||
|
||||
net_probe.discover(:nodes => uids)
|
||||
net_probe.send_probing_frames(:interfaces => data_to_send.to_json)
|
||||
end
|
||||
end
|
||||
|
||||
def self.start_frame_listeners_60(ctx, net_probe, nodes)
|
||||
nodes.each do |node|
|
||||
data_to_send = make_interfaces_to_send(node['networks'])
|
||||
|
||||
Astute.logger.debug(
|
||||
"#{ctx.task_id}: Network checker listen: node: #{node['uid']} data:\n#{data_to_send.pretty_inspect}")
|
||||
|
||||
net_probe.discover(:nodes => [node['uid'].to_s])
|
||||
net_probe.start_frame_listeners(:interfaces => data_to_send.to_json)
|
||||
end
|
||||
end
|
||||
|
||||
def self.send_probing_frames_60(ctx, net_probe, nodes)
|
||||
nodes.each do |node|
|
||||
data_to_send = make_interfaces_to_send(node['networks'])
|
||||
|
||||
Astute.logger.debug(
|
||||
"#{ctx.task_id}: Network checker send: node: #{node['uid']} data:\n#{data_to_send.pretty_inspect}")
|
||||
|
||||
net_probe.discover(:nodes => [node['uid'].to_s])
|
||||
net_probe.send_probing_frames(:interfaces => data_to_send.to_json)
|
||||
end
|
||||
end
|
||||
|
||||
def self.make_interfaces_to_send(networks, joined=true)
|
||||
data_to_send = {}
|
||||
networks.each do |network|
|
||||
if joined
|
||||
data_to_send[network['iface']] = network['vlans'].join(",")
|
||||
else
|
||||
data_to_send[network['iface']] = network['vlans']
|
||||
end
|
||||
end
|
||||
|
||||
data_to_send
|
||||
end
|
||||
|
||||
def self.format_dhcp_response(response)
|
||||
node_result = {:uid => response.results[:sender],
|
||||
:status=>'ready'}
|
||||
if response.results[:data][:out].present?
|
||||
Astute.logger.debug("DHCP checker received:\n#{response.pretty_inspect}")
|
||||
node_result[:data] = JSON.parse(response.results[:data][:out])
|
||||
elsif response.results[:data][:err].present?
|
||||
Astute.logger.debug("DHCP checker errred with:\n#{response.pretty_inspect}")
|
||||
node_result[:status] = 'error'
|
||||
node_result[:error_msg] = 'Error in dhcp checker. Check logs for details'
|
||||
end
|
||||
node_result
|
||||
end
|
||||
|
||||
def self.format_result(stats)
|
||||
uids = stats.map{|node| node.results[:sender]}.sort
|
||||
stats.map do |node|
|
||||
{
|
||||
'uid' => node.results[:sender],
|
||||
'networks' => check_vlans_by_traffic(
|
||||
node.results[:sender],
|
||||
uids,
|
||||
node.results[:data][:neighbours])
|
||||
}
|
||||
end
|
||||
end
|
||||
|
||||
def self.flatten_response(response)
|
||||
response.map do |node|
|
||||
{:out => node.results[:data][:out],
|
||||
:err => node.results[:data][:err],
|
||||
:status => node.results[:data][:status],
|
||||
:uid => node.results[:sender]}
|
||||
end
|
||||
end
|
||||
|
||||
def self.check_vlans_by_traffic(uid, uids, data)
|
||||
data.map do |iface, vlans|
|
||||
{
|
||||
'iface' => iface,
|
||||
'vlans' => remove_extra_data(uid, uids, vlans).select {|k,v|
|
||||
v.keys.present?
|
||||
}.keys.map(&:to_i)
|
||||
}
|
||||
end
|
||||
end
|
||||
|
||||
# Remove unnecessary data
|
||||
def self.remove_extra_data(uid, uids, vlans)
|
||||
vlans.each do |k, data|
|
||||
# remove data sent by node itself
|
||||
data.delete(uid)
|
||||
# remove data sent by nodes from different envs
|
||||
data.keep_if { |k, v| uids.include?(k) }
|
||||
end
|
||||
vlans
|
||||
end
|
||||
|
||||
end
|
||||
end
|
||||
|
|
@ -1,101 +0,0 @@
|
|||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
|
||||
require 'active_support/core_ext/hash/indifferent_access'
|
||||
require 'ostruct'
|
||||
|
||||
module Astute
|
||||
class Node < OpenStruct
|
||||
def initialize(hash=nil)
|
||||
if hash && (uid = hash['uid'])
|
||||
hash = hash.dup
|
||||
hash['uid'] = uid.to_s
|
||||
else
|
||||
raise TypeError.new("Invalid data: #{hash.inspect}")
|
||||
end
|
||||
super hash
|
||||
end
|
||||
|
||||
def [](key)
|
||||
send key
|
||||
end
|
||||
|
||||
def []=(key, value)
|
||||
send "#{key}=", value
|
||||
end
|
||||
|
||||
def uid
|
||||
@table[:uid]
|
||||
end
|
||||
|
||||
def uid=(_)
|
||||
raise TypeError.new('Read-only attribute')
|
||||
end
|
||||
|
||||
def to_hash
|
||||
@table.with_indifferent_access
|
||||
end
|
||||
|
||||
def fetch(key, default)
|
||||
ret = self[key]
|
||||
if ret.nil?
|
||||
ret = default
|
||||
end
|
||||
ret
|
||||
end
|
||||
end
|
||||
|
||||
class NodesHash < Hash
|
||||
alias uids keys
|
||||
alias nodes values
|
||||
|
||||
def self.build(nodes)
|
||||
return nodes if nodes.kind_of? self
|
||||
nodes.inject(self.new) do |hash, node|
|
||||
hash << node
|
||||
hash
|
||||
end
|
||||
end
|
||||
|
||||
def <<(node)
|
||||
node = normalize_value(node)
|
||||
self[node.uid] = node
|
||||
self
|
||||
end
|
||||
|
||||
def push(*nodes)
|
||||
nodes.each{|node| self.<< node }
|
||||
self
|
||||
end
|
||||
|
||||
def [](key)
|
||||
super key.to_s
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def []=(*args)
|
||||
super
|
||||
end
|
||||
|
||||
def normalize_value(node)
|
||||
if node.kind_of? Node
|
||||
node
|
||||
else
|
||||
Node.new(node.to_hash)
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
|
@ -1,197 +0,0 @@
|
|||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
|
||||
module Astute
|
||||
class NodesRemover
|
||||
|
||||
def initialize(ctx, nodes, reboot=true)
|
||||
@ctx = ctx
|
||||
@nodes = NodesHash.build(nodes)
|
||||
@reboot = reboot
|
||||
end
|
||||
|
||||
def remove
|
||||
# TODO(mihgen): 1. Nailgun should process node error message
|
||||
# 2. Should we rename nodes -> removed_nodes array?
|
||||
# 3. If exception is raised here, we should not fully fall into error, but only failed node
|
||||
erased_nodes, error_nodes, inaccessible_nodes = remove_nodes(@nodes)
|
||||
retry_remove_nodes(error_nodes, erased_nodes,
|
||||
Astute.config[:mc_retries], Astute.config[:mc_retry_interval])
|
||||
|
||||
retry_remove_nodes(inaccessible_nodes, erased_nodes,
|
||||
Astute.config[:mc_retries], Astute.config[:mc_retry_interval])
|
||||
|
||||
answer = {'nodes' => serialize_nodes(erased_nodes)}
|
||||
|
||||
if inaccessible_nodes.present?
|
||||
serialized_inaccessible_nodes = serialize_nodes(inaccessible_nodes)
|
||||
answer.merge!({'inaccessible_nodes' => serialized_inaccessible_nodes})
|
||||
|
||||
Astute.logger.warn "#{@ctx.task_id}: Removing of nodes\n#{@nodes.uids.pretty_inspect} finished " \
|
||||
"with errors. Nodes\n#{serialized_inaccessible_nodes.pretty_inspect} are inaccessible"
|
||||
end
|
||||
|
||||
if error_nodes.present?
|
||||
serialized_error_nodes = serialize_nodes(error_nodes)
|
||||
answer.merge!({'status' => 'error', 'error_nodes' => serialized_error_nodes})
|
||||
|
||||
Astute.logger.error "#{@ctx.task_id}: Removing of nodes\n#{@nodes.uids.pretty_inspect} finished " \
|
||||
"with errors:\n#{serialized_error_nodes.pretty_inspect}"
|
||||
end
|
||||
Astute.logger.info "#{@ctx.task_id}: Finished removing of nodes:\n#{@nodes.uids.pretty_inspect}"
|
||||
|
||||
answer
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def serialize_nodes(nodes)
|
||||
nodes.nodes.map(&:to_hash)
|
||||
end
|
||||
|
||||
# When :mclient_remove property is true (the default behavior), we send
|
||||
# the node to mclient for removal (MBR, restarting etc), if it's false
|
||||
# the node is skipped from mclient
|
||||
def skipped_unskipped_mclient_nodes(nodes)
|
||||
mclient_skipped_nodes = NodesHash.build(
|
||||
nodes.values.select { |node| not node.fetch(:mclient_remove, true) }
|
||||
)
|
||||
mclient_nodes = NodesHash.build(
|
||||
nodes.values.select { |node| node.fetch(:mclient_remove, true) }
|
||||
)
|
||||
|
||||
Astute.logger.debug "#{@ctx.task_id}: Split nodes: #{mclient_skipped_nodes}, #{mclient_nodes}"
|
||||
|
||||
[mclient_skipped_nodes, mclient_nodes]
|
||||
end
|
||||
|
||||
def get_already_removed_nodes(nodes)
|
||||
removed_nodes = []
|
||||
control_time = {}
|
||||
|
||||
nodes.uids.sort.each_slice(Astute.config[:max_nodes_per_call]) do |part|
|
||||
control_time.merge!(get_boot_time(part))
|
||||
end
|
||||
|
||||
nodes.each do |uid, node|
|
||||
boot_time = control_time[uid].to_i
|
||||
next if boot_time.zero?
|
||||
if node.boot_time
|
||||
removed_nodes << uid if boot_time != node.boot_time
|
||||
else
|
||||
node.boot_time = boot_time
|
||||
end
|
||||
end
|
||||
removed_nodes
|
||||
end
|
||||
|
||||
def remove_nodes(nodes)
|
||||
if nodes.empty?
|
||||
Astute.logger.info "#{@ctx.task_id}: Nodes to remove are not provided. Do nothing."
|
||||
return Array.new(3){ NodesHash.new }
|
||||
end
|
||||
|
||||
erased_nodes, mclient_nodes = skipped_unskipped_mclient_nodes(nodes)
|
||||
|
||||
removed_nodes = get_already_removed_nodes(mclient_nodes)
|
||||
removed_nodes.each do |uid|
|
||||
erased_node = Node.new('uid' => uid)
|
||||
erased_nodes << erased_node
|
||||
mclient_nodes.delete(uid)
|
||||
Astute.logger.info "#{@ctx.task_id}: Node #{uid} is removed already, skipping"
|
||||
end
|
||||
|
||||
responses = mclient_remove_nodes(mclient_nodes)
|
||||
inaccessible_uids = mclient_nodes.uids - responses.map { |response| response[:sender] }
|
||||
inaccessible_nodes = NodesHash.build(inaccessible_uids.map do |uid|
|
||||
{'uid' => uid, 'error' => 'Node not answered by RPC.', 'boot_time' => mclient_nodes[uid][:boot_time]}
|
||||
end)
|
||||
error_nodes = NodesHash.new
|
||||
|
||||
responses.each do |response|
|
||||
node = Node.new('uid' => response[:sender])
|
||||
if response[:statuscode] != 0
|
||||
node['error'] = "RPC agent 'erase_node' failed. Result:\n#{response.pretty_inspect}"
|
||||
error_nodes << node
|
||||
elsif @reboot && !response[:data][:rebooted]
|
||||
node['error'] = "RPC method 'erase_node' failed with message: #{response[:data][:error_msg]}"
|
||||
error_nodes << node
|
||||
else
|
||||
erased_nodes << node
|
||||
end
|
||||
end
|
||||
[erased_nodes, error_nodes, inaccessible_nodes]
|
||||
end
|
||||
|
||||
def retry_remove_nodes(error_nodes, erased_nodes, retries=3, interval=1)
|
||||
retries.times do
|
||||
retried_erased_nodes = remove_nodes(error_nodes)[0]
|
||||
retried_erased_nodes.each do |uid, node|
|
||||
error_nodes.delete uid
|
||||
erased_nodes << node
|
||||
end
|
||||
return if error_nodes.empty?
|
||||
sleep(interval) if interval > 0
|
||||
end
|
||||
end
|
||||
|
||||
def mclient_remove_nodes(nodes)
|
||||
Astute.logger.info "#{@ctx.task_id}: Starting removing of nodes:\n#{nodes.uids.pretty_inspect}"
|
||||
results = []
|
||||
|
||||
nodes.uids.sort.each_slice(Astute.config[:max_nodes_per_remove_call]).with_index do |part, i|
|
||||
sleep Astute.config[:nodes_remove_interval] if i != 0
|
||||
results += mclient_remove_piece_nodes(part)
|
||||
end
|
||||
results
|
||||
end
|
||||
|
||||
def mclient_remove_piece_nodes(nodes)
|
||||
remover = MClient.new(@ctx, "erase_node", nodes, check_result=false)
|
||||
responses = remover.erase_node(:reboot => @reboot)
|
||||
Astute.logger.debug "#{@ctx.task_id}: Data received from nodes:\n#{responses.pretty_inspect}"
|
||||
responses.map(&:results)
|
||||
end
|
||||
|
||||
def run_shell_without_check(context, node_uids, cmd, timeout=10)
|
||||
shell = MClient.new(
|
||||
context,
|
||||
'execute_shell_command',
|
||||
node_uids,
|
||||
check_result=false,
|
||||
timeout=timeout
|
||||
)
|
||||
results = shell.execute(:cmd => cmd)
|
||||
results.inject({}) do |h, res|
|
||||
Astute.logger.debug(
|
||||
"#{context.task_id}: cmd: #{cmd}\n" \
|
||||
"stdout: #{res.results[:data][:stdout]}\n" \
|
||||
"stderr: #{res.results[:data][:stderr]}\n" \
|
||||
"exit code: #{res.results[:data][:exit_code]}")
|
||||
h.merge({res.results[:sender] => res.results[:data][:stdout].chomp})
|
||||
end
|
||||
end
|
||||
|
||||
def get_boot_time(node_uids)
|
||||
run_shell_without_check(
|
||||
@ctx,
|
||||
node_uids,
|
||||
"stat --printf='%Y' /proc/1",
|
||||
timeout=10
|
||||
)
|
||||
end
|
||||
|
||||
end
|
||||
end
|
|
@ -1,296 +0,0 @@
|
|||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
|
||||
class Orchestrator
|
||||
def initialize(log_parsing=false)
|
||||
@log_parsing = log_parsing
|
||||
end
|
||||
|
||||
def node_type(reporter, task_id, nodes_uids, timeout=nil)
|
||||
provisioner = Provisioner.new(@log_parsing)
|
||||
provisioner.node_type(reporter, task_id, nodes_uids, timeout)
|
||||
end
|
||||
|
||||
def execute_tasks(up_reporter, task_id, tasks)
|
||||
ctx = Context.new(task_id, up_reporter)
|
||||
Astute::NailgunHooks.new(tasks, ctx, 'execute_tasks').process
|
||||
report_result({}, up_reporter)
|
||||
end
|
||||
|
||||
# Deploy method which use small tasks, but run block of tasks for role
|
||||
# instead of run it using full graph. Use from 6.1 to 8.0. Report progress
|
||||
# based on puppet logs
|
||||
def granular_deploy(up_reporter, task_id, deployment_info, pre_deployment=[], post_deployment=[])
|
||||
time_start = Time.now.to_i
|
||||
deploy_cluster(
|
||||
up_reporter,
|
||||
task_id,
|
||||
deployment_info,
|
||||
Astute::DeploymentEngine::GranularDeployment,
|
||||
pre_deployment,
|
||||
post_deployment
|
||||
)
|
||||
ensure
|
||||
Astute.logger.info "Deployment summary: time was spent " \
|
||||
"#{time_summary(time_start)}"
|
||||
end
|
||||
|
||||
# Deploy method which use small tasks in full graph.
|
||||
# Use from 8.0 (experimental). Report progress based on tasks
|
||||
def task_deploy(up_reporter, task_id, deployment_options = {})
|
||||
time_start = Time.now.to_i
|
||||
proxy_reporter = ProxyReporter::TaskProxyReporter.new(
|
||||
up_reporter
|
||||
)
|
||||
context = Context.new(task_id, proxy_reporter)
|
||||
Astute.logger.info "Task based deployment will be used"
|
||||
|
||||
deployment_engine = TaskDeployment.new(context)
|
||||
write_input_data_to_file(context, deployment_options) if Astute.config.enable_graph_file
|
||||
deployment_engine.deploy(deployment_options)
|
||||
ensure
|
||||
Astute.logger.info "Deployment summary: time was spent " \
|
||||
"#{time_summary(time_start)}"
|
||||
end
|
||||
|
||||
def provision(up_reporter, task_id, provisioning_info)
|
||||
time_start = Time.now.to_i
|
||||
proxy_reporter = ProxyReporter::ProvisiningProxyReporter.new(
|
||||
up_reporter,
|
||||
provisioning_info
|
||||
)
|
||||
provisioner = Provisioner.new(@log_parsing)
|
||||
if provisioning_info['pre_provision']
|
||||
image_build_log = "/var/log/fuel-agent-env" \
|
||||
"-#{calculate_cluster_id(provisioning_info)}.log"
|
||||
Astute.logger.info "Please check image build log here: " \
|
||||
"#{image_build_log}"
|
||||
ctx = Context.new(task_id, proxy_reporter)
|
||||
provisioner.report_image_provision(
|
||||
proxy_reporter,
|
||||
task_id,
|
||||
provisioning_info['nodes'],
|
||||
image_log_parser(provisioning_info)
|
||||
) do
|
||||
begin
|
||||
Astute::NailgunHooks.new(
|
||||
provisioning_info['pre_provision'],
|
||||
ctx,
|
||||
'provision'
|
||||
).process
|
||||
rescue Astute::DeploymentEngineError => e
|
||||
raise e, "Image build task failed. Please check " \
|
||||
"build log here for details: #{image_build_log}. " \
|
||||
"Hint: restart deployment can help if no error in build " \
|
||||
"log was found"
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
# NOTE(kozhukalov): Some of our pre-provision tasks need cobbler to be synced
|
||||
# once those tasks are finished. It looks like the easiest way to do this
|
||||
# inside mcollective docker container is to use Astute binding capabilities.
|
||||
cobbler = CobblerManager.new(provisioning_info['engine'], up_reporter)
|
||||
cobbler.sync
|
||||
|
||||
provisioner.provision(
|
||||
proxy_reporter,
|
||||
task_id,
|
||||
provisioning_info
|
||||
)
|
||||
ensure
|
||||
Astute.logger.info "Provision summary: time was spent " \
|
||||
"#{time_summary(time_start)}"
|
||||
end
|
||||
|
||||
def remove_nodes(reporter, task_id, engine_attrs, nodes, options={})
|
||||
# FIXME(vsharshov): bug/1463881: In case of post deployment we mark all nodes
|
||||
# as ready. In this case we will get empty nodes.
|
||||
return if nodes.empty?
|
||||
|
||||
options.reverse_merge!({
|
||||
:reboot => true,
|
||||
:raise_if_error => false,
|
||||
:reset => false
|
||||
})
|
||||
|
||||
result = perform_pre_deletion_tasks(reporter, task_id, nodes, options)
|
||||
return result if result['status'] != 'ready'
|
||||
|
||||
provisioner = Provisioner.new(@log_parsing)
|
||||
provisioner.remove_nodes(
|
||||
reporter,
|
||||
task_id,
|
||||
engine_attrs,
|
||||
nodes,
|
||||
options
|
||||
)
|
||||
end
|
||||
|
||||
def stop_puppet_deploy(reporter, task_id, nodes)
|
||||
# FIXME(vsharshov): bug/1463881: In case of post deployment we mark all nodes
|
||||
# as ready. If we run stop deployment we will get empty nodes.
|
||||
return if nodes.empty?
|
||||
|
||||
nodes_uids = nodes.map { |n| n['uid'] }.uniq
|
||||
puppetd = MClient.new(Context.new(task_id, reporter), "puppetd", nodes_uids, check_result=false)
|
||||
puppetd.stop_and_disable
|
||||
end
|
||||
|
||||
def stop_provision(reporter, task_id, engine_attrs, nodes)
|
||||
provisioner = Provisioner.new(@log_parsing)
|
||||
provisioner.stop_provision(reporter, task_id, engine_attrs, nodes)
|
||||
end
|
||||
|
||||
def dump_environment(reporter, task_id, settings)
|
||||
Dump.dump_environment(Context.new(task_id, reporter), settings)
|
||||
end
|
||||
|
||||
def verify_networks(reporter, task_id, nodes)
|
||||
ctx = Context.new(task_id, reporter)
|
||||
validate_nodes_access(ctx, nodes)
|
||||
Network.check_network(ctx, nodes)
|
||||
end
|
||||
|
||||
def check_dhcp(reporter, task_id, nodes)
|
||||
ctx = Context.new(task_id, reporter)
|
||||
validate_nodes_access(ctx, nodes)
|
||||
Network.check_dhcp(ctx, nodes)
|
||||
end
|
||||
|
||||
def multicast_verification(reporter, task_id, nodes)
|
||||
ctx = Context.new(task_id, reporter)
|
||||
validate_nodes_access(ctx, nodes)
|
||||
Network.multicast_verification(ctx, nodes)
|
||||
end
|
||||
|
||||
def check_repositories(reporter, task_id, nodes, urls)
|
||||
ctx = Context.new(task_id, reporter)
|
||||
validate_nodes_access(ctx, nodes)
|
||||
Network.check_urls_access(ctx, nodes, urls)
|
||||
end
|
||||
|
||||
def check_repositories_with_setup(reporter, task_id, nodes)
|
||||
ctx = Context.new(task_id, reporter)
|
||||
validate_nodes_access(ctx, nodes)
|
||||
Network.check_repositories_with_setup(ctx, nodes)
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def deploy_cluster(up_reporter, task_id, deployment_info, deploy_engine, pre_deployment, post_deployment)
|
||||
proxy_reporter = ProxyReporter::DeploymentProxyReporter.new(up_reporter, deployment_info)
|
||||
log_parser = @log_parsing ? LogParser::ParseDeployLogs.new : LogParser::NoParsing.new
|
||||
context = Context.new(task_id, proxy_reporter, log_parser)
|
||||
deploy_engine_instance = deploy_engine.new(context)
|
||||
Astute.logger.info "Using #{deploy_engine_instance.class} for deployment."
|
||||
|
||||
deploy_engine_instance.deploy(deployment_info, pre_deployment, post_deployment)
|
||||
|
||||
context.status
|
||||
end
|
||||
|
||||
def report_result(result, reporter)
|
||||
default_result = {'status' => 'ready', 'progress' => 100}
|
||||
|
||||
result = {} unless result.instance_of?(Hash)
|
||||
status = default_result.merge(result)
|
||||
reporter.report(status)
|
||||
end
|
||||
|
||||
def validate_nodes_access(ctx, nodes)
|
||||
nodes_types = node_type(ctx.reporter, ctx.task_id, nodes.map{ |n| n['uid'] }, timeout=10)
|
||||
not_available_nodes = nodes.map { |n| n['uid'].to_s } - nodes_types.map { |n| n['uid'].to_s }
|
||||
unless not_available_nodes.empty?
|
||||
raise "Network verification not available because nodes #{not_available_nodes} " \
|
||||
"not available via mcollective"
|
||||
end
|
||||
end
|
||||
|
||||
def image_log_parser(provisioning_info)
|
||||
log_parser = LogParser::ParseImageBuildLogs.new
|
||||
log_parser.cluster_id = calculate_cluster_id(provisioning_info)
|
||||
log_parser
|
||||
end
|
||||
|
||||
def calculate_cluster_id(provisioning_info)
|
||||
return nil unless provisioning_info['pre_provision'].present?
|
||||
cmd = provisioning_info['pre_provision'].first.fetch('parameters', {}).fetch('cmd', "")
|
||||
# find cluster id from cmd using pattern fuel-agent-env-<Integer>.log
|
||||
# FIXME(vsharshov): https://bugs.launchpad.net/fuel/+bug/1449512
|
||||
cluster_id = cmd[/fuel-agent-env-(\d+)/, 1]
|
||||
Astute.logger.debug "Cluster id: #{cluster_id}"
|
||||
cluster_id
|
||||
end
|
||||
|
||||
def check_for_offline_nodes(reporter, task_id, nodes)
|
||||
PreDelete.check_for_offline_nodes(Context.new(task_id, reporter), nodes)
|
||||
end
|
||||
|
||||
def check_ceph_osds(reporter, task_id, nodes)
|
||||
PreDelete.check_ceph_osds(Context.new(task_id, reporter), nodes)
|
||||
end
|
||||
|
||||
def remove_ceph_mons(reporter, task_id, nodes)
|
||||
PreDelete.remove_ceph_mons(Context.new(task_id, reporter), nodes)
|
||||
end
|
||||
|
||||
def perform_pre_deletion_tasks(reporter, task_id, nodes, options={})
|
||||
result = {'status' => 'ready'}
|
||||
# This option is no longer Ceph-specific and should be renamed
|
||||
# FIXME(rmoe): https://bugs.launchpad.net/fuel/+bug/1454377
|
||||
if options[:check_ceph]
|
||||
result = check_for_offline_nodes(reporter, task_id, nodes)
|
||||
return result if result['status'] != 'ready'
|
||||
result = check_ceph_osds(reporter, task_id, nodes)
|
||||
return result if result['status'] != 'ready'
|
||||
result = remove_ceph_mons(reporter, task_id, nodes)
|
||||
end
|
||||
result
|
||||
end
|
||||
|
||||
def time_summary(time)
|
||||
amount_time = (Time.now.to_i - time).to_i
|
||||
Time.at(amount_time).utc.strftime("%H:%M:%S")
|
||||
end
|
||||
|
||||
# Dump the task graph data to a file
|
||||
# @param [Astute::Context] context
|
||||
# @param [Hash] data
|
||||
def write_input_data_to_file(context, data={})
|
||||
yaml_file = File.join(
|
||||
Astute.config.graph_dot_dir,
|
||||
"graph-#{context.task_id}.yaml"
|
||||
)
|
||||
data = filter_sensitive_data(data)
|
||||
File.open(yaml_file, 'w') { |f| f.write(YAML.dump(data)) }
|
||||
Astute.logger.info("Check inpute data file #{yaml_file}")
|
||||
end
|
||||
|
||||
# Remove the potentially sensitive data
|
||||
# from the task parameters before dumping the graph
|
||||
# @param [Hash] data
|
||||
# @return [Hash]
|
||||
def filter_sensitive_data(data)
|
||||
data = data.deep_dup
|
||||
data[:tasks_graph].each do |_node_id, tasks|
|
||||
tasks.each { |task| task.delete('parameters') }
|
||||
end
|
||||
data
|
||||
end
|
||||
|
||||
end # class
|
||||
end # module
|
|
@ -1,40 +0,0 @@
|
|||
# Copyright 2014 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
class PostPatchingHa < PostDeployAction
|
||||
|
||||
def process(deployment_info, context)
|
||||
return if deployment_info.first['openstack_version_prev'].nil? ||
|
||||
deployment_info.first['deployment_mode'] !~ /ha/i
|
||||
|
||||
controller_nodes = deployment_info.select{ |n| n['role'] =~ /controller/i }.map{ |n| n['uid'] }
|
||||
return if controller_nodes.empty?
|
||||
|
||||
Astute.logger.info "Starting unmigration of pacemaker services from " \
|
||||
"nodes\n#{controller_nodes.pretty_inspect}"
|
||||
|
||||
Astute::Pacemaker.commands(action='start', deployment_info).each do |pcmk_unban_cmd|
|
||||
response = run_shell_command(context, controller_nodes, pcmk_unban_cmd)
|
||||
|
||||
if response[:data][:exit_code] != 0
|
||||
Astute.logger.warn "#{context.task_id}: Failed to unban service, "\
|
||||
"check the debugging output for details"
|
||||
end
|
||||
end
|
||||
|
||||
Astute.logger.info "#{context.task_id}: Finished post-patching-ha hook"
|
||||
end #process
|
||||
end #class
|
||||
end
|
|
@ -1,44 +0,0 @@
|
|||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
class RestartRadosgw < PostDeploymentAction
|
||||
|
||||
def process(deployment_info, context)
|
||||
ceph_node = deployment_info.find { |n| n['role'] == 'ceph-osd' }
|
||||
objects_ceph = ceph_node && ceph_node.fetch('storage', {}).fetch('objects_ceph')
|
||||
|
||||
return unless objects_ceph
|
||||
Astute.logger.info "Start restarting radosgw on controller nodes"
|
||||
|
||||
cmd = <<-RESTART_RADOSGW
|
||||
(test -f /etc/init.d/ceph-radosgw && /etc/init.d/ceph-radosgw restart) ||
|
||||
(test -f /etc/init.d/radosgw && /etc/init.d/radosgw restart);
|
||||
RESTART_RADOSGW
|
||||
cmd.tr!("\n"," ")
|
||||
|
||||
controller_nodes = deployment_info.first['nodes'].inject([]) do |c_n, n|
|
||||
c_n << n['uid'] if ['controller', 'primary-controller'].include? n['role']
|
||||
c_n
|
||||
end
|
||||
|
||||
response = run_shell_command(context, controller_nodes, cmd)
|
||||
if response[:data][:exit_code] != 0
|
||||
Astute.logger.warn "#{context.task_id}: Fail to restart radosgw, "\
|
||||
"check the debugging output for details"
|
||||
end
|
||||
Astute.logger.info "#{context.task_id}: Finish restarting radosgw on controller nodes"
|
||||
end #process
|
||||
end #class
|
||||
end
|
|
@ -1,65 +0,0 @@
|
|||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
|
||||
class UpdateClusterHostsInfo < PostDeploymentAction
|
||||
|
||||
def process(deployment_info, context)
|
||||
Astute.logger.info "Updating /etc/hosts in all cluster nodes"
|
||||
return if deployment_info.empty?
|
||||
|
||||
response = nil
|
||||
deployment_info.first['nodes'].each do |node|
|
||||
upload_file(node['uid'],
|
||||
deployment_info.first['nodes'].to_yaml,
|
||||
context)
|
||||
|
||||
cmd = <<-UPDATE_HOSTS
|
||||
ruby -r 'yaml' -e 'y = YAML.load_file("/etc/astute.yaml");
|
||||
y["nodes"] = YAML.load_file("/tmp/astute.yaml");
|
||||
File.open("/etc/astute.yaml", "w") { |f| f.write y.to_yaml }';
|
||||
puppet apply --logdest syslog --debug -e '$settings=parseyaml($::astute_settings_yaml)
|
||||
$nodes_hash=$settings["nodes"]
|
||||
class {"l23network::hosts_file": nodes => $nodes_hash }'
|
||||
UPDATE_HOSTS
|
||||
cmd.tr!("\n"," ")
|
||||
|
||||
response = run_shell_command(context, Array(node['uid']), cmd)
|
||||
if response[:data][:exit_code] != 0
|
||||
Astute.logger.warn "#{context.task_id}: Fail to update /etc/hosts, "\
|
||||
"check the debugging output for node "\
|
||||
"#{node['uid']} for details"
|
||||
end
|
||||
end
|
||||
|
||||
Astute.logger.info "#{context.task_id}: Updating /etc/hosts is done"
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def upload_file(node_uid, content, context)
|
||||
upload_mclient = Astute::MClient.new(context, "uploadfile", Array(node_uid))
|
||||
upload_mclient.upload(:path => "/tmp/astute.yaml",
|
||||
:content => content,
|
||||
:overwrite => true,
|
||||
:parents => true,
|
||||
:permissions => '0600'
|
||||
)
|
||||
rescue MClientTimeout, MClientError => e
|
||||
Astute.logger.error("#{context.task_id}: mcollective upload_file agent error: #{e.message}")
|
||||
end
|
||||
|
||||
end #class
|
||||
end
|
|
@ -1,75 +0,0 @@
|
|||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
class UpdateNoQuorumPolicy < PostDeploymentAction
|
||||
|
||||
def process(deployment_info, context)
|
||||
return if deployment_info.first['deployment_mode'] !~ /ha/i
|
||||
|
||||
# NOTE(bogdando) use 'suicide' if fencing is enabled in corosync
|
||||
xml = <<-EOF
|
||||
<diff>
|
||||
<diff-removed>
|
||||
<cib>
|
||||
<configuration>
|
||||
<crm_config>
|
||||
<cluster_property_set id="cib-bootstrap-options">
|
||||
<nvpair value="ignore" id="cib-bootstrap-options-no-quorum-policy"/>
|
||||
</cluster_property_set>
|
||||
</crm_config>
|
||||
</configuration>
|
||||
</cib>
|
||||
</diff-removed>
|
||||
<diff-added>
|
||||
<cib>
|
||||
<configuration>
|
||||
<crm_config>
|
||||
<cluster_property_set id="cib-bootstrap-options">
|
||||
<nvpair value="stop" id="cib-bootstrap-options-no-quorum-policy"/>
|
||||
</cluster_property_set>
|
||||
</crm_config>
|
||||
</configuration>
|
||||
</cib>
|
||||
</diff-added>
|
||||
</diff>
|
||||
EOF
|
||||
cmd = "/usr/sbin/cibadmin --patch --sync-call --xml-text '#{xml}'"
|
||||
cmd.tr!("\n"," ")
|
||||
|
||||
controllers_count = deployment_info.select {|n|
|
||||
['controller', 'primary-controller'].include? n['role']
|
||||
}.size
|
||||
if controllers_count > 2
|
||||
primary_controller = deployment_info.find {|n| n['role'] == 'primary-controller' }
|
||||
if context.status[primary_controller['uid']] == 'error'
|
||||
Astute.logger.info "Update quorum policy for corosync cluster " \
|
||||
"disabled because of primary-controller status is error"
|
||||
return
|
||||
end
|
||||
|
||||
Astute.logger.info "Started updating no quorum policy for corosync cluster"
|
||||
response = run_shell_command(context, Array(primary_controller['uid']), cmd)
|
||||
if response[:data][:exit_code] != 0
|
||||
Astute.logger.warn "#{context.task_id}: Failed to update no "\
|
||||
"quorum policy for corosync cluster,"
|
||||
end
|
||||
Astute.logger.info "#{context.task_id}: Finished updating "\
|
||||
"no quorum policy for corosync cluster"
|
||||
else
|
||||
Astute.logger.info "No need to update quorum policy for corosync cluster"
|
||||
end
|
||||
end #process
|
||||
end #class
|
||||
end
|
|
@ -1,111 +0,0 @@
|
|||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
|
||||
class CirrosError < AstuteError; end
|
||||
|
||||
class UploadCirrosImage < PostDeploymentAction
|
||||
|
||||
def process(deployment_info, context)
|
||||
# Mark controller node as error if present
|
||||
node = deployment_info.find { |n| n['role'] == 'primary-controller' }
|
||||
node = deployment_info.find { |n| n['role'] == 'controller' } unless node
|
||||
node = deployment_info.last unless node
|
||||
|
||||
controller = node['nodes'].find { |n| n['role'] == 'primary-controller' }
|
||||
controller = node['nodes'].find { |n| n['role'] == 'controller' } unless controller
|
||||
|
||||
if controller.nil?
|
||||
Astute.logger.debug "Could not find controller in nodes in facts! " \
|
||||
"Please check logs to be sure that it is correctly generated."
|
||||
return
|
||||
end
|
||||
# controller['test_vm_image'] contains a hash like that:
|
||||
# controller['test_vm_image'] = {
|
||||
# 'disk_format' => 'qcow2',
|
||||
# 'container_format' => 'bare',
|
||||
# 'public' => 'true',
|
||||
# 'img_name' => 'TestVM',
|
||||
# 'os_name' => 'cirros',
|
||||
# 'img_path' => '/opt/vm/cirros-x86_64-disk.img',
|
||||
# 'glance_properties' => '--property murano_image_info=\'{\"title\": \"Murano Demo\", \"type\": \"cirros.demo\"}\''
|
||||
# }
|
||||
|
||||
os = node['test_vm_image']
|
||||
cmd = ". /root/openrc && /usr/bin/glance image-list"
|
||||
|
||||
# waited until the glance is started because when vCenter used as a glance
|
||||
# backend launch may takes up to 1 minute.
|
||||
response = {}
|
||||
5.times.each do |retries|
|
||||
sleep 10 if retries > 0
|
||||
|
||||
response = run_shell_command(context, Array(controller['uid']), cmd)
|
||||
break if response[:data][:exit_code] == 0
|
||||
end
|
||||
|
||||
if response[:data][:exit_code] != 0
|
||||
msg = 'Disabling the upload of disk image because glance was not installed properly'
|
||||
if context.status[node['uid']] != 'error'
|
||||
raise_cirros_error(
|
||||
context,
|
||||
node,
|
||||
msg
|
||||
)
|
||||
else
|
||||
Astute.logger.error("#{context.task_id}: #{msg}")
|
||||
return
|
||||
end
|
||||
end
|
||||
|
||||
cmd = <<-UPLOAD_IMAGE
|
||||
. /root/openrc &&
|
||||
/usr/bin/glance image-list | grep -q #{os['img_name']} ||
|
||||
/usr/bin/glance image-create
|
||||
--name \'#{os['img_name']}\'
|
||||
--is-public #{os['public']}
|
||||
--container-format=\'#{os['container_format']}\'
|
||||
--disk-format=\'#{os['disk_format']}\'
|
||||
--min-ram=#{os['min_ram']} #{os['glance_properties']}
|
||||
--file \'#{os['img_path']}\'
|
||||
UPLOAD_IMAGE
|
||||
cmd.tr!("\n"," ")
|
||||
|
||||
response = run_shell_command(context, Array(controller['uid']), cmd)
|
||||
if response[:data][:exit_code] == 0
|
||||
Astute.logger.info "#{context.task_id}: Upload cirros " \
|
||||
"image \"#{os['img_name']}\" is done"
|
||||
else
|
||||
raise_cirros_error(context, node, "Upload cirros \"#{os['img_name']}\" image failed")
|
||||
end
|
||||
end # process
|
||||
|
||||
private
|
||||
|
||||
def raise_cirros_error(context, node, msg='')
|
||||
Astute.logger.error("#{context.task_id}: #{msg}")
|
||||
context.report_and_update_status('nodes' => [
|
||||
{'uid' => node['uid'],
|
||||
'status' => 'error',
|
||||
'error_type' => 'deploy',
|
||||
'role' => node['role']
|
||||
}
|
||||
]
|
||||
)
|
||||
raise CirrosError, msg
|
||||
end
|
||||
|
||||
end # class
|
||||
end
|
|
@ -1,177 +0,0 @@
|
|||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
module PreDelete
|
||||
|
||||
def self.check_ceph_osds(ctx, nodes)
|
||||
answer = {"status" => "ready"}
|
||||
ceph_nodes = nodes.select { |n| n["roles"].include? "ceph-osd" }
|
||||
ceph_osds = ceph_nodes.collect{ |n| n["slave_name"] }
|
||||
return answer if ceph_osds.empty?
|
||||
|
||||
cmd = "ceph -f json osd tree"
|
||||
result = {}
|
||||
shell = nil
|
||||
|
||||
ceph_nodes.each do |ceph_node|
|
||||
shell = MClient.new(ctx, "execute_shell_command", [ceph_node["id"]], timeout=60, retries=1)
|
||||
result = shell.execute(:cmd => cmd).first.results
|
||||
break if result[:data][:exit_code] == 0
|
||||
end
|
||||
|
||||
if result[:data][:exit_code] != 0
|
||||
Astute.logger.debug "Ceph has not been found or has not been configured properly" \
|
||||
" Safely removing nodes..."
|
||||
return answer
|
||||
end
|
||||
|
||||
osds = {}
|
||||
|
||||
tree = JSON.parse(result[:data][:stdout])
|
||||
|
||||
tree["nodes"].each do |osd|
|
||||
osds[osd["name"]] = osd["children"] if ceph_osds.include? osd["name"]
|
||||
end
|
||||
|
||||
# pg dump lists all pgs in the cluster and where they are located.
|
||||
# $14 is the 'up set' (the list of OSDs responsible for a particular
|
||||
# pg for an epoch) and $16 is the 'acting set' (list of OSDs who
|
||||
# are [or were at some point] responsible for a pg). These sets
|
||||
# will generally be the same.
|
||||
osd_list = osds.values.flatten.join("|")
|
||||
cmd = "ceph pg dump 2>/dev/null | " \
|
||||
"awk '//{print $14, $16}' | " \
|
||||
"egrep -o '\\<(#{osd_list})\\>' | " \
|
||||
"sort -un"
|
||||
|
||||
result = shell.execute(:cmd => cmd).first.results
|
||||
rs = result[:data][:stdout].split("\n")
|
||||
|
||||
# JSON.parse returns the children as integers, so the result from the
|
||||
# shell command needs to be converted for the set operations to work.
|
||||
rs.map! { |x| x.to_i }
|
||||
|
||||
error_nodes = []
|
||||
osds.each do |name, children|
|
||||
error_nodes << name if rs & children != []
|
||||
end
|
||||
|
||||
if not error_nodes.empty?
|
||||
msg = "Ceph data still exists on: #{error_nodes.join(', ')}. " \
|
||||
"You must manually remove the OSDs from the cluster " \
|
||||
"and allow Ceph to rebalance before deleting these nodes."
|
||||
answer = {"status" => "error", "error" => msg}
|
||||
end
|
||||
|
||||
answer
|
||||
end
|
||||
|
||||
def self.remove_ceph_mons(ctx, nodes)
|
||||
answer = {"status" => "ready"}
|
||||
ceph_mon_nodes = nodes.select { |n| n["roles"].include? "controller" }
|
||||
ceph_mons = ceph_mon_nodes.collect{ |n| n["slave_name"] }
|
||||
return answer if ceph_mon_nodes.empty?
|
||||
|
||||
#Get the list of mon nodes
|
||||
result = {}
|
||||
shell = nil
|
||||
|
||||
ceph_mon_nodes.each do |ceph_mon_node|
|
||||
shell = MClient.new(ctx, "execute_shell_command", [ceph_mon_node["id"]], timeout=120, retries=1)
|
||||
result = shell.execute(:cmd => "ceph -f json mon dump").first.results
|
||||
break if result[:data][:exit_code] == 0
|
||||
end
|
||||
|
||||
if result[:data][:exit_code] != 0
|
||||
Astute.logger.debug "Ceph mon has not been found or has not been configured properly" \
|
||||
" Safely removing nodes..."
|
||||
return answer
|
||||
end
|
||||
|
||||
mon_dump = JSON.parse(result[:data][:stdout])
|
||||
left_mons = mon_dump['mons'].select { | n | n if ! ceph_mons.include? n['name'] }
|
||||
left_mon_names = left_mons.collect { |n| n['name'] }
|
||||
left_mon_ips = left_mons.collect { |n| n['addr'].split(":")[0] }
|
||||
|
||||
#Remove nodes from ceph cluster
|
||||
Astute.logger.info("Removing ceph mons #{ceph_mons} from cluster")
|
||||
ceph_mon_nodes.each do |node|
|
||||
shell = MClient.new(ctx, "execute_shell_command", [node["id"]], timeout=120, retries=1)
|
||||
#remove node from ceph mon list
|
||||
shell.execute(:cmd => "ceph mon remove #{node["slave_name"]}").first.results
|
||||
end
|
||||
|
||||
#Fix the ceph.conf on the left mon nodes
|
||||
left_mon_names.each do |node|
|
||||
mon_initial_members_cmd = "sed -i \"s/mon_initial_members.*/mon_initial_members\ = #{left_mon_names.join(" ")}/g\" /etc/ceph/ceph.conf"
|
||||
mon_host_cmd = "sed -i \"s/mon_host.*/mon_host\ = #{left_mon_ips.join(" ")}/g\" /etc/ceph/ceph.conf"
|
||||
shell = MClient.new(ctx, "execute_shell_command", [node.split('-')[1]], timeout=120, retries=1)
|
||||
shell.execute(:cmd => mon_initial_members_cmd).first.results
|
||||
shell.execute(:cmd => mon_host_cmd).first.results
|
||||
end
|
||||
|
||||
Astute.logger.info("Ceph mons are left in cluster: #{left_mon_names}")
|
||||
|
||||
answer
|
||||
end
|
||||
|
||||
def self.check_for_offline_nodes(ctx, nodes)
|
||||
answer = {"status" => "ready"}
|
||||
# FIXME(vsharshov): We send for node/cluster deletion operation
|
||||
# as integer instead of String
|
||||
mco_nodes = nodes.map { |n| n['uid'].to_s }
|
||||
|
||||
|
||||
online_nodes = detect_available_nodes(ctx, mco_nodes)
|
||||
offline_nodes = mco_nodes - online_nodes
|
||||
|
||||
if offline_nodes.present?
|
||||
offline_nodes.map! { |e| {'uid' => e} }
|
||||
msg = "MCollective is not running on nodes: " \
|
||||
"#{offline_nodes.collect {|n| n['uid'] }.join(',')}. " \
|
||||
"MCollective must be running to properly delete a node."
|
||||
Astute.logger.warn msg
|
||||
answer = {'status' => 'error',
|
||||
'error' => msg,
|
||||
'error_nodes' => offline_nodes}
|
||||
end
|
||||
|
||||
answer
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def self.detect_available_nodes(ctx, uids)
|
||||
all_uids = uids.clone
|
||||
available_uids = []
|
||||
|
||||
# In case of big amount of nodes we should do several calls to be sure
|
||||
# about node status
|
||||
Astute.config[:mc_retries].times.each do
|
||||
systemtype = Astute::MClient.new(ctx, "systemtype", all_uids, check_result=false, 10)
|
||||
available_nodes = systemtype.get_type
|
||||
|
||||
available_uids += available_nodes.map { |node| node.results[:sender] }
|
||||
all_uids -= available_uids
|
||||
break if all_uids.empty?
|
||||
|
||||
sleep Astute.config[:mc_retry_interval]
|
||||
end
|
||||
|
||||
available_uids
|
||||
end
|
||||
|
||||
end
|
||||
end
|
|
@ -1,34 +0,0 @@
|
|||
# Copyright 2014 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
class ConnectFacts < PreDeployAction
|
||||
|
||||
def process(deployment_info, context)
|
||||
deployment_info.each{ |node| connect_facts(context, node) }
|
||||
Astute.logger.info "#{context.task_id}: Connect role facts for nodes"
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def connect_facts(context, node)
|
||||
run_shell_command(
|
||||
context,
|
||||
[node['uid']],
|
||||
"ln -s -f /etc/#{node['role']}.yaml /etc/astute.yaml"
|
||||
)
|
||||
end
|
||||
|
||||
end #class
|
||||
end
|
|
@ -1,25 +0,0 @@
|
|||
# Copyright 2014 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
class EnablePuppetDeploy < PreDeploymentAction
|
||||
|
||||
# Unlock puppet (can be lock if puppet was killed by user)
|
||||
def process(deployment_info, context)
|
||||
nodes_uids = only_uniq_nodes(deployment_info).map{ |n| n['uid'] }
|
||||
puppetd = MClient.new(context, "puppetd", nodes_uids)
|
||||
puppetd.enable
|
||||
end #process
|
||||
end #class
|
||||
end
|
|
@ -1,55 +0,0 @@
|
|||
# Copyright 2014 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
require 'open3'
|
||||
require 'fileutils'
|
||||
|
||||
module Astute
|
||||
class GenerateKeys < PreDeploymentAction
|
||||
|
||||
# Generate ssh keys to future uploading to all cluster nodes
|
||||
def process(deployment_info, context)
|
||||
overwrite = false
|
||||
deployment_id = deployment_info.first['deployment_id']
|
||||
raise "Deployment_id is missing" unless deployment_id
|
||||
|
||||
Astute.config.puppet_keys.each do |key_name|
|
||||
dir_path = File.join(Astute.config.keys_src_dir, deployment_id.to_s, key_name)
|
||||
key_path = File.join(dir_path, key_name + '.key')
|
||||
|
||||
FileUtils.mkdir_p dir_path
|
||||
raise DeploymentEngineError, "Could not create directory #{dir_path}" unless File.directory?(dir_path)
|
||||
|
||||
next if File.exist?(key_path) && !overwrite
|
||||
|
||||
# Generate key(<name>.key) and save it to <KEY_DIR>/<name>/<name>.key
|
||||
File.delete key_path if File.exist? key_path
|
||||
|
||||
cmd = "openssl rand -base64 741 > #{key_path} 2>&1"
|
||||
status, stdout, _stderr = run_system_command cmd
|
||||
|
||||
error_msg = "Could not generate key! Command: #{cmd}, output: #{stdout}, exit code: #{status}"
|
||||
raise DeploymentEngineError, error_msg if status != 0
|
||||
end
|
||||
end #process
|
||||
|
||||
private
|
||||
|
||||
def run_system_command(cmd)
|
||||
stdout, stderr, status = Open3.capture3 cmd
|
||||
return status.exitstatus, stdout, stderr
|
||||
end
|
||||
|
||||
end #class
|
||||
end
|
|
@ -1,55 +0,0 @@
|
|||
# Copyright 2014 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
require 'open3'
|
||||
require 'fileutils'
|
||||
|
||||
module Astute
|
||||
class GenerateSshKeys < PreDeploymentAction
|
||||
|
||||
# Generate ssh keys to future uploading to all cluster nodes
|
||||
def process(deployment_info, context)
|
||||
overwrite = false
|
||||
deployment_id = deployment_info.first['deployment_id']
|
||||
raise "Deployment_id is missing" unless deployment_id
|
||||
|
||||
Astute.config.puppet_ssh_keys.each do |key_name|
|
||||
dir_path = File.join(Astute.config.keys_src_dir, deployment_id.to_s, key_name)
|
||||
key_path = File.join(dir_path, key_name)
|
||||
|
||||
FileUtils.mkdir_p dir_path
|
||||
raise DeploymentEngineError, "Could not create directory #{dir_path}" unless File.directory?(dir_path)
|
||||
|
||||
next if File.exist?(key_path) && !overwrite
|
||||
|
||||
# Generate 2 keys(<name> and <name>.pub) and save it to <KEY_DIR>/<name>/
|
||||
File.delete key_path if File.exist? key_path
|
||||
|
||||
cmd = "ssh-keygen -b 2048 -t rsa -N '' -f #{key_path} 2>&1"
|
||||
status, stdout, _stderr = run_system_command cmd
|
||||
|
||||
error_msg = "Could not generate ssh key! Command: #{cmd}, output: #{stdout}, exit code: #{status}"
|
||||
raise DeploymentEngineError, error_msg if status != 0
|
||||
end
|
||||
end #process
|
||||
|
||||
private
|
||||
|
||||
def run_system_command(cmd)
|
||||
stdout, stderr, status = Open3.capture3 cmd
|
||||
return status.exitstatus, stdout, stderr
|
||||
end
|
||||
|
||||
end #class
|
||||
end
|
|
@ -1,34 +0,0 @@
|
|||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
class InitialConnectFacts < PreDeploymentAction
|
||||
|
||||
def process(deployment_info, context)
|
||||
only_uniq_nodes(deployment_info).each{ |node| connect_facts(context, node) }
|
||||
Astute.logger.info "#{context.task_id}: Initial connect role facts for nodes"
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def connect_facts(context, node)
|
||||
run_shell_command(
|
||||
context,
|
||||
[node['uid']],
|
||||
"ln -s -f /etc/#{node['role']}.yaml /etc/astute.yaml"
|
||||
)
|
||||
end
|
||||
|
||||
end #class
|
||||
end
|
|
@ -1,74 +0,0 @@
|
|||
# Copyright 2014 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
require 'uri'
|
||||
SYNC_RETRIES = 10
|
||||
|
||||
module Astute
|
||||
class SyncPuppetStuff < PreDeploymentAction
|
||||
|
||||
# Sync puppet manifests and modules to every node
|
||||
def process(deployment_info, context)
|
||||
master_ip = deployment_info.first['master_ip']
|
||||
modules_source = deployment_info.first['puppet']['modules']
|
||||
manifests_source = deployment_info.first['puppet']['manifests']
|
||||
# Paths to Puppet modules and manifests at the master node set by Nailgun
|
||||
# Check fuel source code /deployment/puppet/nailgun/manifests/puppetsync.pp
|
||||
schemas = [modules_source, manifests_source].map do |url|
|
||||
begin
|
||||
URI.parse(url).scheme
|
||||
rescue URI::InvalidURIError => e
|
||||
raise DeploymentEngineError, e.message
|
||||
end
|
||||
end
|
||||
|
||||
if schemas.select{ |x| x != schemas.first }.present?
|
||||
raise DeploymentEngineError, "Scheme for puppet modules '#{schemas.first}' and" \
|
||||
" puppet manifests '#{schemas.last}' not equivalent!"
|
||||
end
|
||||
|
||||
nodes_uids = only_uniq_nodes(deployment_info).map{ |n| n['uid'] }
|
||||
|
||||
perform_with_limit(nodes_uids) do |part|
|
||||
sync_puppet_stuff(context, part, schemas, modules_source, manifests_source)
|
||||
end
|
||||
|
||||
end # process
|
||||
|
||||
private
|
||||
|
||||
def sync_puppet_stuff(context, node_uids, schemas, modules_source, manifests_source)
|
||||
sync_mclient = MClient.new(context, "puppetsync", node_uids)
|
||||
|
||||
case schemas.first
|
||||
when 'rsync'
|
||||
begin
|
||||
sync_mclient.rsync(:modules_source => modules_source,
|
||||
:manifests_source => manifests_source
|
||||
)
|
||||
rescue MClientError => e
|
||||
sync_retries ||= 0
|
||||
sync_retries += 1
|
||||
if sync_retries < SYNC_RETRIES
|
||||
Astute.logger.warn("Rsync problem. Try to repeat: #{sync_retries} attempt")
|
||||
retry
|
||||
end
|
||||
raise e
|
||||
end
|
||||
else
|
||||
raise DeploymentEngineError, "Unknown scheme '#{schemas.first}' in #{modules_source}"
|
||||
end
|
||||
end #process
|
||||
end #class
|
||||
end
|
|
@ -1,58 +0,0 @@
|
|||
# Copyright 2014 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
SYNC_RETRIES = 10
|
||||
|
||||
module Astute
|
||||
class SyncTasks < PreDeploymentAction
|
||||
|
||||
# Sync puppet manifests and modules to every node
|
||||
def process(deployment_info, context)
|
||||
return unless deployment_info.first['tasks_source']
|
||||
|
||||
# URI to Tasklib tasks at the master node set by Nailgun
|
||||
master_ip = deployment_info.first['master_ip']
|
||||
tasks_source = deployment_info.first['tasks_source'] || "rsync://#{master_ip}:/puppet/tasks/"
|
||||
source = tasks_source.chomp('/').concat('/')
|
||||
|
||||
nodes_uids = only_uniq_nodes(deployment_info).map{ |n| n['uid'] }
|
||||
|
||||
perform_with_limit(nodes_uids) do |part|
|
||||
rsync_tasks(context, source, part)
|
||||
end
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def rsync_tasks(context, source, nodes_uids)
|
||||
path = '/etc/puppet/tasks/'
|
||||
rsync_options = '-c -r --delete'
|
||||
rsync_cmd = "mkdir -p #{path} && rsync #{rsync_options} #{source} #{path}"
|
||||
|
||||
sync_retries = 0
|
||||
while sync_retries < SYNC_RETRIES
|
||||
sync_retries += 1
|
||||
response = run_shell_command(
|
||||
context,
|
||||
nodes_uids,
|
||||
rsync_cmd,
|
||||
300
|
||||
)
|
||||
break if response[:data][:exit_code] == 0
|
||||
Astute.logger.warn("Rsync problem. Try to repeat: #{sync_retries} attempt")
|
||||
end
|
||||
end #rsync_tasks
|
||||
|
||||
end #class
|
||||
end
|
|
@ -1,45 +0,0 @@
|
|||
# Copyright 2014 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
class SyncTime < PreDeploymentAction
|
||||
|
||||
# Sync time
|
||||
def process(deployment_info, context)
|
||||
nodes_uids = only_uniq_nodes(deployment_info).map{ |n| n['uid'] }
|
||||
cmd = "ntpdate -u $(egrep '^server' /etc/ntp.conf | sed '/^#/d' | awk '{print $2}')"
|
||||
succeeded = false
|
||||
|
||||
Astute.config.mc_retries.times.each do
|
||||
succeeded = run_shell_command_remotely(context, nodes_uids, cmd)
|
||||
return if succeeded
|
||||
sleep Astute.config.mc_retry_interval
|
||||
end
|
||||
|
||||
if !succeeded
|
||||
Astute.logger.warn "Run command: '#{cmd}' in nodes: #{nodes_uids} fail. " \
|
||||
"Check debug output for more information. You can try "\
|
||||
"to fix it problem manually."
|
||||
end
|
||||
end #process
|
||||
|
||||
private
|
||||
|
||||
def run_shell_command_remotely(context, nodes_uids, cmd)
|
||||
response = run_shell_command(context, nodes_uids, cmd)
|
||||
response.fetch(:data, {})[:exit_code] == 0
|
||||
end
|
||||
|
||||
end #class
|
||||
end
|
|
@ -1,103 +0,0 @@
|
|||
# Copyright 2014 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
class UpdateRepoSources < PreDeploymentAction
|
||||
|
||||
# Update packages source list
|
||||
def process(deployment_info, context)
|
||||
return unless deployment_info.first['repo_setup']['repos']
|
||||
content = generate_repo_source(deployment_info)
|
||||
deployment_info = only_uniq_nodes(deployment_info)
|
||||
|
||||
perform_with_limit(deployment_info) do |part|
|
||||
upload_repo_source(context, part, content)
|
||||
regenerate_metadata(context, part)
|
||||
end
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def generate_repo_source(deployment_info)
|
||||
ubuntu_source = -> (repo) { "deb #{repo['uri']} #{repo['suite']} #{repo['section']}" }
|
||||
centos_source = -> (repo) do
|
||||
["[#{repo['name'].downcase}]", "name=#{repo['name']}", "baseurl=#{repo['uri']}", "gpgcheck=0"].join("\n")
|
||||
end
|
||||
|
||||
formatter = case target_os(deployment_info)
|
||||
when 'centos' then centos_source
|
||||
when 'ubuntu' then ubuntu_source
|
||||
end
|
||||
|
||||
content = []
|
||||
deployment_info.first['repo_setup']['repos'].each do |repo|
|
||||
content << formatter.call(repo)
|
||||
end
|
||||
content.join("\n")
|
||||
end
|
||||
|
||||
def upload_repo_source(context, deployment_info, content)
|
||||
upload_mclient = MClient.new(context, "uploadfile", deployment_info.map{ |n| n['uid'] }.uniq)
|
||||
destination_path = case target_os(deployment_info)
|
||||
when 'centos' then '/etc/yum.repos.d/nailgun.repo'
|
||||
when 'ubuntu' then '/etc/apt/sources.list'
|
||||
end
|
||||
upload_mclient.upload(:path => destination_path,
|
||||
:content => content,
|
||||
:user_owner => 'root',
|
||||
:group_owner => 'root',
|
||||
:permissions => '0644',
|
||||
:dir_permissions => '0755',
|
||||
:overwrite => true,
|
||||
:parents => true
|
||||
)
|
||||
end
|
||||
|
||||
def regenerate_metadata(context, deployment_info)
|
||||
cmd = case target_os(deployment_info)
|
||||
when 'centos' then "yum clean all"
|
||||
when 'ubuntu' then "apt-get clean; apt-get update"
|
||||
end
|
||||
|
||||
succeeded = false
|
||||
nodes_uids = deployment_info.map{ |n| n['uid'] }.uniq
|
||||
Astute.config.mc_retries.times.each do
|
||||
succeeded = run_shell_command_remotely(context, nodes_uids, cmd)
|
||||
return if succeeded
|
||||
sleep Astute.config.mc_retry_interval
|
||||
end
|
||||
|
||||
if !succeeded
|
||||
raise DeploymentEngineError, "Run command: '#{cmd}' in nodes: #{nodes_uids} fail." \
|
||||
" Check debug output for more information"
|
||||
end
|
||||
end
|
||||
|
||||
def target_os(deployment_info)
|
||||
os = deployment_info.first['cobbler']['profile']
|
||||
case os
|
||||
when 'centos-x86_64' then 'centos'
|
||||
when 'ubuntu_1404_x86_64' then 'ubuntu'
|
||||
else
|
||||
raise DeploymentEngineError, "Unknown system #{os}"
|
||||
end
|
||||
end
|
||||
|
||||
def run_shell_command_remotely(context, nodes_uids, cmd)
|
||||
response = run_shell_command(context, nodes_uids, cmd)
|
||||
response.fetch(:data, {})[:exit_code] == 0
|
||||
end
|
||||
|
||||
end #class
|
||||
end
|
|
@ -1,60 +0,0 @@
|
|||
# Copyright 2014 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
require 'psych'
|
||||
|
||||
module Astute
|
||||
class UploadFacts < PreDeploymentAction
|
||||
|
||||
def process(deployment_info, context)
|
||||
deployment_info.each{ |node| upload_facts(context, node) }
|
||||
Astute.logger.info "#{context.task_id}: Required attrs/metadata passed via facts extension"
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
# This is simple version of 'YAML::dump' with force quoting of strings started with prefixed numeral values
|
||||
def safe_yaml_dump(obj)
|
||||
visitor = Psych::Visitors::YAMLTree.new({})
|
||||
visitor << obj
|
||||
visitor.tree.grep(Psych::Nodes::Scalar).each do |node|
|
||||
node.style = Psych::Nodes::Scalar::DOUBLE_QUOTED if
|
||||
node.value =~ /^0[xbod0]+/i && node.plain && node.quoted
|
||||
end
|
||||
visitor.tree.yaml
|
||||
end
|
||||
|
||||
def upload_facts(context, node)
|
||||
|
||||
# TODO: Should be changed to the default 'to_yaml' method only after upgrading
|
||||
# to Ruby 2.1 everywhere on client nodes which used this YAML.
|
||||
yaml_data = safe_yaml_dump(node)
|
||||
|
||||
Astute.logger.info "#{context.task_id}: storing metadata for node uid=#{node['uid']} "\
|
||||
"role=#{node['role']}"
|
||||
Astute.logger.debug "#{context.task_id}: stores metadata: #{yaml_data}"
|
||||
|
||||
# This is synchronious RPC call, so we are sure that data were sent and processed remotely
|
||||
upload_mclient = Astute::MClient.new(context, "uploadfile", [node['uid']])
|
||||
upload_mclient.upload(
|
||||
:path => "/etc/#{node['role']}.yaml",
|
||||
:content => yaml_data,
|
||||
:overwrite => true,
|
||||
:parents => true,
|
||||
:permissions => '0600'
|
||||
)
|
||||
end
|
||||
|
||||
end #class
|
||||
end
|
|
@ -1,59 +0,0 @@
|
|||
# Copyright 2014 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
class UploadKeys < PreDeploymentAction
|
||||
|
||||
# Upload ssh keys from master node to all cluster nodes
|
||||
def process(deployment_info, context)
|
||||
deployment_id = deployment_info.first['deployment_id'].to_s
|
||||
nodes_ids = only_uniq_nodes(deployment_info).map{ |n| n['uid'] }
|
||||
perform_with_limit(nodes_ids) do |ids|
|
||||
upload_keys(context, ids, deployment_id)
|
||||
end
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def upload_keys(context, node_uids, deployment_id)
|
||||
Astute.config.puppet_keys.each do |key_name|
|
||||
upload_mclient = MClient.new(context, "uploadfile", node_uids)
|
||||
key = key_name + '.key'
|
||||
source_path = File.join(
|
||||
Astute.config.keys_src_dir,
|
||||
deployment_id,
|
||||
key_name,
|
||||
key
|
||||
)
|
||||
destination_path = File.join(
|
||||
Astute.config.keys_dst_dir,
|
||||
key_name,
|
||||
key
|
||||
)
|
||||
content = File.read(source_path)
|
||||
upload_mclient.upload(
|
||||
:path => destination_path,
|
||||
:content => content,
|
||||
:user_owner => 'root',
|
||||
:group_owner => 'root',
|
||||
:permissions => '0600',
|
||||
:dir_permissions => '0700',
|
||||
:overwrite => true,
|
||||
:parents => true
|
||||
)
|
||||
end
|
||||
end #upload_keys
|
||||
|
||||
end #class
|
||||
end
|
|
@ -1,59 +0,0 @@
|
|||
# Copyright 2014 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
class UploadSshKeys < PreDeploymentAction
|
||||
|
||||
# Upload ssh keys from master node to all cluster nodes
|
||||
def process(deployment_info, context)
|
||||
deployment_id = deployment_info.first['deployment_id'].to_s
|
||||
nodes_ids = only_uniq_nodes(deployment_info).map{ |n| n['uid'] }
|
||||
perform_with_limit(nodes_ids) do |ids|
|
||||
upload_keys(context, ids, deployment_id)
|
||||
end
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def upload_keys(context, node_uids, deployment_id)
|
||||
Astute.config.puppet_ssh_keys.each do |key_name|
|
||||
upload_mclient = MClient.new(context, "uploadfile", node_uids)
|
||||
[key_name, key_name + ".pub"].each do |ssh_key|
|
||||
source_path = File.join(
|
||||
Astute.config.keys_src_dir,
|
||||
deployment_id,
|
||||
key_name,
|
||||
ssh_key
|
||||
)
|
||||
destination_path = File.join(
|
||||
Astute.config.keys_dst_dir,
|
||||
key_name,
|
||||
ssh_key
|
||||
)
|
||||
content = File.read(source_path)
|
||||
upload_mclient.upload(:path => destination_path,
|
||||
:content => content,
|
||||
:user_owner => 'root',
|
||||
:group_owner => 'root',
|
||||
:permissions => '0600',
|
||||
:dir_permissions => '0700',
|
||||
:overwrite => true,
|
||||
:parents => true
|
||||
)
|
||||
end
|
||||
end
|
||||
end #upload_keys
|
||||
|
||||
end #class
|
||||
end
|
|
@ -1,77 +0,0 @@
|
|||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
class PrePatching < PreNodeAction
|
||||
|
||||
def process(deployment_info, context)
|
||||
return unless deployment_info.first['openstack_version_prev']
|
||||
|
||||
# We should stop services with SIGTERM or even SIGKILL.
|
||||
# StopOSTServices do this and should be run before.
|
||||
|
||||
remove_cmd = getremovepackage_cmd(deployment_info)
|
||||
|
||||
nodes = deployment_info.map { |n| n['uid'] }
|
||||
|
||||
Astute.logger.info "Starting removal of error-prone packages"
|
||||
Astute.logger.info "Executing command #{remove_cmd}"
|
||||
Astute.logger.info "On nodes\n#{nodes.pretty_inspect}"
|
||||
|
||||
response = run_shell_command(context, nodes, remove_cmd, 600)
|
||||
|
||||
if response[:data][:exit_code] != 0
|
||||
Astute.logger.error "#{context.task_id}: Fail to remove packages, "\
|
||||
"check the debugging output for details"
|
||||
end
|
||||
|
||||
Astute.logger.info "#{context.task_id}: Finished pre-patching hook"
|
||||
end #process
|
||||
|
||||
def getremovepackage_cmd(deployment_info)
|
||||
os = deployment_info.first['cobbler']['profile']
|
||||
case os
|
||||
when /centos/i then "yum -y remove #{centos_packages}"
|
||||
when /ubuntu/i then "aptitude -y remove #{ubuntu_packages}"
|
||||
else
|
||||
raise DeploymentEngineError, "Unknown system #{os}"
|
||||
end
|
||||
end
|
||||
|
||||
def centos_packages
|
||||
packages = <<-Packages
|
||||
python-oslo-messaging python-oslo-config openstack-heat-common
|
||||
python-nova python-routes python-routes1.12 python-neutron
|
||||
python-django-horizon murano-api sahara sahara-dashboard
|
||||
python-ceilometer openstack-swift openstack-utils
|
||||
python-glance python-glanceclient python-cinder
|
||||
python-sqlalchemy python-testtools
|
||||
Packages
|
||||
packages.tr!("\n"," ")
|
||||
end
|
||||
|
||||
def ubuntu_packages
|
||||
packages = <<-Packages
|
||||
python-oslo.messaging python-oslo.config python-heat python-nova
|
||||
python-routes python-routes1.13 python-neutron python-django-horizon
|
||||
murano-common murano-api sahara sahara-dashboard python-ceilometer
|
||||
python-swift python-cinder python-keystoneclient python-neutronclient
|
||||
python-novaclient python-swiftclient python-troveclient
|
||||
python-sqlalchemy python-testtools
|
||||
Packages
|
||||
packages.tr!("\n"," ")
|
||||
end
|
||||
|
||||
end #class
|
||||
end
|
|
@ -1,50 +0,0 @@
|
|||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
class PrePatchingHa < PreNodeAction
|
||||
|
||||
def process(deployment_info, context)
|
||||
return if deployment_info.first['openstack_version_prev'].nil? ||
|
||||
deployment_info.first['deployment_mode'] !~ /ha/i
|
||||
|
||||
# Run only once for node. If one of role is controller or primary-controller
|
||||
# generate new deployment_info block.
|
||||
# Important for 'mongo' role which run early then 'controller'
|
||||
current_uids = deployment_info.map{ |n| n['uid'] }
|
||||
controllers = deployment_info.first['nodes'].select{ |n| current_uids.include?(n['uid']) && n['role'] =~ /controller/i }
|
||||
c_deployment_info = deployment_info.select { |d_i| controllers.map{ |n| n['uid'] }.include? d_i['uid'] }
|
||||
|
||||
return if c_deployment_info.empty?
|
||||
c_deployment_info.each do |c_d_i|
|
||||
c_d_i['role'] = controllers.find{ |c| c['uid'] == c_d_i['uid'] }['role']
|
||||
end
|
||||
controller_nodes = c_deployment_info.map{ |n| n['uid'] }
|
||||
|
||||
Astute.logger.info "Starting migration of pacemaker services from " \
|
||||
"nodes\n#{controller_nodes.pretty_inspect}"
|
||||
|
||||
Astute::Pacemaker.commands(action='stop', c_deployment_info).each do |pcmk_ban_cmd|
|
||||
response = run_shell_command(context, controller_nodes, pcmk_ban_cmd)
|
||||
|
||||
if response[:data][:exit_code] != 0
|
||||
Astute.logger.warn "#{context.task_id}: Failed to ban service, "\
|
||||
"check the debugging output for details"
|
||||
end
|
||||
end
|
||||
|
||||
Astute.logger.info "#{context.task_id}: Finished pre-patching-ha hook"
|
||||
end #process
|
||||
end #class
|
||||
end
|
|
@ -1,65 +0,0 @@
|
|||
# Copyright 2014 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
class StopOSTServices < PreNodeAction
|
||||
|
||||
def process(deployment_info, context)
|
||||
old_env = deployment_info.first['openstack_version_prev']
|
||||
return unless old_env
|
||||
|
||||
Astute.logger.info "Stop all Openstack services hook start"
|
||||
|
||||
node_uids = deployment_info.collect { |n| n['uid'] }
|
||||
file_content = get_file
|
||||
target_file = '/tmp/stop_services.rb'
|
||||
|
||||
upload_script(context, node_uids, target_file, file_content)
|
||||
|
||||
Astute.logger.info "Running file: #{target_file} on node uids: #{node_uids.join ', '}"
|
||||
|
||||
response = run_shell_command(context, node_uids, "/usr/bin/ruby #{target_file} |tee /tmp/stop_services.log")
|
||||
|
||||
if response[:data][:exit_code] != 0
|
||||
Astute.logger.warn "#{context.task_id}: Script returned error code #{response[:data][:exit_code]}"
|
||||
end
|
||||
|
||||
Astute.logger.info "#{context.task_id}: Finished stop services pre-patching hook"
|
||||
end #process
|
||||
|
||||
private
|
||||
|
||||
def get_file
|
||||
File.read File.join(File.dirname(__FILE__), 'stop_services.script')
|
||||
end
|
||||
|
||||
def upload_script(context, node_uids, target_file, file_content)
|
||||
target_file = '/tmp/stop_services.rb'
|
||||
Astute.logger.info "Uploading file: #{target_file} to nodes uids: #{node_uids.join ', '}"
|
||||
|
||||
MClient.new(context, "uploadfile", node_uids).upload(
|
||||
:path => target_file,
|
||||
:content => file_content,
|
||||
:user_owner => 'root',
|
||||
:group_owner => 'root',
|
||||
:permissions => '0700',
|
||||
:overwrite => true,
|
||||
:parents => true
|
||||
)
|
||||
rescue MClientTimeout, MClientError => e
|
||||
Astute.logger.error("#{context.task_id}: mcollective error: #{e.message}")
|
||||
end
|
||||
|
||||
end #class
|
||||
end #module
|
|
@ -1,213 +0,0 @@
|
|||
#!/usr/bin/env ruby
|
||||
require 'rubygems'
|
||||
require 'facter'
|
||||
|
||||
# pre-deploy hook library
|
||||
module PreDeploy
|
||||
@dry_run = false
|
||||
@process_tree = nil
|
||||
@osfamily = nil
|
||||
@stop_services_regexp = %r{nova|cinder|glance|keystone|neutron|sahara|murano|ceilometer|heat|swift|apache2|httpd}
|
||||
|
||||
# get regexp that selects services and processes to stop
|
||||
# @return [Regexp]
|
||||
def self.stop_services_regexp
|
||||
@stop_services_regexp
|
||||
end
|
||||
|
||||
# set regexp that selects services and processes to stop
|
||||
# @param value [Regexp]
|
||||
def self.stop_services_regexp=(value)
|
||||
@stop_services_regexp = value
|
||||
end
|
||||
|
||||
# get osfamily from facter
|
||||
# @return [String]
|
||||
def self.osfamily
|
||||
return @osfamily if @osfamily
|
||||
@osfamily = Facter.value 'osfamily'
|
||||
end
|
||||
|
||||
# get dry run without doing anything switch
|
||||
# @return [TrueClass,FalseClass]
|
||||
def self.dry_run
|
||||
@dry_run
|
||||
end
|
||||
|
||||
# set dry run without doing anything switch
|
||||
# @param value [TrueClass,FalseClass]
|
||||
def self.dry_run=(value)
|
||||
@dry_run = value
|
||||
end
|
||||
|
||||
# get ps from shell command
|
||||
# @return [String]
|
||||
def self.ps
|
||||
`ps haxo pid,ppid,cmd`
|
||||
end
|
||||
|
||||
# get service statu from shell command
|
||||
# @return String
|
||||
def self.services
|
||||
`service --status-all 2>&1`
|
||||
end
|
||||
|
||||
# same as process_tree but reset mnemoization
|
||||
# @return [Hash<Integer => Hash<Symbol => String,Integer>>]
|
||||
def self.process_tree_with_renew
|
||||
@process_tree = nil
|
||||
self.process_tree
|
||||
end
|
||||
|
||||
# build process tree from process list
|
||||
# @return [Hash<Integer => Hash<Symbol => String,Integer>>]
|
||||
def self.process_tree
|
||||
return @process_tree if @process_tree
|
||||
@process_tree = {}
|
||||
self.ps.split("\n").each do |p|
|
||||
f = p.split
|
||||
pid = f.shift.to_i
|
||||
ppid = f.shift.to_i
|
||||
cmd = f.join ' '
|
||||
|
||||
# create entry for this pid if not present
|
||||
@process_tree[pid] = {
|
||||
:children => []
|
||||
} unless @process_tree.key? pid
|
||||
|
||||
# fill this entry
|
||||
@process_tree[pid][:ppid] = ppid
|
||||
@process_tree[pid][:pid] = pid
|
||||
@process_tree[pid][:cmd] = cmd
|
||||
|
||||
# create entry for parent process if not present
|
||||
@process_tree[ppid] = {
|
||||
:children => []
|
||||
} unless @process_tree.key? ppid
|
||||
|
||||
# fill parent's children
|
||||
@process_tree[ppid][:children] << pid
|
||||
end
|
||||
@process_tree
|
||||
end
|
||||
|
||||
# kill selected pid or array of them
|
||||
# @param pids [Integer,String] Pids to kill
|
||||
# @param signal [Integer,String] Which signal?
|
||||
# @param recursive [TrueClass,FalseClass] Kill children too?
|
||||
# @return [TrueClass,FalseClass] Was the signal sent? Process may still be present even on success.
|
||||
def self.kill_pids(pids, signal = 9, recursive = true)
|
||||
pids = Array pids
|
||||
|
||||
pids_to_kill = pids.inject([]) do |all_pids, pid|
|
||||
pid = pid.to_i
|
||||
if recursive
|
||||
all_pids + self.get_children_pids(pid)
|
||||
else
|
||||
all_pids << pid
|
||||
end
|
||||
end
|
||||
|
||||
pids_to_kill.uniq!
|
||||
pids_to_kill.sort!
|
||||
|
||||
return false unless pids_to_kill.any?
|
||||
puts "Kill these pids: #{pids_to_kill.join ', '} with signal #{signal}"
|
||||
self.run "kill -#{signal} #{pids_to_kill.join ' '}"
|
||||
end
|
||||
|
||||
# recursion to find all children pids
|
||||
# @return [Array<Integer>]
|
||||
def self.get_children_pids(pid)
|
||||
pid = pid.to_i
|
||||
unless self.process_tree.key? pid
|
||||
puts "No such pid: #{pid}"
|
||||
return []
|
||||
end
|
||||
self.process_tree[pid][:children].inject([pid]) do |all_children_pids, child_pid|
|
||||
all_children_pids + self.get_children_pids(child_pid)
|
||||
end
|
||||
end
|
||||
|
||||
# same as services_to_stop but reset mnemoization
|
||||
# @return Array[String]
|
||||
def self.services_to_stop_with_renew
|
||||
@services_to_stop = nil
|
||||
self.services_to_stop
|
||||
end
|
||||
|
||||
# find running services that should be stopped
|
||||
# uses service status and regex to filter
|
||||
# @return [Array<String>]
|
||||
def self.services_to_stop
|
||||
return @services_to_stop if @services_to_stop
|
||||
@services_to_stop = self.services.split("\n").inject([]) do |services_to_stop, service|
|
||||
fields = service.chomp.split
|
||||
running = if fields[4] == 'running...'
|
||||
fields[0]
|
||||
elsif fields[1] == '+'
|
||||
fields[3]
|
||||
else
|
||||
nil
|
||||
end
|
||||
|
||||
if running =~ @stop_services_regexp
|
||||
# replace wrong service name
|
||||
running = 'httpd' if running == 'httpd.event' and self.osfamily == 'RedHat'
|
||||
running = 'openstack-keystone' if running == 'keystone' and self.osfamily == 'RedHat'
|
||||
services_to_stop << running
|
||||
else
|
||||
services_to_stop
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
# stop services that match stop_services_regex
|
||||
def self.stop_services
|
||||
self.services_to_stop.each do |service|
|
||||
puts "Try to stop service: #{service}"
|
||||
self.run "service #{service} stop"
|
||||
end
|
||||
end
|
||||
|
||||
# filter pids which cmd match regexp
|
||||
# @param regexp <Regexp> Search pids by this regexp
|
||||
# @return [Hash<Integer => Hash<Symbol => String,Integer>>]
|
||||
def self.pids_by_regexp(regexp)
|
||||
matched = {}
|
||||
self.process_tree.each do |pid,process|
|
||||
matched[pid] = process if process[:cmd] =~ regexp
|
||||
end
|
||||
matched
|
||||
end
|
||||
|
||||
# kill pids that match stop_services_regexp
|
||||
# @return <TrueClass,FalseClass>
|
||||
def self.kill_pids_by_stop_regexp
|
||||
pids = self.pids_by_regexp(@stop_services_regexp).keys
|
||||
self.kill_pids pids
|
||||
end
|
||||
|
||||
# here be other fixes
|
||||
# TODO: not needed anymore?
|
||||
def self.misc_fixes
|
||||
if self.osfamily == 'Debian'
|
||||
puts 'Enabling WSGI module'
|
||||
self.run 'a2enmod wsgi'
|
||||
end
|
||||
end
|
||||
|
||||
# run the shell command with dry_run support
|
||||
# @param cmd [String] Command to run
|
||||
def self.run(cmd)
|
||||
command = "#{self.dry_run ? 'echo' : ''} #{cmd} 2>&1"
|
||||
system command
|
||||
end
|
||||
end # class
|
||||
|
||||
if __FILE__ == $0
|
||||
# PreDeploy.dry_run = true
|
||||
PreDeploy.misc_fixes
|
||||
PreDeploy.stop_services
|
||||
PreDeploy.kill_pids_by_stop_regexp
|
||||
end
|
|
@ -1,511 +0,0 @@
|
|||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
|
||||
class Provisioner
|
||||
def initialize(log_parsing=false)
|
||||
@log_parsing = log_parsing
|
||||
end
|
||||
|
||||
def node_type(reporter, task_id, nodes_uids, timeout=nil)
|
||||
context = Context.new(task_id, reporter)
|
||||
systemtype = MClient.new(
|
||||
context,
|
||||
"systemtype",
|
||||
nodes_uids,
|
||||
_check_result=false,
|
||||
timeout
|
||||
)
|
||||
systems = systemtype.get_type
|
||||
systems.map do |n|
|
||||
{
|
||||
'uid' => n.results[:sender],
|
||||
'node_type' => n.results[:data][:node_type].to_s.chomp
|
||||
}
|
||||
end
|
||||
end
|
||||
|
||||
def provision(reporter, task_id, provisioning_info)
|
||||
engine_attrs = provisioning_info['engine']
|
||||
nodes = provisioning_info['nodes']
|
||||
|
||||
raise "Nodes to provision are not provided!" if nodes.empty?
|
||||
|
||||
fault_tolerance = provisioning_info.fetch('fault_tolerance', [])
|
||||
|
||||
cobbler = CobblerManager.new(engine_attrs, reporter)
|
||||
result_msg = {'nodes' => []}
|
||||
begin
|
||||
prepare_nodes(reporter, task_id, engine_attrs, nodes, cobbler)
|
||||
failed_uids, timeouted_uids = provision_and_watch_progress(reporter,
|
||||
task_id,
|
||||
Array.new(nodes),
|
||||
engine_attrs,
|
||||
fault_tolerance)
|
||||
rescue => e
|
||||
Astute.logger.error("Error occured while provisioning:\n#{e.pretty_inspect}")
|
||||
reporter.report({
|
||||
'status' => 'error',
|
||||
'error' => e.message,
|
||||
'progress' => 100})
|
||||
unlock_nodes_discovery(reporter, task_id, nodes.map { |n| n['uid'] }, nodes)
|
||||
raise e
|
||||
end
|
||||
|
||||
handle_failed_nodes(failed_uids, result_msg)
|
||||
if failed_uids.count > 0
|
||||
unlock_nodes_discovery(reporter, task_id, failed_uids, nodes)
|
||||
end
|
||||
handle_timeouted_nodes(timeouted_uids, result_msg)
|
||||
|
||||
node_uids = nodes.map { |n| n['uid'] }
|
||||
|
||||
(node_uids - timeouted_uids - failed_uids).each do |uid|
|
||||
result_msg['nodes'] << {'uid' => uid, 'progress' => 100, 'status' => 'provisioned'}
|
||||
end
|
||||
|
||||
if should_fail(failed_uids + timeouted_uids, fault_tolerance)
|
||||
result_msg['status'] = 'error'
|
||||
result_msg['error'] = 'Too many nodes failed to provision'
|
||||
result_msg['progress'] = 100
|
||||
end
|
||||
|
||||
# If there was no errors, then set status to ready
|
||||
result_msg.reverse_merge!({'status' => 'ready', 'progress' => 100})
|
||||
Astute.logger.info "Message: #{result_msg}"
|
||||
|
||||
reporter.report(result_msg)
|
||||
|
||||
result_msg
|
||||
rescue => e
|
||||
Rsyslogd.send_sighup(
|
||||
Context.new(task_id, reporter),
|
||||
engine_attrs["master_ip"]
|
||||
)
|
||||
raise e
|
||||
end
|
||||
|
||||
def provision_and_watch_progress(reporter,
|
||||
task_id,
|
||||
nodes_to_provision,
|
||||
engine_attrs,
|
||||
fault_tolerance)
|
||||
raise "Nodes to provision are not provided!" if nodes_to_provision.empty?
|
||||
|
||||
provision_log_parser = @log_parsing ? LogParser::ParseProvisionLogs.new : LogParser::NoParsing.new
|
||||
|
||||
prepare_logs_for_parsing(provision_log_parser, nodes_to_provision)
|
||||
|
||||
nodes_not_booted = []
|
||||
nodes = []
|
||||
nodes_timeout = {}
|
||||
timeouted_uids = []
|
||||
failed_uids = []
|
||||
max_nodes = Astute.config[:max_nodes_to_provision]
|
||||
Astute.logger.debug("Starting provision")
|
||||
catch :done do
|
||||
loop do
|
||||
sleep_not_greater_than(20) do
|
||||
#provision more
|
||||
if nodes_not_booted.count < max_nodes && nodes_to_provision.count > 0
|
||||
new_nodes = nodes_to_provision.shift(max_nodes - nodes_not_booted.count)
|
||||
|
||||
Astute.logger.debug("Provisioning nodes: #{new_nodes}")
|
||||
failed_uids += provision_piece(reporter, task_id, engine_attrs, new_nodes)
|
||||
Astute.logger.info "Nodes failed to reboot: #{failed_uids} "
|
||||
|
||||
nodes_not_booted += new_nodes.map{ |n| n['uid'] }
|
||||
nodes_not_booted -= failed_uids
|
||||
nodes += new_nodes
|
||||
|
||||
timeout_time = Time.now.utc + Astute.config.provisioning_timeout
|
||||
new_nodes.each {|n| nodes_timeout[n['uid']] = timeout_time}
|
||||
end
|
||||
|
||||
nodes_types = node_type(reporter, task_id, nodes.map {|n| n['uid']}, 5)
|
||||
target_uids, nodes_not_booted, reject_uids = analize_node_types(nodes_types, nodes_not_booted)
|
||||
|
||||
if reject_uids.present?
|
||||
ctx ||= Context.new(task_id, reporter)
|
||||
reject_nodes = reject_uids.map { |uid| {'uid' => uid } }
|
||||
NodesRemover.new(ctx, reject_nodes, _reboot=true).remove
|
||||
end
|
||||
|
||||
#check timouted nodes
|
||||
nodes_not_booted.each do |uid|
|
||||
time_now = Time.now.utc
|
||||
if nodes_timeout[uid] < time_now
|
||||
Astute.logger.info "Node timed out to provision: #{uid} "
|
||||
timeouted_uids.push(uid)
|
||||
end
|
||||
end
|
||||
nodes_not_booted -= timeouted_uids
|
||||
|
||||
if should_fail(failed_uids + timeouted_uids, fault_tolerance)
|
||||
Astute.logger.debug("Aborting provision. To many nodes failed: #{failed_uids + timeouted_uids}")
|
||||
Astute.logger.debug("Those nodes where we not yet started provision will be set to error mode")
|
||||
failed_uids += nodes_to_provision.map{ |n| n['uid'] }
|
||||
return failed_uids, timeouted_uids
|
||||
end
|
||||
|
||||
if nodes_not_booted.empty? and nodes_to_provision.empty?
|
||||
Astute.logger.info "Provisioning finished"
|
||||
throw :done
|
||||
end
|
||||
|
||||
Astute.logger.debug("Still provisioning following nodes: #{nodes_not_booted}")
|
||||
report_about_progress(reporter, provision_log_parser, target_uids, nodes)
|
||||
end
|
||||
end
|
||||
end
|
||||
return failed_uids, timeouted_uids
|
||||
end
|
||||
|
||||
def remove_nodes(reporter, task_id, engine_attrs, nodes, options)
|
||||
options.reverse_merge!({
|
||||
:reboot => true,
|
||||
:raise_if_error => false,
|
||||
:reset => false
|
||||
})
|
||||
|
||||
cobbler = CobblerManager.new(engine_attrs, reporter)
|
||||
if options[:reset]
|
||||
cobbler.edit_nodes(nodes, {'profile' => Astute.config.bootstrap_profile})
|
||||
cobbler.netboot_nodes(nodes, true)
|
||||
else
|
||||
cobbler.remove_nodes(nodes)
|
||||
end
|
||||
|
||||
ctx = Context.new(task_id, reporter)
|
||||
result = NodesRemover.new(ctx, nodes, options[:reboot]).remove
|
||||
|
||||
if (result['error_nodes'] || result['inaccessible_nodes']) && options[:raise_if_error]
|
||||
bad_node_ids = result.fetch('error_nodes', []) +
|
||||
result.fetch('inaccessible_nodes', [])
|
||||
raise "Mcollective problem with nodes #{bad_node_ids}, please check log for details"
|
||||
end
|
||||
|
||||
Rsyslogd.send_sighup(ctx, engine_attrs["master_ip"])
|
||||
result
|
||||
end
|
||||
|
||||
def stop_provision(reporter, task_id, engine_attrs, nodes)
|
||||
ctx = Context.new(task_id, reporter)
|
||||
_provisioned_nodes, result = stop_provision_via_mcollective(ctx, nodes)
|
||||
|
||||
result['status'] = 'error' if result['error_nodes'].present?
|
||||
|
||||
Rsyslogd.send_sighup(
|
||||
Context.new(task_id, reporter),
|
||||
engine_attrs["master_ip"]
|
||||
)
|
||||
|
||||
result
|
||||
end
|
||||
|
||||
def provision_piece(reporter, task_id, engine_attrs, nodes)
|
||||
cobbler = CobblerManager.new(engine_attrs, reporter)
|
||||
failed_uids = []
|
||||
# TODO(kozhukalov): do not forget about execute_shell_command timeout which is 3600
|
||||
# provision_and_watch_progress has provisioning_timeout + 3600 is much longer than provisioning_timeout
|
||||
|
||||
# IBP is implemented in terms of Fuel Agent installed into bootstrap ramdisk
|
||||
# we don't want nodes to be rebooted into OS installer ramdisk
|
||||
cobbler.edit_nodes(nodes, {'profile' => Astute.config.bootstrap_profile})
|
||||
|
||||
# change node type to prevent unexpected erase
|
||||
change_nodes_type(reporter, task_id, nodes)
|
||||
# Run parallel reporter
|
||||
report_image_provision(reporter, task_id, nodes) do
|
||||
failed_uids |= image_provision(reporter, task_id, nodes)
|
||||
end
|
||||
provisioned_nodes = nodes.reject { |n| failed_uids.include? n['uid'] }
|
||||
|
||||
# disabling pxe boot (chain loader) for nodes which succeeded
|
||||
cobbler.netboot_nodes(provisioned_nodes, false)
|
||||
|
||||
# in case of IBP we reboot only those nodes which we managed to provision
|
||||
soft_reboot(
|
||||
reporter,
|
||||
task_id,
|
||||
provisioned_nodes.map{ |n| n['uid'] },
|
||||
'reboot_provisioned_nodes'
|
||||
)
|
||||
|
||||
return failed_uids
|
||||
end
|
||||
|
||||
def report_image_provision(reporter, task_id, nodes,
|
||||
provision_log_parser=LogParser::ParseProvisionLogs.new, &block)
|
||||
prepare_logs_for_parsing(provision_log_parser, nodes)
|
||||
|
||||
watch_and_report = Thread.new do
|
||||
loop do
|
||||
report_about_progress(reporter, provision_log_parser, [], nodes)
|
||||
sleep 1
|
||||
end
|
||||
end
|
||||
|
||||
block.call
|
||||
ensure
|
||||
watch_and_report.exit if defined? watch_and_report
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def image_provision(reporter, task_id, nodes)
|
||||
ImageProvision.provision(Context.new(task_id, reporter), nodes)
|
||||
end
|
||||
|
||||
def soft_reboot(reporter, task_id, nodes, task_name)
|
||||
ImageProvision.reboot(Context.new(task_id, reporter), nodes, task_name)
|
||||
end
|
||||
|
||||
def report_result(result, reporter)
|
||||
default_result = {'status' => 'ready', 'progress' => 100}
|
||||
|
||||
result = {} unless result.instance_of?(Hash)
|
||||
status = default_result.merge(result)
|
||||
reporter.report(status)
|
||||
end
|
||||
|
||||
def prepare_logs_for_parsing(provision_log_parser, nodes)
|
||||
sleep_not_greater_than(10) do # Wait while nodes going to reboot
|
||||
Astute.logger.info "Starting OS provisioning for nodes: #{nodes.map{ |n| n['uid'] }.join(',')}"
|
||||
begin
|
||||
provision_log_parser.prepare(nodes)
|
||||
rescue => e
|
||||
Astute.logger.warn "Some error occurred when prepare LogParser: #{e.message}, trace: #{e.format_backtrace}"
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def analize_node_types(types, nodes_not_booted)
|
||||
types.each { |t| Astute.logger.debug("Got node types: uid=#{t['uid']} type=#{t['node_type']}") }
|
||||
target_uids = types.reject{ |n| n['node_type'] != 'target' }.map{ |n| n['uid'] }
|
||||
reject_uids = types.reject{ |n| ['target', 'image'].include? n['node_type'] }.map{ |n| n['uid'] }
|
||||
Astute.logger.debug("Not target nodes will be rejected: #{reject_uids.join(',')}")
|
||||
|
||||
nodes_not_booted -= target_uids
|
||||
Astute.logger.debug "Not provisioned: #{nodes_not_booted.join(',')}, " \
|
||||
"got target OSes: #{target_uids.join(',')}"
|
||||
return target_uids, nodes_not_booted, reject_uids
|
||||
end
|
||||
|
||||
def sleep_not_greater_than(sleep_time, &block)
|
||||
time = Time.now.to_f
|
||||
block.call
|
||||
time = time + sleep_time - Time.now.to_f
|
||||
sleep(time) if time > 0
|
||||
end
|
||||
|
||||
def report_about_progress(reporter, provision_log_parser, target_uids, nodes)
|
||||
begin
|
||||
nodes_progress = provision_log_parser.progress_calculate(nodes.map{ |n| n['uid'] }, nodes)
|
||||
nodes_progress.each do |n|
|
||||
if target_uids.include?(n['uid'])
|
||||
n['progress'] = 100
|
||||
n['status'] = 'provisioned'
|
||||
else
|
||||
n['status'] = 'provisioning'
|
||||
end
|
||||
end
|
||||
reporter.report({'nodes' => nodes_progress})
|
||||
rescue => e
|
||||
Astute.logger.warn "Some error occurred when parse logs for nodes progress: #{e.message}, trace: #{e.format_backtrace}"
|
||||
end
|
||||
end
|
||||
|
||||
def stop_provision_via_mcollective(ctx, nodes)
|
||||
return [], {} if nodes.empty?
|
||||
|
||||
mco_result = {}
|
||||
nodes_uids = nodes.map{ |n| n['uid'] }
|
||||
|
||||
Astute.config.mc_retries.times do |i|
|
||||
sleep Astute.config.nodes_remove_interval
|
||||
|
||||
Astute.logger.debug "Trying to connect to nodes #{nodes_uids} using mcollective"
|
||||
nodes_types = node_type(ctx.reporter, ctx.task_id, nodes_uids, 2)
|
||||
next if nodes_types.empty?
|
||||
|
||||
provisioned = nodes_types.select{ |n| ['target', 'bootstrap', 'image'].include? n['node_type'] }
|
||||
.map{ |n| {'uid' => n['uid']} }
|
||||
current_mco_result = NodesRemover.new(ctx, provisioned, _reboot=true).remove
|
||||
Astute.logger.debug "Retry result #{i}: "\
|
||||
"mco success nodes: #{current_mco_result['nodes']}, "\
|
||||
"mco error nodes: #{current_mco_result['error_nodes']}, "\
|
||||
"mco inaccessible nodes: #{current_mco_result['inaccessible_nodes']}"
|
||||
|
||||
mco_result = merge_rm_nodes_result(mco_result, current_mco_result)
|
||||
nodes_uids -= provisioned.map{ |n| n['uid'] }
|
||||
|
||||
break if nodes_uids.empty?
|
||||
end
|
||||
|
||||
provisioned_nodes = nodes.map{ |n| {'uid' => n['uid']} } - nodes_uids.map {|n| {'uid' => n} }
|
||||
|
||||
Astute.logger.debug "MCO final result: "\
|
||||
"mco success nodes: #{mco_result['nodes']}, "\
|
||||
"mco error nodes: #{mco_result['error_nodes']}, "\
|
||||
"mco inaccessible nodes: #{mco_result['inaccessible_nodes']}, "\
|
||||
"all mco nodes: #{provisioned_nodes}"
|
||||
|
||||
return provisioned_nodes, mco_result
|
||||
end
|
||||
|
||||
def unlock_nodes_discovery(reporter, task_id="", failed_uids, nodes)
|
||||
nodes_uids = nodes.select{ |n| failed_uids.include?(n['uid']) }
|
||||
.map{ |n| n['uid'] }
|
||||
shell = MClient.new(Context.new(task_id, reporter),
|
||||
'execute_shell_command',
|
||||
nodes_uids,
|
||||
_check_result=false,
|
||||
_timeout=2)
|
||||
mco_result = shell.execute(:cmd => "rm -f #{Astute.config.agent_nodiscover_file}")
|
||||
result = mco_result.map do |n|
|
||||
{
|
||||
'uid' => n.results[:sender],
|
||||
'exit code' => n.results[:data][:exit_code]
|
||||
}
|
||||
end
|
||||
Astute.logger.debug "Unlock discovery for failed nodes. Result: #{result}"
|
||||
end
|
||||
|
||||
def merge_rm_nodes_result(res1, res2)
|
||||
['nodes', 'error_nodes', 'inaccessible_nodes'].inject({}) do |result, node_status|
|
||||
result[node_status] = (res1.fetch(node_status, []) + res2.fetch(node_status, [])).uniq
|
||||
result
|
||||
end
|
||||
end
|
||||
|
||||
def change_nodes_type(reporter, task_id, nodes, type="image")
|
||||
nodes_uids = nodes.map{ |n| n['uid'] }
|
||||
shell = MClient.new(Context.new(task_id, reporter),
|
||||
'execute_shell_command',
|
||||
nodes_uids,
|
||||
_check_result=false,
|
||||
_timeout=5)
|
||||
mco_result = shell.execute(:cmd => "echo '#{type}' > /etc/nailgun_systemtype")
|
||||
result = mco_result.map do |n|
|
||||
{
|
||||
'uid' => n.results[:sender],
|
||||
'exit code' => n.results[:data][:exit_code]
|
||||
}
|
||||
end
|
||||
Astute.logger.debug "Change node type to #{type}. Result: #{result}"
|
||||
end
|
||||
|
||||
def handle_failed_nodes(failed_uids, result_msg)
|
||||
if failed_uids.present?
|
||||
Astute.logger.error("Provision of some nodes failed. Failed nodes: #{failed_uids}")
|
||||
nodes_progress = failed_uids.map do |n|
|
||||
{
|
||||
'uid' => n,
|
||||
'status' => 'error',
|
||||
'error_msg' => "Failed to provision",
|
||||
'progress' => 100,
|
||||
'error_type' => 'provision'
|
||||
}
|
||||
end
|
||||
result_msg['nodes'] += nodes_progress
|
||||
end
|
||||
end
|
||||
|
||||
def handle_timeouted_nodes(timeouted_uids, result_msg)
|
||||
if timeouted_uids.present?
|
||||
Astute.logger.error("Timeout of provisioning is exceeded. Nodes not booted: #{timeouted_uids}")
|
||||
nodes_progress = timeouted_uids.map do |n|
|
||||
{
|
||||
'uid' => n,
|
||||
'status' => 'error',
|
||||
'error_msg' => "Timeout of provisioning is exceeded",
|
||||
'progress' => 100,
|
||||
'error_type' => 'provision'
|
||||
}
|
||||
end
|
||||
result_msg['nodes'] += nodes_progress
|
||||
end
|
||||
end
|
||||
|
||||
def should_fail(failed_uids, fault_tolerance)
|
||||
return failed_uids.present? if fault_tolerance.empty?
|
||||
|
||||
uids_in_groups = []
|
||||
fault_tolerance.each do |group|
|
||||
failed_from_group = failed_uids.select { |uid| group['uids'].include? uid }
|
||||
uids_in_groups += failed_from_group
|
||||
max_to_fail = group['percentage'] / 100.0 * group['uids'].count
|
||||
if failed_from_group.count > max_to_fail
|
||||
return true
|
||||
end
|
||||
end
|
||||
failed_uids -= uids_in_groups
|
||||
if failed_uids.present?
|
||||
return true
|
||||
end
|
||||
false
|
||||
end
|
||||
|
||||
def prepare_nodes(reporter, task_id, engine_attrs, nodes, cobbler)
|
||||
# 1. Erase all nodes
|
||||
# 2. Return already provisioned node to bootstrap state
|
||||
# 3. Delete and add again nodes to Cobbler
|
||||
|
||||
Astute.logger.info "Preparing nodes for installation"
|
||||
existent_nodes = cobbler.get_existent_nodes(nodes)
|
||||
|
||||
# Change node type to prevent wrong node detection as provisioned
|
||||
# Also this type if node will not rebooted, Astute will be allowed
|
||||
# to try to reboot such nodes again
|
||||
if existent_nodes.present?
|
||||
change_nodes_type(reporter, task_id, existent_nodes, 'reprovisioned')
|
||||
end
|
||||
|
||||
remove_nodes(
|
||||
reporter,
|
||||
task_id,
|
||||
engine_attrs,
|
||||
nodes,
|
||||
{:reboot => false,
|
||||
:raise_if_error => true,
|
||||
:reset => true}
|
||||
)
|
||||
|
||||
if existent_nodes.present?
|
||||
soft_reboot(
|
||||
reporter,
|
||||
task_id,
|
||||
existent_nodes.map{ |n| n['uid'] },
|
||||
"reboot_reprovisioned_nodes"
|
||||
)
|
||||
end
|
||||
|
||||
cobbler.remove_nodes(nodes)
|
||||
|
||||
# NOTE(kozhukalov): We try to find out if there are systems
|
||||
# in the Cobbler with the same MAC addresses. If so, Cobbler is going
|
||||
# to throw MAC address duplication error. We need to remove these
|
||||
# nodes.
|
||||
mac_duplicate_names = cobbler.get_mac_duplicate_names(nodes)
|
||||
if mac_duplicate_names.present?
|
||||
cobbler.remove_nodes(mac_duplicate_names.map {|n| {'slave_name' => n}})
|
||||
end
|
||||
|
||||
cobbler.add_nodes(nodes)
|
||||
end
|
||||
|
||||
end
|
||||
end
|
|
@ -1,282 +0,0 @@
|
|||
# Copyright 2016 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
class PuppetJob
|
||||
|
||||
FINAL_JOB_TASK_STATUSES = [
|
||||
'successful', 'failed'
|
||||
]
|
||||
|
||||
JOB_TASK_STATUSES = [
|
||||
'successful', 'failed', 'running'
|
||||
]
|
||||
|
||||
SUCCEED_STATUSES = [
|
||||
'succeed'
|
||||
]
|
||||
|
||||
BUSY_STATUSES = [
|
||||
'running'
|
||||
]
|
||||
|
||||
UNDEFINED_STATUSES = [
|
||||
'undefined'
|
||||
]
|
||||
|
||||
STOPED_STATUSES = [
|
||||
'stopped', 'disabled'
|
||||
]
|
||||
|
||||
def initialize(task, puppet_mclient, options)
|
||||
@task = task
|
||||
@retries = options['retries']
|
||||
@puppet_start_timeout = options['puppet_start_timeout']
|
||||
@puppet_start_interval = options['puppet_start_interval']
|
||||
@time_observer = TimeObserver.new(options['timeout'])
|
||||
@succeed_retries = options['succeed_retries']
|
||||
@undefined_retries = options['undefined_retries']
|
||||
@original_undefined_retries = options['undefined_retries']
|
||||
@puppet_mclient = puppet_mclient
|
||||
end
|
||||
|
||||
# Run selected puppet manifest on node
|
||||
# @return [void]
|
||||
def run
|
||||
Astute.logger.info "Start puppet with timeout "\
|
||||
"#{@time_observer.time_limit} sec. #{task_details_for_log}"
|
||||
|
||||
@time_observer.start
|
||||
puppetd_run
|
||||
self.task_status = 'running'
|
||||
end
|
||||
|
||||
# Return actual status of puppet run
|
||||
# @return [String] Task status: successful, failed or running
|
||||
def status
|
||||
return @task_status if job_ended?
|
||||
|
||||
current_task_status = puppet_to_task_status(puppet_status)
|
||||
|
||||
self.task_status = case current_task_status
|
||||
when 'successful'
|
||||
processing_succeed_task
|
||||
when 'running'
|
||||
processing_running_task
|
||||
when 'failed'
|
||||
processing_error_task
|
||||
when 'undefined'
|
||||
processing_undefined_task
|
||||
end
|
||||
|
||||
time_is_up! if should_stop?
|
||||
@task_status
|
||||
end
|
||||
|
||||
# Return actual last run summary for puppet run
|
||||
# @return [Hash] Puppet summary
|
||||
def summary
|
||||
@puppet_mclient.summary
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
# Should stop process or not: task is still running but we are out of time
|
||||
# @return [true, false]
|
||||
def should_stop?
|
||||
@time_observer.time_is_up? &&
|
||||
FINAL_JOB_TASK_STATUSES.exclude?(@task_status)
|
||||
end
|
||||
|
||||
# Is job has ended or not
|
||||
# @return [true, false]
|
||||
def job_ended?
|
||||
FINAL_JOB_TASK_STATUSES.include?(@task_status)
|
||||
end
|
||||
|
||||
# Set task status to failed and reset retires counter to 0 to avoid
|
||||
# redundant retries
|
||||
# @return [void]
|
||||
def time_is_up!
|
||||
Astute.logger.error "Puppet agent took too long to run puppet task."\
|
||||
" Mark task as failed. #{task_details_for_log}"
|
||||
self.task_status = 'failed'
|
||||
end
|
||||
|
||||
# Setup task status
|
||||
# @param [String] status The task status
|
||||
# @return [void]
|
||||
# @raise [StatusValidationError] Unknown job status
|
||||
def task_status=(status)
|
||||
if JOB_TASK_STATUSES.include?(status)
|
||||
@task_status = status
|
||||
else
|
||||
raise StatusValidationError,
|
||||
"Unknow job status: #{status}. Expected: #{JOB_TASK_STATUSES}"
|
||||
end
|
||||
end
|
||||
|
||||
# Return actual status of puppet using mcollective puppet agent
|
||||
# @return [String]: puppet status
|
||||
def puppet_status
|
||||
actual_status = @puppet_mclient.status
|
||||
log_current_status(actual_status)
|
||||
|
||||
if UNDEFINED_STATUSES.include?(actual_status)
|
||||
Astute.logger.warn "Error to get puppet status. "\
|
||||
"#{task_details_for_log}."
|
||||
end
|
||||
|
||||
actual_status
|
||||
end
|
||||
|
||||
# Run puppet manifest using mcollective puppet agent
|
||||
# @return [true, false] Is puppet run has started or not
|
||||
# TODO(vsharshov): need refactoring to make this be async call
|
||||
def puppetd_run
|
||||
puppet_run_obsorver = TimeObserver.new(@puppet_start_timeout)
|
||||
puppet_run_obsorver.start
|
||||
|
||||
while puppet_run_obsorver.enough_time?
|
||||
is_running = @puppet_mclient.run
|
||||
return true if is_running
|
||||
|
||||
Astute.logger.debug "Could not run puppet process "\
|
||||
"#{task_details_for_log}. Left #{puppet_run_obsorver.left_time} sec"
|
||||
sleep @puppet_start_interval
|
||||
end
|
||||
Astute.logger.error "Problem with puppet start. Time "\
|
||||
"(#{@puppet_start_timeout} sec) is over. #{task_details_for_log}"
|
||||
false
|
||||
end
|
||||
|
||||
# Convert puppet status to task status
|
||||
# @param [String] puppet_status The puppet status of task
|
||||
# @return [String] Task status
|
||||
# @raise [StatusValidationError] Unknown puppet status
|
||||
def puppet_to_task_status(mco_puppet_status)
|
||||
case
|
||||
when SUCCEED_STATUSES.include?(mco_puppet_status)
|
||||
'successful'
|
||||
when BUSY_STATUSES.include?(mco_puppet_status)
|
||||
'running'
|
||||
when STOPED_STATUSES.include?(mco_puppet_status)
|
||||
'failed'
|
||||
when UNDEFINED_STATUSES.include?(mco_puppet_status)
|
||||
'undefined'
|
||||
else
|
||||
raise StatusValidationError,
|
||||
"Unknow puppet status: #{mco_puppet_status}"
|
||||
end
|
||||
end
|
||||
|
||||
# Return short useful info about node and puppet task
|
||||
# @return [String]
|
||||
def task_details_for_log
|
||||
"Node #{@puppet_mclient.node_id}, task #{@task}, manifest "\
|
||||
"#{@puppet_mclient.manifest}"
|
||||
end
|
||||
|
||||
# Write to log with needed message level actual task status
|
||||
# @param [String] status Actual puppet status of task
|
||||
# @return [void]
|
||||
def log_current_status(status)
|
||||
message = "#{task_details_for_log}, status: #{status}"
|
||||
if (UNDEFINED_STATUSES + STOPED_STATUSES).include?(status)
|
||||
Astute.logger.error message
|
||||
else
|
||||
Astute.logger.debug message
|
||||
end
|
||||
end
|
||||
|
||||
# Process additional action in case of puppet succeed
|
||||
# @return [String] Task status: successful or running
|
||||
def processing_succeed_task
|
||||
reset_undefined_retries!
|
||||
Astute.logger.debug "Puppet completed within "\
|
||||
"#{@time_observer.since_start} seconds"
|
||||
if @succeed_retries > 0
|
||||
@succeed_retries -= 1
|
||||
Astute.logger.debug "Succeed puppet on node will be "\
|
||||
"restarted. #{@succeed_retries} retries remained. "\
|
||||
"#{task_details_for_log}"
|
||||
Astute.logger.info "Retrying to run puppet for following succeed "\
|
||||
"node: #{@puppet_mclient.node_id}"
|
||||
puppetd_run
|
||||
'running'
|
||||
else
|
||||
Astute.logger.info "Node #{@puppet_mclient.node_id} has succeed "\
|
||||
"to deploy. #{task_details_for_log}"
|
||||
'successful'
|
||||
end
|
||||
end
|
||||
|
||||
# Process additional action in case of puppet failed
|
||||
# @return [String] Task status: failed or running
|
||||
def processing_error_task
|
||||
reset_undefined_retries!
|
||||
if @retries > 0
|
||||
@retries -= 1
|
||||
Astute.logger.debug "Puppet on node will be "\
|
||||
"restarted because of fail. #{@retries} retries remained."\
|
||||
"#{task_details_for_log}"
|
||||
Astute.logger.info "Retrying to run puppet for following error "\
|
||||
"nodes: #{@puppet_mclient.node_id}"
|
||||
puppetd_run
|
||||
'running'
|
||||
else
|
||||
Astute.logger.error "Node has failed to deploy. There is"\
|
||||
" no more retries for puppet run. #{task_details_for_log}"
|
||||
'failed'
|
||||
end
|
||||
end
|
||||
|
||||
# Process additional action in case of undefined puppet status
|
||||
# @return [String] Task status: failed or running
|
||||
def processing_undefined_task
|
||||
if @undefined_retries > 0
|
||||
@undefined_retries -= 1
|
||||
Astute.logger.debug "Puppet on node has undefined status. "\
|
||||
"#{@undefined_retries} retries remained. "\
|
||||
"#{task_details_for_log}"
|
||||
Astute.logger.info "Retrying to check status for following "\
|
||||
"nodes: #{@puppet_mclient.node_id}"
|
||||
'running'
|
||||
else
|
||||
Astute.logger.error "Node has failed to get status. There is"\
|
||||
" no more retries for status check. #{task_details_for_log}"
|
||||
'failed'
|
||||
end
|
||||
end
|
||||
|
||||
# Process additional action in case of puppet running
|
||||
# @return [String]: Task status: successful, failed or running
|
||||
def processing_running_task
|
||||
reset_undefined_retries!
|
||||
'running'
|
||||
end
|
||||
|
||||
# Reset undefined retries to original value
|
||||
# @return [void]
|
||||
def reset_undefined_retries!
|
||||
return if @undefined_retries == @original_undefined_retries
|
||||
|
||||
Astute.logger.debug "Reset undefined retries to original "\
|
||||
"value: #{@original_undefined_retries}"
|
||||
@undefined_retries = @original_undefined_retries
|
||||
end
|
||||
|
||||
end #PuppetJob
|
||||
|
||||
end
|
|
@ -1,258 +0,0 @@
|
|||
# Copyright 2014 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
require 'timeout'
|
||||
|
||||
module Astute
|
||||
|
||||
# @deprecated Please use {#Astute::PuppetJob} instead. This code is
|
||||
# useful only for Granular or older deployment engines.
|
||||
class PuppetTask
|
||||
|
||||
def initialize(ctx, node, options={})
|
||||
default_options = {
|
||||
:retries => Astute.config.puppet_retries,
|
||||
:puppet_manifest => '/etc/puppet/manifests/site.pp',
|
||||
:puppet_modules => Astute.config.puppet_module_path,
|
||||
:cwd => Astute.config.shell_cwd,
|
||||
:timeout => Astute.config.puppet_timeout,
|
||||
:puppet_debug => false,
|
||||
:succeed_retries => Astute.config.puppet_succeed_retries,
|
||||
:raw_report => Astute.config.puppet_raw_report,
|
||||
:puppet_noop_run => Astute.config.puppet_noop_run,
|
||||
}
|
||||
@options = options.compact.reverse_merge(default_options)
|
||||
@options.freeze
|
||||
|
||||
@ctx = ctx
|
||||
@node = node
|
||||
@retries = @options[:retries]
|
||||
@time_observer = TimeObserver.new(@options[:timeout])
|
||||
@is_hung = false
|
||||
@succeed_retries = @options[:succeed_retries]
|
||||
@summary = {}
|
||||
end
|
||||
|
||||
def run
|
||||
Astute.logger.debug "Waiting for puppet to finish deployment on " \
|
||||
"node #{@node['uid']} (timeout = #{@time_observer.time_limit} sec)..."
|
||||
@time_observer.start
|
||||
puppetd_runonce
|
||||
end
|
||||
|
||||
# expect to run this method with respect of Astute.config.puppet_fade_interval
|
||||
def status
|
||||
raise Timeout::Error if @time_observer.time_is_up?
|
||||
|
||||
@summary = puppet_status
|
||||
status = node_status(@summary)
|
||||
message = "Node #{@node['uid']}(#{@node['role']}) status: #{status}"
|
||||
if status == 'error'
|
||||
Astute.logger.error message
|
||||
else
|
||||
Astute.logger.debug message
|
||||
end
|
||||
|
||||
result = case status
|
||||
when 'succeed'
|
||||
processing_succeed_node(@summary)
|
||||
when 'running'
|
||||
processing_running_node
|
||||
when 'error'
|
||||
processing_error_node(@summary)
|
||||
end
|
||||
|
||||
#TODO(vsharshov): Should we move it to control module?
|
||||
@ctx.report_and_update_status('nodes' => [result]) if result
|
||||
|
||||
# ready, error or deploying
|
||||
result.fetch('status', 'deploying')
|
||||
rescue MClientTimeout, Timeout::Error
|
||||
Astute.logger.warn "Puppet agent #{@node['uid']} " \
|
||||
"didn't respond within the allotted time"
|
||||
'error'
|
||||
end
|
||||
|
||||
def summary
|
||||
@summary
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def puppetd
|
||||
puppetd = MClient.new(
|
||||
@ctx,
|
||||
"puppetd",
|
||||
[@node['uid']],
|
||||
_check_result=true,
|
||||
_timeout=nil,
|
||||
_retries=Astute.config.mc_retries,
|
||||
_enable_result_logging=false
|
||||
)
|
||||
puppetd.on_respond_timeout do |uids|
|
||||
nodes = uids.map do |uid|
|
||||
{
|
||||
'uid' => uid,
|
||||
'status' => 'error',
|
||||
'error_type' => 'deploy',
|
||||
'role' => @node['role']
|
||||
}
|
||||
end
|
||||
@ctx.report_and_update_status('nodes' => nodes)
|
||||
raise MClientTimeout
|
||||
end
|
||||
puppetd
|
||||
end
|
||||
|
||||
def puppet_status
|
||||
puppetd.last_run_summary(
|
||||
:puppet_noop_run => @options[:puppet_noop_run],
|
||||
:raw_report => @options[:raw_report]
|
||||
).first[:data]
|
||||
end
|
||||
|
||||
def puppet_run
|
||||
puppetd.runonce(
|
||||
:puppet_debug => @options[:puppet_debug],
|
||||
:manifest => @options[:puppet_manifest],
|
||||
:modules => @options[:puppet_modules],
|
||||
:cwd => @options[:cwd],
|
||||
:puppet_noop_run => @options[:puppet_noop_run],
|
||||
)
|
||||
end
|
||||
|
||||
def running?(status)
|
||||
['running'].include? status[:status]
|
||||
end
|
||||
|
||||
def idling?(status)
|
||||
['idling'].include? status[:status]
|
||||
end
|
||||
|
||||
def stopped?(status)
|
||||
['stopped', 'disabled'].include? status[:status]
|
||||
end
|
||||
|
||||
def succeed?(status)
|
||||
status[:status] == 'stopped' &&
|
||||
status[:resources]['failed'].to_i == 0 &&
|
||||
status[:resources]['failed_to_restart'].to_i == 0
|
||||
end
|
||||
|
||||
# Runs puppetd.runonce only if puppet is stopped on the host at the time
|
||||
# If it isn't stopped, we wait a bit and try again.
|
||||
# Returns list of nodes uids which appear to be with hung puppet.
|
||||
def puppetd_runonce
|
||||
started = Time.now.to_i
|
||||
while Time.now.to_i - started < Astute.config.puppet_fade_timeout
|
||||
status = puppet_status
|
||||
|
||||
is_stopped = stopped?(status)
|
||||
is_idling = idling?(status)
|
||||
is_running = running?(status)
|
||||
|
||||
#Try to kill 'idling' process and run again by 'runonce' call
|
||||
puppet_run if is_stopped || is_idling
|
||||
|
||||
break if !is_running && !is_idling
|
||||
sleep Astute.config.puppet_fade_interval
|
||||
end
|
||||
|
||||
if is_running || is_idling
|
||||
Astute.logger.warn "Following nodes have puppet hung " \
|
||||
"(#{is_running ? 'running' : 'idling'}): '#{@node['uid']}'"
|
||||
@is_hung = true
|
||||
else
|
||||
@is_hung = false
|
||||
end
|
||||
end
|
||||
|
||||
def node_status(last_run)
|
||||
case
|
||||
when @is_hung
|
||||
'error'
|
||||
when succeed?(last_run) && !@is_hung
|
||||
'succeed'
|
||||
when (running?(last_run) || idling?(last_run)) && !@is_hung
|
||||
'running'
|
||||
when stopped?(last_run) && !succeed?(last_run) && !@is_hung
|
||||
'error'
|
||||
else
|
||||
msg = "Unknow status: " \
|
||||
"is_hung #{@is_hung}, succeed? #{succeed?(last_run)}, " \
|
||||
"running? #{running?(last_run)}, stopped? #{stopped?(last_run)}, " \
|
||||
"idling? #{idling?(last_run)}"
|
||||
raise msg
|
||||
end
|
||||
end
|
||||
|
||||
def processing_succeed_node(last_run)
|
||||
Astute.logger.debug "Puppet completed within "\
|
||||
"#{@time_observer.since_start} seconds"
|
||||
if @succeed_retries > 0
|
||||
@succeed_retries -= 1
|
||||
Astute.logger.debug "Succeed puppet on node #{@node['uid']} will be "\
|
||||
"restarted. #{@succeed_retries} retries remained."
|
||||
Astute.logger.info "Retrying to run puppet for following succeed " \
|
||||
"node: #{@node['uid']}"
|
||||
puppetd_runonce
|
||||
node_report_format('status' => 'deploying')
|
||||
else
|
||||
Astute.logger.debug "Node #{@node['uid']} has succeed to deploy. " \
|
||||
"There is no more retries for puppet run."
|
||||
{ 'uid' => @node['uid'], 'status' => 'ready', 'role' => @node['role'] }
|
||||
end
|
||||
end
|
||||
|
||||
def processing_error_node(last_run)
|
||||
if @retries > 0
|
||||
@retries -= 1
|
||||
Astute.logger.debug "Puppet on node #{@node['uid']} will be "\
|
||||
"restarted. #{@retries} retries remained."
|
||||
Astute.logger.info "Retrying to run puppet for following error " \
|
||||
"nodes: #{@node['uid']}"
|
||||
puppetd_runonce
|
||||
node_report_format('status' => 'deploying')
|
||||
else
|
||||
Astute.logger.debug "Node #{@node['uid']} has failed to deploy. " \
|
||||
"There is no more retries for puppet run."
|
||||
node_report_format('status' => 'error', 'error_type' => 'deploy')
|
||||
end
|
||||
end
|
||||
|
||||
def processing_running_node
|
||||
nodes_to_report = []
|
||||
begin
|
||||
# Pass nodes because logs calculation needs IP address of node, not just uid
|
||||
nodes_progress = @ctx.deploy_log_parser.progress_calculate([@node['uid']], [@node])
|
||||
if nodes_progress.present?
|
||||
Astute.logger.debug "Got progress for nodes:\n#{nodes_progress.pretty_inspect}"
|
||||
|
||||
# Nodes with progress are running, so they are not included in nodes_to_report yet
|
||||
nodes_progress.map! { |x| x.merge!('status' => 'deploying', 'role' => @node['role']) }
|
||||
nodes_to_report = nodes_progress
|
||||
end
|
||||
rescue => e
|
||||
Astute.logger.warn "Some error occurred when parse logs for " \
|
||||
"nodes progress: #{e.message}, trace: #{e.format_backtrace}"
|
||||
end
|
||||
nodes_to_report.first || node_report_format('status' => 'deploying')
|
||||
end
|
||||
|
||||
def node_report_format(add_info={})
|
||||
add_info.merge('uid' => @node['uid'], 'role' => @node['role'])
|
||||
end
|
||||
|
||||
end #PuppetTask
|
||||
end
|
|
@ -1,69 +0,0 @@
|
|||
# Copyright 2014 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
|
||||
require 'json'
|
||||
require 'timeout'
|
||||
|
||||
module Astute
|
||||
module PuppetdDeployer
|
||||
|
||||
def self.deploy(ctx, nodes, retries=2, puppet_manifest=nil, puppet_modules=nil, cwd=nil, puppet_debug=true)
|
||||
@ctx = ctx
|
||||
@retries = retries
|
||||
@nodes = nodes
|
||||
@puppet_manifest = puppet_manifest || '/etc/puppet/manifests/site.pp'
|
||||
@puppet_modules = puppet_modules || '/etc/puppet/modules'
|
||||
@cwd = cwd || '/'
|
||||
@puppet_debug = puppet_debug
|
||||
|
||||
Astute.logger.debug "Waiting for puppet to finish deployment on all
|
||||
nodes (timeout = #{Astute.config.puppet_timeout} sec)..."
|
||||
time_before = Time.now
|
||||
|
||||
deploy_nodes
|
||||
|
||||
time_spent = Time.now - time_before
|
||||
Astute.logger.info "#{@ctx.task_id}: Spent #{time_spent} seconds on puppet run "\
|
||||
"for following nodes(uids): #{@nodes.map {|n| n['uid']}.join(',')}"
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def self.deploy_nodes
|
||||
puppet_tasks = @nodes.map { |n| puppet_task(n) }
|
||||
puppet_tasks.each(&:run)
|
||||
|
||||
loop do
|
||||
sleep Astute.config.puppet_deploy_interval
|
||||
break if !puppet_tasks.any? { |t| t.status == 'deploying' }
|
||||
end
|
||||
end
|
||||
|
||||
def self.puppet_task(n)
|
||||
PuppetTask.new(
|
||||
@ctx,
|
||||
n,
|
||||
{
|
||||
:retries => @retries,
|
||||
:puppet_manifest => @puppet_manifest,
|
||||
:puppet_modules => @puppet_modules,
|
||||
:cwd => @cwd,
|
||||
:timeout => Astute.config.puppet_timeout,
|
||||
:puppet_debug => @puppet_debug
|
||||
})
|
||||
end
|
||||
|
||||
end
|
||||
end
|
|
@ -1,293 +0,0 @@
|
|||
# -*- coding: utf-8 -*-
|
||||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
require 'set'
|
||||
|
||||
STATES = {
|
||||
'offline' => 0,
|
||||
'discover' => 10,
|
||||
'provisioning' => 30,
|
||||
'provisioned' => 40,
|
||||
'deploying' => 50,
|
||||
'ready' => 60,
|
||||
'error' => 70
|
||||
}
|
||||
|
||||
module Astute
|
||||
module ProxyReporter
|
||||
class DeploymentProxyReporter
|
||||
attr_accessor :deploy
|
||||
alias_method :deploy?, :deploy
|
||||
|
||||
def initialize(up_reporter, deployment_info=[])
|
||||
@up_reporter = up_reporter
|
||||
@nodes = deployment_info.inject([]) do |nodes, di|
|
||||
nodes << {'uid' => di['uid'], 'role' => di['role'], 'fail_if_error' => di['fail_if_error']}
|
||||
end
|
||||
@deploy = deployment_info.present?
|
||||
end
|
||||
|
||||
def report(data)
|
||||
Astute.logger.debug("Data received by DeploymentProxyReporter to report it up:\n#{data.pretty_inspect}")
|
||||
report_new_data(data)
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def report_new_data(data)
|
||||
if data['nodes']
|
||||
nodes_to_report = get_nodes_to_report(data['nodes'])
|
||||
return if nodes_to_report.empty? # Let's report only if nodes updated
|
||||
|
||||
# Update nodes attributes in @nodes.
|
||||
update_saved_nodes(nodes_to_report)
|
||||
data['nodes'] = nodes_to_report
|
||||
end
|
||||
data.merge!(get_overall_status(data))
|
||||
Astute.logger.debug("Data send by DeploymentProxyReporter to report it up:\n#{data.pretty_inspect}")
|
||||
@up_reporter.report(data)
|
||||
end
|
||||
|
||||
def get_overall_status(data)
|
||||
status = data['status']
|
||||
error_nodes = @nodes.select { |n| n['status'] == 'error' }.map{ |n| n['uid'] }
|
||||
msg = data['error']
|
||||
|
||||
if status == 'ready' && error_nodes.any?
|
||||
status = 'error'
|
||||
msg = "Some error occured on nodes\n#{error_nodes.pretty_inspect}"
|
||||
end
|
||||
progress = data['progress']
|
||||
|
||||
{'status' => status, 'error' => msg, 'progress' => progress}.reject{|k,v| v.nil?}
|
||||
end
|
||||
|
||||
def get_nodes_to_report(nodes)
|
||||
nodes.map{ |node| node_validate(node) }.compact
|
||||
end
|
||||
|
||||
def update_saved_nodes(new_nodes)
|
||||
# Update nodes attributes in @nodes.
|
||||
new_nodes.each do |node|
|
||||
saved_node = @nodes.find { |n| n['uid'] == node['uid'] && n['role'] == node['role'] }
|
||||
if saved_node
|
||||
node.each {|k, v| saved_node[k] = v}
|
||||
else
|
||||
@nodes << node
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def node_validate(node)
|
||||
validates_basic_fields(node)
|
||||
|
||||
# Ignore hooks report for multiroles progress
|
||||
calculate_multiroles_node_progress(node) if deploy? && node['role'] != 'hook'
|
||||
|
||||
normalization_progress(node)
|
||||
|
||||
compare_with_previous_state(node)
|
||||
end
|
||||
|
||||
# Validate of basic fields in message about nodes
|
||||
def validates_basic_fields(node)
|
||||
err = []
|
||||
case
|
||||
when node['status']
|
||||
err << "Status provided #{node['status']} is not supported" unless STATES[node['status']]
|
||||
when node['progress']
|
||||
err << "progress value provided, but no status"
|
||||
end
|
||||
|
||||
err << "Node role is not provided" if deploy? && !node['role']
|
||||
err << "Node uid is not provided" unless node['uid']
|
||||
|
||||
if err.any?
|
||||
msg = "Validation of node:\n#{node.pretty_inspect} for report failed: #{err.join('; ')}."
|
||||
Astute.logger.error(msg)
|
||||
raise msg
|
||||
end
|
||||
end
|
||||
|
||||
# Proportionally reduce the progress on the number of roles. Based on the
|
||||
# fact that each part makes the same contribution to the progress we divide
|
||||
# 100 to number of roles for this node. Also we prevent send final status for
|
||||
# node before all roles will be deployed. Final result for node:
|
||||
# * any error — error;
|
||||
# * without error — succes.
|
||||
# Example:
|
||||
# Node have 3 roles and already success deploy first role and now deploying
|
||||
# second(50%). Overall progress of the operation for node is
|
||||
# 50 / 3 + 1 * 100 / 3 = 49
|
||||
# We calculate it as 100/3 = 33% for every finished(success or fail) role
|
||||
# Exception: node which have fail_if_error status equal true for some
|
||||
# assigned node role. If this node role fail, we send error state for
|
||||
# the entire node immediately.
|
||||
def calculate_multiroles_node_progress(node)
|
||||
@finish_roles_for_nodes ||= []
|
||||
roles_of_node = @nodes.select { |n| n['uid'] == node['uid'] }
|
||||
all_roles_amount = roles_of_node.size
|
||||
|
||||
return if all_roles_amount == 1 # calculation should only be done for multi roles
|
||||
|
||||
finish_roles_amount = @finish_roles_for_nodes.select do |n|
|
||||
n['uid'] == node['uid'] && ['ready', 'error'].include?(n['status'])
|
||||
end.size
|
||||
return if finish_roles_amount == all_roles_amount # already done all work
|
||||
|
||||
# recalculate progress for node
|
||||
node['progress'] = node['progress'].to_i/all_roles_amount + 100 * finish_roles_amount/all_roles_amount
|
||||
|
||||
# save final state if present
|
||||
if ['ready', 'error'].include? node['status']
|
||||
@finish_roles_for_nodes << { 'uid' => node['uid'], 'role' => node['role'], 'status' => node['status'] }
|
||||
node['progress'] = 100 * (finish_roles_amount + 1)/all_roles_amount
|
||||
end
|
||||
|
||||
# No more status update will be for node which failed and have fail_if_error as true
|
||||
fail_if_error = @nodes.find { |n| n['uid'] == node['uid'] && n['role'] == node['role'] }['fail_if_error']
|
||||
fail_now = fail_if_error && node['status'] == 'error'
|
||||
|
||||
if all_roles_amount - finish_roles_amount != 1 && !fail_now
|
||||
# block 'ready' or 'error' final status for node if not all roles yet deployed
|
||||
node['status'] = 'deploying'
|
||||
node.delete('error_type') # Additional field for error response
|
||||
elsif ['ready', 'error'].include? node['status']
|
||||
node['status'] = @finish_roles_for_nodes.select { |n| n['uid'] == node['uid'] }
|
||||
.select { |n| n['status'] == 'error' }
|
||||
.empty? ? 'ready' : 'error'
|
||||
node['progress'] = 100
|
||||
node['error_type'] = 'deploy' if node['status'] == 'error'
|
||||
end
|
||||
end
|
||||
|
||||
# Normalization of progress field: ensures that the scaling progress was
|
||||
# in range from 0 to 100 and has a value of 100 fot the final node status
|
||||
def normalization_progress(node)
|
||||
if node['progress']
|
||||
if node['progress'] > 100
|
||||
Astute.logger.warn("Passed report for node with progress > 100: "\
|
||||
"#{node.pretty_inspect}. Adjusting progress to 100.")
|
||||
node['progress'] = 100
|
||||
elsif node['progress'] < 0
|
||||
Astute.logger.warn("Passed report for node with progress < 0: "\
|
||||
"#{node.pretty_inspect}. Adjusting progress to 0.")
|
||||
node['progress'] = 0
|
||||
end
|
||||
end
|
||||
if node['status'] && ['provisioned', 'ready'].include?(node['status']) && node['progress'] != 100
|
||||
Astute.logger.warn("In #{node['status']} state node should have progress 100, "\
|
||||
"but node passed:\n#{node.pretty_inspect}. Setting it to 100")
|
||||
node['progress'] = 100
|
||||
end
|
||||
end
|
||||
|
||||
# Comparison information about node with previous state.
|
||||
def compare_with_previous_state(node)
|
||||
saved_node = @nodes.find { |x| x['uid'] == node['uid'] && x['role'] == node['role'] }
|
||||
if saved_node
|
||||
saved_status = STATES[saved_node['status']].to_i
|
||||
node_status = STATES[node['status']] || saved_status
|
||||
saved_progress = saved_node['progress'].to_i
|
||||
node_progress = node['progress'] || saved_progress
|
||||
|
||||
if node_status < saved_status
|
||||
Astute.logger.warn("Attempt to assign lower status detected: "\
|
||||
"Status was: #{saved_node['status']}, attempted to "\
|
||||
"assign: #{node['status']}. Skipping this node (id=#{node['uid']})")
|
||||
return
|
||||
end
|
||||
if node_progress < saved_progress && node_status == saved_status
|
||||
Astute.logger.warn("Attempt to assign lesser progress detected: "\
|
||||
"Progress was: #{saved_node['status']}, attempted to "\
|
||||
"assign: #{node['progress']}. Skipping this node (id=#{node['uid']})")
|
||||
return
|
||||
end
|
||||
|
||||
# We need to update node here only if progress is greater, or status changed
|
||||
return if node.select{|k, v| saved_node[k] != v }.empty?
|
||||
end
|
||||
node
|
||||
end
|
||||
|
||||
end # DeploymentProxyReporter
|
||||
|
||||
class ProvisiningProxyReporter < DeploymentProxyReporter
|
||||
|
||||
def initialize(up_reporter, provisioning_info=[])
|
||||
@up_reporter = up_reporter
|
||||
@nodes = provisioning_info['nodes'].inject([]) do |nodes, di|
|
||||
nodes << {'uid' => di['uid']}
|
||||
end
|
||||
end
|
||||
|
||||
def report(data)
|
||||
Astute.logger.debug("Data received by ProvisiningProxyReporter to report it up:\n#{data.pretty_inspect}")
|
||||
report_new_data(data)
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def report_new_data(data)
|
||||
if data['nodes']
|
||||
nodes_to_report = get_nodes_to_report(data['nodes'])
|
||||
return if nodes_to_report.empty? # Let's report only if nodes updated
|
||||
|
||||
# Update nodes attributes in @nodes.
|
||||
update_saved_nodes(nodes_to_report)
|
||||
data['nodes'] = nodes_to_report
|
||||
end
|
||||
Astute.logger.debug("Data send by DeploymentProxyReporter to report it up:\n#{data.pretty_inspect}")
|
||||
@up_reporter.report(data)
|
||||
end
|
||||
|
||||
def node_validate(node)
|
||||
validates_basic_fields(node)
|
||||
normalization_progress(node)
|
||||
compare_with_previous_state(node)
|
||||
end
|
||||
|
||||
# Comparison information about node with previous state.
|
||||
def compare_with_previous_state(node)
|
||||
saved_node = @nodes.find { |x| x['uid'] == node['uid'] }
|
||||
if saved_node
|
||||
saved_status = STATES[saved_node['status']].to_i
|
||||
node_status = STATES[node['status']] || saved_status
|
||||
saved_progress = saved_node['progress'].to_i
|
||||
node_progress = node['progress'] || saved_progress
|
||||
|
||||
if node_status < saved_status
|
||||
Astute.logger.warn("Attempt to assign lower status detected: "\
|
||||
"Status was: #{saved_node['status']}, attempted to "\
|
||||
"assign: #{node['status']}. Skipping this node (id=#{node['uid']})")
|
||||
return
|
||||
end
|
||||
if node_progress < saved_progress && node_status == saved_status
|
||||
Astute.logger.warn("Attempt to assign lesser progress detected: "\
|
||||
"Progress was: #{saved_node['status']}, attempted to "\
|
||||
"assign: #{node['progress']}. Skipping this node (id=#{node['uid']})")
|
||||
return
|
||||
end
|
||||
|
||||
# We need to update node here only if progress is greater, or status changed
|
||||
return if node.select{|k, v| saved_node[k] != v }.empty?
|
||||
end
|
||||
node
|
||||
end
|
||||
|
||||
end # ProvisiningProxyReporter
|
||||
|
||||
end # ProxyReporter
|
||||
end # Astute
|
|
@ -1,40 +0,0 @@
|
|||
# Copyright 2014 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
class Rsyslogd
|
||||
|
||||
def self.send_sighup(ctx, master_ip)
|
||||
timeout = Astute.config.ssh_retry_timeout
|
||||
shell = MClient.new(ctx, 'execute_shell_command', ['master'],
|
||||
check_result=true, timeout=timeout, retries=1)
|
||||
cmd = "ssh root@#{master_ip} 'pkill -HUP rsyslogd'"
|
||||
|
||||
begin
|
||||
result = shell.execute(:cmd => cmd).first.results
|
||||
|
||||
Astute.logger.info("#{ctx.task_id}: \
|
||||
stdout: #{result[:data][:stdout]} stderr: #{result[:data][:stderr]} \
|
||||
exit code: #{result[:data][:exit_code]}")
|
||||
rescue Timeout::Error
|
||||
msg = "Sending SIGHUP to rsyslogd is timed out."
|
||||
Astute.logger.error("#{ctx.task_id}: #{msg}")
|
||||
rescue => e
|
||||
msg = "Exception occured during sending SIGHUP to rsyslogd, message: #{e.message} \
|
||||
trace:\n#{e.backtrace.pretty_inspect}"
|
||||
Astute.logger.error("#{ctx.task_id}: #{msg}")
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
|
@ -1,23 +0,0 @@
|
|||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
|
||||
class Date
|
||||
def self.day_fraction_to_time(fr)
|
||||
ss, fr = fr.divmod(Rational(1, 86400))
|
||||
h, ss = ss.divmod(3600)
|
||||
min, s = ss.divmod(60)
|
||||
return h, min, s, fr
|
||||
end
|
||||
end
|
|
@ -1,79 +0,0 @@
|
|||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
require 'thread'
|
||||
|
||||
module Astute
|
||||
module Server
|
||||
|
||||
# Asynchronous singleton logger, which should be used
|
||||
# in event callbacks of event machine, it doesn't block
|
||||
# callbacks because writing a message to log takes some time.
|
||||
# Also synchronous logger, potentially could lead to deadlocks.
|
||||
# See:
|
||||
# https://bugs.launchpad.net/fuel/+bug/1453573
|
||||
# https://bugs.launchpad.net/fuel/+bug/1487397
|
||||
module AsyncLogger
|
||||
def self.start_up(logger=Logger.new(STDOUT))
|
||||
@queue ||= Queue.new
|
||||
@log = logger
|
||||
@thread = Thread.new { flush_messages }
|
||||
end
|
||||
|
||||
def self.shutdown
|
||||
@thread.kill
|
||||
end
|
||||
|
||||
def self.add(severity, msg=nil)
|
||||
return if @shutdown
|
||||
|
||||
@queue.push([severity, msg])
|
||||
end
|
||||
|
||||
def self.debug(msg=nil)
|
||||
add(Logger::Severity::DEBUG, msg)
|
||||
end
|
||||
|
||||
def self.info(msg=nil)
|
||||
add(Logger::Severity::INFO, msg)
|
||||
end
|
||||
|
||||
def self.warn(msg=nil)
|
||||
add(Logger::Severity::WARN, msg)
|
||||
end
|
||||
|
||||
def self.error(msg=nil)
|
||||
add(Logger::Severity::ERROR, msg)
|
||||
end
|
||||
|
||||
def self.fatal(msg=nil)
|
||||
add(Logger::Severity::FATAL, msg)
|
||||
end
|
||||
|
||||
def self.unknown(msg=nil)
|
||||
add(Logger::Severity::UNKNOWN, msg)
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def self.flush_messages
|
||||
loop do
|
||||
severity, msg = @queue.pop
|
||||
@log.add(severity, msg)
|
||||
end
|
||||
end
|
||||
|
||||
end
|
||||
end
|
||||
end
|
|
@ -1,332 +0,0 @@
|
|||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
require 'astute/server/reporter'
|
||||
|
||||
module Astute
|
||||
module Server
|
||||
|
||||
class Dispatcher
|
||||
def initialize(producer)
|
||||
@orchestrator = Astute::Orchestrator.new(log_parsing=true)
|
||||
@producer = producer
|
||||
@provisionLogParser = Astute::LogParser::ParseProvisionLogs.new
|
||||
end
|
||||
|
||||
def echo(args)
|
||||
Astute.logger.info('Running echo command')
|
||||
args
|
||||
end
|
||||
|
||||
#
|
||||
# Main worker actions
|
||||
#
|
||||
|
||||
def image_provision(data)
|
||||
provision(data)
|
||||
end
|
||||
|
||||
def provision(data)
|
||||
Astute.logger.debug("'provision' method called with data:\n"\
|
||||
"#{data.pretty_inspect}")
|
||||
|
||||
reporter = create_reporter(data)
|
||||
begin
|
||||
result = @orchestrator.provision(
|
||||
reporter,
|
||||
data['args']['task_uuid'],
|
||||
data['args']['provisioning_info']
|
||||
)
|
||||
rescue => e
|
||||
Astute.logger.error("Error running provisioning: #{e.message}, "\
|
||||
"trace: #{e.format_backtrace}")
|
||||
raise StopIteration
|
||||
end
|
||||
raise StopIteration if result && result['status'] == 'error'
|
||||
end
|
||||
|
||||
def granular_deploy(data)
|
||||
Astute.logger.debug("'granular_deploy' method called with data:\n"\
|
||||
"#{data.pretty_inspect}")
|
||||
|
||||
reporter = create_reporter(data)
|
||||
begin
|
||||
@orchestrator.granular_deploy(
|
||||
reporter,
|
||||
data['args']['task_uuid'],
|
||||
data['args']['deployment_info'],
|
||||
data['args']['pre_deployment'] || [],
|
||||
data['args']['post_deployment'] || []
|
||||
)
|
||||
reporter.report('status' => 'ready', 'progress' => 100)
|
||||
rescue Timeout::Error
|
||||
msg = "Timeout of deployment is exceeded."
|
||||
Astute.logger.error(msg)
|
||||
reporter.report('status' => 'error', 'error' => msg)
|
||||
|
||||
end
|
||||
end
|
||||
|
||||
def task_deploy(data)
|
||||
Astute.logger.debug("'task_deploy' method called with data:\n"\
|
||||
"#{data.pretty_inspect}")
|
||||
|
||||
Thread.current[:gracefully_stop] = false
|
||||
reporter = create_reporter(data)
|
||||
begin
|
||||
@orchestrator.task_deploy(
|
||||
reporter,
|
||||
data['args']['task_uuid'],
|
||||
{
|
||||
:tasks_graph => data['args'].fetch('tasks_graph', {}),
|
||||
:tasks_directory => data['args'].fetch('tasks_directory', {}),
|
||||
:tasks_metadata => data['args'].fetch('tasks_metadata', {}),
|
||||
:dry_run => data['args'].fetch('dry_run', false),
|
||||
:noop_run => data['args'].fetch('noop_run', false),
|
||||
:debug => data['args'].fetch('debug', false)
|
||||
}
|
||||
)
|
||||
rescue Timeout::Error
|
||||
msg = "Timeout of deployment is exceeded."
|
||||
Astute.logger.error(msg)
|
||||
reporter.report('status' => 'error', 'error' => msg)
|
||||
end
|
||||
end
|
||||
|
||||
def verify_networks(data)
|
||||
data.fetch('subtasks', []).each do |subtask|
|
||||
if self.respond_to?(subtask['method'])
|
||||
self.send(subtask['method'], subtask)
|
||||
else
|
||||
Astute.logger.warn("No method for #{subtask}")
|
||||
end
|
||||
end
|
||||
reporter = create_reporter(data)
|
||||
result = @orchestrator.verify_networks(
|
||||
reporter,
|
||||
data['args']['task_uuid'],
|
||||
data['args']['nodes']
|
||||
)
|
||||
report_result(result, reporter)
|
||||
end
|
||||
|
||||
def check_dhcp(data)
|
||||
reporter = create_reporter(data)
|
||||
result = @orchestrator.check_dhcp(
|
||||
reporter,
|
||||
data['args']['task_uuid'],
|
||||
data['args']['nodes']
|
||||
)
|
||||
report_result(result, reporter)
|
||||
end
|
||||
|
||||
def multicast_verification(data)
|
||||
reporter = create_reporter(data)
|
||||
result = @orchestrator.multicast_verification(
|
||||
reporter,
|
||||
data['args']['task_uuid'],
|
||||
data['args']['nodes']
|
||||
)
|
||||
report_result(result, reporter)
|
||||
end
|
||||
|
||||
def check_repositories(data)
|
||||
reporter = create_reporter(data)
|
||||
result = @orchestrator.check_repositories(
|
||||
reporter,
|
||||
data['args']['task_uuid'],
|
||||
data['args']['nodes'],
|
||||
data['args']['urls']
|
||||
)
|
||||
report_result(result, reporter)
|
||||
end
|
||||
|
||||
def check_repositories_with_setup(data)
|
||||
reporter = create_reporter(data)
|
||||
result = @orchestrator.check_repositories_with_setup(
|
||||
reporter,
|
||||
data['args']['task_uuid'],
|
||||
data['args']['nodes']
|
||||
)
|
||||
report_result(result, reporter)
|
||||
end
|
||||
|
||||
def dump_environment(data)
|
||||
@orchestrator.dump_environment(
|
||||
create_reporter(data),
|
||||
data['args']['task_uuid'],
|
||||
data['args']['settings']
|
||||
)
|
||||
end
|
||||
|
||||
def remove_nodes(data, reset=false)
|
||||
task_uuid = data['args']['task_uuid']
|
||||
reporter = create_reporter(data)
|
||||
|
||||
result = if data['args']['nodes'].empty?
|
||||
Astute.logger.debug("#{task_uuid} Node list is empty")
|
||||
nil
|
||||
else
|
||||
@orchestrator.remove_nodes(
|
||||
reporter,
|
||||
task_uuid,
|
||||
data['args']['engine'],
|
||||
data['args']['nodes'],
|
||||
{
|
||||
:reboot => true,
|
||||
:check_ceph => data['args']['check_ceph'],
|
||||
:reset => reset
|
||||
}
|
||||
)
|
||||
end
|
||||
|
||||
report_result(result, reporter)
|
||||
end
|
||||
|
||||
def reset_environment(data)
|
||||
remove_nodes(data, reset=true)
|
||||
end
|
||||
|
||||
def execute_tasks(data)
|
||||
@orchestrator.execute_tasks(
|
||||
create_reporter(data),
|
||||
data['args']['task_uuid'],
|
||||
data['args']['tasks']
|
||||
)
|
||||
end
|
||||
|
||||
#
|
||||
# Service worker actions
|
||||
#
|
||||
|
||||
def stop_deploy_task(data, service_data)
|
||||
Astute.logger.debug("'stop_deploy_task' service method called with"\
|
||||
"data:\n#{data.pretty_inspect}")
|
||||
target_task_uuid = data['args']['stop_task_uuid']
|
||||
task_uuid = data['args']['task_uuid']
|
||||
|
||||
return unless task_in_queue?(target_task_uuid,
|
||||
service_data[:tasks_queue])
|
||||
|
||||
Astute.logger.debug("Cancel task #{target_task_uuid}. Start")
|
||||
if target_task_uuid == service_data[:tasks_queue].current_task_id
|
||||
reporter = create_reporter(data)
|
||||
result = stop_current_task(data, service_data, reporter)
|
||||
report_result(result, reporter)
|
||||
else
|
||||
replace_future_task(data, service_data)
|
||||
end
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def create_reporter(data)
|
||||
Astute::Server::Reporter.new(
|
||||
@producer,
|
||||
data['respond_to'],
|
||||
data['args']['task_uuid']
|
||||
)
|
||||
end
|
||||
|
||||
def task_in_queue?(task_uuid, tasks_queue)
|
||||
tasks_queue.task_in_queue?(task_uuid)
|
||||
end
|
||||
|
||||
def replace_future_task(data, service_data)
|
||||
target_task_uuid = data['args']['stop_task_uuid']
|
||||
task_uuid = data['args']['task_uuid']
|
||||
|
||||
new_task_data = data_for_rm_nodes(data)
|
||||
Astute.logger.debug("Replace running task #{target_task_uuid} to "\
|
||||
"new #{task_uuid} with data:\n"\
|
||||
"#{new_task_data.pretty_inspect}")
|
||||
service_data[:tasks_queue].replace_task(
|
||||
target_task_uuid,
|
||||
new_task_data
|
||||
)
|
||||
end
|
||||
|
||||
def stop_current_task(data, service_data, reporter)
|
||||
target_task_uuid = data['args']['stop_task_uuid']
|
||||
task_uuid = data['args']['task_uuid']
|
||||
nodes = data['args']['nodes']
|
||||
|
||||
result = if ['deploy', 'granular_deploy'].include? (
|
||||
service_data[:tasks_queue].current_task_method)
|
||||
kill_main_process(target_task_uuid, service_data)
|
||||
|
||||
@orchestrator.stop_puppet_deploy(reporter, task_uuid, nodes)
|
||||
@orchestrator.remove_nodes(
|
||||
reporter,
|
||||
task_uuid,
|
||||
data['args']['engine'],
|
||||
nodes
|
||||
)
|
||||
elsif ['task_deploy'].include? (
|
||||
service_data[:tasks_queue].current_task_method)
|
||||
gracefully_stop_main_process(target_task_uuid, service_data)
|
||||
wait_while_process_run(
|
||||
service_data[:main_work_thread],
|
||||
Astute.config.stop_timeout,
|
||||
target_task_uuid,
|
||||
service_data
|
||||
)
|
||||
else
|
||||
kill_main_process(target_task_uuid, service_data)
|
||||
@orchestrator.stop_provision(
|
||||
reporter,
|
||||
task_uuid,
|
||||
data['args']['engine'],
|
||||
nodes
|
||||
)
|
||||
end
|
||||
end
|
||||
|
||||
def kill_main_process(target_task_uuid, service_data)
|
||||
Astute.logger.info("Try to kill running task #{target_task_uuid}")
|
||||
service_data[:main_work_thread].kill
|
||||
end
|
||||
|
||||
def gracefully_stop_main_process(target_task_uuid, service_data)
|
||||
Astute.logger.info("Try to stop gracefully running " \
|
||||
"task #{target_task_uuid}")
|
||||
service_data[:main_work_thread][:gracefully_stop] = true
|
||||
end
|
||||
|
||||
def wait_while_process_run(process, timeout, target_task_uuid, service_data)
|
||||
Astute.logger.info("Wait until process will stop or exit " \
|
||||
"by timeout #{timeout}")
|
||||
Timeout::timeout(timeout) { process.join }
|
||||
{}
|
||||
rescue Timeout::Error => e
|
||||
msg = "Timeout (#{timeout} sec) was reached."
|
||||
Astute.logger.warn(msg)
|
||||
kill_main_process(target_task_uuid, service_data)
|
||||
{'status' => 'error', 'error' => msg}
|
||||
end
|
||||
|
||||
def data_for_rm_nodes(data)
|
||||
data['method'] = 'remove_nodes'
|
||||
data
|
||||
end
|
||||
|
||||
def report_result(result, reporter)
|
||||
result = {} unless result.instance_of?(Hash)
|
||||
status = {'status' => 'ready', 'progress' => 100}.merge(result)
|
||||
reporter.report(status)
|
||||
end
|
||||
end
|
||||
|
||||
end #Server
|
||||
end #Astute
|
|
@ -1,61 +0,0 @@
|
|||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
module Server
|
||||
class Producer
|
||||
def initialize(exchange)
|
||||
@exchange = exchange
|
||||
@publish_queue = Queue.new
|
||||
@publish_consumer = Thread.new do
|
||||
loop do
|
||||
msg = @publish_queue.pop
|
||||
publish_from_queue msg
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def publish_from_queue(message)
|
||||
Astute.logger.info "Casting message to Nailgun:\n"\
|
||||
"#{message[:message].pretty_inspect}"
|
||||
@exchange.publish(message[:message].to_json, message[:options])
|
||||
rescue => e
|
||||
Astute.logger.error "Error publishing message: #{e.message}"
|
||||
end
|
||||
|
||||
def publish(message, options={})
|
||||
default_options = {
|
||||
:routing_key => Astute.config.broker_publisher_queue,
|
||||
:content_type => 'application/json'
|
||||
}
|
||||
|
||||
# Status message manage task status in Nailgun. If we miss some of them,
|
||||
# user need manually delete them or change it status using DB.
|
||||
# Persistent option tell RabbitMQ to save message in case of
|
||||
# unexpected/expected restart.
|
||||
if message.respond_to?(:keys) && message.keys.map(&:to_s).include?('status')
|
||||
default_options.merge!({:persistent => true})
|
||||
end
|
||||
|
||||
options = default_options.merge(options)
|
||||
@publish_queue << {:message => message, :options => options}
|
||||
end
|
||||
|
||||
def stop
|
||||
@publish_consumer.kill
|
||||
end
|
||||
|
||||
end # Producer
|
||||
end #Server
|
||||
end #Astute
|
|
@ -1,33 +0,0 @@
|
|||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
module Server
|
||||
|
||||
class Reporter
|
||||
def initialize(producer, method, task_uuid)
|
||||
@producer = producer
|
||||
@method = method
|
||||
@task_uuid = task_uuid
|
||||
end
|
||||
|
||||
def report(msg)
|
||||
msg_with_task = {'task_uuid' => @task_uuid}.merge(msg)
|
||||
message = {'method' => @method, 'args' => msg_with_task}
|
||||
@producer.publish(message)
|
||||
end
|
||||
end
|
||||
|
||||
end #Server
|
||||
end #Astute
|
|
@ -1,253 +0,0 @@
|
|||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
require 'json'
|
||||
require 'securerandom'
|
||||
require 'astute/server/task_queue'
|
||||
require 'zlib'
|
||||
|
||||
module Astute
|
||||
module Server
|
||||
|
||||
class Server
|
||||
def initialize(channels_and_exchanges, delegate, producer)
|
||||
@channel = channels_and_exchanges[:channel]
|
||||
@exchange = channels_and_exchanges[:exchange]
|
||||
@delegate = delegate
|
||||
@producer = producer
|
||||
@service_channel = channels_and_exchanges[:service_channel]
|
||||
@service_exchange = channels_and_exchanges[:service_exchange]
|
||||
# NOTE(eli): Generate unique name for service queue
|
||||
# See bug: https://bugs.launchpad.net/fuel/+bug/1485895
|
||||
@service_queue_name = "naily_service_#{SecureRandom.uuid}"
|
||||
@watch_thread = nil
|
||||
end
|
||||
|
||||
def run
|
||||
@queue = @channel.queue(
|
||||
Astute.config.broker_queue,
|
||||
:durable => true
|
||||
)
|
||||
@queue.bind(@exchange)
|
||||
|
||||
@service_queue = @service_channel.queue(
|
||||
@service_queue_name,
|
||||
:exclusive => true,
|
||||
:auto_delete => true
|
||||
)
|
||||
@service_queue.bind(@service_exchange)
|
||||
|
||||
@main_work_thread = nil
|
||||
@tasks_queue = TaskQueue.new
|
||||
|
||||
register_callbacks
|
||||
|
||||
run_infinite_loop
|
||||
@watch_thread.join
|
||||
end
|
||||
|
||||
def stop
|
||||
@watch_thread.wakeup
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def run_infinite_loop
|
||||
@watch_thread = Thread.new do
|
||||
Thread.stop
|
||||
Astute.logger.debug "Stop main thread"
|
||||
end
|
||||
end
|
||||
|
||||
def register_callbacks
|
||||
main_worker
|
||||
service_worker
|
||||
end
|
||||
|
||||
def main_worker
|
||||
@queue.subscribe(:manual_ack => true) do |delivery_info, properties, payload|
|
||||
if @main_work_thread.nil? || !@main_work_thread.alive?
|
||||
@channel.acknowledge(delivery_info.delivery_tag, false)
|
||||
perform_main_job(payload, properties)
|
||||
else
|
||||
Astute.logger.debug "Requeue message because worker is busy"
|
||||
# Avoid throttle by consume/reject cycle
|
||||
# if only one worker is running
|
||||
@channel.reject(delivery_info.delivery_tag, true)
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def service_worker
|
||||
@service_queue.subscribe do |_delivery_info, properties, payload|
|
||||
perform_service_job(payload, properties)
|
||||
end
|
||||
end
|
||||
|
||||
def perform_main_job(payload, properties)
|
||||
@main_work_thread = Thread.new do
|
||||
data = parse_data(payload, properties)
|
||||
Astute.logger.debug("Process message from worker queue:\n"\
|
||||
"#{data.pretty_inspect}")
|
||||
@tasks_queue = Astute::Server::TaskQueue.new
|
||||
|
||||
@tasks_queue.add_task(data)
|
||||
dispatch(@tasks_queue)
|
||||
|
||||
# Clean up tasks queue to prevent wrong service job work flow for
|
||||
# already finished tasks
|
||||
@tasks_queue = TaskQueue.new
|
||||
end
|
||||
end
|
||||
|
||||
def perform_service_job(payload, properties)
|
||||
Thread.new do
|
||||
service_data = {
|
||||
:main_work_thread => @main_work_thread,
|
||||
:tasks_queue => @tasks_queue
|
||||
}
|
||||
data = parse_data(payload, properties)
|
||||
Astute.logger.debug("Process message from service queue:\n"\
|
||||
"#{data.pretty_inspect}")
|
||||
dispatch(data, service_data)
|
||||
end
|
||||
end
|
||||
|
||||
def dispatch(data, service_data=nil)
|
||||
data.each_with_index do |message, i|
|
||||
begin
|
||||
send_running_task_status(message)
|
||||
dispatch_message message, service_data
|
||||
rescue StopIteration
|
||||
Astute.logger.debug "Dispatching aborted by #{message['method']}"
|
||||
abort_messages data[(i + 1)..-1]
|
||||
break
|
||||
rescue => ex
|
||||
Astute.logger.error "Error running RPC method "\
|
||||
"#{message['method']}: #{ex.message}, "\
|
||||
"trace: #{ex.format_backtrace}"
|
||||
return_results message, {
|
||||
'status' => 'error',
|
||||
'error' => "Method #{message['method']}. #{ex.message}.\n" \
|
||||
"Inspect Astute logs for the details"
|
||||
}
|
||||
break
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def dispatch_message(data, service_data=nil)
|
||||
if Astute.config.fake_dispatch
|
||||
Astute.logger.debug "Fake dispatch"
|
||||
return
|
||||
end
|
||||
|
||||
unless @delegate.respond_to?(data['method'])
|
||||
Astute.logger.error "Unsupported RPC call '#{data['method']}'"
|
||||
return_results data, {
|
||||
'status' => 'error',
|
||||
'error' => "Unsupported method '#{data['method']}' called."
|
||||
}
|
||||
return
|
||||
end
|
||||
|
||||
if service_data.nil?
|
||||
Astute.logger.debug "Main worker task id is "\
|
||||
"#{@tasks_queue.current_task_id}"
|
||||
end
|
||||
|
||||
Astute.logger.info "Processing RPC call '#{data['method']}'"
|
||||
if !service_data
|
||||
@delegate.send(data['method'], data)
|
||||
else
|
||||
@delegate.send(data['method'], data, service_data)
|
||||
end
|
||||
end
|
||||
|
||||
def send_running_task_status(message)
|
||||
return_results(message, {'status' => 'running'})
|
||||
end
|
||||
|
||||
def return_results(message, results={})
|
||||
if results.is_a?(Hash) && message['respond_to']
|
||||
reporter = Astute::Server::Reporter.new(
|
||||
@producer,
|
||||
message['respond_to'],
|
||||
message['args']['task_uuid']
|
||||
)
|
||||
reporter.report results
|
||||
end
|
||||
end
|
||||
|
||||
def parse_data(data, properties)
|
||||
data = unzip_message(data, properties) if zip?(properties)
|
||||
messages = begin
|
||||
JSON.load(data)
|
||||
rescue => e
|
||||
Astute.logger.error "Error deserializing payload: #{e.message},"\
|
||||
" trace:\n#{e.backtrace.pretty_inspect}"
|
||||
nil
|
||||
end
|
||||
messages.is_a?(Array) ? messages : [messages]
|
||||
end
|
||||
|
||||
def unzip_message(data, properties)
|
||||
Zlib::Inflate.inflate(data)
|
||||
rescue => e
|
||||
msg = "Gzip failure with error #{e.message} in\n"\
|
||||
"#{e.backtrace.pretty_inspect} with properties\n"\
|
||||
"#{properties.pretty_inspect} on data\n#{data.pretty_inspect}"
|
||||
Astute.logger.error(msg)
|
||||
raise e, msg
|
||||
end
|
||||
|
||||
def zip?(properties)
|
||||
properties[:headers]['compression'] == "application/x-gzip"
|
||||
end
|
||||
|
||||
def abort_messages(messages)
|
||||
return unless messages && messages.size > 0
|
||||
messages.each do |message|
|
||||
begin
|
||||
Astute.logger.debug "Aborting '#{message['method']}'"
|
||||
err_msg = {
|
||||
'status' => 'error',
|
||||
'error' => 'Task aborted',
|
||||
'progress' => 100
|
||||
}
|
||||
|
||||
if message['args']['nodes'].instance_of?(Array)
|
||||
err_nodes = message['args']['nodes'].map do |node|
|
||||
{
|
||||
'uid' => node['uid'],
|
||||
'status' => 'error',
|
||||
'error_type' => 'provision',
|
||||
'progress' => 0
|
||||
}
|
||||
end
|
||||
|
||||
err_msg.merge!('nodes' => err_nodes)
|
||||
end
|
||||
|
||||
return_results(message, err_msg)
|
||||
rescue => ex
|
||||
Astute.logger.debug "Failed to abort '#{message['method']}':\n"\
|
||||
"#{ex.pretty_inspect}"
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
end #Server
|
||||
end #Astute
|
|
@ -1,89 +0,0 @@
|
|||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
require 'thread'
|
||||
|
||||
module Astute
|
||||
module Server
|
||||
|
||||
class TaskQueue
|
||||
include Enumerable
|
||||
|
||||
attr_reader :current_task_id
|
||||
attr_reader :current_task_method
|
||||
|
||||
def initialize
|
||||
@queue = []
|
||||
@semaphore = Mutex.new
|
||||
@current_task_id = nil
|
||||
@current_task_method = nil
|
||||
end
|
||||
|
||||
def add_task(data)
|
||||
@semaphore.synchronize { data.compact.each { |t| @queue << t } }
|
||||
end
|
||||
|
||||
def replace_task(replacing_task_id, new_task_data)
|
||||
@semaphore.synchronize do
|
||||
@queue.map! { |x| find_task_id(x) == replacing_task_id ? new_task_data : x }.flatten!
|
||||
end
|
||||
end
|
||||
|
||||
def remove_task(replacing_task_id)
|
||||
replace_task(replacing_task_id, nil)
|
||||
end
|
||||
|
||||
def clear_queue
|
||||
@semaphore.synchronize { @queue.map! { |x| nil } }
|
||||
end
|
||||
|
||||
def task_in_queue?(task_id)
|
||||
@semaphore.synchronize { @queue.find { |t| find_task_id(t) == task_id } }
|
||||
end
|
||||
|
||||
def each(&block)
|
||||
@queue.each do |task|
|
||||
@semaphore.synchronize do
|
||||
next if task.nil?
|
||||
@current_task_id = find_task_id(task)
|
||||
@current_task_method = find_task_method(task)
|
||||
end
|
||||
|
||||
if block_given?
|
||||
block.call task
|
||||
else
|
||||
yield task
|
||||
end
|
||||
end
|
||||
ensure
|
||||
@semaphore.synchronize do
|
||||
@current_task_id = nil
|
||||
@current_task_method = nil
|
||||
end
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def find_task_id(data)
|
||||
data && data['args'] && data['args']['task_uuid'] ? data['args']['task_uuid'] : nil
|
||||
end
|
||||
|
||||
def find_task_method(data)
|
||||
data && data['method']
|
||||
end
|
||||
|
||||
end
|
||||
|
||||
end #Server
|
||||
end #Astute
|
|
@ -1,182 +0,0 @@
|
|||
# Copyright 2013 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
require 'raemon'
|
||||
require 'net/http'
|
||||
require 'bunny'
|
||||
|
||||
module Astute
|
||||
module Server
|
||||
|
||||
class Worker
|
||||
include Raemon::Worker
|
||||
|
||||
DELAY_SEC = 5
|
||||
|
||||
def start
|
||||
super
|
||||
start_heartbeat
|
||||
Astute::Server::AsyncLogger.start_up(Astute.logger)
|
||||
Astute.logger = Astute::Server::AsyncLogger
|
||||
end
|
||||
|
||||
def stop
|
||||
super
|
||||
@connection.stop if defined?(@connection) && @connection.present?
|
||||
@producer.stop if defined?(@producer) && @producer.present?
|
||||
@server.stop if defined?(@server) && @server.present?
|
||||
Astute::Server::AsyncLogger.shutdown
|
||||
end
|
||||
|
||||
def run
|
||||
Astute.logger.info "Worker initialization"
|
||||
run_server
|
||||
rescue Bunny::TCPConnectionFailed => e
|
||||
Astute.logger.warn "TCP connection to AMQP failed: #{e.message}. "\
|
||||
"Retry #{DELAY_SEC} sec later..."
|
||||
sleep DELAY_SEC
|
||||
retry
|
||||
rescue Bunny::PossibleAuthenticationFailureError => e
|
||||
Astute.logger.warn "If problem repeated more than 5 minutes, "\
|
||||
"please check "\
|
||||
"authentication parameters. #{e.message}. "\
|
||||
"Retry #{DELAY_SEC} sec later..."
|
||||
sleep DELAY_SEC
|
||||
retry
|
||||
rescue => e
|
||||
Astute.logger.error "Exception during worker initialization:"\
|
||||
" #{e.message}, trace: #{e.format_backtrace}"
|
||||
Astute.logger.warn "Retry #{DELAY_SEC} sec later..."
|
||||
sleep DELAY_SEC
|
||||
retry
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def start_heartbeat
|
||||
@heartbeat ||= Thread.new do
|
||||
sleep 30
|
||||
heartbeat!
|
||||
end
|
||||
end
|
||||
|
||||
def run_server
|
||||
@connection = Bunny.new(connection_options)
|
||||
@connection.start
|
||||
channels_and_exchanges = declare_channels_and_exchanges(@connection)
|
||||
|
||||
@producer = Astute::Server::Producer.new(
|
||||
channels_and_exchanges[:report_exchange]
|
||||
)
|
||||
delegate = Astute::Server::Dispatcher.new(@producer)
|
||||
@server = Astute::Server::Server.new(
|
||||
channels_and_exchanges,
|
||||
delegate,
|
||||
@producer
|
||||
)
|
||||
|
||||
@server.run
|
||||
end
|
||||
|
||||
def declare_channels_and_exchanges(connection)
|
||||
# WARN: Bunny::Channel are designed to assume they are
|
||||
# not shared between threads.
|
||||
channel = @connection.create_channel
|
||||
exchange = channel.topic(
|
||||
Astute.config.broker_exchange,
|
||||
:durable => true
|
||||
)
|
||||
|
||||
report_channel = @connection.create_channel
|
||||
report_exchange = report_channel.topic(
|
||||
Astute.config.broker_exchange,
|
||||
:durable => true
|
||||
)
|
||||
|
||||
service_channel = @connection.create_channel
|
||||
service_channel.prefetch(0)
|
||||
|
||||
service_exchange = service_channel.fanout(
|
||||
Astute.config.broker_service_exchange,
|
||||
:auto_delete => true
|
||||
)
|
||||
|
||||
return {
|
||||
:exchange => exchange,
|
||||
:service_exchange => service_exchange,
|
||||
:channel => channel,
|
||||
:service_channel => service_channel,
|
||||
:report_channel => report_channel,
|
||||
:report_exchange => report_exchange
|
||||
}
|
||||
rescue Bunny::PreconditionFailed => e
|
||||
Astute.logger.warn "Try to remove problem exchanges and queues"
|
||||
if connection.queue_exists? Astute.config.broker_queue
|
||||
channel.queue_delete Astute.config.broker_queue
|
||||
end
|
||||
if connection.queue_exists? Astute.config.broker_publisher_queue
|
||||
channel.queue_delete Astute.config.broker_publisher_queue
|
||||
end
|
||||
|
||||
cleanup_rabbitmq_stuff
|
||||
raise e
|
||||
end
|
||||
|
||||
def connection_options
|
||||
{
|
||||
:host => Astute.config.broker_host,
|
||||
:port => Astute.config.broker_port,
|
||||
:user => Astute.config.broker_username,
|
||||
:pass => Astute.config.broker_password,
|
||||
:heartbeat => :server
|
||||
}.reject{|k, v| v.nil? }
|
||||
end
|
||||
|
||||
def cleanup_rabbitmq_stuff
|
||||
Astute.logger.warn "Try to remove problem exchanges and queues"
|
||||
|
||||
|
||||
[Astute.config.broker_exchange,
|
||||
Astute.config.broker_service_exchange].each do |exchange|
|
||||
rest_delete("/api/exchanges/%2F/#{exchange}")
|
||||
end
|
||||
end
|
||||
|
||||
def rest_delete(url)
|
||||
http = Net::HTTP.new(
|
||||
Astute.config.broker_host,
|
||||
Astute.config.broker_rest_api_port
|
||||
)
|
||||
request = Net::HTTP::Delete.new(url)
|
||||
request.basic_auth(
|
||||
Astute.config.broker_username,
|
||||
Astute.config.broker_password
|
||||
)
|
||||
|
||||
response = http.request(request)
|
||||
|
||||
case response.code.to_i
|
||||
when 204 then Astute.logger.debug "Successfully delete object at #{url}"
|
||||
when 404 then
|
||||
else
|
||||
Astute.logger.error "Failed to perform delete request. Debug"\
|
||||
" information: http code: #{response.code},"\
|
||||
" message: #{response.message},"\
|
||||
" body #{response.body}"
|
||||
end
|
||||
end
|
||||
|
||||
end # Worker
|
||||
end #Server
|
||||
end #Astute
|
|
@ -1,195 +0,0 @@
|
|||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
class Task
|
||||
|
||||
ALLOWED_STATUSES = [:successful, :failed, :running, :pending, :skipped]
|
||||
attr_reader :task, :ctx
|
||||
def initialize(task, context)
|
||||
# WARNING: this code expect that only one node will be send
|
||||
# on one hook.
|
||||
@task = task
|
||||
@status = :pending
|
||||
@ctx = context
|
||||
@time_start = Time.now.to_i
|
||||
post_initialize(task, context)
|
||||
end
|
||||
|
||||
# Run current task on node, specified in task
|
||||
def run
|
||||
validation
|
||||
setup_default
|
||||
running!
|
||||
process
|
||||
rescue => e
|
||||
Astute.logger.warn("Fail to run task #{task['type']} #{task_name}" \
|
||||
" with error #{e.message} trace: #{e.format_backtrace}")
|
||||
failed!
|
||||
end
|
||||
|
||||
# Polls the status of the task
|
||||
def status
|
||||
calculate_status unless finished?
|
||||
@status
|
||||
rescue => e
|
||||
Astute.logger.warn("Fail to detect status of the task #{task['type']}" \
|
||||
" #{task_name} with error #{e.message} trace: #{e.format_backtrace}")
|
||||
failed!
|
||||
end
|
||||
|
||||
def status=(value)
|
||||
value = value.to_sym
|
||||
unless ALLOWED_STATUSES.include?(value)
|
||||
raise AstuteError::InvalidArgument,
|
||||
"#{self}: Invalid task status: #{value}"
|
||||
end
|
||||
@status = value
|
||||
end
|
||||
|
||||
# Run current task on node, specified in task, using sync mode
|
||||
def sync_run
|
||||
run
|
||||
loop do
|
||||
sleep Astute.config.task_poll_delay
|
||||
status
|
||||
break if finished?
|
||||
end
|
||||
|
||||
successful?
|
||||
end
|
||||
|
||||
# Show additional info about tasks: last run summary, sdtout etc
|
||||
def summary
|
||||
{}
|
||||
end
|
||||
|
||||
def finished?
|
||||
[:successful, :failed, :skipped].include? @status
|
||||
end
|
||||
|
||||
def successful?
|
||||
@status == :successful
|
||||
end
|
||||
|
||||
def pending?
|
||||
@status == :pending
|
||||
end
|
||||
|
||||
def skipped?
|
||||
@status == :skipped
|
||||
end
|
||||
|
||||
def running?
|
||||
@status == :running
|
||||
end
|
||||
|
||||
def failed?
|
||||
@status == :failed
|
||||
end
|
||||
|
||||
def post_initialize(task, context)
|
||||
nil
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
# Run current task on node, specified in task
|
||||
# should be fast and async and do not raise exceptions
|
||||
# @abstract Should be implemented in a subclass
|
||||
def process
|
||||
raise NotImplementedError
|
||||
end
|
||||
|
||||
# Polls the status of the task
|
||||
# should update the task status and do not raise exceptions
|
||||
# @abstract Should be implemented in a subclass
|
||||
def calculate_status
|
||||
raise NotImplementedError
|
||||
end
|
||||
|
||||
def validate_presence(data, key)
|
||||
raise TaskValidationError,
|
||||
"Missing a required parameter #{key}" unless data[key].present?
|
||||
end
|
||||
|
||||
# Pre validation of the task
|
||||
# should check task and raise error if something went wrong
|
||||
# @raise [TaskValidationError] if the object is not a task or has missing fields
|
||||
def validation
|
||||
|
||||
end
|
||||
|
||||
# Setup default value for hook
|
||||
# should not raise any exception
|
||||
def setup_default
|
||||
|
||||
end
|
||||
|
||||
# Run short shell commands
|
||||
# should use only in case of short run command
|
||||
# In other case please use shell task
|
||||
# Synchronous (blocking) call
|
||||
def run_shell_without_check(node_uid, cmd, timeout=2)
|
||||
ShellMClient.new(@ctx, node_uid).run_without_check(cmd, timeout)
|
||||
end
|
||||
|
||||
# Create file with content on selected node
|
||||
# should use only for small file
|
||||
# In other case please use separate thread or
|
||||
# use upload file task.
|
||||
# Synchronous (blocking) call
|
||||
def upload_file(node_uid, mco_params)
|
||||
UploadFileMClient.new(@ctx, node_uid).upload_without_check(mco_params)
|
||||
end
|
||||
|
||||
# Create file with content on selected node
|
||||
# should use only for small file
|
||||
# Synchronous (blocking) call
|
||||
def upload_file_with_check(node_uid, mco_params)
|
||||
UploadFileMClient.new(@ctx, node_uid).upload_with_check(mco_params)
|
||||
end
|
||||
|
||||
def failed!
|
||||
self.status = :failed
|
||||
time_summary
|
||||
end
|
||||
|
||||
def running!
|
||||
self.status = :running
|
||||
end
|
||||
|
||||
def succeed!
|
||||
self.status = :successful
|
||||
time_summary
|
||||
end
|
||||
|
||||
def skipped!
|
||||
self.status = :skipped
|
||||
time_summary
|
||||
end
|
||||
|
||||
def task_name
|
||||
task['id'] || task['diagnostic_name']
|
||||
end
|
||||
|
||||
def time_summary
|
||||
amount_time = (Time.now.to_i - @time_start).to_i
|
||||
wasted_time = Time.at(amount_time).utc.strftime("%H:%M:%S")
|
||||
Astute.logger.debug("Task time summary: #{task_name} with status" \
|
||||
" #{@status.to_s} on node #{task['node_id']} took #{wasted_time}")
|
||||
end
|
||||
|
||||
end
|
||||
end
|
|
@ -1,36 +0,0 @@
|
|||
# Copyright 2016 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
require 'fuel_deployment'
|
||||
|
||||
module Astute
|
||||
class TaskCluster < Deployment::Cluster
|
||||
attr_accessor :noop_run, :debug_run
|
||||
|
||||
def initialize(id=nil)
|
||||
super
|
||||
@node_statuses_transitions = {}
|
||||
end
|
||||
|
||||
attr_accessor :node_statuses_transitions
|
||||
|
||||
def hook_post_gracefully_stop(*args)
|
||||
report_new_node_status(args[0])
|
||||
end
|
||||
|
||||
def report_new_node_status(node)
|
||||
node.report_node_status
|
||||
end
|
||||
|
||||
end
|
||||
end
|
|
@ -1,380 +0,0 @@
|
|||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
require_relative '../fuel_deployment'
|
||||
|
||||
module Astute
|
||||
class TaskDeployment
|
||||
|
||||
|
||||
#TODO(vsharshov): remove this default after adding support of node
|
||||
# status transition to Nailgun
|
||||
NODE_STATUSES_TRANSITIONS = {
|
||||
'successful' => {'status' => 'ready'},
|
||||
'stopped' => {'status' => 'stopped'},
|
||||
'failed' => {'status' => 'error', 'error_type' => 'deploy'}
|
||||
}
|
||||
|
||||
attr_reader :ctx, :cluster_class, :node_class
|
||||
def initialize(context, cluster_class=TaskCluster, node_class=TaskNode)
|
||||
@ctx = context
|
||||
@cluster_class = cluster_class
|
||||
@node_class = node_class
|
||||
end
|
||||
|
||||
def self.munge_task(tasks_names, tasks_graph)
|
||||
result = Set.new
|
||||
tasks_names.each do |task|
|
||||
if task.is_a? Deployment::Task
|
||||
result.add task
|
||||
next
|
||||
end
|
||||
Astute.logger.debug("munging task #{task}")
|
||||
parts = task.split('/')
|
||||
task_name = parts[0]
|
||||
task_range = parts[1]
|
||||
if task_range
|
||||
Astute.logger.debug("expanding task #{task} range to specific nodes #{task_range}")
|
||||
node_ids = expand_node_ids(task_range).flatten
|
||||
Astute.logger.debug("expanded task #{task} range to #{node_ids.to_a}")
|
||||
else
|
||||
Astute.logger.debug("expanding task #{task} range to all_nodes")
|
||||
node_ids = tasks_graph.each_node.collect {|node| node.uid}
|
||||
end
|
||||
exp_t = tasks_graph.each_task.select do |_task|
|
||||
#Astute.logger.debug("task node id comparison is #{_task.node in? node_ids}")
|
||||
rv = (_task.name == task_name and _task.node.uid.in? node_ids)
|
||||
rv
|
||||
end
|
||||
exp_t.each do |t|
|
||||
result.add t
|
||||
end
|
||||
end
|
||||
result
|
||||
end
|
||||
|
||||
def self.expand_node_ids(interval)
|
||||
interval.split(',').collect do |part|
|
||||
if part =~ /^(\d+)-(\d+)$/
|
||||
($1.to_i .. $2.to_i).to_a
|
||||
else
|
||||
part
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def self.munge_list_of_start_end(tasks_graph, subgraphs)
|
||||
subgraphs.each do |subgraph|
|
||||
subgraph['start'] ||= []
|
||||
subgraph['end'] ||= []
|
||||
Astute.logger.debug("munging start tasks #{subgraph['start'].to_a} ")
|
||||
subgraph['start'] = munge_task(subgraph['start'], tasks_graph) unless subgraph['start'].blank?
|
||||
Astute.logger.debug("munged start tasks to #{subgraph['start'].to_a}")
|
||||
Astute.logger.debug("munging end tasks #{subgraph['end'].to_a} ")
|
||||
subgraph['end'] = munge_task(subgraph['end'], tasks_graph) unless subgraph['end'].blank?
|
||||
Astute.logger.debug("munged end tasks to #{subgraph['end'].to_a} ")
|
||||
end
|
||||
end
|
||||
|
||||
def create_cluster(deployment_options={})
|
||||
tasks_graph = deployment_options.fetch(:tasks_graph, {})
|
||||
tasks_directory = deployment_options.fetch(:tasks_directory, {})
|
||||
tasks_metadata = deployment_options.fetch(:tasks_metadata, {})
|
||||
|
||||
raise DeploymentEngineError, 'Deployment graph was not provided!' if tasks_graph.blank?
|
||||
|
||||
support_virtual_node(tasks_graph)
|
||||
unzip_graph(tasks_graph, tasks_directory)
|
||||
|
||||
cluster = cluster_class.new
|
||||
cluster.node_concurrency.maximum = Astute.config.max_nodes_per_call
|
||||
cluster.stop_condition { Thread.current[:gracefully_stop] }
|
||||
|
||||
cluster.noop_run = deployment_options.fetch(:noop_run, false)
|
||||
cluster.debug_run = deployment_options.fetch(:debug, false)
|
||||
|
||||
cluster.node_statuses_transitions = tasks_metadata.fetch(
|
||||
'node_statuses_transitions',
|
||||
NODE_STATUSES_TRANSITIONS
|
||||
)
|
||||
|
||||
setup_fault_tolerance_behavior(
|
||||
tasks_metadata['fault_tolerance_groups'],
|
||||
cluster,
|
||||
tasks_graph.keys
|
||||
)
|
||||
critical_uids = critical_node_uids(cluster.fault_tolerance_groups)
|
||||
offline_uids = detect_offline_nodes(tasks_graph.keys)
|
||||
|
||||
fail_offline_nodes(
|
||||
:offline_uids => offline_uids,
|
||||
:critical_uids => critical_uids,
|
||||
:node_statuses_transitions => cluster.node_statuses_transitions
|
||||
)
|
||||
|
||||
tasks_graph.keys.each do |node_id|
|
||||
node = node_class.new(node_id, cluster)
|
||||
node.context = ctx
|
||||
node.set_critical if critical_uids.include?(node_id)
|
||||
node.set_as_sync_point if sync_point?(node_id)
|
||||
node.set_status_failed if offline_uids.include?(node_id)
|
||||
end
|
||||
|
||||
setup_fail_behavior(tasks_graph, cluster)
|
||||
setup_debug_behavior(tasks_graph, cluster)
|
||||
setup_tasks(tasks_graph, cluster)
|
||||
setup_task_depends(tasks_graph, cluster)
|
||||
setup_task_concurrency(tasks_graph, cluster)
|
||||
subgraphs = self.class.munge_list_of_start_end(cluster, tasks_metadata.fetch('subgraphs', []))
|
||||
cluster.subgraphs = subgraphs unless subgraphs.compact_blank.blank?
|
||||
Astute.logger.debug(cluster.subgraphs)
|
||||
cluster.setup_start_end unless cluster.subgraphs.blank?
|
||||
cluster
|
||||
end
|
||||
|
||||
def deploy(deployment_options={})
|
||||
cluster = create_cluster(deployment_options)
|
||||
dry_run = deployment_options.fetch(:dry_run, false)
|
||||
write_graph_to_file(cluster)
|
||||
result = if dry_run
|
||||
{:success => true}
|
||||
else
|
||||
run_result = cluster.run
|
||||
# imitate dry_run results for noop run after deployment
|
||||
cluster.noop_run ? {:success => true } : run_result
|
||||
end
|
||||
report_final_node_progress(cluster)
|
||||
report_deploy_result(result)
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def sync_point?(node_id)
|
||||
'virtual_sync_node' == node_id
|
||||
end
|
||||
|
||||
def unzip_graph(tasks_graph, tasks_directory)
|
||||
tasks_graph.each do |node_id, tasks|
|
||||
tasks.each do |task|
|
||||
task.merge!({'node_id' => node_id})
|
||||
.reverse_merge(tasks_directory.fetch(task['id'], {}))
|
||||
end
|
||||
end
|
||||
tasks_graph
|
||||
end
|
||||
|
||||
def setup_fault_tolerance_behavior(fault_tolerance_groups, cluster, nodes)
|
||||
fault_tolerance_groups = [] if fault_tolerance_groups.nil?
|
||||
|
||||
defined_nodes = fault_tolerance_groups.map { |g| g['node_ids'] }.flatten.uniq
|
||||
all_nodes = nodes.select{ |n| !sync_point?(n) }
|
||||
undefined_nodes = all_nodes - defined_nodes
|
||||
|
||||
fault_tolerance_groups << {
|
||||
'fault_tolerance' => 0,
|
||||
'name' => 'zero_tolerance_as_default_for_nodes',
|
||||
'node_ids' => undefined_nodes
|
||||
}
|
||||
|
||||
cluster.fault_tolerance_groups = fault_tolerance_groups
|
||||
end
|
||||
|
||||
def setup_fail_behavior(tasks_graph, cluster)
|
||||
return unless cluster.noop_run
|
||||
tasks_graph.each do |node_id, tasks|
|
||||
tasks.each do |task|
|
||||
task['fail_on_error'] = false
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def setup_debug_behavior(tasks_graph, cluster)
|
||||
return unless cluster.debug_run
|
||||
tasks_graph.each do |node_id, tasks|
|
||||
tasks.each do |task|
|
||||
if task['parameters'].present?
|
||||
task['parameters']['debug'] = true
|
||||
else
|
||||
task['parameters'] = { 'debug' => true }
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def setup_tasks(tasks_graph, cluster)
|
||||
tasks_graph.each do |node_id, tasks|
|
||||
tasks.each do |task|
|
||||
cluster[node_id].graph.create_task(task['id'], task)
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def setup_task_depends(tasks_graph, cluster)
|
||||
tasks_graph.each do |node_id, tasks|
|
||||
tasks.each do |task|
|
||||
task.fetch('requires', []).each do |d_t|
|
||||
cluster[node_id][task['id']].depends(
|
||||
cluster[d_t['node_id']][d_t['name']])
|
||||
end
|
||||
|
||||
task.fetch('required_for', []).each do |d_t|
|
||||
cluster[node_id][task['id']].depended_on(
|
||||
cluster[d_t['node_id']][d_t['name']])
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def setup_task_concurrency(tasks_graph, cluster)
|
||||
tasks_graph.each do |_node_id, tasks|
|
||||
tasks.each do |task|
|
||||
cluster.task_concurrency[task['id']].maximum = task_concurrency_value(task)
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def task_concurrency_value(task)
|
||||
strategy = task.fetch('parameters', {}).fetch('strategy', {})
|
||||
value = case strategy['type']
|
||||
when 'one_by_one' then 1
|
||||
when 'parallel' then strategy['amount'].to_i
|
||||
else 0
|
||||
end
|
||||
return value if value >= 0
|
||||
raise DeploymentEngineError, "Task concurrency expect only "\
|
||||
"non-negative integer, but got #{value}. Please check task #{task}"
|
||||
end
|
||||
|
||||
def report_deploy_result(result)
|
||||
if result[:success] && result.fetch(:failed_nodes, []).empty?
|
||||
ctx.report('status' => 'ready', 'progress' => 100)
|
||||
elsif result[:success] && result.fetch(:failed_nodes, []).present?
|
||||
ctx.report('status' => 'ready', 'progress' => 100)
|
||||
else
|
||||
ctx.report(
|
||||
'status' => 'error',
|
||||
'progress' => 100,
|
||||
'error' => result[:status]
|
||||
)
|
||||
end
|
||||
end
|
||||
|
||||
def write_graph_to_file(deployment)
|
||||
return unless Astute.config.enable_graph_file
|
||||
graph_file = File.join(
|
||||
Astute.config.graph_dot_dir,
|
||||
"graph-#{ctx.task_id}.dot"
|
||||
)
|
||||
File.open(graph_file, 'w') { |f| f.write(deployment.to_dot) }
|
||||
Astute.logger.info("Check graph into file #{graph_file}")
|
||||
end
|
||||
|
||||
# Astute use special virtual node for deployment tasks, because
|
||||
# any task must be connected to node. For task, which play
|
||||
# synchronization role, we create virtual_sync_node
|
||||
def support_virtual_node(tasks_graph)
|
||||
tasks_graph['virtual_sync_node'] = tasks_graph['null']
|
||||
tasks_graph.delete('null')
|
||||
|
||||
tasks_graph.each do |_node_id, tasks|
|
||||
tasks.each do |task|
|
||||
task.fetch('requires',[]).each do |d_t|
|
||||
d_t['node_id'] = 'virtual_sync_node' if d_t['node_id'].nil?
|
||||
end
|
||||
|
||||
task.fetch('required_for', []).each do |d_t|
|
||||
d_t['node_id'] = 'virtual_sync_node' if d_t['node_id'].nil?
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
tasks_graph
|
||||
end
|
||||
|
||||
def critical_node_uids(fault_tolerance_groups)
|
||||
return [] if fault_tolerance_groups.blank?
|
||||
critical_nodes = fault_tolerance_groups.inject([]) do |critical_uids, group|
|
||||
critical_uids += group['node_ids'] if group['fault_tolerance'].zero?
|
||||
critical_uids
|
||||
end
|
||||
Astute.logger.info "Critical node #{critical_nodes}" if critical_nodes.present?
|
||||
critical_nodes
|
||||
end
|
||||
|
||||
def fail_offline_nodes(args={})
|
||||
critical_uids = args.fetch(:critical_uids, [])
|
||||
offline_uids = args.fetch(:offline_uids, [])
|
||||
node_statuses_transitions = args.fetch(:node_statuses_transitions, {})
|
||||
|
||||
return if offline_uids.blank?
|
||||
|
||||
nodes = offline_uids.map do |uid|
|
||||
{'uid' => uid,
|
||||
'error_msg' => 'Node is not ready for deployment: '\
|
||||
'mcollective has not answered'
|
||||
}.merge(node_statuses_transitions.fetch('failed', {}))
|
||||
end
|
||||
|
||||
ctx.report_and_update_status(
|
||||
'nodes' => nodes,
|
||||
'error' => 'Node is not ready for deployment'
|
||||
)
|
||||
|
||||
missing_required = critical_uids & offline_uids
|
||||
if missing_required.present?
|
||||
error_message = "Critical nodes are not available for deployment: " \
|
||||
"#{missing_required}"
|
||||
raise Astute::DeploymentEngineError, error_message
|
||||
end
|
||||
|
||||
end
|
||||
|
||||
def detect_offline_nodes(uids)
|
||||
available_uids = []
|
||||
|
||||
uids.delete('master')
|
||||
uids.delete('virtual_sync_node')
|
||||
# In case of big amount of nodes we should do several calls to be sure
|
||||
# about node status
|
||||
if uids.present?
|
||||
Astute.config.mc_retries.times.each do
|
||||
systemtype = MClient.new(
|
||||
ctx,
|
||||
"systemtype",
|
||||
uids,
|
||||
_check_result=false,
|
||||
10
|
||||
)
|
||||
available_nodes = systemtype.get_type
|
||||
|
||||
available_uids += available_nodes.map { |node| node.results[:sender] }
|
||||
uids -= available_uids
|
||||
break if uids.empty?
|
||||
|
||||
sleep Astute.config.mc_retry_interval
|
||||
end
|
||||
end
|
||||
|
||||
Astute.logger.warn "Offline node #{uids}" if uids.present?
|
||||
uids
|
||||
end
|
||||
|
||||
def report_final_node_progress(cluster)
|
||||
node_report = cluster.nodes.inject([]) do |node_progress, node|
|
||||
node_progress += [{'uid' => node[0].to_s, 'progress' => 100}]
|
||||
end
|
||||
ctx.report('nodes' => node_report)
|
||||
end
|
||||
|
||||
end
|
||||
end
|
|
@ -1,51 +0,0 @@
|
|||
# Copyright 2014 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
|
||||
class TaskManager
|
||||
def initialize(nodes)
|
||||
@tasks = nodes.inject({}) do |h, n|
|
||||
h.merge({n['uid'] => n['tasks'].sort_by{ |f| f['priority'] }.each})
|
||||
end
|
||||
|
||||
@current_task = {}
|
||||
Astute.logger.info "The following tasks will be performed on nodes: " \
|
||||
"#{@tasks.map {|k, v| {k => v.to_a}}.to_yaml}"
|
||||
end
|
||||
|
||||
def current_task(node_id)
|
||||
@current_task[node_id]
|
||||
end
|
||||
|
||||
def next_task(node_id)
|
||||
@current_task[node_id] = @tasks[node_id].next
|
||||
rescue StopIteration
|
||||
@current_task[node_id] = nil
|
||||
delete_node(node_id)
|
||||
end
|
||||
|
||||
def delete_node(node_id)
|
||||
@tasks[node_id] = nil
|
||||
end
|
||||
|
||||
def task_in_queue?
|
||||
@tasks.select{ |_k,v| v }.present?
|
||||
end
|
||||
|
||||
def node_uids
|
||||
@tasks.select{ |_k,v| v }.keys
|
||||
end
|
||||
end
|
||||
end
|
|
@ -1,141 +0,0 @@
|
|||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
require 'fuel_deployment'
|
||||
|
||||
module Astute
|
||||
class TaskNode < Deployment::Node
|
||||
def context=(context)
|
||||
@ctx = context
|
||||
end
|
||||
|
||||
def run(inbox_task)
|
||||
self.task = inbox_task
|
||||
@task_engine = select_task_engine(task.data)
|
||||
@task_engine.run
|
||||
task.set_status_running
|
||||
set_status_busy
|
||||
report_node_status if report_running?(task.data)
|
||||
end
|
||||
|
||||
def poll
|
||||
return unless busy?
|
||||
|
||||
debug("Node #{uid}: task #{task.name}, task status #{task.status}")
|
||||
|
||||
# Please be informed that this code define special method
|
||||
# of Deployment::Node class. We use special method `task`
|
||||
# to manage task status, graph of tasks and nodes.
|
||||
task.status = setup_task_status
|
||||
if @task.running?
|
||||
@ctx.report({
|
||||
'nodes' => [{
|
||||
'uid' => uid,
|
||||
'deployment_graph_task_name' => task.name,
|
||||
'task_status' => task.status.to_s,
|
||||
}]
|
||||
})
|
||||
else
|
||||
info "Finished task #{task} #{"with status: #{task.status}" if task}"
|
||||
setup_node_status
|
||||
report_node_status
|
||||
end
|
||||
end
|
||||
|
||||
def report_node_status
|
||||
node_status = {
|
||||
'uid' => uid,
|
||||
'progress' => current_progress_bar,
|
||||
}
|
||||
node_status.merge!(node_report_status)
|
||||
|
||||
if task
|
||||
node_status.merge!(
|
||||
'deployment_graph_task_name' => task.name,
|
||||
'task_status' => task.status.to_s,
|
||||
'summary' => @task_engine.summary
|
||||
)
|
||||
node_status.merge!(
|
||||
'error_msg' => "Task #{task.name} failed on node #{name}"
|
||||
) if task.failed?
|
||||
end
|
||||
|
||||
@ctx.report('nodes' => [node_status])
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
# This method support special task behavior. If task failed
|
||||
# and we do not think that deployment should be stopped, Astute
|
||||
# will mark such task as skipped and do not report error
|
||||
def setup_task_status
|
||||
if !task.data.fetch('fail_on_error', true) && @task_engine.failed?
|
||||
Astute.logger.warn "Task #{task.name} failed, but marked as skipped "\
|
||||
"because of 'fail on error' behavior"
|
||||
return :skipped
|
||||
end
|
||||
@task_engine.status
|
||||
end
|
||||
|
||||
def setup_node_status
|
||||
if task
|
||||
set_status_failed && return if task.failed?
|
||||
set_status_skipped && return if task.dep_failed?
|
||||
end
|
||||
|
||||
set_status_online
|
||||
end
|
||||
|
||||
def current_progress_bar
|
||||
if tasks_total_count != 0
|
||||
100 * tasks_finished_count / tasks_total_count
|
||||
else
|
||||
100
|
||||
end
|
||||
end
|
||||
|
||||
def select_task_engine(data)
|
||||
noop_prefix = noop_run? && not_noop_type?(data) ? "Noop" : ""
|
||||
task_class_name = noop_prefix + data['type'].split('_').collect(&:capitalize).join
|
||||
Object.const_get('Astute::' + task_class_name).new(data, @ctx)
|
||||
rescue => e
|
||||
raise TaskValidationError, "Unknown task type '#{data['type']}'. Detailed: #{e.message}"
|
||||
end
|
||||
|
||||
def report_running?(data)
|
||||
!['noop', 'stage', 'skipped'].include?(data['type'])
|
||||
end
|
||||
|
||||
def noop_run?
|
||||
cluster.noop_run
|
||||
end
|
||||
|
||||
def node_report_status
|
||||
if !finished?
|
||||
{}
|
||||
elsif skipped?
|
||||
cluster.node_statuses_transitions.fetch('stopped', {})
|
||||
elsif successful?
|
||||
cluster.node_statuses_transitions.fetch('successful', {})
|
||||
else
|
||||
cluster.node_statuses_transitions.fetch('failed', {})
|
||||
end
|
||||
end
|
||||
|
||||
def not_noop_type?(data)
|
||||
!['noop', 'stage', 'skipped'].include?(data['type'])
|
||||
end
|
||||
|
||||
end
|
||||
end
|
|
@ -1,155 +0,0 @@
|
|||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
require 'digest/md5'
|
||||
|
||||
module Astute
|
||||
module ProxyReporter
|
||||
class TaskProxyReporter
|
||||
|
||||
INTEGRATED_STATES = ['error', 'stopped']
|
||||
|
||||
REPORT_REAL_TASK_STATE_MAP = {
|
||||
'running' => 'running',
|
||||
'successful' => 'ready',
|
||||
'failed' => 'error',
|
||||
'skipped' => 'skipped'
|
||||
}
|
||||
|
||||
REPORT_REAL_NODE_MAP = {
|
||||
'virtual_sync_node' => nil
|
||||
}
|
||||
|
||||
def initialize(up_reporter)
|
||||
@up_reporter = up_reporter
|
||||
@messages_cache = []
|
||||
end
|
||||
|
||||
def report(original_data)
|
||||
return if duplicate?(original_data)
|
||||
|
||||
data = original_data.deep_dup
|
||||
if data['nodes']
|
||||
nodes_to_report = get_nodes_to_report(data['nodes'])
|
||||
return if nodes_to_report.empty? # Let's report only if nodes updated
|
||||
|
||||
data['nodes'] = nodes_to_report
|
||||
end
|
||||
|
||||
@up_reporter.report(data)
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def get_nodes_to_report(nodes)
|
||||
nodes.map{ |node| node_validate(node) }.compact
|
||||
end
|
||||
|
||||
def node_validate(original_node)
|
||||
node = deep_copy(original_node)
|
||||
return unless node_should_include?(node)
|
||||
convert_node_name_to_original(node)
|
||||
return node unless are_fields_valid?(node)
|
||||
convert_task_status_to_status(node)
|
||||
normalization_progress(node)
|
||||
return node
|
||||
end
|
||||
|
||||
def are_fields_valid?(node)
|
||||
are_node_basic_fields_valid?(node) && are_task_basic_fields_valid?(node)
|
||||
end
|
||||
|
||||
def node_should_include?(node)
|
||||
is_num?(node['uid']) ||
|
||||
['master', 'virtual_sync_node'].include?(node['uid'])
|
||||
end
|
||||
|
||||
def valid_task_status?(status)
|
||||
REPORT_REAL_TASK_STATE_MAP.keys.include? status.to_s
|
||||
end
|
||||
|
||||
def integrated_status?(status)
|
||||
INTEGRATED_STATES.include? status.to_s
|
||||
end
|
||||
|
||||
# Validate of basic fields in message about node
|
||||
def are_node_basic_fields_valid?(node)
|
||||
err = []
|
||||
err << "Node uid is not provided" unless node['uid']
|
||||
|
||||
err.any? ? fail_validation(node, err) : true
|
||||
end
|
||||
|
||||
# Validate of basic fields in message about task
|
||||
def are_task_basic_fields_valid?(node)
|
||||
err = []
|
||||
|
||||
err << "Task status provided '#{node['task_status']}' is not supported" if
|
||||
!valid_task_status?(node['task_status'])
|
||||
err << "Task name is not provided" if node['deployment_graph_task_name'].blank?
|
||||
|
||||
err.any? ? fail_validation(node, err) : true
|
||||
end
|
||||
|
||||
|
||||
def convert_task_status_to_status(node)
|
||||
node['task_status'] = REPORT_REAL_TASK_STATE_MAP.fetch(node['task_status'])
|
||||
end
|
||||
|
||||
# Normalization of progress field: ensures that the scaling progress was
|
||||
# in range from 0 to 100 and has a value of 100 fot the integrated node
|
||||
# status
|
||||
def normalization_progress(node)
|
||||
if node['progress']
|
||||
node['progress'] = 100 if node['progress'] > 100
|
||||
node['progress'] = 0 if node['progress'] < 0
|
||||
else
|
||||
node['progress'] = 100 if integrated_status?(node['status'])
|
||||
end
|
||||
end
|
||||
|
||||
def fail_validation(node, err)
|
||||
msg = "Validation of node:\n#{node.pretty_inspect} for " \
|
||||
"report failed: #{err.join('; ')}"
|
||||
Astute.logger.warn(msg)
|
||||
false
|
||||
end
|
||||
|
||||
def convert_node_name_to_original(node)
|
||||
if REPORT_REAL_NODE_MAP.keys.include?(node['uid'])
|
||||
node['uid'] = REPORT_REAL_NODE_MAP.fetch(node['uid'])
|
||||
end
|
||||
end
|
||||
|
||||
def is_num?(str)
|
||||
Integer(str)
|
||||
rescue ArgumentError, TypeError
|
||||
false
|
||||
end
|
||||
|
||||
# Save message digest to protect server from
|
||||
# message flooding. Sure, because of Hash is complicated structure
|
||||
# which does not respect order and can be generate different strings
|
||||
# but we still catch most of possible duplicates.
|
||||
def duplicate?(data)
|
||||
msg_digest = Digest::MD5.hexdigest(data.to_s)
|
||||
return true if @messages_cache.include?(msg_digest)
|
||||
|
||||
@messages_cache << msg_digest
|
||||
return false
|
||||
end
|
||||
|
||||
end
|
||||
end
|
||||
end
|
|
@ -1,41 +0,0 @@
|
|||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
class CobblerSync < Task
|
||||
|
||||
def post_initialize(task, context)
|
||||
@work_thread = nil
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def process
|
||||
cobbler = CobblerManager.new(
|
||||
task['parameters']['provisioning_info']['engine'],
|
||||
ctx.reporter
|
||||
)
|
||||
@work_thread = Thread.new { cobbler.sync }
|
||||
end
|
||||
|
||||
def calculate_status
|
||||
@work_thread.join and succeed! unless @work_thread.alive?
|
||||
end
|
||||
|
||||
def validation
|
||||
validate_presence(task['parameters'], 'provisioning_info')
|
||||
end
|
||||
|
||||
end
|
||||
end
|
|
@ -1,58 +0,0 @@
|
|||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
class CopyFiles < Task
|
||||
|
||||
def post_initialize(task, context)
|
||||
@work_thread = nil
|
||||
@files_status = task['parameters']['files'].inject({}) do |f_s, n|
|
||||
f_s.merge({ n['src']+n['dst'] => :pending })
|
||||
end
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def process
|
||||
task['parameters']['files'].each do |file|
|
||||
if File.file?(file['src']) && File.readable?(file['src'])
|
||||
parameters = {
|
||||
'content' => File.binread(file['src']),
|
||||
'path' => file['dst'],
|
||||
'permissions' => file['permissions'] || task['parameters']['permissions'],
|
||||
'dir_permissions' => file['dir_permissions'] || task['parameters']['dir_permissions'],
|
||||
}
|
||||
@files_status[file['src']+file['dst']] =
|
||||
upload_file(task['node_id'], parameters)
|
||||
else
|
||||
@files_status[file['src']+file['dst']] = false
|
||||
end
|
||||
end # files
|
||||
end
|
||||
|
||||
def calculate_status
|
||||
if @files_status.values.all?{ |s| s != :pending }
|
||||
failed! if @files_status.values.include?(false)
|
||||
succeed! if @files_status.values.all?{ |s| s == true }
|
||||
return
|
||||
end
|
||||
end
|
||||
|
||||
def validation
|
||||
validate_presence(task, 'node_id')
|
||||
validate_presence(task['parameters'], 'files')
|
||||
end
|
||||
|
||||
end
|
||||
end
|
|
@ -1,53 +0,0 @@
|
|||
# Copyright 2016 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
class EraseNode < Task
|
||||
|
||||
def summary
|
||||
{'task_summary' => "Node #{task['node_id']} was erased without reboot"\
|
||||
" with result #{@status}"}
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def process
|
||||
erase_node
|
||||
end
|
||||
|
||||
def calculate_status
|
||||
succeed!
|
||||
end
|
||||
|
||||
def validation
|
||||
validate_presence(task, 'node_id')
|
||||
end
|
||||
|
||||
def erase_node
|
||||
remover = MClient.new(
|
||||
ctx,
|
||||
"erase_node",
|
||||
Array(task['node_id']),
|
||||
_check_result=false)
|
||||
response = remover.erase_node(:reboot => false)
|
||||
Astute.logger.debug "#{ctx.task_id}: Data received from node "\
|
||||
"#{task['node_id']} :\n#{response.pretty_inspect}"
|
||||
rescue Astute::MClientTimeout, Astute::MClientError => e
|
||||
Astute.logger.error("#{ctx.task_id}: #{task_name} mcollective " \
|
||||
"erase node command failed with error #{e.message}")
|
||||
failed!
|
||||
end
|
||||
|
||||
end
|
||||
end
|
|
@ -1,60 +0,0 @@
|
|||
# Copyright 2016 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
class MasterShell < Task
|
||||
|
||||
# Accept to run shell tasks using existing shell asynchronous
|
||||
# mechanism. It will run task on master node.
|
||||
|
||||
def post_initialize(task, context)
|
||||
@shell_task = nil
|
||||
end
|
||||
|
||||
def summary
|
||||
@shell_task.summary
|
||||
rescue
|
||||
{}
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def process
|
||||
@shell_task = Shell.new(
|
||||
generate_master_shell,
|
||||
ctx
|
||||
)
|
||||
@shell_task.run
|
||||
end
|
||||
|
||||
def calculate_status
|
||||
self.status = @shell_task.status
|
||||
end
|
||||
|
||||
def validation
|
||||
validate_presence(task['parameters'], 'cmd')
|
||||
end
|
||||
|
||||
def setup_default
|
||||
task['parameters']['timeout'] ||= Astute.config.shell_timeout
|
||||
task['parameters']['cwd'] ||= Astute.config.shell_cwd
|
||||
task['parameters']['retries'] ||= Astute.config.shell_retries
|
||||
task['parameters']['interval'] ||= Astute.config.shell_interval
|
||||
end
|
||||
|
||||
def generate_master_shell
|
||||
task.merge('node_id' => 'master')
|
||||
end
|
||||
end
|
||||
end
|
|
@ -1,89 +0,0 @@
|
|||
# Copyright 2016 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
class MoveToBootstrap < Task
|
||||
|
||||
def post_initialize(task, context)
|
||||
@work_thread = nil
|
||||
end
|
||||
|
||||
def summary
|
||||
{'task_summary' => "Node #{task['node_id']} was move to bootstrap with"\
|
||||
" result #{@status}"}
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def process
|
||||
cobbler = CobblerManager.new(
|
||||
task['parameters']['provisioning_info']['engine'],
|
||||
ctx.reporter
|
||||
)
|
||||
|
||||
@work_thread = Thread.new do
|
||||
is_exist = cobbler.existent_node?(task['parameters']['provisioning_info']['slave_name'])
|
||||
|
||||
# Change node type to prevent wrong node detection as provisioned
|
||||
# Also this type if node will not rebooted, Astute will be allowed
|
||||
# to try to reboot such nodes again
|
||||
change_nodes_type('reprovisioned') if is_exist
|
||||
bootstrap_profile = task['parameters']['provisioning_info']['profile'] ||
|
||||
Astute.config.bootstrap_profile
|
||||
cobbler.edit_node(task['parameters']['provisioning_info']['slave_name'],
|
||||
{'profile' => bootstrap_profile})
|
||||
cobbler.netboot_node(task['parameters']['provisioning_info']['slave_name'],
|
||||
true)
|
||||
|
||||
Reboot.new({'node_id' => task['node_id']}, ctx).sync_run if is_exist
|
||||
Rsyslogd.send_sighup(
|
||||
ctx,
|
||||
task['parameters']['provisioning_info']['engine']['master_ip']
|
||||
)
|
||||
|
||||
cobbler.remove_node(task['parameters']['provisioning_info']['slave_name'])
|
||||
# NOTE(kozhukalov): We try to find out if there are systems
|
||||
# in the Cobbler with the same MAC addresses. If so, Cobbler is going
|
||||
# to throw MAC address duplication error. We need to remove these
|
||||
# nodes.
|
||||
mac_duplicate_names = cobbler.node_mac_duplicate_names(task['parameters']['provisioning_info'])
|
||||
cobbler.remove_nodes(mac_duplicate_names.map {|n| {'slave_name' => n}})
|
||||
|
||||
cobbler.add_node(task['parameters']['provisioning_info'])
|
||||
end
|
||||
end
|
||||
|
||||
def calculate_status
|
||||
@work_thread.join and succeed! unless @work_thread.alive?
|
||||
end
|
||||
|
||||
def validation
|
||||
validate_presence(task['parameters'], 'provisioning_info')
|
||||
validate_presence(task, 'node_id')
|
||||
end
|
||||
|
||||
def change_nodes_type(type="image")
|
||||
run_shell_without_check(
|
||||
task['node_id'],
|
||||
"echo '#{type}' > /etc/nailgun_systemtype",
|
||||
_timeout=5
|
||||
)[:stdout]
|
||||
rescue Astute::MClientTimeout, Astute::MClientError => e
|
||||
Astute.logger.debug("#{ctx.task_id}: #{task_name} mcollective " \
|
||||
"change type command failed with error #{e.message}")
|
||||
nil
|
||||
end
|
||||
|
||||
end
|
||||
end
|
|
@ -1,29 +0,0 @@
|
|||
# Copyright 2015 Mirantis, Inc.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
module Astute
|
||||
class Noop < Task
|
||||
|
||||
private
|
||||
|
||||
def process
|
||||
|
||||
end
|
||||
|
||||
def calculate_status
|
||||
skipped!
|
||||
end
|
||||
|
||||
end
|
||||
end
|
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue