openstack
/
fuel-astute
Code Issues Proposed changes

Retire repository 

Fuel (from openstack namespace) and fuel-ccp (in x namespace)
repositories are unused and ready to retire.

This change removes all content from the repository and adds the usual
README file to point out that the repository is retired following the
process from
https://docs.openstack.org/infra/manual/drivers.html#retiring-a-project

See also
http://lists.openstack.org/pipermail/openstack-discuss/2019-December/011647.html

Depends-On: https://review.opendev.org/699362
Change-Id: If0c2011947c9eb63f59093812b5f9f95ce56a6f8
This commit is contained in:
Andreas Jaeger 2019-12-18 09:34:12 +01:00
parent 46d1908d60
commit 3e5483d1dc
230 changed files with 10 additions and 38355 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

23
.gitignore vendored
View File

@ -1,23 +0,0 @@
.idea
*.gem
# SimpleCov
coverage
coverage.data
# Need only on local machine for gem
Gemfile.lock
docs/_build
#Vim swap files
*.swp
# Local raemon copy
raemon/
*.svg
*.png
*.yaml
!examples/example_astute_config.yaml

3
.rspec
View File

@ -1,3 +0,0 @@
--color
-f d
#--profile

1
.ruby-version
View File

@ -1 +0,0 @@
2.1

3
Gemfile
View File

@ -1,3 +0,0 @@
source 'https://rubygems.org'
gem 'raemon', :git => 'https://github.com/pressly/raemon', :ref => 'b78eaae57c8e836b8018386dd96527b8d9971acc'
gemspec

176
LICENSE
View File

@ -1,176 +0,0 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.

67
MAINTAINERS
View File

@ -1,67 +0,0 @@
---
description:
For Fuel team structure and contribution policy, see [1].
This is repository level MAINTAINERS file. All contributions to this
repository must be approved by one or more Core Reviewers [2].
If you are contributing to files (or create new directories) in
root folder of this repository, please contact Core Reviewers for
review and merge requests.
If you are contributing to subfolders of this repository, please
check 'maintainers' section of this file in order to find maintainers
for those specific modules.
It is mandatory to get +1 from one or more maintainers before asking
Core Reviewers for review/merge in order to decrease a load on Core Reviewers [3].
Exceptions are when maintainers are actually cores, or when maintainers
are not available for some reason (e.g. on vacation).
[1] https://specs.openstack.org/openstack/fuel-specs/policy/team-structure
[2] https://review.openstack.org/#/admin/groups/655,members
[3] http://lists.openstack.org/pipermail/openstack-dev/2015-August/072406.html
Please keep this file in YAML format in order to allow helper scripts
to read this as a configuration data.
maintainers:
- ./:
- name: Vladimir Sharshov
email: vsharshov@mirantis.com
IRC: warpc
- name: Vladimir Kozhukalov
email: vkozhukalov@mirantis.com
IRC: kozhukalov
- name: Ivan Ponomarev
email: iponomarev@mirantis.com
IRC: iponomarev
- debian/: &packaging_team
- name: Mikhail Ivanov
email: mivanov@mirantis.com
IRC: mivanov
- name: Artem Silenkov
email: asilenkov@mirantis.com
IRC: asilenkov
- name: Alexander Tsamutali
email: atsamutali@mirantis.com
IRC: astsmtl
- name: Daniil Trishkin
email: dtrishkin@mirantis.com
IRC: dtrishkin
- name: Ivan Udovichenko
email: iudovichenko@mirantis.com
IRC: tlbr
- name: Igor Yozhikov
email: iyozhikov@mirantis.com
IRC: IgorYozhikov
- specs/: *packaging_team

75
README.md
View File

@ -1,75 +0,0 @@
#Astute
[![Team and repository tags](http://governance.openstack.org/badges/fuel-astute.svg)](http://governance.openstack.org/reference/tags/index.html)
<!-- Change things from this point on -->
Astute is orchestrator, which is using data about nodes and deployment settings performs two things:
- provision
- deploy
Provision
-----
OS installation on selected nodes.
Provisioning is done using Cobbler. Astute orchestrator collects data about nodes and creates corresponding Cobbler systems using parameters specified in engine section of provision data. After the systems are created, it connects to Cobbler engine and reboots nodes according to the power management parameters of the node.
Deploy
-----
OpenStack installation in the desired configuration on the selected nodes.
Astute uses data about nodes and deployment settings and recalculates parameters needed for deployment. Calculated parameters are passed to the nodes being deployed by use of nailyfact MCollective agent that uploads these attributes to `/etc/astute.yaml` file of the node. Then puppet parses this file using Facter plugin and uploads these facts into puppet. These facts are used during catalog compilation phase by puppet. Finally catalog is executed and Astute orchestrator passes to the next node in deployment sequence. Fuel Library provides puppet modules for Astute.
Using as library
-----
```ruby
require 'astute'
class ConsoleReporter
def report(msg)
puts msg.inspect
end
end
reporter = ConsoleReporter.new
orchestrator = Astute::Orchestrator.new(log_parsing=false)
# Add systems to cobbler, reboot and start installation process.
orchestrator.provision(reporter, environment['engine'], environment['nodes'])
# Observation OS installation
orchestrator.watch_provision_progress(reporter, environment['task_uuid'], environment['nodes'])
# Deploy OpenStack
orchestrator.deploy(reporter, environment['task_uuid'], environment['nodes'])
```
Example of using Astute as library: lib/astute/server/dispatcher.rb
Using as CLI
-----
CLI interface in Astute no longer supported. Please use new Fuel-CLI. More details you can get by link: https://github.com/openstack/fuel-docs/blob/master/pages/user-guide/cli.rst
-----
- ISO, other materials: http://fuel.mirantis.com/
- User guide: http://docs.mirantis.com/
- Development documentation: http://docs.mirantis.com/fuel-dev/
License
------
Licensed under the Apache License, Version 2.0 (the "License"); you may
not use this file except in compliance with the License. You may obtain
a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
License for the specific language governing permissions and limitations
under the License.

10
README.rst Normal file
View File

@ -0,0 +1,10 @@
This project is no longer maintained.
The contents of this repository are still available in the Git
source code management system. To see the contents of this
repository before it reached its end of life, please check out the
previous commit with "git checkout HEAD^1".
For any further questions, please email
openstack-discuss@lists.openstack.org or join #openstack-dev on
Freenode.

12
Rakefile
View File

@ -1,12 +0,0 @@
require 'rspec/core/rake_task'
namespace :spec do
RSpec::Core::RakeTask.new(:unit) do |t|
t.rspec_opts = "--color --format documentation"
specfile = ENV['S'].to_s.strip.length > 0 ? "*#{ENV['S']}*" : '*'
t.pattern = "spec/unit/**/#{specfile}_spec.rb"
end
end
task :default => 'spec:unit'
task :spec => 'spec:unit'

33
astute.gemspec
View File

@ -1,33 +0,0 @@
$:.unshift File.expand_path('lib', File.dirname(__FILE__))
require 'astute/version'
Gem::Specification.new do |s|
s.name = 'astute'
s.version = Astute::VERSION
s.summary = 'Orchestrator for OpenStack deployment'
s.description = 'Deployment Orchestrator of Puppet via MCollective. Works as a library or from CLI.'
s.authors = ['Mike Scherbakov']
s.email = ['mscherbakov@mirantis.com']
s.add_dependency 'activesupport', '~> 4.1'
s.add_dependency 'mcollective-client', '>= 2.4.1'
s.add_dependency 'symboltable', '>= 1.0.2'
s.add_dependency 'rest-client', '>= 1.6.7'
# Astute as service
s.add_dependency 'bunny', '>= 2.0'
s.add_dependency 'raemon', '>= 0.3'
s.add_development_dependency 'facter'
s.add_development_dependency 'rake', '10.0.4'
s.add_development_dependency 'rspec', '>= 3.4.0'
s.add_development_dependency 'mocha', '0.13.3'
s.add_development_dependency 'simplecov', '~> 0.7.1'
s.add_development_dependency 'simplecov-rcov', '~> 0.2.3'
s.files = Dir.glob("{bin,lib,spec,examples}/**/*")
s.executables = %w(astuted astute-simulator)
s.require_path = 'lib'
end

9
astute.service
View File

@ -1,9 +0,0 @@
[Unit]
Name=Astute daemon
[Service]
EnvironmentFile=-/etc/sysconfig/astute
ExecStart=/usr/bin/astuted $ASTUTE_OPTIONS
[Install]
WantedBy=multi-user.target

1
astute.sysconfig
View File

@ -1 +0,0 @@
ASTUTE_OPTIONS="--config /etc/astute/astuted.conf --logfile /var/log/astute/astute.log --loglevel debug --workers 7"

20
bin/astute-simulator
View File

@ -1,20 +0,0 @@
#!/usr/bin/env ruby
# Copyright 2016 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
require 'fuel_deployment/simulator'
require 'astute'
simulator = Astute::Simulator.new
simulator.run

88
bin/astuted
View File

@ -1,88 +0,0 @@
#!/usr/bin/env ruby
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
require 'astute'
require 'logger'
require 'ostruct'
require 'optparse'
require 'yaml'
require 'raemon'
options = OpenStruct.new
options.daemonize = false
options.pidfile = '/var/run/astuted.pid'
options.config_path = '/etc/astute/astuted.conf'
options.log_path = nil
options.log_level = 'debug'
options.workers = 1
OptionParser.new do |opts|
opts.banner = 'Usage: astuted [options]'
opts.separator "\nOptions:"
opts.on('-d', '--[no-]daemonize', 'Daemonize server') do |flag|
options.daemonize = flag
end
opts.on('-P', '--pidfile PATH', 'Path to pidfile') do |path|
options.pidfile = path
end
opts.on('-w', '--workers NUMBER', 'Number of worker processes') do |number|
options.workers = number.to_i
end
opts.on('-c', '--config PATH', 'Use custom config file') do |path|
unless File.exists?(path)
puts "Error: config file #{path} was not found"
exit
end
options.config_path = path
end
opts.on('-l', '--logfile PATH' 'Log file path') do |path|
options.log_path = path
end
levels = %w{fatal error warn info debug}
opts.on('--loglevel LEVEL', levels, "Logging level (#{levels.join(', ')})") do |level|
options.log_level = level
end
opts.on_tail('-h', '--help', 'Show this message') do
puts opts
exit
end
opts.on_tail('-v', '--version', 'Show version') do
puts Astute::VERSION
exit
end
end.parse!
if options.daemonize
# After daemonize we can't log to STDOUT, pick a default log file
options.log_path ||= "#{Dir.pwd}/astute.log"
end
Astute.config.update(YAML.load(File.read(options.config_path))) if File.exists?(options.config_path)
Astute.logger = options.log_path ? Logger.new(options.log_path) : Logger.new(STDOUT)
Astute.logger.level = Logger.const_get(options.log_level.upcase)
Astute.logger.formatter = proc do |severity, datetime, progname, msg|
severity_map = {'DEBUG' => 'DEBUG', 'INFO' => 'INFO', 'WARN' => 'WARNING', 'ERROR' => 'ERROR', 'FATAL' => 'CRITICAL'}
"#{datetime.strftime("%Y-%m-%d %H:%M:%S")} #{severity_map[severity]} [#{Process.pid}] #{msg}\n"
end
Astute.logger.debug "Starting with settings\n#{Astute.config.to_yaml}"
Raemon::Master.start(options.workers, Astute::Server::Worker,
:detach => options.daemonize,
:name => 'astute',
:pid_file => options.pidfile,
:logger => Astute.logger
)

0
bindep.txt
View File

47
debian/changelog vendored
View File

@ -1,47 +0,0 @@
astute (10.0.0-1) trusty; urgency=low
* Bump version to 10.0
-- Sergey Kulanov <skulanov@mirantis.com> Mon, 21 Mar 2016 13:49:10 +0200
astute (9.0.0-1) trusty; urgency=low
* Update version to 9.0.0
-- Sergey Kulanov <skulanov@mirantis.com> Thu, 17 Dec 2015 15:35:14 +0200
astute (8.0.0-1) trusty; urgency=low
* Update version to 8.0.0
-- Vladimir Sharshov <vsharshov@mirantis.com> Fri, 28 Aug 2015 13:30:00 +0300
astute (7.0.0-1) trusty; urgency=low
* Update version to 7.0.0
-- Aleksandra Fedorova <afedorova@mirantis.com> Mon, 08 Jun 2015 19:30:00 +0300
astute (6.1.0-1) trusty; urgency=low
* Update version to 6.1.0
-- Matthew Mosesohn <mmosesohn@mirantis.com> Wed, 22 Apr 2015 14:44:00 +0300
astute (6.0.0-1) trusty; urgency=low
* Update code from upstream
-- Igor Kalnitsky <ikalnitsky@mirantis.com> Wed, 26 Nov 2014 19:49:00 +0200
astute (0.0.1-ubuntu1) precise; urgency=low
* Update code from upstream
-- OSCI Jenkins <dburmistrov@mirantis.com> Wed, 03 Sep 2014 15:20:13 +0400
astute (0.0.1) unstable; urgency=low
* Initial release.
-- Mirantis Product <product@mirantis.com> Tue, 20 Aug 2013 22:20:46 +0400

1
debian/compat vendored
View File

@ -1 +0,0 @@
7

12
debian/control vendored
View File

@ -1,12 +0,0 @@
Source: astute
Section: admin
Priority: optional
Maintainer: Mirantis Product <product@mirantis.com>
Build-Depends: debhelper (>= 9), gem2deb
Standards-Version: 3.9.2
Package: nailgun-mcagents
Architecture: all
Depends: ${misc:Depends}, ${shlibs:Depends}, mcollective
Description: NailGun mcagents
.

1
debian/nailgun-mcagents.install vendored
View File

@ -1 +0,0 @@
mcagents/* /usr/share/mcollective/plugins/mcollective/agent/

4
debian/rules vendored
View File

@ -1,4 +0,0 @@
#!/usr/bin/make -f
%:
dh $@

1
debian/source/format vendored
View File

@ -1 +0,0 @@
3.0 (quilt)

22
examples/example_astute_config.yaml
View File

@ -1,22 +0,0 @@
# This is example config file for Astute. Your config file should be placed
# to /opt/astute/astute.conf. You can check default values in config.rb file.
---
# mc_retries is used in mclient.rb file.
# MClient tries mc_retries times to call MCagent before failure.
mc_retries: 5
# puppet_timeout is used in puppetd.rb file.
# Maximum time (in seconds) Astute waits for the whole deployment.
puppet_timeout: 3600
# puppet_deploy_interval is used in puppetd.rb file.
# Astute sleeps for puppet_deploy_interval seconds, then check Puppet agents
# statuses again.
puppet_deploy_interval: 2
# puppet_fade_timeout is used in puppetd.rb file.
# After Puppet agent has finished real work it spend some time to graceful exit.
# puppet_fade_timeout means how long (in seconds) Astute can take for Puppet
# to exit after real work has finished.
puppet_fade_timeout: 120
# puppet_fade_interval is used in puppetd.rb file.
# Retry every puppet_fade_interval seconds to check puppet state if it was
# in 'running' state.
puppet_fade_interval: 10

120
lib/astute.rb
View File

@ -1,120 +0,0 @@
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
require 'astute/ruby_removed_functions'
require 'json'
require 'yaml'
require 'logger'
require 'shellwords'
require 'active_support/all'
require 'pp'
require 'bunny'
require 'zlib'
require 'astute/ext/array'
require 'astute/ext/exception'
require 'astute/ext/deep_copy'
require 'astute/ext/hash'
require 'astute/exceptions'
require 'astute/config'
require 'astute/logparser'
require 'astute/orchestrator'
require 'astute/deployment_engine'
require 'astute/network'
require 'astute/puppetd'
require 'astute/provision'
require 'astute/deployment_engine/granular_deployment'
require 'astute/cobbler'
require 'astute/cobbler_manager'
require 'astute/image_provision'
require 'astute/dump'
require 'astute/deploy_actions'
require 'astute/nailgun_hooks'
require 'astute/puppet_task'
require 'astute/puppet_job'
require 'astute/task_manager'
require 'astute/pre_delete'
require 'astute/version'
require 'astute/server/async_logger'
require 'astute/reporter'
require 'astute/mclient'
require 'astute/context'
require 'astute/nodes_remover'
require 'astute/task'
require 'astute/task_deployment'
require 'astute/task_node'
require 'astute/task_proxy_reporter'
require 'astute/task_cluster'
require 'astute/common/reboot'
require 'astute/time_observer'
require 'fuel_deployment'
['/astute/pre_deployment_actions/*.rb',
'/astute/pre_deploy_actions/*.rb',
'/astute/pre_node_actions/*.rb',
'/astute/post_deploy_actions/*.rb',
'/astute/post_deployment_actions/*.rb',
'/astute/common_actions/*.rb',
'/astute/tasks/*.rb',
'/astute/mclients/*.rb'
].each do |path|
Dir[File.dirname(__FILE__) + path].each{ |f| require f }
end
# Server
require 'astute/server/worker'
require 'astute/server/server'
require 'astute/server/producer'
require 'astute/server/dispatcher'
require 'astute/server/reporter'
module Astute
# Library
autoload 'Node', 'astute/node'
autoload 'NodesHash', 'astute/node'
autoload 'Rsyslogd', 'astute/rsyslogd'
LogParser.autoload :ParseDeployLogs, 'astute/logparser/deployment'
LogParser.autoload :ParseProvisionLogs, 'astute/logparser/provision'
LogParser.autoload :ParseImageBuildLogs, 'astute/logparser/provision'
LogParser.autoload :Patterns, 'astute/logparser/parser_patterns'
LOG_PATH = '/var/log/astute.log'
def self.logger
unless @logger
@logger = Logger.new(LOG_PATH)
@logger.formatter = proc do |severity, datetime, progname, msg|
severity_map = {
'DEBUG' => 'DEBUG',
'INFO' => 'INFO',
'WARN' => 'WARNING',
'ERROR' => 'ERROR',
'FATAL' => 'CRITICAL'
}
"#{datetime.strftime("%Y-%m-%d %H:%M:%S")} #{severity_map[severity]} [#{Process.pid}] #{msg}\n"
end
end
@logger
end
def self.logger=(logger)
@logger = logger
Deployment::Log.logger = @logger
end
config_file = '/opt/astute/astute.conf'
Astute.config.update(YAML.load(File.read(config_file))) if File.exists?(config_file)
end

344
lib/astute/cobbler.rb
View File

@ -1,344 +0,0 @@
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
require 'xmlrpc/client'
module Astute
module Provision
class CobblerError < RuntimeError; end
class Cobbler
attr_reader :remote
def initialize(o={})
Astute.logger.debug("Cobbler options:\n#{o.pretty_inspect}")
if (match = /^http:\/\/([^:]+?):?(\d+)?(\/.+)/.match(o['url']))
host = match[1]
port = match[2] || '80'
path = match[3]
else
host = o['host'] || 'localhost'
port = o['port'] || '80'
path = o['path'] || '/cobbler_api'
end
@username = o['username'] || 'cobbler'
@password = o['password'] || 'cobbler'
Astute.logger.debug("Connecting to cobbler with: host: #{host} port: #{port} path: #{path}")
@remote = XMLRPC::Client.new(host, path, port)
@remote.timeout = 120
Astute.logger.debug("Cobbler initialize with username: #{@username}, password: #{@password}")
end
def token
remote.call('login', @username, @password)
end
def item_from_hash(what, name, data, opts = {})
options = {
:item_preremove => true,
}.merge!(opts)
cobsh = Cobsh.new(data.merge({'what' => what, 'name' => name}))
cobblerized = cobsh.cobblerized
Astute.logger.debug("Creating/editing item from hash:\n#{cobsh.pretty_inspect}")
remove_item(what, name) if options[:item_preremove]
# get existent item id or create new one
item_id = get_item_id(what, name)
# defining all item options
cobblerized.each do |opt, value|
next if opt == 'interfaces'
Astute.logger.debug("Setting #{what} #{name} opt: #{opt}=#{value}")
remote.call('modify_item', what, item_id, opt, value, token)
end
# defining system interfaces
if what == 'system' && cobblerized.has_key?('interfaces')
Astute.logger.debug("Defining system interfaces #{name} #{cobblerized['interfaces']}")
remote.call('modify_system', item_id, 'modify_interface',
cobblerized['interfaces'], token)
end
# save item into cobbler database
Astute.logger.debug("Saving #{what} #{name}")
remote.call('save_item', what, item_id, token)
end
def remove_item(what, name, recursive=true)
remote.call('remove_item', what, name, token, recursive) if item_exists(what, name)
end
def remove_system(name)
remove_item('system', name)
end
def item_exists(what, name)
remote.call('has_item', what, name)
end
def items_by_criteria(what, criteria)
remote.call('find_items', what, criteria)
end
def system_by_mac(mac)
items_by_criteria('system', {"mac_address" => mac})[0]
end
def system_exists?(name)
item_exists('system', name)
end
def get_item_id(what, name)
if item_exists(what, name)
item_id = remote.call('get_item_handle', what, name, token)
else
item_id = remote.call('new_item', what, token)
remote.call('modify_item', what, item_id, 'name', name, token)
end
item_id
end
def sync
remote.call('sync', token)
rescue Net::ReadTimeout, XMLRPC::FaultException => e
retries ||= 0
retries += 1
raise e if retries > 2
Astute.logger.warn("Cobbler problem. Try to repeat: #{retries} attempt")
sleep 10
retry
end
def power(name, action)
options = {"systems" => [name], "power" => action}
remote.call('background_power_system', options, token)
end
def power_on(name)
power(name, 'on')
end
def power_off(name)
power(name, 'off')
end
def power_reboot(name)
power(name, 'reboot')
end
def event_status(event_id)
remote.call('get_task_status', event_id)
end
def netboot(name, state)
state = ['on', 'yes', true, 'true', 1, '1'].include?(state)
if system_exists?(name)
system_id = get_item_id('system', name)
else
raise CobblerError, "System #{name} not found."
end
remote.call('modify_system', system_id, 'netboot_enabled', state, token)
remote.call('save_system', system_id, token, 'edit')
end
end
class Cobsh < ::Hash
ALIASES = {
'ks_meta' => ['ksmeta'],
'mac_address' => ['mac'],
'ip_address' => ['ip'],
}
# these fields can be get from the cobbler code
# you can just import cobbler.item_distro.FIELDS
# or cobbler.item_system.FIELDS
FIELDS = {
'system' => {
'fields' => [
'name', 'owners', 'profile', 'image', 'status', 'kernel_options',
'kernel_options_post', 'ks_meta', 'enable_gpxe', 'proxy',
'netboot_enabled', 'kickstart', 'comment', 'server',
'virt_path', 'virt_type', 'virt_cpus', 'virt_file_size',
'virt_disk_driver', 'virt_ram', 'virt_auto_boot', 'power_type',
'power_address', 'power_user', 'power_pass', 'power_id',
'hostname', 'gateway', 'name_servers', 'name_servers_search',
'ipv6_default_device', 'ipv6_autoconfiguration', 'mgmt_classes',
'mgmt_parameters', 'boot_files', 'fetchable_files',
'template_files', 'redhat_management_key', 'redhat_management_server',
'repos_enabled', 'ldap_enabled', 'ldap_type', 'monit_enabled',
],
'interfaces_fields' => [
'mac_address', 'mtu', 'ip_address', 'interface_type',
'interface_master', 'bonding_opts', 'bridge_opts',
'management', 'static', 'netmask', 'dhcp_tag', 'dns_name',
'static_routes', 'virt_bridge', 'ipv6_address', 'ipv6_secondaries',
'ipv6_mtu', 'ipv6_static_routes', 'ipv6_default_gateway'
],
'special' => ['interfaces', 'interfaces_extra']
},
'profile' => {
'fields' => [
'name', 'owners', 'distro', 'parent', 'enable_gpxe',
'enable_menu', 'kickstart', 'kernel_options', 'kernel_options_post',
'ks_meta', 'proxy', 'repos', 'comment', 'virt_auto_boot',
'virt_cpus', 'virt_file_size', 'virt_disk_driver',
'virt_ram', 'virt_type', 'virt_path', 'virt_bridge',
'dhcp_tag', 'server', 'name_servers', 'name_servers_search',
'mgmt_classes', 'mgmt_parameters', 'boot_files', 'fetchable_files',
'template_files', 'redhat_management_key', 'redhat_management_server'
]
},
'distro' => {
'fields' => ['name', 'owners', 'kernel', 'initrd', 'kernel_options',
'kernel_options_post', 'ks_meta', 'arch', 'breed',
'os_version', 'comment', 'mgmt_classes', 'boot_files',
'fetchable_files', 'template_files', 'redhat_management_key',
'redhat_management_server']
}
}
def initialize(h)
Astute.logger.debug("Cobsh is initialized with:\n#{h.pretty_inspect}")
raise CobblerError, "Cobbler hash must have 'name' key" unless h.has_key? 'name'
raise CobblerError, "Cobbler hash must have 'what' key" unless h.has_key? 'what'
raise CobblerError, "Unsupported 'what' value" unless FIELDS.has_key? h['what']
h.each{|k, v| store(k, v)}
end
def cobblerized
Astute.logger.debug("Cobblerizing hash:\n#{pretty_inspect}")
ch = {}
ks_meta = ''
kernel_options = ''
each do |k, v|
k = aliased(k)
if ch.has_key?(k) && ch[k] == v
next
elsif ch.has_key?(k)
raise CobblerError, "Wrong cobbler data: #{k} is duplicated"
end
# skiping not valid item options
unless valid_field?(k)
Astute.logger.warn("Key #{k} is not valid. Will be skipped.")
next
end
ks_meta = serialize_cobbler_parameter(v) if 'ks_meta' == k
kernel_options = serialize_cobbler_parameter(v) if 'kernel_options' == k
# special handling for system interface fields
# which are the only objects in cobbler that will ever work this way
if k == 'interfaces'
ch.store('interfaces', cobblerized_interfaces)
next
end
# here we convert interfaces_extra options into ks_meta format
if k == 'interfaces_extra'
ks_meta << cobblerized_interfaces_extra
next
end
ch.store(k, v)
end # each do |k, v|
ch.store('ks_meta', ks_meta.strip) unless ks_meta.strip.empty?
ch.store('kernel_options', kernel_options.strip) unless kernel_options.strip.empty?
ch
end
def serialize_cobbler_parameter(param)
serialized_param = ''
if param.kind_of?(Hash)
param.each do |ks_meta_key, ks_meta_value|
serialized_param << " #{ks_meta_key}=#{serialize_cobbler_value(ks_meta_value)}"
end
elsif param.kind_of?(String)
param
else
raise CobblerError, "Wrong param format. It must be Hash or String: '#{param}'"
end
serialized_param
end
def serialize_cobbler_value(value)
if value.kind_of?(Hash) || value.kind_of?(Array)
return "\"#{value.to_json.gsub('"', '\"')}\""
end
value
end
def aliased(k)
# converting 'foo-bar' keys into 'foo_bar' keys
k1 = k.gsub(/-/,'_')
# converting orig keys into alias keys
# example: 'ksmeta' into 'ks_meta'
k2 = ALIASES.each_key.select{|ak| ALIASES[ak].include?(k1)}[0] || k1
Astute.logger.debug("Key #{k} aliased with #{k2}") if k != k2
k2
end
def valid_field?(k)
(FIELDS[fetch('what')]['fields'].include?(k) or
(FIELDS[fetch('what')]['special'] or []).include?(k))
end
def valid_interface_field?(k)
(FIELDS[fetch('what')]['interfaces_fields'] or []).include?(k)
end
def cobblerized_interfaces
interfaces = {}
fetch('interfaces').each do |iname, ihash|
ihash.each do |iopt, ivalue|
iopt = aliased(iopt)
if interfaces.has_key?("#{iopt}-#{iname}")
raise CobblerError, "Wrong interface cobbler data: #{iopt} is duplicated"
end
unless valid_interface_field?(iopt)
Astute.logger.debug("Interface key #{iopt} is not valid. Skipping")
next
end
Astute.logger.debug("Defining interfaces[#{iopt}-#{iname}] = #{ivalue}")
interfaces["#{iopt}-#{iname}"] = ivalue
end
end
interfaces
end
def cobblerized_interfaces_extra
# here we just want to convert interfaces_extra into ks_meta
interfaces_extra_str = ""
fetch('interfaces_extra').each do |iname, iextra|
iextra.each do |k, v|
Astute.logger.debug("Adding into ks_meta interface_extra_#{iname}_#{k}=#{v}")
interfaces_extra_str << " interface_extra_#{iname}_#{k}=#{v}"
end
end
interfaces_extra_str
end
end
end
end

260
lib/astute/cobbler_manager.rb
View File

@ -1,260 +0,0 @@
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class CobblerManager
def initialize(engine_attrs, reporter)
raise "Settings for Cobbler must be set" if engine_attrs.blank?
begin
Astute.logger.debug("Trying to instantiate cobbler engine:"\
"\n#{engine_attrs.pretty_inspect}")
@engine = Astute::Provision::Cobbler.new(engine_attrs)
rescue => e
Astute.logger.error("Error occured during cobbler initializing")
reporter.report({
'status' => 'error',
'error' => 'Cobbler can not be initialized',
'progress' => 100
})
raise e
end
end
def add_nodes(nodes)
nodes.each do |node|
cobbler_name = node['slave_name']
begin
Astute.logger.info("Adding #{cobbler_name} into cobbler")
@engine.item_from_hash('system', cobbler_name, node, :item_preremove => true)
rescue RuntimeError => e
Astute.logger.error("Error occured while adding system #{cobbler_name} to cobbler")
raise e
end
end
ensure
sync
end
def remove_nodes(nodes, retries=3, interval=2)
nodes_to_remove = nodes.map {|node| node['slave_name']}.uniq
Astute.logger.info("Total list of nodes to remove: #{nodes_to_remove.pretty_inspect}")
retries.times do
nodes_to_remove.select! do |name|
unless @engine.system_exists?(name)
Astute.logger.info("System is not in cobbler: #{name}")
next
else
Astute.logger.info("Trying to remove system from cobbler: #{name}")
@engine.remove_system(name)
end
@engine.system_exists?(name)
end
return if nodes_to_remove.empty?
sleep(interval) if interval > 0
end
ensure
Astute.logger.error("Cannot remove nodes from cobbler: #{nodes_to_remove.pretty_inspect}") if nodes_to_remove.present?
sync
end
def reboot_nodes(nodes)
splay = calculate_splay_between_nodes(nodes)
nodes.inject({}) do |reboot_events, node|
cobbler_name = node['slave_name']
Astute.logger.debug("Trying to reboot node: #{cobbler_name}")
#Sleep up to splay seconds before reboot for load balancing
sleep splay
reboot_events.merge(cobbler_name => @engine.power_reboot(cobbler_name))
end
end
def check_reboot_nodes(reboot_events)
begin
Astute.logger.debug("Waiting for reboot to be complete: nodes: #{reboot_events.keys}")
failed_nodes = []
Timeout::timeout(Astute.config.reboot_timeout) do
while not reboot_events.empty?
reboot_events.each do |node_name, event_id|
event_status = @engine.event_status(event_id)
Astute.logger.debug("Reboot task status: node: #{node_name} status: #{event_status}")
if event_status[2] =~ /^failed$/
Astute.logger.error("Error occured while trying to reboot: #{node_name}")
reboot_events.delete(node_name)
failed_nodes << node_name
elsif event_status[2] =~ /^complete$/
Astute.logger.debug("Successfully rebooted: #{node_name}")
reboot_events.delete(node_name)
end
end
sleep(5)
end
end
rescue Timeout::Error => e
Astute.logger.debug("Reboot timeout: reboot tasks not completed for nodes #{reboot_events.keys}")
raise e
end
failed_nodes
end
def edit_nodes(nodes, data)
nodes.each do |node|
cobbler_name = node['slave_name']
begin
Astute.logger.info("Changing cobbler system #{cobbler_name}")
@engine.item_from_hash('system', cobbler_name, data, :item_preremove => false)
rescue RuntimeError => e
Astute.logger.error("Error occured while changing cobbler system #{cobbler_name}")
raise e
end
end
ensure
sync
end
def netboot_nodes(nodes, state)
nodes.each do |node|
cobbler_name = node['slave_name']
begin
Astute.logger.info("Changing node netboot state #{cobbler_name}")
@engine.netboot(cobbler_name, state)
rescue RuntimeError => e
Astute.logger.error("Error while changing node netboot state #{cobbler_name}")
raise e
end
end
ensure
sync
end
def get_existent_nodes(nodes)
existent_nodes = []
nodes.each do |node|
cobbler_name = node['slave_name']
if @engine.system_exists?(cobbler_name)
Astute.logger.info("Update #{cobbler_name}, node already exists in cobbler")
existent_nodes << node
end
end
existent_nodes
end
def existent_node?(cobbler_name)
return false unless @engine.system_exists?(cobbler_name)
Astute.logger.info("Node #{cobbler_name} already exists in cobbler")
true
end
def edit_node(cobbler_name, data)
begin
Astute.logger.info("Changing cobbler system #{cobbler_name}")
@engine.item_from_hash('system', cobbler_name, data, :item_preremove => false)
rescue RuntimeError => e
Astute.logger.error("Error occured while changing cobbler system #{cobbler_name}")
raise e
end
ensure
sync
end
def netboot_node(cobbler_name, state)
begin
Astute.logger.info("Changing node netboot state #{cobbler_name}")
@engine.netboot(cobbler_name, state)
rescue RuntimeError => e
Astute.logger.error("Error while changing node netboot state #{cobbler_name}")
raise e
end
ensure
sync
end
def remove_node(cobbler_name, retries=3, interval=2)
Astute.logger.info("Node to remove: #{cobbler_name}")
retries.times do
unless @engine.system_exists?(cobbler_name)
Astute.logger.info("System is not in cobbler: #{cobbler_name}")
return
else
Astute.logger.info("Trying to remove system from cobbler: #{cobbler_name}")
@engine.remove_system(cobbler_name)
end
return unless @engine.system_exists?(cobbler_name)
sleep(interval) if interval > 0
end
ensure
Astute.logger.error("Cannot remove node #{cobbler_name} from cobbler") if @engine.system_exists?(cobbler_name)
sync
end
def add_node(node)
cobbler_name = node['slave_name']
begin
Astute.logger.info("Adding #{cobbler_name} into cobbler")
@engine.item_from_hash('system', cobbler_name, node, :item_preremove => true)
rescue RuntimeError => e
Astute.logger.error("Error occured while adding system #{cobbler_name} to cobbler")
raise e
end
ensure
sync
end
def node_mac_duplicate_names(node)
mac_duplicate_names = []
Astute.logger.info("Trying to find MAC duplicates for node #{node['slave_name']}")
if node['interfaces']
node['interfaces'].each do |iname, ihash|
if ihash['mac_address']
Astute.logger.info("Trying to find system with MAC: #{ihash['mac_address']}")
found_node = @engine.system_by_mac(ihash['mac_address'])
mac_duplicate_names << found_node['name'] if found_node
end
end
end
mac_duplicate_names.uniq
end
def get_mac_duplicate_names(nodes)
mac_duplicate_names = []
nodes.each do |node|
Astute.logger.info("Trying to find MAC duplicates for node #{node['slave_name']}")
if node['interfaces']
node['interfaces'].each do |iname, ihash|
if ihash['mac_address']
Astute.logger.info("Trying to find system with MAC: #{ihash['mac_address']}")
found_node = @engine.system_by_mac(ihash['mac_address'])
mac_duplicate_names << found_node['name'] if found_node
end
end
end
end
mac_duplicate_names.uniq
end
def sync
Astute.logger.debug("Cobbler syncing")
@engine.sync
end
private
def calculate_splay_between_nodes(nodes)
# For 20 nodes, 120 iops and 180 splay_factor splay will be 1.5749
(nodes.size + 1) / Astute.config.iops.to_f * Astute.config.splay_factor / nodes.size
end
end
end

38
lib/astute/common/reboot.rb
View File

@ -1,38 +0,0 @@
# Copyright 2014 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
module RebootCommand
# Reboot immediately if we're in a bootstrap. Wait until system boots
# completely in case of provisioned node. We check it by existense
# of /run/cloud-init/status.json (it's located on tmpfs, so no stale
# file from previous boot can be found). If this file hasn't appeared
# after 60 seconds - reboot as is.
CMD = <<-REBOOT_COMMAND
if [ $(hostname) = bootstrap ]; then
reboot;
fi;
t=0;
while true; do
if [ -f /run/cloud-init/status.json -o $t -gt 60 ]; then
reboot;
else
sleep 1;
t=$((t + 1));
fi;
done
REBOOT_COMMAND
end
end

90
lib/astute/common_actions/pacemaker.rb
View File

@ -1,90 +0,0 @@
# Copyright 2014 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class Pacemaker
def self.commands(behavior, deployment_info)
return [] if deployment_info.first['deployment_mode'] !~ /ha/i
controller_nodes = deployment_info.select{ |n| n['role'] =~ /controller/i }.map{ |n| n['uid'] }
return [] if controller_nodes.empty?
ha_size = deployment_info.first['nodes'].count { |n|
['controller', 'primary-controller'].include? n['role']
}
action = if ha_size < 3
case behavior
when 'stop' then 'stop'
when 'start' then 'start'
end
else
case behavior
when 'stop' then 'ban'
when 'start' then 'clear'
end
end
cmds = pacemaker_services_list(deployment_info).inject([]) do |cmds, pacemaker_service|
if ha_size < 3
cmds << "crm resource #{action} #{pacemaker_service} && sleep 3"
else
cmds << "pcs resource #{action} #{pacemaker_service} `crm_node -n` && sleep 3"
end
end
cmds
end
private
def self.pacemaker_services_list(deployment_info)
services_list = []
#Heat engine service is present everywhere
services_list += heat_service_name(deployment_info)
if deployment_info.first['quantum']
services_list << 'p_neutron-openvswitch-agent'
services_list << 'p_neutron-metadata-agent'
services_list << 'p_neutron-l3-agent'
services_list << 'p_neutron-dhcp-agent'
end
if deployment_info.first.fetch('ceilometer', {})['enabled']
services_list += ceilometer_service_names(deployment_info)
end
return services_list
end
def self.ceilometer_service_names(deployment_info)
case deployment_info.first['cobbler']['profile']
when /centos/i
['p_openstack-ceilometer-compute','p_openstack-ceilometer-central']
when /ubuntu/i
['p_ceilometer-agent-central','p_ceilometer-agent-compute']
end
end
def self.heat_service_name(deployment_info)
case deployment_info.first['cobbler']['profile']
when /centos/i
['openstack-heat-engine', 'p_openstack-heat-engine']
when /ubuntu/i
['heat-engine', 'p_heat-engine']
end
end
end #class
end

130
lib/astute/config.rb
View File

@ -1,130 +0,0 @@
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
require 'symboltable'
require 'singleton'
module Astute
class ConfigError < StandardError; end
class UnknownOptionError < ConfigError
attr_reader :name
def initialize(name)
super("Unknown config option #{name}")
@name = name
end
end
class MyConfig
include Singleton
attr_reader :configtable
def initialize
@configtable = SymbolTable.new
end
end
class ParseError < ConfigError
attr_reader :line
def initialize(message, line)
super(message)
@line = line
end
end
def self.config
config = MyConfig.instance.configtable
config.update(default_config) if config.empty?
return config
end
def self.default_config
conf = {}
# Library settings
conf[:puppet_timeout] = 90 * 60 # maximum time it waits for single puppet run
conf[:puppet_deploy_interval] = 2 # sleep for ## sec, then check puppet status again
conf[:puppet_fade_timeout] = 120 # how long it can take for puppet to exit after dumping to last_run_summary
conf[:puppet_start_timeout] = 10 # how long it can take for puppet to start
conf[:puppet_start_interval] = 2 # interval between attemps to start puppet
conf[:puppet_retries] = 2 # how many times astute will try to run puppet
conf[:upload_retries] = 3 # how many times astute will try to run upload task
conf[:puppet_succeed_retries] = 0 # use this to rerun a puppet task again if it was successful (idempotency)
conf[:puppet_undefined_retries] = 3 # how many times astute will try to get actual status of node before fail
conf[:puppet_module_path] = '/etc/puppet/modules' # where we should find basic modules for puppet
conf[:puppet_noop_run] = false # enable Puppet noop run
conf[:mc_retries] = 10 # MClient tries to call mcagent before failure
conf[:mc_retry_interval] = 1 # MClient sleeps for ## sec between retries
conf[:puppet_fade_interval] = 30 # retry every ## seconds to check puppet state if it was running
conf[:provisioning_timeout] = 90 * 60 # timeout for booting target OS in provision
conf[:reboot_timeout] = 900 # how long it can take for node to reboot
conf[:dump_timeout] = 3600 # maximum time it waits for the dump (meaningles to be larger
# than the specified in timeout of execute_shell_command mcagent
conf[:shell_retries] = 2 # default retries for shell task
conf[:shell_interval] = 2 # default interval for shell task
conf[:shell_timeout] = 300 # default timeout for shell task
conf[:upload_timeout] = 60 # default timeout for upload task
conf[:shell_cwd] = '/' # default cwd for shell task
conf[:stop_timeout] = 600 # how long it can take for stop
conf[:rsync_options] = '-c -r --delete -l' # default rsync options
conf[:keys_src_dir] = '/var/lib/fuel/keys' # path where ssh and openssl keys will be created
conf[:puppet_ssh_keys] = [
'neutron',
'nova',
'ceph',
'mysql',
] # name of ssh keys what will be generated and uploaded to all nodes before deploy
conf[:puppet_keys] = [
'mongodb'
] # name of keys what will be generated and uploaded to all nodes before deploy
conf[:keys_dst_dir] = '/var/lib/astute' # folder where keys will be uploaded. Warning!
conf[:max_nodes_per_call] = 50 # how many nodes to deploy simultaneously
conf[:max_nodes_to_provision] = 50 # how many nodes to provision simultaneously
conf[:ssh_retry_timeout] = 30 # SSH sleeps for ## sec between retries
conf[:max_nodes_per_remove_call] = 10 # how many nodes to remove in one call
conf[:nodes_remove_interval] = 10 # sleeps for ## sec between remove calls
conf[:max_nodes_net_validation] = 10 # how many nodes will send in parallel test packets
# during network verification
conf[:dhcp_repeat] = 3 # Dhcp discover will be sended 3 times
conf[:iops] = 120 # Default IOPS master node IOPS performance
conf[:splay_factor] = 180 # Formula: 20(amount of nodes nodes) div 120(iops) = 0.1667
# 0.1667 / 180 = 30 sec. Delay between reboot command for first
# and last node in group should be 30 sec. Empirical observation.
# Please increase if nodes could not provisioning
conf[:agent_nodiscover_file] = '/etc/nailgun-agent/nodiscover' # if this file in place, nailgun-agent will do nothing
conf[:bootstrap_profile] = 'ubuntu_bootstrap' # use the Ubuntu based bootstrap by default
conf[:graph_dot_dir] = "/var/lib/astute/graphs" # default dir patch for debug graph file
conf[:enable_graph_file] = true # enable debug graph records to file
conf[:puppet_raw_report] = false # enable puppet detailed report
conf[:task_poll_delay] = 1 # sleeps for ## sec between task status calls
# Server settings
conf[:broker_host] = 'localhost'
conf[:broker_port] = 5672
conf[:broker_rest_api_port] = 15672
conf[:broker_username] = 'mcollective'
conf[:broker_password] = 'mcollective'
conf[:broker_service_exchange] = 'naily_service'
conf[:broker_queue] = 'naily'
conf[:broker_publisher_queue] = 'nailgun'
conf[:broker_exchange] = 'nailgun'
conf
end
end

43
lib/astute/context.rb
View File

@ -1,43 +0,0 @@
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class Context
attr_accessor :reporter, :deploy_log_parser
attr_reader :task_id, :status
def initialize(task_id, reporter, deploy_log_parser=nil)
@task_id = task_id
@reporter = reporter
@status = {}
@deploy_log_parser = deploy_log_parser
end
def report_and_update_status(data)
if data['nodes']
data['nodes'].each do |node|
#TODO(vsharshov): save node role to hash
@status.merge! node['uid'] => node['status'] if node['uid'] && node['status']
end
end
reporter.report(data)
end
def report(msg)
@reporter.report msg
end
end
end

215
lib/astute/deploy_actions.rb
View File

@ -1,215 +0,0 @@
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class DeployActions
def initialize(deployment_info, context)
@deployment_info = deployment_info
@context = context
@actions = []
end
def process
@actions.each { |action| action.process(@deployment_info, @context) }
end
end
class PreDeployActions < DeployActions
def initialize(deployment_info, context)
super
@actions = [
ConnectFacts.new
]
end
end
class GranularPreDeployActions < DeployActions
def initialize(deployment_info, context)
super
@actions = [
ConnectFacts.new
]
end
end
class PostDeployActions < DeployActions
def initialize(deployment_info, context)
super
@actions = [
PostPatchingHa.new
]
end
end
class GranularPostDeployActions < DeployActions
def initialize(deployment_info, context)
super
@actions = [
PostPatchingHa.new
]
end
end
class PreNodeActions
def initialize(context)
@node_uids = []
@context = context
@actions = [
PrePatchingHa.new,
StopOSTServices.new,
PrePatching.new
]
end
def process(deployment_info)
nodes_to_process = deployment_info.select { |n| !@node_uids.include?(n['uid']) }
return if nodes_to_process.empty?
@actions.each { |action| action.process(nodes_to_process, @context) }
@node_uids += nodes_to_process.map { |n| n['uid'] }
end
end
class GranularPreNodeActions
def initialize(context)
@node_uids = []
@context = context
@actions = [
PrePatchingHa.new,
StopOSTServices.new,
PrePatching.new
]
end
def process(deployment_info)
nodes_to_process = deployment_info.select { |n| !@node_uids.include?(n['uid']) }
return if nodes_to_process.empty?
@actions.each { |action| action.process(nodes_to_process, @context) }
@node_uids += nodes_to_process.map { |n| n['uid'] }
end
end
class PreDeploymentActions < DeployActions
def initialize(deployment_info, context)
super
@actions = [
SyncTime.new,
GenerateSshKeys.new,
GenerateKeys.new,
UploadSshKeys.new,
UploadKeys.new,
UpdateRepoSources.new,
SyncPuppetStuff.new,
SyncTasks.new,
EnablePuppetDeploy.new,
UploadFacts.new,
InitialConnectFacts.new
]
end
end
class GranularPreDeploymentActions < DeployActions
def initialize(deployment_info, context)
super
@actions = [
EnablePuppetDeploy.new,
UploadFacts.new,
InitialConnectFacts.new
]
end
end
class TaskPreDeploymentActions < DeployActions
def initialize(deployment_info, context)
super
@actions = [
EnablePuppetDeploy.new,
UploadFacts.new,
InitialConnectFacts.new
]
end
end
class PostDeploymentActions < DeployActions
def initialize(deployment_info, context)
super
@actions = [
UpdateNoQuorumPolicy.new,
UploadCirrosImage.new,
RestartRadosgw.new,
UpdateClusterHostsInfo.new
]
end
end
class DeployAction
def process(deployment_info, context)
raise "Should be implemented!"
end
def run_shell_command(context, node_uids, cmd, timeout=60)
shell = MClient.new(context,
'execute_shell_command',
node_uids,
check_result=true,
timeout=timeout,
retries=1)
#TODO: return result for all nodes not only for first
response = shell.execute(:cmd => cmd).first
Astute.logger.debug("#{context.task_id}: cmd: #{cmd}
stdout: #{response[:data][:stdout]}
stderr: #{response[:data][:stderr]}
exit code: #{response[:data][:exit_code]}")
response
rescue MClientTimeout, MClientError => e
Astute.logger.error("#{context.task_id}: cmd: #{cmd}
mcollective error: #{e.message}")
{:data => {}}
end
def only_uniq_nodes(nodes)
nodes.uniq { |n| n['uid'] }
end
# Prevent high load for tasks
def perform_with_limit(nodes, &block)
nodes.each_slice(Astute.config[:max_nodes_per_call]) do |part|
block.call(part)
end
end
end # DeployAction
class PreDeployAction < DeployAction; end
class PostDeployAction < DeployAction; end
class PreNodeAction < DeployAction; end
class PostNodeAction < DeployAction; end
class PreDeploymentAction < DeployAction; end
class PostDeploymentAction < DeployAction; end
end

269
lib/astute/deployment_engine.rb
View File

@ -1,269 +0,0 @@
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class DeploymentEngine
def initialize(context)
if self.class.superclass.name == 'Object'
raise "Instantiation of this superclass is not allowed. Please subclass from #{self.class.name}."
end
@ctx = context
end
def deploy(deployment_info, pre_deployment=[], post_deployment=[])
raise "Deployment info are not provided!" if deployment_info.blank?
deployment_info, pre_deployment, post_deployment = remove_failed_nodes(deployment_info,
pre_deployment,
post_deployment)
@ctx.deploy_log_parser.deploy_type = deployment_info.first['deployment_mode']
Astute.logger.info "Deployment mode #{@ctx.deploy_log_parser.deploy_type}"
begin
pre_deployment_actions(deployment_info, pre_deployment)
rescue => e
Astute.logger.error("Unexpected error #{e.message} traceback #{e.format_backtrace}")
raise e
end
failed = []
# Sort by priority (the lower the number, the higher the priority)
# and send groups to deploy
deployment_info.sort_by { |f| f['priority'] }.group_by{ |f| f['priority'] }.each do |_, nodes|
# Prevent attempts to run several deploy on a single node.
# This is possible because one node
# can perform multiple roles.
group_by_uniq_values(nodes).each do |nodes_group|
# Prevent deploy too many nodes at once
nodes_group.each_slice(Astute.config[:max_nodes_per_call]) do |part|
# for each chunk run group deployment pipeline
# create links to the astute.yaml
pre_deploy_actions(part)
# run group deployment
deploy_piece(part)
failed = critical_failed_nodes(part)
# if any of the node are critical and failed
# raise an error and mark all other nodes as error
if failed.any?
# TODO(dshulyak) maybe we should print all failed tasks for this nodes
# but i am not sure how it will look like
raise Astute::DeploymentEngineError, "Deployment failed on nodes #{failed.join(', ')}"
end
end
end
end
# Post deployment hooks
post_deployment_actions(deployment_info, post_deployment)
end
protected
def validate_nodes(nodes)
return true unless nodes.empty?
Astute.logger.info "#{@ctx.task_id}: Nodes to deploy are not provided. Do nothing."
false
end
private
# Transform nodes source array to array of nodes arrays where subarray
# contain only uniq elements from source
# Source: [
# {'uid' => 1, 'role' => 'cinder'},
# {'uid' => 2, 'role' => 'cinder'},
# {'uid' => 2, 'role' => 'compute'}]
# Result: [
# [{'uid' =>1, 'role' => 'cinder'},
# {'uid' => 2, 'role' => 'cinder'}],
# [{'uid' => 2, 'role' => 'compute'}]]
def group_by_uniq_values(nodes_array)
nodes_array = deep_copy(nodes_array)
sub_arrays = []
while !nodes_array.empty?
sub_arrays << uniq_nodes(nodes_array)
uniq_nodes(nodes_array).clone.each {|e| nodes_array.slice!(nodes_array.index(e)) }
end
sub_arrays
end
def uniq_nodes(nodes_array)
nodes_array.inject([]) { |result, node| result << node unless include_node?(result, node); result }
end
def include_node?(nodes_array, node)
nodes_array.find { |n| node['uid'] == n['uid'] }
end
def nodes_status(nodes, status, data_to_merge)
{
'nodes' => nodes.map do |n|
{'uid' => n['uid'], 'status' => status, 'role' => n['role']}.merge(data_to_merge)
end
}
end
def critical_failed_nodes(part)
part.select{ |n| n['fail_if_error'] }.map{ |n| n['uid'] } &
@ctx.status.select { |k, v| v == 'error' }.keys
end
def pre_deployment_actions(deployment_info, pre_deployment)
raise "Should be implemented"
end
def pre_node_actions(part)
raise "Should be implemented"
end
def pre_deploy_actions(part)
raise "Should be implemented"
end
def post_deploy_actions(part)
raise "Should be implemented"
end
def post_deployment_actions(deployment_info, post_deployment)
raise "Should be implemented"
end
# Removes nodes which failed to provision
def remove_failed_nodes(deployment_info, pre_deployment, post_deployment)
uids = get_uids_from_deployment_info deployment_info
required_nodes = deployment_info.select { |node| node["fail_if_error"] }
required_uids = required_nodes.map { |node| node["uid"]}
available_uids = detect_available_nodes(uids)
offline_uids = uids - available_uids
if offline_uids.present?
# set status for all failed nodes to error
nodes = (uids - available_uids).map do |uid|
{'uid' => uid,
'status' => 'error',
'error_type' => 'provision',
# Avoid deployment reporter param validation
'role' => 'hook',
'error_msg' => 'Node is not ready for deployment: mcollective has not answered'
}
end
@ctx.report_and_update_status('nodes' => nodes, 'error' => 'Node is not ready for deployment')
# check if all required nodes are online
# if not, raise error
missing_required = required_uids - available_uids
if missing_required.present?
error_message = "Critical nodes are not available for deployment: #{missing_required}"
raise Astute::DeploymentEngineError, error_message
end
end
return remove_offline_nodes(
uids,
available_uids,
pre_deployment,
deployment_info,
post_deployment,
offline_uids)
end
def remove_offline_nodes(uids, available_uids, pre_deployment, deployment_info, post_deployment, offline_uids)
if offline_uids.blank?
return [deployment_info, pre_deployment, post_deployment]
end
Astute.logger.info "Removing nodes which failed to provision: #{offline_uids}"
deployment_info = cleanup_nodes_block(deployment_info, offline_uids)
deployment_info = deployment_info.select { |node| available_uids.include? node['uid'] }
available_uids += ["master"]
pre_deployment.each do |task|
task['uids'] = task['uids'].select { |uid| available_uids.include? uid }
end
post_deployment.each do |task|
task['uids'] = task['uids'].select { |uid| available_uids.include? uid }
end
[pre_deployment, post_deployment].each do |deployment_task|
deployment_task.select! do |task|
if task['uids'].present?
true
else
Astute.logger.info "Task(hook) was deleted because there is no " \
"node where it should be run \n#{task.to_yaml}"
false
end
end
end
[deployment_info, pre_deployment, post_deployment]
end
def cleanup_nodes_block(deployment_info, offline_uids)
return deployment_info if offline_uids.blank?
nodes = deployment_info.first['nodes']
# In case of deploy in already existing cluster in nodes block
# we will have all cluster nodes. We should remove only missing
# nodes instead of stay only available.
# Example: deploy 3 nodes, after it deploy 2 nodes.
# In 1 of 2 seconds nodes missing, in nodes block we should
# contain only 4 nodes.
nodes_wthout_missing = nodes.select { |node| !offline_uids.include?(node['uid']) }
deployment_info.each { |node| node['nodes'] = nodes_wthout_missing }
deployment_info
end
def detect_available_nodes(uids)
all_uids = uids.clone
available_uids = []
# In case of big amount of nodes we should do several calls to be sure
# about node status
Astute.config[:mc_retries].times.each do
systemtype = Astute::MClient.new(@ctx, "systemtype", all_uids, check_result=false, 10)
available_nodes = systemtype.get_type.select do |node|
node.results[:data][:node_type].chomp == "target"
end
available_uids += available_nodes.map { |node| node.results[:sender] }
all_uids -= available_uids
break if all_uids.empty?
sleep Astute.config[:mc_retry_interval]
end
available_uids
end
def get_uids_from_deployment_info(deployment_info)
top_level_uids = deployment_info.map{ |node| node["uid"] }
inside_uids = deployment_info.inject([]) do |uids, node|
uids += node.fetch('nodes', []).map{ |n| n['uid'] }
end
top_level_uids | inside_uids
end
end
end

271
lib/astute/deployment_engine/granular_deployment.rb
View File

@ -1,271 +0,0 @@
# Copyright 2014 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
class Astute::DeploymentEngine::GranularDeployment < Astute::DeploymentEngine
NAILGUN_STATUS = ['ready', 'error', 'deploying']
def deploy_piece(nodes, retries=1)
report_ready_for_nodes_without_tasks(nodes)
nodes = filter_nodes_with_tasks(nodes)
return false unless validate_nodes(nodes)
@ctx.reporter.report(nodes_status(nodes, 'deploying', {'progress' => 0}))
log_preparation(nodes)
Astute.logger.info "#{@ctx.task_id}: Starting deployment"
@running_tasks = {}
@start_times = {}
@nodes_roles = nodes.inject({}) { |h, n| h.merge({n['uid'] => n['role']}) }
@nodes_by_uid = nodes.inject({}) { |h, n| h.merge({ n['uid'] => n }) }
@puppet_debug = nodes.first.fetch('puppet_debug', true)
begin
@task_manager = Astute::TaskManager.new(nodes)
@hook_context = Astute::Context.new(
@ctx.task_id,
HookReporter.new,
Astute::LogParser::NoParsing.new
)
deploy_nodes(nodes)
rescue => e
# We should fail all nodes in case of post deployment
# process. In other case they will not sending back
# for redeploy
report_nodes = nodes.uniq{ |n| n['uid'] }.map do |node|
{ 'uid' => node['uid'],
'status' => 'error',
'role' => node['role'],
'error_type' => 'deploy'
}
end
@ctx.report_and_update_status('nodes' => report_nodes)
raise e
end
Astute.logger.info "#{@ctx.task_id}: Finished deployment of nodes" \
" => roles: #{@nodes_roles.pretty_inspect}"
end
def puppet_task(node_id, task)
# Use fake reporter because of logic. We need to handle report here
Astute::PuppetTask.new(
@hook_context,
@nodes_by_uid[node_id], # Use single node uid instead of task['uids']
{
:retries => task['parameters']['retries'],
:puppet_manifest => task['parameters']['puppet_manifest'],
:puppet_modules => task['parameters']['puppet_modules'],
:cwd => task['parameters']['cwd'],
:timeout => task['parameters']['timeout'],
:puppet_debug => @puppet_debug
}
)
end
def run_task(node_id, task)
@start_times[node_id] = {
'time_start' => Time.now.to_i,
'task_name' => task_name(task)
}
Astute.logger.info "#{@ctx.task_id}: run task '#{task.to_yaml}' on " \
"node #{node_id}"
@running_tasks[node_id] = puppet_task(node_id, task)
@running_tasks[node_id].run
end
def check_status(node_id)
status = @running_tasks[node_id].status
if NAILGUN_STATUS.include? status
status
else
raise "Internal error. Unknown status '#{status}'"
end
end
def deploy_nodes(nodes)
@task_manager.node_uids.each { |n| task = @task_manager.next_task(n) and run_task(n, task) }
while @task_manager.task_in_queue?
nodes_to_report = []
sleep Astute.config.puppet_deploy_interval
@task_manager.node_uids.each do |node_id|
if task = @task_manager.current_task(node_id)
case status = check_status(node_id)
when 'ready'
Astute.logger.info "Task '#{task}' on node uid=#{node_id} " \
"ended successfully"
time_summary(node_id, status)
new_task = @task_manager.next_task(node_id)
if new_task
run_task(node_id, new_task)
else
nodes_to_report << process_success_node(node_id, task)
end
when 'deploying'
progress_report = process_running_node(node_id, task, nodes)
nodes_to_report << progress_report if progress_report
when 'error'
Astute.logger.error "Task '#{task}' failed on node #{node_id}"
nodes_to_report << process_fail_node(node_id, task)
time_summary(node_id, status)
else
raise "Internal error. Known status '#{status}', but " \
"handler not provided"
end
else
Astute.logger.debug "No more tasks provided for node #{node_id}"
end
end
@ctx.report_and_update_status('nodes' => nodes_to_report) if nodes_to_report.present?
break unless @task_manager.task_in_queue?
end
end
def process_success_node(node_id, task)
Astute.logger.info "No more tasks provided for node #{node_id}. All node " \
"tasks completed successfully"
{
"uid" => node_id,
'status' => 'ready',
'role' => @nodes_roles[node_id],
"progress" => 100,
'task' => task
}
end
def process_fail_node(node_id, task)
Astute.logger.error "No more tasks will be executed on the node #{node_id}"
@task_manager.delete_node(node_id)
{
'uid' => node_id,
'status' => 'error',
'error_type' => 'deploy',
'role' => @nodes_roles[node_id],
'task' => task
}
end
def process_running_node(node_id, task, nodes)
Astute.logger.debug "Task '#{task}' on node uid=#{node_id} deploying"
begin
# Pass nodes because logs calculation needs IP address of node, not just uid
nodes_progress = @ctx.deploy_log_parser.progress_calculate(Array(node_id), nodes)
if nodes_progress.present?
nodes_progress.map! { |x| x.merge!(
'status' => 'deploying',
'role' => @nodes_roles[x['uid']],
'task' => task
) }
nodes_progress.first
else
nil
end
rescue => e
Astute.logger.warn "Some error occurred when parse logs for nodes progress: #{e.message}, "\
"trace: #{e.format_backtrace}"
nil
end
end
def log_preparation(nodes)
@ctx.deploy_log_parser.prepare(nodes)
rescue => e
Astute.logger.warn "Some error occurred when prepare LogParser: " \
"#{e.message}, trace: #{e.format_backtrace}"
end
# If node doesn't have tasks, it means that node
# is ready, because it doesn't require deployment
def report_ready_for_nodes_without_tasks(nodes)
nodes_without_tasks = filter_nodes_without_tasks(nodes)
@ctx.reporter.report(nodes_status(nodes_without_tasks, 'ready', {'progress' => 100}))
end
def filter_nodes_with_tasks(nodes)
nodes.select { |n| node_with_tasks?(n) }
end
def filter_nodes_without_tasks(nodes)
nodes.select { |n| !node_with_tasks?(n) }
end
def node_with_tasks?(node)
node['tasks'].present?
end
# Pre/post hooks
def pre_deployment_actions(deployment_info, pre_deployment)
Astute::GranularPreDeploymentActions.new(deployment_info, @ctx).process
Astute::NailgunHooks.new(pre_deployment, @ctx).process
end
def pre_node_actions(part)
@action ||= Astute::GranularPreNodeActions.new(@ctx)
@action.process(part)
end
def pre_deploy_actions(part)
Astute::GranularPreDeployActions.new(part, @ctx).process
end
def post_deploy_actions(part)
Astute::GranularPostDeployActions.new(part, @ctx).process
end
def post_deployment_actions(deployment_info, post_deployment)
begin
Astute::NailgunHooks.new(post_deployment, @ctx).process
rescue => e
# We should fail all nodes in case of post deployment
# process. In other case they will not sending back
# for redeploy
nodes = deployment_info.uniq {|n| n['uid']}.map do |node|
{ 'uid' => node['uid'],
'status' => 'error',
'role' => 'hook',
'error_type' => 'deploy',
}
end
@ctx.report_and_update_status('nodes' => nodes)
raise e
end
end
def time_summary(node_id, status)
return unless @start_times.fetch(node_id, {}).fetch('time_start', nil)
amount_time = (Time.now.to_i - @start_times[node_id]['time_start']).to_i
wasted_time = Time.at(amount_time).utc.strftime("%H:%M:%S")
Astute.logger.debug("Task time summary:" \
" #{@start_times[node_id]['task_name']} with status" \
" #{status.to_s} on node #{node_id} took #{wasted_time}")
end
def task_name(task)
task['id'] || task['diagnostic_name'] || task['type']
end
class HookReporter
def report(msg)
Astute.logger.debug msg
end
end
end

86
lib/astute/dump.rb
View File

@ -1,86 +0,0 @@
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
module Dump
def self.dump_environment(ctx, settings)
shell = MClient.new(
ctx,
'execute_shell_command',
['master'],
check_result=true,
settings['timeout'] || Astute.config.dump_timeout,
retries=0,
enable_result_logging=false
)
begin
log_file = "/var/log/timmy.log"
snapshot = File.basename(settings['target'])
if settings['timestamp']
snapshot = DateTime.now.strftime("#{snapshot}-%Y-%m-%d_%H-%M-%S")
end
base_dir = File.dirname(settings['target'])
dest_dir = File.join(base_dir, snapshot)
dest_file = File.join(dest_dir, "config.tar.gz")
token = settings['auth-token']
dump_cmd = "mkdir -p #{dest_dir} && "\
"timmy --logs --days 3 --dest-file #{dest_file}"\
" --fuel-token #{token} --log-file #{log_file} && "\
"tar --directory=#{base_dir} -cf #{dest_dir}.tar #{snapshot} && "\
"echo #{dest_dir}.tar > #{settings['lastdump']} && "\
"rm -rf #{dest_dir}"
Astute.logger.debug("Try to execute command: #{dump_cmd}")
result = shell.execute(:cmd => dump_cmd).first.results
Astute.logger.debug("#{ctx.task_id}: exit code: #{result[:data][:exit_code]}")
if result[:data][:exit_code] == 0
Astute.logger.info("#{ctx.task_id}: Snapshot is done.")
report_success(ctx, "#{dest_dir}.tar")
elsif result[:data][:exit_code] == 28
Astute.logger.error("#{ctx.task_id}: Disk space for creating snapshot exceeded.")
report_error(ctx, "Timmy exit code: #{result[:data][:exit_code]}. Disk space for creating snapshot exceeded.")
elsif result[:data][:exit_code] == 100
Astute.logger.error("#{ctx.task_id}: Not enough free space for logs. Decrease logs coefficient via CLI or config or free up space.")
report_error(ctx, "Timmy exit code: #{result[:data][:exit_code]}. Not enough free space for logs.")
else
Astute.logger.error("#{ctx.task_id}: Dump command returned non zero exit code. For details see #{log_file}")
report_error(ctx, "Timmy exit code: #{result[:data][:exit_code]}")
end
rescue Timeout::Error
msg = "Dump is timed out"
Astute.logger.error("#{ctx.task_id}: #{msg}")
report_error(ctx, msg)
rescue => e
msg = "Exception occured during dump task: message: #{e.message} \
trace:\n#{e.backtrace.pretty_inspect}"
Astute.logger.error("#{ctx.task_id}: #{msg}")
report_error(ctx, msg)
end
end
def self.report_success(ctx, msg=nil)
success_msg = {'status' => 'ready', 'progress' => 100}
success_msg.merge!({'msg' => msg}) if msg
ctx.reporter.report(success_msg)
end
def self.report_error(ctx, msg)
ctx.reporter.report({'status' => 'error', 'error' => msg, 'progress' => 100})
end
end
end

36
lib/astute/exceptions.rb
View File

@ -1,36 +0,0 @@
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
require 'timeout'
module Astute
# Base class for all errors
class AstuteError < StandardError; end
# Provisioning log errors
class ParseProvisionLogsError < AstuteError; end
# Image provisioning errors
class FailedImageProvisionError < AstuteError; end
# Deployment engine error
class DeploymentEngineError < AstuteError; end
# MClient errors
class MClientError < AstuteError; end
# MClient timeout error
class MClientTimeout < Timeout::Error; end
# Task validation error
class TaskValidationError < AstuteError; end
# Status error
class StatusValidationError < AstuteError; end
end

28
lib/astute/ext/array.rb
View File

@ -1,28 +0,0 @@
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
class Array
def compact_blank
reject do |val|
case val
when Hash then val.compact_blank.blank?
when Array then val.map { |v| v.respond_to?(:compact_blank) ? v.compact_blank : v }.blank?
when String then val.blank?
else val.blank?
end
end
end
end

18
lib/astute/ext/deep_copy.rb
View File

@ -1,18 +0,0 @@
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
def deep_copy(data)
Marshal.load(Marshal.dump(data))
end

20
lib/astute/ext/exception.rb
View File

@ -1,20 +0,0 @@
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
class Exception
def format_backtrace
"\n" + backtrace.pretty_inspect
end
end

32
lib/astute/ext/hash.rb
View File

@ -1,32 +0,0 @@
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
class Hash
def absent_keys(array)
array.select { |key| self[key].blank? }
end
def compact_blank
delete_if do |_key, val|
case val
when Hash then val.compact_blank.blank?
when Array then val.map { |v| v.respond_to?(:compact_blank) ? v.compact_blank : v }.blank?
when String then val.blank?
else val.blank?
end
end
end
end

160
lib/astute/image_provision.rb
View File

@ -1,160 +0,0 @@
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
module ImageProvision
def self.provision(ctx, nodes)
failed_uids = []
uids_to_provision, failed_uids = upload_provision(ctx, nodes)
run_provision(ctx, uids_to_provision, failed_uids)
rescue => e
msg = "Error while provisioning: message: #{e.message}" \
" trace\n: #{e.format_backtrace}"
Astute.logger.error("#{ctx.task_id}: #{msg}")
report_error(ctx, msg)
failed_uids
end
def self.upload_provision(ctx, nodes)
failed_uids = []
nodes.each do |node|
succees = upload_provision_data(ctx, node)
next if succees
failed_uids << node['uid']
Astute.logger.error("#{ctx.task_id}: Upload provisioning data " \
"failed on node #{node['uid']}. Provision on such node will " \
"not start")
end
uids_to_provision = nodes.select { |n| !failed_uids.include?(n['uid']) }
.map { |n| n['uid'] }
[uids_to_provision, failed_uids]
end
def self.upload_provision_data(ctx, node)
Astute.logger.debug("#{ctx.task_id}: uploading provision " \
"data on node #{node['uid']}: #{node.to_json}")
upload_task = Astute::UploadFile.new(
generate_upload_provision_task(node),
ctx
)
upload_task.sync_run
end
def self.generate_upload_provision_task(node)
{
"id" => 'upload_provision_data',
"node_id" => node['uid'],
"parameters" => {
"path" => '/tmp/provision.json',
"data" => node.to_json,
"user_owner" => 'root',
"group_owner" => 'root',
"overwrite" => true,
"timeout" => Astute.config.upload_timeout
}
}
end
def self.run_provision(ctx, uids, failed_uids)
Astute.logger.debug("#{ctx.task_id}: running provision script: " \
"#{uids.join(', ')}")
failed_uids |= run_shell_task(
ctx,
uids,
'flock -n /var/lock/provision.lock provision',
Astute.config.provisioning_timeout
)
failed_uids
end
def self.report_error(ctx, msg)
ctx.reporter.report({
'status' => 'error',
'error' => msg,
'progress' => 100
})
end
def self.reboot(ctx, node_ids, task_id="reboot_provisioned_nodes")
if node_ids.empty?
Astute.logger.warn("No nodes were sent to reboot for " \
"task: #{task_id}")
return
end
Astute::NailgunHooks.new(
[{
"priority" => 100,
"type" => "reboot",
"fail_on_error" => false,
"id" => task_id,
"uids" => node_ids,
"parameters" => {
"timeout" => Astute.config.reboot_timeout
}
}],
ctx,
'provision'
).process
end
def self.run_shell_task(ctx, node_uids, cmd, timeout=3600)
shell_tasks = node_uids.inject([]) do |tasks, node_id|
tasks << Shell.new(generate_shell_hook(node_id, cmd, timeout), ctx)
end
shell_tasks.each(&:run)
while shell_tasks.any? { |t| !t.finished? } do
shell_tasks.select { |t| !t.finished? }.each(&:status)
sleep Astute.config.task_poll_delay
end
failed_uids = shell_tasks.select{ |t| t.failed? }
.inject([]) do |failed_nodes, task|
Astute.logger.error("#{ctx.task_id}: Provision command returned " \
"non zero exit code on node: #{task.node_id}")
failed_nodes << task.node_id
end
failed_uids
rescue => e
Astute.logger.error("#{ctx.task_id}: cmd: #{cmd} " \
"error: #{e.message}, trace #{e.backtrace}")
node_uids
end
def self.generate_shell_hook(node_id, cmd, timeout)
{
"node_id" => node_id,
"id" => "provision_#{node_id}",
"parameters" => {
"cmd" => cmd,
"cwd" => "/",
"timeout" => timeout,
"retries" => 0
}
}
end
end
end

261
lib/astute/logparser.rb
View File

@ -1,261 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
require 'erb'
module Astute
module LogParser
LOG_PORTION = 10000
# Default values. Can be overrided by pattern_spec.
# E.g. pattern_spec = {'separator' => 'new_separator', ...}
PATH_PREFIX = '/var/log/remote/'
SEPARATOR = "SEPARATOR\n"
class NoParsing
def initialize(*args)
end
def method_missing(*args)
# We just eat the call if we don't want to deal with logs
end
def progress_calculate(*args)
[]
end
end
class DirSizeCalculation
attr_reader :nodes
def initialize(nodes)
@nodes = nodes.map{|n| n.dup}
@nodes.each{|node| node[:path_items] = weight_reassignment(node[:path_items])}
end
def deploy_type=(*args)
# Because we mimic the DeploymentParser, we should define all auxiliary method
# even they do nothing.
end
def prepare(nodes)
# Because we mimic the DeploymentParser, we should define all auxiliary method
# even they do nothing.
end
def progress_calculate(uids_to_calc, nodes)
uids_to_calc.map do |uid|
node = @nodes.find{|n| n[:uid] == uid}
node[:path_items] ||= []
progress = 0
node[:path_items].each do |item|
size = recursive_size(item[:path])
sub_progress = 100 * size / item[:max_size]
sub_progress = 0 if sub_progress < 0
sub_progress = 100 if sub_progress > 100
progress += sub_progress * item[:weight]
end
{'uid' => uid, 'progress' => progress.to_i}
end
end
private
def recursive_size(path, opts={})
return File.size?(path).to_i if not File.directory?(path)
total_size = 0
Dir[File.join("#{path}", '**/*')].each do |f|
# Option :files_only used when you want to calculate total size of
# regular files only. The default :files_only is false, so the function will
# include inode size of each dir (4096 bytes in most cases) to total value
# as the unix util 'du' does it.
total_size += File.size?(f).to_i if File.file?(f) || ! opts[:files_only]
end
total_size
end
def weight_reassignment(items)
# The finction normalizes the weights of each item in order to make sum of
# all weights equal to one.
# It divides items as wighted and unweighted depending on the existence of
# the :weight key in the item.
# - Each unweighted item will be weighted as a one N-th part of the total number of items.
# - All weights of weighted items are summed up and then each weighted item
# gets a new weight as a multiplication of a relative weight among all
# weighted items and the ratio of the number of the weighted items to
# the total number of items.
# E.g. we have four items: one with weight 0.5, another with weight 1.5, and
# two others as unweighted. All unweighted items will get the weight 1/4.
# Weight's sum of weighted items is 2. So the first item will get the weight:
# (relative weight 0.5/2) * (weighted items ratio 2/4) = 1/8.
# Finally all items will be normalised with next weights:
# 1/8, 3/8, 1/4, and 1/4.
ret_items = items.reject do |item|
weight = item[:weight]
# Save an item if it unweighted.
next if weight.nil?
raise "Weight should be a non-negative number" unless [Fixnum, Float].include?(weight.class) && weight >= 0
# Drop an item if it weighted as zero.
item[:weight] == 0
end
return [] if ret_items.empty?
ret_items.map!{|n| n.dup}
partial_weight = 1.0 / ret_items.length
weighted_items = ret_items.select{|n| n[:weight]}
weighted_sum = 0.0
weighted_items.each{|n| weighted_sum += n[:weight]}
weighted_sum = weighted_sum * ret_items.length / weighted_items.length if weighted_items.any?
raise "Unexpectedly a summary weight of weighted items is a non-positive" if weighted_items.any? && weighted_sum <= 0
ret_items.each do |item|
weight = item[:weight]
item[:weight] = weight ? weight / weighted_sum : partial_weight
end
ret_items
end
end
class ParseNodeLogs
attr_reader :pattern_spec
def initialize
@pattern_spec = {}
@pattern_spec['path_prefix'] ||= PATH_PREFIX.to_s
@pattern_spec['separator'] ||= SEPARATOR.to_s
@nodes_patterns = {}
end
def progress_calculate(uids_to_calc, nodes)
nodes_progress = []
patterns = patterns_for_nodes(nodes, uids_to_calc)
uids_to_calc.each do |uid|
node = nodes.find {|n| n['uid'] == uid}
@nodes_patterns[uid] ||= patterns[uid]
node_pattern_spec = @nodes_patterns[uid]
# FIXME(eli): this var is required for binding() below
@pattern_spec = @nodes_patterns[uid]
erb_path = node_pattern_spec['path_format']
path = ERB.new(erb_path).result(binding())
progress = 0
begin
# Return percent of progress
progress = (get_log_progress(path, node_pattern_spec) * 100).to_i
rescue => e
Astute.logger.warn "Some error occurred when calculate progress " \
"for node '#{uid}': #{e.message}, trace: #{e.format_backtrace}"
end
nodes_progress << {
'uid' => uid,
'progress' => progress
}
end
nodes_progress
end
def prepare(nodes)
patterns = patterns_for_nodes(nodes)
nodes.each do |node|
pattern = patterns[node['uid']]
path = "#{pattern['path_prefix']}#{node['ip']}/#{pattern['filename']}"
File.open(path, 'a') { |fo| fo.write pattern['separator'] } if File.writable?(path)
end
end
# Get patterns for selected nodes
# if uids_to_calc is nil, then
# patterns for all nodes will be returned
def patterns_for_nodes(nodes, uids_to_calc=nil)
uids_to_calc = nodes.map { |node| node['uid'] } if uids_to_calc.nil?
nodes_to_calc = nodes.select { |node| uids_to_calc.include?(node['uid']) }
patterns = {}
nodes_to_calc.map do |node|
patterns[node['uid']] = get_pattern_for_node(node)
end
patterns
end
private
def get_log_progress(path, node_pattern_spec)
unless File.readable?(path)
Astute.logger.debug "Can't read file with logs: #{path}"
return 0
end
if node_pattern_spec.nil?
Astute.logger.warn "Can't parse logs. Pattern_spec is empty."
return 0
end
progress = nil
File.open(path) do |fo|
# Try to find well-known ends of log.
endlog = find_endlog_patterns(fo, node_pattern_spec)
return endlog if endlog
# Start reading from end of file.
fo.pos = fo.stat.size
# Method 'calculate' should be defined at child classes.
progress = calculate(fo, node_pattern_spec)
node_pattern_spec['file_pos'] = fo.pos
end
unless progress
Astute.logger.warn("Wrong pattern\n#{node_pattern_spec.pretty_inspect}\ndefined for calculating progress via logs.")
return 0
end
progress
end
def find_endlog_patterns(fo, pattern_spec)
# Pattern example:
# pattern_spec = {...,
# 'endlog_patterns' => [{'pattern' => /Finished catalog run in [0-9]+\.[0-9]* seconds\n/, 'progress' => 1.0}],
# }
endlog_patterns = pattern_spec['endlog_patterns']
return nil unless endlog_patterns
fo.pos = fo.stat.size
chunk = get_chunk(fo, 100)
return nil unless chunk
endlog_patterns.each do |pattern|
return pattern['progress'] if Regexp.new("#{pattern['pattern']}$").match(chunk)
end
nil
end
def get_chunk(fo, size=nil, pos=nil)
if pos
fo.pos = pos
return fo.read
end
size = LOG_PORTION unless size
return nil if fo.pos == 0
size = fo.pos if fo.pos < size
next_pos = fo.pos - size
fo.pos = next_pos
block = fo.read(size)
fo.pos = next_pos
block
end
end
end
end

160
lib/astute/logparser/deployment.rb
View File

@ -1,160 +0,0 @@
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
module LogParser
class ParseDeployLogs < ParseNodeLogs
attr_reader :deploy_type
def deploy_type=(deploy_type)
@deploy_type = deploy_type
@nodes_patterns = {}
end
def get_pattern_for_node(node)
role = node['role']
node_pattern = Patterns::get_default_pattern(
"puppet-log-components-list-#{@deploy_type}-#{role}")
node_pattern['path_prefix'] ||= PATH_PREFIX.to_s
node_pattern['separator'] ||= SEPARATOR.to_s
node_pattern
end
private
def calculate(fo, node_pattern_spec)
case node_pattern_spec['type']
when 'count-lines'
progress = simple_line_counter(fo, node_pattern_spec)
when 'components-list'
progress = component_parser(fo, node_pattern_spec)
end
return progress
end
def simple_line_counter(fo, pattern_spec)
# Pattern specification example:
# pattern_spec = {'type' => 'count-lines',
# 'endlog_patterns' => [{'pattern' => /Finished catalog run in [0-9]+\.[0-9]* seconds\n/, 'progress' => 1.0}],
# 'expected_line_number' => 500}
# Use custom separator if defined.
separator = pattern_spec['separator']
counter = 0
end_of_scope = false
previous_subchunk = ''
until end_of_scope
chunk = get_chunk(fo, pattern_spec['chunk_size'])
break unless chunk
# Trying to find separator on border between chunks.
subchunk = chunk.slice((1-separator.size)..-1)
# End of file reached. Exit from cycle.
end_of_scope = true unless subchunk
if subchunk and (subchunk + previous_subchunk).include?(separator)
# Separator found on border between chunks. Exit from cycle.
end_of_scope = true
continue
end
pos = chunk.rindex(separator)
if pos
end_of_scope = true
chunk = chunk.slice((pos + separator.size)..-1)
end
counter += chunk.count("\n")
end
number = pattern_spec['expected_line_number']
unless number
Astute.logger.warn("Wrong pattern\n#{pattern_spec.pretty_inspect} defined for calculating progress via log.")
return 0
end
progress = counter.to_f / number
progress = 1 if progress > 1
return progress
end
def component_parser(fo, pattern_spec)
# Pattern specification example:
# pattern_spec = {'type' => 'components-list',
# 'chunk_size' => 40000,
# 'components_list' => [
# {'name' => 'Horizon', 'weight' => 10, 'patterns' => [
# {'pattern' => '/Stage[main]/Horizon/Package[mod_wsgi]/ensure) created', 'progress' => 0.1},
# {'pattern' => '/Stage[main]/Horizon/File_line[horizon_redirect_rule]/ensure) created', 'progress' => 0.3},
# {'pattern' => '/Stage[main]/Horizon/File[/etc/openstack-dashboard/local_settings]/group)', 'progress' => 0.7},
# {'pattern' => '/Stage[main]/Horizon/Service[$::horizon::params::http_service]/ensure)'\
# ' ensure changed \'stopped\' to \'running\'', 'progress' => 1},
# ]
# },
# ]
# }
# Use custom separator if defined.
separator = pattern_spec['separator']
components_list = pattern_spec['components_list']
unless components_list
Astute.logger.warn("Wrong pattern\n#{pattern_spec.pretty_inspect} defined for calculating progress via logs.")
return 0
end
chunk = get_chunk(fo, pos=pattern_spec['file_pos'])
return 0 unless chunk
pos = chunk.rindex(separator)
chunk = chunk.slice((pos + separator.size)..-1) if pos
block = chunk.split("\n")
# Update progress of each component.
while block.any?
string = block.pop
components_list.each do |component|
matched_pattern = nil
component['patterns'].each do |pattern|
if pattern['regexp']
matched_pattern = pattern if string.match(pattern['pattern'])
else
matched_pattern = pattern if string.include?(pattern['pattern'])
end
break if matched_pattern
end
if matched_pattern and
(not component['_progress'] or matched_pattern['progress'] > component['_progress'])
component['_progress'] = matched_pattern['progress']
end
end
end
# Calculate integral progress.
weighted_components = components_list.select{|n| n['weight']}
weight_sum = 0.0
if weighted_components.any?
weighted_components.each{|n| weight_sum += n['weight']}
weight_sum = weight_sum * components_list.length / weighted_components.length
raise "Total weight of weighted components equal to zero." if weight_sum == 0
end
nonweighted_delta = 1.0 / components_list.length
progress = 0
components_list.each do |component|
component['_progress'] = 0.0 unless component['_progress']
weight = component['weight']
if weight
progress += component['_progress'] * weight / weight_sum
else
progress += component['_progress'] * nonweighted_delta
end
end
return progress
end
end
end
end

75
lib/astute/logparser/parser_patterns.rb
View File

@ -1,75 +0,0 @@
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
module LogParser
module Patterns
def self.get_default_pattern(key)
pattern_key = key
pattern_key = 'default' unless @default_patterns.has_key?(key)
deep_copy(@default_patterns[pattern_key])
end
def self.list_default_patterns
return @default_patterns.keys
end
@default_patterns = {
'provisioning-image-building' =>
{'type' => 'supposed-time',
'chunk_size' => 10000,
'date_format' => '%Y-%m-%d %H:%M:%S',
'date_regexp' => '^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}',
'pattern_list' => [
{'pattern' => '--- Building image (do_build_image) ---', 'supposed_time' => 12},
{'pattern' => '*** Shipping image content ***', 'supposed_time' => 12},
{'pattern' => 'Running deboostrap completed', 'supposed_time' => 270},
{'pattern' => 'Running apt-get install completed', 'supposed_time' => 480},
{'pattern' => '--- Building image END (do_build_image) ---', 'supposed_time' => 240},
{'pattern' => 'All necessary images are available.', 'supposed_time' => 10}
].reverse,
'filename' => "fuel-agent-env",
'path_format' => "<%= @pattern_spec['path_prefix']%><%= @pattern_spec['filename']%>-<%= @pattern_spec['cluster_id']%>.log"
},
'image-based-provisioning' =>
{'type' => 'pattern-list',
'chunk_size' => 10000,
'pattern_list' => [
{'pattern' => '--- Provisioning (do_provisioning) ---', 'progress' => 0.81},
{'pattern' => '--- Partitioning disks (do_partitioning) ---', 'progress' => 0.82},
{'pattern' => '--- Creating configdrive (do_configdrive) ---', 'progress' => 0.92},
{'pattern' => 'Next chunk',
'number' => 600,
'p_min' => 0.92,
'p_max' => 0.98},
{'pattern' => '--- Installing bootloader (do_bootloader) ---', 'progress' => 0.99},
{'pattern' => '--- Provisioning END (do_provisioning) ---', 'progress' => 1}
],
'filename' => 'bootstrap/fuel-agent.log',
'path_format' => "<%= @pattern_spec['path_prefix'] %><%= node['hostname'] %>/<%= @pattern_spec['filename'] %>",
},
'default' => {
'type' => 'count-lines',
'endlog_patterns' => [{'pattern' => /Finished catalog run in [0-9]+\.[0-9]* seconds\n/, 'progress' => 1.0}],
'expected_line_number' => 345,
'filename' => 'puppet-apply.log',
'path_format' => "<%= @pattern_spec['path_prefix'] %><%= node['fqdn'] %>/<%= @pattern_spec['filename'] %>"
},
}
end
end
end

262
lib/astute/logparser/provision.rb
View File

@ -1,262 +0,0 @@
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
require 'date'
module Astute
module LogParser
class ParseProvisionLogs < ParseNodeLogs
def get_pattern_for_node(node)
os = node['profile']
pattern_spec_name = if node.fetch('ks_meta', {}).key?('image_data')
'image-based-provisioning'
elsif ['centos-x86_64'].include?(os)
'centos-anaconda-log-supposed-time-kvm'
elsif os == 'ubuntu_1404_x86_64'
'ubuntu-provisioning'
else
raise Astute::ParseProvisionLogsError, "Cannot find profile for os with: #{os}"
end
pattern_spec = deep_copy(Patterns::get_default_pattern(pattern_spec_name))
pattern_spec['path_prefix'] ||= PATH_PREFIX.to_s
pattern_spec['separator'] ||= SEPARATOR.to_s
pattern_spec
end
private
def calculate(fo, node_pattern_spec)
case node_pattern_spec['type']
when 'pattern-list'
progress = simple_pattern_finder(fo, node_pattern_spec)
when 'supposed-time'
progress = supposed_time_parser(fo, node_pattern_spec)
end
progress
end
# Pattern specification example:
# pattern_spec = {'type' => 'supposed-time',
# 'chunk_size' => 10000,
# 'date_format' => '%Y-%m-%dT%H:%M:%S',
# 'date_regexp' => '^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}',
# 'pattern_list' => [
# {'pattern' => 'Running anaconda script', 'supposed_time' => 60},
# ....
# {'pattern' => 'leaving (1) step postscripts', 'supposed_time' => 130},
# ].reverse,
# 'filename' => 'install/anaconda.log'
# }
# Use custom separator if defined.
def supposed_time_parser(fo, pattern_spec)
separator = pattern_spec['separator']
log_patterns = pattern_spec['pattern_list']
date_format = pattern_spec['date_format']
date_regexp = pattern_spec['date_regexp']
unless date_regexp and date_format and log_patterns
Astute.logger.warn("Wrong pattern_spec\n#{pattern_spec.pretty_inspect} defined for calculating progress via logs.")
return 0
end
def self.get_elapsed_time(patterns)
elapsed_time = 0
patterns.each do |p|
if p['_progress']
break
else
elapsed_time += p['supposed_time']
end
end
return elapsed_time
end
def self.get_progress(base_progress, elapsed_time, delta_time, supposed_time=nil)
return 1.0 if elapsed_time.zero?
k = (1.0 - base_progress) / elapsed_time
supposed_time ? surplus = delta_time - supposed_time : surplus = nil
if surplus and surplus > 0
progress = supposed_time * k + surplus * k/3 + base_progress
else
progress = delta_time * k + base_progress
end
progress = 1.0 if progress > 1
return progress
end
def self.get_seconds_from_time(date)
hours, mins, secs, _frac = Date::day_fraction_to_time(date)
return hours*60*60 + mins*60 + secs
end
chunk = get_chunk(fo, pattern_spec['chunk_size'])
return 0 unless chunk
pos = chunk.rindex(separator)
chunk = chunk.slice((pos + separator.size)..-1) if pos
block = chunk.split("\n")
now = DateTime.now()
prev_time = pattern_spec['_prev_time'] ||= now
prev_progress = pattern_spec['_prev_progress'] ||= 0
elapsed_time = pattern_spec['_elapsed_time'] ||= get_elapsed_time(log_patterns)
seconds_since_prev = get_seconds_from_time(now - prev_time)
until block.empty?
string = block.pop
log_patterns.each do |pattern|
if string.include?(pattern['pattern'])
if pattern['_progress']
# We not found any new messages. Calculate progress with old data.
progress = get_progress(prev_progress, elapsed_time,
seconds_since_prev, pattern['supposed_time'])
return progress
else
# We found message that we never find before. We need to:
# calculate progress for this message;
# recalculate control point and elapsed_time;
# calculate progress for current time.
# Trying to find timestamp of message.
date_string = string.match(date_regexp)
if date_string
# Get relative time when the message realy occured.
date = DateTime.strptime(date_string[0], date_format) - prev_time.offset
real_time = get_seconds_from_time(date - prev_time)
# Update progress of the message.
prev_supposed_time = log_patterns.select{|n| n['_progress'] == prev_progress}[0]
prev_supposed_time = prev_supposed_time['supposed_time'] if prev_supposed_time
progress = get_progress(prev_progress, elapsed_time, real_time, prev_supposed_time)
pattern['_progress'] = progress
# Recalculate elapsed time.
elapsed_time = pattern_spec['_elapsed_time'] = get_elapsed_time(log_patterns)
# Update time and progress for control point.
prev_time = pattern_spec['_prev_time'] = date
prev_progress = pattern_spec['_prev_progress'] = progress
seconds_since_prev = get_seconds_from_time(now - date)
# Calculate progress for current time.
progress = get_progress(prev_progress, elapsed_time,
seconds_since_prev, pattern['supposed_time'])
return progress
else
Astute.logger.info("Can't gather date (format: '#{date_regexp}') from string: #{string}")
end
end
end
end
end
# We found nothing.
progress = get_progress(prev_progress, elapsed_time, seconds_since_prev, log_patterns[0]['supposed_time'])
return progress
end
def simple_pattern_finder(fo, pattern_spec)
# Pattern specification example:
# pattern_spec = {'type' => 'pattern-list', 'separator' => "custom separator\n",
# 'chunk_size' => 40000,
# 'pattern_list' => [
# {'pattern' => 'Running kickstart %%pre script', 'progress' => 0.08},
# {'pattern' => 'to step enablefilesystems', 'progress' => 0.09},
# {'pattern' => 'to step reposetup', 'progress' => 0.13},
# {'pattern' => 'to step installpackages', 'progress' => 0.16},
# {'pattern' => 'Installing',
# 'number' => 210, # Now it install 205 packets. Add 5 packets for growth in future.
# 'p_min' => 0.16, # min percent
# 'p_max' => 0.87 # max percent
# },
# {'pattern' => 'to step postinstallconfig', 'progress' => 0.87},
# {'pattern' => 'to step dopostaction', 'progress' => 0.92},
# ]
# }
# Use custom separator if defined.
separator = pattern_spec['separator']
log_patterns = pattern_spec['pattern_list']
unless log_patterns
Astute.logger.warn("Wrong pattern\n#{pattern_spec.pretty_inspect} defined for calculating progress via logs.")
return 0
end
chunk = get_chunk(fo, pattern_spec['chunk_size'])
# NOTE(mihgen): Following line fixes "undefined method `rindex' for nil:NilClass" for empty log file
return 0 unless chunk
pos = chunk.rindex(separator)
chunk = chunk.slice((pos + separator.size)..-1) if pos
block = chunk.split("\n")
return 0 unless block
while true
string = block.pop
return 0 unless string # If we found nothing
log_patterns.each do |pattern|
if string.include?(pattern['pattern'])
return pattern['progress'] if pattern['progress']
if pattern['number']
string = block.pop
counter = 1
while string
counter += 1 if string.include?(pattern['pattern'])
string = block.pop
end
progress = counter.to_f / pattern['number']
progress = 1 if progress > 1
progress = pattern['p_min'] + progress * (pattern['p_max'] - pattern['p_min'])
return progress
end
Astute.logger.warn("Wrong pattern\n#{pattern_spec.pretty_inspect} defined for calculating progress via log.")
end
end
end
end
end # ParseProvisionLogs
class ParseImageBuildLogs < ParseProvisionLogs
PATH_PREFIX = '/var/log/'
attr_accessor :cluster_id
def get_pattern_for_node(node)
os = node['profile']
pattern_spec_name = 'provisioning-image-building'
pattern_spec = deep_copy(Patterns::get_default_pattern(pattern_spec_name))
pattern_spec['path_prefix'] ||= PATH_PREFIX.to_s
pattern_spec['separator'] ||= SEPARATOR.to_s
pattern_spec['cluster_id'] = cluster_id
pattern_spec
end
def prepare(nodes)
# This is common file for all nodes
pattern_spec = get_pattern_for_node(nodes.first)
path = pattern_spec['path_format']
File.open(path, 'a') { |fo| fo.write pattern['separator'] } if File.writable?(path)
end
def progress_calculate(uids_to_calc, nodes)
result = super
# Limit progress for this part to 80% as max
result.map { |h| h['progress'] = (h['progress'] * 0.8).to_i }
result
end
end # ParseImageProvisionLogs
end
end

190
lib/astute/mclient.rb
View File

@ -1,190 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
require 'mcollective'
require 'timeout'
module Astute
class MClient
include MCollective::RPC
attr_accessor :retries
def initialize(
ctx,
agent,
nodes=nil,
check_result=true,
timeout=nil,
retries=Astute.config.mc_retries,
enable_result_logging=true
)
@task_id = ctx.task_id
@agent = agent
@nodes = nodes.map { |n| n.to_s } if nodes
@check_result = check_result
# Will be used a minimum of two things: the specified parameter(timeout)
# and timeout from DDL (10 sec by default if not explicitly specified in DDL)
# If timeout here is nil will be used value from DDL.
# Examples:
# timeout - 10 sec, DDL - 20 sec. Result — 10 sec.
# timeout - 30 sec, DDL - 20 sec. Result — 20 sec.
# timeout - 20 sec, DDL - not set. Result — 10 sec.
@timeout = timeout
@retries = retries
@enable_result_logging = enable_result_logging
initialize_mclient
end
def on_respond_timeout(&block)
@on_respond_timeout = block
self
end
def method_missing(method, *args)
@mc_res = mc_send(method, *args)
if method == :discover
@nodes = args[0][:nodes]
return @mc_res
end
# Enable if needed. In normal case it eats the screen pretty fast
log_result(@mc_res, method) if @enable_result_logging
check_results_with_retries(method, args) if @check_result
@mc_res
end
private
def check_results_with_retries(method, args)
err_msg = ''
timeout_nodes_count = 0
# Following error might happen because of misconfiguration, ex. direct_addressing = 1 only on client
# or.. could be just some hang? Let's retry if @retries is set
if @mc_res.length < @nodes.length
# some nodes didn't respond
retry_index = 1
while retry_index <= @retries
sleep rand
nodes_responded = @mc_res.map { |n| n.results[:sender] }
not_responded = @nodes - nodes_responded
Astute.logger.debug "Retry ##{retry_index} to run mcollective agent on nodes: '#{not_responded.join(',')}'"
mc_send :discover, :nodes => not_responded
@new_res = mc_send(method, *args)
log_result(@new_res, method) if @enable_result_logging
# @new_res can have some nodes which finally responded
@mc_res += @new_res
break if @mc_res.length == @nodes.length
retry_index += 1
end
if @mc_res.length < @nodes.length
nodes_responded = @mc_res.map { |n| n.results[:sender] }
not_responded = @nodes - nodes_responded
if @on_respond_timeout
@on_respond_timeout.call not_responded
else
err_msg += "MCollective agents '#{@agent}' " \
"'#{not_responded.join(',')}' didn't respond within the " \
"allotted time.\n"
timeout_nodes_count += not_responded.size
end
end
end
failed = @mc_res.select { |x| x.results[:statuscode] != 0 }
if failed.any?
err_msg += "MCollective call failed in agent '#{@agent}', "\
"method '#{method}', failed nodes: \n"
failed.each do |n|
err_msg += "ID: #{n.results[:sender]} - Reason: #{n.results[:statusmsg]}\n"
end
end
if err_msg.present?
Astute.logger.error err_msg
expired_size = failed.count { |n| n.results[:statusmsg] == 'execution expired' }
# Detect TimeOut: 1 condition - fail because of DDL timeout, 2 - fail because of custom timeout
if (failed.present? && failed.size == expired_size) || (timeout_nodes_count > 0 && failed.empty?)
raise MClientTimeout, "#{@task_id}: #{err_msg}"
else
raise MClientError, "#{@task_id}: #{err_msg}"
end
end
end
def mc_send(*args)
retries = 1
begin
@mc.send(*args)
rescue => ex
case ex
when Stomp::Error::NoCurrentConnection
# stupid stomp cannot recover severed connection
stomp = MCollective::PluginManager["connector_plugin"]
stomp.disconnect rescue nil
stomp.instance_variable_set :@connection, nil
initialize_mclient
end
if retries < 3
Astute.logger.error "Retrying MCollective call after exception:\n#{ex.pretty_inspect}"
sleep rand
retries += 1
retry
else
Astute.logger.error "No more retries for MCollective call after exception: " \
"#{ex.format_backtrace}"
raise MClientError, "#{ex.pretty_inspect}"
end
end
end
def initialize_mclient
retries = 1
begin
@mc = rpcclient(@agent, :exit_on_failure => false)
@mc.timeout = @timeout if @timeout
@mc.progress = false
if @nodes
@mc.discover :nodes => @nodes
end
rescue => ex
if retries < 3
Astute.logger.error "Retrying RPC client instantiation after exception:\n#{ex.pretty_inspect}"
sleep 5
retries += 1
retry
else
Astute.logger.error "No more retries for MCollective client instantiation after exception: " \
"#{ex.format_backtrace}"
raise MClientError, "#{ex.pretty_inspect}"
end
end
end
def log_result(result, method)
result.each do |node|
Astute.logger.debug "#{@task_id}: MC agent '#{node.agent}', method '#{method}', "\
"results:\n#{node.results.pretty_inspect}"
end
end
end
end

138
lib/astute/mclients/puppet_mclient.rb
View File

@ -1,138 +0,0 @@
# Copyright 2016 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class PuppetMClient
PUPPET_STATUSES = [
'running', 'stopped', 'disabled'
]
attr_reader :summary, :node_id
def initialize(ctx, node_id, options)
@ctx = ctx
@node_id = node_id
@options = options
@summary = {}
end
# Return actual status of puppet using mcollective puppet agent
# @return [String] status: succeed, one of PUPPET_STATUSES or undefined
def status
last_run_summary
succeed? ? 'succeed' : @summary.fetch(:status, 'undefined')
end
# Run puppet on node if available
# @return [true, false]
def run
is_succeed, err_msg = runonce
return true if is_succeed
Astute.logger.warn "Fail to start puppet on node #{@node_id}. "\
"Reason: #{err_msg}"
false
end
# Return path to manifest using by mcollective puppet agent
# @return [String] path to manifest
def manifest
File.join(@options['cwd'], @options['puppet_manifest'])
end
private
# Create configured puppet mcollective agent
# @return [Astute::MClient]
def puppetd(timeout=nil, retries=1)
puppetd = MClient.new(
@ctx,
"puppetd",
[@node_id],
_check_result=true,
_timeout=timeout,
_retries=retries,
_enable_result_logging=false
)
puppetd.on_respond_timeout do |uids|
msg = "Nodes #{uids} reached the response timeout"
Astute.logger.error msg
raise MClientTimeout, msg
end
puppetd
end
# Run last_run_summary action using mcollective puppet agent
# @return [Hash] return hash with status and resources
def last_run_summary
@summary = puppetd(_timeout=10, _retries=6).last_run_summary(
:puppet_noop_run => @options['puppet_noop_run'],
:raw_report => @options['raw_report']
).first[:data]
validate_status!(@summary[:status])
@summary
rescue MClientError, MClientTimeout => e
Astute.logger.warn "Unable to get actual status of puppet on "\
"node #{@node_id}. Reason: #{e.message}"
@summary = {}
end
# Run runonce action using mcollective puppet agent
# @return [[true, false], String] boolean status of run and error message
def runonce
result = puppetd.runonce(
:puppet_debug => @options['puppet_debug'],
:manifest => @options['puppet_manifest'],
:modules => @options['puppet_modules'],
:cwd => @options['cwd'],
:command_prefix => @options['command_prefix'],
:puppet_noop_run => @options['puppet_noop_run'],
).first
return result[:statuscode] == 0, result[:statusmsg]
rescue MClientError, MClientTimeout => e
return false, e.message
end
# Validate puppet status
# @param [String] status The puppet status
# @return [void]
# @raise [MClientError] Unknown status
def validate_status!(status)
unless PUPPET_STATUSES.include?(status)
raise MClientError, "Unknow status '#{status}' from mcollective agent"
end
end
# Detect succeed of puppet run using summary from last_run_summary call
# @return [true, false]
def succeed?
return false if @summary.blank?
@summary[:status] == 'stopped' &&
@summary[:resources] &&
@summary[:resources]['failed'].to_i == 0 &&
@summary[:resources]['failed_to_restart'].to_i == 0
end
# Generate shell cmd for file deletion
# @return [String] shell cmd for deletion
def rm_cmd
PUPPET_FILES_TO_CLEANUP.inject([]) do
|cmd, file| cmd << "rm -f #{file}"
end.join(" && ")
end
end # PuppetMclient
end

84
lib/astute/mclients/shell_mclient.rb
View File

@ -1,84 +0,0 @@
# Copyright 2016 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class ShellMClient
def initialize(ctx, node_id)
@ctx = ctx
@node_id = node_id
end
# Run shell cmd without check using mcollective agent
# @param [String] cmd Shell command for run
# @param [Integer] timeout Timeout for shell command
# @return [Hash] shell result
def run_without_check(cmd, timeout=2)
Astute.logger.debug("Executing shell command without check: "\
"#{details_for_log(cmd, timeout)}")
results = shell(_check_result=false, timeout).execute(:cmd => cmd)
Astute.logger.debug("Mcollective shell #{details_for_log(cmd, timeout)}"\
" result: #{results.pretty_inspect}")
if results.present?
result = results.first
log_result(result, cmd, timeout)
{
:stdout => result.results[:data][:stdout].chomp,
:stderr => result.results[:data][:stderr].chomp,
:exit_code => result.results[:data][:exit_code]
}
else
Astute.logger.warn("#{@ctx.task_id}: Failed to run shell "\
"#{details_for_log(cmd, timeout)}. Error will not raise "\
"because shell was run without check")
{}
end
end
private
# Create configured shell mcollective agent
# @return [Astute::MClient]
def shell(check_result=false, timeout=2)
MClient.new(
@ctx,
'execute_shell_command',
[@node_id],
check_result,
timeout
)
end
# Return short useful info about node and shell task
# @return [String] detail info about cmd
def details_for_log(cmd, timeout)
"command '#{cmd}' on node #{@node_id} with timeout #{timeout}"
end
# Write to log shell command result including exit code
# @param [Hash] result Actual magent shell result
# @return [void]
def log_result(result, cmd, timeout)
return if result.results[:data].blank?
Astute.logger.debug(
"#{@ctx.task_id}: #{details_for_log(cmd, timeout)}\n" \
"stdout: #{result.results[:data][:stdout]}\n" \
"stderr: #{result.results[:data][:stderr]}\n" \
"exit code: #{result.results[:data][:exit_code]}")
end
end
end

133
lib/astute/mclients/upload_file_mclient.rb
View File

@ -1,133 +0,0 @@
# Copyright 2016 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class UploadFileMClient
attr_reader :ctx, :node_id
def initialize(ctx, node_id)
@ctx = ctx
@node_id = node_id
end
# Run upload without check using mcollective agent
# @param [Hash] mco_params Upload file options
# @return [true, false] upload result
def upload_without_check(mco_params)
upload_mclient = upload_mclient(
:check_result => false,
:timeout => mco_params['timeout']
)
upload(mco_params, upload_mclient)
end
# Run upload with check using mcollective agent
# @param [Hash] mco_params Upload file options
# @return [true, false] upload result
def upload_with_check(mco_params)
upload_mclient = upload_mclient(
:check_result => false,
:timeout => mco_params['timeout'],
:retries => mco_params['retries']
)
process_with_retries(:retries => mco_params['retries']) do
upload(mco_params, upload_mclient)
end
end
private
def upload(mco_params, magent)
mco_params = setup_default(mco_params)
results = magent.upload(
:path => mco_params['path'],
:content => mco_params['content'],
:overwrite => mco_params['overwrite'],
:parents => mco_params['parents'],
:permissions => mco_params['permissions'],
:user_owner => mco_params['user_owner'],
:group_owner => mco_params['group_owner'],
:dir_permissions => mco_params['dir_permissions']
)
if results.present? && results.first[:statuscode] == 0
Astute.logger.debug("#{ctx.task_id}: file was uploaded "\
"#{details_for_log(mco_params)} successfully")
true
else
Astute.logger.error("#{ctx.task_id}: file was not uploaded "\
"#{details_for_log(mco_params)}: "\
"#{results.present? ? results.first[:msg] : "node has not answered" }")
false
end
rescue MClientTimeout, MClientError => e
Astute.logger.error("#{ctx.task_id}: file was not uploaded "\
"#{details_for_log(mco_params)}: #{e.message}")
false
end
# Create configured shell mcollective agent
# @return [Astute::MClient]
def upload_mclient(args={})
MClient.new(
ctx,
"uploadfile",
[node_id],
args.fetch(:check_result, false),
args.fetch(:timeout, 2),
args.fetch(:retries, Astute.config.upload_retries)
)
end
# Setup default value for upload mcollective agent
# @param [Hash] mco_params Upload file options
# @return [Hash] mco_params
def setup_default(mco_params)
mco_params['retries'] ||= Astute.config.upload_retries
mco_params['timeout'] ||= Astute.config.upload_timeout
mco_params['overwrite'] = true if mco_params['overwrite'].nil?
mco_params['parents'] = true if mco_params['parents'].nil?
mco_params['permissions'] ||= '0644'
mco_params['user_owner'] ||= 'root'
mco_params['group_owner'] ||= 'root'
mco_params['dir_permissions'] ||= '0755'
mco_params
end
# Return short useful info about node and shell task
# @return [String] detail info about upload task
def details_for_log(mco_params)
"#{mco_params['path']} on node #{node_id} "\
"with timeout #{mco_params['timeout']}"
end
def process_with_retries(args={}, &block)
retries = args.fetch(:retries, 1) + 1
result = false
retries.times do |attempt|
result = block.call
break if result
Astute.logger.warn("#{ctx.task_id} Upload retry for node "\
"#{node_id}: attempt № #{attempt + 1}/#{retries}")
end
result
end
end
end

467
lib/astute/nailgun_hooks.rb
View File

@ -1,467 +0,0 @@
# Copyright 2014 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class NailgunHooks
def initialize(nailgun_hooks, context, type='deploy')
@nailgun_hooks = nailgun_hooks
@ctx = context
@type = type
end
def process
@nailgun_hooks.sort_by { |f| f['priority'] }.each do |hook|
Astute.logger.debug "Run hook #{hook.to_yaml}"
time_start = Time.now.to_i
hook_return = case hook['type']
when 'copy_files' then copy_files_hook(hook)
when 'upload_files' then upload_files_hook(hook)
when 'sync' then sync_hook(hook)
when 'shell' then shell_hook(hook)
when 'upload_file' then upload_file_hook(hook)
when 'puppet' then puppet_hook(hook)
when 'reboot' then reboot_hook(hook)
when 'cobbler_sync' then cobbler_sync_hook(hook)
else raise "Unknown hook type #{hook['type']}"
end
time_summary(time_start, hook, hook_return)
hook_name = task_name(hook)
is_raise_on_error = hook.fetch('fail_on_error', true)
if hook_return['error'] && is_raise_on_error
nodes = hook['uids'].map do |uid|
{ 'uid' => uid,
'status' => 'error',
'error_type' => @type,
'role' => 'hook',
'hook' => hook_name,
'error_msg' => hook_return['error']
}
end
error_message = 'Failed to execute hook'
error_message += " '#{hook_name}'" if hook_name
error_message += "\n\n#{hook_return['error']}"
@ctx.report_and_update_status('nodes' => nodes, 'error' => error_message)
error_message += "#{hook.to_yaml}"
raise Astute::DeploymentEngineError, error_message
end
end
end
private
def task_name(hook)
hook['id'] || hook['diagnostic_name'] || hook['type']
end
def time_summary(time_start, hook, hook_return)
status = hook_return && !hook_return['error'] ? 'successful' : 'error'
amount_time = (Time.now.to_i - time_start).to_i
wasted_time = Time.at(amount_time).utc.strftime("%H:%M:%S")
hook['uids'].each do |node_id|
Astute.logger.debug("Task time summary: #{task_name(hook)} with status" \
" #{status} on node #{node_id} took #{wasted_time}")
end
end
def copy_files_hook(hook)
validate_presence(hook, 'uids')
validate_presence(hook['parameters'], 'files')
ret = {'error' => nil}
hook['parameters']['files'].each do |file|
if File.file?(file['src']) && File.readable?(file['src'])
parameters = {
'content' => File.binread(file['src']),
'path' => file['dst'],
'permissions' => file['permissions'] || hook['parameters']['permissions'],
'dir_permissions' => file['dir_permissions'] || hook['parameters']['dir_permissions'],
}
perform_with_limit(hook['uids']) do |node_uids|
status = upload_file(@ctx, node_uids, parameters)
if !status
ret['error'] = 'Upload not successful'
end
end
else
ret['error'] = "File does not exist or is not readable #{file['src']}"
Astute.logger.warn(ret['error'])
end
end
ret
end #copy_file_hook
def puppet_hook(hook)
validate_presence(hook, 'uids')
validate_presence(hook['parameters'], 'puppet_manifest')
validate_presence(hook['parameters'], 'puppet_modules')
validate_presence(hook['parameters'], 'cwd')
timeout = hook['parameters']['timeout'] || 300
retries = hook['parameters']['retries'] || Astute.config.puppet_retries
ret = {'error' => nil}
perform_with_limit(hook['uids']) do |node_uids|
result = run_puppet(
@ctx,
node_uids,
hook['parameters']['puppet_manifest'],
hook['parameters']['puppet_modules'],
hook['parameters']['cwd'],
timeout,
retries
)
unless result
ret['error'] = "Puppet run failed. Check puppet logs for details"
Astute.logger.warn(ret['error'])
end
end
ret
end #puppet_hook
def upload_file_hook(hook)
validate_presence(hook, 'uids')
validate_presence(hook['parameters'], 'path')
validate_presence(hook['parameters'], 'data')
hook['parameters']['content'] = hook['parameters']['data']
ret = {'error' => nil}
perform_with_limit(hook['uids']) do |node_uids|
status = upload_file(@ctx, node_uids, hook['parameters'])
if status == false
ret['error'] = 'File upload failed'
end
end
ret
end
def upload_files_hook(hook)
validate_presence(hook['parameters'], 'nodes')
ret = {'error' => nil}
hook['parameters']['nodes'].each do |node|
node['files'].each do |file|
parameters = {
'content' => file['data'],
'path' => file['dst'],
'permissions' => file['permissions'] || '0644',
'dir_permissions' => file['dir_permissions'] || '0755',
}
status = upload_file(@ctx, node['uid'], parameters)
if !status
ret['error'] = 'File upload failed'
end
end
end
ret
end
def shell_hook(hook)
validate_presence(hook, 'uids')
validate_presence(hook['parameters'], 'cmd')
timeout = hook['parameters']['timeout'] || 300
cwd = hook['parameters']['cwd'] || "/"
retries = hook['parameters']['retries'] || Astute.config.mc_retries
interval = hook['parameters']['interval'] || Astute.config.mc_retry_interval
shell_command = "cd #{cwd} && #{hook['parameters']['cmd']}"
ret = {'error' => nil}
perform_with_limit(hook['uids']) do |node_uids|
Timeout::timeout(timeout) do
err_msg = run_shell_command(
@ctx,
node_uids,
shell_command,
retries,
interval,
timeout,
cwd
)
ret['error'] = "Failed to run command #{shell_command} (#{err_msg})." if err_msg
end
end
ret
rescue Astute::MClientTimeout, Astute::MClientError, Timeout::Error => e
err = case [e.class]
when [Astute::MClientTimeout] then 'mcollective client timeout error'
when [Astute::MClientError] then 'mcollective client error'
when [Timeout::Error] then 'overall timeout error'
end
ret['error'] = "command: #{shell_command}" \
"\n\nTask: #{@ctx.task_id}: " \
"#{err}: #{e.message}\n" \
"Task timeout: #{timeout}, " \
"Retries: #{hook['parameters']['retries']}"
Astute.logger.error(ret['error'])
ret
end # shell_hook
def cobbler_sync_hook(hook)
validate_presence(hook['parameters'], 'provisioning_info')
ret = {'error' => nil}
cobbler = CobblerManager.new(
hook['parameters']['provisioning_info']['engine'],
@ctx.reporter
)
cobbler.sync
ret
end # cobbler_sync_hook
def sync_hook(hook)
validate_presence(hook, 'uids')
validate_presence(hook['parameters'], 'dst')
validate_presence(hook['parameters'], 'src')
path = hook['parameters']['dst']
source = hook['parameters']['src']
timeout = hook['parameters']['timeout'] || 300
rsync_cmd = "mkdir -p #{path} && rsync #{Astute.config.rsync_options} " \
"#{source} #{path}"
ret = {'error' => nil}
perform_with_limit(hook['uids']) do |node_uids|
err_msg = run_shell_command(
@ctx,
node_uids,
rsync_cmd,
10,
Astute.config.mc_retry_interval,
timeout
)
ret = {'error' => "Failed to perform sync from #{source} to #{path} (#{err_msg})"} if err_msg
end
ret
end # sync_hook
def reboot_hook(hook)
validate_presence(hook, 'uids')
hook_timeout = hook['parameters']['timeout'] || 300
control_time = {}
perform_with_limit(hook['uids']) do |node_uids|
control_time.merge!(boot_time(node_uids))
end
perform_with_limit(hook['uids']) do |node_uids|
run_shell_without_check(@ctx, node_uids, RebootCommand::CMD, timeout=60)
end
already_rebooted = Hash[hook['uids'].collect { |uid| [uid, false] }]
ret = {'error' => nil}
begin
Timeout::timeout(hook_timeout) do
while already_rebooted.values.include?(false)
sleep hook_timeout/10
results = boot_time(already_rebooted.select { |k, v| !v }.keys)
results.each do |node_id, time|
next if already_rebooted[node_id]
already_rebooted[node_id] = (time.to_i != control_time[node_id].to_i)
end
end
end
rescue Timeout::Error => e
Astute.logger.warn("Time detection (#{hook_timeout} sec) for node reboot has expired")
end
if already_rebooted.values.include?(false)
fail_nodes = already_rebooted.select {|k, v| !v }.keys
ret['error'] = "Reboot command failed for nodes #{fail_nodes}. Check debug output for details"
Astute.logger.warn(ret['error'])
end
update_node_status(already_rebooted.select { |node, rebooted| rebooted }.keys)
ret
end # reboot_hook
def validate_presence(data, key)
raise "Missing a required parameter #{key}" unless data[key].present?
end
def run_puppet(context, node_uids, puppet_manifest, puppet_modules, cwd, timeout, retries)
# Prevent send report status to Nailgun
hook_context = Context.new(context.task_id, HookReporter.new, LogParser::NoParsing.new)
nodes = node_uids.map { |node_id| {'uid' => node_id.to_s, 'role' => 'hook'} }
Timeout::timeout(timeout) {
PuppetdDeployer.deploy(
hook_context,
nodes,
retries=retries,
puppet_manifest,
puppet_modules,
cwd
)
}
!hook_context.status.has_value?('error')
rescue Astute::MClientTimeout, Astute::MClientError, Timeout::Error => e
Astute.logger.error("#{context.task_id}: puppet timeout error: #{e.message}")
false
end
def run_shell_command(context, node_uids, cmd, shell_retries, interval, timeout=60, cwd="/tmp")
responses = nil
(shell_retries + 1).times.each do |retry_number|
shell = MClient.new(context,
'execute_shell_command',
node_uids,
check_result=true,
timeout=timeout,
retries=1)
begin
responses = shell.execute(:cmd => cmd, :cwd => cwd)
rescue MClientTimeout, MClientError => e
Astute.logger.error "#{context.task_id}: cmd: #{cmd} \n" \
"mcollective error: #{e.message}"
next
end
responses.each do |response|
Astute.logger.debug(
"#{context.task_id}: cmd: #{cmd}\n" \
"cwd: #{cwd}\n" \
"stdout: #{response[:data][:stdout]}\n" \
"stderr: #{response[:data][:stderr]}\n" \
"exit code: #{response[:data][:exit_code]}")
end
node_uids -= responses.select { |response| response[:data][:exit_code] == 0 }
.map { |response| response[:sender] }
return nil if node_uids.empty?
Astute.logger.warn "Problem while performing cmd on nodes: #{node_uids}. " \
"Retrying... Attempt #{retry_number} of #{shell_retries}"
sleep interval
end
if responses
responses.select { |response| response[:data][:exit_code] != 0 }
.map {|r| "node #{r[:sender]} returned #{r[:data][:exit_code]}"}
.join(", ")
else
"mclient failed to execute command"
end
end
def upload_file(context, node_uids, mco_params={})
upload_mclient = Astute::MClient.new(context, "uploadfile", Array(node_uids))
mco_params['overwrite'] = true if mco_params['overwrite'].nil?
mco_params['parents'] = true if mco_params['parents'].nil?
mco_params['permissions'] ||= '0644'
mco_params['user_owner'] ||= 'root'
mco_params['group_owner'] ||= 'root'
mco_params['dir_permissions'] ||= '0755'
upload_mclient.upload(
:path => mco_params['path'],
:content => mco_params['content'],
:overwrite => mco_params['overwrite'],
:parents => mco_params['parents'],
:permissions => mco_params['permissions'],
:user_owner => mco_params['user_owner'],
:group_owner => mco_params['group_owner'],
:dir_permissions => mco_params['dir_permissions']
)
true
rescue MClientTimeout, MClientError => e
Astute.logger.error("#{context.task_id}: mcollective upload_file agent error: #{e.message}")
false
end
def perform_with_limit(nodes, &block)
nodes.each_slice(Astute.config[:max_nodes_per_call]) do |part|
block.call(part)
end
end
def run_shell_without_check(context, node_uids, cmd, timeout=10)
shell = MClient.new(
context,
'execute_shell_command',
node_uids,
check_result=false,
timeout=timeout
)
results = shell.execute(:cmd => cmd)
results.inject({}) do |h, res|
Astute.logger.debug(
"#{context.task_id}: cmd: #{cmd}\n" \
"stdout: #{res.results[:data][:stdout]}\n" \
"stderr: #{res.results[:data][:stderr]}\n" \
"exit code: #{res.results[:data][:exit_code]}")
h.merge({res.results[:sender] => res.results[:data][:stdout].chomp})
end
end
def boot_time(uids)
run_shell_without_check(
@ctx,
uids,
"stat --printf='%Y' /proc/1",
timeout=10
)
end
def update_node_status(uids)
run_shell_without_check(
@ctx,
uids,
"flock -w 0 -o /var/lock/nailgun-agent.lock -c '/usr/bin/nailgun-agent"\
" 2>&1 | tee -a /var/log/nailgun-agent.log | "\
"/usr/bin/logger -t nailgun-agent'",
_timeout=60 # nailgun-agent start with random (30) delay
)
end
end # class
class HookReporter
def report(msg)
Astute.logger.debug msg
end
end
end # module

280
lib/astute/network.rb
View File

@ -1,280 +0,0 @@
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
module Network
def self.check_network(ctx, nodes)
if nodes.empty?
Astute.logger.info(
"#{ctx.task_id}: Network checker: nodes list is empty. Nothing to check.")
return {
'status' => 'error',
'error' => "Network verification requires a minimum of two nodes."
}
elsif nodes.length == 1
Astute.logger.info(
"#{ctx.task_id}: Network checker: nodes list contains one node only. Do nothing.")
return {'nodes' => [{
'uid' => nodes[0]['uid'],
'networks' => nodes[0]['networks']
}]}
end
uids = nodes.map { |node| node['uid'].to_s }
# TODO Everything breakes if agent not found. We have to handle that
net_probe = MClient.new(ctx, "net_probe", uids)
versioning = Versioning.new(ctx)
old_version, new_version = versioning.split_on_version(uids, '6.1.0')
old_uids = old_version.map { |node| node['uid'].to_i }
new_uids = new_version.map { |node| node['uid'].to_i }
old_nodes = nodes.select { |node| old_uids.include? node['uid'] }
new_nodes = nodes.select { |node| new_uids.include? node['uid'] }
if old_uids.present?
start_frame_listeners_60(ctx, net_probe, old_nodes)
end
if new_uids.present?
start_frame_listeners(ctx, net_probe, new_nodes)
end
ctx.reporter.report({'progress' => 30})
if old_uids.present?
send_probing_frames_60(ctx, net_probe, old_nodes)
end
if new_uids.present?
send_probing_frames(ctx, net_probe, new_nodes)
end
ctx.reporter.report({'progress' => 60})
net_probe.discover(:nodes => uids)
stats = net_probe.get_probing_info
result = format_result(stats)
Astute.logger.debug "#{ctx.task_id}: Network checking is done. Results:\n#{result.pretty_inspect}"
{'nodes' => result}
end
def self.check_dhcp(ctx, nodes)
uids = nodes.map { |node| node['uid'].to_s }
net_probe = MClient.new(ctx, "net_probe", uids)
data_to_send = {}
nodes.each do |node|
data_to_send[node['uid'].to_s] = make_interfaces_to_send(node['networks'], joined=false).to_json
end
repeat = Astute.config.dhcp_repeat
result = net_probe.dhcp_discover(:interfaces => data_to_send,
:timeout => 10, :repeat => repeat).map do |response|
format_dhcp_response(response)
end
status = result.any?{|node| node[:status] == 'error'} && 'error' || 'ready'
{'nodes' => result, 'status'=> status}
end
def self.multicast_verification(ctx, nodes)
uids = nodes.map { |node| node['uid'].to_s }
net_probe = MClient.new(ctx, "net_probe", uids)
data_to_send = {}
nodes.each do |node|
data_to_send[node['uid']] = node
end
listen_resp = net_probe.multicast_listen(:nodes => data_to_send.to_json)
Astute.logger.debug("Mutlicast verification listen:\n#{listen_resp.pretty_inspect}")
ctx.reporter.report({'progress' => 30})
send_resp = net_probe.multicast_send()
Astute.logger.debug("Mutlicast verification send:\n#{send_resp.pretty_inspect}")
ctx.reporter.report({'progress' => 60})
results = net_probe.multicast_info()
Astute.logger.debug("Mutlicast verification info:\n#{results.pretty_inspect}")
response = {}
results.each do |node|
if node.results[:data][:out].present?
response[node.results[:sender].to_i] = JSON.parse(node.results[:data][:out])
end
end
{'nodes' => response}
end
def self.check_urls_access(ctx, nodes, urls)
uids = nodes.map { |node| node['uid'].to_s }
net_probe = MClient.new(ctx, "net_probe", uids)
result = net_probe.check_url_retrieval(:urls => urls)
{'nodes' => flatten_response(result), 'status'=> 'ready'}
end
def self.check_repositories_with_setup(ctx, nodes)
uids = nodes.map { |node| node['uid'].to_s }
net_probe = MClient.new(ctx, "net_probe", uids, check_result=false)
data = nodes.inject({}) { |h, node| h.merge({node['uid'].to_s => node}) }
result = net_probe.check_repositories_with_setup(:data => data)
bad_nodes = nodes.map { |n| n['uid'] } - result.map { |n| n.results[:sender] }
if bad_nodes.present?
error_msg = "Astute could not get result from nodes #{bad_nodes}. " \
"Please check mcollective log on problem nodes " \
"for more details. Hint: try to execute check manually " \
"using command from mcollective log on nodes, also " \
"check nodes availability using command `mco ping` " \
"on master node."
raise MClientTimeout, error_msg
end
{'nodes' => flatten_response(result), 'status'=> 'ready'}
end
private
def self.start_frame_listeners(ctx, net_probe, nodes)
data_to_send = {}
nodes.each do |node|
data_to_send[node['uid'].to_s] = make_interfaces_to_send(node['networks'])
end
uids = nodes.map { |node| node['uid'].to_s }
Astute.logger.debug(
"#{ctx.task_id}: Network checker listen: nodes: #{uids} data:\n#{data_to_send.pretty_inspect}")
net_probe.discover(:nodes => uids)
net_probe.start_frame_listeners(:interfaces => data_to_send.to_json)
end
def self.send_probing_frames(ctx, net_probe, nodes)
nodes.each_slice(Astute.config[:max_nodes_net_validation]) do |nodes_part|
data_to_send = {}
nodes_part.each do |node|
data_to_send[node['uid'].to_s] = make_interfaces_to_send(node['networks'])
end
uids = nodes_part.map { |node| node['uid'].to_s }
Astute.logger.debug(
"#{ctx.task_id}: Network checker send: nodes: #{uids} data:\n#{data_to_send.pretty_inspect}")
net_probe.discover(:nodes => uids)
net_probe.send_probing_frames(:interfaces => data_to_send.to_json)
end
end
def self.start_frame_listeners_60(ctx, net_probe, nodes)
nodes.each do |node|
data_to_send = make_interfaces_to_send(node['networks'])
Astute.logger.debug(
"#{ctx.task_id}: Network checker listen: node: #{node['uid']} data:\n#{data_to_send.pretty_inspect}")
net_probe.discover(:nodes => [node['uid'].to_s])
net_probe.start_frame_listeners(:interfaces => data_to_send.to_json)
end
end
def self.send_probing_frames_60(ctx, net_probe, nodes)
nodes.each do |node|
data_to_send = make_interfaces_to_send(node['networks'])
Astute.logger.debug(
"#{ctx.task_id}: Network checker send: node: #{node['uid']} data:\n#{data_to_send.pretty_inspect}")
net_probe.discover(:nodes => [node['uid'].to_s])
net_probe.send_probing_frames(:interfaces => data_to_send.to_json)
end
end
def self.make_interfaces_to_send(networks, joined=true)
data_to_send = {}
networks.each do |network|
if joined
data_to_send[network['iface']] = network['vlans'].join(",")
else
data_to_send[network['iface']] = network['vlans']
end
end
data_to_send
end
def self.format_dhcp_response(response)
node_result = {:uid => response.results[:sender],
:status=>'ready'}
if response.results[:data][:out].present?
Astute.logger.debug("DHCP checker received:\n#{response.pretty_inspect}")
node_result[:data] = JSON.parse(response.results[:data][:out])
elsif response.results[:data][:err].present?
Astute.logger.debug("DHCP checker errred with:\n#{response.pretty_inspect}")
node_result[:status] = 'error'
node_result[:error_msg] = 'Error in dhcp checker. Check logs for details'
end
node_result
end
def self.format_result(stats)
uids = stats.map{|node| node.results[:sender]}.sort
stats.map do |node|
{
'uid' => node.results[:sender],
'networks' => check_vlans_by_traffic(
node.results[:sender],
uids,
node.results[:data][:neighbours])
}
end
end
def self.flatten_response(response)
response.map do |node|
{:out => node.results[:data][:out],
:err => node.results[:data][:err],
:status => node.results[:data][:status],
:uid => node.results[:sender]}
end
end
def self.check_vlans_by_traffic(uid, uids, data)
data.map do |iface, vlans|
{
'iface' => iface,
'vlans' => remove_extra_data(uid, uids, vlans).select {|k,v|
v.keys.present?
}.keys.map(&:to_i)
}
end
end
# Remove unnecessary data
def self.remove_extra_data(uid, uids, vlans)
vlans.each do |k, data|
# remove data sent by node itself
data.delete(uid)
# remove data sent by nodes from different envs
data.keep_if { |k, v| uids.include?(k) }
end
vlans
end
end
end

101
lib/astute/node.rb
View File

@ -1,101 +0,0 @@
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
require 'active_support/core_ext/hash/indifferent_access'
require 'ostruct'
module Astute
class Node < OpenStruct
def initialize(hash=nil)
if hash && (uid = hash['uid'])
hash = hash.dup
hash['uid'] = uid.to_s
else
raise TypeError.new("Invalid data: #{hash.inspect}")
end
super hash
end
def [](key)
send key
end
def []=(key, value)
send "#{key}=", value
end
def uid
@table[:uid]
end
def uid=(_)
raise TypeError.new('Read-only attribute')
end
def to_hash
@table.with_indifferent_access
end
def fetch(key, default)
ret = self[key]
if ret.nil?
ret = default
end
ret
end
end
class NodesHash < Hash
alias uids keys
alias nodes values
def self.build(nodes)
return nodes if nodes.kind_of? self
nodes.inject(self.new) do |hash, node|
hash << node
hash
end
end
def <<(node)
node = normalize_value(node)
self[node.uid] = node
self
end
def push(*nodes)
nodes.each{|node| self.<< node }
self
end
def [](key)
super key.to_s
end
private
def []=(*args)
super
end
def normalize_value(node)
if node.kind_of? Node
node
else
Node.new(node.to_hash)
end
end
end
end

197
lib/astute/nodes_remover.rb
View File

@ -1,197 +0,0 @@
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class NodesRemover
def initialize(ctx, nodes, reboot=true)
@ctx = ctx
@nodes = NodesHash.build(nodes)
@reboot = reboot
end
def remove
# TODO(mihgen): 1. Nailgun should process node error message
# 2. Should we rename nodes -> removed_nodes array?
# 3. If exception is raised here, we should not fully fall into error, but only failed node
erased_nodes, error_nodes, inaccessible_nodes = remove_nodes(@nodes)
retry_remove_nodes(error_nodes, erased_nodes,
Astute.config[:mc_retries], Astute.config[:mc_retry_interval])
retry_remove_nodes(inaccessible_nodes, erased_nodes,
Astute.config[:mc_retries], Astute.config[:mc_retry_interval])
answer = {'nodes' => serialize_nodes(erased_nodes)}
if inaccessible_nodes.present?
serialized_inaccessible_nodes = serialize_nodes(inaccessible_nodes)
answer.merge!({'inaccessible_nodes' => serialized_inaccessible_nodes})
Astute.logger.warn "#{@ctx.task_id}: Removing of nodes\n#{@nodes.uids.pretty_inspect} finished " \
"with errors. Nodes\n#{serialized_inaccessible_nodes.pretty_inspect} are inaccessible"
end
if error_nodes.present?
serialized_error_nodes = serialize_nodes(error_nodes)
answer.merge!({'status' => 'error', 'error_nodes' => serialized_error_nodes})
Astute.logger.error "#{@ctx.task_id}: Removing of nodes\n#{@nodes.uids.pretty_inspect} finished " \
"with errors:\n#{serialized_error_nodes.pretty_inspect}"
end
Astute.logger.info "#{@ctx.task_id}: Finished removing of nodes:\n#{@nodes.uids.pretty_inspect}"
answer
end
private
def serialize_nodes(nodes)
nodes.nodes.map(&:to_hash)
end
# When :mclient_remove property is true (the default behavior), we send
# the node to mclient for removal (MBR, restarting etc), if it's false
# the node is skipped from mclient
def skipped_unskipped_mclient_nodes(nodes)
mclient_skipped_nodes = NodesHash.build(
nodes.values.select { |node| not node.fetch(:mclient_remove, true) }
)
mclient_nodes = NodesHash.build(
nodes.values.select { |node| node.fetch(:mclient_remove, true) }
)
Astute.logger.debug "#{@ctx.task_id}: Split nodes: #{mclient_skipped_nodes}, #{mclient_nodes}"
[mclient_skipped_nodes, mclient_nodes]
end
def get_already_removed_nodes(nodes)
removed_nodes = []
control_time = {}
nodes.uids.sort.each_slice(Astute.config[:max_nodes_per_call]) do |part|
control_time.merge!(get_boot_time(part))
end
nodes.each do |uid, node|
boot_time = control_time[uid].to_i
next if boot_time.zero?
if node.boot_time
removed_nodes << uid if boot_time != node.boot_time
else
node.boot_time = boot_time
end
end
removed_nodes
end
def remove_nodes(nodes)
if nodes.empty?
Astute.logger.info "#{@ctx.task_id}: Nodes to remove are not provided. Do nothing."
return Array.new(3){ NodesHash.new }
end
erased_nodes, mclient_nodes = skipped_unskipped_mclient_nodes(nodes)
removed_nodes = get_already_removed_nodes(mclient_nodes)
removed_nodes.each do |uid|
erased_node = Node.new('uid' => uid)
erased_nodes << erased_node
mclient_nodes.delete(uid)
Astute.logger.info "#{@ctx.task_id}: Node #{uid} is removed already, skipping"
end
responses = mclient_remove_nodes(mclient_nodes)
inaccessible_uids = mclient_nodes.uids - responses.map { |response| response[:sender] }
inaccessible_nodes = NodesHash.build(inaccessible_uids.map do |uid|
{'uid' => uid, 'error' => 'Node not answered by RPC.', 'boot_time' => mclient_nodes[uid][:boot_time]}
end)
error_nodes = NodesHash.new
responses.each do |response|
node = Node.new('uid' => response[:sender])
if response[:statuscode] != 0
node['error'] = "RPC agent 'erase_node' failed. Result:\n#{response.pretty_inspect}"
error_nodes << node
elsif @reboot && !response[:data][:rebooted]
node['error'] = "RPC method 'erase_node' failed with message: #{response[:data][:error_msg]}"
error_nodes << node
else
erased_nodes << node
end
end
[erased_nodes, error_nodes, inaccessible_nodes]
end
def retry_remove_nodes(error_nodes, erased_nodes, retries=3, interval=1)
retries.times do
retried_erased_nodes = remove_nodes(error_nodes)[0]
retried_erased_nodes.each do |uid, node|
error_nodes.delete uid
erased_nodes << node
end
return if error_nodes.empty?
sleep(interval) if interval > 0
end
end
def mclient_remove_nodes(nodes)
Astute.logger.info "#{@ctx.task_id}: Starting removing of nodes:\n#{nodes.uids.pretty_inspect}"
results = []
nodes.uids.sort.each_slice(Astute.config[:max_nodes_per_remove_call]).with_index do |part, i|
sleep Astute.config[:nodes_remove_interval] if i != 0
results += mclient_remove_piece_nodes(part)
end
results
end
def mclient_remove_piece_nodes(nodes)
remover = MClient.new(@ctx, "erase_node", nodes, check_result=false)
responses = remover.erase_node(:reboot => @reboot)
Astute.logger.debug "#{@ctx.task_id}: Data received from nodes:\n#{responses.pretty_inspect}"
responses.map(&:results)
end
def run_shell_without_check(context, node_uids, cmd, timeout=10)
shell = MClient.new(
context,
'execute_shell_command',
node_uids,
check_result=false,
timeout=timeout
)
results = shell.execute(:cmd => cmd)
results.inject({}) do |h, res|
Astute.logger.debug(
"#{context.task_id}: cmd: #{cmd}\n" \
"stdout: #{res.results[:data][:stdout]}\n" \
"stderr: #{res.results[:data][:stderr]}\n" \
"exit code: #{res.results[:data][:exit_code]}")
h.merge({res.results[:sender] => res.results[:data][:stdout].chomp})
end
end
def get_boot_time(node_uids)
run_shell_without_check(
@ctx,
node_uids,
"stat --printf='%Y' /proc/1",
timeout=10
)
end
end
end

296
lib/astute/orchestrator.rb
View File

@ -1,296 +0,0 @@
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class Orchestrator
def initialize(log_parsing=false)
@log_parsing = log_parsing
end
def node_type(reporter, task_id, nodes_uids, timeout=nil)
provisioner = Provisioner.new(@log_parsing)
provisioner.node_type(reporter, task_id, nodes_uids, timeout)
end
def execute_tasks(up_reporter, task_id, tasks)
ctx = Context.new(task_id, up_reporter)
Astute::NailgunHooks.new(tasks, ctx, 'execute_tasks').process
report_result({}, up_reporter)
end
# Deploy method which use small tasks, but run block of tasks for role
# instead of run it using full graph. Use from 6.1 to 8.0. Report progress
# based on puppet logs
def granular_deploy(up_reporter, task_id, deployment_info, pre_deployment=[], post_deployment=[])
time_start = Time.now.to_i
deploy_cluster(
up_reporter,
task_id,
deployment_info,
Astute::DeploymentEngine::GranularDeployment,
pre_deployment,
post_deployment
)
ensure
Astute.logger.info "Deployment summary: time was spent " \
"#{time_summary(time_start)}"
end
# Deploy method which use small tasks in full graph.
# Use from 8.0 (experimental). Report progress based on tasks
def task_deploy(up_reporter, task_id, deployment_options = {})
time_start = Time.now.to_i
proxy_reporter = ProxyReporter::TaskProxyReporter.new(
up_reporter
)
context = Context.new(task_id, proxy_reporter)
Astute.logger.info "Task based deployment will be used"
deployment_engine = TaskDeployment.new(context)
write_input_data_to_file(context, deployment_options) if Astute.config.enable_graph_file
deployment_engine.deploy(deployment_options)
ensure
Astute.logger.info "Deployment summary: time was spent " \
"#{time_summary(time_start)}"
end
def provision(up_reporter, task_id, provisioning_info)
time_start = Time.now.to_i
proxy_reporter = ProxyReporter::ProvisiningProxyReporter.new(
up_reporter,
provisioning_info
)
provisioner = Provisioner.new(@log_parsing)
if provisioning_info['pre_provision']
image_build_log = "/var/log/fuel-agent-env" \
"-#{calculate_cluster_id(provisioning_info)}.log"
Astute.logger.info "Please check image build log here: " \
"#{image_build_log}"
ctx = Context.new(task_id, proxy_reporter)
provisioner.report_image_provision(
proxy_reporter,
task_id,
provisioning_info['nodes'],
image_log_parser(provisioning_info)
) do
begin
Astute::NailgunHooks.new(
provisioning_info['pre_provision'],
ctx,
'provision'
).process
rescue Astute::DeploymentEngineError => e
raise e, "Image build task failed. Please check " \
"build log here for details: #{image_build_log}. " \
"Hint: restart deployment can help if no error in build " \
"log was found"
end
end
end
# NOTE(kozhukalov): Some of our pre-provision tasks need cobbler to be synced
# once those tasks are finished. It looks like the easiest way to do this
# inside mcollective docker container is to use Astute binding capabilities.
cobbler = CobblerManager.new(provisioning_info['engine'], up_reporter)
cobbler.sync
provisioner.provision(
proxy_reporter,
task_id,
provisioning_info
)
ensure
Astute.logger.info "Provision summary: time was spent " \
"#{time_summary(time_start)}"
end
def remove_nodes(reporter, task_id, engine_attrs, nodes, options={})
# FIXME(vsharshov): bug/1463881: In case of post deployment we mark all nodes
# as ready. In this case we will get empty nodes.
return if nodes.empty?
options.reverse_merge!({
:reboot => true,
:raise_if_error => false,
:reset => false
})
result = perform_pre_deletion_tasks(reporter, task_id, nodes, options)
return result if result['status'] != 'ready'
provisioner = Provisioner.new(@log_parsing)
provisioner.remove_nodes(
reporter,
task_id,
engine_attrs,
nodes,
options
)
end
def stop_puppet_deploy(reporter, task_id, nodes)
# FIXME(vsharshov): bug/1463881: In case of post deployment we mark all nodes
# as ready. If we run stop deployment we will get empty nodes.
return if nodes.empty?
nodes_uids = nodes.map { |n| n['uid'] }.uniq
puppetd = MClient.new(Context.new(task_id, reporter), "puppetd", nodes_uids, check_result=false)
puppetd.stop_and_disable
end
def stop_provision(reporter, task_id, engine_attrs, nodes)
provisioner = Provisioner.new(@log_parsing)
provisioner.stop_provision(reporter, task_id, engine_attrs, nodes)
end
def dump_environment(reporter, task_id, settings)
Dump.dump_environment(Context.new(task_id, reporter), settings)
end
def verify_networks(reporter, task_id, nodes)
ctx = Context.new(task_id, reporter)
validate_nodes_access(ctx, nodes)
Network.check_network(ctx, nodes)
end
def check_dhcp(reporter, task_id, nodes)
ctx = Context.new(task_id, reporter)
validate_nodes_access(ctx, nodes)
Network.check_dhcp(ctx, nodes)
end
def multicast_verification(reporter, task_id, nodes)
ctx = Context.new(task_id, reporter)
validate_nodes_access(ctx, nodes)
Network.multicast_verification(ctx, nodes)
end
def check_repositories(reporter, task_id, nodes, urls)
ctx = Context.new(task_id, reporter)
validate_nodes_access(ctx, nodes)
Network.check_urls_access(ctx, nodes, urls)
end
def check_repositories_with_setup(reporter, task_id, nodes)
ctx = Context.new(task_id, reporter)
validate_nodes_access(ctx, nodes)
Network.check_repositories_with_setup(ctx, nodes)
end
private
def deploy_cluster(up_reporter, task_id, deployment_info, deploy_engine, pre_deployment, post_deployment)
proxy_reporter = ProxyReporter::DeploymentProxyReporter.new(up_reporter, deployment_info)
log_parser = @log_parsing ? LogParser::ParseDeployLogs.new : LogParser::NoParsing.new
context = Context.new(task_id, proxy_reporter, log_parser)
deploy_engine_instance = deploy_engine.new(context)
Astute.logger.info "Using #{deploy_engine_instance.class} for deployment."
deploy_engine_instance.deploy(deployment_info, pre_deployment, post_deployment)
context.status
end
def report_result(result, reporter)
default_result = {'status' => 'ready', 'progress' => 100}
result = {} unless result.instance_of?(Hash)
status = default_result.merge(result)
reporter.report(status)
end
def validate_nodes_access(ctx, nodes)
nodes_types = node_type(ctx.reporter, ctx.task_id, nodes.map{ |n| n['uid'] }, timeout=10)
not_available_nodes = nodes.map { |n| n['uid'].to_s } - nodes_types.map { |n| n['uid'].to_s }
unless not_available_nodes.empty?
raise "Network verification not available because nodes #{not_available_nodes} " \
"not available via mcollective"
end
end
def image_log_parser(provisioning_info)
log_parser = LogParser::ParseImageBuildLogs.new
log_parser.cluster_id = calculate_cluster_id(provisioning_info)
log_parser
end
def calculate_cluster_id(provisioning_info)
return nil unless provisioning_info['pre_provision'].present?
cmd = provisioning_info['pre_provision'].first.fetch('parameters', {}).fetch('cmd', "")
# find cluster id from cmd using pattern fuel-agent-env-<Integer>.log
# FIXME(vsharshov): https://bugs.launchpad.net/fuel/+bug/1449512
cluster_id = cmd[/fuel-agent-env-(\d+)/, 1]
Astute.logger.debug "Cluster id: #{cluster_id}"
cluster_id
end
def check_for_offline_nodes(reporter, task_id, nodes)
PreDelete.check_for_offline_nodes(Context.new(task_id, reporter), nodes)
end
def check_ceph_osds(reporter, task_id, nodes)
PreDelete.check_ceph_osds(Context.new(task_id, reporter), nodes)
end
def remove_ceph_mons(reporter, task_id, nodes)
PreDelete.remove_ceph_mons(Context.new(task_id, reporter), nodes)
end
def perform_pre_deletion_tasks(reporter, task_id, nodes, options={})
result = {'status' => 'ready'}
# This option is no longer Ceph-specific and should be renamed
# FIXME(rmoe): https://bugs.launchpad.net/fuel/+bug/1454377
if options[:check_ceph]
result = check_for_offline_nodes(reporter, task_id, nodes)
return result if result['status'] != 'ready'
result = check_ceph_osds(reporter, task_id, nodes)
return result if result['status'] != 'ready'
result = remove_ceph_mons(reporter, task_id, nodes)
end
result
end
def time_summary(time)
amount_time = (Time.now.to_i - time).to_i
Time.at(amount_time).utc.strftime("%H:%M:%S")
end
# Dump the task graph data to a file
# @param [Astute::Context] context
# @param [Hash] data
def write_input_data_to_file(context, data={})
yaml_file = File.join(
Astute.config.graph_dot_dir,
"graph-#{context.task_id}.yaml"
)
data = filter_sensitive_data(data)
File.open(yaml_file, 'w') { |f| f.write(YAML.dump(data)) }
Astute.logger.info("Check inpute data file #{yaml_file}")
end
# Remove the potentially sensitive data
# from the task parameters before dumping the graph
# @param [Hash] data
# @return [Hash]
def filter_sensitive_data(data)
data = data.deep_dup
data[:tasks_graph].each do |_node_id, tasks|
tasks.each { |task| task.delete('parameters') }
end
data
end
end # class
end # module

40
lib/astute/post_deploy_actions/post_patching_ha.rb
View File

@ -1,40 +0,0 @@
# Copyright 2014 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class PostPatchingHa < PostDeployAction
def process(deployment_info, context)
return if deployment_info.first['openstack_version_prev'].nil? ||
deployment_info.first['deployment_mode'] !~ /ha/i
controller_nodes = deployment_info.select{ |n| n['role'] =~ /controller/i }.map{ |n| n['uid'] }
return if controller_nodes.empty?
Astute.logger.info "Starting unmigration of pacemaker services from " \
"nodes\n#{controller_nodes.pretty_inspect}"
Astute::Pacemaker.commands(action='start', deployment_info).each do |pcmk_unban_cmd|
response = run_shell_command(context, controller_nodes, pcmk_unban_cmd)
if response[:data][:exit_code] != 0
Astute.logger.warn "#{context.task_id}: Failed to unban service, "\
"check the debugging output for details"
end
end
Astute.logger.info "#{context.task_id}: Finished post-patching-ha hook"
end #process
end #class
end

44
lib/astute/post_deployment_actions/restart_radosgw.rb
View File

@ -1,44 +0,0 @@
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class RestartRadosgw < PostDeploymentAction
def process(deployment_info, context)
ceph_node = deployment_info.find { |n| n['role'] == 'ceph-osd' }
objects_ceph = ceph_node && ceph_node.fetch('storage', {}).fetch('objects_ceph')
return unless objects_ceph
Astute.logger.info "Start restarting radosgw on controller nodes"
cmd = <<-RESTART_RADOSGW
(test -f /etc/init.d/ceph-radosgw && /etc/init.d/ceph-radosgw restart) ||
(test -f /etc/init.d/radosgw && /etc/init.d/radosgw restart);
RESTART_RADOSGW
cmd.tr!("\n"," ")
controller_nodes = deployment_info.first['nodes'].inject([]) do |c_n, n|
c_n << n['uid'] if ['controller', 'primary-controller'].include? n['role']
c_n
end
response = run_shell_command(context, controller_nodes, cmd)
if response[:data][:exit_code] != 0
Astute.logger.warn "#{context.task_id}: Fail to restart radosgw, "\
"check the debugging output for details"
end
Astute.logger.info "#{context.task_id}: Finish restarting radosgw on controller nodes"
end #process
end #class
end

65
lib/astute/post_deployment_actions/update_cluster_hosts_info.rb
View File

@ -1,65 +0,0 @@
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class UpdateClusterHostsInfo < PostDeploymentAction
def process(deployment_info, context)
Astute.logger.info "Updating /etc/hosts in all cluster nodes"
return if deployment_info.empty?
response = nil
deployment_info.first['nodes'].each do |node|
upload_file(node['uid'],
deployment_info.first['nodes'].to_yaml,
context)
cmd = <<-UPDATE_HOSTS
ruby -r 'yaml' -e 'y = YAML.load_file("/etc/astute.yaml");
y["nodes"] = YAML.load_file("/tmp/astute.yaml");
File.open("/etc/astute.yaml", "w") { |f| f.write y.to_yaml }';
puppet apply --logdest syslog --debug -e '$settings=parseyaml($::astute_settings_yaml)
$nodes_hash=$settings["nodes"]
class {"l23network::hosts_file": nodes => $nodes_hash }'
UPDATE_HOSTS
cmd.tr!("\n"," ")
response = run_shell_command(context, Array(node['uid']), cmd)
if response[:data][:exit_code] != 0
Astute.logger.warn "#{context.task_id}: Fail to update /etc/hosts, "\
"check the debugging output for node "\
"#{node['uid']} for details"
end
end
Astute.logger.info "#{context.task_id}: Updating /etc/hosts is done"
end
private
def upload_file(node_uid, content, context)
upload_mclient = Astute::MClient.new(context, "uploadfile", Array(node_uid))
upload_mclient.upload(:path => "/tmp/astute.yaml",
:content => content,
:overwrite => true,
:parents => true,
:permissions => '0600'
)
rescue MClientTimeout, MClientError => e
Astute.logger.error("#{context.task_id}: mcollective upload_file agent error: #{e.message}")
end
end #class
end

75
lib/astute/post_deployment_actions/update_no_quorum_policy.rb
View File

@ -1,75 +0,0 @@
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class UpdateNoQuorumPolicy < PostDeploymentAction
def process(deployment_info, context)
return if deployment_info.first['deployment_mode'] !~ /ha/i
# NOTE(bogdando) use 'suicide' if fencing is enabled in corosync
xml = <<-EOF
<diff>
<diff-removed>
<cib>
<configuration>
<crm_config>
<cluster_property_set id="cib-bootstrap-options">
<nvpair value="ignore" id="cib-bootstrap-options-no-quorum-policy"/>
</cluster_property_set>
</crm_config>
</configuration>
</cib>
</diff-removed>
<diff-added>
<cib>
<configuration>
<crm_config>
<cluster_property_set id="cib-bootstrap-options">
<nvpair value="stop" id="cib-bootstrap-options-no-quorum-policy"/>
</cluster_property_set>
</crm_config>
</configuration>
</cib>
</diff-added>
</diff>
EOF
cmd = "/usr/sbin/cibadmin --patch --sync-call --xml-text '#{xml}'"
cmd.tr!("\n"," ")
controllers_count = deployment_info.select {|n|
['controller', 'primary-controller'].include? n['role']
}.size
if controllers_count > 2
primary_controller = deployment_info.find {|n| n['role'] == 'primary-controller' }
if context.status[primary_controller['uid']] == 'error'
Astute.logger.info "Update quorum policy for corosync cluster " \
"disabled because of primary-controller status is error"
return
end
Astute.logger.info "Started updating no quorum policy for corosync cluster"
response = run_shell_command(context, Array(primary_controller['uid']), cmd)
if response[:data][:exit_code] != 0
Astute.logger.warn "#{context.task_id}: Failed to update no "\
"quorum policy for corosync cluster,"
end
Astute.logger.info "#{context.task_id}: Finished updating "\
"no quorum policy for corosync cluster"
else
Astute.logger.info "No need to update quorum policy for corosync cluster"
end
end #process
end #class
end

111
lib/astute/post_deployment_actions/upload_cirros_image.rb
View File

@ -1,111 +0,0 @@
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class CirrosError < AstuteError; end
class UploadCirrosImage < PostDeploymentAction
def process(deployment_info, context)
# Mark controller node as error if present
node = deployment_info.find { |n| n['role'] == 'primary-controller' }
node = deployment_info.find { |n| n['role'] == 'controller' } unless node
node = deployment_info.last unless node
controller = node['nodes'].find { |n| n['role'] == 'primary-controller' }
controller = node['nodes'].find { |n| n['role'] == 'controller' } unless controller
if controller.nil?
Astute.logger.debug "Could not find controller in nodes in facts! " \
"Please check logs to be sure that it is correctly generated."
return
end
# controller['test_vm_image'] contains a hash like that:
# controller['test_vm_image'] = {
# 'disk_format' => 'qcow2',
# 'container_format' => 'bare',
# 'public' => 'true',
# 'img_name' => 'TestVM',
# 'os_name' => 'cirros',
# 'img_path' => '/opt/vm/cirros-x86_64-disk.img',
# 'glance_properties' => '--property murano_image_info=\'{\"title\": \"Murano Demo\", \"type\": \"cirros.demo\"}\''
# }
os = node['test_vm_image']
cmd = ". /root/openrc && /usr/bin/glance image-list"
# waited until the glance is started because when vCenter used as a glance
# backend launch may takes up to 1 minute.
response = {}
5.times.each do |retries|
sleep 10 if retries > 0
response = run_shell_command(context, Array(controller['uid']), cmd)
break if response[:data][:exit_code] == 0
end
if response[:data][:exit_code] != 0
msg = 'Disabling the upload of disk image because glance was not installed properly'
if context.status[node['uid']] != 'error'
raise_cirros_error(
context,
node,
msg
)
else
Astute.logger.error("#{context.task_id}: #{msg}")
return
end
end
cmd = <<-UPLOAD_IMAGE
. /root/openrc &&
/usr/bin/glance image-list | grep -q #{os['img_name']} ||
/usr/bin/glance image-create
--name \'#{os['img_name']}\'
--is-public #{os['public']}
--container-format=\'#{os['container_format']}\'
--disk-format=\'#{os['disk_format']}\'
--min-ram=#{os['min_ram']} #{os['glance_properties']}
--file \'#{os['img_path']}\'
UPLOAD_IMAGE
cmd.tr!("\n"," ")
response = run_shell_command(context, Array(controller['uid']), cmd)
if response[:data][:exit_code] == 0
Astute.logger.info "#{context.task_id}: Upload cirros " \
"image \"#{os['img_name']}\" is done"
else
raise_cirros_error(context, node, "Upload cirros \"#{os['img_name']}\" image failed")
end
end # process
private
def raise_cirros_error(context, node, msg='')
Astute.logger.error("#{context.task_id}: #{msg}")
context.report_and_update_status('nodes' => [
{'uid' => node['uid'],
'status' => 'error',
'error_type' => 'deploy',
'role' => node['role']
}
]
)
raise CirrosError, msg
end
end # class
end

177
lib/astute/pre_delete.rb
View File

@ -1,177 +0,0 @@
# Copyright 2015 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
module PreDelete
def self.check_ceph_osds(ctx, nodes)
answer = {"status" => "ready"}
ceph_nodes = nodes.select { |n| n["roles"].include? "ceph-osd" }
ceph_osds = ceph_nodes.collect{ |n| n["slave_name"] }
return answer if ceph_osds.empty?
cmd = "ceph -f json osd tree"
result = {}
shell = nil
ceph_nodes.each do |ceph_node|
shell = MClient.new(ctx, "execute_shell_command", [ceph_node["id"]], timeout=60, retries=1)
result = shell.execute(:cmd => cmd).first.results
break if result[:data][:exit_code] == 0
end
if result[:data][:exit_code] != 0
Astute.logger.debug "Ceph has not been found or has not been configured properly" \
" Safely removing nodes..."
return answer
end
osds = {}
tree = JSON.parse(result[:data][:stdout])
tree["nodes"].each do |osd|
osds[osd["name"]] = osd["children"] if ceph_osds.include? osd["name"]
end
# pg dump lists all pgs in the cluster and where they are located.
# $14 is the 'up set' (the list of OSDs responsible for a particular
# pg for an epoch) and $16 is the 'acting set' (list of OSDs who
# are [or were at some point] responsible for a pg). These sets
# will generally be the same.
osd_list = osds.values.flatten.join("|")
cmd = "ceph pg dump 2>/dev/null | " \
"awk '//{print $14, $16}' | " \
"egrep -o '\\<(#{osd_list})\\>' | " \
"sort -un"
result = shell.execute(:cmd => cmd).first.results
rs = result[:data][:stdout].split("\n")
# JSON.parse returns the children as integers, so the result from the
# shell command needs to be converted for the set operations to work.
rs.map! { |x| x.to_i }
error_nodes = []
osds.each do |name, children|
error_nodes << name if rs & children != []
end
if not error_nodes.empty?
msg = "Ceph data still exists on: #{error_nodes.join(', ')}. " \
"You must manually remove the OSDs from the cluster " \
"and allow Ceph to rebalance before deleting these nodes."
answer = {"status" => "error", "error" => msg}
end
answer
end
def self.remove_ceph_mons(ctx, nodes)
answer = {"status" => "ready"}
ceph_mon_nodes = nodes.select { |n| n["roles"].include? "controller" }
ceph_mons = ceph_mon_nodes.collect{ |n| n["slave_name"] }
return answer if ceph_mon_nodes.empty?
#Get the list of mon nodes
result = {}
shell = nil
ceph_mon_nodes.each do |ceph_mon_node|
shell = MClient.new(ctx, "execute_shell_command", [ceph_mon_node["id"]], timeout=120, retries=1)
result = shell.execute(:cmd => "ceph -f json mon dump").first.results
break if result[:data][:exit_code] == 0
end
if result[:data][:exit_code] != 0
Astute.logger.debug "Ceph mon has not been found or has not been configured properly" \
" Safely removing nodes..."
return answer
end
mon_dump = JSON.parse(result[:data][:stdout])
left_mons = mon_dump['mons'].select { | n | n if ! ceph_mons.include? n['name'] }
left_mon_names = left_mons.collect { |n| n['name'] }
left_mon_ips = left_mons.collect { |n| n['addr'].split(":")[0] }
#Remove nodes from ceph cluster
Astute.logger.info("Removing ceph mons #{ceph_mons} from cluster")
ceph_mon_nodes.each do |node|
shell = MClient.new(ctx, "execute_shell_command", [node["id"]], timeout=120, retries=1)
#remove node from ceph mon list
shell.execute(:cmd => "ceph mon remove #{node["slave_name"]}").first.results
end
#Fix the ceph.conf on the left mon nodes
left_mon_names.each do |node|
mon_initial_members_cmd = "sed -i \"s/mon_initial_members.*/mon_initial_members\ = #{left_mon_names.join(" ")}/g\" /etc/ceph/ceph.conf"
mon_host_cmd = "sed -i \"s/mon_host.*/mon_host\ = #{left_mon_ips.join(" ")}/g\" /etc/ceph/ceph.conf"
shell = MClient.new(ctx, "execute_shell_command", [node.split('-')[1]], timeout=120, retries=1)
shell.execute(:cmd => mon_initial_members_cmd).first.results
shell.execute(:cmd => mon_host_cmd).first.results
end
Astute.logger.info("Ceph mons are left in cluster: #{left_mon_names}")
answer
end
def self.check_for_offline_nodes(ctx, nodes)
answer = {"status" => "ready"}
# FIXME(vsharshov): We send for node/cluster deletion operation
# as integer instead of String
mco_nodes = nodes.map { |n| n['uid'].to_s }
online_nodes = detect_available_nodes(ctx, mco_nodes)
offline_nodes = mco_nodes - online_nodes
if offline_nodes.present?
offline_nodes.map! { |e| {'uid' => e} }
msg = "MCollective is not running on nodes: " \
"#{offline_nodes.collect {|n| n['uid'] }.join(',')}. " \
"MCollective must be running to properly delete a node."
Astute.logger.warn msg
answer = {'status' => 'error',
'error' => msg,
'error_nodes' => offline_nodes}
end
answer
end
private
def self.detect_available_nodes(ctx, uids)
all_uids = uids.clone
available_uids = []
# In case of big amount of nodes we should do several calls to be sure
# about node status
Astute.config[:mc_retries].times.each do
systemtype = Astute::MClient.new(ctx, "systemtype", all_uids, check_result=false, 10)
available_nodes = systemtype.get_type
available_uids += available_nodes.map { |node| node.results[:sender] }
all_uids -= available_uids
break if all_uids.empty?
sleep Astute.config[:mc_retry_interval]
end
available_uids
end
end
end

34
lib/astute/pre_deploy_actions/connect_facts.rb
View File

@ -1,34 +0,0 @@
# Copyright 2014 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class ConnectFacts < PreDeployAction
def process(deployment_info, context)
deployment_info.each{ |node| connect_facts(context, node) }
Astute.logger.info "#{context.task_id}: Connect role facts for nodes"
end
private
def connect_facts(context, node)
run_shell_command(
context,
[node['uid']],
"ln -s -f /etc/#{node['role']}.yaml /etc/astute.yaml"
)
end
end #class
end

25
lib/astute/pre_deployment_actions/enable_puppet_deploy.rb
View File

@ -1,25 +0,0 @@
# Copyright 2014 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class EnablePuppetDeploy < PreDeploymentAction
# Unlock puppet (can be lock if puppet was killed by user)
def process(deployment_info, context)
nodes_uids = only_uniq_nodes(deployment_info).map{ |n| n['uid'] }
puppetd = MClient.new(context, "puppetd", nodes_uids)
puppetd.enable
end #process
end #class
end

55
lib/astute/pre_deployment_actions/generate_keys.rb
View File

@ -1,55 +0,0 @@
# Copyright 2014 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
require 'open3'
require 'fileutils'
module Astute
class GenerateKeys < PreDeploymentAction
# Generate ssh keys to future uploading to all cluster nodes
def process(deployment_info, context)
overwrite = false
deployment_id = deployment_info.first['deployment_id']
raise "Deployment_id is missing" unless deployment_id
Astute.config.puppet_keys.each do |key_name|
dir_path = File.join(Astute.config.keys_src_dir, deployment_id.to_s, key_name)
key_path = File.join(dir_path, key_name + '.key')
FileUtils.mkdir_p dir_path
raise DeploymentEngineError, "Could not create directory #{dir_path}" unless File.directory?(dir_path)
next if File.exist?(key_path) && !overwrite
# Generate key(<name>.key) and save it to <KEY_DIR>/<name>/<name>.key
File.delete key_path if File.exist? key_path
cmd = "openssl rand -base64 741 > #{key_path} 2>&1"
status, stdout, _stderr = run_system_command cmd
error_msg = "Could not generate key! Command: #{cmd}, output: #{stdout}, exit code: #{status}"
raise DeploymentEngineError, error_msg if status != 0
end
end #process
private
def run_system_command(cmd)
stdout, stderr, status = Open3.capture3 cmd
return status.exitstatus, stdout, stderr
end
end #class
end

55
lib/astute/pre_deployment_actions/generate_ssh_keys.rb
View File

@ -1,55 +0,0 @@
# Copyright 2014 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
require 'open3'
require 'fileutils'
module Astute
class GenerateSshKeys < PreDeploymentAction
# Generate ssh keys to future uploading to all cluster nodes
def process(deployment_info, context)
overwrite = false
deployment_id = deployment_info.first['deployment_id']
raise "Deployment_id is missing" unless deployment_id
Astute.config.puppet_ssh_keys.each do |key_name|
dir_path = File.join(Astute.config.keys_src_dir, deployment_id.to_s, key_name)
key_path = File.join(dir_path, key_name)
FileUtils.mkdir_p dir_path
raise DeploymentEngineError, "Could not create directory #{dir_path}" unless File.directory?(dir_path)
next if File.exist?(key_path) && !overwrite
# Generate 2 keys(<name> and <name>.pub) and save it to <KEY_DIR>/<name>/
File.delete key_path if File.exist? key_path
cmd = "ssh-keygen -b 2048 -t rsa -N '' -f #{key_path} 2>&1"
status, stdout, _stderr = run_system_command cmd
error_msg = "Could not generate ssh key! Command: #{cmd}, output: #{stdout}, exit code: #{status}"
raise DeploymentEngineError, error_msg if status != 0
end
end #process
private
def run_system_command(cmd)
stdout, stderr, status = Open3.capture3 cmd
return status.exitstatus, stdout, stderr
end
end #class
end

34
lib/astute/pre_deployment_actions/initial_connect_facts.rb
View File

@ -1,34 +0,0 @@
# Copyright 2015 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class InitialConnectFacts < PreDeploymentAction
def process(deployment_info, context)
only_uniq_nodes(deployment_info).each{ |node| connect_facts(context, node) }
Astute.logger.info "#{context.task_id}: Initial connect role facts for nodes"
end
private
def connect_facts(context, node)
run_shell_command(
context,
[node['uid']],
"ln -s -f /etc/#{node['role']}.yaml /etc/astute.yaml"
)
end
end #class
end

74
lib/astute/pre_deployment_actions/sync_puppet_stuff.rb
View File

@ -1,74 +0,0 @@
# Copyright 2014 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
require 'uri'
SYNC_RETRIES = 10
module Astute
class SyncPuppetStuff < PreDeploymentAction
# Sync puppet manifests and modules to every node
def process(deployment_info, context)
master_ip = deployment_info.first['master_ip']
modules_source = deployment_info.first['puppet']['modules']
manifests_source = deployment_info.first['puppet']['manifests']
# Paths to Puppet modules and manifests at the master node set by Nailgun
# Check fuel source code /deployment/puppet/nailgun/manifests/puppetsync.pp
schemas = [modules_source, manifests_source].map do |url|
begin
URI.parse(url).scheme
rescue URI::InvalidURIError => e
raise DeploymentEngineError, e.message
end
end
if schemas.select{ |x| x != schemas.first }.present?
raise DeploymentEngineError, "Scheme for puppet modules '#{schemas.first}' and" \
" puppet manifests '#{schemas.last}' not equivalent!"
end
nodes_uids = only_uniq_nodes(deployment_info).map{ |n| n['uid'] }
perform_with_limit(nodes_uids) do |part|
sync_puppet_stuff(context, part, schemas, modules_source, manifests_source)
end
end # process
private
def sync_puppet_stuff(context, node_uids, schemas, modules_source, manifests_source)
sync_mclient = MClient.new(context, "puppetsync", node_uids)
case schemas.first
when 'rsync'
begin
sync_mclient.rsync(:modules_source => modules_source,
:manifests_source => manifests_source
)
rescue MClientError => e
sync_retries ||= 0
sync_retries += 1
if sync_retries < SYNC_RETRIES
Astute.logger.warn("Rsync problem. Try to repeat: #{sync_retries} attempt")
retry
end
raise e
end
else
raise DeploymentEngineError, "Unknown scheme '#{schemas.first}' in #{modules_source}"
end
end #process
end #class
end

58
lib/astute/pre_deployment_actions/sync_tasks.rb
View File

@ -1,58 +0,0 @@
# Copyright 2014 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
SYNC_RETRIES = 10
module Astute
class SyncTasks < PreDeploymentAction
# Sync puppet manifests and modules to every node
def process(deployment_info, context)
return unless deployment_info.first['tasks_source']
# URI to Tasklib tasks at the master node set by Nailgun
master_ip = deployment_info.first['master_ip']
tasks_source = deployment_info.first['tasks_source'] || "rsync://#{master_ip}:/puppet/tasks/"
source = tasks_source.chomp('/').concat('/')
nodes_uids = only_uniq_nodes(deployment_info).map{ |n| n['uid'] }
perform_with_limit(nodes_uids) do |part|
rsync_tasks(context, source, part)
end
end
private
def rsync_tasks(context, source, nodes_uids)
path = '/etc/puppet/tasks/'
rsync_options = '-c -r --delete'
rsync_cmd = "mkdir -p #{path} && rsync #{rsync_options} #{source} #{path}"
sync_retries = 0
while sync_retries < SYNC_RETRIES
sync_retries += 1
response = run_shell_command(
context,
nodes_uids,
rsync_cmd,
300
)
break if response[:data][:exit_code] == 0
Astute.logger.warn("Rsync problem. Try to repeat: #{sync_retries} attempt")
end
end #rsync_tasks
end #class
end

45
lib/astute/pre_deployment_actions/sync_time.rb
View File

@ -1,45 +0,0 @@
# Copyright 2014 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class SyncTime < PreDeploymentAction
# Sync time
def process(deployment_info, context)
nodes_uids = only_uniq_nodes(deployment_info).map{ |n| n['uid'] }
cmd = "ntpdate -u $(egrep '^server' /etc/ntp.conf | sed '/^#/d' | awk '{print $2}')"
succeeded = false
Astute.config.mc_retries.times.each do
succeeded = run_shell_command_remotely(context, nodes_uids, cmd)
return if succeeded
sleep Astute.config.mc_retry_interval
end
if !succeeded
Astute.logger.warn "Run command: '#{cmd}' in nodes: #{nodes_uids} fail. " \
"Check debug output for more information. You can try "\
"to fix it problem manually."
end
end #process
private
def run_shell_command_remotely(context, nodes_uids, cmd)
response = run_shell_command(context, nodes_uids, cmd)
response.fetch(:data, {})[:exit_code] == 0
end
end #class
end

103
lib/astute/pre_deployment_actions/update_repo_sources.rb
View File

@ -1,103 +0,0 @@
# Copyright 2014 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class UpdateRepoSources < PreDeploymentAction
# Update packages source list
def process(deployment_info, context)
return unless deployment_info.first['repo_setup']['repos']
content = generate_repo_source(deployment_info)
deployment_info = only_uniq_nodes(deployment_info)
perform_with_limit(deployment_info) do |part|
upload_repo_source(context, part, content)
regenerate_metadata(context, part)
end
end
private
def generate_repo_source(deployment_info)
ubuntu_source = -> (repo) { "deb #{repo['uri']} #{repo['suite']} #{repo['section']}" }
centos_source = -> (repo) do
["[#{repo['name'].downcase}]", "name=#{repo['name']}", "baseurl=#{repo['uri']}", "gpgcheck=0"].join("\n")
end
formatter = case target_os(deployment_info)
when 'centos' then centos_source
when 'ubuntu' then ubuntu_source
end
content = []
deployment_info.first['repo_setup']['repos'].each do |repo|
content << formatter.call(repo)
end
content.join("\n")
end
def upload_repo_source(context, deployment_info, content)
upload_mclient = MClient.new(context, "uploadfile", deployment_info.map{ |n| n['uid'] }.uniq)
destination_path = case target_os(deployment_info)
when 'centos' then '/etc/yum.repos.d/nailgun.repo'
when 'ubuntu' then '/etc/apt/sources.list'
end
upload_mclient.upload(:path => destination_path,
:content => content,
:user_owner => 'root',
:group_owner => 'root',
:permissions => '0644',
:dir_permissions => '0755',
:overwrite => true,
:parents => true
)
end
def regenerate_metadata(context, deployment_info)
cmd = case target_os(deployment_info)
when 'centos' then "yum clean all"
when 'ubuntu' then "apt-get clean; apt-get update"
end
succeeded = false
nodes_uids = deployment_info.map{ |n| n['uid'] }.uniq
Astute.config.mc_retries.times.each do
succeeded = run_shell_command_remotely(context, nodes_uids, cmd)
return if succeeded
sleep Astute.config.mc_retry_interval
end
if !succeeded
raise DeploymentEngineError, "Run command: '#{cmd}' in nodes: #{nodes_uids} fail." \
" Check debug output for more information"
end
end
def target_os(deployment_info)
os = deployment_info.first['cobbler']['profile']
case os
when 'centos-x86_64' then 'centos'
when 'ubuntu_1404_x86_64' then 'ubuntu'
else
raise DeploymentEngineError, "Unknown system #{os}"
end
end
def run_shell_command_remotely(context, nodes_uids, cmd)
response = run_shell_command(context, nodes_uids, cmd)
response.fetch(:data, {})[:exit_code] == 0
end
end #class
end

60
lib/astute/pre_deployment_actions/upload_facts.rb
View File

@ -1,60 +0,0 @@
# Copyright 2014 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
require 'psych'
module Astute
class UploadFacts < PreDeploymentAction
def process(deployment_info, context)
deployment_info.each{ |node| upload_facts(context, node) }
Astute.logger.info "#{context.task_id}: Required attrs/metadata passed via facts extension"
end
private
# This is simple version of 'YAML::dump' with force quoting of strings started with prefixed numeral values
def safe_yaml_dump(obj)
visitor = Psych::Visitors::YAMLTree.new({})
visitor << obj
visitor.tree.grep(Psych::Nodes::Scalar).each do |node|
node.style = Psych::Nodes::Scalar::DOUBLE_QUOTED if
node.value =~ /^0[xbod0]+/i && node.plain && node.quoted
end
visitor.tree.yaml
end
def upload_facts(context, node)
# TODO: Should be changed to the default 'to_yaml' method only after upgrading
# to Ruby 2.1 everywhere on client nodes which used this YAML.
yaml_data = safe_yaml_dump(node)
Astute.logger.info "#{context.task_id}: storing metadata for node uid=#{node['uid']} "\
"role=#{node['role']}"
Astute.logger.debug "#{context.task_id}: stores metadata: #{yaml_data}"
# This is synchronious RPC call, so we are sure that data were sent and processed remotely
upload_mclient = Astute::MClient.new(context, "uploadfile", [node['uid']])
upload_mclient.upload(
:path => "/etc/#{node['role']}.yaml",
:content => yaml_data,
:overwrite => true,
:parents => true,
:permissions => '0600'
)
end
end #class
end

59
lib/astute/pre_deployment_actions/upload_keys.rb
View File

@ -1,59 +0,0 @@
# Copyright 2014 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class UploadKeys < PreDeploymentAction
# Upload ssh keys from master node to all cluster nodes
def process(deployment_info, context)
deployment_id = deployment_info.first['deployment_id'].to_s
nodes_ids = only_uniq_nodes(deployment_info).map{ |n| n['uid'] }
perform_with_limit(nodes_ids) do |ids|
upload_keys(context, ids, deployment_id)
end
end
private
def upload_keys(context, node_uids, deployment_id)
Astute.config.puppet_keys.each do |key_name|
upload_mclient = MClient.new(context, "uploadfile", node_uids)
key = key_name + '.key'
source_path = File.join(
Astute.config.keys_src_dir,
deployment_id,
key_name,
key
)
destination_path = File.join(
Astute.config.keys_dst_dir,
key_name,
key
)
content = File.read(source_path)
upload_mclient.upload(
:path => destination_path,
:content => content,
:user_owner => 'root',
:group_owner => 'root',
:permissions => '0600',
:dir_permissions => '0700',
:overwrite => true,
:parents => true
)
end
end #upload_keys
end #class
end

59
lib/astute/pre_deployment_actions/upload_ssh_keys.rb
View File

@ -1,59 +0,0 @@
# Copyright 2014 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class UploadSshKeys < PreDeploymentAction
# Upload ssh keys from master node to all cluster nodes
def process(deployment_info, context)
deployment_id = deployment_info.first['deployment_id'].to_s
nodes_ids = only_uniq_nodes(deployment_info).map{ |n| n['uid'] }
perform_with_limit(nodes_ids) do |ids|
upload_keys(context, ids, deployment_id)
end
end
private
def upload_keys(context, node_uids, deployment_id)
Astute.config.puppet_ssh_keys.each do |key_name|
upload_mclient = MClient.new(context, "uploadfile", node_uids)
[key_name, key_name + ".pub"].each do |ssh_key|
source_path = File.join(
Astute.config.keys_src_dir,
deployment_id,
key_name,
ssh_key
)
destination_path = File.join(
Astute.config.keys_dst_dir,
key_name,
ssh_key
)
content = File.read(source_path)
upload_mclient.upload(:path => destination_path,
:content => content,
:user_owner => 'root',
:group_owner => 'root',
:permissions => '0600',
:dir_permissions => '0700',
:overwrite => true,
:parents => true
)
end
end
end #upload_keys
end #class
end

77
lib/astute/pre_node_actions/pre_patching.rb
View File

@ -1,77 +0,0 @@
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class PrePatching < PreNodeAction
def process(deployment_info, context)
return unless deployment_info.first['openstack_version_prev']
# We should stop services with SIGTERM or even SIGKILL.
# StopOSTServices do this and should be run before.
remove_cmd = getremovepackage_cmd(deployment_info)
nodes = deployment_info.map { |n| n['uid'] }
Astute.logger.info "Starting removal of error-prone packages"
Astute.logger.info "Executing command #{remove_cmd}"
Astute.logger.info "On nodes\n#{nodes.pretty_inspect}"
response = run_shell_command(context, nodes, remove_cmd, 600)
if response[:data][:exit_code] != 0
Astute.logger.error "#{context.task_id}: Fail to remove packages, "\
"check the debugging output for details"
end
Astute.logger.info "#{context.task_id}: Finished pre-patching hook"
end #process
def getremovepackage_cmd(deployment_info)
os = deployment_info.first['cobbler']['profile']
case os
when /centos/i then "yum -y remove #{centos_packages}"
when /ubuntu/i then "aptitude -y remove #{ubuntu_packages}"
else
raise DeploymentEngineError, "Unknown system #{os}"
end
end
def centos_packages
packages = <<-Packages
python-oslo-messaging python-oslo-config openstack-heat-common
python-nova python-routes python-routes1.12 python-neutron
python-django-horizon murano-api sahara sahara-dashboard
python-ceilometer openstack-swift openstack-utils
python-glance python-glanceclient python-cinder
python-sqlalchemy python-testtools
Packages
packages.tr!("\n"," ")
end
def ubuntu_packages
packages = <<-Packages
python-oslo.messaging python-oslo.config python-heat python-nova
python-routes python-routes1.13 python-neutron python-django-horizon
murano-common murano-api sahara sahara-dashboard python-ceilometer
python-swift python-cinder python-keystoneclient python-neutronclient
python-novaclient python-swiftclient python-troveclient
python-sqlalchemy python-testtools
Packages
packages.tr!("\n"," ")
end
end #class
end

50
lib/astute/pre_node_actions/pre_patching_ha.rb
View File

@ -1,50 +0,0 @@
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class PrePatchingHa < PreNodeAction
def process(deployment_info, context)
return if deployment_info.first['openstack_version_prev'].nil? ||
deployment_info.first['deployment_mode'] !~ /ha/i
# Run only once for node. If one of role is controller or primary-controller
# generate new deployment_info block.
# Important for 'mongo' role which run early then 'controller'
current_uids = deployment_info.map{ |n| n['uid'] }
controllers = deployment_info.first['nodes'].select{ |n| current_uids.include?(n['uid']) && n['role'] =~ /controller/i }
c_deployment_info = deployment_info.select { |d_i| controllers.map{ |n| n['uid'] }.include? d_i['uid'] }
return if c_deployment_info.empty?
c_deployment_info.each do |c_d_i|
c_d_i['role'] = controllers.find{ |c| c['uid'] == c_d_i['uid'] }['role']
end
controller_nodes = c_deployment_info.map{ |n| n['uid'] }
Astute.logger.info "Starting migration of pacemaker services from " \
"nodes\n#{controller_nodes.pretty_inspect}"
Astute::Pacemaker.commands(action='stop', c_deployment_info).each do |pcmk_ban_cmd|
response = run_shell_command(context, controller_nodes, pcmk_ban_cmd)
if response[:data][:exit_code] != 0
Astute.logger.warn "#{context.task_id}: Failed to ban service, "\
"check the debugging output for details"
end
end
Astute.logger.info "#{context.task_id}: Finished pre-patching-ha hook"
end #process
end #class
end

65
lib/astute/pre_node_actions/stop_ost_services.rb
View File

@ -1,65 +0,0 @@
# Copyright 2014 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class StopOSTServices < PreNodeAction
def process(deployment_info, context)
old_env = deployment_info.first['openstack_version_prev']
return unless old_env
Astute.logger.info "Stop all Openstack services hook start"
node_uids = deployment_info.collect { |n| n['uid'] }
file_content = get_file
target_file = '/tmp/stop_services.rb'
upload_script(context, node_uids, target_file, file_content)
Astute.logger.info "Running file: #{target_file} on node uids: #{node_uids.join ', '}"
response = run_shell_command(context, node_uids, "/usr/bin/ruby #{target_file} |tee /tmp/stop_services.log")
if response[:data][:exit_code] != 0
Astute.logger.warn "#{context.task_id}: Script returned error code #{response[:data][:exit_code]}"
end
Astute.logger.info "#{context.task_id}: Finished stop services pre-patching hook"
end #process
private
def get_file
File.read File.join(File.dirname(__FILE__), 'stop_services.script')
end
def upload_script(context, node_uids, target_file, file_content)
target_file = '/tmp/stop_services.rb'
Astute.logger.info "Uploading file: #{target_file} to nodes uids: #{node_uids.join ', '}"
MClient.new(context, "uploadfile", node_uids).upload(
:path => target_file,
:content => file_content,
:user_owner => 'root',
:group_owner => 'root',
:permissions => '0700',
:overwrite => true,
:parents => true
)
rescue MClientTimeout, MClientError => e
Astute.logger.error("#{context.task_id}: mcollective error: #{e.message}")
end
end #class
end #module

213
lib/astute/pre_node_actions/stop_services.script
View File

@ -1,213 +0,0 @@
#!/usr/bin/env ruby
require 'rubygems'
require 'facter'
# pre-deploy hook library
module PreDeploy
@dry_run = false
@process_tree = nil
@osfamily = nil
@stop_services_regexp = %r{nova|cinder|glance|keystone|neutron|sahara|murano|ceilometer|heat|swift|apache2|httpd}
# get regexp that selects services and processes to stop
# @return [Regexp]
def self.stop_services_regexp
@stop_services_regexp
end
# set regexp that selects services and processes to stop
# @param value [Regexp]
def self.stop_services_regexp=(value)
@stop_services_regexp = value
end
# get osfamily from facter
# @return [String]
def self.osfamily
return @osfamily if @osfamily
@osfamily = Facter.value 'osfamily'
end
# get dry run without doing anything switch
# @return [TrueClass,FalseClass]
def self.dry_run
@dry_run
end
# set dry run without doing anything switch
# @param value [TrueClass,FalseClass]
def self.dry_run=(value)
@dry_run = value
end
# get ps from shell command
# @return [String]
def self.ps
`ps haxo pid,ppid,cmd`
end
# get service statu from shell command
# @return String
def self.services
`service --status-all 2>&1`
end
# same as process_tree but reset mnemoization
# @return [Hash<Integer => Hash<Symbol => String,Integer>>]
def self.process_tree_with_renew
@process_tree = nil
self.process_tree
end
# build process tree from process list
# @return [Hash<Integer => Hash<Symbol => String,Integer>>]
def self.process_tree
return @process_tree if @process_tree
@process_tree = {}
self.ps.split("\n").each do |p|
f = p.split
pid = f.shift.to_i
ppid = f.shift.to_i
cmd = f.join ' '
# create entry for this pid if not present
@process_tree[pid] = {
:children => []
} unless @process_tree.key? pid
# fill this entry
@process_tree[pid][:ppid] = ppid
@process_tree[pid][:pid] = pid
@process_tree[pid][:cmd] = cmd
# create entry for parent process if not present
@process_tree[ppid] = {
:children => []
} unless @process_tree.key? ppid
# fill parent's children
@process_tree[ppid][:children] << pid
end
@process_tree
end
# kill selected pid or array of them
# @param pids [Integer,String] Pids to kill
# @param signal [Integer,String] Which signal?
# @param recursive [TrueClass,FalseClass] Kill children too?
# @return [TrueClass,FalseClass] Was the signal sent? Process may still be present even on success.
def self.kill_pids(pids, signal = 9, recursive = true)
pids = Array pids
pids_to_kill = pids.inject([]) do |all_pids, pid|
pid = pid.to_i
if recursive
all_pids + self.get_children_pids(pid)
else
all_pids << pid
end
end
pids_to_kill.uniq!
pids_to_kill.sort!
return false unless pids_to_kill.any?
puts "Kill these pids: #{pids_to_kill.join ', '} with signal #{signal}"
self.run "kill -#{signal} #{pids_to_kill.join ' '}"
end
# recursion to find all children pids
# @return [Array<Integer>]
def self.get_children_pids(pid)
pid = pid.to_i
unless self.process_tree.key? pid
puts "No such pid: #{pid}"
return []
end
self.process_tree[pid][:children].inject([pid]) do |all_children_pids, child_pid|
all_children_pids + self.get_children_pids(child_pid)
end
end
# same as services_to_stop but reset mnemoization
# @return Array[String]
def self.services_to_stop_with_renew
@services_to_stop = nil
self.services_to_stop
end
# find running services that should be stopped
# uses service status and regex to filter
# @return [Array<String>]
def self.services_to_stop
return @services_to_stop if @services_to_stop
@services_to_stop = self.services.split("\n").inject([]) do |services_to_stop, service|
fields = service.chomp.split
running = if fields[4] == 'running...'
fields[0]
elsif fields[1] == '+'
fields[3]
else
nil
end
if running =~ @stop_services_regexp
# replace wrong service name
running = 'httpd' if running == 'httpd.event' and self.osfamily == 'RedHat'
running = 'openstack-keystone' if running == 'keystone' and self.osfamily == 'RedHat'
services_to_stop << running
else
services_to_stop
end
end
end
# stop services that match stop_services_regex
def self.stop_services
self.services_to_stop.each do |service|
puts "Try to stop service: #{service}"
self.run "service #{service} stop"
end
end
# filter pids which cmd match regexp
# @param regexp <Regexp> Search pids by this regexp
# @return [Hash<Integer => Hash<Symbol => String,Integer>>]
def self.pids_by_regexp(regexp)
matched = {}
self.process_tree.each do |pid,process|
matched[pid] = process if process[:cmd] =~ regexp
end
matched
end
# kill pids that match stop_services_regexp
# @return <TrueClass,FalseClass>
def self.kill_pids_by_stop_regexp
pids = self.pids_by_regexp(@stop_services_regexp).keys
self.kill_pids pids
end
# here be other fixes
# TODO: not needed anymore?
def self.misc_fixes
if self.osfamily == 'Debian'
puts 'Enabling WSGI module'
self.run 'a2enmod wsgi'
end
end
# run the shell command with dry_run support
# @param cmd [String] Command to run
def self.run(cmd)
command = "#{self.dry_run ? 'echo' : ''} #{cmd} 2>&1"
system command
end
end # class
if __FILE__ == $0
# PreDeploy.dry_run = true
PreDeploy.misc_fixes
PreDeploy.stop_services
PreDeploy.kill_pids_by_stop_regexp
end

511
lib/astute/provision.rb
View File

@ -1,511 +0,0 @@
# Copyright 2015 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class Provisioner
def initialize(log_parsing=false)
@log_parsing = log_parsing
end
def node_type(reporter, task_id, nodes_uids, timeout=nil)
context = Context.new(task_id, reporter)
systemtype = MClient.new(
context,
"systemtype",
nodes_uids,
_check_result=false,
timeout
)
systems = systemtype.get_type
systems.map do |n|
{
'uid' => n.results[:sender],
'node_type' => n.results[:data][:node_type].to_s.chomp
}
end
end
def provision(reporter, task_id, provisioning_info)
engine_attrs = provisioning_info['engine']
nodes = provisioning_info['nodes']
raise "Nodes to provision are not provided!" if nodes.empty?
fault_tolerance = provisioning_info.fetch('fault_tolerance', [])
cobbler = CobblerManager.new(engine_attrs, reporter)
result_msg = {'nodes' => []}
begin
prepare_nodes(reporter, task_id, engine_attrs, nodes, cobbler)
failed_uids, timeouted_uids = provision_and_watch_progress(reporter,
task_id,
Array.new(nodes),
engine_attrs,
fault_tolerance)
rescue => e
Astute.logger.error("Error occured while provisioning:\n#{e.pretty_inspect}")
reporter.report({
'status' => 'error',
'error' => e.message,
'progress' => 100})
unlock_nodes_discovery(reporter, task_id, nodes.map { |n| n['uid'] }, nodes)
raise e
end
handle_failed_nodes(failed_uids, result_msg)
if failed_uids.count > 0
unlock_nodes_discovery(reporter, task_id, failed_uids, nodes)
end
handle_timeouted_nodes(timeouted_uids, result_msg)
node_uids = nodes.map { |n| n['uid'] }
(node_uids - timeouted_uids - failed_uids).each do |uid|
result_msg['nodes'] << {'uid' => uid, 'progress' => 100, 'status' => 'provisioned'}
end
if should_fail(failed_uids + timeouted_uids, fault_tolerance)
result_msg['status'] = 'error'
result_msg['error'] = 'Too many nodes failed to provision'
result_msg['progress'] = 100
end
# If there was no errors, then set status to ready
result_msg.reverse_merge!({'status' => 'ready', 'progress' => 100})
Astute.logger.info "Message: #{result_msg}"
reporter.report(result_msg)
result_msg
rescue => e
Rsyslogd.send_sighup(
Context.new(task_id, reporter),
engine_attrs["master_ip"]
)
raise e
end
def provision_and_watch_progress(reporter,
task_id,
nodes_to_provision,
engine_attrs,
fault_tolerance)
raise "Nodes to provision are not provided!" if nodes_to_provision.empty?
provision_log_parser = @log_parsing ? LogParser::ParseProvisionLogs.new : LogParser::NoParsing.new
prepare_logs_for_parsing(provision_log_parser, nodes_to_provision)
nodes_not_booted = []
nodes = []
nodes_timeout = {}
timeouted_uids = []
failed_uids = []
max_nodes = Astute.config[:max_nodes_to_provision]
Astute.logger.debug("Starting provision")
catch :done do
loop do
sleep_not_greater_than(20) do
#provision more
if nodes_not_booted.count < max_nodes && nodes_to_provision.count > 0
new_nodes = nodes_to_provision.shift(max_nodes - nodes_not_booted.count)
Astute.logger.debug("Provisioning nodes: #{new_nodes}")
failed_uids += provision_piece(reporter, task_id, engine_attrs, new_nodes)
Astute.logger.info "Nodes failed to reboot: #{failed_uids} "
nodes_not_booted += new_nodes.map{ |n| n['uid'] }
nodes_not_booted -= failed_uids
nodes += new_nodes
timeout_time = Time.now.utc + Astute.config.provisioning_timeout
new_nodes.each {|n| nodes_timeout[n['uid']] = timeout_time}
end
nodes_types = node_type(reporter, task_id, nodes.map {|n| n['uid']}, 5)
target_uids, nodes_not_booted, reject_uids = analize_node_types(nodes_types, nodes_not_booted)
if reject_uids.present?
ctx ||= Context.new(task_id, reporter)
reject_nodes = reject_uids.map { |uid| {'uid' => uid } }
NodesRemover.new(ctx, reject_nodes, _reboot=true).remove
end
#check timouted nodes
nodes_not_booted.each do |uid|
time_now = Time.now.utc
if nodes_timeout[uid] < time_now
Astute.logger.info "Node timed out to provision: #{uid} "
timeouted_uids.push(uid)
end
end
nodes_not_booted -= timeouted_uids
if should_fail(failed_uids + timeouted_uids, fault_tolerance)
Astute.logger.debug("Aborting provision. To many nodes failed: #{failed_uids + timeouted_uids}")
Astute.logger.debug("Those nodes where we not yet started provision will be set to error mode")
failed_uids += nodes_to_provision.map{ |n| n['uid'] }
return failed_uids, timeouted_uids
end
if nodes_not_booted.empty? and nodes_to_provision.empty?
Astute.logger.info "Provisioning finished"
throw :done
end
Astute.logger.debug("Still provisioning following nodes: #{nodes_not_booted}")
report_about_progress(reporter, provision_log_parser, target_uids, nodes)
end
end
end
return failed_uids, timeouted_uids
end
def remove_nodes(reporter, task_id, engine_attrs, nodes, options)
options.reverse_merge!({
:reboot => true,
:raise_if_error => false,
:reset => false
})
cobbler = CobblerManager.new(engine_attrs, reporter)
if options[:reset]
cobbler.edit_nodes(nodes, {'profile' => Astute.config.bootstrap_profile})
cobbler.netboot_nodes(nodes, true)
else
cobbler.remove_nodes(nodes)
end
ctx = Context.new(task_id, reporter)
result = NodesRemover.new(ctx, nodes, options[:reboot]).remove
if (result['error_nodes'] || result['inaccessible_nodes']) && options[:raise_if_error]
bad_node_ids = result.fetch('error_nodes', []) +
result.fetch('inaccessible_nodes', [])
raise "Mcollective problem with nodes #{bad_node_ids}, please check log for details"
end
Rsyslogd.send_sighup(ctx, engine_attrs["master_ip"])
result
end
def stop_provision(reporter, task_id, engine_attrs, nodes)
ctx = Context.new(task_id, reporter)
_provisioned_nodes, result = stop_provision_via_mcollective(ctx, nodes)
result['status'] = 'error' if result['error_nodes'].present?
Rsyslogd.send_sighup(
Context.new(task_id, reporter),
engine_attrs["master_ip"]
)
result
end
def provision_piece(reporter, task_id, engine_attrs, nodes)
cobbler = CobblerManager.new(engine_attrs, reporter)
failed_uids = []
# TODO(kozhukalov): do not forget about execute_shell_command timeout which is 3600
# provision_and_watch_progress has provisioning_timeout + 3600 is much longer than provisioning_timeout
# IBP is implemented in terms of Fuel Agent installed into bootstrap ramdisk
# we don't want nodes to be rebooted into OS installer ramdisk
cobbler.edit_nodes(nodes, {'profile' => Astute.config.bootstrap_profile})
# change node type to prevent unexpected erase
change_nodes_type(reporter, task_id, nodes)
# Run parallel reporter
report_image_provision(reporter, task_id, nodes) do
failed_uids |= image_provision(reporter, task_id, nodes)
end
provisioned_nodes = nodes.reject { |n| failed_uids.include? n['uid'] }
# disabling pxe boot (chain loader) for nodes which succeeded
cobbler.netboot_nodes(provisioned_nodes, false)
# in case of IBP we reboot only those nodes which we managed to provision
soft_reboot(
reporter,
task_id,
provisioned_nodes.map{ |n| n['uid'] },
'reboot_provisioned_nodes'
)
return failed_uids
end
def report_image_provision(reporter, task_id, nodes,
provision_log_parser=LogParser::ParseProvisionLogs.new, &block)
prepare_logs_for_parsing(provision_log_parser, nodes)
watch_and_report = Thread.new do
loop do
report_about_progress(reporter, provision_log_parser, [], nodes)
sleep 1
end
end
block.call
ensure
watch_and_report.exit if defined? watch_and_report
end
private
def image_provision(reporter, task_id, nodes)
ImageProvision.provision(Context.new(task_id, reporter), nodes)
end
def soft_reboot(reporter, task_id, nodes, task_name)
ImageProvision.reboot(Context.new(task_id, reporter), nodes, task_name)
end
def report_result(result, reporter)
default_result = {'status' => 'ready', 'progress' => 100}
result = {} unless result.instance_of?(Hash)
status = default_result.merge(result)
reporter.report(status)
end
def prepare_logs_for_parsing(provision_log_parser, nodes)
sleep_not_greater_than(10) do # Wait while nodes going to reboot
Astute.logger.info "Starting OS provisioning for nodes: #{nodes.map{ |n| n['uid'] }.join(',')}"
begin
provision_log_parser.prepare(nodes)
rescue => e
Astute.logger.warn "Some error occurred when prepare LogParser: #{e.message}, trace: #{e.format_backtrace}"
end
end
end
def analize_node_types(types, nodes_not_booted)
types.each { |t| Astute.logger.debug("Got node types: uid=#{t['uid']} type=#{t['node_type']}") }
target_uids = types.reject{ |n| n['node_type'] != 'target' }.map{ |n| n['uid'] }
reject_uids = types.reject{ |n| ['target', 'image'].include? n['node_type'] }.map{ |n| n['uid'] }
Astute.logger.debug("Not target nodes will be rejected: #{reject_uids.join(',')}")
nodes_not_booted -= target_uids
Astute.logger.debug "Not provisioned: #{nodes_not_booted.join(',')}, " \
"got target OSes: #{target_uids.join(',')}"
return target_uids, nodes_not_booted, reject_uids
end
def sleep_not_greater_than(sleep_time, &block)
time = Time.now.to_f
block.call
time = time + sleep_time - Time.now.to_f
sleep(time) if time > 0
end
def report_about_progress(reporter, provision_log_parser, target_uids, nodes)
begin
nodes_progress = provision_log_parser.progress_calculate(nodes.map{ |n| n['uid'] }, nodes)
nodes_progress.each do |n|
if target_uids.include?(n['uid'])
n['progress'] = 100
n['status'] = 'provisioned'
else
n['status'] = 'provisioning'
end
end
reporter.report({'nodes' => nodes_progress})
rescue => e
Astute.logger.warn "Some error occurred when parse logs for nodes progress: #{e.message}, trace: #{e.format_backtrace}"
end
end
def stop_provision_via_mcollective(ctx, nodes)
return [], {} if nodes.empty?
mco_result = {}
nodes_uids = nodes.map{ |n| n['uid'] }
Astute.config.mc_retries.times do |i|
sleep Astute.config.nodes_remove_interval
Astute.logger.debug "Trying to connect to nodes #{nodes_uids} using mcollective"
nodes_types = node_type(ctx.reporter, ctx.task_id, nodes_uids, 2)
next if nodes_types.empty?
provisioned = nodes_types.select{ |n| ['target', 'bootstrap', 'image'].include? n['node_type'] }
.map{ |n| {'uid' => n['uid']} }
current_mco_result = NodesRemover.new(ctx, provisioned, _reboot=true).remove
Astute.logger.debug "Retry result #{i}: "\
"mco success nodes: #{current_mco_result['nodes']}, "\
"mco error nodes: #{current_mco_result['error_nodes']}, "\
"mco inaccessible nodes: #{current_mco_result['inaccessible_nodes']}"
mco_result = merge_rm_nodes_result(mco_result, current_mco_result)
nodes_uids -= provisioned.map{ |n| n['uid'] }
break if nodes_uids.empty?
end
provisioned_nodes = nodes.map{ |n| {'uid' => n['uid']} } - nodes_uids.map {|n| {'uid' => n} }
Astute.logger.debug "MCO final result: "\
"mco success nodes: #{mco_result['nodes']}, "\
"mco error nodes: #{mco_result['error_nodes']}, "\
"mco inaccessible nodes: #{mco_result['inaccessible_nodes']}, "\
"all mco nodes: #{provisioned_nodes}"
return provisioned_nodes, mco_result
end
def unlock_nodes_discovery(reporter, task_id="", failed_uids, nodes)
nodes_uids = nodes.select{ |n| failed_uids.include?(n['uid']) }
.map{ |n| n['uid'] }
shell = MClient.new(Context.new(task_id, reporter),
'execute_shell_command',
nodes_uids,
_check_result=false,
_timeout=2)
mco_result = shell.execute(:cmd => "rm -f #{Astute.config.agent_nodiscover_file}")
result = mco_result.map do |n|
{
'uid' => n.results[:sender],
'exit code' => n.results[:data][:exit_code]
}
end
Astute.logger.debug "Unlock discovery for failed nodes. Result: #{result}"
end
def merge_rm_nodes_result(res1, res2)
['nodes', 'error_nodes', 'inaccessible_nodes'].inject({}) do |result, node_status|
result[node_status] = (res1.fetch(node_status, []) + res2.fetch(node_status, [])).uniq
result
end
end
def change_nodes_type(reporter, task_id, nodes, type="image")
nodes_uids = nodes.map{ |n| n['uid'] }
shell = MClient.new(Context.new(task_id, reporter),
'execute_shell_command',
nodes_uids,
_check_result=false,
_timeout=5)
mco_result = shell.execute(:cmd => "echo '#{type}' > /etc/nailgun_systemtype")
result = mco_result.map do |n|
{
'uid' => n.results[:sender],
'exit code' => n.results[:data][:exit_code]
}
end
Astute.logger.debug "Change node type to #{type}. Result: #{result}"
end
def handle_failed_nodes(failed_uids, result_msg)
if failed_uids.present?
Astute.logger.error("Provision of some nodes failed. Failed nodes: #{failed_uids}")
nodes_progress = failed_uids.map do |n|
{
'uid' => n,
'status' => 'error',
'error_msg' => "Failed to provision",
'progress' => 100,
'error_type' => 'provision'
}
end
result_msg['nodes'] += nodes_progress
end
end
def handle_timeouted_nodes(timeouted_uids, result_msg)
if timeouted_uids.present?
Astute.logger.error("Timeout of provisioning is exceeded. Nodes not booted: #{timeouted_uids}")
nodes_progress = timeouted_uids.map do |n|
{
'uid' => n,
'status' => 'error',
'error_msg' => "Timeout of provisioning is exceeded",
'progress' => 100,
'error_type' => 'provision'
}
end
result_msg['nodes'] += nodes_progress
end
end
def should_fail(failed_uids, fault_tolerance)
return failed_uids.present? if fault_tolerance.empty?
uids_in_groups = []
fault_tolerance.each do |group|
failed_from_group = failed_uids.select { |uid| group['uids'].include? uid }
uids_in_groups += failed_from_group
max_to_fail = group['percentage'] / 100.0 * group['uids'].count
if failed_from_group.count > max_to_fail
return true
end
end
failed_uids -= uids_in_groups
if failed_uids.present?
return true
end
false
end
def prepare_nodes(reporter, task_id, engine_attrs, nodes, cobbler)
# 1. Erase all nodes
# 2. Return already provisioned node to bootstrap state
# 3. Delete and add again nodes to Cobbler
Astute.logger.info "Preparing nodes for installation"
existent_nodes = cobbler.get_existent_nodes(nodes)
# Change node type to prevent wrong node detection as provisioned
# Also this type if node will not rebooted, Astute will be allowed
# to try to reboot such nodes again
if existent_nodes.present?
change_nodes_type(reporter, task_id, existent_nodes, 'reprovisioned')
end
remove_nodes(
reporter,
task_id,
engine_attrs,
nodes,
{:reboot => false,
:raise_if_error => true,
:reset => true}
)
if existent_nodes.present?
soft_reboot(
reporter,
task_id,
existent_nodes.map{ |n| n['uid'] },
"reboot_reprovisioned_nodes"
)
end
cobbler.remove_nodes(nodes)
# NOTE(kozhukalov): We try to find out if there are systems
# in the Cobbler with the same MAC addresses. If so, Cobbler is going
# to throw MAC address duplication error. We need to remove these
# nodes.
mac_duplicate_names = cobbler.get_mac_duplicate_names(nodes)
if mac_duplicate_names.present?
cobbler.remove_nodes(mac_duplicate_names.map {|n| {'slave_name' => n}})
end
cobbler.add_nodes(nodes)
end
end
end

282
lib/astute/puppet_job.rb
View File

@ -1,282 +0,0 @@
# Copyright 2016 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class PuppetJob
FINAL_JOB_TASK_STATUSES = [
'successful', 'failed'
]
JOB_TASK_STATUSES = [
'successful', 'failed', 'running'
]
SUCCEED_STATUSES = [
'succeed'
]
BUSY_STATUSES = [
'running'
]
UNDEFINED_STATUSES = [
'undefined'
]
STOPED_STATUSES = [
'stopped', 'disabled'
]
def initialize(task, puppet_mclient, options)
@task = task
@retries = options['retries']
@puppet_start_timeout = options['puppet_start_timeout']
@puppet_start_interval = options['puppet_start_interval']
@time_observer = TimeObserver.new(options['timeout'])
@succeed_retries = options['succeed_retries']
@undefined_retries = options['undefined_retries']
@original_undefined_retries = options['undefined_retries']
@puppet_mclient = puppet_mclient
end
# Run selected puppet manifest on node
# @return [void]
def run
Astute.logger.info "Start puppet with timeout "\
"#{@time_observer.time_limit} sec. #{task_details_for_log}"
@time_observer.start
puppetd_run
self.task_status = 'running'
end
# Return actual status of puppet run
# @return [String] Task status: successful, failed or running
def status
return @task_status if job_ended?
current_task_status = puppet_to_task_status(puppet_status)
self.task_status = case current_task_status
when 'successful'
processing_succeed_task
when 'running'
processing_running_task
when 'failed'
processing_error_task
when 'undefined'
processing_undefined_task
end
time_is_up! if should_stop?
@task_status
end
# Return actual last run summary for puppet run
# @return [Hash] Puppet summary
def summary
@puppet_mclient.summary
end
private
# Should stop process or not: task is still running but we are out of time
# @return [true, false]
def should_stop?
@time_observer.time_is_up? &&
FINAL_JOB_TASK_STATUSES.exclude?(@task_status)
end
# Is job has ended or not
# @return [true, false]
def job_ended?
FINAL_JOB_TASK_STATUSES.include?(@task_status)
end
# Set task status to failed and reset retires counter to 0 to avoid
# redundant retries
# @return [void]
def time_is_up!
Astute.logger.error "Puppet agent took too long to run puppet task."\
" Mark task as failed. #{task_details_for_log}"
self.task_status = 'failed'
end
# Setup task status
# @param [String] status The task status
# @return [void]
# @raise [StatusValidationError] Unknown job status
def task_status=(status)
if JOB_TASK_STATUSES.include?(status)
@task_status = status
else
raise StatusValidationError,
"Unknow job status: #{status}. Expected: #{JOB_TASK_STATUSES}"
end
end
# Return actual status of puppet using mcollective puppet agent
# @return [String]: puppet status
def puppet_status
actual_status = @puppet_mclient.status
log_current_status(actual_status)
if UNDEFINED_STATUSES.include?(actual_status)
Astute.logger.warn "Error to get puppet status. "\
"#{task_details_for_log}."
end
actual_status
end
# Run puppet manifest using mcollective puppet agent
# @return [true, false] Is puppet run has started or not
# TODO(vsharshov): need refactoring to make this be async call
def puppetd_run
puppet_run_obsorver = TimeObserver.new(@puppet_start_timeout)
puppet_run_obsorver.start
while puppet_run_obsorver.enough_time?
is_running = @puppet_mclient.run
return true if is_running
Astute.logger.debug "Could not run puppet process "\
"#{task_details_for_log}. Left #{puppet_run_obsorver.left_time} sec"
sleep @puppet_start_interval
end
Astute.logger.error "Problem with puppet start. Time "\
"(#{@puppet_start_timeout} sec) is over. #{task_details_for_log}"
false
end
# Convert puppet status to task status
# @param [String] puppet_status The puppet status of task
# @return [String] Task status
# @raise [StatusValidationError] Unknown puppet status
def puppet_to_task_status(mco_puppet_status)
case
when SUCCEED_STATUSES.include?(mco_puppet_status)
'successful'
when BUSY_STATUSES.include?(mco_puppet_status)
'running'
when STOPED_STATUSES.include?(mco_puppet_status)
'failed'
when UNDEFINED_STATUSES.include?(mco_puppet_status)
'undefined'
else
raise StatusValidationError,
"Unknow puppet status: #{mco_puppet_status}"
end
end
# Return short useful info about node and puppet task
# @return [String]
def task_details_for_log
"Node #{@puppet_mclient.node_id}, task #{@task}, manifest "\
"#{@puppet_mclient.manifest}"
end
# Write to log with needed message level actual task status
# @param [String] status Actual puppet status of task
# @return [void]
def log_current_status(status)
message = "#{task_details_for_log}, status: #{status}"
if (UNDEFINED_STATUSES + STOPED_STATUSES).include?(status)
Astute.logger.error message
else
Astute.logger.debug message
end
end
# Process additional action in case of puppet succeed
# @return [String] Task status: successful or running
def processing_succeed_task
reset_undefined_retries!
Astute.logger.debug "Puppet completed within "\
"#{@time_observer.since_start} seconds"
if @succeed_retries > 0
@succeed_retries -= 1
Astute.logger.debug "Succeed puppet on node will be "\
"restarted. #{@succeed_retries} retries remained. "\
"#{task_details_for_log}"
Astute.logger.info "Retrying to run puppet for following succeed "\
"node: #{@puppet_mclient.node_id}"
puppetd_run
'running'
else
Astute.logger.info "Node #{@puppet_mclient.node_id} has succeed "\
"to deploy. #{task_details_for_log}"
'successful'
end
end
# Process additional action in case of puppet failed
# @return [String] Task status: failed or running
def processing_error_task
reset_undefined_retries!
if @retries > 0
@retries -= 1
Astute.logger.debug "Puppet on node will be "\
"restarted because of fail. #{@retries} retries remained."\
"#{task_details_for_log}"
Astute.logger.info "Retrying to run puppet for following error "\
"nodes: #{@puppet_mclient.node_id}"
puppetd_run
'running'
else
Astute.logger.error "Node has failed to deploy. There is"\
" no more retries for puppet run. #{task_details_for_log}"
'failed'
end
end
# Process additional action in case of undefined puppet status
# @return [String] Task status: failed or running
def processing_undefined_task
if @undefined_retries > 0
@undefined_retries -= 1
Astute.logger.debug "Puppet on node has undefined status. "\
"#{@undefined_retries} retries remained. "\
"#{task_details_for_log}"
Astute.logger.info "Retrying to check status for following "\
"nodes: #{@puppet_mclient.node_id}"
'running'
else
Astute.logger.error "Node has failed to get status. There is"\
" no more retries for status check. #{task_details_for_log}"
'failed'
end
end
# Process additional action in case of puppet running
# @return [String]: Task status: successful, failed or running
def processing_running_task
reset_undefined_retries!
'running'
end
# Reset undefined retries to original value
# @return [void]
def reset_undefined_retries!
return if @undefined_retries == @original_undefined_retries
Astute.logger.debug "Reset undefined retries to original "\
"value: #{@original_undefined_retries}"
@undefined_retries = @original_undefined_retries
end
end #PuppetJob
end

258
lib/astute/puppet_task.rb
View File

@ -1,258 +0,0 @@
# Copyright 2014 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
require 'timeout'
module Astute
# @deprecated Please use {#Astute::PuppetJob} instead. This code is
# useful only for Granular or older deployment engines.
class PuppetTask
def initialize(ctx, node, options={})
default_options = {
:retries => Astute.config.puppet_retries,
:puppet_manifest => '/etc/puppet/manifests/site.pp',
:puppet_modules => Astute.config.puppet_module_path,
:cwd => Astute.config.shell_cwd,
:timeout => Astute.config.puppet_timeout,
:puppet_debug => false,
:succeed_retries => Astute.config.puppet_succeed_retries,
:raw_report => Astute.config.puppet_raw_report,
:puppet_noop_run => Astute.config.puppet_noop_run,
}
@options = options.compact.reverse_merge(default_options)
@options.freeze
@ctx = ctx
@node = node
@retries = @options[:retries]
@time_observer = TimeObserver.new(@options[:timeout])
@is_hung = false
@succeed_retries = @options[:succeed_retries]
@summary = {}
end
def run
Astute.logger.debug "Waiting for puppet to finish deployment on " \
"node #{@node['uid']} (timeout = #{@time_observer.time_limit} sec)..."
@time_observer.start
puppetd_runonce
end
# expect to run this method with respect of Astute.config.puppet_fade_interval
def status
raise Timeout::Error if @time_observer.time_is_up?
@summary = puppet_status
status = node_status(@summary)
message = "Node #{@node['uid']}(#{@node['role']}) status: #{status}"
if status == 'error'
Astute.logger.error message
else
Astute.logger.debug message
end
result = case status
when 'succeed'
processing_succeed_node(@summary)
when 'running'
processing_running_node
when 'error'
processing_error_node(@summary)
end
#TODO(vsharshov): Should we move it to control module?
@ctx.report_and_update_status('nodes' => [result]) if result
# ready, error or deploying
result.fetch('status', 'deploying')
rescue MClientTimeout, Timeout::Error
Astute.logger.warn "Puppet agent #{@node['uid']} " \
"didn't respond within the allotted time"
'error'
end
def summary
@summary
end
private
def puppetd
puppetd = MClient.new(
@ctx,
"puppetd",
[@node['uid']],
_check_result=true,
_timeout=nil,
_retries=Astute.config.mc_retries,
_enable_result_logging=false
)
puppetd.on_respond_timeout do |uids|
nodes = uids.map do |uid|
{
'uid' => uid,
'status' => 'error',
'error_type' => 'deploy',
'role' => @node['role']
}
end
@ctx.report_and_update_status('nodes' => nodes)
raise MClientTimeout
end
puppetd
end
def puppet_status
puppetd.last_run_summary(
:puppet_noop_run => @options[:puppet_noop_run],
:raw_report => @options[:raw_report]
).first[:data]
end
def puppet_run
puppetd.runonce(
:puppet_debug => @options[:puppet_debug],
:manifest => @options[:puppet_manifest],
:modules => @options[:puppet_modules],
:cwd => @options[:cwd],
:puppet_noop_run => @options[:puppet_noop_run],
)
end
def running?(status)
['running'].include? status[:status]
end
def idling?(status)
['idling'].include? status[:status]
end
def stopped?(status)
['stopped', 'disabled'].include? status[:status]
end
def succeed?(status)
status[:status] == 'stopped' &&
status[:resources]['failed'].to_i == 0 &&
status[:resources]['failed_to_restart'].to_i == 0
end
# Runs puppetd.runonce only if puppet is stopped on the host at the time
# If it isn't stopped, we wait a bit and try again.
# Returns list of nodes uids which appear to be with hung puppet.
def puppetd_runonce
started = Time.now.to_i
while Time.now.to_i - started < Astute.config.puppet_fade_timeout
status = puppet_status
is_stopped = stopped?(status)
is_idling = idling?(status)
is_running = running?(status)
#Try to kill 'idling' process and run again by 'runonce' call
puppet_run if is_stopped || is_idling
break if !is_running && !is_idling
sleep Astute.config.puppet_fade_interval
end
if is_running || is_idling
Astute.logger.warn "Following nodes have puppet hung " \
"(#{is_running ? 'running' : 'idling'}): '#{@node['uid']}'"
@is_hung = true
else
@is_hung = false
end
end
def node_status(last_run)
case
when @is_hung
'error'
when succeed?(last_run) && !@is_hung
'succeed'
when (running?(last_run) || idling?(last_run)) && !@is_hung
'running'
when stopped?(last_run) && !succeed?(last_run) && !@is_hung
'error'
else
msg = "Unknow status: " \
"is_hung #{@is_hung}, succeed? #{succeed?(last_run)}, " \
"running? #{running?(last_run)}, stopped? #{stopped?(last_run)}, " \
"idling? #{idling?(last_run)}"
raise msg
end
end
def processing_succeed_node(last_run)
Astute.logger.debug "Puppet completed within "\
"#{@time_observer.since_start} seconds"
if @succeed_retries > 0
@succeed_retries -= 1
Astute.logger.debug "Succeed puppet on node #{@node['uid']} will be "\
"restarted. #{@succeed_retries} retries remained."
Astute.logger.info "Retrying to run puppet for following succeed " \
"node: #{@node['uid']}"
puppetd_runonce
node_report_format('status' => 'deploying')
else
Astute.logger.debug "Node #{@node['uid']} has succeed to deploy. " \
"There is no more retries for puppet run."
{ 'uid' => @node['uid'], 'status' => 'ready', 'role' => @node['role'] }
end
end
def processing_error_node(last_run)
if @retries > 0
@retries -= 1
Astute.logger.debug "Puppet on node #{@node['uid']} will be "\
"restarted. #{@retries} retries remained."
Astute.logger.info "Retrying to run puppet for following error " \
"nodes: #{@node['uid']}"
puppetd_runonce
node_report_format('status' => 'deploying')
else
Astute.logger.debug "Node #{@node['uid']} has failed to deploy. " \
"There is no more retries for puppet run."
node_report_format('status' => 'error', 'error_type' => 'deploy')
end
end
def processing_running_node
nodes_to_report = []
begin
# Pass nodes because logs calculation needs IP address of node, not just uid
nodes_progress = @ctx.deploy_log_parser.progress_calculate([@node['uid']], [@node])
if nodes_progress.present?
Astute.logger.debug "Got progress for nodes:\n#{nodes_progress.pretty_inspect}"
# Nodes with progress are running, so they are not included in nodes_to_report yet
nodes_progress.map! { |x| x.merge!('status' => 'deploying', 'role' => @node['role']) }
nodes_to_report = nodes_progress
end
rescue => e
Astute.logger.warn "Some error occurred when parse logs for " \
"nodes progress: #{e.message}, trace: #{e.format_backtrace}"
end
nodes_to_report.first || node_report_format('status' => 'deploying')
end
def node_report_format(add_info={})
add_info.merge('uid' => @node['uid'], 'role' => @node['role'])
end
end #PuppetTask
end

69
lib/astute/puppetd.rb
View File

@ -1,69 +0,0 @@
# Copyright 2014 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
require 'json'
require 'timeout'
module Astute
module PuppetdDeployer
def self.deploy(ctx, nodes, retries=2, puppet_manifest=nil, puppet_modules=nil, cwd=nil, puppet_debug=true)
@ctx = ctx
@retries = retries
@nodes = nodes
@puppet_manifest = puppet_manifest || '/etc/puppet/manifests/site.pp'
@puppet_modules = puppet_modules || '/etc/puppet/modules'
@cwd = cwd || '/'
@puppet_debug = puppet_debug
Astute.logger.debug "Waiting for puppet to finish deployment on all
nodes (timeout = #{Astute.config.puppet_timeout} sec)..."
time_before = Time.now
deploy_nodes
time_spent = Time.now - time_before
Astute.logger.info "#{@ctx.task_id}: Spent #{time_spent} seconds on puppet run "\
"for following nodes(uids): #{@nodes.map {|n| n['uid']}.join(',')}"
end
private
def self.deploy_nodes
puppet_tasks = @nodes.map { |n| puppet_task(n) }
puppet_tasks.each(&:run)
loop do
sleep Astute.config.puppet_deploy_interval
break if !puppet_tasks.any? { |t| t.status == 'deploying' }
end
end
def self.puppet_task(n)
PuppetTask.new(
@ctx,
n,
{
:retries => @retries,
:puppet_manifest => @puppet_manifest,
:puppet_modules => @puppet_modules,
:cwd => @cwd,
:timeout => Astute.config.puppet_timeout,
:puppet_debug => @puppet_debug
})
end
end
end

293
lib/astute/reporter.rb
View File

@ -1,293 +0,0 @@
# -*- coding: utf-8 -*-
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
require 'set'
STATES = {
'offline' => 0,
'discover' => 10,
'provisioning' => 30,
'provisioned' => 40,
'deploying' => 50,
'ready' => 60,
'error' => 70
}
module Astute
module ProxyReporter
class DeploymentProxyReporter
attr_accessor :deploy
alias_method :deploy?, :deploy
def initialize(up_reporter, deployment_info=[])
@up_reporter = up_reporter
@nodes = deployment_info.inject([]) do |nodes, di|
nodes << {'uid' => di['uid'], 'role' => di['role'], 'fail_if_error' => di['fail_if_error']}
end
@deploy = deployment_info.present?
end
def report(data)
Astute.logger.debug("Data received by DeploymentProxyReporter to report it up:\n#{data.pretty_inspect}")
report_new_data(data)
end
private
def report_new_data(data)
if data['nodes']
nodes_to_report = get_nodes_to_report(data['nodes'])
return if nodes_to_report.empty? # Let's report only if nodes updated
# Update nodes attributes in @nodes.
update_saved_nodes(nodes_to_report)
data['nodes'] = nodes_to_report
end
data.merge!(get_overall_status(data))
Astute.logger.debug("Data send by DeploymentProxyReporter to report it up:\n#{data.pretty_inspect}")
@up_reporter.report(data)
end
def get_overall_status(data)
status = data['status']
error_nodes = @nodes.select { |n| n['status'] == 'error' }.map{ |n| n['uid'] }
msg = data['error']
if status == 'ready' && error_nodes.any?
status = 'error'
msg = "Some error occured on nodes\n#{error_nodes.pretty_inspect}"
end
progress = data['progress']
{'status' => status, 'error' => msg, 'progress' => progress}.reject{|k,v| v.nil?}
end
def get_nodes_to_report(nodes)
nodes.map{ |node| node_validate(node) }.compact
end
def update_saved_nodes(new_nodes)
# Update nodes attributes in @nodes.
new_nodes.each do |node|
saved_node = @nodes.find { |n| n['uid'] == node['uid'] && n['role'] == node['role'] }
if saved_node
node.each {|k, v| saved_node[k] = v}
else
@nodes << node
end
end
end
def node_validate(node)
validates_basic_fields(node)
# Ignore hooks report for multiroles progress
calculate_multiroles_node_progress(node) if deploy? && node['role'] != 'hook'
normalization_progress(node)
compare_with_previous_state(node)
end
# Validate of basic fields in message about nodes
def validates_basic_fields(node)
err = []
case
when node['status']
err << "Status provided #{node['status']} is not supported" unless STATES[node['status']]
when node['progress']
err << "progress value provided, but no status"
end
err << "Node role is not provided" if deploy? && !node['role']
err << "Node uid is not provided" unless node['uid']
if err.any?
msg = "Validation of node:\n#{node.pretty_inspect} for report failed: #{err.join('; ')}."
Astute.logger.error(msg)
raise msg
end
end
# Proportionally reduce the progress on the number of roles. Based on the
# fact that each part makes the same contribution to the progress we divide
# 100 to number of roles for this node. Also we prevent send final status for
# node before all roles will be deployed. Final result for node:
# * any error — error;
# * without error — succes.
# Example:
# Node have 3 roles and already success deploy first role and now deploying
# second(50%). Overall progress of the operation for node is
# 50 / 3 + 1 * 100 / 3 = 49
# We calculate it as 100/3 = 33% for every finished(success or fail) role
# Exception: node which have fail_if_error status equal true for some
# assigned node role. If this node role fail, we send error state for
# the entire node immediately.
def calculate_multiroles_node_progress(node)
@finish_roles_for_nodes ||= []
roles_of_node = @nodes.select { |n| n['uid'] == node['uid'] }
all_roles_amount = roles_of_node.size
return if all_roles_amount == 1 # calculation should only be done for multi roles
finish_roles_amount = @finish_roles_for_nodes.select do |n|
n['uid'] == node['uid'] && ['ready', 'error'].include?(n['status'])
end.size
return if finish_roles_amount == all_roles_amount # already done all work
# recalculate progress for node
node['progress'] = node['progress'].to_i/all_roles_amount + 100 * finish_roles_amount/all_roles_amount
# save final state if present
if ['ready', 'error'].include? node['status']
@finish_roles_for_nodes << { 'uid' => node['uid'], 'role' => node['role'], 'status' => node['status'] }
node['progress'] = 100 * (finish_roles_amount + 1)/all_roles_amount
end
# No more status update will be for node which failed and have fail_if_error as true
fail_if_error = @nodes.find { |n| n['uid'] == node['uid'] && n['role'] == node['role'] }['fail_if_error']
fail_now = fail_if_error && node['status'] == 'error'
if all_roles_amount - finish_roles_amount != 1 && !fail_now
# block 'ready' or 'error' final status for node if not all roles yet deployed
node['status'] = 'deploying'
node.delete('error_type') # Additional field for error response
elsif ['ready', 'error'].include? node['status']
node['status'] = @finish_roles_for_nodes.select { |n| n['uid'] == node['uid'] }
.select { |n| n['status'] == 'error' }
.empty? ? 'ready' : 'error'
node['progress'] = 100
node['error_type'] = 'deploy' if node['status'] == 'error'
end
end
# Normalization of progress field: ensures that the scaling progress was
# in range from 0 to 100 and has a value of 100 fot the final node status
def normalization_progress(node)
if node['progress']
if node['progress'] > 100
Astute.logger.warn("Passed report for node with progress > 100: "\
"#{node.pretty_inspect}. Adjusting progress to 100.")
node['progress'] = 100
elsif node['progress'] < 0
Astute.logger.warn("Passed report for node with progress < 0: "\
"#{node.pretty_inspect}. Adjusting progress to 0.")
node['progress'] = 0
end
end
if node['status'] && ['provisioned', 'ready'].include?(node['status']) && node['progress'] != 100
Astute.logger.warn("In #{node['status']} state node should have progress 100, "\
"but node passed:\n#{node.pretty_inspect}. Setting it to 100")
node['progress'] = 100
end
end
# Comparison information about node with previous state.
def compare_with_previous_state(node)
saved_node = @nodes.find { |x| x['uid'] == node['uid'] && x['role'] == node['role'] }
if saved_node
saved_status = STATES[saved_node['status']].to_i
node_status = STATES[node['status']] || saved_status
saved_progress = saved_node['progress'].to_i
node_progress = node['progress'] || saved_progress
if node_status < saved_status
Astute.logger.warn("Attempt to assign lower status detected: "\
"Status was: #{saved_node['status']}, attempted to "\
"assign: #{node['status']}. Skipping this node (id=#{node['uid']})")
return
end
if node_progress < saved_progress && node_status == saved_status
Astute.logger.warn("Attempt to assign lesser progress detected: "\
"Progress was: #{saved_node['status']}, attempted to "\
"assign: #{node['progress']}. Skipping this node (id=#{node['uid']})")
return
end
# We need to update node here only if progress is greater, or status changed
return if node.select{|k, v| saved_node[k] != v }.empty?
end
node
end
end # DeploymentProxyReporter
class ProvisiningProxyReporter < DeploymentProxyReporter
def initialize(up_reporter, provisioning_info=[])
@up_reporter = up_reporter
@nodes = provisioning_info['nodes'].inject([]) do |nodes, di|
nodes << {'uid' => di['uid']}
end
end
def report(data)
Astute.logger.debug("Data received by ProvisiningProxyReporter to report it up:\n#{data.pretty_inspect}")
report_new_data(data)
end
private
def report_new_data(data)
if data['nodes']
nodes_to_report = get_nodes_to_report(data['nodes'])
return if nodes_to_report.empty? # Let's report only if nodes updated
# Update nodes attributes in @nodes.
update_saved_nodes(nodes_to_report)
data['nodes'] = nodes_to_report
end
Astute.logger.debug("Data send by DeploymentProxyReporter to report it up:\n#{data.pretty_inspect}")
@up_reporter.report(data)
end
def node_validate(node)
validates_basic_fields(node)
normalization_progress(node)
compare_with_previous_state(node)
end
# Comparison information about node with previous state.
def compare_with_previous_state(node)
saved_node = @nodes.find { |x| x['uid'] == node['uid'] }
if saved_node
saved_status = STATES[saved_node['status']].to_i
node_status = STATES[node['status']] || saved_status
saved_progress = saved_node['progress'].to_i
node_progress = node['progress'] || saved_progress
if node_status < saved_status
Astute.logger.warn("Attempt to assign lower status detected: "\
"Status was: #{saved_node['status']}, attempted to "\
"assign: #{node['status']}. Skipping this node (id=#{node['uid']})")
return
end
if node_progress < saved_progress && node_status == saved_status
Astute.logger.warn("Attempt to assign lesser progress detected: "\
"Progress was: #{saved_node['status']}, attempted to "\
"assign: #{node['progress']}. Skipping this node (id=#{node['uid']})")
return
end
# We need to update node here only if progress is greater, or status changed
return if node.select{|k, v| saved_node[k] != v }.empty?
end
node
end
end # ProvisiningProxyReporter
end # ProxyReporter
end # Astute

40
lib/astute/rsyslogd.rb
View File

@ -1,40 +0,0 @@
# Copyright 2014 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class Rsyslogd
def self.send_sighup(ctx, master_ip)
timeout = Astute.config.ssh_retry_timeout
shell = MClient.new(ctx, 'execute_shell_command', ['master'],
check_result=true, timeout=timeout, retries=1)
cmd = "ssh root@#{master_ip} 'pkill -HUP rsyslogd'"
begin
result = shell.execute(:cmd => cmd).first.results
Astute.logger.info("#{ctx.task_id}: \
stdout: #{result[:data][:stdout]} stderr: #{result[:data][:stderr]} \
exit code: #{result[:data][:exit_code]}")
rescue Timeout::Error
msg = "Sending SIGHUP to rsyslogd is timed out."
Astute.logger.error("#{ctx.task_id}: #{msg}")
rescue => e
msg = "Exception occured during sending SIGHUP to rsyslogd, message: #{e.message} \
trace:\n#{e.backtrace.pretty_inspect}"
Astute.logger.error("#{ctx.task_id}: #{msg}")
end
end
end
end

23
lib/astute/ruby_removed_functions.rb
View File

@ -1,23 +0,0 @@
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
class Date
def self.day_fraction_to_time(fr)
ss, fr = fr.divmod(Rational(1, 86400))
h, ss = ss.divmod(3600)
min, s = ss.divmod(60)
return h, min, s, fr
end
end

79
lib/astute/server/async_logger.rb
View File

@ -1,79 +0,0 @@
# Copyright 2015 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
require 'thread'
module Astute
module Server
# Asynchronous singleton logger, which should be used
# in event callbacks of event machine, it doesn't block
# callbacks because writing a message to log takes some time.
# Also synchronous logger, potentially could lead to deadlocks.
# See:
# https://bugs.launchpad.net/fuel/+bug/1453573
# https://bugs.launchpad.net/fuel/+bug/1487397
module AsyncLogger
def self.start_up(logger=Logger.new(STDOUT))
@queue ||= Queue.new
@log = logger
@thread = Thread.new { flush_messages }
end
def self.shutdown
@thread.kill
end
def self.add(severity, msg=nil)
return if @shutdown
@queue.push([severity, msg])
end
def self.debug(msg=nil)
add(Logger::Severity::DEBUG, msg)
end
def self.info(msg=nil)
add(Logger::Severity::INFO, msg)
end
def self.warn(msg=nil)
add(Logger::Severity::WARN, msg)
end
def self.error(msg=nil)
add(Logger::Severity::ERROR, msg)
end
def self.fatal(msg=nil)
add(Logger::Severity::FATAL, msg)
end
def self.unknown(msg=nil)
add(Logger::Severity::UNKNOWN, msg)
end
private
def self.flush_messages
loop do
severity, msg = @queue.pop
@log.add(severity, msg)
end
end
end
end
end

332
lib/astute/server/dispatcher.rb
View File

@ -1,332 +0,0 @@
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
require 'astute/server/reporter'
module Astute
module Server
class Dispatcher
def initialize(producer)
@orchestrator = Astute::Orchestrator.new(log_parsing=true)
@producer = producer
@provisionLogParser = Astute::LogParser::ParseProvisionLogs.new
end
def echo(args)
Astute.logger.info('Running echo command')
args
end
#
# Main worker actions
#
def image_provision(data)
provision(data)
end
def provision(data)
Astute.logger.debug("'provision' method called with data:\n"\
"#{data.pretty_inspect}")
reporter = create_reporter(data)
begin
result = @orchestrator.provision(
reporter,
data['args']['task_uuid'],
data['args']['provisioning_info']
)
rescue => e
Astute.logger.error("Error running provisioning: #{e.message}, "\
"trace: #{e.format_backtrace}")
raise StopIteration
end
raise StopIteration if result && result['status'] == 'error'
end
def granular_deploy(data)
Astute.logger.debug("'granular_deploy' method called with data:\n"\
"#{data.pretty_inspect}")
reporter = create_reporter(data)
begin
@orchestrator.granular_deploy(
reporter,
data['args']['task_uuid'],
data['args']['deployment_info'],
data['args']['pre_deployment'] || [],
data['args']['post_deployment'] || []
)
reporter.report('status' => 'ready', 'progress' => 100)
rescue Timeout::Error
msg = "Timeout of deployment is exceeded."
Astute.logger.error(msg)
reporter.report('status' => 'error', 'error' => msg)
end
end
def task_deploy(data)
Astute.logger.debug("'task_deploy' method called with data:\n"\
"#{data.pretty_inspect}")
Thread.current[:gracefully_stop] = false
reporter = create_reporter(data)
begin
@orchestrator.task_deploy(
reporter,
data['args']['task_uuid'],
{
:tasks_graph => data['args'].fetch('tasks_graph', {}),
:tasks_directory => data['args'].fetch('tasks_directory', {}),
:tasks_metadata => data['args'].fetch('tasks_metadata', {}),
:dry_run => data['args'].fetch('dry_run', false),
:noop_run => data['args'].fetch('noop_run', false),
:debug => data['args'].fetch('debug', false)
}
)
rescue Timeout::Error
msg = "Timeout of deployment is exceeded."
Astute.logger.error(msg)
reporter.report('status' => 'error', 'error' => msg)
end
end
def verify_networks(data)
data.fetch('subtasks', []).each do |subtask|
if self.respond_to?(subtask['method'])
self.send(subtask['method'], subtask)
else
Astute.logger.warn("No method for #{subtask}")
end
end
reporter = create_reporter(data)
result = @orchestrator.verify_networks(
reporter,
data['args']['task_uuid'],
data['args']['nodes']
)
report_result(result, reporter)
end
def check_dhcp(data)
reporter = create_reporter(data)
result = @orchestrator.check_dhcp(
reporter,
data['args']['task_uuid'],
data['args']['nodes']
)
report_result(result, reporter)
end
def multicast_verification(data)
reporter = create_reporter(data)
result = @orchestrator.multicast_verification(
reporter,
data['args']['task_uuid'],
data['args']['nodes']
)
report_result(result, reporter)
end
def check_repositories(data)
reporter = create_reporter(data)
result = @orchestrator.check_repositories(
reporter,
data['args']['task_uuid'],
data['args']['nodes'],
data['args']['urls']
)
report_result(result, reporter)
end
def check_repositories_with_setup(data)
reporter = create_reporter(data)
result = @orchestrator.check_repositories_with_setup(
reporter,
data['args']['task_uuid'],
data['args']['nodes']
)
report_result(result, reporter)
end
def dump_environment(data)
@orchestrator.dump_environment(
create_reporter(data),
data['args']['task_uuid'],
data['args']['settings']
)
end
def remove_nodes(data, reset=false)
task_uuid = data['args']['task_uuid']
reporter = create_reporter(data)
result = if data['args']['nodes'].empty?
Astute.logger.debug("#{task_uuid} Node list is empty")
nil
else
@orchestrator.remove_nodes(
reporter,
task_uuid,
data['args']['engine'],
data['args']['nodes'],
{
:reboot => true,
:check_ceph => data['args']['check_ceph'],
:reset => reset
}
)
end
report_result(result, reporter)
end
def reset_environment(data)
remove_nodes(data, reset=true)
end
def execute_tasks(data)
@orchestrator.execute_tasks(
create_reporter(data),
data['args']['task_uuid'],
data['args']['tasks']
)
end
#
# Service worker actions
#
def stop_deploy_task(data, service_data)
Astute.logger.debug("'stop_deploy_task' service method called with"\
"data:\n#{data.pretty_inspect}")
target_task_uuid = data['args']['stop_task_uuid']
task_uuid = data['args']['task_uuid']
return unless task_in_queue?(target_task_uuid,
service_data[:tasks_queue])
Astute.logger.debug("Cancel task #{target_task_uuid}. Start")
if target_task_uuid == service_data[:tasks_queue].current_task_id
reporter = create_reporter(data)
result = stop_current_task(data, service_data, reporter)
report_result(result, reporter)
else
replace_future_task(data, service_data)
end
end
private
def create_reporter(data)
Astute::Server::Reporter.new(
@producer,
data['respond_to'],
data['args']['task_uuid']
)
end
def task_in_queue?(task_uuid, tasks_queue)
tasks_queue.task_in_queue?(task_uuid)
end
def replace_future_task(data, service_data)
target_task_uuid = data['args']['stop_task_uuid']
task_uuid = data['args']['task_uuid']
new_task_data = data_for_rm_nodes(data)
Astute.logger.debug("Replace running task #{target_task_uuid} to "\
"new #{task_uuid} with data:\n"\
"#{new_task_data.pretty_inspect}")
service_data[:tasks_queue].replace_task(
target_task_uuid,
new_task_data
)
end
def stop_current_task(data, service_data, reporter)
target_task_uuid = data['args']['stop_task_uuid']
task_uuid = data['args']['task_uuid']
nodes = data['args']['nodes']
result = if ['deploy', 'granular_deploy'].include? (
service_data[:tasks_queue].current_task_method)
kill_main_process(target_task_uuid, service_data)
@orchestrator.stop_puppet_deploy(reporter, task_uuid, nodes)
@orchestrator.remove_nodes(
reporter,
task_uuid,
data['args']['engine'],
nodes
)
elsif ['task_deploy'].include? (
service_data[:tasks_queue].current_task_method)
gracefully_stop_main_process(target_task_uuid, service_data)
wait_while_process_run(
service_data[:main_work_thread],
Astute.config.stop_timeout,
target_task_uuid,
service_data
)
else
kill_main_process(target_task_uuid, service_data)
@orchestrator.stop_provision(
reporter,
task_uuid,
data['args']['engine'],
nodes
)
end
end
def kill_main_process(target_task_uuid, service_data)
Astute.logger.info("Try to kill running task #{target_task_uuid}")
service_data[:main_work_thread].kill
end
def gracefully_stop_main_process(target_task_uuid, service_data)
Astute.logger.info("Try to stop gracefully running " \
"task #{target_task_uuid}")
service_data[:main_work_thread][:gracefully_stop] = true
end
def wait_while_process_run(process, timeout, target_task_uuid, service_data)
Astute.logger.info("Wait until process will stop or exit " \
"by timeout #{timeout}")
Timeout::timeout(timeout) { process.join }
{}
rescue Timeout::Error => e
msg = "Timeout (#{timeout} sec) was reached."
Astute.logger.warn(msg)
kill_main_process(target_task_uuid, service_data)
{'status' => 'error', 'error' => msg}
end
def data_for_rm_nodes(data)
data['method'] = 'remove_nodes'
data
end
def report_result(result, reporter)
result = {} unless result.instance_of?(Hash)
status = {'status' => 'ready', 'progress' => 100}.merge(result)
reporter.report(status)
end
end
end #Server
end #Astute

61
lib/astute/server/producer.rb
View File

@ -1,61 +0,0 @@
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
module Server
class Producer
def initialize(exchange)
@exchange = exchange
@publish_queue = Queue.new
@publish_consumer = Thread.new do
loop do
msg = @publish_queue.pop
publish_from_queue msg
end
end
end
def publish_from_queue(message)
Astute.logger.info "Casting message to Nailgun:\n"\
"#{message[:message].pretty_inspect}"
@exchange.publish(message[:message].to_json, message[:options])
rescue => e
Astute.logger.error "Error publishing message: #{e.message}"
end
def publish(message, options={})
default_options = {
:routing_key => Astute.config.broker_publisher_queue,
:content_type => 'application/json'
}
# Status message manage task status in Nailgun. If we miss some of them,
# user need manually delete them or change it status using DB.
# Persistent option tell RabbitMQ to save message in case of
# unexpected/expected restart.
if message.respond_to?(:keys) && message.keys.map(&:to_s).include?('status')
default_options.merge!({:persistent => true})
end
options = default_options.merge(options)
@publish_queue << {:message => message, :options => options}
end
def stop
@publish_consumer.kill
end
end # Producer
end #Server
end #Astute

33
lib/astute/server/reporter.rb
View File

@ -1,33 +0,0 @@
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
module Server
class Reporter
def initialize(producer, method, task_uuid)
@producer = producer
@method = method
@task_uuid = task_uuid
end
def report(msg)
msg_with_task = {'task_uuid' => @task_uuid}.merge(msg)
message = {'method' => @method, 'args' => msg_with_task}
@producer.publish(message)
end
end
end #Server
end #Astute

253
lib/astute/server/server.rb
View File

@ -1,253 +0,0 @@
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
require 'json'
require 'securerandom'
require 'astute/server/task_queue'
require 'zlib'
module Astute
module Server
class Server
def initialize(channels_and_exchanges, delegate, producer)
@channel = channels_and_exchanges[:channel]
@exchange = channels_and_exchanges[:exchange]
@delegate = delegate
@producer = producer
@service_channel = channels_and_exchanges[:service_channel]
@service_exchange = channels_and_exchanges[:service_exchange]
# NOTE(eli): Generate unique name for service queue
# See bug: https://bugs.launchpad.net/fuel/+bug/1485895
@service_queue_name = "naily_service_#{SecureRandom.uuid}"
@watch_thread = nil
end
def run
@queue = @channel.queue(
Astute.config.broker_queue,
:durable => true
)
@queue.bind(@exchange)
@service_queue = @service_channel.queue(
@service_queue_name,
:exclusive => true,
:auto_delete => true
)
@service_queue.bind(@service_exchange)
@main_work_thread = nil
@tasks_queue = TaskQueue.new
register_callbacks
run_infinite_loop
@watch_thread.join
end
def stop
@watch_thread.wakeup
end
private
def run_infinite_loop
@watch_thread = Thread.new do
Thread.stop
Astute.logger.debug "Stop main thread"
end
end
def register_callbacks
main_worker
service_worker
end
def main_worker
@queue.subscribe(:manual_ack => true) do |delivery_info, properties, payload|
if @main_work_thread.nil? || !@main_work_thread.alive?
@channel.acknowledge(delivery_info.delivery_tag, false)
perform_main_job(payload, properties)
else
Astute.logger.debug "Requeue message because worker is busy"
# Avoid throttle by consume/reject cycle
# if only one worker is running
@channel.reject(delivery_info.delivery_tag, true)
end
end
end
def service_worker
@service_queue.subscribe do |_delivery_info, properties, payload|
perform_service_job(payload, properties)
end
end
def perform_main_job(payload, properties)
@main_work_thread = Thread.new do
data = parse_data(payload, properties)
Astute.logger.debug("Process message from worker queue:\n"\
"#{data.pretty_inspect}")
@tasks_queue = Astute::Server::TaskQueue.new
@tasks_queue.add_task(data)
dispatch(@tasks_queue)
# Clean up tasks queue to prevent wrong service job work flow for
# already finished tasks
@tasks_queue = TaskQueue.new
end
end
def perform_service_job(payload, properties)
Thread.new do
service_data = {
:main_work_thread => @main_work_thread,
:tasks_queue => @tasks_queue
}
data = parse_data(payload, properties)
Astute.logger.debug("Process message from service queue:\n"\
"#{data.pretty_inspect}")
dispatch(data, service_data)
end
end
def dispatch(data, service_data=nil)
data.each_with_index do |message, i|
begin
send_running_task_status(message)
dispatch_message message, service_data
rescue StopIteration
Astute.logger.debug "Dispatching aborted by #{message['method']}"
abort_messages data[(i + 1)..-1]
break
rescue => ex
Astute.logger.error "Error running RPC method "\
"#{message['method']}: #{ex.message}, "\
"trace: #{ex.format_backtrace}"
return_results message, {
'status' => 'error',
'error' => "Method #{message['method']}. #{ex.message}.\n" \
"Inspect Astute logs for the details"
}
break
end
end
end
def dispatch_message(data, service_data=nil)
if Astute.config.fake_dispatch
Astute.logger.debug "Fake dispatch"
return
end
unless @delegate.respond_to?(data['method'])
Astute.logger.error "Unsupported RPC call '#{data['method']}'"
return_results data, {
'status' => 'error',
'error' => "Unsupported method '#{data['method']}' called."
}
return
end
if service_data.nil?
Astute.logger.debug "Main worker task id is "\
"#{@tasks_queue.current_task_id}"
end
Astute.logger.info "Processing RPC call '#{data['method']}'"
if !service_data
@delegate.send(data['method'], data)
else
@delegate.send(data['method'], data, service_data)
end
end
def send_running_task_status(message)
return_results(message, {'status' => 'running'})
end
def return_results(message, results={})
if results.is_a?(Hash) && message['respond_to']
reporter = Astute::Server::Reporter.new(
@producer,
message['respond_to'],
message['args']['task_uuid']
)
reporter.report results
end
end
def parse_data(data, properties)
data = unzip_message(data, properties) if zip?(properties)
messages = begin
JSON.load(data)
rescue => e
Astute.logger.error "Error deserializing payload: #{e.message},"\
" trace:\n#{e.backtrace.pretty_inspect}"
nil
end
messages.is_a?(Array) ? messages : [messages]
end
def unzip_message(data, properties)
Zlib::Inflate.inflate(data)
rescue => e
msg = "Gzip failure with error #{e.message} in\n"\
"#{e.backtrace.pretty_inspect} with properties\n"\
"#{properties.pretty_inspect} on data\n#{data.pretty_inspect}"
Astute.logger.error(msg)
raise e, msg
end
def zip?(properties)
properties[:headers]['compression'] == "application/x-gzip"
end
def abort_messages(messages)
return unless messages && messages.size > 0
messages.each do |message|
begin
Astute.logger.debug "Aborting '#{message['method']}'"
err_msg = {
'status' => 'error',
'error' => 'Task aborted',
'progress' => 100
}
if message['args']['nodes'].instance_of?(Array)
err_nodes = message['args']['nodes'].map do |node|
{
'uid' => node['uid'],
'status' => 'error',
'error_type' => 'provision',
'progress' => 0
}
end
err_msg.merge!('nodes' => err_nodes)
end
return_results(message, err_msg)
rescue => ex
Astute.logger.debug "Failed to abort '#{message['method']}':\n"\
"#{ex.pretty_inspect}"
end
end
end
end
end #Server
end #Astute

89
lib/astute/server/task_queue.rb
View File

@ -1,89 +0,0 @@
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
require 'thread'
module Astute
module Server
class TaskQueue
include Enumerable
attr_reader :current_task_id
attr_reader :current_task_method
def initialize
@queue = []
@semaphore = Mutex.new
@current_task_id = nil
@current_task_method = nil
end
def add_task(data)
@semaphore.synchronize { data.compact.each { |t| @queue << t } }
end
def replace_task(replacing_task_id, new_task_data)
@semaphore.synchronize do
@queue.map! { |x| find_task_id(x) == replacing_task_id ? new_task_data : x }.flatten!
end
end
def remove_task(replacing_task_id)
replace_task(replacing_task_id, nil)
end
def clear_queue
@semaphore.synchronize { @queue.map! { |x| nil } }
end
def task_in_queue?(task_id)
@semaphore.synchronize { @queue.find { |t| find_task_id(t) == task_id } }
end
def each(&block)
@queue.each do |task|
@semaphore.synchronize do
next if task.nil?
@current_task_id = find_task_id(task)
@current_task_method = find_task_method(task)
end
if block_given?
block.call task
else
yield task
end
end
ensure
@semaphore.synchronize do
@current_task_id = nil
@current_task_method = nil
end
end
private
def find_task_id(data)
data && data['args'] && data['args']['task_uuid'] ? data['args']['task_uuid'] : nil
end
def find_task_method(data)
data && data['method']
end
end
end #Server
end #Astute

182
lib/astute/server/worker.rb
View File

@ -1,182 +0,0 @@
# Copyright 2013 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
require 'raemon'
require 'net/http'
require 'bunny'
module Astute
module Server
class Worker
include Raemon::Worker
DELAY_SEC = 5
def start
super
start_heartbeat
Astute::Server::AsyncLogger.start_up(Astute.logger)
Astute.logger = Astute::Server::AsyncLogger
end
def stop
super
@connection.stop if defined?(@connection) && @connection.present?
@producer.stop if defined?(@producer) && @producer.present?
@server.stop if defined?(@server) && @server.present?
Astute::Server::AsyncLogger.shutdown
end
def run
Astute.logger.info "Worker initialization"
run_server
rescue Bunny::TCPConnectionFailed => e
Astute.logger.warn "TCP connection to AMQP failed: #{e.message}. "\
"Retry #{DELAY_SEC} sec later..."
sleep DELAY_SEC
retry
rescue Bunny::PossibleAuthenticationFailureError => e
Astute.logger.warn "If problem repeated more than 5 minutes, "\
"please check "\
"authentication parameters. #{e.message}. "\
"Retry #{DELAY_SEC} sec later..."
sleep DELAY_SEC
retry
rescue => e
Astute.logger.error "Exception during worker initialization:"\
" #{e.message}, trace: #{e.format_backtrace}"
Astute.logger.warn "Retry #{DELAY_SEC} sec later..."
sleep DELAY_SEC
retry
end
private
def start_heartbeat
@heartbeat ||= Thread.new do
sleep 30
heartbeat!
end
end
def run_server
@connection = Bunny.new(connection_options)
@connection.start
channels_and_exchanges = declare_channels_and_exchanges(@connection)
@producer = Astute::Server::Producer.new(
channels_and_exchanges[:report_exchange]
)
delegate = Astute::Server::Dispatcher.new(@producer)
@server = Astute::Server::Server.new(
channels_and_exchanges,
delegate,
@producer
)
@server.run
end
def declare_channels_and_exchanges(connection)
# WARN: Bunny::Channel are designed to assume they are
# not shared between threads.
channel = @connection.create_channel
exchange = channel.topic(
Astute.config.broker_exchange,
:durable => true
)
report_channel = @connection.create_channel
report_exchange = report_channel.topic(
Astute.config.broker_exchange,
:durable => true
)
service_channel = @connection.create_channel
service_channel.prefetch(0)
service_exchange = service_channel.fanout(
Astute.config.broker_service_exchange,
:auto_delete => true
)
return {
:exchange => exchange,
:service_exchange => service_exchange,
:channel => channel,
:service_channel => service_channel,
:report_channel => report_channel,
:report_exchange => report_exchange
}
rescue Bunny::PreconditionFailed => e
Astute.logger.warn "Try to remove problem exchanges and queues"
if connection.queue_exists? Astute.config.broker_queue
channel.queue_delete Astute.config.broker_queue
end
if connection.queue_exists? Astute.config.broker_publisher_queue
channel.queue_delete Astute.config.broker_publisher_queue
end
cleanup_rabbitmq_stuff
raise e
end
def connection_options
{
:host => Astute.config.broker_host,
:port => Astute.config.broker_port,
:user => Astute.config.broker_username,
:pass => Astute.config.broker_password,
:heartbeat => :server
}.reject{|k, v| v.nil? }
end
def cleanup_rabbitmq_stuff
Astute.logger.warn "Try to remove problem exchanges and queues"
[Astute.config.broker_exchange,
Astute.config.broker_service_exchange].each do |exchange|
rest_delete("/api/exchanges/%2F/#{exchange}")
end
end
def rest_delete(url)
http = Net::HTTP.new(
Astute.config.broker_host,
Astute.config.broker_rest_api_port
)
request = Net::HTTP::Delete.new(url)
request.basic_auth(
Astute.config.broker_username,
Astute.config.broker_password
)
response = http.request(request)
case response.code.to_i
when 204 then Astute.logger.debug "Successfully delete object at #{url}"
when 404 then
else
Astute.logger.error "Failed to perform delete request. Debug"\
" information: http code: #{response.code},"\
" message: #{response.message},"\
" body #{response.body}"
end
end
end # Worker
end #Server
end #Astute

195
lib/astute/task.rb
View File

@ -1,195 +0,0 @@
# Copyright 2015 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class Task
ALLOWED_STATUSES = [:successful, :failed, :running, :pending, :skipped]
attr_reader :task, :ctx
def initialize(task, context)
# WARNING: this code expect that only one node will be send
# on one hook.
@task = task
@status = :pending
@ctx = context
@time_start = Time.now.to_i
post_initialize(task, context)
end
# Run current task on node, specified in task
def run
validation
setup_default
running!
process
rescue => e
Astute.logger.warn("Fail to run task #{task['type']} #{task_name}" \
" with error #{e.message} trace: #{e.format_backtrace}")
failed!
end
# Polls the status of the task
def status
calculate_status unless finished?
@status
rescue => e
Astute.logger.warn("Fail to detect status of the task #{task['type']}" \
" #{task_name} with error #{e.message} trace: #{e.format_backtrace}")
failed!
end
def status=(value)
value = value.to_sym
unless ALLOWED_STATUSES.include?(value)
raise AstuteError::InvalidArgument,
"#{self}: Invalid task status: #{value}"
end
@status = value
end
# Run current task on node, specified in task, using sync mode
def sync_run
run
loop do
sleep Astute.config.task_poll_delay
status
break if finished?
end
successful?
end
# Show additional info about tasks: last run summary, sdtout etc
def summary
{}
end
def finished?
[:successful, :failed, :skipped].include? @status
end
def successful?
@status == :successful
end
def pending?
@status == :pending
end
def skipped?
@status == :skipped
end
def running?
@status == :running
end
def failed?
@status == :failed
end
def post_initialize(task, context)
nil
end
private
# Run current task on node, specified in task
# should be fast and async and do not raise exceptions
# @abstract Should be implemented in a subclass
def process
raise NotImplementedError
end
# Polls the status of the task
# should update the task status and do not raise exceptions
# @abstract Should be implemented in a subclass
def calculate_status
raise NotImplementedError
end
def validate_presence(data, key)
raise TaskValidationError,
"Missing a required parameter #{key}" unless data[key].present?
end
# Pre validation of the task
# should check task and raise error if something went wrong
# @raise [TaskValidationError] if the object is not a task or has missing fields
def validation
end
# Setup default value for hook
# should not raise any exception
def setup_default
end
# Run short shell commands
# should use only in case of short run command
# In other case please use shell task
# Synchronous (blocking) call
def run_shell_without_check(node_uid, cmd, timeout=2)
ShellMClient.new(@ctx, node_uid).run_without_check(cmd, timeout)
end
# Create file with content on selected node
# should use only for small file
# In other case please use separate thread or
# use upload file task.
# Synchronous (blocking) call
def upload_file(node_uid, mco_params)
UploadFileMClient.new(@ctx, node_uid).upload_without_check(mco_params)
end
# Create file with content on selected node
# should use only for small file
# Synchronous (blocking) call
def upload_file_with_check(node_uid, mco_params)
UploadFileMClient.new(@ctx, node_uid).upload_with_check(mco_params)
end
def failed!
self.status = :failed
time_summary
end
def running!
self.status = :running
end
def succeed!
self.status = :successful
time_summary
end
def skipped!
self.status = :skipped
time_summary
end
def task_name
task['id'] || task['diagnostic_name']
end
def time_summary
amount_time = (Time.now.to_i - @time_start).to_i
wasted_time = Time.at(amount_time).utc.strftime("%H:%M:%S")
Astute.logger.debug("Task time summary: #{task_name} with status" \
" #{@status.to_s} on node #{task['node_id']} took #{wasted_time}")
end
end
end

36
lib/astute/task_cluster.rb
View File

@ -1,36 +0,0 @@
# Copyright 2016 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
require 'fuel_deployment'
module Astute
class TaskCluster < Deployment::Cluster
attr_accessor :noop_run, :debug_run
def initialize(id=nil)
super
@node_statuses_transitions = {}
end
attr_accessor :node_statuses_transitions
def hook_post_gracefully_stop(*args)
report_new_node_status(args[0])
end
def report_new_node_status(node)
node.report_node_status
end
end
end

380
lib/astute/task_deployment.rb
View File

@ -1,380 +0,0 @@
# Copyright 2015 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
require_relative '../fuel_deployment'
module Astute
class TaskDeployment
#TODO(vsharshov): remove this default after adding support of node
# status transition to Nailgun
NODE_STATUSES_TRANSITIONS = {
'successful' => {'status' => 'ready'},
'stopped' => {'status' => 'stopped'},
'failed' => {'status' => 'error', 'error_type' => 'deploy'}
}
attr_reader :ctx, :cluster_class, :node_class
def initialize(context, cluster_class=TaskCluster, node_class=TaskNode)
@ctx = context
@cluster_class = cluster_class
@node_class = node_class
end
def self.munge_task(tasks_names, tasks_graph)
result = Set.new
tasks_names.each do |task|
if task.is_a? Deployment::Task
result.add task
next
end
Astute.logger.debug("munging task #{task}")
parts = task.split('/')
task_name = parts[0]
task_range = parts[1]
if task_range
Astute.logger.debug("expanding task #{task} range to specific nodes #{task_range}")
node_ids = expand_node_ids(task_range).flatten
Astute.logger.debug("expanded task #{task} range to #{node_ids.to_a}")
else
Astute.logger.debug("expanding task #{task} range to all_nodes")
node_ids = tasks_graph.each_node.collect {|node| node.uid}
end
exp_t = tasks_graph.each_task.select do |_task|
#Astute.logger.debug("task node id comparison is #{_task.node in? node_ids}")
rv = (_task.name == task_name and _task.node.uid.in? node_ids)
rv
end
exp_t.each do |t|
result.add t
end
end
result
end
def self.expand_node_ids(interval)
interval.split(',').collect do |part|
if part =~ /^(\d+)-(\d+)$/
($1.to_i .. $2.to_i).to_a
else
part
end
end
end
def self.munge_list_of_start_end(tasks_graph, subgraphs)
subgraphs.each do |subgraph|
subgraph['start'] ||= []
subgraph['end'] ||= []
Astute.logger.debug("munging start tasks #{subgraph['start'].to_a} ")
subgraph['start'] = munge_task(subgraph['start'], tasks_graph) unless subgraph['start'].blank?
Astute.logger.debug("munged start tasks to #{subgraph['start'].to_a}")
Astute.logger.debug("munging end tasks #{subgraph['end'].to_a} ")
subgraph['end'] = munge_task(subgraph['end'], tasks_graph) unless subgraph['end'].blank?
Astute.logger.debug("munged end tasks to #{subgraph['end'].to_a} ")
end
end
def create_cluster(deployment_options={})
tasks_graph = deployment_options.fetch(:tasks_graph, {})
tasks_directory = deployment_options.fetch(:tasks_directory, {})
tasks_metadata = deployment_options.fetch(:tasks_metadata, {})
raise DeploymentEngineError, 'Deployment graph was not provided!' if tasks_graph.blank?
support_virtual_node(tasks_graph)
unzip_graph(tasks_graph, tasks_directory)
cluster = cluster_class.new
cluster.node_concurrency.maximum = Astute.config.max_nodes_per_call
cluster.stop_condition { Thread.current[:gracefully_stop] }
cluster.noop_run = deployment_options.fetch(:noop_run, false)
cluster.debug_run = deployment_options.fetch(:debug, false)
cluster.node_statuses_transitions = tasks_metadata.fetch(
'node_statuses_transitions',
NODE_STATUSES_TRANSITIONS
)
setup_fault_tolerance_behavior(
tasks_metadata['fault_tolerance_groups'],
cluster,
tasks_graph.keys
)
critical_uids = critical_node_uids(cluster.fault_tolerance_groups)
offline_uids = detect_offline_nodes(tasks_graph.keys)
fail_offline_nodes(
:offline_uids => offline_uids,
:critical_uids => critical_uids,
:node_statuses_transitions => cluster.node_statuses_transitions
)
tasks_graph.keys.each do |node_id|
node = node_class.new(node_id, cluster)
node.context = ctx
node.set_critical if critical_uids.include?(node_id)
node.set_as_sync_point if sync_point?(node_id)
node.set_status_failed if offline_uids.include?(node_id)
end
setup_fail_behavior(tasks_graph, cluster)
setup_debug_behavior(tasks_graph, cluster)
setup_tasks(tasks_graph, cluster)
setup_task_depends(tasks_graph, cluster)
setup_task_concurrency(tasks_graph, cluster)
subgraphs = self.class.munge_list_of_start_end(cluster, tasks_metadata.fetch('subgraphs', []))
cluster.subgraphs = subgraphs unless subgraphs.compact_blank.blank?
Astute.logger.debug(cluster.subgraphs)
cluster.setup_start_end unless cluster.subgraphs.blank?
cluster
end
def deploy(deployment_options={})
cluster = create_cluster(deployment_options)
dry_run = deployment_options.fetch(:dry_run, false)
write_graph_to_file(cluster)
result = if dry_run
{:success => true}
else
run_result = cluster.run
# imitate dry_run results for noop run after deployment
cluster.noop_run ? {:success => true } : run_result
end
report_final_node_progress(cluster)
report_deploy_result(result)
end
private
def sync_point?(node_id)
'virtual_sync_node' == node_id
end
def unzip_graph(tasks_graph, tasks_directory)
tasks_graph.each do |node_id, tasks|
tasks.each do |task|
task.merge!({'node_id' => node_id})
.reverse_merge(tasks_directory.fetch(task['id'], {}))
end
end
tasks_graph
end
def setup_fault_tolerance_behavior(fault_tolerance_groups, cluster, nodes)
fault_tolerance_groups = [] if fault_tolerance_groups.nil?
defined_nodes = fault_tolerance_groups.map { |g| g['node_ids'] }.flatten.uniq
all_nodes = nodes.select{ |n| !sync_point?(n) }
undefined_nodes = all_nodes - defined_nodes
fault_tolerance_groups << {
'fault_tolerance' => 0,
'name' => 'zero_tolerance_as_default_for_nodes',
'node_ids' => undefined_nodes
}
cluster.fault_tolerance_groups = fault_tolerance_groups
end
def setup_fail_behavior(tasks_graph, cluster)
return unless cluster.noop_run
tasks_graph.each do |node_id, tasks|
tasks.each do |task|
task['fail_on_error'] = false
end
end
end
def setup_debug_behavior(tasks_graph, cluster)
return unless cluster.debug_run
tasks_graph.each do |node_id, tasks|
tasks.each do |task|
if task['parameters'].present?
task['parameters']['debug'] = true
else
task['parameters'] = { 'debug' => true }
end
end
end
end
def setup_tasks(tasks_graph, cluster)
tasks_graph.each do |node_id, tasks|
tasks.each do |task|
cluster[node_id].graph.create_task(task['id'], task)
end
end
end
def setup_task_depends(tasks_graph, cluster)
tasks_graph.each do |node_id, tasks|
tasks.each do |task|
task.fetch('requires', []).each do |d_t|
cluster[node_id][task['id']].depends(
cluster[d_t['node_id']][d_t['name']])
end
task.fetch('required_for', []).each do |d_t|
cluster[node_id][task['id']].depended_on(
cluster[d_t['node_id']][d_t['name']])
end
end
end
end
def setup_task_concurrency(tasks_graph, cluster)
tasks_graph.each do |_node_id, tasks|
tasks.each do |task|
cluster.task_concurrency[task['id']].maximum = task_concurrency_value(task)
end
end
end
def task_concurrency_value(task)
strategy = task.fetch('parameters', {}).fetch('strategy', {})
value = case strategy['type']
when 'one_by_one' then 1
when 'parallel' then strategy['amount'].to_i
else 0
end
return value if value >= 0
raise DeploymentEngineError, "Task concurrency expect only "\
"non-negative integer, but got #{value}. Please check task #{task}"
end
def report_deploy_result(result)
if result[:success] && result.fetch(:failed_nodes, []).empty?
ctx.report('status' => 'ready', 'progress' => 100)
elsif result[:success] && result.fetch(:failed_nodes, []).present?
ctx.report('status' => 'ready', 'progress' => 100)
else
ctx.report(
'status' => 'error',
'progress' => 100,
'error' => result[:status]
)
end
end
def write_graph_to_file(deployment)
return unless Astute.config.enable_graph_file
graph_file = File.join(
Astute.config.graph_dot_dir,
"graph-#{ctx.task_id}.dot"
)
File.open(graph_file, 'w') { |f| f.write(deployment.to_dot) }
Astute.logger.info("Check graph into file #{graph_file}")
end
# Astute use special virtual node for deployment tasks, because
# any task must be connected to node. For task, which play
# synchronization role, we create virtual_sync_node
def support_virtual_node(tasks_graph)
tasks_graph['virtual_sync_node'] = tasks_graph['null']
tasks_graph.delete('null')
tasks_graph.each do |_node_id, tasks|
tasks.each do |task|
task.fetch('requires',[]).each do |d_t|
d_t['node_id'] = 'virtual_sync_node' if d_t['node_id'].nil?
end
task.fetch('required_for', []).each do |d_t|
d_t['node_id'] = 'virtual_sync_node' if d_t['node_id'].nil?
end
end
end
tasks_graph
end
def critical_node_uids(fault_tolerance_groups)
return [] if fault_tolerance_groups.blank?
critical_nodes = fault_tolerance_groups.inject([]) do |critical_uids, group|
critical_uids += group['node_ids'] if group['fault_tolerance'].zero?
critical_uids
end
Astute.logger.info "Critical node #{critical_nodes}" if critical_nodes.present?
critical_nodes
end
def fail_offline_nodes(args={})
critical_uids = args.fetch(:critical_uids, [])
offline_uids = args.fetch(:offline_uids, [])
node_statuses_transitions = args.fetch(:node_statuses_transitions, {})
return if offline_uids.blank?
nodes = offline_uids.map do |uid|
{'uid' => uid,
'error_msg' => 'Node is not ready for deployment: '\
'mcollective has not answered'
}.merge(node_statuses_transitions.fetch('failed', {}))
end
ctx.report_and_update_status(
'nodes' => nodes,
'error' => 'Node is not ready for deployment'
)
missing_required = critical_uids & offline_uids
if missing_required.present?
error_message = "Critical nodes are not available for deployment: " \
"#{missing_required}"
raise Astute::DeploymentEngineError, error_message
end
end
def detect_offline_nodes(uids)
available_uids = []
uids.delete('master')
uids.delete('virtual_sync_node')
# In case of big amount of nodes we should do several calls to be sure
# about node status
if uids.present?
Astute.config.mc_retries.times.each do
systemtype = MClient.new(
ctx,
"systemtype",
uids,
_check_result=false,
10
)
available_nodes = systemtype.get_type
available_uids += available_nodes.map { |node| node.results[:sender] }
uids -= available_uids
break if uids.empty?
sleep Astute.config.mc_retry_interval
end
end
Astute.logger.warn "Offline node #{uids}" if uids.present?
uids
end
def report_final_node_progress(cluster)
node_report = cluster.nodes.inject([]) do |node_progress, node|
node_progress += [{'uid' => node[0].to_s, 'progress' => 100}]
end
ctx.report('nodes' => node_report)
end
end
end

51
lib/astute/task_manager.rb
View File

@ -1,51 +0,0 @@
# Copyright 2014 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class TaskManager
def initialize(nodes)
@tasks = nodes.inject({}) do |h, n|
h.merge({n['uid'] => n['tasks'].sort_by{ |f| f['priority'] }.each})
end
@current_task = {}
Astute.logger.info "The following tasks will be performed on nodes: " \
"#{@tasks.map {|k, v| {k => v.to_a}}.to_yaml}"
end
def current_task(node_id)
@current_task[node_id]
end
def next_task(node_id)
@current_task[node_id] = @tasks[node_id].next
rescue StopIteration
@current_task[node_id] = nil
delete_node(node_id)
end
def delete_node(node_id)
@tasks[node_id] = nil
end
def task_in_queue?
@tasks.select{ |_k,v| v }.present?
end
def node_uids
@tasks.select{ |_k,v| v }.keys
end
end
end

141
lib/astute/task_node.rb
View File

@ -1,141 +0,0 @@
# Copyright 2015 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
require 'fuel_deployment'
module Astute
class TaskNode < Deployment::Node
def context=(context)
@ctx = context
end
def run(inbox_task)
self.task = inbox_task
@task_engine = select_task_engine(task.data)
@task_engine.run
task.set_status_running
set_status_busy
report_node_status if report_running?(task.data)
end
def poll
return unless busy?
debug("Node #{uid}: task #{task.name}, task status #{task.status}")
# Please be informed that this code define special method
# of Deployment::Node class. We use special method `task`
# to manage task status, graph of tasks and nodes.
task.status = setup_task_status
if @task.running?
@ctx.report({
'nodes' => [{
'uid' => uid,
'deployment_graph_task_name' => task.name,
'task_status' => task.status.to_s,
}]
})
else
info "Finished task #{task} #{"with status: #{task.status}" if task}"
setup_node_status
report_node_status
end
end
def report_node_status
node_status = {
'uid' => uid,
'progress' => current_progress_bar,
}
node_status.merge!(node_report_status)
if task
node_status.merge!(
'deployment_graph_task_name' => task.name,
'task_status' => task.status.to_s,
'summary' => @task_engine.summary
)
node_status.merge!(
'error_msg' => "Task #{task.name} failed on node #{name}"
) if task.failed?
end
@ctx.report('nodes' => [node_status])
end
private
# This method support special task behavior. If task failed
# and we do not think that deployment should be stopped, Astute
# will mark such task as skipped and do not report error
def setup_task_status
if !task.data.fetch('fail_on_error', true) && @task_engine.failed?
Astute.logger.warn "Task #{task.name} failed, but marked as skipped "\
"because of 'fail on error' behavior"
return :skipped
end
@task_engine.status
end
def setup_node_status
if task
set_status_failed && return if task.failed?
set_status_skipped && return if task.dep_failed?
end
set_status_online
end
def current_progress_bar
if tasks_total_count != 0
100 * tasks_finished_count / tasks_total_count
else
100
end
end
def select_task_engine(data)
noop_prefix = noop_run? && not_noop_type?(data) ? "Noop" : ""
task_class_name = noop_prefix + data['type'].split('_').collect(&:capitalize).join
Object.const_get('Astute::' + task_class_name).new(data, @ctx)
rescue => e
raise TaskValidationError, "Unknown task type '#{data['type']}'. Detailed: #{e.message}"
end
def report_running?(data)
!['noop', 'stage', 'skipped'].include?(data['type'])
end
def noop_run?
cluster.noop_run
end
def node_report_status
if !finished?
{}
elsif skipped?
cluster.node_statuses_transitions.fetch('stopped', {})
elsif successful?
cluster.node_statuses_transitions.fetch('successful', {})
else
cluster.node_statuses_transitions.fetch('failed', {})
end
end
def not_noop_type?(data)
!['noop', 'stage', 'skipped'].include?(data['type'])
end
end
end

155
lib/astute/task_proxy_reporter.rb
View File

@ -1,155 +0,0 @@
# Copyright 2015 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
require 'digest/md5'
module Astute
module ProxyReporter
class TaskProxyReporter
INTEGRATED_STATES = ['error', 'stopped']
REPORT_REAL_TASK_STATE_MAP = {
'running' => 'running',
'successful' => 'ready',
'failed' => 'error',
'skipped' => 'skipped'
}
REPORT_REAL_NODE_MAP = {
'virtual_sync_node' => nil
}
def initialize(up_reporter)
@up_reporter = up_reporter
@messages_cache = []
end
def report(original_data)
return if duplicate?(original_data)
data = original_data.deep_dup
if data['nodes']
nodes_to_report = get_nodes_to_report(data['nodes'])
return if nodes_to_report.empty? # Let's report only if nodes updated
data['nodes'] = nodes_to_report
end
@up_reporter.report(data)
end
private
def get_nodes_to_report(nodes)
nodes.map{ |node| node_validate(node) }.compact
end
def node_validate(original_node)
node = deep_copy(original_node)
return unless node_should_include?(node)
convert_node_name_to_original(node)
return node unless are_fields_valid?(node)
convert_task_status_to_status(node)
normalization_progress(node)
return node
end
def are_fields_valid?(node)
are_node_basic_fields_valid?(node) && are_task_basic_fields_valid?(node)
end
def node_should_include?(node)
is_num?(node['uid']) ||
['master', 'virtual_sync_node'].include?(node['uid'])
end
def valid_task_status?(status)
REPORT_REAL_TASK_STATE_MAP.keys.include? status.to_s
end
def integrated_status?(status)
INTEGRATED_STATES.include? status.to_s
end
# Validate of basic fields in message about node
def are_node_basic_fields_valid?(node)
err = []
err << "Node uid is not provided" unless node['uid']
err.any? ? fail_validation(node, err) : true
end
# Validate of basic fields in message about task
def are_task_basic_fields_valid?(node)
err = []
err << "Task status provided '#{node['task_status']}' is not supported" if
!valid_task_status?(node['task_status'])
err << "Task name is not provided" if node['deployment_graph_task_name'].blank?
err.any? ? fail_validation(node, err) : true
end
def convert_task_status_to_status(node)
node['task_status'] = REPORT_REAL_TASK_STATE_MAP.fetch(node['task_status'])
end
# Normalization of progress field: ensures that the scaling progress was
# in range from 0 to 100 and has a value of 100 fot the integrated node
# status
def normalization_progress(node)
if node['progress']
node['progress'] = 100 if node['progress'] > 100
node['progress'] = 0 if node['progress'] < 0
else
node['progress'] = 100 if integrated_status?(node['status'])
end
end
def fail_validation(node, err)
msg = "Validation of node:\n#{node.pretty_inspect} for " \
"report failed: #{err.join('; ')}"
Astute.logger.warn(msg)
false
end
def convert_node_name_to_original(node)
if REPORT_REAL_NODE_MAP.keys.include?(node['uid'])
node['uid'] = REPORT_REAL_NODE_MAP.fetch(node['uid'])
end
end
def is_num?(str)
Integer(str)
rescue ArgumentError, TypeError
false
end
# Save message digest to protect server from
# message flooding. Sure, because of Hash is complicated structure
# which does not respect order and can be generate different strings
# but we still catch most of possible duplicates.
def duplicate?(data)
msg_digest = Digest::MD5.hexdigest(data.to_s)
return true if @messages_cache.include?(msg_digest)
@messages_cache << msg_digest
return false
end
end
end
end

41
lib/astute/tasks/cobbler_sync.rb
View File

@ -1,41 +0,0 @@
# Copyright 2015 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class CobblerSync < Task
def post_initialize(task, context)
@work_thread = nil
end
private
def process
cobbler = CobblerManager.new(
task['parameters']['provisioning_info']['engine'],
ctx.reporter
)
@work_thread = Thread.new { cobbler.sync }
end
def calculate_status
@work_thread.join and succeed! unless @work_thread.alive?
end
def validation
validate_presence(task['parameters'], 'provisioning_info')
end
end
end

58
lib/astute/tasks/copy_files.rb
View File

@ -1,58 +0,0 @@
# Copyright 2015 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class CopyFiles < Task
def post_initialize(task, context)
@work_thread = nil
@files_status = task['parameters']['files'].inject({}) do |f_s, n|
f_s.merge({ n['src']+n['dst'] => :pending })
end
end
private
def process
task['parameters']['files'].each do |file|
if File.file?(file['src']) && File.readable?(file['src'])
parameters = {
'content' => File.binread(file['src']),
'path' => file['dst'],
'permissions' => file['permissions'] || task['parameters']['permissions'],
'dir_permissions' => file['dir_permissions'] || task['parameters']['dir_permissions'],
}
@files_status[file['src']+file['dst']] =
upload_file(task['node_id'], parameters)
else
@files_status[file['src']+file['dst']] = false
end
end # files
end
def calculate_status
if @files_status.values.all?{ |s| s != :pending }
failed! if @files_status.values.include?(false)
succeed! if @files_status.values.all?{ |s| s == true }
return
end
end
def validation
validate_presence(task, 'node_id')
validate_presence(task['parameters'], 'files')
end
end
end

53
lib/astute/tasks/erase_node.rb
View File

@ -1,53 +0,0 @@
# Copyright 2016 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class EraseNode < Task
def summary
{'task_summary' => "Node #{task['node_id']} was erased without reboot"\
" with result #{@status}"}
end
private
def process
erase_node
end
def calculate_status
succeed!
end
def validation
validate_presence(task, 'node_id')
end
def erase_node
remover = MClient.new(
ctx,
"erase_node",
Array(task['node_id']),
_check_result=false)
response = remover.erase_node(:reboot => false)
Astute.logger.debug "#{ctx.task_id}: Data received from node "\
"#{task['node_id']} :\n#{response.pretty_inspect}"
rescue Astute::MClientTimeout, Astute::MClientError => e
Astute.logger.error("#{ctx.task_id}: #{task_name} mcollective " \
"erase node command failed with error #{e.message}")
failed!
end
end
end

60
lib/astute/tasks/master_shell.rb
View File

@ -1,60 +0,0 @@
# Copyright 2016 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class MasterShell < Task
# Accept to run shell tasks using existing shell asynchronous
# mechanism. It will run task on master node.
def post_initialize(task, context)
@shell_task = nil
end
def summary
@shell_task.summary
rescue
{}
end
private
def process
@shell_task = Shell.new(
generate_master_shell,
ctx
)
@shell_task.run
end
def calculate_status
self.status = @shell_task.status
end
def validation
validate_presence(task['parameters'], 'cmd')
end
def setup_default
task['parameters']['timeout'] ||= Astute.config.shell_timeout
task['parameters']['cwd'] ||= Astute.config.shell_cwd
task['parameters']['retries'] ||= Astute.config.shell_retries
task['parameters']['interval'] ||= Astute.config.shell_interval
end
def generate_master_shell
task.merge('node_id' => 'master')
end
end
end

89
lib/astute/tasks/move_to_bootstrap.rb
View File

@ -1,89 +0,0 @@
# Copyright 2016 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class MoveToBootstrap < Task
def post_initialize(task, context)
@work_thread = nil
end
def summary
{'task_summary' => "Node #{task['node_id']} was move to bootstrap with"\
" result #{@status}"}
end
private
def process
cobbler = CobblerManager.new(
task['parameters']['provisioning_info']['engine'],
ctx.reporter
)
@work_thread = Thread.new do
is_exist = cobbler.existent_node?(task['parameters']['provisioning_info']['slave_name'])
# Change node type to prevent wrong node detection as provisioned
# Also this type if node will not rebooted, Astute will be allowed
# to try to reboot such nodes again
change_nodes_type('reprovisioned') if is_exist
bootstrap_profile = task['parameters']['provisioning_info']['profile'] ||
Astute.config.bootstrap_profile
cobbler.edit_node(task['parameters']['provisioning_info']['slave_name'],
{'profile' => bootstrap_profile})
cobbler.netboot_node(task['parameters']['provisioning_info']['slave_name'],
true)
Reboot.new({'node_id' => task['node_id']}, ctx).sync_run if is_exist
Rsyslogd.send_sighup(
ctx,
task['parameters']['provisioning_info']['engine']['master_ip']
)
cobbler.remove_node(task['parameters']['provisioning_info']['slave_name'])
# NOTE(kozhukalov): We try to find out if there are systems
# in the Cobbler with the same MAC addresses. If so, Cobbler is going
# to throw MAC address duplication error. We need to remove these
# nodes.
mac_duplicate_names = cobbler.node_mac_duplicate_names(task['parameters']['provisioning_info'])
cobbler.remove_nodes(mac_duplicate_names.map {|n| {'slave_name' => n}})
cobbler.add_node(task['parameters']['provisioning_info'])
end
end
def calculate_status
@work_thread.join and succeed! unless @work_thread.alive?
end
def validation
validate_presence(task['parameters'], 'provisioning_info')
validate_presence(task, 'node_id')
end
def change_nodes_type(type="image")
run_shell_without_check(
task['node_id'],
"echo '#{type}' > /etc/nailgun_systemtype",
_timeout=5
)[:stdout]
rescue Astute::MClientTimeout, Astute::MClientError => e
Astute.logger.debug("#{ctx.task_id}: #{task_name} mcollective " \
"change type command failed with error #{e.message}")
nil
end
end
end

29
lib/astute/tasks/noop.rb
View File

@ -1,29 +0,0 @@
# Copyright 2015 Mirantis, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
module Astute
class Noop < Task
private
def process
end
def calculate_status
skipped!
end
end
end

Some files were not shown because too many files have changed in this diff Show More