On the effort to make Sahara more user and operators friendly
we are splitting the plugins from Sahara core.
The main goal of this change is to facilitate installation,
maintainance and upgrade of plugins. With the plugins outside
the main Sahara code, operators will be able to install a subset
of plugins, as well as upgrade to newer versions of the plugins
without having to wait for a new version of OpenStack to be
released. As well, it aims to facilitate new contributors to
develop and maintain their own plugins.
Sahara Spec: https://specs.openstack.org/openstack/sahara-specs/specs/rocky/plugins-outside-sahara-core.html
Change-Id: I7ed0945fd82e37daaf6b29f947d3dba06db9d158
It seems to be easy to hit timeout errors, so let's increase
the value of the timeout when ambari-server is started.
The option is recognized only from Ambari 2.5 onwards, but
the plugin uses Ambari 2.6 for all versions but HDP 2.3.
Story: 2004102
Task: 27509
Change-Id: I47c6dd14585ed20998fcf4a068cf1e7144eaf026
Often time ambari fails during cluster installation/service
starting stage. This is quiet prominent when there is
a large number of nodes in the cluster. Review of the
logs from the cluster indicates that ambari installation
scripts has a timeout parameter set to 1800 sec, this
requires adjustment depending on the environment and
speed of package installation.
This fix provides one parameter named
"agent.package.install.task.timeout" inside the Ambari tab
of the HDP cluster template UI . User may change the
values and accordingly the ambari server will be setup
for package installation timeout.'
Change-Id: I826dbebb446d49e01e3cd6d7e525b43aa4523434
Story: #2003176
Task: #23320
When hdp2.5 cluster installation is done with off-line
repo,hdp-utils repo name should be HDP-UTILS-1.1.0.21.
However this is hardcoded wrongly in sahara repo_id_map
as HDP-UTILS-1.1.0.20.As a result Ambari HDP repo set up
fails.
This fix will correct the repo_id_map
Change-Id: Ibf7d341cc2d2a53be521039d4f843cdbc5ee880b
Story: #2003654
Task: #26067
Log messages are no longer being translated. This removes all use of
the _LE, _LI, _LW and _LC translation markers to simplify logging and to
avoid confusion with new contributions.
See:
http://lists.openstack.org/pipermail/openstack-i18n/2016-November/002574.htmlhttp://lists.openstack.org/pipermail/openstack-dev/2017-March/113365.html
It will be a giant patch if all these _LE, _LI, _LW and _LC being deleted
within one patch, so this patch only delete _LW, _LC and _LE has been
handled in previous patch; deletion of _LI will be in the following patches
related with this one.
Change-Id: Ia696ee76f22e488b9a219305f4f220eca2161cee
'Preparing policy files' step had incorrect count: '6/5'. It's because
this step was inside of 'Prepare Kerberos' step.
Change-Id: I2909085d7035d89d4a07cc249390c2e6589c0c16
closes-bug: 1627081
Fix some errors by deploying Ubuntu clusters with
Ambari plugin.
Done:
1. Fix errors by setup ambari-agents in 109l of
sahara/plugins/ambari/deploy.py
2. Added "ranger-hdfs-plugin-properties" to configs
2.4 Ambari version.
3. Added method "_resolve_package_conflicts" for
uninstall packages which conflict with Oozie
packages.
4. Disable base repos usage for ubuntu in case of cluster
deploing with mirror.
co-authored-by: Michael Ionkin <mionkin@mirantis.com>
Change-Id: I57c18c4905314f4bf2b8f9fe3a1ff6035c5a9088
Closes-bug: 1567955
this change reimplements oozie client to support running
of edp jobs on top of the kerberized clusters.
additionally, we will setup keytabs and principals in order
to have ability to process authentication in clients.
these principals will be refreshed by additional cron
jobs once per hour.
implements bp: initial-kerberos-integration
Change-Id: I55b4267c09c43fdef75a443410f4ae11f56127e1
Per [0] there are four strategies that can be used
for autoconfiguration in Ambari. We are providing ability to specify that.
[0] https://cwiki.apache.org/confluence/display/AMBARI/Blueprints
Change-Id: Ia73d289cfe7eb230fd86f43865eb392f0d4c02da
Rack awareness is added in HDP plugin for maintain data locality
feature of Hadoop.
Also renamed decomission_helper.py to requests_helper.py
note: Ambari allows to set up rack info through cluster creation
blueprint, but it doesn't work for scaling so that is why we manually
adds rack info for new instances (when scaling) through Ambari API and
that is why we manually restarts HDFS and MAPREDUCE2 services after
scaling.
closes-bug: 1618831
Change-Id: I2a696ea534d0a93f7e88c5e22ef99b008248c875
Add ability to automaticaly generate better configurations for
cluster. All calculations of recommended config values is executed
by Ambari itself according to 'ALWAYS_APPLY' strategy.
Change-Id: I98f14e4578397d6a8d14438e368485d5dbeaeed2
Using an centos7 image and changing the HDP and HDP-UTILS URL location
resulted in an error of
"os_type=os_type_map[client.get_host_info[ambari.fqdn())["os_type"]]
KeyError: u'centos7'. This results in only being able to change the
HDP and HDP-UTILS URL if you were using centos6 or redhat6.This change
adds centos7 and redhat7 to the map to allow custom HDP and HDP-UTILS URLs.
Should we add additional operating systems to the list?
Closes-Bug: #1599252
Change-Id: Iaa5cd773c5daeabeda287048a2d3f8696faa1436
this change will remove config groups associated with hosts
which were removed during decommission.
Change-Id: Ie345784fad0a3c258645f91ff0708f01e3530886
Closes-bug: 1573512
Namenode(s) and resourcemanager(s) need to be restarted after down
scaling of ambari cluster. This patch adds this functionality.
Change-Id: I15c838d0e1f056dda8e468d5e2012ba6941b9782
closes-bug: 1573517
Hive service reqiuires a config file hive-site.xml to be uploaded to
HDFS befor running the jobs. However Ambari stores that file in the
local filesystem.
This patch uploads hive-site.xml to HDFS.
Closes-bug: 1564422
Change-Id: If50aadb071d661a8c7e29962f66375d98c8d124d
This patch adds our custom hadoop swiftfs implementation to Ambari
cluster instances after their start in order to allow usage of
swift with Keystone API v3.
p.s. required jar file should be previously saved in the
/opt folder of the base images.
Closes-bug: 1558064
Change-Id: Ie6df4a542a16b4417b505b4a621e8b4b921364d3
There have been some missed and incorrect configurations which don't
allow to properly launch Resourcemanager HA
Also made some code refactoring - added string constants
Change-Id: Ia8d4c67ea850d94b76e7c65ea41c29e550d712ac
Closes-bug: 1553209
The general configs added to enable HA mode for both NameNode and
ResourceManager.
The HA is enabled by adding a set of configs to the Blueprint.
implements bp: hdp-22-support
Change-Id: Ie9336ca4f5bd24856fa2f3017b0cd61ce5145d1d
Adding "# nosec" comment per bandit documentation to avoid having it
be reported during the gate test.
Change-Id: I9b18c6ed5ada636350c8dc2b4d5ea1869750caa4
Closes-Bug: #1548991
Closes-Bug: #1548948
Closes-Bug: #1548965
Scale up and down operation is now supported by HDP 2.2 plugin.
The NodeManagers and DataNodes are decommissioned when necessary.
Added scaling support in scenario tests.
partially implements bp: hdp-22-support
Co-Authored-By: Vitaly Gridnev <vgridnev@mirantis.com>
Co-Authored-By: Michael Ionkin <mionkin@mirantis.com>
Change-Id: I931e7874e2502b2d711cc8754c21683deeb4a248
Ambari plugin didn't enabled by default in Sahara.
And current supported version of HDP is 2.3.
Hence we can remove support of HDP 2.2
Change-Id: Ic453e889a55d5e321d63bdb8bd0a4bbf4c384797
* use run_as_root=True instead of 'sudo' when calling execute_command
* format multi-line string value of init_db_template
Closes-Bug: #1530284
Change-Id: I8a3b6a069693abf0b3b726c9e221c378f350e19d
It can useful to allow users to disable base repos in case
of no internet access. In such case we should prevent
possible fetching base Centos mirrors.
Closes-bug: 1508701
Change-Id: If6003d7a050dd6646c02d3c655054b01e0861174
Currently only cinder volumes can be formatted and mounted.
This patch provides an ability of formatting and mounting
any attached storage devices. Also storage_paths method was
moved from node group objects to instance and corresponding
fixes in plugins was made.
Closes-bug: #1478899
Closes-bug: #1478904
Change-Id: If92d6fdea25e374d6d5c404be5ac3677cb60c057
This patch added support of base Hadoop services:
HDFS:
* NameNode
* DataNode
* SecondaryNameNode
YARN:
* ResourceManager
* NodeManager
* MapReduce History Server
* YARN Timeline Server
ZooKeeper:
* ZooKeeper Server
Supported features:
* Cinder
* Configs provisioning
* Swift integration
Also added scenario tests for plugin.
partially implements bp: hdp-22-support
Change-Id: I2e1188735132881cef1026624cca580ccc7bb2ee
Current features:
* Set up Ambari server and Ambari agents from pre-built image
* Support Ambari 2.0 / 2.1
* Password generation for management console
partially implements bp: hdp-22-support
Change-Id: I53084a83cccde52654a42911f449cc8b7d769bea
Telles Nobrega