On the effort to make Sahara more user and operators friendly
we are splitting the plugins from Sahara core.
The main goal of this change is to facilitate installation,
maintainance and upgrade of plugins. With the plugins outside
the main Sahara code, operators will be able to install a subset
of plugins, as well as upgrade to newer versions of the plugins
without having to wait for a new version of OpenStack to be
released. As well, it aims to facilitate new contributors to
develop and maintain their own plugins.
Sahara Spec: https://specs.openstack.org/openstack/sahara-specs/specs/rocky/plugins-outside-sahara-core.html
Change-Id: I7ed0945fd82e37daaf6b29f947d3dba06db9d158
port number for HBase configured in Sahara starts with 600xx
where as hortonworks ambari configures these ports on the
hbase master and region servers with 160xx.Although Ambari
will start the master server with the 160xx port, clients
can't connect them due to improper security rules.As a
result hbase master web info port link in the cluster
general info page doesn't work.HBase region server won't start.
This patch sets the HBase ports correctly for Ambari Plugin.
Change-Id: Ic13944ed729c0840578784f50a53f17b0706b62c
Story: #2003562
Task: #24853
With the new validation system, image versions on get_image_arguments()
has to contain all available versions in order to allow cluster creation
and validation with them all.
Story: #2001888
Change-Id: I88c2a553512a797099a4b2fbd9e9d204475e755b
Fix some errors by deploying Ubuntu clusters with
Ambari plugin.
Done:
1. Fix errors by setup ambari-agents in 109l of
sahara/plugins/ambari/deploy.py
2. Added "ranger-hdfs-plugin-properties" to configs
2.4 Ambari version.
3. Added method "_resolve_package_conflicts" for
uninstall packages which conflict with Oozie
packages.
4. Disable base repos usage for ubuntu in case of cluster
deploing with mirror.
co-authored-by: Michael Ionkin <mionkin@mirantis.com>
Change-Id: I57c18c4905314f4bf2b8f9fe3a1ff6035c5a9088
Closes-bug: 1567955
this change reimplements oozie client to support running
of edp jobs on top of the kerberized clusters.
additionally, we will setup keytabs and principals in order
to have ability to process authentication in clients.
these principals will be refreshed by additional cron
jobs once per hour.
implements bp: initial-kerberos-integration
Change-Id: I55b4267c09c43fdef75a443410f4ae11f56127e1
Rack awareness is added in HDP plugin for maintain data locality
feature of Hadoop.
Also renamed decomission_helper.py to requests_helper.py
note: Ambari allows to set up rack info through cluster creation
blueprint, but it doesn't work for scaling so that is why we manually
adds rack info for new instances (when scaling) through Ambari API and
that is why we manually restarts HDFS and MAPREDUCE2 services after
scaling.
closes-bug: 1618831
Change-Id: I2a696ea534d0a93f7e88c5e22ef99b008248c875
This patch implements designate integration. Added:
* 'use_designate' and 'nameservers' configs
* 'domain_name' column to Cluster and Cluster Templates and
'dns_hostname' to Instance in DB
* get_management_ip() method for Instance for properly setup
hostnames in cluster info and use it in ambari, cdh, mapr, spark,
storm and vanilla plugins
* method for change /etc/resolv.conf on all cluster instances
according to 'nameservers' config
* method for generate heat template for designate A and PTR records
* validation check for cluster template creation because now it
contains new field - 'domain_name'
* fix unit tests appropriatly and new tests added
* updated heat_template_version to '2016-04-08' (because we use
'str_split' method)
NOTE: in spec we proposed to use two domains: one for internal
resolution and another for external. But we decided to use only one
domain for both sides so an instance will have one hostname in the
domain and two appropriate A records because it's more convenient.
bp: designate-integration
Change-Id: I66525c0305450d8ba08840be1c95bfe79f21fff7
this change will remove config groups associated with hosts
which were removed during decommission.
Change-Id: Ie345784fad0a3c258645f91ff0708f01e3530886
Closes-bug: 1573512
Namenode(s) and resourcemanager(s) need to be restarted after down
scaling of ambari cluster. This patch adds this functionality.
Change-Id: I15c838d0e1f056dda8e468d5e2012ba6941b9782
closes-bug: 1573517
Hive service reqiuires a config file hive-site.xml to be uploaded to
HDFS befor running the jobs. However Ambari stores that file in the
local filesystem.
This patch uploads hive-site.xml to HDFS.
Closes-bug: 1564422
Change-Id: If50aadb071d661a8c7e29962f66375d98c8d124d
This patch adds our custom hadoop swiftfs implementation to Ambari
cluster instances after their start in order to allow usage of
swift with Keystone API v3.
p.s. required jar file should be previously saved in the
/opt folder of the base images.
Closes-bug: 1558064
Change-Id: Ie6df4a542a16b4417b505b4a621e8b4b921364d3
The general configs added to enable HA mode for both NameNode and
ResourceManager.
The HA is enabled by adding a set of configs to the Blueprint.
implements bp: hdp-22-support
Change-Id: Ie9336ca4f5bd24856fa2f3017b0cd61ce5145d1d
this change implements ambari alerts health checks.
ambari service checks seems to be harmful in case of unhealthy
clusters, so implementation of that will be postponed.
Partially-implements blueprint: cluster-verification
Change-Id: I4898d44e392944cdddbd14154738b8935a9b6da4
Scale up and down operation is now supported by HDP 2.2 plugin.
The NodeManagers and DataNodes are decommissioned when necessary.
Added scaling support in scenario tests.
partially implements bp: hdp-22-support
Co-Authored-By: Vitaly Gridnev <vgridnev@mirantis.com>
Co-Authored-By: Michael Ionkin <mionkin@mirantis.com>
Change-Id: I931e7874e2502b2d711cc8754c21683deeb4a248
Ambari plugin didn't enabled by default in Sahara.
And current supported version of HDP is 2.3.
Hence we can remove support of HDP 2.2
Change-Id: Ic453e889a55d5e321d63bdb8bd0a4bbf4c384797
* add hacking check: no_mutable_default_args()
* remove the mutable default arguments "[]" or "{}" when the function
is defined.
ref: http://docs.python-guide.org/en/latest/writing/gotchas/
Closes-Bug: #1530282
Change-Id: Ice6f7654211b73d7f8bc3ca0e4dfae3dca354397
Start support Spark EDP jobs in Ambari plugin.
Also added sparkPi job to test this support.
Current issues:
* Spark with Swift doesn't work
Change-Id: I2374d387054efa20876fbcee46ede038e0f3d520
Partially-implements-blueprint: hdp-22-support
It can useful to allow users to disable base repos in case
of no internet access. In such case we should prevent
possible fetching base Centos mirrors.
Closes-bug: 1508701
Change-Id: If6003d7a050dd6646c02d3c655054b01e0861174
Open ports for Kafka Broker (default 6667) and also
open port for ZooKeeper (default 2181).
Change-Id: Ifabdcfa295de03a2b1a120bf291c7b876948e921
Closes-bug: 1512661
Ambari plugin should return dictionary with
versions in job types method of Ambari plugin.
Closes-bug: 1502841
Change-Id: Icdfd1baf8543977c7f7c90847b6d75534a077d1b
This patch added support of the following services:
* Oozie
* Hive
Known issues:
* Pig job with Swift doesn't work
partially implements bp: hdp-22-support
Change-Id: I92e6f09e0c9685f24cdcf3ca386c76ea20134e3e
This patch added support of base Hadoop services:
HDFS:
* NameNode
* DataNode
* SecondaryNameNode
YARN:
* ResourceManager
* NodeManager
* MapReduce History Server
* YARN Timeline Server
ZooKeeper:
* ZooKeeper Server
Supported features:
* Cinder
* Configs provisioning
* Swift integration
Also added scenario tests for plugin.
partially implements bp: hdp-22-support
Change-Id: I2e1188735132881cef1026624cca580ccc7bb2ee
Current features:
* Set up Ambari server and Ambari agents from pre-built image
* Support Ambari 2.0 / 2.1
* Password generation for management console
partially implements bp: hdp-22-support
Change-Id: I53084a83cccde52654a42911f449cc8b7d769bea
Telles Nobrega