On the effort to make Sahara more user and operators friendly
we are splitting the plugins from Sahara core.
The main goal of this change is to facilitate installation,
maintainance and upgrade of plugins. With the plugins outside
the main Sahara code, operators will be able to install a subset
of plugins, as well as upgrade to newer versions of the plugins
without having to wait for a new version of OpenStack to be
released. As well, it aims to facilitate new contributors to
develop and maintain their own plugins.
Sahara Spec: https://specs.openstack.org/openstack/sahara-specs/specs/rocky/plugins-outside-sahara-core.html
Change-Id: I7ed0945fd82e37daaf6b29f947d3dba06db9d158
Spark 1.3.1 came out in Liberty, and was quickly succeeded in Mitaka
with Spark 1.6.0, so it is safe to deprecate.
Change-Id: Ic1a1cf579c5e73a71e44992ea49434db7c4bcf96
Previously RequiredServiceMissingException was raised for
all cases of invalid count for Spark master, even if the
service was present but count was otherwise incorrect. Now
the correct exception will be raised depending on whether
master is missing or master has wrong count.
Also, add missing unit tests for validation.
Change-Id: Ide7cc8414086e377c86306630920a3c753811458
Log messages are no longer being translated. This removes all use of
the _LE, _LI, _LW and _LC translation markers to simplify logging and to
avoid confusion with new contributions.
See:
http://lists.openstack.org/pipermail/openstack-i18n/2016-November/002574.htmlhttp://lists.openstack.org/pipermail/openstack-dev/2017-March/113365.html
It will be a giant patch if all these _LE, _LI, _LW and _LC being deleted
within one patch, so this patch only delete _LI; _LW, _LC and _LE has been
handled in previous patchies. This is the last patch for sahara to delete
log translations.
Change-Id: I73e99ca7fd5bc0cd8df7cce30b5f12e00e70149c
This patch implements designate integration. Added:
* 'use_designate' and 'nameservers' configs
* 'domain_name' column to Cluster and Cluster Templates and
'dns_hostname' to Instance in DB
* get_management_ip() method for Instance for properly setup
hostnames in cluster info and use it in ambari, cdh, mapr, spark,
storm and vanilla plugins
* method for change /etc/resolv.conf on all cluster instances
according to 'nameservers' config
* method for generate heat template for designate A and PTR records
* validation check for cluster template creation because now it
contains new field - 'domain_name'
* fix unit tests appropriatly and new tests added
* updated heat_template_version to '2016-04-08' (because we use
'str_split' method)
NOTE: in spec we proposed to use two domains: one for internal
resolution and another for external. But we decided to use only one
domain for both sides so an instance will have one hostname in the
domain and two appropriate A records because it's more convenient.
bp: designate-integration
Change-Id: I66525c0305450d8ba08840be1c95bfe79f21fff7
this change contains following changes:
1. get_labels implementation for stable plugins;
2. improvements in exposing list of plugins: now
it returns detailed info about labels also;
3. fixed minor validation issue with name parameter
of validator.
4. validations on labels on cluster creation,
cluster scaling and launching jobs.
blueprint: plugin-management-api
Change-Id: I8baf6b60f3be1c5ed81758712258b65c4d30e443
It was deprecated in Liberty and is marked
for removal in Mitaka.
Change-Id: I3ba6941b1e1aa6900b5f59ea52a0370577729d9e
Implements: blueprint remove-spark-100
Spark 1.6.0 is available now for deployment.
Also added the current working directory to the driver class path for
proper reading of the spark.xml file
Change-Id: I9a46a503c7e52d756c7de8c8694dbfc51f80f2be
Co-Authored-By: Vitaly Gridnev <vgridnev@mirantis.com>
bp: support-spark-160
* add hacking check: no_mutable_default_args()
* remove the mutable default arguments "[]" or "{}" when the function
is defined.
ref: http://docs.python-guide.org/en/latest/writing/gotchas/
Closes-Bug: #1530282
Change-Id: Ice6f7654211b73d7f8bc3ca0e4dfae3dca354397
Currently only cinder volumes can be formatted and mounted.
This patch provides an ability of formatting and mounting
any attached storage devices. Also storage_paths method was
moved from node group objects to instance and corresponding
fixes in plugins was made.
Closes-bug: #1478899
Closes-bug: #1478904
Change-Id: If92d6fdea25e374d6d5c404be5ac3677cb60c057
Implemented option for disabling scaling for old clusters.
After autoconfiguration we will put 'auto-configured' flag into
cluster.extra field. That will allow us to disable autotune configs
for old created clusters.
Partial-Implements blueprint: recommend-configuration
Change-Id: I7d09ef20da51a170e9c021032c8bf13c34ae8d85
We can autoconfigure dfs.replication in Spark plugin, so let's do it.
Partial-Implements blueprint: recommend-configuration
Change-Id: I0f4ce35fdf71c424c9ff6ca96e04ba5d63aeb1b0
As discussed in the Sahara weekly meeting, this change request
removes 0.9.1 from the list of supported Spark versions and adds
the deprecation message to the validation phase for clusters
created for Spark 1.0.0, the same way as is done in the vanilla
plugin for version 2.4.1
Change-Id: I13965c09c083823511e66c7ae42317ceae1059cb
Implements: blueprint deprecate-spark-100
Add version "1.3.1" to the list of supported versions, in tandem with
https://review.openstack.org/#/c/195054/ that adds support in the DIB.
This one-liner is sufficient, from Sahara's point of view Spark 1.0
and 1.3 are the same.
This patch set Updates also the Spark plugin documentation and the
tests.
Change-Id: I8fa482b6d1d6abaa6633aec309a3ba826a8b7ebb
Implements: blueprint support-spark-1-3
The EDP Spark engine was importing a config helper from the Spark
plugin.
The helper was moved to common plugin utils and now is imported from
there by both the plugin and the engine.
This is the part of sahara and plugins split.
Partially-implements bp: move-plugins-to-separate-repo
Change-Id: Ie84cc163a09bf1e7b58fcdb08e0647a85492593b
Currently some ids in Sahara log messages are noticed twice.
All this messages should be checked and fixed to avoid information
duplication.
partially implements bp: logs-improvement
Change-Id: I1fd3f8b76ed6f56d9e8116ea453d05246a68f7dc
This change implements the optional methods in the Plugins SPI
to support the job-types endpoint for the Spark plugin.
Config hints at this point are unchanged. Additional work may be
needed to provide config-hints specific to Spark plugin versions.
Partial-Implements: blueprint edp-job-types-endpoint
Change-Id: I1cd318da11c997119b192e7396969f89d8f0f216
This is an initial commit to make sahara logs relevant to new
logging spec.
We need to change some log levels to become closer to the spec, also
we need to change some log messages (e.g. for INFO level).
partially-implement bp new-style-logging
Change-Id: I8e7b7c4c3f375648d2999b1d0e19b68390cc22a8
* scaling down now works correctly
* renaming _start_slave_datanode_processes to
_start_datanode_processes to avoid confusion
Change-Id: I1bf3dea47793e7358d9ba3d632828639cd128c40
Closes-bug: #1376228
Changes:
* using oslo_config instead of oslo.config
* using oslo_concurrency instead of oslo.concurrency
* using oslo_db instead of oslo.db
* using oslo_i18n instead of oslo.i18n
* using oslo_messaging instead of oslo.messaging
* using oslo_middleware instead of oslo.middleware
* using oslo_serialization instead of oslo.serialization
* using oslo_utils instead of oslo.utils
Change-Id: Ib0f18603ca5b0885256a39a96a3620d05260a272
Closes-bug: #1414587
This change allows Spark jobs to access Swift URLs without
any need to modify the Spark job code itself. There are a
number of things necessary to make this work:
* add a "edp.spark.adapt_for_swift" config value to control the
feature
* generate a modified spark-submit command when the feature is
enabled
* add the hadoop-swift.jar to the Spark classpaths for the
driver and executors (cluster launch)
* include the general Swift configs in the Hadoop core-site-xml
and make Spark read the Hadoop core-site.xml (cluster launch)
* upload an xml file containing the Swift authentication configs
for Hadoop
* run a wrapper class that reads the extra Hadoop configuration
and adds it to the configuration for the job
Changes in other CRs:
* add the hadoop-swift.jar to the Spark images
* add the SparkWrapper code to sahara-extra
Partial-Implements: blueprint edp-spark-swift-integration
Change-Id: I03dca4400c832f3ba8bc508d4fb2aa98dede8d80
Introduces a periodic task for the cleanup of data from Spark jobs, in order
to ease maintenance of long-lived clusters.
Change-Id: Ia7dc2dde54ab62199a630c3d1b64c76f08698181
Implements: blueprint spark-cleanup
oslo.log was added to global requirements, so we can migrate to this module.
Also we need this migration to be ensure that it works correctly with
log-improvements.
Since openstack.common.log is not dropped in oslo-incubator,
we shouldn't remove it.
Change-Id: I90468e4db812ae0b5d8a43a37206b236f8904661
Closes-bug: #1412673
Plugins dir contains 'general' module which looks like yet another
plugin along with vanilla, fake, hdp, spark and cdh. But it doesn't
and contains two files only. Moved them one level up to avoid such
confusion.
Closes-Bug: #1378178
Change-Id: Ia600e4c584d48a3227552f0051cc3bf906206bed
Now EDP engine is fully responsible on validation of data for
job execution.
Other changes:
* Removed API calls from validation to remove circular dependancy
* Removed plugins patching in validation to allow non-vanilla
plugins testing
* Renamed job_executor to job_execution
Change-Id: I14c86f33b355cb4317e96a70109d8d72d52d3c00
Closes-Bug: #1357512
+ Moved 'get_hdfs_user' method from plugin SPI to EDP engine
Futher steps: move other EDP-specific method to EDP engine
Change-Id: I0537397894012f496ea4abc2661aa8331fbf6bd3
Partial-Bug: #1357512
Telles Nobrega