On the effort to make Sahara more user and operators friendly
we are splitting the plugins from Sahara core.
The main goal of this change is to facilitate installation,
maintainance and upgrade of plugins. With the plugins outside
the main Sahara code, operators will be able to install a subset
of plugins, as well as upgrade to newer versions of the plugins
without having to wait for a new version of OpenStack to be
released. As well, it aims to facilitate new contributors to
develop and maintain their own plugins.
Sahara Spec: https://specs.openstack.org/openstack/sahara-specs/specs/rocky/plugins-outside-sahara-core.html
Change-Id: I7ed0945fd82e37daaf6b29f947d3dba06db9d158
It was deprecated in Liberty and is marked
for removal in Mitaka.
Change-Id: I3ba6941b1e1aa6900b5f59ea52a0370577729d9e
Implements: blueprint remove-spark-100
For Spark/Swift integration, we use a wrapper class to set up
the hadoop environment. For this to succeed, the current
working directory must be on the classpath. Newer versions of
Spark have changed how the default classpath is generated, so
Sahara must ensure explicitly that the working dir will be
included.
Change-Id: I6680bf8736cada93e87821ef37de3c3b4202ead4
Close-Bug: #1486544
Spark jobs in Cloudera 5.3.0 and 5.4.0 plugins are now supported.
Required unit tests have been added. Merged with current
master HEAD.
Change-Id: Ic8fde97e424e45c6f31f7794749793b26c844915
Implements: blueprint spark-jobs-for-cdh-5-3-0
This change implements the optional methods in the Plugins SPI
to support the job-types endpoint for the Spark plugin.
Config hints at this point are unchanged. Additional work may be
needed to provide config-hints specific to Spark plugin versions.
Partial-Implements: blueprint edp-job-types-endpoint
Change-Id: I1cd318da11c997119b192e7396969f89d8f0f216
Now EDP engine is fully responsible on validation of data for
job execution.
Other changes:
* Removed API calls from validation to remove circular dependancy
* Removed plugins patching in validation to allow non-vanilla
plugins testing
* Renamed job_executor to job_execution
Change-Id: I14c86f33b355cb4317e96a70109d8d72d52d3c00
Closes-Bug: #1357512
+ Moved 'get_hdfs_user' method from plugin SPI to EDP engine
Futher steps: move other EDP-specific method to EDP engine
Change-Id: I0537397894012f496ea4abc2661aa8331fbf6bd3
Partial-Bug: #1357512
Telles Nobrega