* Create S3 data source type for EDP
* Support storing S3 secret key in Castellan
* Unit tests for new data source type
* Document new data source type and related ideas
* Add support of S3 configs into Spark and Oozie workflows
* Hide S3 credentials in job execution info, like for Swift
* Release note
Change-Id: I3ae5b9879b54f81d34bc7cd6a6f754347ce82f33
This patch overwrites migration from novaclient.images to
glanceclient.v2.images. Glanceclient uses Warlock library
and we want to be independent of this library because its 1.3.0
release had broken all our CI.
The idea is we speak to glanceclient which returns image dict,
then we transform this dict by adding our sahara related key-values.
Change-Id: I07e9875622ace6b0aa3cd098d36b2eeed59c6d54
co-authored-by: Vitaly Gridnev <vgridnev@mirantis.com>
this change implements basic things requried for
cluster verifications:
* API modifications;
* DB implementation and conductor ops;
* basic health checks;
* periodic job.
Partially implements blueprint: cluster-verification
Change-Id: I1b975c9a5e5241c660de66a04bf559fc8f960873
This change adds an "interface" map to the API for job creation, such that
the operator registering a job can define a unified, human-readable way to
pass in all arguments, parameters, and configurations that the execution
of that job may require or accept. This will allow platform-agnostic
wizarding at the job execution phase and allows users to document use of
their own jobs once in a persistent, standardized format.
Change-Id: I59b9b679a650361ddcd30975891496fdfbabb93c
Partially Implements: blueprint unified-job-interface-map
All non-plugin exceptions are now located in one place. The constructor
usage is now brought to one pattern.
Change-Id: I794143889edf89a8ab79d0c6b8fb410da39081a0
Attached volumes is a technical information and should not be in
cluster output. Moreover this informatio is not used by Sahara in
most cases. Currently it is used by direct engine only to detach
volumes it attached before. After removing direct provisioning
engine "volumes" table should be removed too.
Change-Id: I10d99aed74be1126065be68082673a2e8aa1027c
Closes-Bug: #1459447
* Generates a cluster-scoped proxy-user and keystone TRUST
* Pass a proxy-user auth and trust id to hive-server using hive-site.xml
Partially implements: blueprint edp-hive-vanilla-swift
Change-Id: I65592421b29428d3e726aaf0aaa4977378849ef0
+ Added validation check on scaling using other engine
Change-Id: I3f36b949d1388f809c33334be6fcd0bdb30ade7a
Implements: blueprint cluster-persist-sahara-configuration
Changes
* adding sahara.utils.proxy module
* adding functions to create and delete proxy users
* adding proxy user creation during job execution
* adding proxy user deletion during job execution status update
* adding sanitization for proxy configs from job execution
* adding unit test for proxy usage detection function
Partial-implements: blueprint edp-swift-trust-authentication
Change-Id: I551a14a3cb5320a27fc6104b35c3b9a08a03abda
Added ability to ask Sahara to create security group for node group. Feature
is only implemented in vanilla plugin 1.2.1 so far.
Partially implements: blueprint cluster-secgroups
Change-Id: I21d0196396bb966fe3d88f5445e98aebe90ad94b
1. Removed all error handling in provisioning engine and moved it
to ops.
2. Implemented general logic on handling with deleted cluster
Implements blueprint error-handling-in-provisioning
Change-Id: Ia161dea1c726bd95157a279eb0f2917af2516b07
Although there is no data model update to JobExecution there are some
minor changes to ensure that trust ids don't leak from the JobExecution
representations.
Changes
* adding sanitizing for trusts to the JobExecution Resource
* adding testing for trusts filter from JobExecution
Partially implement: blueprint edp-swift-trust-authentication
Change-Id: If88bba39efdab01195878a5b1d43bb6c975531b9
Basic filtering is available via Resource._filter_fields and
Resource._children but partial filtering of fields is not supported.
This change allows a Resource to define custom sanitization methods
for individual fields. The specified method is applied after the
field has been generated.
For nested structures, particularly when elements are not Resource
types, it is easier to scan the field after generation and remove
specific elements than it is to modify the recursive descent already
implemented in to_wrapped_dict().
JobExecutions are filtered with this mechanism to set swift credential
values to empty strings and to remove the 'conf' section reported
for running Oozie actions.
Closes-Bug: #1300291
Change-Id: Ifb0b99a3e13d40306139b0d0021aead69e870205
The 'job_configs' field will contain swift credentials used by Hadoop
when running jobs that use swift data sources. The 'extra' field may
contain a token for use with neutron.
The current filtering mechanism in Sahara allows filtering fields by name,
but it's not set up to support programmatic filtering of things which
can be variable. It may not be necessary to filter out 'job_configs'
entirely from a security perspective, but there currently is no other
option when a field might or might not contain particular values.
Additionally, sensitive information could potentially be passed in 'args'
within 'job_configs' and it is impossible to know anything about the
content of that field.
Closes-Bug: #1273661
Change-Id: Idb2e68a2d42e45bab04c62c740cbbaf5e51b2719
The 'mains' and 'libs' elements of a Job contain references
to JobBinary objects, which can contain credentials in the
'extra' field. Filter 'extra' from 'mains' and 'libs' when
returning a wrapped Job object.
Closes-Bug: #1272452
Change-Id: Ieccadbbda1f081595374469cb938b7bffbc08e17
Rename the subdirectory and replace all instances
of "import savanna" with "import sahara" and all
instances of "from savanna" with "from sahara".
* Replaced mock patches like mock.patch('savanna...
* Updated config generator script
* Renamed entry points in setup.cfg
* Hacking checks also fixed
* Manual renaming in alembic scripts to force work migrations
* Fix doc building
* Renamed itests directories
* Some changes in gitignore
* Removed locale dir after rebase
Co-Authored-By: Alexander Ignatov <aignatov@mirantis.com>
Change-Id: Ia77252c24046c3e7283c0a7b96d11636020b949c
Partially implements: blueprint savanna-renaming-service
Jeremy Freudberg