Attached volumes already had support for explicit volume type,
availability zone, and instance locality. Extend this capability to
boot volumes.
(For posterity's sake, a summary of why other options related to
attached volumes were not in fact appropriate for boot volumes:
- volumes_per_node: there can only be one boot volume
- volumes_size: boot volume size given by flavor
- volume_mount_prefix: boot volume handled differently)
Change-Id: I009c0da4179c880fd87a7c1903c93825e8997a38
Story: 2004507
Task: 28229
* Create S3 data source type for EDP
* Support storing S3 secret key in Castellan
* Unit tests for new data source type
* Document new data source type and related ideas
* Add support of S3 configs into Spark and Oozie workflows
* Hide S3 credentials in job execution info, like for Swift
* Release note
Change-Id: I3ae5b9879b54f81d34bc7cd6a6f754347ce82f33
* Create common module for managing S3 job binaries
* Add new dependency on botocore
* Use common S3 library to create S3 job binary type for EDP
* Use common S3 library to create S3 job binary retriever
* Support storing S3 secret key in Castellan
* Document new job binary type (and foreshadow the S3 data source type)
* Unit tests for new code
Change-Id: I6781203d802305446ba1418ed6999186db4dfe9b
Partially-Implements: bp sahara-support-s3
Changes to make the integration of the existing code with the data source
and job binary abstractions possible.
Change-Id: I524f25ac95bb634b0583113792460c17217acc34
Implements: blueprint data-source-plugin
Introduces a new field called anti_affinity_ratio.
Creates more server groups in case the
user defines the anti_affinity_ratio more than 1.
Default value of anti_affinity_ratio is kept 1.
Instances are placed in the server groups in round-robin
fashion.
Partially-Implements: blueprint improving-anti-affinity
Change-Id: I86bda256ea661fa0ba9d6d3819477edff080bcee
this change is only provides a basic implementation
of db objects related to plugin's api
Change-Id: I320a5b46f9d1e14cc3db81757808f062d6aa60a6
bp: plugin-management-api
This patch fix cluster creation with public template that has been
created in another tenant (not in current).
The problem is that at the particular time configs of cluster and node
groups are merging. And 'tenant_id' property of node group replaces the
same property of cluster config. Therefore, further sahara db api
cannot find this new cluster with current tenant id if node group has
been created in another tenant.
Change-Id: I8eaa4350ec873cca73a6e828efd54200912bacaa
closes-bug: 1585211
this change implements basic things requried for
cluster verifications:
* API modifications;
* DB implementation and conductor ops;
* basic health checks;
* periodic job.
Partially implements blueprint: cluster-verification
Change-Id: I1b975c9a5e5241c660de66a04bf559fc8f960873
This change implements regex matching for filters on string
values passed to job_executions_list() in the REST api.
No existing internal calls that use job_execution_get_all()
are affected.
Partial-bug: #1503345
Change-Id: Id78d4af41da23b093dafc434c6852e893593d605
This change implements regex matching for filters on string
values passed to job_binary_internal_list() in the REST api.
No existing internal calls that use job_binary_internal_get_all()
are affected.
Partial-bug: #1503345
Change-Id: I268c439ab83587375dbff7e125e23f07c9381d83
This change implements regex matching for filters on string
values passed to job_list() in the REST api.
No existing internal calls that use job_get_all()
are affected.
Partial-bug: #1503345
Change-Id: Id19eeae194d7e4c97f041e067f1ee8a57a675815
This change implements regex matching for filters on string
values passed to job_binary_list() in the REST api.
No existing internal calls that use job_binary_get_all()
are affected.
Partial-bug: #1503345
Change-Id: Id34696ca128214ebb555980ca1f9f068dabc1a11
This change implements regex matching for filters on string
values passed to node_group_templates_list() in the REST api.
No existing internal calls that use node_group_templates_get_all()
are affected.
Partial-bug: #1503345
Change-Id: I8e8465332c58ed657f22b64016c40e041bd3a54e
This change implements regex matching for filters
on string values passed to clusters_list() in the REST api.
No existing internal calls that use cluster_get_all()
are affected.
Change-Id: Ifac63bef69cdf8c94132bfb5f14b646ae587b9ab
Partial-bug: #1503345
This change implements regex matching for filters on string
values passed to data_sources_list() in the REST api.
No existing internal calls that use data_sources_get_all()
are affected.
Partial-bug: #1503345
Change-Id: I906ddf6dc27336c39b7ccf298b0e3558ec86178b
This change implements regex matching for filters on string
values passed to cluster_templates_list() in the REST api.
No existing internal calls that use cluster_templates_get_all()
are affected.
Partial-bug: #1503345
Change-Id: I1a320ff8c01ef5fdc5e66ed35a2b7a6ec9173175
With the change of semantics for templates with
'is_default' and 'is_protected' set, this flag
should be changed everywhere to have a more
appropriate name.
Change-Id: Ifc913e34e94dc59604f733c991858e7011bd728f
Partial-implements: blueprint replace-is-default
this change adds a utils module to the castellan service package. this
module contains 3 wrapper functions to help reduce the overhead for
working with castellan.
* add sahara.service.castellan.utils module
* fixup previous usages of the castellan key manager
Change-Id: I6ad4e98ab41788022104ad2886e0ab74e4061ec3
Partial-Implements: blueprint improved-secret-storage
This change adds the sahara key manager and converts the proxy passwords
and swift passwords to use the castellan interface.
* adding sahara key manager
* adding castellan to requirements
* removing barbicanclient from requirements
* removing sahara.utils.keymgr and related tests
* adding castellan wrapper configs to sahara list_opts
* creating a castellan validate_config to help setup
* updating documentation for castellan usage
* fixing up tests to work with castellan
* converting all proxy password usages to use castellan
* converting job binaries to use castellan when user credentials are
applied
* converting data source to use castellan when user credentials are
applied
Change-Id: I8cb08a365c6175744970b1037501792fe1ddb0c7
Partial-Implements: blueprint improved-secret-storage
Closes-Bug: #1431944
When we update the shares on a cluster, we are now
calling the unmount_shares routine anytime we have
removed shares from the cluster.
Change-Id: Id028df9d09b8d603500b80af14fdf3be2ddde4d3
Partial-Implements: bp sahara-add-shares-to-clusters
When we update the shares on a cluster, we are now
calling the mount_shares routine anytime we have
new or changed shares so that the shares will be
mounted. Unmounting of shares will be handled in
another patch.
Change-Id: I3456ea089f9eb005c842aa3a43bdce6da5daafe0
Partial-Implements: bp sahara-add-shares-to-clusters
Currently only cinder volumes can be formatted and mounted.
This patch provides an ability of formatting and mounting
any attached storage devices. Also storage_paths method was
moved from node group objects to instance and corresponding
fixes in plugins was made.
Closes-bug: #1478899
Closes-bug: #1478904
Change-Id: If92d6fdea25e374d6d5c404be5ac3677cb60c057
Adding shared across tenants and protected from modifictaion
resources support. Implemented for clusters, cluster templates,
node group templates, data sources, job executions, jobs,
job binaries and job binary internals.
Changes:
* Added validation checks to service.validations.acl
* Changed validation schema's to support "is_public"
and "is_protected" fields
* Added validation of "is_protected" field for update(scale/cancel)
and delete methods of corresponding resources
* Extended get and list methods outputs with "is_public" resources
* Added unit tests for "is_public" and "is_protected" resources
Change-Id: I1a3cb14b8de70256e6aa27312dde341e85fc376c
Partially-Implements: blueprint shared-protected-resources
Adding new fields to support shared resources and
ability to prevent unintentional changes.
Partially-Implements: blueprint shared-protected-resources
Change-Id: I200944ac876e476ea0659d502a55e5f9b2a158ee
This change will allow data sources with urls of the form
"manila://share-id/path", similar to manila urls for job binaries.
The Sahara native url will be logged in the JobExecution, but
the true runtime url (file:///mnt/path) for manila shares will
be used in the cluster.
Partial-implements: blueprint manila-as-a-data-source
Change-Id: I0b43491decbe6cb0ec0b84314cf9b407b9e3fb4a
Adding ability to update job binary internals in order
to change their public/protected attributes later.
APIImpact
PATCH /v1.1/{tenant_id}/job-binary-internals/{job_binary_internal_id}
Partially implements: blueprint api-for-objects-update
Change-Id: I2d311ad5e06e42784bc0843eb38a9154f41be754
Job Binary objects can now be updated as long as they
are not referenced by an active job in the PENDING state.
Change-Id: I9e1559e3ab5ff07557f284ae59bb8161591b6e6c
Implements: bp edp-edit-job-binaries
Data Source objects can now be updated as long as they
are not referenced by an active job in the PENDING state.
Change-Id: I1d29ed73e635c1314cdd01fdbd6dd84645cabec5
Partial-Implements: bp edp-edit-data-sources
* Change 'type' for optional fields to list including 'null'
* Allow the default template tool to use "None" to
unset fields with replacement values when no replacement has been
specified.
* Fix cluster template update code to check for explicit setting of
the 'node_groups' field in a cluster template update
Partial-Bug: #1438077
Closes-Bug: #1439839
Change-Id: Ia180867c8e3ffb502da40f169a49dd3a54c23072
This change fixes the following issues:
* Remove _appy_defaults for cluster template fields on update
Fields that are not specified should not be given default values
on update like they are on creation. If a field is not specified,
it should be left alone
* Do not delete node groups if the cluster template update does
specify a node_group field
Closes-Bug: #1438077
Change-Id: I65dd15e02871f9c7227d2b51acedd6e06cb04950
Changes:
* Move updating provision progress to conductor;
* Remove redundant conductor ops;
* Improve all updates operations.
This change would improve complexity of all event log operations
and decrease number of updates of provisioning steps.
Also we remove unneeded fields in base, because we can calculate
it very easy from UI.
partially implements: bp event-log
Change-Id: I63c593d5ba27879edc2142a1e2ab310628ee045c
* 'is_default' field (default False) prevents update or deletion of templates
* 'ignore_default' flag (default False) on database operations defeats
the 'is_default' field on a template
Partial-implements: blueprint default-templates
Change-Id: I973df823981ca1a92d0dd116e32e1eacd79c56a7
API was extended and db schema was changed to support
instance locality
partially implements bp: volume-instance-locality
Change-Id: I245ebe7d7704013e8d11afc5893d46d7d9c74467
Changes to allow the updating of existing cluster templates.
Change-Id: I040fdcad6e3bd5ba8e8f841e0de554d4c296ec3f
Partial-Implements: bp support-template-editing
Jeremy Freudberg