Commit Graph

266 Commits

Author SHA1 Message Date
guolimin.lc a806a536b6 Remove redundant "=" to imporve doc format
Change-Id: Ia24a706d93fa0d717978b127d53135e225f05da3
2023-08-09 13:48:53 +08:00
Takashi Kajinami d9693fee50 Enable healthcheck middleware by default
Healthcheck middleware provides the /healthcheck endpoint which is
useful for loadbalancer or monitoring tools to check health of api
services. This change enables the middleware by default.

Note that this endpoint has no authentication. The /healthcheck entry
should be removed from api-paste.ini to disable the middleware again.

Change-Id: I1bc61afff2851bb8efac3ec5101a5198f204f41e
2021-08-21 11:04:23 +09:00
Jeremy Freudberg 865858460c Remove some files not worth maintaining
Change-Id: I4d7fc4b212a42c29839e1c6915ec6aab3ee4ed15
2019-06-20 09:59:19 -04:00
Luigi Toscano 8c941ed187 Replace git.openstack.org URLs with opendev.org URLs
Thorough replacement of git.openstack.org URLs with their opendev.org
counterparts.

Depends-On: https://review.opendev.org/653988
Change-Id: I6c7fffc1a32cb35ff7fe2ba964510b4f32805842
2019-04-20 14:55:03 +02:00
Jeremy Freudberg b1503b36c4 Tweak Sahara to make version discovery easier
Or at least, tweak Sahara to be more accomodating to keystoneauth's way
of doing version discovery.

* Don't require auth to do version discovery
* Make project ID in URL optional for APIv1 (and also for APIv2, but
  don't go around advertising that... *wink*)

Change-Id: Idb6f734aee26cab5bd629963a66ba01c92760864
Closes-Bug: #1744350
2018-01-25 05:56:15 +00:00
Jeremy Freudberg 991a3995e2 [APIv2]Enable APIv2, experimentally
Pretty much all the changes from v1->v2 regarding REST semantics,
payload changes, etc have been completed.

So let's enable APIv2 in an experimental state. It will be needed for
exposing some new Sahara features like force-delete, decomission of
a specific node, and hopefully more.

bp v2-api-experimental-impl

Depends-On: I8397ed7c134f0742de8c38466ed3f9035d8103d4
Depends-On: I102fb2ad16d0256e3a9aa364586332a13826cc90
Change-Id: I6ee89d52ab6679c34f0a089df4b111796922e171
2017-12-14 17:33:07 +00:00
zhangxuanyuan eaaa239240 Remove use_neutron from config
Nova network has been fully removed, remove use_neutron config
and related codes.

Change-Id: Ib9d87dd339d637b69fb27315d92228cbc523c8eb
Closes-Bug: #1693891
Implements: bp sahara-remove-nova-network
2017-11-24 23:27:33 +08:00
Telles Nobrega 5d208ed486 Policy in code for Sahara
This patch is part of the community goal to move policies into code.

https://governance.openstack.org/tc/goals/queens/policy-in-code.html

Change-Id: I8f228ca40acc589b9aa41fc1e80a1c0308f179cf
2017-10-23 11:46:31 -03:00
ChangBo Guo(gcb) e4d57aa863 Remove doc about config option verbose
oslo.log deprecate config option verbose since aug 1, 2015 in review: in
https://review.openstack.org/#/c/206437/, and will remove it.

sahara doesn't use verbose but includes documents about config option
verbose, need clean up.

Change-Id: I3bf0cca4513ea0904c461df4f3089900ab2d8715
2017-02-14 20:37:16 +08:00
Shu Yingya 5fdba5c322 [APIv2] Update registry images tagging
The endpoint of image tag should be changed to follow guidelines
on tag[*]. Main ideas are:
1. To add, remove, or change tags, a PUT request should be sent to
   the root tags URL, the server responds with a 200 status code.
2. To delete the entire tag list associated with a resource, a
   DELETE request must be sent to the root tags URL.
3. the resource identified by URL must expose its tags with root
   URL like: http://example.com:8386/servers/1234567890/tags.
4. There also add a get tags method and a dozen of unit test

*: http://specs.openstack.org/openstack/api-wg/guidelines/tags.html

Partial-Implements: bp v2-api-experimental-impl
Change-Id: Ia70322538f966ad032ae804c99f6f07463461e7d
2017-01-23 10:24:31 +00:00
Luigi Toscano 176b480f53 Switch use_neutron=true by default
The old nova-network was deprecated in Newton and, according nova
release notes, no longer supported for regular deployment in Ocata.

Change the unit tests with the assumption that neutron is the default.

Change-Id: I67b23fe8797425bf47e4b2370fb82885b74318ae
2016-12-09 14:28:53 +01:00
pangliye ac0413aa8c Updating list of plugins in config sample
The list of plugins in config sample was updated to use value
by default.

Change-Id: Ibe505c8008fe74dda8e632745127c4bfeb126352
2016-11-29 12:22:26 +00:00
Jenkins 6783a33c7b Merge "Use http_proxy_to_wsgi middleware" 2016-11-01 14:07:39 +00:00
melissaml 591c02fdfc Fix a typo in rootwrap.conf
TrivialFix

Change-Id: I29fbfd84eb64ea2d304fd95f3cd31dcacc2a9ff4
2016-10-20 00:10:15 +08:00
Jeremy Liu c0f43c2c5f Use http_proxy_to_wsgi middleware
This sets up the HTTPProxyToWSGI middleware in front of Sahara.
The purpose of this middleware is to set up the request URL
correctly in case there is a proxy (For instance, a loadbalancer
such as HAProxy) in front of Sahara.

The HTTPProxyToWSGI is off by default and needs to be enabled
via a configuration value.

Change-Id: Ica7e8671e3880c0db90d382bec89b0994f75b36d
Closes-bug: #1590608
2016-10-16 23:25:29 +08:00
Vitaly Gridnev ce5b034d96 plugins:patch is now admin only operation
and also enabling all important plugins by default. there
is no longer sense to keep them outside of enabled stuff.

Change-Id: I98354232a467d72b8a3713697dac5c76db081862
bp: plugin-management-api
2016-08-16 16:57:08 +03:00
Michael Ionkin d9a2f00d2d Designate integration
This patch implements designate integration. Added:
* 'use_designate' and 'nameservers' configs
* 'domain_name' column to Cluster and Cluster Templates and
  'dns_hostname' to Instance in DB
* get_management_ip() method for Instance for properly setup
  hostnames in cluster info and use it in ambari, cdh, mapr, spark,
  storm and vanilla plugins
* method for change /etc/resolv.conf on all cluster instances
  according to 'nameservers' config
* method for generate heat template for designate A and PTR records
* validation check for cluster template creation because now it
  contains new field - 'domain_name'
* fix unit tests appropriatly and new tests added
* updated heat_template_version to '2016-04-08' (because we use
  'str_split' method)

NOTE: in spec we proposed to use two domains: one for internal
resolution and another for external. But we decided to use only one
domain for both sides so an instance will have one hostname in the
domain and two appropriate A records because it's more convenient.

bp: designate-integration
Change-Id: I66525c0305450d8ba08840be1c95bfe79f21fff7
2016-08-02 18:53:25 +03:00
Luigi Toscano d9acf6eb05 Fix doc about scenario and Tempest tests
- properly reference sahara-tests;
- change a bit the wording of Tempest tests (they are not all in the
  Tempest repository);
- use the official git.openstack.org URL.

Change-Id: I59a5c9190da2c05f8b1122cae184303fb46b6e64
2016-04-22 12:58:38 +02:00
Michael Krotscheck 8b4b3abdaf Moved CORS middleware configuration into oslo-config-generator
The default values needed for sahara's implementation of cors
middleware have been moved from paste.ini into the configuration
hooks provided by oslo.config. Furthermore, these values have been
added to the default initialization procedure. This ensures
that if a value remains unset in the configuration file, it will
fallback to using sane defaults. It also ensures that an operator
modifying the configuration will be presented with that same
set of defaults.

Change-Id: Iedcaa002ff3d40cf61168769bc3946f8c6e42b87
Closes-Bug: 1551836
2016-03-03 14:01:09 -08:00
Michael McCune a9137addb4 add initial v2 api
this change implements the initial v2 experimental api. it is
implemented as an optional series of class that can be configured
through the paste deploy configuration file.

* add wsgi auth validator
* add middleware router
* add v2 endpoints
* add v2 flask blueprint object
* add optional paste filter and composite
* add developer docs for v2 api

Change-Id: I74627c0879851b354b5043f8a6ff91bae8438bb1
Partial-Implements: bp v2-api-experimental-impl
2016-02-10 09:56:07 -05:00
Evgeny Sikachev 7a29eda040 Remove scenario tests and related files
Removed etc/scenario,
        etc/edp-examples,
        sahara/tests/scenario,
        sahara/tests/scenario_unit,
        tools/gate
From tox.ini removed envs scenario, py27-scenario-unit
bp: move-scenario-tests-to-separate-repo

Change-Id: I36917d56a2ea233213f2913d17589bdc66fd4fec
2016-01-22 14:04:41 +03:00
Michael Krotscheck 030705eb04 Added Keystone and RequestID headers to CORS middleware
CORS middleware's latent configuration feature, new in 3.0.0,
allows adding headers that apply to all valid origins.
This patch adds headers commonly used in openstack to sahara's paste
pipeline, so that operators do not have to be aware of additional
configuration magic to ensure that browsers can talk to the API.

Change-Id: Ibf89e69f6e9a747043c6d79e20805f66a9c7aefa
2016-01-08 11:30:17 -08:00
Evgeny Sikachev a1c3dad505 Update HA scenario for CDH
Fix HA scenario for CDH

Change-Id: I231369c6132fd5638ae0a098fd5dbd14263b9ba9
2015-12-28 07:26:39 +00:00
Vitaly Gridnev e9a7a3858a Implement custom check for Kafka Service
This change implements custom check for Kafka Service.
This creates test topic, several messages are sending.

Change-Id: If6013ecc6a173b99ced68722775fbe30702943c5
2015-12-17 10:31:49 +00:00
Luigi Toscano ad45ef623b Scenario templates: make is_proxy_gateway configurable
is_proxy_gateway is part of the environmental settings, so it should
be possible to override it from the scenario file. Nevertheless,
keep the default value as the current one, so that the existing jobs
work without changes.

Change-Id: I4cc20154c9ee0b126a1ea21ee72090f76cf6b6e1
2015-12-11 15:34:12 +01:00
Vitaly Gridnev b194572aa5 Support of Spark EDP in Ambari plugin
Start support Spark EDP jobs in Ambari plugin.
Also added sparkPi job to test this support.

Current issues:
 * Spark with Swift doesn't work

Change-Id: I2374d387054efa20876fbcee46ede038e0f3d520
Partially-implements-blueprint: hdp-22-support
2015-11-26 12:44:57 +03:00
luhuichun 06dc55f5b9 Plugin version error in scenario test for vanilla2.6.0
As 2.6.0 is deprecated, this file can be removed.

Change-Id: Id7b238978ae63b1fc95da8c34455784091ec9e66
Closes-bug: 1514322
2015-11-10 02:19:10 +08:00
Jenkins fd57a02a10 Merge "Add CDH test enabling HDFS HA" 2015-11-05 21:19:45 +00:00
Ken Chen a937f5b891 Add CDH test enabling HDFS HA
We add three HDFS_JOURNALNODE roles in the cluster template to
enable HDFS NameNode HA. We did not directly modified in
cdh-5.4.0.yaml.mako file, or it makes CI test too long and may
exceed the timeout.

Partial-implements: blueprint cdh-ha-support
Change-Id: I173027c0511ee5e7f76ea485b13089a3a5839ece
2015-11-04 11:00:25 +08:00
Vitaly Gridnev b5d2f87ce1 Disable security for Oozie in Ambari
Temporary disable security for Oozie in Ambari plugin.
That can be returned again when Sahara Oozie client will
be able to authenticate.

Partially-implements-blueprint: hdp-22-support
Change-Id: I661e2cf55243c661103c807e5fba7ad74978f215
2015-11-01 15:25:00 +00:00
Evgeny Sikachev 60632be680 Fix Mapr on ci
1. Flavor for Mapr master is small and CLDB
cannot start. Need large flavor.
2. Add testing the use of ephemeral drive.
3. Disable check of EDP on Mapr, because of scenario tests
cannot write to MaprFS

Change-Id: I44537773517bf98fc83a54947efe02663212eb2d
2015-10-23 14:48:27 +00:00
Jenkins 6d13dbf4d5 Merge "Hive job type support on CI" 2015-10-21 13:54:03 +00:00
Michael Krotscheck 4e96378f33 Switched CORS configuration to use oslo_config
When cors middleware was moved into paste.ini, genconfig was not
updated to indicate that the configuration method had changed.
This breaks backwards-compatibility, however since the middleware
supports automatic oslo config detection via pastedeploy, it is
easily corrected.

This patch updates the api-paste.ini file to encourage CORS
middleware to draw its configuration from sahara.conf. It also
moves cors to the beginning of the middleware pipeline.

Change-Id: I4cff577aeff21286d0f9ce13142babec39db5e13
2015-10-20 11:02:58 -07:00
Vitaly Gridnev 357059e40b Hive job type support on CI
Support hive job type on Sahara CI. Also using
datasources placeholders in output url for correct
work of job.

Change-Id: I1177a7d12e380b6eb032f6e6ba025aa86451d846
2015-10-17 12:53:40 +00:00
Jenkins 8910d7d826 Merge "Use api-paste.ini for loading middleware" 2015-10-16 17:20:06 +00:00
Vitaly Gridnev 08b9396676 Reformat job flows
Scenario framework started support of several edp job
flows, so it's better to use explicit jobs in plugins
flows.

Change-Id: If0f6a182fbf34ed7b2283f2d8bbf61b32064d4bf
2015-10-14 09:57:19 -04:00
Sergey Reshetnyak 64583e4b83 Use api-paste.ini for loading middleware
Changes:
* use api-paste.ini config for loading middleware
* refactoring middlewares to support loading via pastedeploy
* use debug middleware from oslo_middleware library instead log_exchange
  middleware

Closes-bug: #1503983
Closes-bug: #1361360

Change-Id: I444c1799ef53dbb19a601e51dd95cd8509fb1c0c
2015-10-13 16:04:22 +03:00
Jenkins f8e6907299 Merge "Add put data in HDFS for EDP testcase" 2015-10-06 17:20:05 +00:00
Javier Pena ee6d3fd032 Add /usr/share/sahara/rootwrap to filters_path
All other OpenStack services using rootwrap filters include
/usr/share/${project}/rootwrap in filters_path, and consistence is
helpful for distributions.

Change-Id: I19fa431283a1f42778a4ab5a0794e18f6b96fb02
2015-10-02 12:19:29 +02:00
Vitaly Gridnev cb3b6fcb82 Cleanup databases during execution of hive example
We should drop databases and tables created in hive
examples to prevent already exists errors.

Closes-bug: 1413602

Change-Id: I155e4cbfc437b5753d7c7ea7fcac74d2592f3618
2015-09-25 11:17:18 +00:00
Sergey Reshetnyak a4fb58043f Add put data in HDFS for EDP testcase
Now you can use hdfs as input datasource method

Change-Id: I08c179e3ffb60306c90178080b45e6443d03ecb6
2015-09-22 15:37:01 +03:00
Jenkins 5deababe20 Merge "Adding check of indirect access" 2015-09-18 15:29:02 +00:00
Evgeny Sikachev 3b89217a8b Adding check of indirect access
If in scenario file set is_proxy_gateway: true, need
check of indirect access

Change-Id: Icf779a9e8cd1145de68f5341720d32c6b0d1ef55
2015-09-16 17:40:22 +03:00
Vitaly Gridnev 2b25dfa07f Create ResourceGroup with volumes only if it is required
If volumes_per_node or volumes_size is 0, there is no sense
in creating empty ResourceGroup.

Also remove volumes from one node group in gate to be sure
that sahara correctly handle such case.

Closes-bug: 1494650
Change-Id: Ic592710773b9338f5857e431b0567bc5c87af2bd
2015-09-15 11:42:55 +00:00
Jenkins 65212a92fd Merge "Drop HDP 1.3.2 plugin" 2015-09-14 18:21:03 +00:00
Jenkins 3b37989912 Merge "Drop Vanilla Hadoop 1" 2015-09-14 18:20:57 +00:00
Vitaly Gridnev 4824ca3dc6 [CDH] Fix problem with launching Spark jobs
Sahara EDP should select SPARK_YARN_HISTORY_SERVER node
as the master node for running Spark jobs.
Also uploaded spark job to cdh_flow on CI.

Closes-bug: 1490012
Change-Id: I09a745b8c84e6cca2fb2924e85d845aafc8e1134
2015-09-05 01:24:18 +03:00
Vitaly Gridnev 27e428f9bf Drop HDP 1.3.2 plugin
Drop HDP 1.3.2 with all related to that plugin unit tests,
scenario tests.

Implements blueprint: drop-hadoop-1

Change-Id: I25c5fc046e78d347c8bfe059129f4dcd14511c99
2015-09-03 17:29:20 +03:00
Nikita Konovalov 6b5f0d0b1b Drop Vanilla Hadoop 1
Dropping support of unpopular and unused Hadoop v1 in Vanilla plugin.

Partially-implements bp: drop-hadoop-1

Change-Id: I0a322fb7b8db50941c4854f45077fe6232e2c766
2015-09-03 16:02:29 +03:00
Jenkins 12ccb88434 Merge "Add EDP services to new HDP plugin" 2015-09-01 20:42:13 +00:00