APIv2 - api-ref documentation for APIv2

Writing api-ref for APIv2.

Story: #2002102
Task: #19780

Change-Id: Ib9e855c11f03239e70306d12e96194549d2dc0f3
This commit is contained in:
Telles Nobrega 2018-12-10 07:52:57 -03:00 committed by Telles Nobrega
parent 9aa67c6206
commit 0b8002a99d
81 changed files with 6669 additions and 5 deletions

View File

@ -10,3 +10,4 @@ API content can be searched using the :ref:`search`.
:maxdepth: 2
v1.1/index
v2/index

View File

@ -1,8 +1,8 @@
:tocdepth: 2
:tocdepth: 3
========================
------------------------
Data Processing API v1.1
========================
------------------------
.. rest_expand_all::

View File

@ -0,0 +1,241 @@
.. -*- rst -*-
=================
Cluster templates
=================
A cluster template configures a cluster. A cluster template
lists node groups with the number of instances in each group. You
can also define cluster-scoped configurations in a cluster
template.
Show cluster template details
=============================
.. rest_method:: GET /v2/cluster-templates/{cluster_template_id}
Shows details for a cluster template.
Normal response codes: 200
Request
-------
.. rest_parameters:: parameters.yaml
- cluster_template_id: url_cluster_template_id
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- description: cluster_template_description
- use_autoconfig: use_autoconfig
- cluster_configs: cluster_configs
- created_at: created_at
- default_image_id: default_image_id
- updated_at: updated_at
- plugin_name: plugin_name
- is_default: is_default
- is_protected: object_is_protected
- shares: object_shares
- domain_name: domain_name
- project_id: project_id
- node_groups: node_groups
- is_public: object_is_public
- plugin_version: plugin_version
- id: cluster_template_id
- name: cluster_template_name
Response Example
----------------
.. literalinclude:: samples/cluster-templates/cluster-templates-list-response.json
:language: javascript
Update cluster templates
========================
.. rest_method:: PATCH /v2/cluster-templates/{cluster_template_id}
Updates a cluster template.
Normal response codes:202
Request
-------
.. rest_parameters:: parameters.yaml
- cluster_template_id: cluster_template_id
Request Example
---------------
.. literalinclude:: samples/cluster-templates/cluster-template-update-request.json
:language: javascript
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- description: cluster_template_description
- use_autoconfig: use_autoconfig
- cluster_configs: cluster_configs
- created_at: created_at
- default_image_id: default_image_id
- updated_at: updated_at
- plugin_name: plugin_name
- is_default: is_default
- is_protected: object_is_protected
- shares: object_shares
- domain_name: domain_name
- project_id: project_id
- node_groups: node_groups
- is_public: object_is_public
- plugin_version: plugin_version
- id: cluster_template_id
- name: cluster_template_name
Delete cluster template
=======================
.. rest_method:: DELETE /v2/cluster-templates/{cluster_template_id}
Deletes a cluster template.
Normal response codes:204
Request
-------
.. rest_parameters:: parameters.yaml
- cluster_template_id: cluster_template_id
List cluster templates
======================
.. rest_method:: GET /v2/cluster-templates
Lists available cluster templates.
Normal response codes: 200
Request
-------
.. rest_parameters:: parameters.yaml
- limit: limit
- marker: marker
- sort_by: sort_by_cluster_templates
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- markers: markers
- prev: prev
- next: next
- description: cluster_template_description
- use_autoconfig: use_autoconfig
- cluster_configs: cluster_configs
- created_at: created_at
- default_image_id: default_image_id
- updated_at: updated_at
- plugin_name: plugin_name
- is_default: is_default
- is_protected: object_is_protected
- shares: object_shares
- domain_name: domain_name
- project_id: project_id
- node_groups: node_groups
- is_public: object_is_public
- plugin_version: plugin_version
- id: cluster_template_id
- name: cluster_template_name
Response Example
----------------
.. rest_method:: GET /v2/cluster-templates?limit=2
.. literalinclude:: samples/cluster-templates/cluster-templates-list-response.json
:language: javascript
Create cluster templates
========================
.. rest_method:: POST /v2/cluster-templates
Creates a cluster template.
Normal response codes:202
Request Example
---------------
.. literalinclude:: samples/cluster-templates/cluster-template-create-request.json
:language: javascript
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- description: cluster_template_description
- use_autoconfig: use_autoconfig
- cluster_configs: cluster_configs
- created_at: created_at
- default_image_id: default_image_id
- updated_at: updated_at
- plugin_name: plugin_name
- is_default: is_default
- is_protected: object_is_protected
- shares: object_shares
- domain_name: domain_name
- project_id: project_id
- node_groups: node_groups
- is_public: object_is_public
- plugin_version: plugin_version
- id: cluster_template_id
- name: cluster_template_name

View File

@ -0,0 +1,293 @@
.. -*- rst -*-
========
Clusters
========
A cluster is a group of nodes with the same configuration.
List available clusters
=======================
.. rest_method:: GET /v2/clusters
Lists available clusters.
Normal response codes: 200
Request
-------
.. rest_parameters:: parameters.yaml
- limit: limit
- marker: marker
- sort_by: sort_by_clusters
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- markers: markers
- prev: prev
- next: next
- count: count
- info: info
- cluster_template_id: cluster_template_id
- is_transient: is_transient
- provision_progress: provision_progress
- status: status
- neutron_management_network: neutron_management_network
- clusters: clusters
- management_public_key: management_public_key
- status_description: status_description
- trust_id: trust_id
- domain_name: domain_name
Response Example
----------------
.. rest_method:: GET /v2/clusters
.. literalinclude:: samples/clusters/clusters-list-response.json
:language: javascript
Create cluster
==============
.. rest_method:: POST /v2/clusters
Creates a cluster.
Normal response codes: 202
Request Example
---------------
.. literalinclude:: samples/clusters/cluster-create-request.json
:language: javascript
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- count: count
- info: info
- cluster_template_id: cluster_template_id
- is_transient: is_transient
- provision_progress: provision_progress
- status: status
- neutron_management_network: neutron_management_network
- management_public_key: management_public_key
- status_description: status_description
- trust_id: trust_id
- domain_name: domain_name
Show details of a cluster
=========================
.. rest_method:: GET /v2/clusters/{cluster_id}
Shows details for a cluster, by ID.
Normal response codes: 200
Request
-------
.. rest_parameters:: parameters.yaml
- cluster_id: url_cluster_id
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- count: count
- info: info
- cluster_template_id: cluster_template_id
- is_transient: is_transient
- provision_progress: provision_progress
- status: status
- neutron_management_network: neutron_management_network
- management_public_key: management_public_key
- status_description: status_description
- trust_id: trust_id
- domain_name: domain_name
Response Example
----------------
.. literalinclude:: samples/clusters/cluster-show-response.json
:language: javascript
Delete a cluster
================
.. rest_method:: DELETE /v2/clusters/{cluster_id}
Deletes a cluster.
Normal response codes: 204 or 200
Request
-------
.. rest_parameters:: parameters.yaml
- cluster_id: url_cluster_id
- force: force
Scale cluster
=============
.. rest_method:: PUT /v2/clusters/{cluster_id}
Scales a cluster.
Normal response codes: 202
Request
-------
.. rest_parameters:: parameters.yaml
- cluster_id: cluster_id
Request Example
---------------
.. literalinclude:: samples/clusters/cluster-scale-request.json
:language: javascript
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- count: count
- info: info
- cluster_template_id: cluster_template_id
- is_transient: is_transient
- provision_progress: provision_progress
- status: status
- neutron_management_network: neutron_management_network
- management_public_key: management_public_key
- status_description: status_description
- trust_id: trust_id
- domain_name: domain_name
Update cluster
==============
.. rest_method:: PATCH /v2/clusters/{cluster_id}
Updates a cluster.
Normal response codes: 202
Request
-------
.. rest_parameters:: parameters.yaml
- cluster_id: url_cluster_id
Request Example
---------------
.. literalinclude:: samples/clusters/cluster-update-request.json
:language: javascript
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- count: count
- info: info
- cluster_template_id: cluster_template_id
- is_transient: is_transient
- provision_progress: provision_progress
- status: status
- neutron_management_network: neutron_management_network
- management_public_key: management_public_key
- status_description: status_description
- trust_id: trust_id
- domain_name: domain_name
Show progress
=============
.. rest_method:: GET /v2/clusters/{cluster_id}
Shows provisioning progress for a cluster.
Normal response codes: 200
Request
-------
.. rest_parameters:: parameters.yaml
- cluster_id: url_cluster_id
Response Example
----------------
.. literalinclude:: samples/event-log/cluster-progress-response.json
:language: javascript

View File

@ -0,0 +1,202 @@
.. -*- rst -*-
============
Data sources
============
A data source object defines the location of input or output for
MapReduce jobs and might reference different types of storage.
The Data Processing service does not validate data source
locations.
Show data source details
========================
.. rest_method:: GET /v2/data-sources/{data_source_id}
Shows details for a data source.
Normal response codes: 200
Request
-------
.. rest_parameters:: parameters.yaml
- data_source_id: url_data_source_id
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- description: data_source_description
- url: url
- project_id: project_id
- created_at: created_at
- updated_at: updated_at
- is_protected: object_is_protected
- is_public: object_is_public
- type: type
- id: data_source_id
- name: data_source_name
Response Example
----------------
.. literalinclude:: samples/data-sources/data-source-show-response.json
:language: javascript
Delete data source
==================
.. rest_method:: DELETE /v2/data-sources/{data_source_id}
Deletes a data source.
Normal response codes:204
Request
-------
.. rest_parameters:: parameters.yaml
- data_source_id: url_data_source_id
Update data source
==================
.. rest_method:: PATCH /v2/data-sources/{data_source_id}
Updates a data source.
Normal response codes:202
Request
-------
.. rest_parameters:: parameters.yaml
- data_source_id: url_data_source_id
Request Example
---------------
.. literalinclude:: samples/data-sources/data-source-update-request.json
:language: javascript
List data sources
=================
.. rest_method:: GET /v2/data-sources
Lists all data sources.
Normal response codes: 200
Request
-------
.. rest_parameters:: parameters.yaml
- limit: limit
- marker: marker
- sort_by: sort_by_data_sources
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- markers: markers
- prev: prev
- next: next
- description: data_source_description
- url: url
- project_id: project_id
- created_at: created_at
- updated_at: updated_at
- is_protected: object_is_protected
- is_public: object_is_public
- type: type
- id: data_source_id
- name: data_source_name
Response Example
----------------
.. rest_method:: GET /v2/data-sourses?sort_by=-name
.. literalinclude:: samples/data-sources/data-sources-list-response.json
:language: javascript
Create data source
==================
.. rest_method:: POST /v2/data-sources
Creates a data source.
Normal response codes:202
Request Example
---------------
.. literalinclude:: samples/data-sources/data-source-register-hdfs-request.json
:language: javascript
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- description: data_source_description
- url: url
- project_id: project_id
- created_at: created_at
- updated_at: updated_at
- is_protected: object_is_protected
- is_public: object_is_public
- type: type
- id: data_source_id
- name: data_source_name

View File

@ -0,0 +1,42 @@
.. -*- rst -*-
=========
Event log
=========
The event log feature provides information about cluster
provisioning. In the event of errors, the event log shows the
reason for the failure.
Show progress
=============
.. rest_method:: GET /v2/clusters/{cluster_id}
Shows provisioning progress of cluster.
Normal response codes: 200
Error response codes:
Request
-------
.. rest_parameters:: parameters.yaml
- cluster_id: cluster_id
Response Example
----------------
.. literalinclude:: samples/event-log/cluster-progress-response.json
:language: javascript

View File

@ -0,0 +1,244 @@
.. -*- rst -*-
==============
Image registry
==============
Use the image registry tool to manage images, add tags to and
remove tags from images, and define the user name for an instance
operating system. Each plugin lists required tags for an image. To
run remote operations, the Data Processing service requires a user
name with which to log in to the operating system for an instance.
Add tags to image
=================
.. rest_method:: PUT /v2/images/{image_id}/tags
Adds tags to an image.
Normal response codes:202
Request
-------
.. rest_parameters:: parameters.yaml
- tags: tags
- image_id: url_image_id
Request Example
---------------
.. literalinclude:: samples/image-registry/image-tags-add-request.json
:language: javascript
Show image details
==================
.. rest_method:: GET /v2/images/{image_id}
Shows details for an image.
Normal response codes: 200
Request
-------
.. rest_parameters:: parameters.yaml
- image_id: url_image_id
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- status: status
- username: username
- updated: updated
- description: image_description
- created: created
- image: image
- tags: tags
- minDisk: minDisk
- name: image_name
- progress: progress
- minRam: minRam
- id: image_id
- metadata: metadata
Response Example
----------------
.. literalinclude:: samples/image-registry/image-show-response.json
:language: javascript
Register image
==============
.. rest_method:: POST /v2/images/{image_id}
Registers an image in the registry.
Normal response codes:202
Request
-------
.. rest_parameters:: parameters.yaml
- username: username
- description: image_description
- image_id: url_image_id
Request Example
---------------
.. literalinclude:: samples/image-registry/image-register-request.json
:language: javascript
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- status: status
- username: username
- updated: updated
- description: image_description
- created: created
- image: image
- tags: tags
- minDisk: minDisk
- name: image_name
- progress: progress
- minRam: minRam
- id: image_id
- metadata: metadata
Unregister image
================
.. rest_method:: DELETE /v2/images/{image_id}
Removes an image from the registry.
Normal response codes:204
Request
-------
.. rest_parameters:: parameters.yaml
- image_id: url_image_id
Remove tags from image
======================
.. rest_method:: DELETE /v2/images/{image_id}/tag
Removes tags from an image.
Normal response codes:202
Request
-------
.. rest_parameters:: parameters.yaml
- tags: tags
- image_id: url_image_id
Request Example
---------------
.. literalinclude:: samples/image-registry/image-tags-delete-request.json
:language: javascript
List images
===========
.. rest_method:: GET /v2/images
Lists all images registered in the registry.
Normal response codes: 200
Request
-------
.. rest_parameters:: parameters.yaml
- tags: tags
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- status: status
- username: username
- updated: updated
- description: image_description
- created: created
- image: image
- tags: tags
- minDisk: minDisk
- name: image_name
- images: images
- progress: progress
- minRam: minRam
- id: image_id
- metadata: metadata
Response Example
----------------
.. literalinclude:: samples/image-registry/images-list-response.json
:language: javascript

View File

@ -0,0 +1,20 @@
:tocdepth: 3
----------------------
Data Processing API v2
----------------------
.. rest_expand_all::
.. include:: cluster-templates.inc
.. include:: clusters.inc
.. include:: data-sources.inc
.. include:: event-log.inc
.. include:: image-registry.inc
.. include:: job-binaries.inc
.. include:: job-templates.inc
.. include:: job-types.inc
.. include:: jobs.inc
.. include:: node-group-templates.inc
.. include:: plugins.inc

View File

@ -0,0 +1,256 @@
.. -*- rst -*-
============
Job binaries
============
Job binary objects represent data processing applications and
libraries that are stored in Object Storage service(S3 or Swift) or
in Manila Shares.
List job binaries
=================
.. rest_method:: GET /v2/job-binaries
Lists the available job binaries.
Normal response codes: 200
Request
-------
.. rest_parameters:: parameters.yaml
- limit: limit
- marker: marker
- sort_by: sort_by_job_binary
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- markers: markers
- prev: prev
- next: next
- description: job_binary_description
- url: url
- project_id: project_id
- created_at: created_at
- updated_at: updated_at
- is_protected: object_is_protected
- is_public: object_is_public
- binaries: binaries
- id: job_binary_id
- name: job_binary_name
Response Example
----------------
.. rest_method:: GET /v2/job-binaries?sort_by=created_at
.. literalinclude:: samples/job-binaries/list-response.json
:language: javascript
Create job binary
=================
.. rest_method:: POST /v2/job-binaries
Creates a job binary.
Normal response codes:202
Request Example
---------------
.. literalinclude:: samples/job-binaries/create-request.json
:language: javascript
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- description: job_binary_description
- url: url
- project_id: project_id
- created_at: created_at
- updated_at: updated_at
- is_protected: object_is_protected
- is_public: object_is_public
- id: job_binary_id
- name: job_binary_name
Show job binary details
=======================
.. rest_method:: GET /v2/job-binaries/{job_binary_id}
Shows details for a job binary.
Normal response codes: 200
Request
-------
.. rest_parameters:: parameters.yaml
- job_binary_id: url_job_binary_id
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- description: job_binary_description
- url: url
- project_id: project_id
- created_at: created_at
- updated_at: updated_at
- is_protected: object_is_protected
- is_public: object_is_public
- id: job_binary_id
- name: job_binary_name
Response Example
----------------
.. literalinclude:: samples/job-binaries/show-response.json
:language: javascript
Delete job binary
=================
.. rest_method:: DELETE /v2/job-binaries/{job_binary_id}
Deletes a job binary.
Normal response codes:204
Request
-------
.. rest_parameters:: parameters.yaml
- job_binary_id: url_job_binary_id
Update job binary
=================
.. rest_method:: PATCH /v2/job-binaries/{job_binary_id}
Updates a job binary.
Normal response codes:202
Request
-------
.. rest_parameters:: parameters.yaml
- job_binary_id: url_job_binary_id
Request Example
---------------
.. literalinclude:: samples/job-binaries/update-request.json
:language: javascript
Show job binary data
====================
.. rest_method:: GET /v2/job-binaries/{job_binary_id}/data
Shows data for a job binary.
The response body shows the job binary raw data and the response
headers show the data length.
Example response:
::
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 161
Content-Type: text/html; charset=utf-8
Date: Sat, 28 Mar 2016 02:42:48 GMT
A = load '$INPUT' using PigStorage(':') as (fruit: chararray);
B = foreach A generate com.hadoopbook.pig.Trim(fruit);
store B into '$OUTPUT' USING PigStorage();
Normal response codes: 200
Request
-------
.. rest_parameters:: parameters.yaml
- job_binary_id: url_job_binary_id
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- Content-Length: Content-Length
Response Example
----------------
.. literalinclude:: samples/job-binaries/show-data-response
:language: text

View File

@ -0,0 +1,257 @@
.. -*- rst -*-
=============
Job templates
=============
A job templates object lists the binaries that a job needs to run.
To run a job, you must specify data sources and job parameters.
You can run a job on an existing or new transient cluster.
List job templates
==================
.. rest_method:: GET /v2/job-templates
Lists all job templates.
Normal response codes: 200
Request
-------
.. rest_parameters:: parameters.yaml
- limit: limit
- marker: marker
- sort_by: sort_by_job_templates
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- job_templates: job_templates
- description: job_description
- project_id: project_id
- created_at: created_at
- mains: mains
- updated_at: updated_at
- libs: libs
- is_protected: object_is_protected
- interface: interface
- is_public: object_is_public
- type: type
- id: job_template_id
- name: job_template_name
- markers: markers
- prev: prev
- next: next
Response Example
----------------
..rest_method:: GET /v2/job-templates?limit=2
.. literalinclude:: samples/job-templates/job-templates-list-response.json
:language: javascript
Create job template
===================
.. rest_method:: POST /v2/job-templates
Creates a job object.
Normal response codes:202
Request Example
---------------
.. literalinclude:: samples/job-templates/job-template-create-request.json
:language: javascript
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- description: job_description
- project_id: project_id
- created_at: created_at
- mains: mains
- updated_at: updated_at
- libs: libs
- is_protected: object_is_protected
- interface: interface
- is_public: object_is_public
- type: type
- id: job_template_id
- name: job_template_name
Show job template details
=========================
.. rest_method:: GET /v2/job-templates/{job_template_id}
Shows details for a job template.
Normal response codes: 200
Request
-------
.. rest_parameters:: parameters.yaml
- job_template_id: url_job_template_id
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- description: job_description
- project_id: project_id
- created_at: created_at
- mains: mains
- updated_at: updated_at
- libs: libs
- is_protected: object_is_protected
- interface: interface
- is_public: object_is_public
- type: type
- id: job_template_id
- name: job_template_name
Response Example
----------------
.. literalinclude:: samples/job-templates/job-template-show-response.json
:language: javascript
Remove job template
===================
.. rest_method:: DELETE /v2/job-templates/{job_template_id}
Removes a job.
Normal response codes:204
Request
-------
.. rest_parameters:: parameters.yaml
- job_template_id: url_job_template_id
Update job template object
==========================
.. rest_method:: PATCH /v2/job-templates/{job_template_id}
Updates a job template object.
Normal response codes:202
Request
-------
.. rest_parameters:: parameters.yaml
- job_template_id: url_job_template_id
Request Example
---------------
.. literalinclude:: samples/job-templates/job-template-update-request.json
:language: javascript
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- description: job_description
- project_id: project_id
- created_at: created_at
- mains: mains
- updated_at: updated_at
- libs: libs
- is_protected: object_is_protected
- interface: interface
- is_public: object_is_public
- type: type
- id: job_template_id
- name: job_template_name
Get job template config hints
=============================
.. rest_method:: GET /v2/job-templates/config-hints/{job_type}
Get job template config hints
Normal response codes:202
Request
-------
.. rest_parameters:: parameters.yaml
- job_type: url_job_type
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- job_config: job_config
- args: args
- configs: configs

View File

@ -0,0 +1,61 @@
.. -*- rst -*-
=========
Job types
=========
Each plugin that supports EDP also supports specific job types.
Different versions of a plugin might actually support different job
types. Configuration options vary by plugin, version, and job type.
The job types provide information about which plugins support which
job types and how to configure the job types.
List job types
==============
.. rest_method:: GET /v2/job-types
Lists all job types.
You can use query parameters to filter the response.
Normal response codes: 200
Error response codes:
Request
-------
.. rest_parameters:: parameters.yaml
- plugin: plugin
- version: version
- type: type
- hints: hints
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- versions: versions
- title: title
- description: description_plugin
- job_types: job_types
- name: plugin_name
Response Example
----------------
.. literalinclude:: samples/job-types/job-types-list-response.json
:language: javascript

262
api-ref/source/v2/jobs.inc Normal file
View File

@ -0,0 +1,262 @@
.. -*- rst -*-
====
Jobs
====
A job object represents a job that runs on a cluster.
A job polls the status of a running job and reports it to the user.
Execute Job
===========
.. rest_method:: POST /v2/jobs
Executes a job.
Normal response codes: 200
Request Example
----------------
.. rest_method:: /v2/jobs
.. literalinclude:: samples/jobs/job-request.json
:language: javascript
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- info: info
- output_id: output_id
- start_time: start_time
- job_template_id: job_template_id
- updated_at: updated_at
- project_id: project_id
- created_at: created_at
- args: args
- data_source_urls: data_source_urls
- return_code: return_code
- oozie_job_id: oozie_job_id
- is_protected: is_protected_3
- cluster_id: cluster_id
- end_time: end_time
- params: params
- is_public: job_is_public
- input_id: input_id
- configs: configs
- job: job
- id: job_id
Response Example
----------------
.. literalinclude:: samples/jobs/job-response.json
:language: javascript
List jobs
=========
.. rest_method:: GET /v2/jobs
Lists available jobs.
Normal response codes: 200
Request
-------
.. rest_parameters:: parameters.yaml
- limit: limit
- marker: marker
- sort_by: sort_by_job
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- markers: markers
- prev: prev
- next: next
- info: info
- output_id: output_id
- start_time: start_time
- job_template_id: job_template_id
- updated_at: updated_at
- project_id: project_id
- created_at: created_at
- args: args
- data_source_urls: data_source_urls
- return_code: return_code
- oozie_job_id: oozie_job_id
- is_protected: is_protected_3
- cluster_id: cluster_id
- end_time: end_time
- params: params
- is_public: job_is_public
- input_id: input_id
- configs: configs
- job: job
- id: job_id
- jobs: jobs
Response Example
----------------
.. rest_method:: /v2/jobs
.. literalinclude:: samples/jobs/list-response.json
:language: javascript
Show job
========
.. rest_method:: GET /v2/jobs/{job_id}
Shows details for a job, by ID.
Normal response codes: 200
Request
-------
.. rest_parameters:: parameters.yaml
- job_id: url_job_id
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- info: info
- output_id: output_id
- start_time: start_time
- job_template_id: job_template_id
- updated_at: updated_at
- project_id: project_id
- created_at: created_at
- args: args
- data_source_urls: data_source_urls
- return_code: return_code
- oozie_job_id: oozie_job_id
- is_protected: is_protected_3
- cluster_id: cluster_id
- end_time: end_time
- params: params
- is_public: job_is_public
- input_id: input_id
- configs: configs
- job: job
- id: job_id
Response Example
----------------
.. literalinclude:: samples/jobs/job-response.json
:language: javascript
Delete job
==========
.. rest_method:: DELETE /v2/jobs/{job_id}
Deletes a job.
Normal response codes:204
Request
-------
.. rest_parameters:: parameters.yaml
- job_id: url_job_id
Update job
==========
.. rest_method:: PATCH /v2/jobs/{job_id}
Updates a job.
Normal response codes:202
Request
-------
.. rest_parameters:: parameters.yaml
- job_id: url_job_id
Request Example
---------------
.. literalinclude:: samples/jobs/job-update-request.json
:language: javascript
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- info: info
- output_id: output_id
- start_time: start_time
- job_template_id: job_template_id
- updated_at: updated_at
- project_id: project_id
- created_at: created_at
- args: args
- data_source_urls: data_source_urls
- return_code: return_code
- oozie_job_id: oozie_job_id
- is_protected: is_protected_3
- cluster_id: cluster_id
- end_time: end_time
- params: params
- is_public: job_is_public
- input_id: input_id
- configs: configs
- job: job
- id: job_id

View File

@ -0,0 +1,289 @@
.. -*- rst -*-
====================
Node group templates
====================
A cluster is a group of nodes with the same configuration. A node
group template configures a node in the cluster.
A template configures Hadoop processes and VM characteristics, such
as the number of reduced slots for task tracker, the number of
CPUs, and the amount of RAM. The template specifies the VM
characteristics through an OpenStack flavor.
List node group templates
=========================
.. rest_method:: GET /v2/node-group-templates
Lists available node group templates.
Normal response codes: 200
Request
-------
.. rest_parameters:: parameters.yaml
- limit: limit
- marker: marker
- sort_by: sort_by_node_group_templates
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- markers: markers
- prev: prev
- next: next
- volume_local_to_instance: volume_local_to_instance
- availability_zone: availability_zone
- updated_at: updated_at
- use_autoconfig: use_autoconfig
- volumes_per_node: volumes_per_node
- id: node_group_template_id
- security_groups: security_groups
- shares: object_shares
- node_configs: node_configs
- auto_security_group: auto_security_group
- volumes_availability_zone: volumes_availability_zone
- description: node_group_template_description
- volume_mount_prefix: volume_mount_prefix
- plugin_name: plugin_name
- floating_ip_pool: floating_ip_pool
- is_default: is_default
- image_id: image_id
- volumes_size: volumes_size
- is_proxy_gateway: is_proxy_gateway
- is_public: object_is_public
- plugin_version: plugin_version
- name: node_group_template_name
- project_id: project_id
- created_at: created_at
- volume_type: volume_type
- is_protected: object_is_protected
- node_processes: node_processes
- flavor_id: flavor_id
Response Example
----------------
.. rest_method:: GET /v2/node-group-templates?limit=2&marker=38b4e146-1d39-4822-bad2-fef1bf304a52&sort_by=name
.. literalinclude:: samples/node-group-templates/node-group-templates-list-response.json
:language: javascript
Create node group template
==========================
.. rest_method:: POST /v2/node-group-templates
Creates a node group template.
Normal response codes: 202
Request Example
---------------
.. literalinclude:: samples/node-group-templates/node-group-template-create-request.json
:language: javascript
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- volume_local_to_instance: volume_local_to_instance
- availability_zone: availability_zone
- updated_at: updated_at
- use_autoconfig: use_autoconfig
- volumes_per_node: volumes_per_node
- id: node_group_template_id
- security_groups: security_groups
- shares: object_shares
- node_configs: node_configs
- auto_security_group: auto_security_group
- volumes_availability_zone: volumes_availability_zone
- description: node_group_template_description
- volume_mount_prefix: volume_mount_prefix
- plugin_name: plugin_name
- floating_ip_pool: floating_ip_pool
- is_default: is_default
- image_id: image_id
- volumes_size: volumes_size
- is_proxy_gateway: is_proxy_gateway
- is_public: object_is_public
- plugin_version: plugin_version
- name: node_group_template_name
- project_id: project_id
- created_at: created_at
- volume_type: volume_type
- is_protected: object_is_protected
- node_processes: node_processes
- flavor_id: flavor_id
Show node group template details
================================
.. rest_method:: GET /v2/node-group-templates/{node_group_template_id}
Shows a node group template, by ID.
Normal response codes: 200
Request
-------
.. rest_parameters:: parameters.yaml
- node_group_template_id: url_node_group_template_id
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- volume_local_to_instance: volume_local_to_instance
- availability_zone: availability_zone
- updated_at: updated_at
- use_autoconfig: use_autoconfig
- volumes_per_node: volumes_per_node
- id: node_group_template_id
- security_groups: security_groups
- shares: object_shares
- node_configs: node_configs
- auto_security_group: auto_security_group
- volumes_availability_zone: volumes_availability_zone
- description: node_group_template_description
- volume_mount_prefix: volume_mount_prefix
- plugin_name: plugin_name
- floating_ip_pool: floating_ip_pool
- is_default: is_default
- image_id: image_id
- volumes_size: volumes_size
- is_proxy_gateway: is_proxy_gateway
- is_public: object_is_public
- plugin_version: plugin_version
- name: node_group_template_name
- project_id: project_id
- created_at: created_at
- volume_type: volume_type
- is_protected: object_is_protected
- node_processes: node_processes
- flavor_id: flavor_id
Response Example
----------------
.. literalinclude:: samples/node-group-templates/node-group-template-show-response.json
:language: javascript
Delete node group template
==========================
.. rest_method:: DELETE /v2/node-group-templates/{node_group_template_id}
Deletes a node group template.
Normal response codes:204
Request
-------
.. rest_parameters:: parameters.yaml
- node_group_template_id: url_node_group_template_id
Update node group template
==========================
.. rest_method:: PATCH /v2/node-group-templates/{node_group_template_id}
Updates a node group template.
Normal respose codes:202
Request
-------
.. rest_parameters:: parameters.yaml
- node_group_template_id: url_node_group_template_id
Request Example
---------------
.. literalinclude:: samples/node-group-templates/node-group-template-update-request.json
:language: javascript
Export node group template
==========================
.. rest_method:: GET /v2/node-group-templates/{node_group_template_id}/export
Exports a node group template.
Normal respose codes:202
Request
-------
.. rest_parameters:: parameters.yaml
- node_group_template_id: url_node_group_template_id
Request Example
---------------
.. literalinclude:: samples/node-group-templates/node-group-template-update-request.json
:language: javascript

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,179 @@
.. -*- rst -*-
=======
Plugins
=======
A plugin object defines the Hadoop or Spark version that it can
install and which configurations can be set for the cluster.
Show plugin details
===================
.. rest_method:: GET /v2/plugins/{plugin_name}
Shows details for a plugin.
Normal response codes: 200
Error response codes: 400, 500
Request
-------
.. rest_parameters:: parameters.yaml
- plugin_name: url_plugin_name
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- versions: versions
- title: title
- description: description_plugin
- name: plugin_name
Response Example
----------------
.. literalinclude:: samples/plugins/plugin-show-response.json
:language: javascript
List plugins
============
.. rest_method:: GET /v2/plugins
Lists all registered plugins.
Normal response codes: 200
Error response codes: 400, 500
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- title: title
- versions: versions
- plugins: plugins
- description: description_plugin
- name: plugin_name
Response Example
----------------
.. literalinclude:: samples/plugins/plugins-list-response.json
:language: javascript
Show plugin version details
===========================
.. rest_method:: GET /v2/plugins/{plugin_name}/{version}
Shows details for a plugin version.
Normal response codes: 200
Error response codes: 400, 500
Request
-------
.. rest_parameters:: parameters.yaml
- plugin_name: url_plugin_name
- version: version
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- versions: versions
- title: title
- description: description_plugin
- name: plugin_name
Response Example
----------------
.. literalinclude:: samples/plugins/plugin-version-show-response.json
:language: javascript
Update plugin details
=====================
.. rest_method:: PATCH /v2/plugins/{plugin_name}
Updates details for a plugin.
Normal response codes: 202
Error response codes: 400, 500
Request
-------
.. rest_parameters:: parameters.yaml
- plugin_name: url_plugin_name
Request Example
---------------
.. literalinclude:: samples/plugins/plugin-update-request.json
:language: javascript
Response Parameters
-------------------
.. rest_parameters:: parameters.yaml
- title: title
- versions: versions
- description: description_plugin
- name: plugin_name
Response Example
----------------
.. literalinclude:: samples/plugins/plugin-update-response.json
:language: javascript

View File

@ -0,0 +1,17 @@
{
"plugin_name": "vanilla",
"plugin_version": "2.7.1",
"node_groups": [
{
"name": "worker",
"count": 3,
"node_group_template_id": "846edb31-add5-46e6-a4ee-a4c339f99251"
},
{
"name": "master",
"count": 1,
"node_group_template_id": "0bb9f1a4-0c44-4dc5-9452-6741c62ed9ae"
}
],
"name": "cluster-template"
}

View File

@ -0,0 +1,82 @@
{
"cluster_template": {
"is_public": false,
"anti_affinity": [],
"name": "cluster-template",
"created_at": "2015-09-14T10:38:44",
"project_id": "808d5032ea0446889097723bfc8e919d",
"cluster_configs": {},
"shares": null,
"id": "57c92a7c-5c6a-42ea-9c6f-9f40a5aa4b36",
"default_image_id": null,
"is_default": false,
"updated_at": null,
"plugin_name": "vanilla",
"node_groups": [
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {},
"auto_security_group": false,
"availability_zone": null,
"count": 1,
"flavor_id": "2",
"id": "1751c04e-8f39-467e-a421-480961172d4b",
"security_groups": null,
"use_autoconfig": true,
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:38:44",
"node_group_template_id": "0bb9f1a4-0c44-4dc5-9452-6741c62ed9ae",
"updated_at": null,
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "master",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"namenode",
"resourcemanager",
"oozie",
"historyserver"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"volume_type": null
},
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {},
"auto_security_group": false,
"availability_zone": null,
"count": 3,
"flavor_id": "2",
"id": "3ee85068-c455-4391-9db2-b54a20b99df3",
"security_groups": null,
"use_autoconfig": true,
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:38:44",
"node_group_template_id": "846edb31-add5-46e6-a4ee-a4c339f99251",
"updated_at": null,
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "worker",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"datanode",
"nodemanager"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"volume_type": null
}
],
"neutron_management_network": null,
"domain_name": null,
"plugin_version": "2.7.1",
"use_autoconfig": true,
"description": null,
"is_protected": false
}
}

View File

@ -0,0 +1,82 @@
{
"cluster_template": {
"is_public": false,
"anti_affinity": [],
"name": "cluster-template",
"created_at": "2015-09-14T10:38:44",
"project_id": "808d5032ea0446889097723bfc8e919d",
"cluster_configs": {},
"shares": null,
"id": "57c92a7c-5c6a-42ea-9c6f-9f40a5aa4b36",
"default_image_id": null,
"is_default": false,
"updated_at": null,
"plugin_name": "vanilla",
"node_groups": [
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {},
"auto_security_group": false,
"availability_zone": null,
"count": 1,
"flavor_id": "2",
"id": "1751c04e-8f39-467e-a421-480961172d4b",
"security_groups": null,
"use_autoconfig": true,
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:38:44",
"node_group_template_id": "0bb9f1a4-0c44-4dc5-9452-6741c62ed9ae",
"updated_at": null,
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "master",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"namenode",
"resourcemanager",
"oozie",
"historyserver"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"volume_type": null
},
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {},
"auto_security_group": false,
"availability_zone": null,
"count": 3,
"flavor_id": "2",
"id": "3ee85068-c455-4391-9db2-b54a20b99df3",
"security_groups": null,
"use_autoconfig": true,
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:38:44",
"node_group_template_id": "846edb31-add5-46e6-a4ee-a4c339f99251",
"updated_at": null,
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "worker",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"datanode",
"nodemanager"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"volume_type": null
}
],
"neutron_management_network": "b1610452-2933-46b0-bf31-660cfa5621bd",
"domain_name": null,
"plugin_version": "2.7.1",
"use_autoconfig": true,
"description": null,
"is_protected": false
}
}

View File

@ -0,0 +1,11 @@
{
"description": "Updated template",
"plugin_name": "vanilla",
"plugin_version": "2.7.1",
"name": "vanilla-updated",
"cluster_configs": {
"HDFS": {
"dfs.replication": 2
}
}
}

View File

@ -0,0 +1,67 @@
{
"cluster_template": {
"is_public": false,
"anti_affinity": [],
"name": "vanilla-updated",
"created_at": "2015-08-21T08:41:24",
"project_id": "808d5032ea0446889097723bfc8e919d",
"cluster_configs": {
"HDFS": {
"dfs.replication": 2
}
},
"shares": null,
"id": "84d47e85-6094-473f-bf6d-5a7e6e86564e",
"default_image_id": null,
"is_default": false,
"updated_at": "2015-09-14T10:45:57",
"plugin_name": "vanilla",
"node_groups": [
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {
"YARN": {},
"JobFlow": {},
"MapReduce": {},
"Hive": {},
"Hadoop": {},
"HDFS": {}
},
"auto_security_group": true,
"availability_zone": "",
"count": 1,
"flavor_id": "3",
"id": "57b966ab-617e-4735-bf60-0cb991208a52",
"security_groups": [],
"use_autoconfig": true,
"volumes_availability_zone": null,
"created_at": "2015-08-21T08:41:24",
"node_group_template_id": "a5533187-3f14-42c3-ba3a-196c13fe0fb5",
"updated_at": null,
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "all",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"namenode",
"datanode",
"historyserver",
"resourcemanager",
"nodemanager",
"oozie"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"volume_type": null
}
],
"neutron_management_network": null,
"domain_name": null,
"plugin_version": "2.7.1",
"use_autoconfig": true,
"description": "Updated template",
"is_protected": false
}
}

View File

@ -0,0 +1,140 @@
{
"cluster_templates": [
{
"is_public": false,
"anti_affinity": [],
"name": "cluster-template",
"created_at": "2015-09-14T10:38:44",
"project_id": "808d5032ea0446889097723bfc8e919d",
"cluster_configs": {},
"shares": null,
"id": "57c92a7c-5c6a-42ea-9c6f-9f40a5aa4b36",
"default_image_id": null,
"is_default": false,
"updated_at": null,
"plugin_name": "vanilla",
"node_groups": [
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {},
"auto_security_group": false,
"availability_zone": null,
"count": 1,
"flavor_id": "2",
"id": "1751c04e-8f39-467e-a421-480961172d4b",
"security_groups": null,
"use_autoconfig": true,
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:38:44",
"node_group_template_id": "0bb9f1a4-0c44-4dc5-9452-6741c62ed9ae",
"updated_at": null,
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "master",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"namenode",
"resourcemanager",
"oozie",
"historyserver"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"volume_type": null
},
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {},
"auto_security_group": false,
"availability_zone": null,
"count": 3,
"flavor_id": "2",
"id": "3ee85068-c455-4391-9db2-b54a20b99df3",
"security_groups": null,
"use_autoconfig": true,
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:38:44",
"node_group_template_id": "846edb31-add5-46e6-a4ee-a4c339f99251",
"updated_at": null,
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "worker",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"datanode",
"nodemanager"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"volume_type": null
}
],
"neutron_management_network": "b1610452-2933-46b0-bf31-660cfa5621bd",
"domain_name": null,
"plugin_version": "2.7.1",
"use_autoconfig": true,
"description": null,
"is_protected": false
},
{
"is_public": true,
"anti_affinity": [],
"name": "asd",
"created_at": "2015-08-18T08:39:39",
"project_id": "808d5032ea0446889097723bfc8e919d",
"cluster_configs": {
"general": {}
},
"shares": null,
"id": "5a9c787c-2078-4f7d-9a66-27759be9051b",
"default_image_id": null,
"is_default": false,
"updated_at": "2015-09-14T08:41:15",
"plugin_name": "vanilla",
"node_groups": [
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {},
"auto_security_group": true,
"availability_zone": "",
"count": 1,
"flavor_id": "2",
"id": "a65864dd-3f99-4d29-a011-f7711cc23fa0",
"security_groups": [],
"use_autoconfig": true,
"volumes_availability_zone": null,
"created_at": "2015-08-18T08:39:39",
"node_group_template_id": "42ce49de-1b8f-41d5-8f4a-244ec0826d92",
"updated_at": null,
"volumes_per_node": 1,
"is_proxy_gateway": false,
"name": "asd",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"namenode",
"jobtracker"
],
"volumes_size": 10,
"volume_local_to_instance": false,
"volume_type": null
}
],
"neutron_management_network": null,
"domain_name": null,
"plugin_version": "2.7.1",
"use_autoconfig": true,
"description": "",
"is_protected": false
}
],
"markers": {
"prev": null,
"next": "2c76e0d3-56cd-4d28-bb4f-4808e538c7b9"
}
}

View File

@ -0,0 +1,9 @@
{
"plugin_name": "vanilla",
"plugin_version": "2.7.1",
"cluster_template_id": "57c92a7c-5c6a-42ea-9c6f-9f40a5aa4b36",
"default_image_id": "4118a476-dfdc-4b0e-8d5c-463cba08e9ae",
"user_keypair_id": "test",
"name": "vanilla-cluster",
"neutron_management_network": "b1610452-2933-46b0-bf31-660cfa5621bd"
}

View File

@ -0,0 +1,128 @@
{
"cluster": {
"is_public": false,
"project_id": "808d5032ea0446889097723bfc8e919d",
"shares": null,
"domain_name": null,
"status_description": "",
"plugin_name": "vanilla",
"neutron_management_network": "b1610452-2933-46b0-bf31-660cfa5621bd",
"info": {},
"user_keypair_id": "test",
"management_public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfe9ARO+t9CybtuC1+cusDTeQL7wos1+U2dKPlCUJvNUn0PcunGefqWI4MUZPY9yGmvRqfINy7/xRQCzL0AwgqzwcCXamcK8JCC80uH7j8Vxa4kJheG1jxMoz/FpDSdRnzNZ+m7H5rjOwAQANhL7KatGLyCPQg9fqOoaIyCZE/A3fztm/XjJMpWnuANpUZubZtISEfu4UZKVk/DPSlBrbTZkTOvEog1LwZCZoTt0rq6a7PJFzJJkq0YecRudu/f3tpXbNe/F84sd9PhOSqcrRbm72WzglyEE8PuS1kuWpEz8G+Y5/0tQxnoh6khj9mgflrdCFuvpdutFLH4eN5MFDh Generated-by-Sahara\n",
"id": "e172d86c-906d-418e-a29c-6189f53bfa42",
"cluster_template_id": "57c92a7c-5c6a-42ea-9c6f-9f40a5aa4b36",
"node_groups": [
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {
"YARN": {
"yarn.nodemanager.vmem-check-enabled": "false",
"yarn.scheduler.maximum-allocation-mb": 2048,
"yarn.scheduler.minimum-allocation-mb": 256,
"yarn.nodemanager.resource.memory-mb": 2048
},
"MapReduce": {
"yarn.app.mapreduce.am.resource.mb": 256,
"mapreduce.task.io.sort.mb": 102,
"mapreduce.reduce.java.opts": "-Xmx409m",
"mapreduce.reduce.memory.mb": 512,
"mapreduce.map.memory.mb": 256,
"yarn.app.mapreduce.am.command-opts": "-Xmx204m",
"mapreduce.map.java.opts": "-Xmx204m"
}
},
"auto_security_group": false,
"availability_zone": null,
"count": 1,
"flavor_id": "2",
"id": "0fe07f2a-0275-4bc0-93b2-c3c1e48e2815",
"security_groups": null,
"use_autoconfig": true,
"instances": [],
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:57:11",
"node_group_template_id": "0bb9f1a4-0c44-4dc5-9452-6741c62ed9ae",
"updated_at": "2015-09-14T10:57:12",
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "master",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"namenode",
"resourcemanager",
"oozie",
"historyserver"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"volume_type": null
},
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {
"YARN": {
"yarn.nodemanager.vmem-check-enabled": "false",
"yarn.scheduler.maximum-allocation-mb": 2048,
"yarn.scheduler.minimum-allocation-mb": 256,
"yarn.nodemanager.resource.memory-mb": 2048
},
"MapReduce": {
"yarn.app.mapreduce.am.resource.mb": 256,
"mapreduce.task.io.sort.mb": 102,
"mapreduce.reduce.java.opts": "-Xmx409m",
"mapreduce.reduce.memory.mb": 512,
"mapreduce.map.memory.mb": 256,
"yarn.app.mapreduce.am.command-opts": "-Xmx204m",
"mapreduce.map.java.opts": "-Xmx204m"
}
},
"auto_security_group": false,
"availability_zone": null,
"count": 3,
"flavor_id": "2",
"id": "c7a3bea4-c898-446b-8c67-6d378d4c06c4",
"security_groups": null,
"use_autoconfig": true,
"instances": [],
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:57:11",
"node_group_template_id": "846edb31-add5-46e6-a4ee-a4c339f99251",
"updated_at": "2015-09-14T10:57:12",
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "worker",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"datanode",
"nodemanager"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"volume_type": null
}
],
"provision_progress": [],
"plugin_version": "2.7.1",
"use_autoconfig": true,
"trust_id": null,
"description": null,
"created_at": "2015-09-14T10:57:11",
"is_protected": false,
"updated_at": "2015-09-14T10:57:12",
"is_transient": false,
"cluster_configs": {
"HDFS": {
"dfs.replication": 3
}
},
"anti_affinity": [],
"name": "vanilla-cluster",
"default_image_id": "4118a476-dfdc-4b0e-8d5c-463cba08e9ae",
"status": "Validating"
}
}

View File

@ -0,0 +1,15 @@
{
"add_node_groups": [
{
"count": 1,
"name": "b-worker",
"node_group_template_id": "bc270ffe-a086-4eeb-9baa-2f5a73504622"
}
],
"resize_node_groups": [
{
"count": 4,
"name": "worker"
}
]
}

View File

@ -0,0 +1,370 @@
{
"cluster": {
"info": {
"YARN": {
"Web UI": "http://172.18.168.115:8088",
"ResourceManager": "http://172.18.168.115:8032"
},
"HDFS": {
"Web UI": "http://172.18.168.115:50070",
"NameNode": "hdfs://vanilla-cluster-master-0:9000"
},
"MapReduce JobHistory Server": {
"Web UI": "http://172.18.168.115:19888"
},
"JobFlow": {
"Oozie": "http://172.18.168.115:11000"
}
},
"plugin_name": "vanilla",
"plugin_version": "2.7.1",
"updated_at": "2015-09-14T11:01:15",
"name": "vanilla-cluster",
"id": "e172d86c-906d-418e-a29c-6189f53bfa42",
"management_public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfe9ARO+t9CybtuC1+cusDTeQL7wos1+U2dKPlCUJvNUn0PcunGefqWI4MUZPY9yGmvRqfINy7/xRQCzL0AwgqzwcCXamcK8JCC80uH7j8Vxa4kJheG1jxMoz/FpDSdRnzNZ+m7H5rjOwAQANhL7KatGLyCPQg9fqOoaIyCZE/A3fztm/XjJMpWnuANpUZubZtISEfu4UZKVk/DPSlBrbTZkTOvEog1LwZCZoTt0rq6a7PJFzJJkq0YecRudu/f3tpXbNe/F84sd9PhOSqcrRbm72WzglyEE8PuS1kuWpEz8G+Y5/0tQxnoh6khj9mgflrdCFuvpdutFLH4eN5MFDh Generated-by-Sahara\n",
"trust_id": null,
"status_description": "",
"default_image_id": "4118a476-dfdc-4b0e-8d5c-463cba08e9ae",
"cluster_template_id": "57c92a7c-5c6a-42ea-9c6f-9f40a5aa4b36",
"is_protected": false,
"is_transient": false,
"provision_progress": [
{
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42",
"total": 1,
"successful": true,
"step_name": "Create Heat stack",
"step_type": "Engine: create cluster",
"updated_at": "2015-09-14T10:57:38",
"project_id": "808d5032ea0446889097723bfc8e919d",
"created_at": "2015-09-14T10:57:18",
"id": "0a6d95f9-30f4-4434-823a-a38a7999a5af"
},
{
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42",
"total": 4,
"successful": true,
"step_name": "Configure instances",
"step_type": "Engine: create cluster",
"updated_at": "2015-09-14T10:58:22",
"project_id": "808d5032ea0446889097723bfc8e919d",
"created_at": "2015-09-14T10:58:16",
"id": "29f2b587-c34c-4871-9ed9-9235b411cd9a"
},
{
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42",
"total": 1,
"successful": true,
"step_name": "Start the following process(es): Oozie",
"step_type": "Plugin: start cluster",
"updated_at": "2015-09-14T11:01:15",
"project_id": "808d5032ea0446889097723bfc8e919d",
"created_at": "2015-09-14T11:00:27",
"id": "36f1efde-90f9-41c1-b409-aa1cf9623e3e"
},
{
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42",
"total": 4,
"successful": true,
"step_name": "Configure instances",
"step_type": "Plugin: configure cluster",
"updated_at": "2015-09-14T10:59:21",
"project_id": "808d5032ea0446889097723bfc8e919d",
"created_at": "2015-09-14T10:58:22",
"id": "602bcc27-3a2d-42c8-8aca-ebc475319c72"
},
{
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42",
"total": 1,
"successful": true,
"step_name": "Configure topology data",
"step_type": "Plugin: configure cluster",
"updated_at": "2015-09-14T10:59:37",
"project_id": "808d5032ea0446889097723bfc8e919d",
"created_at": "2015-09-14T10:59:21",
"id": "7e291df1-2d32-410d-ae89-33ab6f83cf17"
},
{
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42",
"total": 3,
"successful": true,
"step_name": "Start the following process(es): DataNodes, NodeManagers",
"step_type": "Plugin: start cluster",
"updated_at": "2015-09-14T11:00:11",
"project_id": "808d5032ea0446889097723bfc8e919d",
"created_at": "2015-09-14T11:00:01",
"id": "8ab7933c-ad61-4a4f-88db-23ce78ee10f6"
},
{
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42",
"total": 1,
"successful": true,
"step_name": "Await DataNodes start up",
"step_type": "Plugin: start cluster",
"updated_at": "2015-09-14T11:00:21",
"project_id": "808d5032ea0446889097723bfc8e919d",
"created_at": "2015-09-14T11:00:11",
"id": "9c8dc016-8c5b-4e80-9857-80c41f6bd971"
},
{
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42",
"total": 1,
"successful": true,
"step_name": "Start the following process(es): HistoryServer",
"step_type": "Plugin: start cluster",
"updated_at": "2015-09-14T11:00:27",
"project_id": "808d5032ea0446889097723bfc8e919d",
"created_at": "2015-09-14T11:00:21",
"id": "c6327532-222b-416c-858f-73dbb32b8e97"
},
{
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42",
"total": 4,
"successful": true,
"step_name": "Wait for instance accessibility",
"step_type": "Engine: create cluster",
"updated_at": "2015-09-14T10:58:14",
"project_id": "808d5032ea0446889097723bfc8e919d",
"created_at": "2015-09-14T10:57:41",
"id": "d3eca726-8b44-473a-ac29-fba45a893725"
},
{
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42",
"total": 0,
"successful": true,
"step_name": "Mount volumes to instances",
"step_type": "Engine: create cluster",
"updated_at": "2015-09-14T10:58:15",
"project_id": "808d5032ea0446889097723bfc8e919d",
"created_at": "2015-09-14T10:58:14",
"id": "d7a875ff-64bf-41aa-882d-b5061c8ee152"
},
{
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42",
"total": 1,
"successful": true,
"step_name": "Start the following process(es): ResourceManager",
"step_type": "Plugin: start cluster",
"updated_at": "2015-09-14T11:00:00",
"project_id": "808d5032ea0446889097723bfc8e919d",
"created_at": "2015-09-14T10:59:55",
"id": "ded7d227-10b8-4cb0-ab6c-25da1462bb7a"
},
{
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42",
"total": 1,
"successful": true,
"step_name": "Start the following process(es): NameNode",
"step_type": "Plugin: start cluster",
"updated_at": "2015-09-14T10:59:54",
"project_id": "808d5032ea0446889097723bfc8e919d",
"created_at": "2015-09-14T10:59:38",
"id": "e1701ff5-930a-4212-945a-43515dfe24d1"
},
{
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42",
"total": 4,
"successful": true,
"step_name": "Assign IPs",
"step_type": "Engine: create cluster",
"updated_at": "2015-09-14T10:57:41",
"project_id": "808d5032ea0446889097723bfc8e919d",
"created_at": "2015-09-14T10:57:38",
"id": "eaf0ab1b-bf8f-48f0-8f2c-fa4f82f539b9"
}
],
"status": "Active",
"description": null,
"use_autoconfig": true,
"shares": null,
"domain_name": null,
"neutron_management_network": "b1610452-2933-46b0-bf31-660cfa5621bd",
"is_public": false,
"project_id": "808d5032ea0446889097723bfc8e919d",
"node_groups": [
{
"volumes_per_node": 0,
"volume_type": null,
"updated_at": "2015-09-14T10:57:37",
"name": "b-worker",
"id": "b7a6dea4-c898-446b-8c67-4f378d4c06c4",
"node_group_template_id": "bc270ffe-a086-4eeb-9baa-2f5a73504622",
"node_configs": {
"YARN": {
"yarn.nodemanager.vmem-check-enabled": "false",
"yarn.scheduler.minimum-allocation-mb": 256,
"yarn.nodemanager.resource.memory-mb": 2048,
"yarn.scheduler.maximum-allocation-mb": 2048
},
"MapReduce": {
"mapreduce.map.memory.mb": 256,
"yarn.app.mapreduce.am.command-opts": "-Xmx204m",
"mapreduce.map.java.opts": "-Xmx204m",
"mapreduce.reduce.memory.mb": 512,
"mapreduce.task.io.sort.mb": 102,
"mapreduce.reduce.java.opts": "-Xmx409m",
"yarn.app.mapreduce.am.resource.mb": 256
}
},
"auto_security_group": false,
"volumes_availability_zone": null,
"use_autoconfig": true,
"security_groups": null,
"shares": null,
"node_processes": [
"datanode",
"nodemanager"
],
"availability_zone": null,
"flavor_id": "2",
"image_id": null,
"volume_local_to_instance": false,
"count": 1,
"volumes_size": 0,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"volume_mount_prefix": "/volumes/disk",
"instances": [],
"is_proxy_gateway": false,
"created_at": "2015-09-14T10:57:11"
},
{
"volumes_per_node": 0,
"volume_type": null,
"updated_at": "2015-09-14T10:57:36",
"name": "master",
"id": "0fe07f2a-0275-4bc0-93b2-c3c1e48e2815",
"node_group_template_id": "0bb9f1a4-0c44-4dc5-9452-6741c62ed9ae",
"node_configs": {
"YARN": {
"yarn.nodemanager.vmem-check-enabled": "false",
"yarn.scheduler.minimum-allocation-mb": 256,
"yarn.nodemanager.resource.memory-mb": 2048,
"yarn.scheduler.maximum-allocation-mb": 2048
},
"MapReduce": {
"mapreduce.map.memory.mb": 256,
"yarn.app.mapreduce.am.command-opts": "-Xmx204m",
"mapreduce.map.java.opts": "-Xmx204m",
"mapreduce.reduce.memory.mb": 512,
"mapreduce.task.io.sort.mb": 102,
"mapreduce.reduce.java.opts": "-Xmx409m",
"yarn.app.mapreduce.am.resource.mb": 256
}
},
"auto_security_group": false,
"volumes_availability_zone": null,
"use_autoconfig": true,
"security_groups": null,
"shares": null,
"node_processes": [
"namenode",
"resourcemanager",
"oozie",
"historyserver"
],
"availability_zone": null,
"flavor_id": "2",
"image_id": null,
"volume_local_to_instance": false,
"count": 1,
"volumes_size": 0,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"volume_mount_prefix": "/volumes/disk",
"instances": [
{
"instance_id": "b9f16a07-88fc-423e-83a3-489598fe6737",
"internal_ip": "10.50.0.60",
"instance_name": "vanilla-cluster-master-0",
"updated_at": "2015-09-14T10:57:39",
"management_ip": "172.18.168.115",
"created_at": "2015-09-14T10:57:36",
"id": "4867d92e-cc7b-4cde-9a1a-149e91caa491"
}
],
"is_proxy_gateway": false,
"created_at": "2015-09-14T10:57:11"
},
{
"volumes_per_node": 0,
"volume_type": null,
"updated_at": "2015-09-14T10:57:37",
"name": "worker",
"id": "c7a3bea4-c898-446b-8c67-6d378d4c06c4",
"node_group_template_id": "846edb31-add5-46e6-a4ee-a4c339f99251",
"node_configs": {
"YARN": {
"yarn.nodemanager.vmem-check-enabled": "false",
"yarn.scheduler.minimum-allocation-mb": 256,
"yarn.nodemanager.resource.memory-mb": 2048,
"yarn.scheduler.maximum-allocation-mb": 2048
},
"MapReduce": {
"mapreduce.map.memory.mb": 256,
"yarn.app.mapreduce.am.command-opts": "-Xmx204m",
"mapreduce.map.java.opts": "-Xmx204m",
"mapreduce.reduce.memory.mb": 512,
"mapreduce.task.io.sort.mb": 102,
"mapreduce.reduce.java.opts": "-Xmx409m",
"yarn.app.mapreduce.am.resource.mb": 256
}
},
"auto_security_group": false,
"volumes_availability_zone": null,
"use_autoconfig": true,
"security_groups": null,
"shares": null,
"node_processes": [
"datanode",
"nodemanager"
],
"availability_zone": null,
"flavor_id": "2",
"image_id": null,
"volume_local_to_instance": false,
"count": 4,
"volumes_size": 0,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"volume_mount_prefix": "/volumes/disk",
"instances": [
{
"instance_id": "0cf1ee81-aa72-48da-be2c-65bc2fa51f8f",
"internal_ip": "10.50.0.63",
"instance_name": "vanilla-cluster-worker-0",
"updated_at": "2015-09-14T10:57:39",
"management_ip": "172.18.168.118",
"created_at": "2015-09-14T10:57:37",
"id": "f3633b30-c1e4-4144-930b-ab5b780b87be"
},
{
"instance_id": "4a937391-b594-4ad0-9a53-00a99a691383",
"internal_ip": "10.50.0.62",
"instance_name": "vanilla-cluster-worker-1",
"updated_at": "2015-09-14T10:57:40",
"management_ip": "172.18.168.117",
"created_at": "2015-09-14T10:57:37",
"id": "0d66fd93-f277-4a94-b46a-f5866aa0c38f"
},
{
"instance_id": "839b1d56-6d0d-4aa4-9d05-30e029c276f8",
"internal_ip": "10.50.0.61",
"instance_name": "vanilla-cluster-worker-2",
"updated_at": "2015-09-14T10:57:40",
"management_ip": "172.18.168.116",
"created_at": "2015-09-14T10:57:37",
"id": "0982cefd-5c58-436e-8f1e-c1d0830f18a7"
}
],
"is_proxy_gateway": false,
"created_at": "2015-09-14T10:57:11"
}
],
"cluster_configs": {
"HDFS": {
"dfs.replication": 3
}
},
"user_keypair_id": "apavlov",
"anti_affinity": [],
"created_at": "2015-09-14T10:57:11"
}
}

View File

@ -0,0 +1,128 @@
{
"cluster": {
"is_public": false,
"project_id": "808d5032ea0446889097723bfc8e919d",
"shares": null,
"domain_name": null,
"status_description": "",
"plugin_name": "vanilla",
"neutron_management_network": "b1610452-2933-46b0-bf31-660cfa5621bd",
"info": {},
"user_keypair_id": "test",
"management_public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfe9ARO+t9CybtuC1+cusDTeQL7wos1+U2dKPlCUJvNUn0PcunGefqWI4MUZPY9yGmvRqfINy7/xRQCzL0AwgqzwcCXamcK8JCC80uH7j8Vxa4kJheG1jxMoz/FpDSdRnzNZ+m7H5rjOwAQANhL7KatGLyCPQg9fqOoaIyCZE/A3fztm/XjJMpWnuANpUZubZtISEfu4UZKVk/DPSlBrbTZkTOvEog1LwZCZoTt0rq6a7PJFzJJkq0YecRudu/f3tpXbNe/F84sd9PhOSqcrRbm72WzglyEE8PuS1kuWpEz8G+Y5/0tQxnoh6khj9mgflrdCFuvpdutFLH4eN5MFDh Generated-by-Sahara\n",
"id": "e172d86c-906d-418e-a29c-6189f53bfa42",
"cluster_template_id": "57c92a7c-5c6a-42ea-9c6f-9f40a5aa4b36",
"node_groups": [
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {
"YARN": {
"yarn.nodemanager.vmem-check-enabled": "false",
"yarn.scheduler.maximum-allocation-mb": 2048,
"yarn.scheduler.minimum-allocation-mb": 256,
"yarn.nodemanager.resource.memory-mb": 2048
},
"MapReduce": {
"yarn.app.mapreduce.am.resource.mb": 256,
"mapreduce.task.io.sort.mb": 102,
"mapreduce.reduce.java.opts": "-Xmx409m",
"mapreduce.reduce.memory.mb": 512,
"mapreduce.map.memory.mb": 256,
"yarn.app.mapreduce.am.command-opts": "-Xmx204m",
"mapreduce.map.java.opts": "-Xmx204m"
}
},
"auto_security_group": false,
"availability_zone": null,
"count": 1,
"flavor_id": "2",
"id": "0fe07f2a-0275-4bc0-93b2-c3c1e48e2815",
"security_groups": null,
"use_autoconfig": true,
"instances": [],
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:57:11",
"node_group_template_id": "0bb9f1a4-0c44-4dc5-9452-6741c62ed9ae",
"updated_at": "2015-09-14T10:57:12",
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "master",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"namenode",
"resourcemanager",
"oozie",
"historyserver"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"volume_type": null
},
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {
"YARN": {
"yarn.nodemanager.vmem-check-enabled": "false",
"yarn.scheduler.maximum-allocation-mb": 2048,
"yarn.scheduler.minimum-allocation-mb": 256,
"yarn.nodemanager.resource.memory-mb": 2048
},
"MapReduce": {
"yarn.app.mapreduce.am.resource.mb": 256,
"mapreduce.task.io.sort.mb": 102,
"mapreduce.reduce.java.opts": "-Xmx409m",
"mapreduce.reduce.memory.mb": 512,
"mapreduce.map.memory.mb": 256,
"yarn.app.mapreduce.am.command-opts": "-Xmx204m",
"mapreduce.map.java.opts": "-Xmx204m"
}
},
"auto_security_group": false,
"availability_zone": null,
"count": 3,
"flavor_id": "2",
"id": "c7a3bea4-c898-446b-8c67-6d378d4c06c4",
"security_groups": null,
"use_autoconfig": true,
"instances": [],
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:57:11",
"node_group_template_id": "846edb31-add5-46e6-a4ee-a4c339f99251",
"updated_at": "2015-09-14T10:57:12",
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "worker",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"datanode",
"nodemanager"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"volume_type": null
}
],
"provision_progress": [],
"plugin_version": "2.7.1",
"use_autoconfig": true,
"trust_id": null,
"description": null,
"created_at": "2015-09-14T10:57:11",
"is_protected": false,
"updated_at": "2015-09-14T10:57:12",
"is_transient": false,
"cluster_configs": {
"HDFS": {
"dfs.replication": 3
}
},
"anti_affinity": [],
"name": "vanilla-cluster",
"default_image_id": "4118a476-dfdc-4b0e-8d5c-463cba08e9ae",
"status": "Validating"
}
}

View File

@ -0,0 +1,4 @@
{
"name": "public-vanilla-cluster",
"is_public": true
}

View File

@ -0,0 +1,128 @@
{
"cluster": {
"is_public": true,
"project_id": "808d5032ea0446889097723bfc8e919d",
"shares": null,
"domain_name": null,
"status_description": "",
"plugin_name": "vanilla",
"neutron_management_network": "b1610452-2933-46b0-bf31-660cfa5621bd",
"info": {},
"user_keypair_id": "test",
"management_public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfe9ARO+t9CybtuC1+cusDTeQL7wos1+U2dKPlCUJvNUn0PcunGefqWI4MUZPY9yGmvRqfINy7/xRQCzL0AwgqzwcCXamcK8JCC80uH7j8Vxa4kJheG1jxMoz/FpDSdRnzNZ+m7H5rjOwAQANhL7KatGLyCPQg9fqOoaIyCZE/A3fztm/XjJMpWnuANpUZubZtISEfu4UZKVk/DPSlBrbTZkTOvEog1LwZCZoTt0rq6a7PJFzJJkq0YecRudu/f3tpXbNe/F84sd9PhOSqcrRbm72WzglyEE8PuS1kuWpEz8G+Y5/0tQxnoh6khj9mgflrdCFuvpdutFLH4eN5MFDh Generated-by-Sahara\n",
"id": "e172d86c-906d-418e-a29c-6189f53bfa42",
"cluster_template_id": "57c92a7c-5c6a-42ea-9c6f-9f40a5aa4b36",
"node_groups": [
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {
"YARN": {
"yarn.nodemanager.vmem-check-enabled": "false",
"yarn.scheduler.maximum-allocation-mb": 2048,
"yarn.scheduler.minimum-allocation-mb": 256,
"yarn.nodemanager.resource.memory-mb": 2048
},
"MapReduce": {
"yarn.app.mapreduce.am.resource.mb": 256,
"mapreduce.task.io.sort.mb": 102,
"mapreduce.reduce.java.opts": "-Xmx409m",
"mapreduce.reduce.memory.mb": 512,
"mapreduce.map.memory.mb": 256,
"yarn.app.mapreduce.am.command-opts": "-Xmx204m",
"mapreduce.map.java.opts": "-Xmx204m"
}
},
"auto_security_group": false,
"availability_zone": null,
"count": 1,
"flavor_id": "2",
"id": "0fe07f2a-0275-4bc0-93b2-c3c1e48e2815",
"security_groups": null,
"use_autoconfig": true,
"instances": [],
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:57:11",
"node_group_template_id": "0bb9f1a4-0c44-4dc5-9452-6741c62ed9ae",
"updated_at": "2015-09-14T10:57:12",
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "master",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"namenode",
"resourcemanager",
"oozie",
"historyserver"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"volume_type": null
},
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {
"YARN": {
"yarn.nodemanager.vmem-check-enabled": "false",
"yarn.scheduler.maximum-allocation-mb": 2048,
"yarn.scheduler.minimum-allocation-mb": 256,
"yarn.nodemanager.resource.memory-mb": 2048
},
"MapReduce": {
"yarn.app.mapreduce.am.resource.mb": 256,
"mapreduce.task.io.sort.mb": 102,
"mapreduce.reduce.java.opts": "-Xmx409m",
"mapreduce.reduce.memory.mb": 512,
"mapreduce.map.memory.mb": 256,
"yarn.app.mapreduce.am.command-opts": "-Xmx204m",
"mapreduce.map.java.opts": "-Xmx204m"
}
},
"auto_security_group": false,
"availability_zone": null,
"count": 3,
"flavor_id": "2",
"id": "c7a3bea4-c898-446b-8c67-6d378d4c06c4",
"security_groups": null,
"use_autoconfig": true,
"instances": [],
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:57:11",
"node_group_template_id": "846edb31-add5-46e6-a4ee-a4c339f99251",
"updated_at": "2015-09-14T10:57:12",
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "worker",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"datanode",
"nodemanager"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"volume_type": null
}
],
"provision_progress": [],
"plugin_version": "2.7.1",
"use_autoconfig": true,
"trust_id": null,
"description": null,
"created_at": "2015-09-14T10:57:11",
"is_protected": false,
"updated_at": "2015-09-14T10:57:12",
"is_transient": false,
"cluster_configs": {
"HDFS": {
"dfs.replication": 3
}
},
"anti_affinity": [],
"name": "public-vanilla-cluster",
"default_image_id": "4118a476-dfdc-4b0e-8d5c-463cba08e9ae",
"status": "Validating"
}
}

View File

@ -0,0 +1,327 @@
{
"clusters": [
{
"is_public": false,
"project_id": "808d5032ea0446889097723bfc8e919d",
"shares": null,
"domain_name": null,
"status_description": "",
"plugin_name": "vanilla",
"neutron_management_network": "b1610452-2933-46b0-bf31-660cfa5621bd",
"info": {
"YARN": {
"Web UI": "http://172.18.168.115:8088",
"ResourceManager": "http://172.18.168.115:8032"
},
"HDFS": {
"Web UI": "http://172.18.168.115:50070",
"NameNode": "hdfs://vanilla-cluster-master-0:9000"
},
"JobFlow": {
"Oozie": "http://172.18.168.115:11000"
},
"MapReduce JobHistory Server": {
"Web UI": "http://172.18.168.115:19888"
}
},
"user_keypair_id": "apavlov",
"management_public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfe9ARO+t9CybtuC1+cusDTeQL7wos1+U2dKPlCUJvNUn0PcunGefqWI4MUZPY9yGmvRqfINy7/xRQCzL0AwgqzwcCXamcK8JCC80uH7j8Vxa4kJheG1jxMoz/FpDSdRnzNZ+m7H5rjOwAQANhL7KatGLyCPQg9fqOoaIyCZE/A3fztm/XjJMpWnuANpUZubZtISEfu4UZKVk/DPSlBrbTZkTOvEog1LwZCZoTt0rq6a7PJFzJJkq0YecRudu/f3tpXbNe/F84sd9PhOSqcrRbm72WzglyEE8PuS1kuWpEz8G+Y5/0tQxnoh6khj9mgflrdCFuvpdutFLH4eN5MFDh Generated-by-Sahara\n",
"id": "e172d86c-906d-418e-a29c-6189f53bfa42",
"cluster_template_id": "57c92a7c-5c6a-42ea-9c6f-9f40a5aa4b36",
"node_groups": [
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {
"YARN": {
"yarn.nodemanager.vmem-check-enabled": "false",
"yarn.scheduler.maximum-allocation-mb": 2048,
"yarn.scheduler.minimum-allocation-mb": 256,
"yarn.nodemanager.resource.memory-mb": 2048
},
"MapReduce": {
"yarn.app.mapreduce.am.resource.mb": 256,
"mapreduce.task.io.sort.mb": 102,
"mapreduce.reduce.java.opts": "-Xmx409m",
"mapreduce.reduce.memory.mb": 512,
"mapreduce.map.memory.mb": 256,
"yarn.app.mapreduce.am.command-opts": "-Xmx204m",
"mapreduce.map.java.opts": "-Xmx204m"
}
},
"auto_security_group": false,
"availability_zone": null,
"count": 1,
"flavor_id": "2",
"id": "0fe07f2a-0275-4bc0-93b2-c3c1e48e2815",
"security_groups": null,
"use_autoconfig": true,
"instances": [
{
"created_at": "2015-09-14T10:57:36",
"id": "4867d92e-cc7b-4cde-9a1a-149e91caa491",
"management_ip": "172.18.168.115",
"updated_at": "2015-09-14T10:57:39",
"instance_id": "b9f16a07-88fc-423e-83a3-489598fe6737",
"internal_ip": "10.50.0.60",
"instance_name": "vanilla-cluster-master-0"
}
],
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:57:11",
"node_group_template_id": "0bb9f1a4-0c44-4dc5-9452-6741c62ed9ae",
"updated_at": "2015-09-14T10:57:36",
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "master",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"namenode",
"resourcemanager",
"oozie",
"historyserver"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"volume_type": null
},
{
"image_id": null,
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {
"YARN": {
"yarn.nodemanager.vmem-check-enabled": "false",
"yarn.scheduler.maximum-allocation-mb": 2048,
"yarn.scheduler.minimum-allocation-mb": 256,
"yarn.nodemanager.resource.memory-mb": 2048
},
"MapReduce": {
"yarn.app.mapreduce.am.resource.mb": 256,
"mapreduce.task.io.sort.mb": 102,
"mapreduce.reduce.java.opts": "-Xmx409m",
"mapreduce.reduce.memory.mb": 512,
"mapreduce.map.memory.mb": 256,
"yarn.app.mapreduce.am.command-opts": "-Xmx204m",
"mapreduce.map.java.opts": "-Xmx204m"
}
},
"auto_security_group": false,
"availability_zone": null,
"count": 3,
"flavor_id": "2",
"id": "c7a3bea4-c898-446b-8c67-6d378d4c06c4",
"security_groups": null,
"use_autoconfig": true,
"instances": [
{
"created_at": "2015-09-14T10:57:37",
"id": "f3633b30-c1e4-4144-930b-ab5b780b87be",
"management_ip": "172.18.168.118",
"updated_at": "2015-09-14T10:57:39",
"instance_id": "0cf1ee81-aa72-48da-be2c-65bc2fa51f8f",
"internal_ip": "10.50.0.63",
"instance_name": "vanilla-cluster-worker-0"
},
{
"created_at": "2015-09-14T10:57:37",
"id": "0d66fd93-f277-4a94-b46a-f5866aa0c38f",
"management_ip": "172.18.168.117",
"updated_at": "2015-09-14T10:57:40",
"instance_id": "4a937391-b594-4ad0-9a53-00a99a691383",
"internal_ip": "10.50.0.62",
"instance_name": "vanilla-cluster-worker-1"
},
{
"created_at": "2015-09-14T10:57:37",
"id": "0982cefd-5c58-436e-8f1e-c1d0830f18a7",
"management_ip": "172.18.168.116",
"updated_at": "2015-09-14T10:57:40",
"instance_id": "839b1d56-6d0d-4aa4-9d05-30e029c276f8",
"internal_ip": "10.50.0.61",
"instance_name": "vanilla-cluster-worker-2"
}
],
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:57:11",
"node_group_template_id": "846edb31-add5-46e6-a4ee-a4c339f99251",
"updated_at": "2015-09-14T10:57:37",
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "worker",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"datanode",
"nodemanager"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"volume_type": null
}
],
"provision_progress": [
{
"created_at": "2015-09-14T10:57:18",
"project_id": "808d5032ea0446889097723bfc8e919d",
"id": "0a6d95f9-30f4-4434-823a-a38a7999a5af",
"step_type": "Engine: create cluster",
"step_name": "Create Heat stack",
"updated_at": "2015-09-14T10:57:38",
"successful": true,
"total": 1,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T10:58:16",
"project_id": "808d5032ea0446889097723bfc8e919d",
"id": "29f2b587-c34c-4871-9ed9-9235b411cd9a",
"step_type": "Engine: create cluster",
"step_name": "Configure instances",
"updated_at": "2015-09-14T10:58:22",
"successful": true,
"total": 4,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T11:00:27",
"project_id": "808d5032ea0446889097723bfc8e919d",
"id": "36f1efde-90f9-41c1-b409-aa1cf9623e3e",
"step_type": "Plugin: start cluster",
"step_name": "Start the following process(es): Oozie",
"updated_at": "2015-09-14T11:01:15",
"successful": true,
"total": 1,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T10:58:22",
"project_id": "808d5032ea0446889097723bfc8e919d",
"id": "602bcc27-3a2d-42c8-8aca-ebc475319c72",
"step_type": "Plugin: configure cluster",
"step_name": "Configure instances",
"updated_at": "2015-09-14T10:59:21",
"successful": true,
"total": 4,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T10:59:21",
"project_id": "808d5032ea0446889097723bfc8e919d",
"id": "7e291df1-2d32-410d-ae89-33ab6f83cf17",
"step_type": "Plugin: configure cluster",
"step_name": "Configure topology data",
"updated_at": "2015-09-14T10:59:37",
"successful": true,
"total": 1,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T11:00:01",
"project_id": "808d5032ea0446889097723bfc8e919d",
"id": "8ab7933c-ad61-4a4f-88db-23ce78ee10f6",
"step_type": "Plugin: start cluster",
"step_name": "Start the following process(es): DataNodes, NodeManagers",
"updated_at": "2015-09-14T11:00:11",
"successful": true,
"total": 3,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T11:00:11",
"project_id": "808d5032ea0446889097723bfc8e919d",
"id": "9c8dc016-8c5b-4e80-9857-80c41f6bd971",
"step_type": "Plugin: start cluster",
"step_name": "Await DataNodes start up",
"updated_at": "2015-09-14T11:00:21",
"successful": true,
"total": 1,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T11:00:21",
"project_id": "808d5032ea0446889097723bfc8e919d",
"id": "c6327532-222b-416c-858f-73dbb32b8e97",
"step_type": "Plugin: start cluster",
"step_name": "Start the following process(es): HistoryServer",
"updated_at": "2015-09-14T11:00:27",
"successful": true,
"total": 1,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T10:57:41",
"project_id": "808d5032ea0446889097723bfc8e919d",
"id": "d3eca726-8b44-473a-ac29-fba45a893725",
"step_type": "Engine: create cluster",
"step_name": "Wait for instance accessibility",
"updated_at": "2015-09-14T10:58:14",
"successful": true,
"total": 4,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T10:58:14",
"project_id": "808d5032ea0446889097723bfc8e919d",
"id": "d7a875ff-64bf-41aa-882d-b5061c8ee152",
"step_type": "Engine: create cluster",
"step_name": "Mount volumes to instances",
"updated_at": "2015-09-14T10:58:15",
"successful": true,
"total": 0,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T10:59:55",
"project_id": "808d5032ea0446889097723bfc8e919d",
"id": "ded7d227-10b8-4cb0-ab6c-25da1462bb7a",
"step_type": "Plugin: start cluster",
"step_name": "Start the following process(es): ResourceManager",
"updated_at": "2015-09-14T11:00:00",
"successful": true,
"total": 1,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T10:59:38",
"project_id": "808d5032ea0446889097723bfc8e919d",
"id": "e1701ff5-930a-4212-945a-43515dfe24d1",
"step_type": "Plugin: start cluster",
"step_name": "Start the following process(es): NameNode",
"updated_at": "2015-09-14T10:59:54",
"successful": true,
"total": 1,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
},
{
"created_at": "2015-09-14T10:57:38",
"project_id": "808d5032ea0446889097723bfc8e919d",
"id": "eaf0ab1b-bf8f-48f0-8f2c-fa4f82f539b9",
"step_type": "Engine: create cluster",
"step_name": "Assign IPs",
"updated_at": "2015-09-14T10:57:41",
"successful": true,
"total": 4,
"cluster_id": "e172d86c-906d-418e-a29c-6189f53bfa42"
}
],
"plugin_version": "2.7.1",
"use_autoconfig": true,
"trust_id": null,
"description": null,
"created_at": "2015-09-14T10:57:11",
"is_protected": false,
"updated_at": "2015-09-14T11:01:15",
"is_transient": false,
"cluster_configs": {
"HDFS": {
"dfs.replication": 3
}
},
"anti_affinity": [],
"name": "vanilla-cluster",
"default_image_id": "4118a476-dfdc-4b0e-8d5c-463cba08e9ae",
"status": "Active"
}
]
}

View File

@ -0,0 +1,11 @@
{
"plugin_name": "vanilla",
"plugin_version": "2.6.0",
"cluster_template_id": "9951f86d-57ba-43d6-9cb0-14ed2ec7a6cf",
"default_image_id": "bc3c3d3c-2684-4bf8-a9fa-388fb71288a9",
"user_keypair_id": "test",
"name": "def-cluster",
"count": 2,
"cluster_configs": {},
"neutron_management_network": "7e31648b-4b2e-4f32-9b0a-113581c27076"
}

View File

@ -0,0 +1,6 @@
{
"clusters": [
"a007a3e7-658f-4568-b0f2-fe2fd5efc554",
"b012a6et-65hf-4566-b0f2-fe3fd7efc567"
]
}

View File

@ -0,0 +1,6 @@
{
"description": "This is hdfs input",
"url": "hdfs://test-master-node:8020/user/hadoop/input",
"type": "hdfs",
"name": "hdfs_input"
}

View File

@ -0,0 +1,14 @@
{
"data_source": {
"is_public": false,
"project_id": "9cd1314a0a31493282b6712b76a8fcda",
"is_protected": false,
"created_at": "2015-03-26 11:09:36.148464",
"id": "d7fffe9c-3b42-46a9-8be8-e98f586fa7a9",
"updated_at": null,
"name": "hdfs_input",
"description": "This is hdfs input",
"url": "hdfs://test-master-node:8020/user/hadoop/input",
"type": "hdfs"
}
}

View File

@ -0,0 +1,10 @@
{
"description": "This is input",
"url": "swift://container/text",
"credentials": {
"password": "swordfish",
"user": "dev"
},
"type": "swift",
"name": "swift_input"
}

View File

@ -0,0 +1,14 @@
{
"data_source": {
"is_public": false,
"project_id": "9cd1314a0a31493282b6712b76a8fcda",
"is_protected": false,
"created_at": "2015-03-26 11:18:10.691493",
"id": "953831f2-0852-49d8-ac71-af5805e25256",
"updated_at": null,
"name": "swift_input",
"description": "This is input",
"url": "swift://container/text",
"type": "swift"
}
}

View File

@ -0,0 +1,14 @@
{
"data_source": {
"is_public": false,
"project_id": "9cd1314a0a31493282b6712b76a8fcda",
"is_protected": false,
"created_at": "2015-03-26 11:18:10.691493",
"id": "953831f2-0852-49d8-ac71-af5805e25256",
"updated_at": null,
"name": "swift_input",
"description": "This is input",
"url": "swift://container/text",
"type": "swift"
}
}

View File

@ -0,0 +1,4 @@
{
"description": "This is public input",
"is_protected": true
}

View File

@ -0,0 +1,14 @@
{
"data_source": {
"is_public": true,
"project_id": "9cd1314a0a31493282b6712b76a8fcda",
"is_protected": false,
"created_at": "2015-09-15 12:32:24.847493",
"id": "953831f2-0852-49d8-ac71-af5805e25256",
"updated_at": "2015-09-15 12:34:42.597435",
"name": "swift_input",
"description": "This is public input",
"url": "swift://container/text",
"type": "swift"
}
}

View File

@ -0,0 +1,28 @@
{
"data_sources": [
{
"is_public": false,
"project_id": "9cd1314a0a31493282b6712b76a8fcda",
"is_protected": false,
"created_at": "2015-03-26 11:18:10",
"id": "953831f2-0852-49d8-ac71-af5805e25256",
"name": "swift_input",
"updated_at": null,
"description": "This is input",
"url": "swift://container/text",
"type": "swift"
},
{
"is_public": false,
"project_id": "9cd1314a0a31493282b6712b76a8fcda",
"is_protected": false,
"created_at": "2015-03-26 11:09:36",
"id": "d7fffe9c-3b42-46a9-8be8-e98f586fa7a9",
"name": "hdfs_input",
"updated_at": null,
"description": "This is hdfs input",
"url": "hdfs://test-master-node:8020/user/hadoop/input",
"type": "hdfs"
}
]
}

View File

@ -0,0 +1,72 @@
{
"status": "Error",
"neutron_management_network": "7e31648b-4b2e-4f32-9b0a-113581c27076",
"is_transient": false,
"description": "",
"user_keypair_id": "vgridnev",
"updated_at": "2015-03-31 14:10:59",
"plugin_name": "spark",
"provision_progress": [
{
"successful": false,
"project_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-03-31 14:10:20",
"step_type": "Engine: create cluster",
"updated_at": "2015-03-31 14:10:35",
"events": [
{
"instance_name": "sample-worker-spark-004",
"successful": false,
"created_at": "2015-03-31 14:10:35",
"updated_at": null,
"event_info": "Node sample-worker-spark-004 has error status\nError ID: 3e238c82-d1f5-4560-8ed8-691e923e16a0",
"instance_id": "b5ba5ba8-e9c1-47f7-9355-3ce0ec0e449d",
"node_group_id": "145cf2fb-dcdf-42af-a4b9-a4047d2919d4",
"step_id": "3f243c67-2c27-47c7-a0c0-0834ad17f8b6",
"id": "34afcfc7-bdb0-43cb-b142-283d560dc6ad"
},
{
"instance_name": "sample-worker-spark-001",
"successful": true,
"created_at": "2015-03-31 14:10:35",
"updated_at": null,
"event_info": null,
"instance_id": "c532ab71-38da-475a-95f8-f8eb93b8f1c2",
"node_group_id": "145cf2fb-dcdf-42af-a4b9-a4047d2919d4",
"step_id": "3f243c67-2c27-47c7-a0c0-0834ad17f8b6",
"id": "4ba50414-5216-4161-bc7a-12716122b99d"
}
],
"cluster_id": "c26ec982-ba6b-4d75-818c-a50240164af0",
"step_name": "Wait for instances to become active",
"total": 5,
"id": "3f243c67-2c27-47c7-a0c0-0834ad17f8b6"
},
{
"successful": true,
"project_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-03-31 14:10:12",
"step_type": "Engine: create cluster",
"updated_at": "2015-03-31 14:10:19",
"events": [],
"cluster_id": "c26ec982-ba6b-4d75-818c-a50240164af0",
"step_name": "Run instances",
"total": 5,
"id": "407ba50a-c799-46af-9dfb-6aa5f6ade426"
}
],
"anti_affinity": [],
"node_groups": [],
"management_public_key": "Sahara",
"status_description": "Creating cluster failed for the following reason(s): Node sample-worker-spark-004 has error status\nError ID: 3e238c82-d1f5-4560-8ed8-691e923e16a0",
"plugin_version": "1.0.0",
"id": "c26ec982-ba6b-4d75-1f8c-a50240164af0",
"trust_id": null,
"info": {},
"cluster_template_id": "5a9a09a3-9349-43bd-9058-16c401fad2d5",
"name": "sample",
"cluster_configs": {},
"created_at": "2015-03-31 14:10:07",
"default_image_id": "e6a6c5da-67be-4017-a7d2-81f466efe67e",
"project_id": "9cd1314a0a31493282b6712b76a8fcda"
}

View File

@ -0,0 +1,4 @@
{
"username": "ubuntu",
"description": "Ubuntu image for Hadoop 2.7.1"
}

View File

@ -0,0 +1,25 @@
{
"image": {
"updated": "2015-03-24T10:05:10Z",
"metadata": {
"_sahara_description": "Ubuntu image for Hadoop 2.7.1",
"_sahara_username": "ubuntu",
"_sahara_tag_vanilla": true,
"_sahara_tag_2.7.1": true
},
"id": "bb8d12b5-f9bb-49f0-aecb-739b8a9bec89",
"minDisk": 0,
"status": "ACTIVE",
"tags": [
"vanilla",
"2.7.1"
],
"minRam": 0,
"progress": 100,
"username": "ubuntu",
"created": "2015-02-03T10:28:39Z",
"name": "sahara-vanilla-2.7.1-ubuntu-14.04",
"description": "Ubuntu image for Hadoop 2.7.1",
"OS-EXT-IMG-SIZE:size": 1101856768
}
}

View File

@ -0,0 +1,24 @@
{
"image": {
"updated": "2015-02-03T10:29:32Z",
"metadata": {
"_sahara_username": "ubuntu",
"_sahara_tag_vanilla": true,
"_sahara_tag_2.6.0": true
},
"id": "bb8d12b5-f9bb-49f0-aecb-739b8a9bec89",
"minDisk": 0,
"status": "ACTIVE",
"tags": [
"vanilla",
"2.6.0"
],
"minRam": 0,
"progress": 100,
"username": "ubuntu",
"created": "2015-02-03T10:28:39Z",
"name": "sahara-vanilla-2.6.0-ubuntu-14.04",
"description": null,
"OS-EXT-IMG-SIZE:size": 1101856768
}
}

View File

@ -0,0 +1,7 @@
{
"tags": [
"vanilla",
"2.7.1",
"some_other_tag"
]
}

View File

@ -0,0 +1,27 @@
{
"image": {
"updated": "2015-03-24T10:18:33Z",
"metadata": {
"_sahara_tag_vanilla": true,
"_sahara_description": "Ubuntu image for Hadoop 2.7.1",
"_sahara_username": "ubuntu",
"_sahara_tag_some_other_tag": true,
"_sahara_tag_2.7.1": true
},
"id": "bb8d12b5-f9bb-49f0-aecb-739b8a9bec89",
"minDisk": 0,
"status": "ACTIVE",
"tags": [
"vanilla",
"some_other_tag",
"2.7.1"
],
"minRam": 0,
"progress": 100,
"username": "ubuntu",
"created": "2015-02-03T10:28:39Z",
"name": "sahara-vanilla-2.6.0-ubuntu-14.04",
"description": "Ubuntu image for Hadoop 2.7.1",
"OS-EXT-IMG-SIZE:size": 1101856768
}
}

View File

@ -0,0 +1,5 @@
{
"tags": [
"some_other_tag"
]
}

View File

@ -0,0 +1,25 @@
{
"image": {
"updated": "2015-03-24T10:19:28Z",
"metadata": {
"_sahara_description": "Ubuntu image for Hadoop 2.7.1",
"_sahara_username": "ubuntu",
"_sahara_tag_vanilla": true,
"_sahara_tag_2.7.1": true
},
"id": "bb8d12b5-f9bb-49f0-aecb-739b8a9bec89",
"minDisk": 0,
"status": "ACTIVE",
"tags": [
"vanilla",
"2.7.1"
],
"minRam": 0,
"progress": 100,
"username": "ubuntu",
"created": "2015-02-03T10:28:39Z",
"name": "sahara-vanilla-2.7.1-ubuntu-14.04",
"description": "Ubuntu image for Hadoop 2.7.1",
"OS-EXT-IMG-SIZE:size": 1101856768
}
}

View File

@ -0,0 +1,48 @@
{
"images": [
{
"name": "ubuntu-vanilla-2.7.1",
"id": "4118a476-dfdc-4b0e-8d5c-463cba08e9ae",
"created": "2015-08-06T08:17:14Z",
"metadata": {
"_sahara_tag_2.7.1": true,
"_sahara_username": "ubuntu",
"_sahara_tag_vanilla": true
},
"username": "ubuntu",
"progress": 100,
"OS-EXT-IMG-SIZE:size": 998716928,
"status": "ACTIVE",
"minDisk": 0,
"tags": [
"vanilla",
"2.7.1"
],
"updated": "2015-09-04T09:35:09Z",
"minRam": 0,
"description": null
},
{
"name": "cdh-latest",
"id": "ff74035b-9da7-4edf-981d-57f270ed337d",
"created": "2015-09-04T11:56:44Z",
"metadata": {
"_sahara_username": "ubuntu",
"_sahara_tag_5.4.0": true,
"_sahara_tag_cdh": true
},
"username": "ubuntu",
"progress": 100,
"OS-EXT-IMG-SIZE:size": 3281453056,
"status": "ACTIVE",
"minDisk": 0,
"tags": [
"5.4.0",
"cdh"
],
"updated": "2015-09-04T12:46:42Z",
"minRam": 0,
"description": null
}
]
}

View File

@ -0,0 +1,9 @@
{
"url": "swift://container/jar-example.jar",
"name": "jar-example.jar",
"description": "This is a job binary",
"extra": {
"password": "swordfish",
"user": "admin"
}
}

View File

@ -0,0 +1,13 @@
{
"job_binary": {
"is_public": false,
"description": "This is a job binary",
"url": "swift://container/jar-example.jar",
"project_id": "11587919cc534bcbb1027a161c82cf58",
"created_at": "2013-10-15 14:49:20.106452",
"id": "07f86352-ee8a-4b08-b737-d705ded5ff9c",
"updated_at": null,
"name": "jar-example.jar",
"is_protected": false
}
}

View File

@ -0,0 +1,37 @@
{
"binaries": [
{
"is_public": false,
"description": "",
"url": "internal-db://d2498cbf-4589-484a-a814-81436c18beb3",
"project_id": "11587919cc534bcbb1027a161c82cf58",
"created_at": "2013-10-15 12:36:59.375060",
"updated_at": null,
"id": "84248975-3c82-4206-a58d-6e7fb3a563fd",
"name": "example.pig",
"is_protected": false
},
{
"is_public": false,
"description": "",
"url": "internal-db://22f1d87a-23c8-483e-a0dd-cb4a16dde5f9",
"project_id": "11587919cc534bcbb1027a161c82cf58",
"created_at": "2013-10-15 12:43:52.265899",
"updated_at": null,
"id": "508fc62d-1d58-4412-b603-bdab307bb926",
"name": "udf.jar",
"is_protected": false
},
{
"is_public": false,
"description": "",
"url": "swift://container/jar-example.jar",
"project_id": "11587919cc534bcbb1027a161c82cf58",
"created_at": "2013-10-15 14:25:04.970513",
"updated_at": null,
"id": "a716a9cd-9add-4b12-b1b6-cdb71aaef350",
"name": "jar-example.jar",
"is_protected": false
}
]
}

View File

@ -0,0 +1,3 @@
A = load '$INPUT' using PigStorage(':') as (fruit: chararray);
B = foreach A generate com.hadoopbook.pig.Trim(fruit);
store B into '$OUTPUT' USING PigStorage();

View File

@ -0,0 +1,13 @@
{
"job_binary": {
"is_public": false,
"description": "an example jar file",
"url": "swift://container/jar-example.jar",
"project_id": "11587919cc534bcbb1027a161c82cf58",
"created_at": "2013-10-15 14:25:04.970513",
"updated_at": null,
"id": "a716a9cd-9add-4b12-b1b6-cdb71aaef350",
"name": "jar-example.jar",
"is_protected": false
}
}

View File

@ -0,0 +1,5 @@
{
"url": "swift://container/new-jar-example.jar",
"name": "new-jar-example.jar",
"description": "This is a new job binary"
}

View File

@ -0,0 +1,13 @@
{
"job_binary": {
"is_public": false,
"description": "This is a new job binary",
"url": "swift://container/new-jar-example.jar",
"project_id": "11587919cc534bcbb1027a161c82cf58",
"created_at": "2015-09-15 12:42:51.421542",
"updated_at": null,
"id": "b713d7ad-4add-4f12-g1b6-cdg71aaef350",
"name": "new-jar-example.jar",
"is_protected": false
}
}

View File

@ -0,0 +1,11 @@
{
"description": "This is pig job example",
"mains": [
"90d9d5ec-11aa-48bd-bc8c-34936ce0db6e"
],
"libs": [
"320a2ca7-25fd-4b48-9bc3-4fb1b6c4ff27"
],
"type": "Pig",
"name": "pig-job-example"
}

View File

@ -0,0 +1,35 @@
{
"job_template": {
"is_public": false,
"project_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-03-27 08:48:38.630827",
"id": "71defc8f-d005-484f-9d86-1aedf644d1ef",
"name": "pig-job-example",
"description": "This is pig job example",
"interface": [],
"libs": [
{
"project_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-02-10 14:25:53",
"id": "320a2ca7-25fd-4b48-9bc3-4fb1b6c4ff27",
"name": "binary-job",
"updated_at": null,
"description": "",
"url": "internal-db://c6a925fa-ac1d-4b2e-b88a-7054e1927521"
}
],
"type": "Pig",
"is_protected": false,
"mains": [
{
"project_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-02-03 10:47:51",
"id": "90d9d5ec-11aa-48bd-bc8c-34936ce0db6e",
"name": "pig",
"updated_at": null,
"description": "",
"url": "internal-db://872878f6-72ea-44db-8d1d-e6a6396d2df0"
}
]
}
}

View File

@ -0,0 +1,26 @@
{
"job_template": {
"is_public": false,
"project_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-02-10 14:25:48",
"id": "1a674c31-9aaa-4d07-b844-2bf200a1b836",
"name": "Edp-test-job",
"updated_at": null,
"description": "",
"interface": [],
"libs": [
{
"project_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-02-10 14:25:48",
"id": "0ff4ac10-94a4-4e25-9ac9-603afe27b100",
"name": "binary-job.jar",
"updated_at": null,
"description": "",
"url": "swift://Edp-test-c71e6bce.sahara/binary-job.jar"
}
],
"type": "MapReduce",
"mains": [],
"is_protected": false
}
}

View File

@ -0,0 +1,4 @@
{
"description": "This is public pig job example",
"name": "public-pig-job-example"
}

View File

@ -0,0 +1,26 @@
{
"job_template": {
"is_public": false,
"project_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-02-10 14:25:48",
"id": "1a674c31-9aaa-4d07-b844-2bf200a1b836",
"name": "public-pig-job-example",
"updated_at": null,
"description": "This is public pig job example",
"interface": [],
"libs": [
{
"project_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-02-10 14:25:48",
"id": "0ff4ac10-94a4-4e25-9ac9-603afe27b100",
"name": "binary-job.jar",
"updated_at": null,
"description": "",
"url": "swift://Edp-test-c71e6bce.sahara/binary-job.jar"
}
],
"type": "MapReduce",
"mains": [],
"is_protected": false
}
}

View File

@ -0,0 +1,66 @@
{
"job_templates": [
{
"is_public": false,
"project_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-02-10 14:25:48",
"id": "1a674c31-9aaa-4d07-b844-2bf200a1b836",
"name": "Edp-test-job-3d60854e",
"updated_at": null,
"description": "",
"interface": [],
"libs": [
{
"project_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-02-10 14:25:48",
"id": "0ff4ac10-94a4-4e25-9ac9-603afe27b100",
"name": "binary-job-339c2d1a.jar",
"updated_at": null,
"description": "",
"url": "swift://Edp-test-c71e6bce.sahara/binary-job-339c2d1a.jar"
}
],
"type": "MapReduce",
"mains": [],
"is_protected": false
},
{
"is_public": false,
"project_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-02-10 14:25:44",
"id": "4d1f3759-3497-4927-8352-910bacf24e62",
"name": "Edp-test-job-6b6953c8",
"updated_at": null,
"description": "",
"interface": [],
"libs": [
{
"project_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-02-10 14:25:44",
"id": "e0d47800-4ac1-4d63-a2e1-c92d669a44e2",
"name": "binary-job-6f21a2f8.jar",
"updated_at": null,
"description": "",
"url": "swift://Edp-test-b409ec68.sahara/binary-job-6f21a2f8.jar"
}
],
"type": "Pig",
"mains": [
{
"project_id": "9cd1314a0a31493282b6712b76a8fcda",
"created_at": "2015-02-10 14:25:44",
"id": "e073e896-f123-4b76-995f-901d786262df",
"name": "binary-job-d4f8bd75.pig",
"updated_at": null,
"description": "",
"url": "swift://Edp-test-b409ec68.sahara/binary-job-d4f8bd75.pig"
}
],
"is_protected": false
}
],
"markers": {
"prev": null,
"next": "c53832da-6e7b-449e-a166-9f9ce1718d03"
}
}

View File

@ -0,0 +1,209 @@
{
"job_types": [
{
"plugins": [
{
"description": "The Apache Vanilla plugin provides the ability to launch upstream Vanilla Apache Hadoop cluster without any management consoles. It can also deploy the Oozie component.",
"versions": {
"1.2.1": {},
"2.6.0": {}
},
"title": "Vanilla Apache Hadoop",
"name": "vanilla"
},
{
"description": "The Hortonworks Sahara plugin automates the deployment of the Hortonworks Data Platform (HDP) on OpenStack.",
"versions": {
"1.3.2": {},
"2.0.6": {}
},
"title": "Hortonworks Data Platform",
"name": "hdp"
},
{
"description": "The Cloudera Sahara plugin provides the ability to launch the Cloudera distribution of Apache Hadoop (CDH) with Cloudera Manager management console.",
"versions": {
"5": {},
"5.3.0": {}
},
"title": "Cloudera Plugin",
"name": "cdh"
}
],
"name": "Hive"
},
{
"plugins": [
{
"description": "The Apache Vanilla plugin provides the ability to launch upstream Vanilla Apache Hadoop cluster without any management consoles. It can also deploy the Oozie component.",
"versions": {
"1.2.1": {},
"2.6.0": {}
},
"title": "Vanilla Apache Hadoop",
"name": "vanilla"
},
{
"description": "The Hortonworks Sahara plugin automates the deployment of the Hortonworks Data Platform (HDP) on OpenStack.",
"versions": {
"1.3.2": {},
"2.0.6": {}
},
"title": "Hortonworks Data Platform",
"name": "hdp"
},
{
"description": "The Cloudera Sahara plugin provides the ability to launch the Cloudera distribution of Apache Hadoop (CDH) with Cloudera Manager management console.",
"versions": {
"5": {},
"5.3.0": {}
},
"title": "Cloudera Plugin",
"name": "cdh"
}
],
"name": "Java"
},
{
"plugins": [
{
"description": "The Apache Vanilla plugin provides the ability to launch upstream Vanilla Apache Hadoop cluster without any management consoles. It can also deploy the Oozie component.",
"versions": {
"1.2.1": {},
"2.6.0": {}
},
"title": "Vanilla Apache Hadoop",
"name": "vanilla"
},
{
"description": "The Hortonworks Sahara plugin automates the deployment of the Hortonworks Data Platform (HDP) on OpenStack.",
"versions": {
"1.3.2": {},
"2.0.6": {}
},
"title": "Hortonworks Data Platform",
"name": "hdp"
},
{
"description": "The Cloudera Sahara plugin provides the ability to launch the Cloudera distribution of Apache Hadoop (CDH) with Cloudera Manager management console.",
"versions": {
"5": {},
"5.3.0": {}
},
"title": "Cloudera Plugin",
"name": "cdh"
}
],
"name": "MapReduce"
},
{
"plugins": [
{
"description": "The Apache Vanilla plugin provides the ability to launch upstream Vanilla Apache Hadoop cluster without any management consoles. It can also deploy the Oozie component.",
"versions": {
"1.2.1": {},
"2.6.0": {}
},
"title": "Vanilla Apache Hadoop",
"name": "vanilla"
},
{
"description": "The Hortonworks Sahara plugin automates the deployment of the Hortonworks Data Platform (HDP) on OpenStack.",
"versions": {
"1.3.2": {},
"2.0.6": {}
},
"title": "Hortonworks Data Platform",
"name": "hdp"
},
{
"description": "The Cloudera Sahara plugin provides the ability to launch the Cloudera distribution of Apache Hadoop (CDH) with Cloudera Manager management console.",
"versions": {
"5": {},
"5.3.0": {}
},
"title": "Cloudera Plugin",
"name": "cdh"
}
],
"name": "MapReduce.Streaming"
},
{
"plugins": [
{
"description": "The Apache Vanilla plugin provides the ability to launch upstream Vanilla Apache Hadoop cluster without any management consoles. It can also deploy the Oozie component.",
"versions": {
"1.2.1": {},
"2.6.0": {}
},
"title": "Vanilla Apache Hadoop",
"name": "vanilla"
},
{
"description": "The Hortonworks Sahara plugin automates the deployment of the Hortonworks Data Platform (HDP) on OpenStack.",
"versions": {
"1.3.2": {},
"2.0.6": {}
},
"title": "Hortonworks Data Platform",
"name": "hdp"
},
{
"description": "The Cloudera Sahara plugin provides the ability to launch the Cloudera distribution of Apache Hadoop (CDH) with Cloudera Manager management console.",
"versions": {
"5": {},
"5.3.0": {}
},
"title": "Cloudera Plugin",
"name": "cdh"
}
],
"name": "Pig"
},
{
"plugins": [
{
"description": "The Apache Vanilla plugin provides the ability to launch upstream Vanilla Apache Hadoop cluster without any management consoles. It can also deploy the Oozie component.",
"versions": {
"1.2.1": {},
"2.6.0": {}
},
"title": "Vanilla Apache Hadoop",
"name": "vanilla"
},
{
"description": "The Hortonworks Sahara plugin automates the deployment of the Hortonworks Data Platform (HDP) on OpenStack.",
"versions": {
"1.3.2": {},
"2.0.6": {}
},
"title": "Hortonworks Data Platform",
"name": "hdp"
},
{
"description": "The Cloudera Sahara plugin provides the ability to launch the Cloudera distribution of Apache Hadoop (CDH) with Cloudera Manager management console.",
"versions": {
"5": {},
"5.3.0": {}
},
"title": "Cloudera Plugin",
"name": "cdh"
}
],
"name": "Shell"
},
{
"plugins": [
{
"description": "This plugin provides an ability to launch Spark on Hadoop CDH cluster without any management consoles.",
"versions": {
"1.0.0": {}
},
"title": "Apache Spark",
"name": "spark"
}
],
"name": "Spark"
}
]
}

View File

@ -0,0 +1,120 @@
{
"job": {
"job_configs": {
"configs": {
"mapred.reduce.tasks": "1",
"mapred.map.tasks": "1"
},
"args": [
"arg1",
"arg2"
],
"params": {
"param2": "value2",
"param1": "value1"
}
},
"is_protected": false,
"input_id": "3e1bc8e6-8c69-4749-8e52-90d9341d15bc",
"job_id": "310b0fc6-e1db-408e-8798-312e7500f3ac",
"cluster_id": "811e1134-666f-4c48-bc92-afb5b10c9d8c",
"created_at": "2015-09-15T09:49:24",
"end_time": "2015-09-15T12:50:46",
"output_id": "52146b52-6540-4aac-a024-fee253cf52a9",
"is_public": false,
"updated_at": "2015-09-15T09:50:46",
"return_code": null,
"data_source_urls": {
"3e1bc8e6-8c69-4749-8e52-90d9341d15bc": "swift://ap-cont/input",
"52146b52-6540-4aac-a024-fee253cf52a9": "swift://ap-cont/output"
},
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"start_time": "2015-09-15T12:49:43",
"id": "20da9edb-12ce-4b45-a473-41baeefef997",
"oozie_job_id": "0000001-150915094349962-oozie-hado-W",
"info": {
"user": "hadoop",
"actions": [
{
"name": ":start:",
"trackerUri": "-",
"externalStatus": "OK",
"status": "OK",
"externalId": "-",
"transition": "job-node",
"data": null,
"endTime": "Tue, 15 Sep 2015 09:49:59 GMT",
"errorCode": null,
"id": "0000001-150915094349962-oozie-hado-W@:start:",
"consoleUrl": "-",
"errorMessage": null,
"toString": "Action name[:start:] status[OK]",
"stats": null,
"type": ":START:",
"retries": 0,
"startTime": "Tue, 15 Sep 2015 09:49:59 GMT",
"externalChildIDs": null,
"cred": "null"
},
{
"name": "job-node",
"trackerUri": "http://172.18.168.119:8032",
"externalStatus": "FAILED/KILLED",
"status": "ERROR",
"externalId": "job_1442310173665_0002",
"transition": "fail",
"data": null,
"endTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"errorCode": "JA018",
"id": "0000001-150915094349962-oozie-hado-W@job-node",
"consoleUrl": "http://ap-cluster-all-0:8088/proxy/application_1442310173665_0002/",
"errorMessage": "Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]",
"toString": "Action name[job-node] status[ERROR]",
"stats": null,
"type": "pig",
"retries": 0,
"startTime": "Tue, 15 Sep 2015 09:49:59 GMT",
"externalChildIDs": null,
"cred": "null"
},
{
"name": "fail",
"trackerUri": "-",
"externalStatus": "OK",
"status": "OK",
"externalId": "-",
"transition": null,
"data": null,
"endTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"errorCode": "E0729",
"id": "0000001-150915094349962-oozie-hado-W@fail",
"consoleUrl": "-",
"errorMessage": "Workflow failed, error message[Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]]",
"toString": "Action name[fail] status[OK]",
"stats": null,
"type": ":KILL:",
"retries": 0,
"startTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"externalChildIDs": null,
"cred": "null"
}
],
"createdTime": "Tue, 15 Sep 2015 09:49:58 GMT",
"status": "KILLED",
"group": null,
"externalId": null,
"acl": null,
"run": 0,
"appName": "job-wf",
"parentId": null,
"conf": "<configuration>\r\n <property>\r\n <name>user.name</name>\r\n <value>hadoop</value>\r\n </property>\r\n <property>\r\n <name>oozie.use.system.libpath</name>\r\n <value>true</value>\r\n </property>\r\n <property>\r\n <name>mapreduce.job.user.name</name>\r\n <value>hadoop</value>\r\n </property>\r\n <property>\r\n <name>nameNode</name>\r\n <value>hdfs://ap-cluster-all-0:9000</value>\r\n </property>\r\n <property>\r\n <name>jobTracker</name>\r\n <value>http://172.18.168.119:8032</value>\r\n </property>\r\n <property>\r\n <name>oozie.wf.application.path</name>\r\n <value>hdfs://ap-cluster-all-0:9000/user/hadoop/pig-job-example/3038025d-9974-4993-a778-26a074cdfb8d/workflow.xml</value>\r\n </property>\r\n</configuration>",
"id": "0000001-150915094349962-oozie-hado-W",
"startTime": "Tue, 15 Sep 2015 09:49:59 GMT",
"appPath": "hdfs://ap-cluster-all-0:9000/user/hadoop/pig-job-example/3038025d-9974-4993-a778-26a074cdfb8d/workflow.xml",
"endTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"toString": "Workflow id[0000001-150915094349962-oozie-hado-W] status[KILLED]",
"lastModTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"consoleUrl": "http://ap-cluster-all-0.novalocal:11000/oozie?job=0000001-150915094349962-oozie-hado-W"
}
}
}

View File

@ -0,0 +1,20 @@
{
"cluster_id": "811e1134-666f-4c48-bc92-afb5b10c9d8c",
"job_template_id": "548ea8d4-a5sd-33a4-bt22-asf4n87a8e2dh",
"input_id": "3e1bc8e6-8c69-4749-8e52-90d9341d15bc",
"output_id": "52146b52-6540-4aac-a024-fee253cf52a9",
"job_configs": {
"configs": {
"mapred.map.tasks": "1",
"mapred.reduce.tasks": "1"
},
"args": [
"arg1",
"arg2"
],
"params": {
"param2": "value2",
"param1": "value1"
}
}
}

View File

@ -0,0 +1,30 @@
{
"job": {
"input_id": "3e1bc8e6-8c69-4749-8e52-90d9341d15bc",
"is_protected": false,
"job_id": "310b0fc6-e1db-408e-8798-312e7500f3ac",
"cluster_id": "811e1134-666f-4c48-bc92-afb5b10c9d8c",
"output_id": "52146b52-6540-4aac-a024-fee253cf52a9",
"created_at": "2015-09-15T09:49:24",
"is_public": false,
"id": "20da9edb-12ce-4b45-a473-41baeefef997",
"project_id": "808d5032ea0446889097723bfc8e919d",
"job_configs": {
"configs": {
"mapred.reduce.tasks": "1",
"mapred.map.tasks": "1"
},
"args": [
"arg1",
"arg2"
],
"params": {
"param2": "value2",
"param1": "value1"
}
},
"info": {
"status": "PENDING"
}
}
}

View File

@ -0,0 +1,3 @@
{
"is_public": true
}

View File

@ -0,0 +1,120 @@
{
"job: {
"job_configs": {
"configs": {
"mapred.reduce.tasks": "1",
"mapred.map.tasks": "1"
},
"args": [
"arg1",
"arg2"
],
"params": {
"param2": "value2",
"param1": "value1"
}
},
"is_protected": false,
"input_id": "3e1bc8e6-8c69-4749-8e52-90d9341d15bc",
"job_id": "310b0fc6-e1db-408e-8798-312e7500f3ac",
"cluster_id": "811e1134-666f-4c48-bc92-afb5b10c9d8c",
"created_at": "2015-09-15T09:49:24",
"end_time": "2015-09-15T12:50:46",
"output_id": "52146b52-6540-4aac-a024-fee253cf52a9",
"is_public": true,
"updated_at": "2015-09-15T09:50:46",
"return_code": null,
"data_source_urls": {
"3e1bc8e6-8c69-4749-8e52-90d9341d15bc": "swift://ap-cont/input",
"52146b52-6540-4aac-a024-fee253cf52a9": "swift://ap-cont/output"
},
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"start_time": "2015-09-15T12:49:43",
"id": "20da9edb-12ce-4b45-a473-41baeefef997",
"oozie_job_id": "0000001-150915094349962-oozie-hado-W",
"info": {
"user": "hadoop",
"actions": [
{
"name": ":start:",
"trackerUri": "-",
"externalStatus": "OK",
"status": "OK",
"externalId": "-",
"transition": "job-node",
"data": null,
"endTime": "Tue, 15 Sep 2015 09:49:59 GMT",
"errorCode": null,
"id": "0000001-150915094349962-oozie-hado-W@:start:",
"consoleUrl": "-",
"errorMessage": null,
"toString": "Action name[:start:] status[OK]",
"stats": null,
"type": ":START:",
"retries": 0,
"startTime": "Tue, 15 Sep 2015 09:49:59 GMT",
"externalChildIDs": null,
"cred": "null"
},
{
"name": "job-node",
"trackerUri": "http://172.18.168.119:8032",
"externalStatus": "FAILED/KILLED",
"status": "ERROR",
"externalId": "job_1442310173665_0002",
"transition": "fail",
"data": null,
"endTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"errorCode": "JA018",
"id": "0000001-150915094349962-oozie-hado-W@job-node",
"consoleUrl": "http://ap-cluster-all-0:8088/proxy/application_1442310173665_0002/",
"errorMessage": "Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]",
"toString": "Action name[job-node] status[ERROR]",
"stats": null,
"type": "pig",
"retries": 0,
"startTime": "Tue, 15 Sep 2015 09:49:59 GMT",
"externalChildIDs": null,
"cred": "null"
},
{
"name": "fail",
"trackerUri": "-",
"externalStatus": "OK",
"status": "OK",
"externalId": "-",
"transition": null,
"data": null,
"endTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"errorCode": "E0729",
"id": "0000001-150915094349962-oozie-hado-W@fail",
"consoleUrl": "-",
"errorMessage": "Workflow failed, error message[Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]]",
"toString": "Action name[fail] status[OK]",
"stats": null,
"type": ":KILL:",
"retries": 0,
"startTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"externalChildIDs": null,
"cred": "null"
}
],
"createdTime": "Tue, 15 Sep 2015 09:49:58 GMT",
"status": "KILLED",
"group": null,
"externalId": null,
"acl": null,
"run": 0,
"appName": "job-wf",
"parentId": null,
"conf": "<configuration>\r\n <property>\r\n <name>user.name</name>\r\n <value>hadoop</value>\r\n </property>\r\n <property>\r\n <name>oozie.use.system.libpath</name>\r\n <value>true</value>\r\n </property>\r\n <property>\r\n <name>mapreduce.job.user.name</name>\r\n <value>hadoop</value>\r\n </property>\r\n <property>\r\n <name>nameNode</name>\r\n <value>hdfs://ap-cluster-all-0:9000</value>\r\n </property>\r\n <property>\r\n <name>jobTracker</name>\r\n <value>http://172.18.168.119:8032</value>\r\n </property>\r\n <property>\r\n <name>oozie.wf.application.path</name>\r\n <value>hdfs://ap-cluster-all-0:9000/user/hadoop/pig-job-example/3038025d-9974-4993-a778-26a074cdfb8d/workflow.xml</value>\r\n </property>\r\n</configuration>",
"id": "0000001-150915094349962-oozie-hado-W",
"startTime": "Tue, 15 Sep 2015 09:49:59 GMT",
"appPath": "hdfs://ap-cluster-all-0:9000/user/hadoop/pig-job-example/3038025d-9974-4993-a778-26a074cdfb8d/workflow.xml",
"endTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"toString": "Workflow id[0000001-150915094349962-oozie-hado-W] status[KILLED]",
"lastModTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"consoleUrl": "http://ap-cluster-all-0.novalocal:11000/oozie?job=0000001-150915094349962-oozie-hado-W"
}
}
}

View File

@ -0,0 +1,122 @@
{
"jobs": [
{
"job_configs": {
"configs": {
"mapred.reduce.tasks": "1",
"mapred.map.tasks": "1"
},
"args": [
"arg1",
"arg2"
],
"params": {
"param2": "value2",
"param1": "value1"
}
},
"is_protected": false,
"input_id": "3e1bc8e6-8c69-4749-8e52-90d9341d15bc",
"job_id": "310b0fc6-e1db-408e-8798-312e7500f3ac",
"cluster_id": "811e1134-666f-4c48-bc92-afb5b10c9d8c",
"created_at": "2015-09-15T09:49:24",
"end_time": "2015-09-15T12:50:46",
"output_id": "52146b52-6540-4aac-a024-fee253cf52a9",
"is_public": false,
"updated_at": "2015-09-15T09:50:46",
"return_code": null,
"data_source_urls": {
"3e1bc8e6-8c69-4749-8e52-90d9341d15bc": "swift://ap-cont/input",
"52146b52-6540-4aac-a024-fee253cf52a9": "swift://ap-cont/output"
},
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"start_time": "2015-09-15T12:49:43",
"id": "20da9edb-12ce-4b45-a473-41baeefef997",
"oozie_job_id": "0000001-150915094349962-oozie-hado-W",
"info": {
"user": "hadoop",
"actions": [
{
"name": ":start:",
"trackerUri": "-",
"externalStatus": "OK",
"status": "OK",
"externalId": "-",
"transition": "job-node",
"data": null,
"endTime": "Tue, 15 Sep 2015 09:49:59 GMT",
"errorCode": null,
"id": "0000001-150915094349962-oozie-hado-W@:start:",
"consoleUrl": "-",
"errorMessage": null,
"toString": "Action name[:start:] status[OK]",
"stats": null,
"type": ":START:",
"retries": 0,
"startTime": "Tue, 15 Sep 2015 09:49:59 GMT",
"externalChildIDs": null,
"cred": "null"
},
{
"name": "job-node",
"trackerUri": "http://172.18.168.119:8032",
"externalStatus": "FAILED/KILLED",
"status": "ERROR",
"externalId": "job_1442310173665_0002",
"transition": "fail",
"data": null,
"endTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"errorCode": "JA018",
"id": "0000001-150915094349962-oozie-hado-W@job-node",
"consoleUrl": "http://ap-cluster-all-0:8088/proxy/application_1442310173665_0002/",
"errorMessage": "Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]",
"toString": "Action name[job-node] status[ERROR]",
"stats": null,
"type": "pig",
"retries": 0,
"startTime": "Tue, 15 Sep 2015 09:49:59 GMT",
"externalChildIDs": null,
"cred": "null"
},
{
"name": "fail",
"trackerUri": "-",
"externalStatus": "OK",
"status": "OK",
"externalId": "-",
"transition": null,
"data": null,
"endTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"errorCode": "E0729",
"id": "0000001-150915094349962-oozie-hado-W@fail",
"consoleUrl": "-",
"errorMessage": "Workflow failed, error message[Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]]",
"toString": "Action name[fail] status[OK]",
"stats": null,
"type": ":KILL:",
"retries": 0,
"startTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"externalChildIDs": null,
"cred": "null"
}
],
"createdTime": "Tue, 15 Sep 2015 09:49:58 GMT",
"status": "KILLED",
"group": null,
"externalId": null,
"acl": null,
"run": 0,
"appName": "job-wf",
"parentId": null,
"conf": "<configuration>\r\n <property>\r\n <name>user.name</name>\r\n <value>hadoop</value>\r\n </property>\r\n <property>\r\n <name>oozie.use.system.libpath</name>\r\n <value>true</value>\r\n </property>\r\n <property>\r\n <name>mapreduce.job.user.name</name>\r\n <value>hadoop</value>\r\n </property>\r\n <property>\r\n <name>nameNode</name>\r\n <value>hdfs://ap-cluster-all-0:9000</value>\r\n </property>\r\n <property>\r\n <name>jobTracker</name>\r\n <value>http://172.18.168.119:8032</value>\r\n </property>\r\n <property>\r\n <name>oozie.wf.application.path</name>\r\n <value>hdfs://ap-cluster-all-0:9000/user/hadoop/pig-job-example/3038025d-9974-4993-a778-26a074cdfb8d/workflow.xml</value>\r\n </property>\r\n</configuration>",
"id": "0000001-150915094349962-oozie-hado-W",
"startTime": "Tue, 15 Sep 2015 09:49:59 GMT",
"appPath": "hdfs://ap-cluster-all-0:9000/user/hadoop/pig-job-example/3038025d-9974-4993-a778-26a074cdfb8d/workflow.xml",
"endTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"toString": "Workflow id[0000001-150915094349962-oozie-hado-W] status[KILLED]",
"lastModTime": "Tue, 15 Sep 2015 09:50:17 GMT",
"consoleUrl": "http://ap-cluster-all-0.novalocal:11000/oozie?job=0000001-150915094349962-oozie-hado-W"
}
}
]
}

View File

@ -0,0 +1,13 @@
{
"plugin_name": "vanilla",
"plugin_version": "2.7.1",
"node_processes": [
"namenode",
"resourcemanager",
"oozie",
"historyserver"
],
"name": "master",
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"flavor_id": "2"
}

View File

@ -0,0 +1,33 @@
{
"node_group_template": {
"is_public": false,
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {},
"auto_security_group": false,
"is_default": false,
"availability_zone": null,
"plugin_name": "vanilla",
"is_protected": false,
"flavor_id": "2",
"id": "0bb9f1a4-0c44-4dc5-9452-6741c62ed9ae",
"plugin_version": "2.7.1",
"use_autoconfig": true,
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:20:11",
"security_groups": null,
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "master",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"namenode",
"resourcemanager",
"oozie",
"historyserver"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"volume_type": null
}
}

View File

@ -0,0 +1,37 @@
{
"node_group_template": {
"is_public": false,
"image_id": null,
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {},
"auto_security_group": false,
"is_default": false,
"availability_zone": null,
"plugin_name": "vanilla",
"flavor_id": "2",
"id": "0bb9f1a4-0c44-4dc5-9452-6741c62ed9ae",
"description": null,
"plugin_version": "2.7.1",
"use_autoconfig": true,
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:20:11",
"is_protected": false,
"updated_at": null,
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "master",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"namenode",
"resourcemanager",
"oozie",
"historyserver"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"security_groups": null,
"volume_type": null
}
}

View File

@ -0,0 +1,10 @@
{
"plugin_name": "vanilla",
"plugin_version": "2.7.1",
"node_processes": [
"datanode"
],
"name": "new",
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"flavor_id": "2"
}

View File

@ -0,0 +1,30 @@
{
"node_group_template": {
"is_public": false,
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {},
"auto_security_group": false,
"is_default": false,
"availability_zone": null,
"plugin_name": "vanilla",
"is_protected": false,
"flavor_id": "2",
"id": "0bb9f1a4-0c44-4dc5-9452-6741c62ed9ae",
"plugin_version": "2.7.1",
"use_autoconfig": true,
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:20:11",
"security_groups": null,
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "new",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"datanode"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"volume_type": null
}
}

View File

@ -0,0 +1,76 @@
{
"node_group_templates": [
{
"is_public": false,
"image_id": null,
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {},
"auto_security_group": false,
"is_default": false,
"availability_zone": null,
"plugin_name": "vanilla",
"flavor_id": "2",
"id": "0bb9f1a4-0c44-4dc5-9452-6741c62ed9ae",
"description": null,
"plugin_version": "2.7.1",
"use_autoconfig": true,
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:20:11",
"is_protected": false,
"updated_at": null,
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "master",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"namenode",
"resourcemanager",
"oozie",
"historyserver"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"security_groups": null,
"volume_type": null
},
{
"is_public": false,
"image_id": null,
"tenant_id": "808d5032ea0446889097723bfc8e919d",
"shares": null,
"floating_ip_pool": "033debed-aeb8-488c-b7d0-adb74c61faa5",
"node_configs": {},
"auto_security_group": false,
"is_default": false,
"availability_zone": null,
"plugin_name": "vanilla",
"flavor_id": "2",
"id": "846edb31-add5-46e6-a4ee-a4c339f99251",
"description": null,
"hadoop_version": "2.7.1",
"use_autoconfig": true,
"volumes_availability_zone": null,
"created_at": "2015-09-14T10:27:00",
"is_protected": false,
"updated_at": null,
"volumes_per_node": 0,
"is_proxy_gateway": false,
"name": "worker",
"volume_mount_prefix": "/volumes/disk",
"node_processes": [
"datanode",
"nodemanager"
],
"volumes_size": 0,
"volume_local_to_instance": false,
"security_groups": null,
"volume_type": null
}
],
"markers": {
"prev":"39dfc852-8588-4b61-8d2b-eb08a67ab240",
"next":"eaa0bd97-ab54-43df-83ab-77a9774d7358"
}
}

View File

@ -0,0 +1,12 @@
{
"plugin": {
"name": "vanilla",
"versions": [
"1.2.1",
"2.4.1",
"2.6.0"
],
"title": "Vanilla Apache Hadoop",
"description": "The Apache Vanilla plugin provides the ability to launch upstream Vanilla Apache Hadoop cluster without any management consoles. It can also deploy the Oozie component."
}
}

View File

@ -0,0 +1,7 @@
{
"plugin_labels": {
"enabled": {
"status": false
}
}
}

View File

@ -0,0 +1,32 @@
{
"plugin": {
"plugin_labels": {
"hidden": {
"status": true,
"mutable": true,
"description": "Existence of plugin or its version is hidden, but still can be used for cluster creation by CLI and directly by client."
},
"enabled": {
"status": false,
"mutable": true,
"description": "Plugin or its version is enabled and can be used by user."
}
},
"description": "It's a fake plugin that aimed to work on the CirrOS images. It doesn't install Hadoop. It's needed to be able to test provisioning part of Sahara codebase itself.",
"versions": [
"0.1"
],
"tenant_id": "993f53c1f51845e48e013aeb632358d8",
"title": "Fake Plugin",
"version_labels": {
"0.1": {
"enabled": {
"status": true,
"mutable": true,
"description": "Plugin or its version is enabled and can be used by user."
}
}
},
"name": "fake"
}
}

View File

@ -0,0 +1,92 @@
{
"plugin": {
"name": "vanilla",
"versions": [
"1.2.1",
"2.4.1",
"2.6.0"
],
"description": "The Apache Vanilla plugin provides the ability to launch upstream Vanilla Apache Hadoop cluster without any management consoles. It can also deploy the Oozie component.",
"required_image_tags": [
"vanilla",
"2.6.0"
],
"node_processes": {
"JobFlow": [
"oozie"
],
"HDFS": [
"namenode",
"datanode",
"secondarynamenode"
],
"YARN": [
"resourcemanager",
"nodemanager"
],
"MapReduce": [
"historyserver"
],
"Hadoop": [],
"Hive": [
"hiveserver"
]
},
"configs": [
{
"default_value": "/tmp/hadoop-${user.name}",
"name": "hadoop.tmp.dir",
"priority": 2,
"config_type": "string",
"applicable_target": "HDFS",
"is_optional": true,
"scope": "node",
"description": "A base for other temporary directories."
},
{
"default_value": true,
"name": "hadoop.native.lib",
"priority": 2,
"config_type": "bool",
"applicable_target": "HDFS",
"is_optional": true,
"scope": "node",
"description": "Should native hadoop libraries, if present, be used."
},
{
"default_value": 1024,
"name": "NodeManager Heap Size",
"config_values": null,
"priority": 1,
"config_type": "int",
"applicable_target": "YARN",
"is_optional": false,
"scope": "node",
"description": null
},
{
"default_value": true,
"name": "Enable Swift",
"config_values": null,
"priority": 1,
"config_type": "bool",
"applicable_target": "general",
"is_optional": false,
"scope": "cluster",
"description": null
},
{
"default_value": true,
"name": "Enable MySQL",
"config_values": null,
"priority": 1,
"config_type": "bool",
"applicable_target": "general",
"is_optional": true,
"scope": "cluster",
"description": null
}
],
"title": "Vanilla Apache Hadoop"
}
}

View File

@ -0,0 +1,41 @@
{
"plugins": [
{
"name": "vanilla",
"description": "The Apache Vanilla plugin provides the ability to launch upstream Vanilla Apache Hadoop cluster without any management consoles. It can also deploy the Oozie component.",
"versions": [
"1.2.1",
"2.4.1",
"2.6.0"
],
"title": "Vanilla Apache Hadoop"
},
{
"name": "hdp",
"description": "The Hortonworks Sahara plugin automates the deployment of the Hortonworks Data Platform (HDP) on OpenStack.",
"versions": [
"1.3.2",
"2.0.6"
],
"title": "Hortonworks Data Platform"
},
{
"name": "spark",
"description": "This plugin provides an ability to launch Spark on Hadoop CDH cluster without any management consoles.",
"versions": [
"1.0.0",
"0.9.1"
],
"title": "Apache Spark"
},
{
"name": "cdh",
"description": "The Cloudera Sahara plugin provides the ability to launch the Cloudera distribution of Apache Hadoop (CDH) with Cloudera Manager management console.",
"versions": [
"5",
"5.3.0"
],
"title": "Cloudera Plugin"
}
]
}

View File

@ -2,7 +2,7 @@
# of appearance. Changing the order has an impact on the overall integration
# process, which may cause wedges in the gate later.
openstackdocstheme>=1.18.1 # Apache-2.0
os-api-ref>=1.4.0 # Apache-2.0
os-api-ref>=1.6.0 # Apache-2.0
reno>=2.5.0 # Apache-2.0
sphinx!=1.6.6,!=1.6.7,>=1.6.2 # BSD
sphinxcontrib-httpdomain>=1.3.0 # BSD

View File

@ -66,7 +66,7 @@ netaddr==0.7.19
netifaces==0.10.6
openstackdocstheme==1.18.1
openstacksdk==0.12.0
os-api-ref==1.4.0
os-api-ref==1.6.0
os-client-config==1.29.0
os-service-types==1.2.0
osc-lib==1.10.0