Commit Graph

474 Commits

Author SHA1 Message Date
Andreas Jaeger e88f6b747a Retire craton
Remove everything, add a README with explanation.

Change-Id: I7c8fcfac0d904f144b6a3838d25db0987d1e74a5
2018-01-18 15:22:28 +01:00
Jim Baker 555d0c8856 Updates Alembic migration to better match SQLAlchemy models
The original Alembic migration script was autogenerated, then
subsequently modified by hand. Between these two aspects, some
correspondence to the corresponding SQLAlchemy models was lost:

* Columns were specified independently of corresponding foreign key
  and primary key constraints; besides making it more difficult to
  follow the code between models and the migration script, there was a
  loss of integrity constraints: a variable association must exist for
  every entity that mixes in VariableMixin as a base was not being
  enforced at the database level with an appropriate not-null; this
  causes problems for use of variables for any such object that is not
  constructed by models code, most notably via direct usage of the
  database. (See specifically
  https://bugs.launchpad.net/craton/+bug/1668251, fixed with this
  change.)

* The column created_at would always be set by model-using code; but
  this was not enforced at the database level by ensuring it was not
  null.

* Children should be deleted if the parent is deleted (we do not
  support "soft delete" in Craton).

In addition, all constraints are now named (the original intent of
this change, so as to ensure the feasibility of future migrations),
and they are also named in a consistent fashion.

Note that this change does not support many-to-many deletions (as seen
in https://bugs.launchpad.net/craton/+bug/1668308), but SQLAlchemy
does provide support for this at the model level. A future change can
address this without requiring Alembic support.

There's no direct testing of this change, but it is implicitly, and
rather strongly, tested by the overall functional tests that are used;
and to a lesser extent by some of the unit tests. This is in part
because this change has strengthened constraints, not weakened them.

Change-Id: I1f84c29610127de12c292a210fd003ae07bd6462
Closes-bug: 1665066
Closes-bug: 1668251
2017-03-29 14:11:45 -06:00
Thomas Maddox 96b6a40288 Add documentation for JSON Path-like variable searching
Change-Id: I8a3648ba5c94bc27bc0bcf0c7ebb9de19a9c4818
2017-03-29 14:33:33 +00:00
Thomas Maddox 606926b89d JSON Path-like querying for variables
This patch implements the ability to query for values
in nested variable documents. For instance, if we had
hosts with a variables that look like:

{
    "hardware_profiles": {
        "disks": [
            {
                "manufacturer": "seagate",
                "capacity_quantity": 2,
                "capacity_unit": "TB"
            },
            {
                "manufacturer": "western",
                "capacity_quantity": 3,
                "capacity_unit": "TB"
            }
        ]
    }
}

and we wanted to grab all of them with a disk that was
manufactured by Seagate, we would like to be able to
query like so:

GET /v1/hosts?vars=hardware_profiles.disks[*].manufacturer:"seagate"

This does modify the variables query to expect valid JSON
values. So, strings need to be quoted, for instance.

Change-Id: Id597d3e57d2e28766fecd1b314f53176543e1b9d
Closes-Bug: 1671116
2017-03-28 15:23:52 +00:00
Jenkins 0e4280ede3 Merge "Ensure JSON responses result from failure" 2017-03-28 04:51:31 +00:00
Jenkins 1b680b2532 Merge "Adding wrapper functions to tools" 2017-03-28 04:51:25 +00:00
git-harry 533492b4d3 Ensure JSON responses result from failure
Responses generated from failure should be JSON formatted.

flask_restful.Api is subclassed to control the format of all error
responses generated by the API Flask app.

The API middleware is modified to use the same error response function
given that it is not part of the Flask app for the API. Any Response
object created by the middleware has been converted to an exception to
ensure that all error responses are generated using the same code.

The use of abort has been replaced with raising exceptions, this further
consolidates the error response mechanism.

response_filter has been modified so that it always expects to receive a
response tuple now that http_codes is no longer there to return a
response object.

The error filters in schemas.py have been removed. These were not used
by the existing code.

Closes-bug: 1665015

Change-Id: I2be40e9493f313b3fe0173c34d371659039c1bae
2017-03-27 22:18:39 +01:00
Michael Porras bb515d45f2 Adding wrapper functions to tools
Adding wrapper functions to the tools directory. Updating development docs
advising users to clone the sample config to a dev config. The dev config
allows users to launch craton directly and in docker without constant config
changes.

Change-Id: I6c32c86ef84fdcf02077fea226e20110826b1f21
Closes-Bug: #1675113
2017-03-27 13:35:43 -05:00
Jenkins bd0a175f34 Merge "Remove link to modindex" 2017-03-27 17:52:12 +00:00
Ian Cordasco 14901b20b6 Update the link to Craton's documentation
ReadTheDocs hosts user content on readthedocs.io and provides TLS for
all projects.

Change-Id: I19be4de52c4290d9fe064f102e65c72eccc014c7
2017-03-24 11:37:13 -05:00
Ian Cordasco 0bbc4edc82 Update status of existing specifications
These two specifications are actually implemented and so can be moved
from approved to implemented.

Change-Id: Id78dffdf286e6d1e3eeba50e2d686f25c8257b1e
2017-03-23 08:48:06 -05:00
Jenkins a2d645c6ee Merge "Docker and dev Documentation update" 2017-03-22 19:35:15 +00:00
Jenkins 034255e750 Merge "Modify docs to use python3 -m pip instead" 2017-03-22 19:26:28 +00:00
Thomas Maddox 1e1e8da171 Return variables in response on Network Device create
If a user creates a network device and also passes
variables during the create process the variables
get created but are not present in the response.

This patch fixes that by adding variables in response
for a create calls on network devices when variable
was in the create request body.

Change-Id: Id9818d5c30a9a0c17039191335f1b75525003a3d
Closes-Bug: 1675087
2017-03-22 16:50:34 +00:00
Michael Porras b571719f67 Docker and dev Documentation update
Updating dev install documentation to include pymysql database connection.
Updating docker documentation to gather bootstrap auth credentials and
change api port.

Change-Id: I5df061749f7c4143c7d2b3f7f6fecddd5be0223b
Closes-Bug: #1674731
2017-03-22 11:28:09 -05:00
Antony Messerli 4aa1e5acb7 Modify docs to use python3 -m pip instead
Adds a few doc suggestions from previos review

Change-Id: I1b1950cd46001fdc6bc34d38aa14f824c9aa490d
2017-03-22 11:15:45 -05:00
Jenkins 6eaa61ccc0 Merge "Create user under project_id passed in request" 2017-03-22 14:08:17 +00:00
Sulochan Acharya 5f0e8d7622 Create user under project_id passed in request
Currently when user is created, its taking the project_id
parameter from the user who is creating the request.
However, given root users are expected to be ablet to create
users for all projects this patch modifies the behaviour
and ensure that the user creation is always under the
project_id parameter that is passed in.

Functional test for create call is added. However,
function test coverage is not complete. It needs to
be addressed as a separate commit, tacked in
issue: 1674287

Closes Bug: 1666695
Partial-Bug: 1674287
Parent: Ie5fac9ef4c305ff7e67fe32855ca8100a8adfa82

Change-Id: I19fb901ad41edeeb0a92f41264867a768a207fce
2017-03-22 11:32:41 +00:00
Jenkins 637250ee6a Merge "Move to MySQL 5.7 and SQLAlchemy>=1.1.0" 2017-03-21 21:41:45 +00:00
Jenkins 2fe08e9f4c Merge "Return variables in response on Network create" 2017-03-21 21:41:14 +00:00
Jenkins 1d9e1b1b86 Merge "Remove log translations" 2017-03-21 21:24:15 +00:00
Thomas Maddox 6884f8ff8c Move to MySQL 5.7 and SQLAlchemy>=1.1.0
In an effort to support JSON Path variable
queries, we need to:
* Pin to MySQL 5.7
* Use JSON column type
* Use a newer SQLAlchemy library.

Change-Id: I98ae021c9cd9755f51e67e56135db164e8ae1f50
Partial-Bug: 1671116
Depends-On: I36b7ae9cf68aeef0e5dbcb1f17db52b81efd6ffe
2017-03-21 21:22:55 +00:00
Thomas Maddox 5852565dcd Return variables in response on Network create
If a user creates a network and also passes variables
during the create process the variables get created
but are not present in the response.

This patch fixes that by adding variables in response
for a create calls on networks when variable was in
the create request body.

Change-Id: I2320f5b07d3544286234ecc5b1cee4eea999b59e
Closes-Bug: 1674815
2017-03-21 21:21:13 +00:00
Jenkins 12b31e49ac Merge "Variable search for resources now uses resolved variables." 2017-03-21 16:41:38 +00:00
Jenkins 35ff3cf96f Merge "Updates documentation for setting up a dev env" 2017-03-21 16:16:40 +00:00
Jim Baker 3d0506b69c Variable search for resources now uses resolved variables.
Implements a filter addition for vars such that each key-value pair in
the vars is searched for a possible match, regardless of associated
resource; then chases the associated resources for the specific
resource being searched for, thereby implementing the resolution
algorithm (but in reverse). Finally all such resources are checked to
see if they intersect with respect to their rooting key-value pairs in
the original search.

Examples:

$ GET v1/hosts?vars=openstack_release:juno

$ GET v1/hosts?vars=foo:abc,bar:xyz

To disable this feature in your request, you can supply
'resolved-values=false', for example:

$ GET /v1/hosts?vars=foo:bar,baz:zoo&resolved-value=false

Filter values are not currently encoded for JSON; but this will fixed
in a subsequent patch that is addressing generalized JSON matching via
JSON path.

Change-Id: I1d40d734e60b5563dfb01da05ffb6494ed9a919c
Closes-bug: 1661226
Closes-bug: 1669493
2017-03-21 13:45:13 +00:00
Antony Messerli ec52862ec6 Updates documentation for setting up a dev env
Updates docs for Ubuntu Xenial, CentOS 7, and
Fedora 25 setup of Craton for setting up an
initial development environment.

Change-Id: I8769860db0ea20f04f193de5e4f06218e89959d0
2017-03-21 08:22:00 -05:00
Jenkins 3d5677c92b Merge "Include project_id only where needed" 2017-03-21 13:03:14 +00:00
Tao Li fda7af32fa Remove log translations
Log messages are no longer being translated. This removes all use of the
_LE, _LI, and _LW translation markers to simplify logging and to avoid
confusion with new contributions. See:
http://lists.openstack.org/pipermail/openstack-i18n/2016-November/002574.html
http://lists.openstack.org/pipermail/openstack-dev/2017-March/113365.html

Change-Id: I9d94735ace6f0dde822bfb37e5c94195a010d837
2017-03-21 16:03:41 +08:00
Sulochan Acharya 9ddf2c4cd5 Include project_id only where needed
Currently project_id is allowed in request body
for all request, however, its not honered since project_id
is pulled from headers.

This commit separates the need for project_id only where
its needed. For instance, in users we expect project_id
to allow creating users for different projects or we
expect project_id in response body.

Partial-Bug: 1666695

Change-Id: Ie5fac9ef4c305ff7e67fe32855ca8100a8adfa82
2017-03-20 09:33:59 +00:00
lidong 584f0551fa Fix a typo
Change-Id: Ia66f2c4501686d8bf7e2b09e17fd77046e212078
2017-03-15 23:03:26 -07:00
lidong ce1311ecfc Correct some typo errors
Change-Id: Id4d7d86bdfda5c83e0ca611fe653a1727870bc22
2017-03-15 03:44:08 -07:00
Jenkins c6eb57c2b0 Merge "Remove tox usage of upper-constraints" 2017-03-13 21:25:38 +00:00
Jenkins 667ac16008 Merge "Adds project/user bootstrap command to dbsync" 2017-03-13 20:48:06 +00:00
Sulochan Acharya 109b4b6dd2 Adds project/user bootstrap command to dbsync
So far we have been using external mysql command
to insert project/user to bootstrap the process.
This caused much frustration since operators needed
to insert records in the db. This patch adds commands
to dbsync to allow creating project/users. This method
also takes care of association_id.

To initiate a bootstrap:
craton-dbsync --config-file=craton.conf bootstrap

which will create a project with a root user.

Users using Dockerfile can look at docker logs
after creating the container to find this info.

Closes Bug: 1670561

Change-Id: I9372961ca6623d530d7844b9f38aade544d961e8
2017-03-13 20:30:03 +00:00
Ian Cordasco b8114d68cb Remove tox usage of upper-constraints
Craton has opted out of global-requirements. This means that we will
need to stop using upper-constraints as those are generated from
openstack/requirements's global-requirements file.

This will allow our work that depends on SQLAlchemy > 1.1 to continue.

Change-Id: Ic5d76d189968c843f1a2fc7bae7f7ad07be785cc
2017-03-13 14:02:11 -05:00
git-harry a690e1997c Validate parent_id is neither id nor a descendant
A device's parent_id represents the id of another device. A parent_id
cannot be a device's own id nor can it be the id of a descendant.

This commit adds a validator decorator to ensure that the parent_id
supplied meets the requirements before the database is updated. Failure
to properly validate the parent_id can cause the database to become
unresponsive.

If an invalid parent_id is supplied the API will return a 400 Bad
Request.

Change-Id: I81f6bff5bf64b7c441fef08a8e03731cc17bb2f1
Closes-bug: 1666536
2017-03-09 10:45:43 +00:00
git-harry cbb246111b Add http_codes decorator where missing
This commit adds the decorator http_codes to a number of the API put
methods so that all now use it. This ensures any exceptions generated by
the methods return the expected status code and the response body is
JSON encoded.

Change-Id: Id24ef3687afe21e36adae225efdef7150b5d5318
Partial-bug: 1665015
2017-03-09 09:16:51 +00:00
Tomi Juvonen c18917cdc4 docker install documentation not consistent
Order of commands and usage of ContainerId and ContainerIP should be
consistent and examples need to work.

Change-Id: I7bb7cee608aeef1916ed9d17fa67566943291af9
Closes-bug: 1671067
2017-03-08 19:35:53 +02:00
git-harry 0f7ee6d791 Pretty-print format all JSON response bodies
Flask-RESTful automatically converts a response body to JSON when it
creates the Response object. The Flask app configuration option
RESTFUL_JSON is added by this commit to appropriately modify how the
JSON body is constructed.

response_filter has been modified so that it returns a tuple of
information that Flask-RESTful will use to create an appropriately
formatted response.

Resource.error_response is creating the Response object itself and so
has been modified to format the JSON in the same way as Flask-RESTful.
Ideally, error_response would not create a separate Response object
however addressing this is a separate issue.

Change-Id: I02eafaf48d7c6e2ceab0add2f78cf6aa1b890f18
Closes-bug: 1664328
2017-03-07 15:57:46 +00:00
Jenkins 7258e8e739 Merge "Minor Documentation Cleanup" 2017-03-06 18:41:00 +00:00
Jenkins 1de24ce983 Merge "Documentation Fix for Region Create" 2017-03-06 18:40:54 +00:00
Jenkins c044bed5d0 Merge "Fake Data: Add labels to devices" 2017-03-06 18:39:11 +00:00
git-harry 61a805b185 Remove networks validation duplicate details key
This commit consolidates the values of a duplicate key and removes the
duplicate.

Change-Id: I0b955fe52c9471de09f8a73fe95961037c570653
2017-03-06 16:03:06 +00:00
Sulochan Acharya dfeb340ab5 Fake Data: Add labels to devices
Adds labels to device and container creation
such that we can use --label query to filter
by labels.

Change-Id: Icd47cbaca0067fad66e0f3b75150653b6a5bde3d
2017-03-06 15:13:19 +00:00
Syed Ahsan Shamim Zaidi 7318f067ad Documentation Fix for Region Create
Currently the region create command doesn't work because of
Cloud_id change. This patch fixes this since in order to create
a region, we first need a cloud id to associate with.

Change-Id: I6aaef86cb1960b3fee77c48d0c53754177d3b9cf
2017-03-03 23:02:07 +00:00
Jenkins 8570377429 Merge "Allow resolved vars in details list calls" 2017-03-03 19:19:26 +00:00
sulochan acharya 1d3fcad786 Allow resolved vars in details list calls
Currently the details=all flag allows us to only
get local variables. This changes that to be resolved
variables such that it matches the rest of the variables
calls.

It also adds details calls to /clouds and /projects
endpoints where it was missing.

Closes Bug: 1667767

Change-Id: Ia2de9110aa118c3fe7cd9374ceed2ce74f13c74e
2017-03-03 11:10:47 +00:00
Jenkins 39683822da Merge "Move craton to a non-conflicting port" 2017-03-03 09:55:30 +00:00
Jenkins e89c1fb40f Merge "Update schema definition for network interfaces port" 2017-03-02 19:52:53 +00:00