The original Alembic migration script was autogenerated, then
subsequently modified by hand. Between these two aspects, some
correspondence to the corresponding SQLAlchemy models was lost:
* Columns were specified independently of corresponding foreign key
and primary key constraints; besides making it more difficult to
follow the code between models and the migration script, there was a
loss of integrity constraints: a variable association must exist for
every entity that mixes in VariableMixin as a base was not being
enforced at the database level with an appropriate not-null; this
causes problems for use of variables for any such object that is not
constructed by models code, most notably via direct usage of the
database. (See specifically
https://bugs.launchpad.net/craton/+bug/1668251, fixed with this
change.)
* The column created_at would always be set by model-using code; but
this was not enforced at the database level by ensuring it was not
null.
* Children should be deleted if the parent is deleted (we do not
support "soft delete" in Craton).
In addition, all constraints are now named (the original intent of
this change, so as to ensure the feasibility of future migrations),
and they are also named in a consistent fashion.
Note that this change does not support many-to-many deletions (as seen
in https://bugs.launchpad.net/craton/+bug/1668308), but SQLAlchemy
does provide support for this at the model level. A future change can
address this without requiring Alembic support.
There's no direct testing of this change, but it is implicitly, and
rather strongly, tested by the overall functional tests that are used;
and to a lesser extent by some of the unit tests. This is in part
because this change has strengthened constraints, not weakened them.
Change-Id: I1f84c29610127de12c292a210fd003ae07bd6462
Closes-bug: 1665066
Closes-bug: 1668251
This patch implements the ability to query for values
in nested variable documents. For instance, if we had
hosts with a variables that look like:
{
"hardware_profiles": {
"disks": [
{
"manufacturer": "seagate",
"capacity_quantity": 2,
"capacity_unit": "TB"
},
{
"manufacturer": "western",
"capacity_quantity": 3,
"capacity_unit": "TB"
}
]
}
}
and we wanted to grab all of them with a disk that was
manufactured by Seagate, we would like to be able to
query like so:
GET /v1/hosts?vars=hardware_profiles.disks[*].manufacturer:"seagate"
This does modify the variables query to expect valid JSON
values. So, strings need to be quoted, for instance.
Change-Id: Id597d3e57d2e28766fecd1b314f53176543e1b9d
Closes-Bug: 1671116
Responses generated from failure should be JSON formatted.
flask_restful.Api is subclassed to control the format of all error
responses generated by the API Flask app.
The API middleware is modified to use the same error response function
given that it is not part of the Flask app for the API. Any Response
object created by the middleware has been converted to an exception to
ensure that all error responses are generated using the same code.
The use of abort has been replaced with raising exceptions, this further
consolidates the error response mechanism.
response_filter has been modified so that it always expects to receive a
response tuple now that http_codes is no longer there to return a
response object.
The error filters in schemas.py have been removed. These were not used
by the existing code.
Closes-bug: 1665015
Change-Id: I2be40e9493f313b3fe0173c34d371659039c1bae
If a user creates a network device and also passes
variables during the create process the variables
get created but are not present in the response.
This patch fixes that by adding variables in response
for a create calls on network devices when variable
was in the create request body.
Change-Id: Id9818d5c30a9a0c17039191335f1b75525003a3d
Closes-Bug: 1675087
Currently when user is created, its taking the project_id
parameter from the user who is creating the request.
However, given root users are expected to be ablet to create
users for all projects this patch modifies the behaviour
and ensure that the user creation is always under the
project_id parameter that is passed in.
Functional test for create call is added. However,
function test coverage is not complete. It needs to
be addressed as a separate commit, tacked in
issue: 1674287
Closes Bug: 1666695
Partial-Bug: 1674287
Parent: Ie5fac9ef4c305ff7e67fe32855ca8100a8adfa82
Change-Id: I19fb901ad41edeeb0a92f41264867a768a207fce
In an effort to support JSON Path variable
queries, we need to:
* Pin to MySQL 5.7
* Use JSON column type
* Use a newer SQLAlchemy library.
Change-Id: I98ae021c9cd9755f51e67e56135db164e8ae1f50
Partial-Bug: 1671116
Depends-On: I36b7ae9cf68aeef0e5dbcb1f17db52b81efd6ffe
If a user creates a network and also passes variables
during the create process the variables get created
but are not present in the response.
This patch fixes that by adding variables in response
for a create calls on networks when variable was in
the create request body.
Change-Id: I2320f5b07d3544286234ecc5b1cee4eea999b59e
Closes-Bug: 1674815
Implements a filter addition for vars such that each key-value pair in
the vars is searched for a possible match, regardless of associated
resource; then chases the associated resources for the specific
resource being searched for, thereby implementing the resolution
algorithm (but in reverse). Finally all such resources are checked to
see if they intersect with respect to their rooting key-value pairs in
the original search.
Examples:
$ GET v1/hosts?vars=openstack_release:juno
$ GET v1/hosts?vars=foo:abc,bar:xyz
To disable this feature in your request, you can supply
'resolved-values=false', for example:
$ GET /v1/hosts?vars=foo:bar,baz:zoo&resolved-value=false
Filter values are not currently encoded for JSON; but this will fixed
in a subsequent patch that is addressing generalized JSON matching via
JSON path.
Change-Id: I1d40d734e60b5563dfb01da05ffb6494ed9a919c
Closes-bug: 1661226
Closes-bug: 1669493
Currently project_id is allowed in request body
for all request, however, its not honered since project_id
is pulled from headers.
This commit separates the need for project_id only where
its needed. For instance, in users we expect project_id
to allow creating users for different projects or we
expect project_id in response body.
Partial-Bug: 1666695
Change-Id: Ie5fac9ef4c305ff7e67fe32855ca8100a8adfa82
So far we have been using external mysql command
to insert project/user to bootstrap the process.
This caused much frustration since operators needed
to insert records in the db. This patch adds commands
to dbsync to allow creating project/users. This method
also takes care of association_id.
To initiate a bootstrap:
craton-dbsync --config-file=craton.conf bootstrap
which will create a project with a root user.
Users using Dockerfile can look at docker logs
after creating the container to find this info.
Closes Bug: 1670561
Change-Id: I9372961ca6623d530d7844b9f38aade544d961e8
A device's parent_id represents the id of another device. A parent_id
cannot be a device's own id nor can it be the id of a descendant.
This commit adds a validator decorator to ensure that the parent_id
supplied meets the requirements before the database is updated. Failure
to properly validate the parent_id can cause the database to become
unresponsive.
If an invalid parent_id is supplied the API will return a 400 Bad
Request.
Change-Id: I81f6bff5bf64b7c441fef08a8e03731cc17bb2f1
Closes-bug: 1666536
This commit adds the decorator http_codes to a number of the API put
methods so that all now use it. This ensures any exceptions generated by
the methods return the expected status code and the response body is
JSON encoded.
Change-Id: Id24ef3687afe21e36adae225efdef7150b5d5318
Partial-bug: 1665015
Flask-RESTful automatically converts a response body to JSON when it
creates the Response object. The Flask app configuration option
RESTFUL_JSON is added by this commit to appropriately modify how the
JSON body is constructed.
response_filter has been modified so that it returns a tuple of
information that Flask-RESTful will use to create an appropriately
formatted response.
Resource.error_response is creating the Response object itself and so
has been modified to format the JSON in the same way as Flask-RESTful.
Ideally, error_response would not create a separate Response object
however addressing this is a separate issue.
Change-Id: I02eafaf48d7c6e2ceab0add2f78cf6aa1b890f18
Closes-bug: 1664328
Currently the details=all flag allows us to only
get local variables. This changes that to be resolved
variables such that it matches the rest of the variables
calls.
It also adds details calls to /clouds and /projects
endpoints where it was missing.
Closes Bug: 1667767
Change-Id: Ia2de9110aa118c3fe7cd9374ceed2ce74f13c74e
Previously, listing or creating a network interface required the port be
an integer. Updating the port, however, allowed it to be a string
instead of similarly requiring it to be an integer. This updates the
constraint and adds the basis of functional testing for those endpoints.
Closes-bug: #1630276
Change-Id: Ic99d73dfd63fb3fe9a64d41ca2b8eafe4ff1c253
This change moves craton to a port that is not in use according to
/etc/services and is not being consumed by any other OpenStack service
as defined here: [
https://docs.openstack.org/newton/config-reference/firewalls-default-ports.html
]. This should allow other deployers to leverage craton in an way that
does not conflict with other OpenStack services already running in
production.
Change-Id: I2d0432b4c608cf11538d7b8f267eaf7be763204c
Signed-off-by: Kevin Carter <kevin.carter@rackspace.com>
We were only using resolved vars in hosts/devics.
However, given the new structure where we have a
cloud -> region -> cell -> host ... it makes sense
to have top level vars on cloud level. This patch
therefore enables variable resolution on regions and
cells now.
Closes Bug: 1669038
Change-Id: I9f4c7dbce0082b1ce0b1f03460c51e39b7a12309
This patch corrects the schema to include requirements
for region_id and cloud_id when creating a network.
Change-Id: I0edbf5efa51a55c425051cbe8d7a60c6ed0a2232
Closes-Bug: 1661761
This patch includes the Cloud resource to allow
for a project o have multiple clouds under it. This
modifies the environment hierarchy to look like:
Project -> Cloud -> Region (-> Cell) -> ...
Change-Id: Ib1c16a504430760f2a7234221f91429a7ea72596
Closes-Bug: 1662576
This commit adds support for a devices endpoint to allow the listing of
heterogeneous collections of devices. This complements the hosts and
network-devices endpoints that only allow the listing of devices of
their respective types.
The devices endpoint supports the query parameter "descendants", this
allows the user to choose whether or not a query includes the
descendants of a specified parent_id. For example
'/v1/devices?parent_id=3&descendants=true' would return any devices with
a parent_id of 3 as well as any descendants of those devices.
The devices DB model has been extended to include a descendants property
to allow a device's descendants to be determined.
The functional testing database clean-up has been modified to enable to
tables to be deleted when devices have been given parent_id's.
create_region and create_host methods have been moved to the base
TestCase class to prevent duplicating this code in the new tests.
Change-Id: Ia6f8dca6cbbf3d583bdff69559be0e803d2cf6de
Co-Authored-By: Jim Baker <jim.baker@python.org>
Implements: bp/list-devices
- This patch adds a new parameter `details` as query
parameter such that, when doing a list call on any
resource we can do ?details=all to get the variables
associated with the particualr resource.
Closes Bug: 1662618
Change-Id: Ic8423e30ca05d265ca6419fa179d5569a1995e3e
Currently we have validators and filters that ensure
certain propeties on GET/POST/PUT calls. So far we
have been using the same schema for GET and POST calls
as a result it was possible to post id, created_at etc
when creating resources. This patch fixes this by dividing
the schema to match get and creates.
This adds black listed properties on creates, which can
be expanded according to the resource in question in the
future in needed.
Functional tests are addded.
Closes Bug: 1662848
Change-Id: I9226c18187064b310c811164e146da4909295768
This also refactors how we add pagination related query parameters to
each schema set of request args.
Closes-Bug: 1667027
Change-Id: Iecb628f05de02d1a153b3b63ab6dd67c33c53053
This patch sets the schema to accept a string
for Project ID since it's actually a UUID
Change-Id: Iad8733594ca83c3bc061017fd9eae6861f31ee5d
Closes-Bug: 1665780
This patch includes a branch to tell _link_params_for
to skip params that have a None value when generating
its links.
Change-Id: I6182028dec94b010ea79a0f281f17d3f70f3b202
Closes-Bug: 1665779
- After commit aad5fd33c3
search for host devices by label was broken because the
pagination function was unable to handle joins on relationsip.
This patch switch to filter rule on query so that it plays
nice with the pagination patch.
Unit test on db and functional tests are added to test
filtering by label.
Closes-Bug: 1662842
Change-Id: I476a5efcc1e5ef8735833dd37583653f2c9d56c5
sort_dir and sort_keys are used in the pagination work. Adding them to
the schemata allows users to specify them in their queries without
receiving 400 Bad Request errors.
Change-Id: If743958afc578be0ffa48c9b0a2670f76855424c
Partial-Bug: #1664786
Implements: bp pagination-of-resources
For our default CI jobs, we can add a test-setup.sh script that will run
before OpenStack CI revokes sudo for the "jenkins" user.
At the moment, we cannot add the jenkins user to the docker group to
allow the jenkins user to use docker. Instead, we have to make docker
launch itself so that the jenkins group can access the socket.
Change-Id: I732d144e4635dffe8034c0d374eee591df078f52
This patch re-applies RBAC-lite to the /projects
endpoint. For the top-level /projects endpoint,
permissions have been narrowed to only root
project users. Anything pertaining to a specific
project has been narrowed to project admin users.
This patch implements a decorator to afford us the
same features for the generic resources functions
that @require_project_admin_context and
@require_admin_context decorators afforded us
piecemeal for other DB API functions.
Change-Id: I875a9a8cb68c76d27f613bac6f4b87f3a1e2b411
Closes-Bug: 1665237
Closes-Bug: 1648626
This fix removes some left-over debug statements
that made it through gate on accident.
Change-Id: I95f0f9f20704c0f05b8f2c280e579b6064d4f89f
Related-Bug: 1648626
Includes Variables support, which is already present
in several other Craton resources, in the Projects
resource as well.
Project variables will serve as a top-most scope
of variables for resources that belong to a specific
project.
Partial-Bug: 1648626
Closes-Bug: 1661670
Change-Id: I574e834ce409efd8ceafb15fe30a86ae93df2a80
Andreas Jaeger