Add json schema validation for Manila APIs
This spec proposes adding json schema validation for request body of Manila APIs. Change-Id: If6e7a16aff7a4bcc64981ed21c566aca3138b044 Partial-Implements: json-schema-validation
This commit is contained in:
parent
3403c81984
commit
9b0407df8e
|
@ -0,0 +1,355 @@
|
|||
..
|
||||
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||
License.
|
||||
|
||||
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||
|
||||
==============
|
||||
API Validation
|
||||
==============
|
||||
|
||||
https://blueprints.launchpad.net/manila/+spec/json-schema-validation
|
||||
|
||||
Currently, Manila has different implementations for validating request bodies.
|
||||
The purpose of this blueprint is to track the progress of validating the
|
||||
request bodies sent to the Manila server, accepting requests that fit the
|
||||
resource schema and rejecting requests that do not fit the schema. Depending
|
||||
on the content of the request body, the request should be accepted or rejected
|
||||
consistently.
|
||||
|
||||
|
||||
Problem description
|
||||
===================
|
||||
|
||||
Currently Manila doesn't have a consistent request validation layer. Some
|
||||
resources validate input at the resource controller and some fail out in the
|
||||
backend. Ideally, Manila would have some validation in place to catch
|
||||
disallowed parameters and return a validation error to the user.
|
||||
|
||||
The end user will benefit from having consistent and helpful feedback,
|
||||
regardless of which resource they are interacting with.
|
||||
|
||||
|
||||
Use Cases
|
||||
=========
|
||||
|
||||
As a user or developer, I want to observe consistent API validation and values
|
||||
passed to the Manila API server.
|
||||
|
||||
|
||||
Proposed change
|
||||
===============
|
||||
|
||||
One possible way to validate the Manila API is to use jsonschema similar to
|
||||
Nova, Cinder, Keystone and Glance (https://pypi.python.org/pypi/jsonschema).
|
||||
A jsonschema validator object can be used to check each resource against an
|
||||
appropriate schema for that resource. If the validation passes, the request
|
||||
can follow the existing flow of control through the resource manager to the
|
||||
backend. If the request body parameters fails the validation specified by the
|
||||
resource schema, a validation error wrapped in HTTPBadRequest will be returned
|
||||
from the server.
|
||||
|
||||
Example:
|
||||
"Invalid input for field 'name'. The value is 'some invalid name value'.
|
||||
|
||||
Each API definition should be added with the following ways:
|
||||
|
||||
* Create definition files under ./manila/api/schemas/.
|
||||
* Each definition should be described with JSON Schema.
|
||||
* Each parameter of definitions(type, minLength, etc.) can be defined from
|
||||
current validation code, DB schema, unit tests, Tempest code, or so on.
|
||||
|
||||
Some notes on doing this implementation:
|
||||
|
||||
* Common parameter types can be leveraged across all Manila resources. An
|
||||
example of this would be as follows::
|
||||
|
||||
# share create schema
|
||||
|
||||
manila/api/schema/shares.py:
|
||||
|
||||
from manila.api.validation import parameter_types
|
||||
create_v231 = {
|
||||
'type': 'object',
|
||||
'properties': {
|
||||
'share': {
|
||||
'type': 'object',
|
||||
'properties': {
|
||||
'description': parameter_types.description,
|
||||
'share_type': {
|
||||
'format': 'uuid'
|
||||
},
|
||||
'share_proto': parameter_types.proto
|
||||
'share_network_id': {
|
||||
'format': 'uuid'
|
||||
},
|
||||
'share_group_id': {
|
||||
'format': 'uuid'
|
||||
},
|
||||
'name': parameter_types.name,
|
||||
'snapshot_id': {
|
||||
'format': 'uuid'
|
||||
},
|
||||
'size': parameter_types.positive_integer,
|
||||
'metadata': {
|
||||
'type': 'object'
|
||||
},
|
||||
},
|
||||
'required': ['size'],
|
||||
'additionalProperties': False,
|
||||
}
|
||||
'required': ['share'],
|
||||
'additionalProperties': False,
|
||||
}
|
||||
}
|
||||
|
||||
manila/api/validation/parameter_types.py:
|
||||
|
||||
description = {
|
||||
'type': 'string', 'minLength': 0, 'maxLength': 255,
|
||||
}
|
||||
|
||||
name = description
|
||||
|
||||
# This registers a FormatChecker on the jsonschema module.
|
||||
# It might appear that nothing is using the decorated method but it gets
|
||||
# used in JSON schema validations to check uuid formatted strings.
|
||||
from oslo_utils import uuidutils
|
||||
|
||||
@jsonschema.FormatChecker.cls_checks('uuid')
|
||||
def _validate_uuid_format(instance):
|
||||
return uuidutils.is_uuid_like(instance)
|
||||
|
||||
positive_integer = {
|
||||
'type': ['integer', 'string'],
|
||||
'pattern': '^[0-9]*$', 'minimum': 1, 'minLength': 1
|
||||
}
|
||||
|
||||
proto = {
|
||||
'type': ['string'],
|
||||
'enum': ['NFS', 'CIFS', 'GlusterFS', 'HDFS', 'CephFS']
|
||||
}
|
||||
|
||||
* The validation can take place at the controller layer using below decorator::
|
||||
|
||||
manila/api/v2/shares.py:
|
||||
|
||||
from manila.api.schemas import share
|
||||
|
||||
@wsgi.Controller.api_version("2.31")
|
||||
@validation.schema(shares.create_v231)
|
||||
def create(self, req, body):
|
||||
"""creates a share."""
|
||||
|
||||
* Initial work will include capturing the Shared File API Spec for existing
|
||||
resources in a schema. This should be a one time operation for each
|
||||
major version of the API. This will be applied to the Shared File V2 API.
|
||||
|
||||
* When adding a new extension to Manila, the new extension must be proposed
|
||||
with its appropriate schema.
|
||||
|
||||
|
||||
Alternatives
|
||||
------------
|
||||
|
||||
Before the API validation framework, we need to add the validation code into
|
||||
each API method in ad-hoc. These changes would make the API method code messy
|
||||
and we needed to create multiple patches due to incomplete validation.
|
||||
|
||||
If using JSON Schema definitions instead, acceptable request formats are clear
|
||||
and we don’t need to do ad-hoc works in the future.
|
||||
|
||||
Pecan Framework:
|
||||
`Pecan <http://pecan.readthedocs.org/en/latest/>`_
|
||||
Some projects(Ironic, Ceilometer, etc.) are implemented with Pecan/WSME
|
||||
frameworks and we can get API documents automatically from the frameworks.
|
||||
In WSME implementation, the developers should define API parameters for
|
||||
each API. Pecan would make the implementations of API routes(URL, METHOD) easy.
|
||||
|
||||
|
||||
Data model impact
|
||||
-----------------
|
||||
|
||||
None
|
||||
|
||||
|
||||
REST API impact
|
||||
---------------
|
||||
|
||||
In Manila, since Liberty, we strive to maintain complete API
|
||||
backwards-compatibility. So, if we have a client that's talking to different
|
||||
manila releases, they should experience the same behavior if using the same
|
||||
API version. We break this rule rarely, but have consistently maintained
|
||||
that if we were correcting buggy behavior, we will consider breaking
|
||||
backwards-compatibility.
|
||||
|
||||
API Response code changes:
|
||||
|
||||
There are some occurrences where API response code will change while adding
|
||||
schema layer for them. For example, On current master 'services' table has
|
||||
'host' and 'binary' of maximum 255 characters in database table. While updating
|
||||
service user can pass 'host' and 'binary' of more than 255 characters which
|
||||
obviously fails with 404 ServiceNotFound wasting a database call. For this we
|
||||
can restrict the 'host' and 'binary' of maximum 255 characters only in schema
|
||||
definition of 'services'. If user passes more than 255 characters, he/she will
|
||||
get 400 BadRequest in response.
|
||||
|
||||
API Response error messages:
|
||||
|
||||
There will be change in the error message returned to user. For example,
|
||||
On current master if user passes more than 255 characters for share name
|
||||
then below error message is returned to user from manila-api:
|
||||
|
||||
Invalid input received: name has <actual no of characters user passed>
|
||||
characters, more than 255.
|
||||
|
||||
With schema validation below error message will be returned to user for this
|
||||
case:
|
||||
|
||||
Invalid input for field/attribute name. Value: <value passed by user>.
|
||||
'<value passed by user>' is too long.
|
||||
|
||||
|
||||
Security impact
|
||||
---------------
|
||||
|
||||
The output from the request validation layer should not compromise data or
|
||||
expose private data to an external user. Request validation should not
|
||||
return information upon successful validation. In the event a request
|
||||
body is not valid, the validation layer should return the invalid values
|
||||
and/or the values required by the request, of which the end user should know.
|
||||
The parameters of the resources being validated are public information,
|
||||
described in the Shared File API spec, with the exception of private data.
|
||||
In the event the user's private data fails validation, a check can be built
|
||||
into the error handling of the validator to not return the actual value of the
|
||||
private data.
|
||||
|
||||
jsonschema documentation notes security considerations for both schemas and
|
||||
instances:
|
||||
http://json-schema.org/latest/json-schema-core.html#anchor21
|
||||
|
||||
Better up front input validation will reduce the ability for malicious user
|
||||
input to exploit security bugs.
|
||||
|
||||
|
||||
Notifications impact
|
||||
--------------------
|
||||
|
||||
None
|
||||
|
||||
Other end user impact
|
||||
---------------------
|
||||
|
||||
None
|
||||
|
||||
Performance Impact
|
||||
------------------
|
||||
|
||||
Manila will pay some performance cost for this comprehensive request
|
||||
parameters validation, because the checks will be increased for API parameters
|
||||
which are not validated now.
|
||||
|
||||
|
||||
Other deployer impact
|
||||
---------------------
|
||||
|
||||
None
|
||||
|
||||
|
||||
Developer impact
|
||||
----------------
|
||||
|
||||
This will require developers contributing new extensions to Manila to have
|
||||
a proper schema representing the extension's API.
|
||||
|
||||
|
||||
Implementation
|
||||
==============
|
||||
|
||||
Assignee(s)
|
||||
-----------
|
||||
|
||||
Primary assignee:
|
||||
dongdongpei (Dongdong Pei <peidongdong121@163.com>)
|
||||
|
||||
Other contributors:
|
||||
wosunoozzy (Yang Zhang <wosunoozzy@gmail.com>)
|
||||
|
||||
Work Items
|
||||
----------
|
||||
|
||||
1. Initial validator implementation, which will contain common validator code
|
||||
designed to be shared across all resource controllers validating request
|
||||
bodies.
|
||||
2. Introduce validation schemas for existing core API resources.
|
||||
3. Introduce validation schemas for existing API extensions.
|
||||
4. Enforce validation on proposed core API additions and extensions.
|
||||
5. Remove duplicated ad-hoc validation code.
|
||||
6. Add unit and end-to-end tests of related API's.
|
||||
7. Add/Update Manila documentation.
|
||||
|
||||
Dependencies
|
||||
============
|
||||
|
||||
The code under manila/api/v1 are getting called by v2. So if we add schema
|
||||
validation for v2 then we will have to remove the existing validation of
|
||||
parameters which is there inside of controller methods which will again
|
||||
break the v2 apis.
|
||||
|
||||
Solution:
|
||||
|
||||
1. Do the schema validation for v2 apis using the @validation.schema decorator
|
||||
similar to Nova and also keep the validation code which is there inside of
|
||||
method to keep v1 working.
|
||||
|
||||
2. Once the decision is made to remove the support to v1 we should remove the
|
||||
validation code from inside of method.
|
||||
|
||||
|
||||
Testing
|
||||
=======
|
||||
|
||||
Tempest tests can be added as each resource is validated against its schema.
|
||||
These tests should walk through invalid request types.
|
||||
|
||||
We can follow some of the validation work already done in the Nova V3 API:
|
||||
|
||||
* `Validation Testing <http://git.openstack.org/cgit/openstack/tempest/tree/etc/schemas/compute/flavors/flavors_list.json?id=24eb89cd3efd9e9873c78aacde804870962ddcbb>`_
|
||||
|
||||
* `Negative Validation Testing <http://git.openstack.org/cgit/openstack/tempest/tree/tempest/api/compute/flavors/test_flavors_negative.py?id=b2978da5ab52e461b06a650e038df52e6ceb5cd6>`_
|
||||
|
||||
Negative validation tests should use tempest.test.NegativeAutoTest
|
||||
|
||||
|
||||
Documentation Impact
|
||||
====================
|
||||
|
||||
1. The Manila API documentation will need to be updated to reflect the
|
||||
REST API changes.
|
||||
2. The Manila developer documentation will need to be updated to explain
|
||||
how the schema validation will work and how to add json schema for
|
||||
new API's.
|
||||
|
||||
|
||||
References
|
||||
==========
|
||||
|
||||
* Understanding JSON Schema:
|
||||
|
||||
http://spacetelescope.github.io/understanding-json-schema/reference/object.html
|
||||
|
||||
* Nova Validation Examples:
|
||||
|
||||
http://git.openstack.org/cgit/openstack/nova/tree/nova/api/validation
|
||||
|
||||
* JSON Schema on PyPI:
|
||||
|
||||
https://pypi.python.org/pypi/jsonschema
|
||||
|
||||
* JSON Schema core definitions and terminology:
|
||||
|
||||
http://tools.ietf.org/html/draft-zyp-json-schema-04
|
||||
|
||||
* JSON Schema Documentation:
|
||||
|
||||
http://json-schema.org/documentation.html
|
Loading…
Reference in New Issue