This better reflects its actual meaning.
The 'hostname_or_ip_address' parameter type is removed as it has been
unused since support for support for the 'os-cells' API was removed in
change Iddb519008515f591cf1d884872a5887afbe766f2.
Change-Id: Ia99aeb37785284e129b01d336da17ea907db95b7
Signed-off-by: Stephen Finucane <stephenfin@redhat.com>
Replace six.text_type with str.
This patch completes six removal.
Change-Id: I779bd1446dc1f070fa5100ccccda7881fa508d79
Implements: blueprint six-removal
Signed-off-by: Takashi Natsume <takanattie@gmail.com>
Remove six.PY2 and six.PY3.
Subsequent patches will replace other six usages.
Change-Id: Iccce0ab50eee515e533ab36c8e7adc10cb3f7019
Implements: blueprint six-removal
Signed-off-by: Takashi Natsume <takanattie@gmail.com>
As noted in change If8184c190e76d8cefb5b097f8fa8cb7564207103, the format
of an internal error message changed in Python 3. There's no need to
manually list each of these versions though - just ensure it's Python 3.
Change-Id: Ie1a97d251c6098054bdc963acf53c3182b52b2aa
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Closes-Bug: #1855493
Drop support for the os-cells REST APIs, which are part of the cells v1
feature which has been deprecated since Pike.
This API now returns a 410 response for all routes.
Unit tests are removed and the functional API sample tests are just
asserting the 410 response now. The latter are also expanded to cover
APIs that weren't previously tested.
The API sample docs are left intact since the API reference still builds
from those and can be considered more or less branchless, so people
looking at the API reference can apply it to older deployments of nova
before os-cells was removed.
A release note added for previous cells v1 removals is amended to note
this additional change.
Part of blueprint remove-cells-v1
Change-Id: Iddb519008515f591cf1d884872a5887afbe766f2
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Without these, if you try to run tox -epy37,functional-py37 you'll
get a successful tox run, but no actual tests are run, which is
rather misleading. Given the generaly availability of python 3.7
this is a bad thing.
Running the tests under python 3.7 identified a few minor tests
failures, also fixed here. Each is a result of a change in behavior in
python 3.7:
* printable unicode changes with a new Unicode 11-based unicodedata
package
* intentionally raising StopIteration in a generator is now considered a
RuntimeError, 'return' should be used instead
* an exception message is different beween python 3 and python 2, and the
guard for it was mapping python 3.5 and 3.6 but not 3.7.
zuul configuration is adjusted to add an experimental job for python 3.7
unit. A functional test job is not added, because we don't have 3.6 yet,
and we probably want to get through that first.
Closes-Bug: #1807976
Closes-Bug: #1807970
Change-Id: I37779a12d3b36eb3dc7e2733d07fe0ed23ab3da6
- _build_regex_range is called 17 times on
import of nova.api.validation.parameters_types.
_build_regex_range internally calls re.escape
and valid_char on every char returned
from _get_all_chars.
_get_all_chars yields all chars up to 0xffff.
As a result re.escape and valid_char are called
1.1 million times when
nova.api.validation.parameters_types is imported.
- This change add a memorize decorator and uses
it to cache _build_regex_range
- This change does not cache valid_char,
_is_printable or re.escape as hashing and
caching them for each invocation would
be far more costly both in time and memory
than computing the result.
Change-Id: Ic1f2c560a6da815b26fdf770450bbe439d18d4f9
Closes-Bug: #1790195
A python version check was only checking for 3.5. As noted
in the pre-existing comment, an exception message is not a
particularly stable interface so specific-to-minor-version
check is maintained.
Change-Id: I441b90e911fbac033e8cdea96114db22cba96ac5
Closes-Bug: #1750355
Invalid query parameter could lead to HTTP 500, although
Nova used JSON Schema verification to check input query
params, but query like: GET /servers?limit=%88 will still
lead to HTTP 500, as it failed to parse at webob which is
pre JSON Schema check.
Partial-Bug: #1746202
Change-Id: I11b94a1aaeb67dc1a5abdcf0af5961ee8942a50a
This patch adds `key_name` param to instance rebuild
API. Then the user could reset the instance keypair
when rebuilding. If set key_name to None, the API
will unset the keypair of the instance.
APIImpact
Implements blueprint: rebuild-keypair-reset
Change-Id: I23886a89c25f811cfbe7e2500ce7ff52f9162966
The below test case is not checking all validator schema formats
defined in list "checks":
nova.tests.unit.test_api_validation.FormatCheckerTestCase.
test_format_checker_failed_with_non_string
The loop variable "check" is never used in the for loop block instead
the same format "name" is tested for four times.
This patch seperates the test cases for each validator schema format
as the exception message is differemt for each one.
NOTE: Not used DDT to combine tests as with DDT it will be difficult to
figure out which validator format has failed.
Change-Id: Ic87128a5362d64534b50e8d1450bd3eca832afbd
The 'message' attribute has been deprecated and removed
from Python3.
For more details, please check:
https://www.python.org/dev/peps/pep-0352/
Change-Id: I6b5596d2376a4013754471ce834e2ac651e377ba
Currently the error message returned by failed query parameter validation
didn't include correct parameter name. This patch correct that.
Change-Id: Ifd0b37084709c2728749b5b7b317f304b0488269
Closes-Bug: #1655710
There are 25 occurences in the code where post method is
defined in the setUp method of the inherited classes.
This patch is intended to move the post method to the
APIValidationTestCase base class to reduce size of code.
TrivialFix
Co-author-by: Dinesh Bhor <dinesh.bhor@nttdata.com>
Change-Id: I509e6e9dbc52f9715276247d8e8c5bb7683d2254
Use the flake8 plugin flake8-import-order to check import ordering. It
can do it automatically and don't need reviewers to check it.
Change-Id: Ia3d81bbbb44b40804b3268c0e648276a36cb4805
Latest jsonschema changes the exception messages. In this
changeset, we assert that the exception message should be
either the one generated by the old library or one from the
new library. Ideally, we should not be string comparing the
error messages at all.
Change-Id: I2faca143c72005c42ef9ce5576013390d0475fb6
This patch enables a white list for query parameter.
The parameters, which are out of white list, will be ignored.
The sort_key still accepts open value. The later patch will add white
list to it.
Co-Authored-By: Zhenyu Zheng <zhengzhenyu@huawei.com>
partial implement of bp add-whitelist-for-server-list-filter-sort-parameters
Change-Id: I7141eef6a1c85ec6d6e8ee170911572535652978
The new decorator 'query_schemas' supports the validation of
query parameters. It also supports specifying the minimum and maximum
API microversion of the schema used for validation.
The query parameters will be converted to a dict of lists. For single
value or multiple values parameter, its value will always be a list.
For example, the 'name' parameter is single value parameter and the
'sort_key' is multiple values parameter. They will be converted to the
dict as:
{'name': ['VM1'], 'sort_key': ['display_name', 'updated_at']}
Their JSON-Schema are as below:
name = {'type': 'array', 'items': {'type': 'string'}, 'maxItems': 1}
sort_key = {'type': 'array', 'items': {'type': 'string'}}
To get rid of the duplicated part of JSON-Schema, the macro function
'single_param' and 'multi_params' are added. Their schema can be just
as below:
name = single_param({'type': 'string'})
sort_key = multi_param({'type': 'string'})
Partial of implementation blueprint consistent-query-parameters-validation
Change-Id: I7d5c353680cafbd59692bd462e9d3f8766021310
* Convert argument to base64.decodestring() to bytes for PY3.
* Fix an issue with python3.5 where the format of an internal error
message changed.
Change-Id: If8184c190e76d8cefb5b097f8fa8cb7564207103
Closes-Bug: 1598062
There are some regexes used in the json-schema for complex string validation.
When the validation failed, json-schema lib didn't generate a useful error
info for the user, it shows the regex to the user. But regex is really
unreadable for normal user. This patch override the default FormatChecker to
support passed in custom error message. This required using custom format
checker instead of using 'pattern'.
For aggregates API, it enabled 'null' in the name input. As the 'format'
keyword works for all allowed types and name format check will only validate
string type, so this patch change the schema to use 'oneOf' keyword, then
the 'format' will only against on string type.
Change-Id: Ic0e608b8a18b635bfcd936f57f14c9f54e1ef8b4
Partial-Bug: #1541691
This reverts commit 8093549e58.
some command like nova aggregate-create foo will fail
guess the reason is we honor the format of the parameter
but it's possible to set the type like following
availability_zone['type'] = ['string', 'null']
which makes the input
{"aggregate": {"name": "foo", "availability_zone": null}}
is also valid
so in the following code, re.search encounter a NONE value
and raise a TypeError
@jsonschema.FormatChecker.cls_checks('name', exception.InvalidName)
def _validate_name(instance):
regex = parameter_types.valid_name_regex
if re.search(regex.regex, instance):
guess we can copy additional code from jsonschema but that need additional effort.
Change-Id: I69619c1b5a5dd3b239100fc52cb65488fbc72609
There are some regexes used in the json-schema for complex string validation.
When the validation failed, json-schema lib didn't generate a useful error
info for the user, it shows the regex to the user. But regex is really
unreadable for normal user. This patch override the default FormatChecker to
support passed in custom error message. This required using custom format
checker instead of using 'pattern'.
Change-Id: Ie6d789e38f64e04a3a76618639c34d36591d6187
Partial-Bug: #1541691
The previous name regexes were built by walking the entire utf8 4 byte
space and building out a regex with all characters that were
printable. When dumped over the wire on a validation error this
creates a nearly 2 MB text string (which is also pushed to the logs).
We can be smarter and assemble character ranges because a-z in regex
means all character values between those 2 character positions. This
makes a validate regex approximately 4% the size of the previous one
(len 1920 vs. len 54263). It thus mitigates the data dump (though it
does not really give a more clear message).
Tests were added to ensure we were building the ranges correctly, as
this is sufficiently tricky.
There will be follow up to create a better error message. A few
additional low level validation tests of the regex themselves were
added in the process of debugging the exclusion support for cells.
Change-Id: I1d3cc40928d80b4397756846ab749b8bacf17fc9
Partial-Bug: #1541691
This patch adds a microversions schema unit test for passing
None as min_version of schema() decorator.
Change-Id: I7d05e6331c07c659f6a97267579f154a7be269f6
In the V2 API, there are three cases for the name field:
1. disallow any space in the name: server_groups.
2. allow leading/trailing whitespaces, strip spaces and disallow
all characters are spaces: flavor_manage, servers.
3. allow leading/trailing whitespacess, no strip spaces and allow
all characters are spaces: aggregates, cells, create_backup,
security_groups, create_image, rebuild
But currently in the V2.1 API and V2.1 API compat mode disallows
leading/trailing in the name field.
For the V2.1 compat mode, we should relax the validation to avoid
breaking the user, although leading/trailing is unclear usecase. This
patch allows leading/trailing spaces but will strip them, and still
disallows that all characters are spaces in the name fields for
flavor_mange, servers, aggregates(and availability_zones),
create_backup, create_image, rebuild.
Due to the server_groups and security_groups(no jsons-schema in v2.1)
have consistent behavior between v2 and v2.1. So this patch won't
change server_groups.
But when creating servers, the name of security_groups, availability_zone
and keyapir isn't stripped the leading/trailing spaces. This is for
backward compatible with users who already use legacy V2 API created
security_group, availabilit_zone and keypair with leading/trailing
in the name, otherwise the users can't use those resource anymore.
For supporting servers schema extension point returns legacy v2 schema,
this patch adds version parameter to the schema extension point. Then
extension point can return different schema based on the version
parameter.
Change-Id: I9442891272284d395ea0dd8cfa302d3f74bf13ec
Partial-Bug: #1498075
We are facing v2 compatibility problems for v2.0 compatible API which
became the default API on /v2 endpoint in Liberty. For fixing these
problems, we need to change some API schemas but we cannot change them
directly because these schemas are used for v2.1 API also and we need
to bump a new microversion for changing them as microversions contract.
To fix these problem without v2.1 API schema changes, this patch
separates the API schemas of v2.0 compatible API from v2.1 API ones.
If we need to separate v2.0 schemas from v2.1, we can specify the
separated schema with '2.0' to the decorator @validation.schema like:
@validation.schema(schema_v20, '2.0', '2.0')
@validation.schema(schema, '2.1')
def update(self, req, id, body):
...
Change-Id: If35306b6a9dfc0da355b9edbf4451bf72516da24
LegacyV2CompatibleWrapper adds environ variable 'openstack.legacy_v2'
to indicate request coming from v2 request.
Based on 'openstack.legacy_v2' will skip additionalProperties checks
in JSON-Schema validation.
Partial implements blueprint api-relax-validation
Change-Id: Ic95dd0d1b77994ce6e47498c8ae0d7ae079706be
This patch adds json-schema for v2.1 fixed-ips. The json-schame limits
the input for actions reserve/unreserve. Also fixed the API sample
file at same time and add parameter_type none.
Co-Authored-By: Ken'ichi Ohmichi <ken1ohmichi@gmail.com>
Change-Id: Id2d0416c3ccc2a50d3cb66bed8747082f98fb194
Closes-Bug: #1438480
This patch changes, where it's posible, tests-package
to using test.NoDBTestCase saving test case running time.
Co-Authored-By: yuntongjin <yuntongjin@gmail.com>
Change-Id: I95945f0bf23c9ef963a4b8634ac4ed6236d97ac7
This allows all printable unicode characters and horizontal whitespace
characters in resource names (flavors, aggregates, cells, etc.), as
opposed to the rather limited set that was previously available. This
follows the principle of not creating unnecessary restrictions.
Implements: blueprint relax-resource-name-restrictions
Closes-Bug: 1366778
Change-Id: I35104852797dcba4594af4361bf9226e16bfb114
Adds the ability to specify minimum and maximum API microversion
versions on jsonschema validation decorators. Validation will
only occur through the decorator if the incoming request version
matches the version range specified. If no range is specified then
validation will always occur.
Partially implements blueprint api-microversions
Change-Id: Ia71963161966af3ca0e6e30e2245f12120f8f8d1
Nova accepts CIDR attributes on network REST APIs and checks its format
on each API. On v2.1 API, the attributes will be checked with JSONSchema.
In addition, Neutron contains its own "cidr" check for its REST APIs.
This patch add a validation format "cidr" by porting Neutron's check from
https://github.com/openstack/neutron/blob/master/neutron/api/v2/attributes.py#L293
Partially implements blueprint v2-on-v3-api
Change-Id: I03601f8fa34fe7617e4f963cdfd4b178eae36565
As part of the split of functional and unit tests we need to isolate
the unit tests into a separate directory for having multiple test
targets in a sane way.
Part of bp:functional-tests-for-nova
Change-Id: Id42ba373c1bda6a312b673ab2b489ca56da8c628
Stephen Finucane