Commit Graph

75 Commits

Author SHA1 Message Date
Zuul c7d4c0c74b Merge "pre-commit: Bump versions" 2024-01-28 09:00:01 +00:00
Stephen Finucane 5587e9555e pre-commit: Bump versions
hacking now has a native hook.

Change-Id: Ibdf50f4e1e1cfe6056f25322f66d4e95f5930dc0
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
2024-01-03 12:12:14 +00:00
Takashi Kajinami c83a4c9968 jsonutils: Raise ValueError in case the input can't be converted
According to the warning it was planned that this change was made in
3.0.

Change-Id: I0c1011b935f52f5ea1329e0316f935a034fdfd0e
2023-11-08 17:41:29 +00:00
Hervé Beraud 61c460f3f2 Fix compatibility with Python 3.8
We removed compatibility with Python 3.8 once but it was added back to
tested runtimes for 2023.2. Thus we have to make sure the code works
with Python 3.8, which was broken by [1].

Note that pytz is added back to requirmenets.txt and is now required
regardless of the python version. This is a short term solution until
we figure out the way to fix the requirement-check job (or we again
remove python 3.8 support).

[1] a326ec5eea

Change-Id: I3b222bb59260dff7a06a5ed48720df3dc8c74ea7
2023-05-16 10:24:51 +02:00
Hervé Beraud a326ec5eea Implement zoneinfo support to drop dependency to pytz
Zoneinfo was introduced within python 3.9.

The support of pytz will be removed within RHEL 10 [1].

2023.2 (bobcat) will move our testing runtime to py3.9 and py3.10
so we want to see pytz removed within this series.

tzdata is required at runtime in our gates, because, by default,
zoneinfo uses the system’s time zone data if available; if no system
time zone data is available, the library will fall back to using the
first-party tzdata package available on PyPI. Apparently our gates have no
time zone data available nor tzdata installed by default because we get the
following error without tzdata installed [3]:
`ModuleNotFoundError: No module named 'tzdata'

So I prefer to add tzdata in our requirements to avoid runtime failure
related to time zone and ensure that time zone are always available.

[1] https://issues.redhat.com/browse/RHEL-219
[2] https://review.opendev.org/c/openstack/governance/+/872232
[3]
https://zuul.opendev.org/t/openstack/build/0a1576775e894b09bc31269fea00ba03/log/job-output.txt#1445`

Depends-on: https://review.opendev.org/c/openstack/requirements/+/875854
Change-Id: I8d87d54f6f5ded8caee6cb780bacb39afea0fea1
2023-03-23 17:17:59 +01:00
songwenping 0f5451555b Remove unnecessary unicode prefixes
Change-Id: I0910f62e7feaf928fe0afe4721c5b271d4748449
2022-04-19 19:48:16 +08:00
Stephen Finucane 967a36fdbe Drop use of deprecated collections classes
These were moved in Python 3.3 and the aliases will be removed in 3.10.

Change-Id: If31797f8fc758468d153c364700db47452edf3dd
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
2021-02-01 11:05:36 +00:00
Sofia Enriquez 02037330d8 Fix json to_primitive when using IO OBjects
Currently, using Cinder's backup service with RBD the
backup-create operation gets stuck when logging
('use_json=True' must be set in the config file).

The oslo.log JSONFormatter gets stuck when passing an
RBDVolumeIOWrapper from os-brick. This happens via os-brick's
utils.trace() method which passes a connector containing
{'path': RBDVolumeIOWrapper}.
The oslo.log JSONFormatter format() method calls
oslo_serialization's jsonutils.to_primitive and passes in
this RBDVolumeIOWrapper object.
 
Therefore the to_primitive method eventually calls
RBDVolumeIOWrapper.read(). In order to fix this the current
path avoids mapping io.IOBase objects and fallback the wrapper
RBD volume object.

Co-authored-by: Eric Harney <eharney@redhat.com>
Closes-Bug: #1908607
Change-Id: I3c416e855cb5f0dc32d14b2749ba92aba8964574
2021-01-18 21:54:02 +00:00
Zuul d94ba92406 Merge "Remove the yamlutils module." 2020-06-08 12:42:49 +00:00
Daniel Bengtsson 3847688ab7 Remove the yamlutils module.
The pyyaml is now safe by default:

https://github.com/yaml/pyyaml/pull/74/files
https://access.redhat.com/security/cve/CVE-2017-18342
https://github.com/search?q=oslo.serialization+yamlutils+user%3Aopenstack+language%3APython&type=Code

So the yamlutils is now useless. No project in OpenStack used this
module. We can remove it.

Sem-Ver: api-break
Change-Id: If0a98f1e15d83131cb309e02f892372f353fa5da
2020-06-08 11:44:27 +02:00
Hervé Beraud 00a87b5059 Stop to use the __future__ module.
The __future__ module [1] was used in this context to ensure compatibility
between python 2 and python 3.

We previously dropped the support of python 2.7 [2] and now we only support
python 3 so we don't need to continue to use this module and the imports
listed below.

Imports commonly used and their related PEPs:
- `division` is related to PEP 238 [3]
- `print_function` is related to PEP 3105 [4]
- `unicode_literals` is related to PEP 3112 [5]
- `with_statement` is related to PEP 343 [6]
- `absolute_import` is related to PEP 328 [7]

[1] https://docs.python.org/3/library/__future__.html
[2] https://governance.openstack.org/tc/goals/selected/ussuri/drop-py27.html
[3] https://www.python.org/dev/peps/pep-0238
[4] https://www.python.org/dev/peps/pep-3105
[5] https://www.python.org/dev/peps/pep-3112
[6] https://www.python.org/dev/peps/pep-0343
[7] https://www.python.org/dev/peps/pep-0328

Change-Id: I973fc9523a32651fa016a6e09fdffc4540aacd87
2020-06-02 20:40:37 +02:00
Sean McGinnis d3159d4b60
Use unittest.mock instead of third party mock
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib.

Change-Id: I5b6f25b1154b9d655843615ca8402fe4f73e923f
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
2020-03-13 11:46:12 -05:00
Ben Nemec 909c17a779 Remove Python 3-specific conditional in dump_as_bytes
Now that we only support Python 3 we don't need logic to handle
Python 2.

Change-Id: I33747d25bc4c67ee13654ed132edf7413371c1dc
2020-03-06 17:17:33 +00:00
Stephen Finucane ad356ea9c2 Drop use of six
Another one bites the dust.

Change-Id: I1fadcad8219322b569eeecd81e454a44641e8b1e
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
2020-03-02 10:10:22 +00:00
Stephen Finucane 9fee0c4f48 Remove long-dead workaround
We don't need to worry about versions of Mox released in 2011 any more

https://bugs.launchpad.net/nova/+bug/852095

Change-Id: Ifd2228071c70e516f11eefde29932813b7cf6a97
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
2020-02-27 15:41:09 +00:00
Daniel Bengtsson 84363e91a0 Deprecated the yamlutils module.
The pyyaml is now safe by default:

https://github.com/yaml/pyyaml/pull/74/files
https://access.redhat.com/security/cve/CVE-2017-18342

So the yamlutils is now useless. We can depracated it and then remove it.

Change-Id: I4ecb34eee942c714d09d2258db80f5b8d61dec89
2020-02-14 13:03:14 +01:00
Ben Nemec 448807a706 Explicitly set default_flow_style to False
In PyYAML 5.1 they changed this default in the library. Because
we weren't explicitly setting it, this broke our unit tests.
Since default_flow_style=False is usually what you want (hence why
PyYAML changed it), let's set it explicitly.

Also note that this was already being set for dump, but not for
dumps. This is also more consistent.

Change-Id: I3f7dfce5ceb5d3b802878c410406994cbb0988b6
2019-05-02 17:05:24 +00:00
Zuul 2d2d84e26d Merge "Introduce a base yaml parser for all openstack components" 2019-03-25 17:40:26 +00:00
Hervé Beraud 35dae9c288 Introduce a base yaml parser for all openstack components
A lot of openstack components like:
- solum
- tosca-parser
- heat
- murano
- etc...

reimplement theirs own yaml parser for loading and dumping.

These implementations sometimes forgot to use a safe loader
or safe dumper, our implementation use safe by default.

You can deactive safe by passing the argument is_safe to false when
you call oslo_serialization.yamlutils.load or oslo_serialization.yamlutils.dump.

Change-Id: I63e85a2b4fc999e6acac12ae51c2ab8c64bddbc6
Co-Authored-By: Natal Ngétal <hobbestigrou@erakis.eu>
2019-02-21 17:34:38 +01:00
ZhijunWei c9b464c099 Update hacking version
Use latest release 1.1.0 and compatible changes w.r.t pep8

Change-Id: I95d62bbce7fe20dd2f9ec8a96cdffeeefcda75be
2019-02-17 10:28:24 +08:00
Corey Bryant 254e0ae839 py37: deal with Exception repr changes
Under Python 3.7, a trailing comma is no longer added to
the init parameters generated by a repr() call:

    >>> repr(Exception('It Works'))
    "Exception('It Works')"

vs

    >>> repr(Exception('It Works'))
    "Exception('It Works',)"

Support pre and post Python 3.7 formats in test cases.

Change-Id: I45bdf565e170793d0342a907628638369d4d0f2f
Closes-Bug: #1783638
2018-07-30 16:46:43 -04:00
Morgan d5855294a8 Stop using encoding='utf-8' with msgpack
msgpack is throwing a warning that the unpacker has deprecated the use
of encoding argument. Instead the unpacker uses `raw=False`, which
does an unpack with similar behavior to the old encoding='utf-8'
mechanism.

This will reduce a lot of warning spam for projects using msgpack
such as Keystone. The change to the kwarg "raw" from "bytes_as_raw"
happned between 0.5.1 and 0.5.2. Changing the lower constraint is
required to ensure the new arguments are matching the method
signature.

Change-Id: Iadbee3ec8def9512369a415fb4603dc05d0cbc56
2018-06-03 00:25:50 -07:00
John L. Villalovos c1a7079c26 When serializing an exception return its 'repr'
When serializing an exception return its 'repr'. A 'repr' is a
printable representation of an object.

For example the exception: ValueError("an exception") will be returned
as the string: "ValueError('an exception',)"

Change-Id: Iac9f4624bcc4ff65e27c9ca20c6cbbe9481cf334
2017-10-03 09:41:52 -07:00
Victor Stinner cdb2f60d26 jsonutils.to_primitive(): add fallback parameter
For example, to_primitive(fallback=repr) can be used to prevent
serialialization error like "ValueError: Circular reference detected"
when using the JSONFormatter of oslo.log.

If fallback is set, it is also used to convert itertools.count(),
"nasty" objects like types, and to handle TypeError.

Use fallback=six.text_type to convert objects to text.

This patch doesn't change the default behaviour.

Related-Bug: #1593641
Change-Id: Ie0f7f2d09355c3d2a9f7c5ee8f7e02dfea3b073b
2017-09-26 11:20:08 +02:00
ChangBo Guo(gcb) fecad3c31f warn developers when can't convert value into primitive
We try to raise ValueError directly when can't convert the value into
proimitive in If9e8dd5cc2634168910d5f9f8d9302aeefa16097, but revert it
due to some reasons The better way is that notify consuming projects
with the future API behavior change before the change. It will raise
ValureError in version 3.0.

Partial-Bug: #1593641
Change-Id: I2eff07cd25f0565b380cb6e76628a896c8d0ec61
2017-09-19 09:37:39 +08:00
ChangBo Guo(gcb) bbee6d1aa1 Revert "Explicitly raise ValueError in to_primitive"
This reverts commit 3727b2d6e0.

It breaks nova's test, need log warning before we make major release.

Change-Id: Ibbfde6207fc8c066155758c7c1aa1521b95d4db1
2017-07-17 23:28:58 +08:00
Jenkins 2a29a5e836 Merge "Explicitly raise ValueError in to_primitive" 2017-07-05 11:26:43 +00:00
Ihar Hrachyshka 38ac21b523 Don't iterate through addresses in netaddr.IPNetwork
Currently, to_primitive tries to iterate through all addresses in the
network, because the type doesn't have a special handling that would
short curcuit it, but also has __iter__. This may be detrimental to
performance, up to the point of node crash due to memory exhaustion if
the passed network range is too large (think of 0.0.0.0/0 or even
2001::/64). This behavior also makes it impossible to restore the
original data format (CIDR).

This patch short curcuits the iteration by handling the IPNetwork type
as a special case, same as we do for IPAddress.

Change-Id: I6aecd2d057d282a655ff9e4918c164253142b188
Closes-Bug: #1698355
2017-06-16 11:46:36 -07:00
ChangBo Guo(gcb) 3727b2d6e0 Explicitly raise ValueError in to_primitive
The problem in the current version of to_primitive function
from jsonutils module is in the situation when the function
doesn't know how to convert an object to primitive. In that
case the function simply returns the same object which causes
the following exception later in json.dumps:
ValueError: Circular reference detected. This exception is not
obvious and is quite misleading. So I think it would be better
to explicitly raise ValueError here.

Closes-Bug: #1593641

Change-Id: If9e8dd5cc2634168910d5f9f8d9302aeefa16097
2017-05-17 15:55:14 +08:00
Joshua Harlow ac5787d0e4 Prefer raising the python2.x type error for b64 decode errors
The current change made in de68f08d37 breaks cinder and other
projects that were expecting a TypeError, so for now and to keep
those projects operating translate the py3.x exception from binascii
into a type error to prefer consistency with existing code.

Change-Id: I4575ea3dad51be9bb2278eb0bfa31cef54c300d5
2017-01-09 16:37:28 -08:00
Jenkins 004bc47522 Merge "Replace six.iteritems() with .items()" 2017-01-02 21:28:38 +00:00
Stephen Finucane ede68f08d3 Don't raise TypeError for invalid b64
In Python 2, the 'b64decode' function calls the 'binascii.a2b_base64'
function but catches any 'binascii.Error' exceptions raised and raises a
TypeError instead [1]. In Python 3, a 'binascii.Error' error is raised
instead [2]. Rather than forcing users to handle two types of exception,
we can allow them to catch only the 'bisascii.Error'. Python 2 provides
a function that does just this - 'decodestring' - which we can use. Make
it so.

[1] https://github.com/python/cpython/blob/2.7/Lib/base64.py#L78
[2] https://github.com/python/cpython/blob/3.5/Lib/base64.py#L87
[3] https://github.com/python/cpython/blob/2.7/Lib/base64.py#L326

Change-Id: I72c6de71b174181292427128d20e03756f85fb97
2016-12-14 16:16:41 +00:00
gecong1973 1df76ee42d Replace six.iteritems() with .items()
1.As mentioned in [1], we should avoid using
six.iteritems to achieve iterators. We can use
dict.items instead, as it will return iterators
in PY3 as well. And dict.items/keys will more readable.
2.In py2, the performance about list should be negligible,
see the link [2].
[1] https://wiki.openstack.org/wiki/Python3
[2] http://lists.openstack.org/pipermail/openstack-dev/2015-June/066391.html

Change-Id: Ifbe7929580ec61a2cd50232794c0cbd0cbf37edc
2016-12-12 11:11:28 +08:00
Luong Anh Tuan 6704d5f8ba Replaces uuid.uuid4 with uuidutils.generate_uuid()
Openstack common has a wrapper for generating uuids.We should
use that function when generating uuids for consistency.

Change-Id: Ifb450c91a1b91f016b4f9ad3a4367e8117b89a0d
Closes-Bug: #1082248
2016-12-05 11:12:56 +07:00
Jenkins 1f86a66a36 Merge "Fix serialization of binary strings in Python3" 2016-08-04 13:31:57 +00:00
Gevorg Davoian a73120aad6 Fix serialization of binary strings in Python3
Change-Id: I30e7c7b7cb2356a39bc51eb8c93f14c8881a38c2
Closes-Bug: #1599387
2016-08-01 16:58:08 +03:00
yanheven 5ae04325e6 Fix parameters of assertEqual are misplaced
Many assertEqual sentences don't follow assertEqual(expected, actual),
These misplaces have 2 impacts:
1, giving confusing messages when some tests failed.
2, mislead other developers, new test modules may follow these wrong pattern.

This patch fix all of them.

Change-Id: Ice3148685b81bf906672b91159fbb29bd618abb0
Closes-Bug: #1604213
2016-07-23 19:38:43 +08:00
Edan David 1ec85e90cd Use {} instead of dict()
There is a performance impact when using dict() instead of {}
in CPython (up to 6 times longer).
Considering the to_primitive function is recursive this can
have quite an effect.

Measuring:
$ python2.7 -m timeit -n 1000000 -r 5 -v 'dict()'
raw times: 0.24 0.24 0.24 0.239 0.24
1000000 loops, best of 5: 0.239 usec per loop

$ python2.7 -m timeit -n 1000000 -r 5 -v '{}'
raw times: 0.0417 0.0413 0.0407 0.0411 0.042
1000000 loops, best of 5: 0.0407 usec per loop

For more information:
https://doughellmann.com/blog/2012/11/12/
the-performance-impact-of-using-dict-instead-of-in-cpython-2-7-2/

Change-Id: Ia0b5892773a19cbabe40313a3bc788580a943f53
2016-06-21 10:58:06 -04:00
Jenkins 0aa1638408 Merge "Support serializing ipaddress objs with jsonutils" 2016-06-09 08:18:25 +00:00
Gevorg Davoian 4fdaeff758 Replace TypeError by ValueError in msgpackutils
There are some differences in the behaviour of json and msgpack
serializers. Msgpack throws TypeError when it doesn't know
how to handle an object. Json throws ValueError instead. Regarding
the fact that serialization should be configurable (in particular,
serializers should have similar behaviour; right now transition from
json to msgpack leads to errors and fails) this patch proposes to
raise ValueError instead of TypeError on failures so that libraries
already using jsonutils would be able to also work with msgpackutils.

Change-Id: I3d21b7d136e5a426a3c4a70a953c82ddcd6ef5af
2016-06-08 08:41:36 +00:00
Jim Baker 8943c73662 Support serializing ipaddress objs with jsonutils
Fixes jsonutils.to_primitive so that it can serialize ipaddress
objects, as used by Python 3 or through the backport in
global-requirements.txt. Current support for netaddr is maintained.

Change-Id: I6aa1ecd5be754f61cdda4e47bc0e0180e232b366
Closes-bug: 1590076
2016-06-07 14:05:50 -06:00
Jenkins 4217a25d40 Merge "Unified and simplified API for all serializers" 2016-04-08 01:55:32 +00:00
Gevorg Davoian 7ac405ce65 Unified and simplified API for all serializers
This patch is a proposition to the oslo.serialization for making a
more convenient API. Let me explain briefly why the current library
isn't very convenient for its users. Imagine that a library user
want to implement a function having some (de-)serialization inside.
The user also wants to make serialization configurable through
an additional argument to the function (e.g. some id like "json",
"msgpack" etc.). One can achieve this behaviour by importing all
necessary serializers (an import per a serializer) and adding
some logic for selecting an appropriate serialization mechanism
depending on id. But what if some new serializer is added to the
oslo.serialization? Then all users of the library will have to make
changes in their code. It's not good, and I think it would be better
if all changes concerning serializers were in the oslo.serialization
library. Therefore I think it is a good idea to have a module which
will bring together all implemented serializers under a unified and
somewhat simplified (because in most cases default values for some
parameters are fine) object oriented API, so that future users of the
library will be able to get any available serializer by its id
(importing only one module). Also it is worth saying that the patch
doesn't affect already written code, so there won't be any problems
with the backward compatibility.

May be this implementation isn't the best one, but I hope
the community will appreciate the idea and propose possible
improvements.

Change-Id: Idb12666255a990dfc8f8ff6b43e941b3481b9c1c
2016-04-04 15:27:16 +03:00
Jenkins 8cb124e149 Merge "Make msgpack registries copyable (and add __contains__)" 2016-03-24 02:51:19 +00:00
Joshua Harlow 056d701620 Make msgpack registries copyable (and add __contains__)
In order to make it easy to extend the default msgpack extension
registry add a copy method to registries and select type handlers
so that a registry with additional values can be easily provided
to the various msgpack utils loads/dumps code.

This also re-adds the msgpackutils tests which appear to have
not shifted over (or otherwise disappeared somehow?) from when the
tests used to live in the root directory.

Closes-Bug: #1537716

Change-Id: I9785a9071044d0e8c044a36ae9af9912fd06c49d
2016-03-14 22:16:51 -07:00
Mehdi Abaakouk da1475a28f msgpack: fix datetime serialization
If the datetime is serialized with python2 and deserializer
with python3 we got KeyError, because python3 will read
the data written with python2 as bytes and doesn't convert it in unicode.

So ensure we use always unicode everywhere.

Change-Id: I4bee7369a278fa7705cde860c68e133cee2fa79e
Closes-bug: #1556081
2016-03-14 16:56:18 +01:00
Ian Cordasco 29a7702a99 Remove fallback to simplejson in jsonutils
OpenStack has ended support for Python 2.6 in all services and
libraries. We no longer need to support a fallback to simplejson on
Python 2.6 so we can remove this fallback logic.

Closes-bug: 1533337
Change-Id: I613541a37229ff2f3ce269cfa7496223d1ff37d0
2016-01-12 13:53:47 -06:00
ChangBo Guo(gcb) 550afa9c0e msgpackutils: drop Python 2.6 support
We doesn't support python 2.6 now, this commit
removes Python 2.6 related code in module msgpackutls.

Change-Id: Id148239ee6c1ad461188fa38b06ca0d7913d7454
2015-11-21 12:18:30 +08:00
Victor Stinner 2d085d2c17 Use versionadded and versionchanged in doc
Document in which version new types and functions were added using
".. versionadded:: x.y". Document changes using
".. versionchanged:: x.y."

For base64 module, add the versionadded tag in the module top
docstring, not on each type/function.

I used "git blame" + "git tag --contains=SHA1" to find these version,
and then I checked manually each version.

Change-Id: I4a891b18064fe7b857a79c57030ff31f7a0370b4
2015-10-15 14:38:50 +02:00
Victor Stinner e49e812900 Add utilities for base64
Writing a code using the base64 module (of the Python standard
library) working on Python 2.7 and 3.4 requires many checks on the
input and/or output type:

* base64.b64encode() only accepts byte string: text must be
  explicitly encoded to ASCII
* base64.b64decode() returns bytes: output must be decoded from UTF-8
  when text is expected

This change adds two pairs of encode/decode functions:

* encode_as_bytes(), decode_as_bytes(): always return the result as
  a byte string
* encode_as_text(), decode_as_text(): always return the result as a
  text string

Encode functions accept text: text is encoded to UTF-8 by default,
but the encoding is configurable.

Decode functions accept text: text is decoded from ASCII.

decode_as_text() decodes the result from UTF-8 by default, but again
the encoding is configurable.


The new submodule is called "base64" to be able to replace:

    import base64

with:

    from oslo_serialization import base64

If the base64 module of the stdlib is needed, it can be imported as a
different name. Example:

    import base64 as std_base64


The encoding example:

    if isinstance(text, six.text_type):
       text = text.encode('utf-8')
    text_b64 = base64.b64encode(text)
    text_b64 = text_b64.decode('ascii')

can be replaced with:

    text_b64 = base64.encode_as_text(text)


The decoding example:

    if isinstance(encoded, six.text_type):
       encoded = encoded.decode('ascii')
    text = base64.b64decode(encoded)
    text = text.decode('utf-8')

can be replaced with:

    text = base64.decode_as_text(text)

Change-Id: Icf8df9c947bc0c5f4838508b756ed8f53efd9fc4
2015-10-01 14:32:29 +00:00