We removed compatibility with Python 3.8 once but it was added back to
tested runtimes for 2023.2. Thus we have to make sure the code works
with Python 3.8, which was broken by [1].
Note that pytz is added back to requirmenets.txt and is now required
regardless of the python version. This is a short term solution until
we figure out the way to fix the requirement-check job (or we again
remove python 3.8 support).
[1] a326ec5eea
Change-Id: I3b222bb59260dff7a06a5ed48720df3dc8c74ea7
Zoneinfo was introduced within python 3.9.
The support of pytz will be removed within RHEL 10 [1].
2023.2 (bobcat) will move our testing runtime to py3.9 and py3.10
so we want to see pytz removed within this series.
tzdata is required at runtime in our gates, because, by default,
zoneinfo uses the system’s time zone data if available; if no system
time zone data is available, the library will fall back to using the
first-party tzdata package available on PyPI. Apparently our gates have no
time zone data available nor tzdata installed by default because we get the
following error without tzdata installed [3]:
`ModuleNotFoundError: No module named 'tzdata'
So I prefer to add tzdata in our requirements to avoid runtime failure
related to time zone and ensure that time zone are always available.
[1] https://issues.redhat.com/browse/RHEL-219
[2] https://review.opendev.org/c/openstack/governance/+/872232
[3]
https://zuul.opendev.org/t/openstack/build/0a1576775e894b09bc31269fea00ba03/log/job-output.txt#1445`
Depends-on: https://review.opendev.org/c/openstack/requirements/+/875854
Change-Id: I8d87d54f6f5ded8caee6cb780bacb39afea0fea1
These were moved in Python 3.3 and the aliases will be removed in 3.10.
Change-Id: If31797f8fc758468d153c364700db47452edf3dd
Signed-off-by: Stephen Finucane <sfinucan@redhat.com>
Currently, using Cinder's backup service with RBD the
backup-create operation gets stuck when logging
('use_json=True' must be set in the config file).
The oslo.log JSONFormatter gets stuck when passing an
RBDVolumeIOWrapper from os-brick. This happens via os-brick's
utils.trace() method which passes a connector containing
{'path': RBDVolumeIOWrapper}.
The oslo.log JSONFormatter format() method calls
oslo_serialization's jsonutils.to_primitive and passes in
this RBDVolumeIOWrapper object.
Therefore the to_primitive method eventually calls
RBDVolumeIOWrapper.read(). In order to fix this the current
path avoids mapping io.IOBase objects and fallback the wrapper
RBD volume object.
Co-authored-by: Eric Harney <eharney@redhat.com>
Closes-Bug: #1908607
Change-Id: I3c416e855cb5f0dc32d14b2749ba92aba8964574
Now that we no longer support py27, we can use the standard library
unittest.mock module instead of the third party mock lib.
Change-Id: I5b6f25b1154b9d655843615ca8402fe4f73e923f
Signed-off-by: Sean McGinnis <sean.mcginnis@gmail.com>
In PyYAML 5.1 they changed this default in the library. Because
we weren't explicitly setting it, this broke our unit tests.
Since default_flow_style=False is usually what you want (hence why
PyYAML changed it), let's set it explicitly.
Also note that this was already being set for dump, but not for
dumps. This is also more consistent.
Change-Id: I3f7dfce5ceb5d3b802878c410406994cbb0988b6
A lot of openstack components like:
- solum
- tosca-parser
- heat
- murano
- etc...
reimplement theirs own yaml parser for loading and dumping.
These implementations sometimes forgot to use a safe loader
or safe dumper, our implementation use safe by default.
You can deactive safe by passing the argument is_safe to false when
you call oslo_serialization.yamlutils.load or oslo_serialization.yamlutils.dump.
Change-Id: I63e85a2b4fc999e6acac12ae51c2ab8c64bddbc6
Co-Authored-By: Natal Ngétal <hobbestigrou@erakis.eu>
Under Python 3.7, a trailing comma is no longer added to
the init parameters generated by a repr() call:
>>> repr(Exception('It Works'))
"Exception('It Works')"
vs
>>> repr(Exception('It Works'))
"Exception('It Works',)"
Support pre and post Python 3.7 formats in test cases.
Change-Id: I45bdf565e170793d0342a907628638369d4d0f2f
Closes-Bug: #1783638
When serializing an exception return its 'repr'. A 'repr' is a
printable representation of an object.
For example the exception: ValueError("an exception") will be returned
as the string: "ValueError('an exception',)"
Change-Id: Iac9f4624bcc4ff65e27c9ca20c6cbbe9481cf334
For example, to_primitive(fallback=repr) can be used to prevent
serialialization error like "ValueError: Circular reference detected"
when using the JSONFormatter of oslo.log.
If fallback is set, it is also used to convert itertools.count(),
"nasty" objects like types, and to handle TypeError.
Use fallback=six.text_type to convert objects to text.
This patch doesn't change the default behaviour.
Related-Bug: #1593641
Change-Id: Ie0f7f2d09355c3d2a9f7c5ee8f7e02dfea3b073b
We try to raise ValueError directly when can't convert the value into
proimitive in If9e8dd5cc2634168910d5f9f8d9302aeefa16097, but revert it
due to some reasons The better way is that notify consuming projects
with the future API behavior change before the change. It will raise
ValureError in version 3.0.
Partial-Bug: #1593641
Change-Id: I2eff07cd25f0565b380cb6e76628a896c8d0ec61
This reverts commit 3727b2d6e0.
It breaks nova's test, need log warning before we make major release.
Change-Id: Ibbfde6207fc8c066155758c7c1aa1521b95d4db1
Currently, to_primitive tries to iterate through all addresses in the
network, because the type doesn't have a special handling that would
short curcuit it, but also has __iter__. This may be detrimental to
performance, up to the point of node crash due to memory exhaustion if
the passed network range is too large (think of 0.0.0.0/0 or even
2001::/64). This behavior also makes it impossible to restore the
original data format (CIDR).
This patch short curcuits the iteration by handling the IPNetwork type
as a special case, same as we do for IPAddress.
Change-Id: I6aecd2d057d282a655ff9e4918c164253142b188
Closes-Bug: #1698355
The problem in the current version of to_primitive function
from jsonutils module is in the situation when the function
doesn't know how to convert an object to primitive. In that
case the function simply returns the same object which causes
the following exception later in json.dumps:
ValueError: Circular reference detected. This exception is not
obvious and is quite misleading. So I think it would be better
to explicitly raise ValueError here.
Closes-Bug: #1593641
Change-Id: If9e8dd5cc2634168910d5f9f8d9302aeefa16097
The current change made in de68f08d37 breaks cinder and other
projects that were expecting a TypeError, so for now and to keep
those projects operating translate the py3.x exception from binascii
into a type error to prefer consistency with existing code.
Change-Id: I4575ea3dad51be9bb2278eb0bfa31cef54c300d5
Openstack common has a wrapper for generating uuids.We should
use that function when generating uuids for consistency.
Change-Id: Ifb450c91a1b91f016b4f9ad3a4367e8117b89a0d
Closes-Bug: #1082248
Many assertEqual sentences don't follow assertEqual(expected, actual),
These misplaces have 2 impacts:
1, giving confusing messages when some tests failed.
2, mislead other developers, new test modules may follow these wrong pattern.
This patch fix all of them.
Change-Id: Ice3148685b81bf906672b91159fbb29bd618abb0
Closes-Bug: #1604213
There are some differences in the behaviour of json and msgpack
serializers. Msgpack throws TypeError when it doesn't know
how to handle an object. Json throws ValueError instead. Regarding
the fact that serialization should be configurable (in particular,
serializers should have similar behaviour; right now transition from
json to msgpack leads to errors and fails) this patch proposes to
raise ValueError instead of TypeError on failures so that libraries
already using jsonutils would be able to also work with msgpackutils.
Change-Id: I3d21b7d136e5a426a3c4a70a953c82ddcd6ef5af
Fixes jsonutils.to_primitive so that it can serialize ipaddress
objects, as used by Python 3 or through the backport in
global-requirements.txt. Current support for netaddr is maintained.
Change-Id: I6aa1ecd5be754f61cdda4e47bc0e0180e232b366
Closes-bug: 1590076
In order to make it easy to extend the default msgpack extension
registry add a copy method to registries and select type handlers
so that a registry with additional values can be easily provided
to the various msgpack utils loads/dumps code.
This also re-adds the msgpackutils tests which appear to have
not shifted over (or otherwise disappeared somehow?) from when the
tests used to live in the root directory.
Closes-Bug: #1537716
Change-Id: I9785a9071044d0e8c044a36ae9af9912fd06c49d
OpenStack has ended support for Python 2.6 in all services and
libraries. We no longer need to support a fallback to simplejson on
Python 2.6 so we can remove this fallback logic.
Closes-bug: 1533337
Change-Id: I613541a37229ff2f3ce269cfa7496223d1ff37d0
Writing a code using the base64 module (of the Python standard
library) working on Python 2.7 and 3.4 requires many checks on the
input and/or output type:
* base64.b64encode() only accepts byte string: text must be
explicitly encoded to ASCII
* base64.b64decode() returns bytes: output must be decoded from UTF-8
when text is expected
This change adds two pairs of encode/decode functions:
* encode_as_bytes(), decode_as_bytes(): always return the result as
a byte string
* encode_as_text(), decode_as_text(): always return the result as a
text string
Encode functions accept text: text is encoded to UTF-8 by default,
but the encoding is configurable.
Decode functions accept text: text is decoded from ASCII.
decode_as_text() decodes the result from UTF-8 by default, but again
the encoding is configurable.
The new submodule is called "base64" to be able to replace:
import base64
with:
from oslo_serialization import base64
If the base64 module of the stdlib is needed, it can be imported as a
different name. Example:
import base64 as std_base64
The encoding example:
if isinstance(text, six.text_type):
text = text.encode('utf-8')
text_b64 = base64.b64encode(text)
text_b64 = text_b64.decode('ascii')
can be replaced with:
text_b64 = base64.encode_as_text(text)
The decoding example:
if isinstance(encoded, six.text_type):
encoded = encoded.decode('ascii')
text = base64.b64decode(encoded)
text = text.decode('utf-8')
can be replaced with:
text = base64.decode_as_text(text)
Change-Id: Icf8df9c947bc0c5f4838508b756ed8f53efd9fc4
The jsonutils.dumps() function returns bytes on Python 2 and Unicode
oon Python 3. In some cases, we always want bytes. For example, a
HTTP body must be bytes. This function avoids an condition call to
.encode() depending on the type or on the Python version.
For example:
body = jsonutils.dumps(data)
if isinstance(body, six.text_type):
body = body.encode('utf-8')
can be replaced with:
body = jsonutils.dump_as_bytes(data)
Change-Id: Ib9d8f1309982762b54d8a91b1f24f64d0ae6723a
This is a wire format change that the projects are not ready for yet (example Nova).
We should do this as part of a major version bump (2.x) with ample time for folks to move up.
Nova review to prepare the Nova code base for this change:
Ib7e60ed94edd1eb409baf8c29c61237d85944787
This reverts commit 5b0827a3df.
Change-Id: Ic4bf047b40c1dad2f8d2bc0afda934e47dba1f9d
Following discussion in change I8b5119e64369ccac3423dccc04421f99912df733
in oslo.utils, we plan to deprecated and remove strtime() which is to be
considered dangerous.
This changes the json serialization code to embed the timezone in the
encoded datetime string if it's available.
Change-Id: I262391987d25b483d2bec4ac3abe94b6e9d032ad
Move the public API out of oslo.serialization to
oslo_serialization. Retain the ability to import from the old namespace
package for backwards compatibility for this release cycle.
bp/drop-namespace-packages
Change-Id: Ic60f809ea00ac77b0753556c6fd00b97e64d57ff
Takashi Kajinami