openstack
/
openstack-specs
Code Issues Proposed changes
openstack-specs/specs/enable-python-3-int-func-te...

9.2 KiB
Raw Blame History

Enabling Python 3 for Integration and Functional Tests

The 2.x series of the C Python interpreter on which OpenStack releases through Kilo are built is reaching the end of its extended support period, defined by the upstream developers. This spec describes motivation for porting fully to Python 3 and some of the work we will need to enable testing applications as they move to Python 3.

Problem description

There are a lot of small motivations for moving to Python 3, including better unicode support and new features in the language and standard library. The primary motivation, however, is that Python 2 is reaching its end-of-life for support from its developers.

Just as we expect our users to update to new versions of OpenStack in order to continue to receive support, the python-dev team expects users of the language to update to reasonably modern and supported versions of the interpreter in order to receive bug and security fixes. When Python 3 was introduced, the support period for Python 2 was extended beyond the normal length of time to allow projects plenty of time to migrate, and to allow the python-dev team to receive feedback to make changes to the language so that migration is easier. That period is coming to an end, and we need to consider migration seriously.

"Python 3.0 was released in 2008. The final 2.x version 2.7 release came out in mid-2010, with a statement of extended support for this end-of-life release. The 2.x branch will see no new major releases after that. 3.x is under active development and has already seen over five years of stable releases, including version 3.3 in 2012 and 3.4 in 2014. This means that all recent standard library improvements, for example, are only available by default in Python 3.x." -- Python2orPython3

That said, we cannot expect all of OpenStack to be ported at one time. It's likely that we could not port everything in a single release cycle, given the other work going on. So we need a way to stage the porting work so that projects can port when they are ready, without having to wait for any other projects to finish their ports.

Proposed change

Our services communicate through REST APIs and the message bus. This means they are decoupled enough that we can port them one at a time, if our tools support running some services on Python 2 and some on Python 3. Our unit test tool, tox, supports multiple Python versions already, and in fact most of our library projects are testing under Python 2.6, 2.7, and 3.4 today. Our integration tests, however, do not yet support multiple Python versions, so that's the next step to take.

General Strategy

  1. Update devstack to install apps with the "right" version of the interpreter.
    • Use the version declared to be supported by the project through its trove classifiers.
    • Allowing apps to be installed with the right version of the interpreter independently of other apps means we can port one app at a time.
  2. Port each application to 3.4, but support both 2.7 and 3.4.
    • Set up an appropriate devstack-gate job using Python 3 as non-voting for projects when they start to port.
    • Make incremental changes to the applications until the non-voting job passes reliably, then update it to make it a voting job.
    • Technically there is no need to run integration tests for an application under both versions, since they only need to be deployed under one version at a time. However, different packagers and deployers may want to choose to wait to move to Python 3 and so we can continue to run the tests under both versions.

Note

Even after all applications are on 3.x, we need to maintain some python 2.7 support for client libraries and the Oslo libraries they use. We should consider the deprecation policy of Python 2 for the client libraries independently of porting the applications to 3.

Which version of Python to use?

We have discussed this before, and it continues to be a moving target. Version 3.4 seems to be our best goal for now.

  • 3.0 - 3.2 are no longer actively supported
  • 3.3 is not available on all distros
  • 3.4 is (or soon will be) available on all distros
  • 3.5 is in beta and so is not ready for us to use, yet

Functional Tests for Libraries

Besides the functional and integration tests for applications, we also have functional tests for libraries. I propose that we configure the test jobs to run those only under Python 3, to avoid duplication and expose porting issues that would have an impact on applications as early as possible.

Alternatives

Stay with C Python 2

Commercial support is likely to be available from distros for longer than it is available upstream, but even that will stop at some point.

Use PyPy or Another Implementation

Some applications may benefit from PyPy's JIT compiler. It currently supports 2.7.8 and 3.2.5, which means our Python 2 code would probably run but code designed for Python 3.4 will not. I'm not aware of any large deployments using PyPy to run services, so I'm not sure this is really a problem. Given the expected long time frame for porting to Python 3, it is likely that PyPy will be able to catch up to the language level needed to run OpenStack by the time we are fully moved to Python 3.

Wait for Python 3.5

Moving from 3.4 to 3.5 should require much less work than moving from 2.7 to 3.4. We can therefore start now, and monitor adoption of 3.5 by distributions to decide whether to ultimately use 3.4 or a later version.

Use Separate Virtualenvs in devstack

We have discussed installing applications into virtualenvs a couple of times. Doing that is orthogonal to these proposed changes, since we would still need to use the correct version of Python within the virtualenv.

Functional tests for libraries on 2 and 3

We could run parallel test jobs configured to run the functional tests for libraries under both Python 2 and 3. This would largely duplicate effort, though it might uncover some inconsistent handling of bytes vs. strings. We shouldn't start out trying to do this, but if we do uncover problems we can add more test jobs.

Implementation

Assignee(s)

Primary assignee: Doug Hellmann

Work Items

  1. Update devstack to install pip for both Python 2 and Python 3.

  2. Update devstack to look at the supported Python versions for a project, and choose the correct copy of pip to install it and its dependencies.

    This may be as simple as:

    python setup.py --classifiers | grep 'Language' | cut -f5 -d: | grep '\.'

  3. When installing libraries from source using the LIBS_FROM_GIT feature of devstack, ensure that the libraries are installed for both Python 2 and Python 3.

  4. Begin porting applications to Python 3.

    • Unit tests can be run under Python 3 for applications just as they are for libraries, by enabling the appropriate job. Having the unit tests working with Python 3 is a good first step, before enabling the integration tests.
    • Integration tests can be run by submitting a patch updating the trove classifier.
    • Some projects will have dependencies blocking them from moving to Python 3 at first, and those should be tracked separately from this proposal.

Some functions in Oslo libraries have been identified as having incompatibilities with Python 3. As these cases are reported, we will need to decide, on a case-by-case basis whether it is feasible to create versions of those functions that work for both Python 2 and 3, or if we will need to create some new APIs for use under Python 3 (see oslo_utils.encodeutils.safe_decode, oslo_utils.strutils.mask_password, and oslo_concurrency.processutils.execute as examples).

References

History

Revisions
Release Name Description
Liberty Introduced

Note

This work is licensed under a Creative Commons Attribution 3.0 Unported License. http://creativecommons.org/licenses/by/3.0/legalcode