This is motivated by OpenDev's desire to drop the old Bullseye container
images as well as container images for older versions of python. We bump
testing to python3.11 on the high end of the range and convert the
container image to python3.11 + Bookworm.
Python 2.7 testing is removed because tox + virtualenv can't actually
create python2.7 virtualenvs any longer. There are workarounds but
dropping the tests is simpler.
Python 3.5 testing is replaced with 3.6 testing beacuse the version of
easy_install on ubuntu xenial does not understand TLS + SNI. A while
back pypi.org dropped non SNI connection support which means we can't
install PBR for to run the setup.py for the projcet. There are
workarounds but 3.5 is old enough taht we should be able to move on.
Change-Id: I2f2a2d0cf71c69d7babd6df9bfdb41a759e9c0ee
Latest setuptools has deprecated the use of setup.py commands like
`setup.py testr`. For some reason python3.9 hangs and fails to work at
all when you run `setup.py testr`. Switch to running testr directly and
not bother debugging this too aggressively as this functionality is
going away eventually.
Change-Id: I3ad9e0c00990fbb7d26b03674833666f32b3bcae
This is long overdue now that buster is out and python3.9 has been
around for a bit. This update allows us to remove the buster 3.7 images
that the gear images were based on.
Depends-On: https://review.opendev.org/c/opendev/gear/+/838562
Change-Id: I8d57c06fe28276d6e15934a785b1f97125f0958f
Modernize our package metadata in the following ways:
* switch from description-file to long_description with the file
attribute, and specify an explicit content type and encoding
* replace the home-page parameter with the newer general url one
* use the specific license metadata in addition to the corresponding
trove classifier for it
* make sure wheels when built also incorporate the LICENSE and
AUTHORS files so that we're not distributing them without a copy
of the license text
* indicate support for all recent Python releases in trove
classifiers
* drop Python 3.4 cruft from the bindep list
https://setuptools.readthedocs.io/en/latest/userguide/declarative_config.html
Also replace the contributor documentation with a more up to date
copy from opendev/bindep, and adjust the copyright assertions in the
built Sphinx docs to refer to "OpenDev Contributors" and drop the
unnecessary year.
Change-Id: I39c5f5afc66edec0cf51709218f143b2a749eddd
Gear (indirectly) relies on cffi, which sometimes isn't built for
the platforms on which we would like to install it. In those cases,
the Python installation has to occur from source, and needs the
headers for libffi to link against when compiling its extensions.
Change-Id: Ifc876d93f95941236b78a88d5741467a10142d54
In preparation for an upcoming release, add testing for latest
Python (3.9). Switch tox to use Python 3 by default, and rename the
testenv for flake8 from pep8 to linters, consistent with other tools
and libraries OpenDev maintains. Update to a newer hacking plugin,
which will use newer flake8 as well. Ignore rules about line breaks
around comparison operators, as well as those related to ambiguous
variable names, at least for now. Also build distribution artifacts
on a more recent platform so we get newer Setuptools with support
for the latest package metadata, in preparation for a coming change
to update that.
Change-Id: I2130d66fc9aadaa9fe09635b59475be71938132e
Zuul test: tests.unit.test_scheduler.TestSchedulerSSL.test_jobs_executed
fails on ubuntu focal with the following exception:
Traceback (most recent call last):
File "/home/gchauvel/zuul/zuul/.tox/py38/lib/python3.8/site-packages/gear/__init__.py", line 2835, in _doConnectLoop
self.connectLoop()
File "/home/gchauvel/zuul/zuul/.tox/py38/lib/python3.8/site-packages/gear/__init__.py", line 2865, in connectLoop
c = context.wrap_socket(c, server_side=True)
File "/usr/lib/python3.8/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "/usr/lib/python3.8/ssl.py", line 1040, in _create
self.do_handshake()
File "/usr/lib/python3.8/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL] internal error (_ssl.c:1108)
This is due to libssl1.1 being compiled with
"-DOPENSSL_TLS_SECURITY_LEVEL=2" and gear forcing TLSv1.0
Extracted from ubuntu source package:
The default security level for TLS connections was increased from
level 1 to level 2. This moves from the 80 bit security level to the
112 bit security level and will require 2048 bit or larger RSA and
DHE keys, 224 bit or larger ECC keys, SHA-2, TLSv1.2 or DTLSv1.2.
Allowing to negotiate TLS to the highest available version between
server and client solves the issue, provided that TLSv1.2 is useable.
The option is supported by in the latest version of all pythons >=3.5
[1][2][3]. Unfortunately Xenial doesn't have latest 3.5 and lacks the
ssl.PROTOCOL_TLS definition. We provide a fallback to select the highest
version of TLS supported in that case.
There is some risk using the fallback beacuse both the client and server
need to agree on the version supported in this case. Xenial python 3.5
does support TLSv1_2 which means that for all practical purposes TLS
v1.2 should be available on all platforms that gear runs avoiding this
problem.
Disable TLSv1.3:
According to https://bugs.python.org/issue43622#msg389497, an event on
ssl socket can happen without data being available at application level.
As gear is using a polling loop with multiple file descriptors and ssl
socket used as a blocking one, a blocked state could happen.
This is highlighted by Zuul SSL test: TestSchedulerSSL, where such
blocked state appears consistently.
note: gear tests and zuul tests are ok when using TLSv1.2 but the
previous behavior could also happen
[1] https://docs.python.org/2.7/library/ssl.html?highlight=protocol_tls#ssl.PROTOCOL_TLS
[2] https://docs.python.org/3.5/library/ssl.html?highlight=protocol_tls#ssl.PROTOCOL_TLS
[3] https://docs.python.org/3/library/ssl.html?highlight=protocol_tls#ssl.PROTOCOL_TLS
Change-Id: I5efb6c0576987815c5b93f8bc4020cdee2898d04
gear/__init__.py:
Modified Server.wakeConnections() so that it randomizes the
list of active connections before sending out NOOP's to them.
This will hopefully spread workload across machines more evenly
when there are multiple workers per machine.
Reference: https://phabricator.wikimedia.org/T258630
Change-Id: I05dcb9fa383f3aefc8b5b1bb9dd8b3ff6ff7f37d
Depends-on: https://review.opendev.org/742165
On Fedora rawhide the gear package no longer build.
https://koschei.fedoraproject.org/package/python-gear?
This patch ensures that the ssl engine does not complains about:
- ssl.SSLError: [SSL: EE_KEY_TOO_SMALL] ee key too small (_ssl.c:2951)
- ssl.SSLError: [SSL: CA_MD_TOO_WEAK] ca md too weak (_ssl.c:2951)
To reproduce the issue:
podman run -it --root fedora:rawhide
dnf install git libffi-devel python-devel tox gcc
git clone https://opendev.org/opendev/gear.git && cd gear
tox -epy39
tox -epy38
Change-Id: I57cd9c4750f27b7b76e92a0eef03e7de70c13dd5
This floods the zuul test logs with poll messages which indicates that
something changed to a busy loop.
This reverts commit 103ad3e8ed.
Change-Id: Id3347136507e7e65ccde937f1c2fd303aa3dfbbe
Gear currently supports keepalive only on linux platforms. On mac the
socket must be configured differently. For now just ignore the
keepalive flag in this case and emit a warning.
Change-Id: I276967b720742fa64e5bc6eb769c75590141275c
Switch between Epoll and Poll depending on the OS capabilities.
Change-Id: Iaf1324d0c9d43c76e3f228f1a176e453a82a71a4
Co-Authored-By: Jan Kubovy <jan.kubovy@bmw.de>
Co-Authored-By: Tobias Henkel <tobias.henkel@bmw.de>
We're moving this from the openstack to the opendev tenant. We
should move the jobs in-repo at the same time.
Depends-On: https://review.opendev.org/688451
Change-Id: I23b4a0d78d515557bfb676f206dcc0a91ecb8502
This patch does a few things to fix the documentation builds for
this project
- Move requirements to doc/requirements.txt for building docs to
avoid installing extra dependencies.
- Bump sphinx version to a newer release which is compatible with
sphinxcontrib-programoutput
- Remove default theme option to use the latest Sphinx theme that
is shipped directly from upstream.
- Bumped basepython for documentation jobs to Python 3.
These are all squashed because the job is currently broken.
Change-Id: Ib998923a5daaa5e9d3ddc748b76b6304e5c39b22
According to the python docs [1] it is recommended to use
SSLContext.wrap_socket to create an ssl connection since Python 3.2
and 2.7.9. This enables us to also leverage server name indication
(SNI).
One use case where SNI is beneficial is an easy and standard way to
route traffic into an Openshift cluster. The most common way to get
traffic into an Openshift cluster is using a routes. The routes in an
openshift cluster work with either HTTP, HTTPS with SNI or TLS with
SNI [2]. TLS with SNI in this case also works with non-http
connections like gearman is using.
[1] https://docs.python.org/3/library/ssl.html#socket-creation
[2] https://docs.okd.io/3.11/dev_guide/expose_service/expose_internal_ip_router.html#overview
Change-Id: I19c1edc4a14a303d2a91894e0065c8d31f89ce24
If the listen address allows for ipv4 or ipv6 values we want to prefer
ipv6 if the host is configured with working ivp6. We add the ai_flag
AF_ADDRCONFIG to filter out ipv6 (and ipv4) if the host isn't configure
for this AF_INET version. Then we sort based on the family to prefer
ipv6 over ipv4.
The reason for this is clients will prefer ipv6 before falling back to
ipv4 when attempting to connect to a hostname. If the server isn't
listening on ipv6 this makes new connections happen slowly.
Change-Id: I9f7a235b04068856c6cceeb2c230f3b56945572e
A gearman client only waiting for jobs will wait indefinitely if the
gearman server vanishes (e.g. due to a VM crash). In this case there
is no traffic on the connection and the client blocks forever if there
is nothing in between that forcefully terminates the connection.
Adding tcp keepalive can mitigate that and the connection will be
terminated by the kernel in this situation which then triggers a
reconnect.
Change-Id: I8589cd45450245a25539c051355b38d16ee9f4b9
Using Python3, Gearman unexpectedly closed the socket with a client when typing the 'workers' command.
gearman-debug.log:
File "/usr/local/lib/python3.5/dist-packages/gear/__init__.py", line 3250, in handleWorkers
(fd, ip, client_id.decode('utf8'),
AttributeError: 'str' object has no attribute 'decode'
Change-Id: I610bd44c76a0e52f8d4e8f24c82c636d4ebef0ae
Add the ability for an operator to control which interface to listen
on. By default we use None to maintain backwards compatibility.
Change-Id: I14c13ff500317d5a7b580e1b2a7f798a8db5de1d
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
It is possible to use python3 for gear, so support both python2 and
python3 wheels.
Also update classifier to indicate which versions of python we
support.
Change-Id: I74384871cabc8d5b22f2d7555201c21f1bf37099
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
After registering a function, if a connection is sleeping (ie,
waiting for a NOOP to wake it up), cause it to send another
grab_job in case the newly registered job is something it can
handle.
Change-Id: Ibea13726f4a451ebc67850b17e168bd4accfbc0b
When connection to gearman server is timeout, it's necessary to
release lock before raising a exception, otherwise the client may
be dead locked.
Change-Id: Idc9c9c4bd439b122dc7855ca05d962c4e6687829
Signed-off-by: shangxdy <shang.xiaodong@zte.com.cn>
Keep track of the three main queue stats we report, so that we don't
have to iterate over all the queued jobs to determine whether they
are running.
Also, drop the workers stat because it's not very useful and not
entirely trivial to calculate.
Change-Id: Id42a05e5626096d1100a9cb9e8166c8ec5103b41
The sendRaw method (and therefore sendPacket) was originally
thread safe by virtue of consisting of a single socket.send()
call, but when we added SSL, we added a loop within the method
to handle the increased likelihood that not all data would be
sent in one call. Of course, the method should have been written
this way to start with.
However, this means that we can end up with partial packets being
sent before a context switch to another thread which may also
want to send a packet. To handle that case, place the entire
method in a lock.
Note, this doesn't affect server connections as they use a
non-blocking connection which has a send queue, so only one thread
ever actuall transmits.
Change-Id: I3bda6fda5f762d18f28b56a43b7dc28f37dbc427