Commit Graph

220 Commits

Author SHA1 Message Date
Clark Boylan 97d2061226 Update gear to python3.11 and Bookworm container
This is motivated by OpenDev's desire to drop the old Bullseye container
images as well as container images for older versions of python. We bump
testing to python3.11 on the high end of the range and convert the
container image to python3.11 + Bookworm.

Python 2.7 testing is removed because tox + virtualenv can't actually
create python2.7 virtualenvs any longer. There are workarounds but
dropping the tests is simpler.

Python 3.5 testing is replaced with 3.6 testing beacuse the version of
easy_install on ubuntu xenial does not understand TLS + SNI. A while
back pypi.org dropped non SNI connection support which means we can't
install PBR for to run the setup.py for the projcet. There are
workarounds but 3.5 is old enough taht we should be able to move on.

Change-Id: I2f2a2d0cf71c69d7babd6df9bfdb41a759e9c0ee
2023-09-20 15:12:18 -07:00
Zuul 76324ab5fa Merge "Update the docker image to python3.9 and buster" 2022-04-19 18:51:57 +00:00
Clark Boylan 9f2d08328e Fix python3.9 testing
Latest setuptools has deprecated the use of setup.py commands like
`setup.py testr`. For some reason python3.9 hangs and fails to work at
all when you run `setup.py testr`. Switch to running testr directly and
not bother debugging this too aggressively as this functionality is
going away eventually.

Change-Id: I3ad9e0c00990fbb7d26b03674833666f32b3bcae
2022-04-19 10:47:50 -07:00
Clark Boylan 2c8e6764cc Update the docker image to python3.9 and buster
This is long overdue now that buster is out and python3.9 has been
around for a bit. This update allows us to remove the buster 3.7 images
that the gear images were based on.

Depends-On: https://review.opendev.org/c/opendev/gear/+/838562
Change-Id: I8d57c06fe28276d6e15934a785b1f97125f0958f
2022-04-19 17:28:49 +00:00
Jeremy Stanley aa21a0c61b Overhaul package metadata and contributor info
Modernize our package metadata in the following ways:

* switch from description-file to long_description with the file
  attribute, and specify an explicit content type and encoding

* replace the home-page parameter with the newer general url one

* use the specific license metadata in addition to the corresponding
  trove classifier for it

* make sure wheels when built also incorporate the LICENSE and
  AUTHORS files so that we're not distributing them without a copy
  of the license text

* indicate support for all recent Python releases in trove
  classifiers

* drop Python 3.4 cruft from the bindep list

https://setuptools.readthedocs.io/en/latest/userguide/declarative_config.html

Also replace the contributor documentation with a more up to date
copy from opendev/bindep, and adjust the copyright assertions in the
built Sphinx docs to refer to "OpenDev Contributors" and drop the
unnecessary year.

Change-Id: I39c5f5afc66edec0cf51709218f143b2a749eddd
2021-07-10 16:45:05 +00:00
Jeremy Stanley 8e93a6b7b8 Add libffi header dependency
Gear (indirectly) relies on cffi, which sometimes isn't built for
the platforms on which we would like to install it. In those cases,
the Python installation has to occur from source, and needs the
headers for libffi to link against when compiling its extensions.

Change-Id: Ifc876d93f95941236b78a88d5741467a10142d54
2021-07-10 16:22:14 +00:00
Jeremy Stanley f995fba9a6 Update testing to Python 3.9 and linters
In preparation for an upcoming release, add testing for latest
Python (3.9). Switch tox to use Python 3 by default, and rename the
testenv for flake8 from pep8 to linters, consistent with other tools
and libraries OpenDev maintains. Update to a newer hacking plugin,
which will use newer flake8 as well. Ignore rules about line breaks
around comparison operators, as well as those related to ambiguous
variable names, at least for now. Also build distribution artifacts
on a more recent platform so we get newer Setuptools with support
for the latest package metadata, in preparation for a coming change
to update that.

Change-Id: I2130d66fc9aadaa9fe09635b59475be71938132e
2021-03-30 14:56:40 +02:00
Guillaume Chauvel 66ba8442dc Create SSL context using PROTOCOL_TLS, fallback to highest supported version
Zuul test: tests.unit.test_scheduler.TestSchedulerSSL.test_jobs_executed
fails on ubuntu focal with the following exception:

Traceback (most recent call last):
  File "/home/gchauvel/zuul/zuul/.tox/py38/lib/python3.8/site-packages/gear/__init__.py", line 2835, in _doConnectLoop
    self.connectLoop()
  File "/home/gchauvel/zuul/zuul/.tox/py38/lib/python3.8/site-packages/gear/__init__.py", line 2865, in connectLoop
    c = context.wrap_socket(c, server_side=True)
  File "/usr/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL] internal error (_ssl.c:1108)

This is due to libssl1.1 being compiled with
"-DOPENSSL_TLS_SECURITY_LEVEL=2" and gear forcing TLSv1.0

Extracted from ubuntu source package:

  The default security level for TLS connections was increased from
  level 1 to level 2. This moves from the 80 bit security level to the
  112 bit security level and will require 2048 bit or larger RSA and
  DHE keys, 224 bit or larger ECC keys, SHA-2, TLSv1.2 or DTLSv1.2.

Allowing to negotiate TLS to the highest available version between
server and client solves the issue, provided that TLSv1.2 is useable.
The option is supported by in the latest version of all pythons >=3.5
[1][2][3]. Unfortunately Xenial doesn't have latest 3.5 and lacks the
ssl.PROTOCOL_TLS definition. We provide a fallback to select the highest
version of TLS supported in that case.

There is some risk using the fallback beacuse both the client and server
need to agree on the version supported in this case. Xenial python 3.5
does support TLSv1_2 which means that for all practical purposes TLS
v1.2 should be available on all platforms that gear runs avoiding this
problem.

Disable TLSv1.3:
According to https://bugs.python.org/issue43622#msg389497, an event on
ssl socket can happen without data being available at application level.
As gear is using a polling loop with multiple file descriptors and ssl
socket used as a blocking one, a blocked state could happen.
This is highlighted by Zuul SSL test: TestSchedulerSSL, where such
blocked state appears consistently.
note: gear tests and zuul tests are ok when using TLSv1.2 but the
previous behavior could also happen

[1] https://docs.python.org/2.7/library/ssl.html?highlight=protocol_tls#ssl.PROTOCOL_TLS
[2] https://docs.python.org/3.5/library/ssl.html?highlight=protocol_tls#ssl.PROTOCOL_TLS
[3] https://docs.python.org/3/library/ssl.html?highlight=protocol_tls#ssl.PROTOCOL_TLS

Change-Id: I5efb6c0576987815c5b93f8bc4020cdee2898d04
2021-03-30 14:56:38 +02:00
Zuul 29e9d1fa99 Merge "wakeConnections: Randomize connections before scanning them" 2021-03-10 19:41:24 +00:00
Zuul edcc55b1e9 Merge "Move handleDisconnect into BaseClientServer" 2021-03-10 19:41:21 +00:00
Zuul 1c4da386d2 Merge "remove unicode from code" 2021-03-10 19:26:20 +00:00
Zuul 3f6fa2da29 Merge "Modify connection timeout process" 2021-03-10 19:05:27 +00:00
Ahmon Dancy a160d10735 wakeConnections: Randomize connections before scanning them
gear/__init__.py:
 Modified Server.wakeConnections() so that it randomizes the
 list of active connections before sending out NOOP's to them.
 This will hopefully spread workload across machines more evenly
 when there are multiple workers per machine.

Reference: https://phabricator.wikimedia.org/T258630

Change-Id: I05dcb9fa383f3aefc8b5b1bb9dd8b3ff6ff7f37d
2021-03-10 18:15:07 +01:00
Mohammed Naser 089471c8fb Added Docker image builds
Change-Id: If0b5c982020faf0f512a1c0d9b7b495789b9aa4f
2021-02-11 19:11:48 +00:00
liyou01 370be17fc5 remove unicode from code
Change-Id: If93af8c01cbbaad4e64dc78fb651815a4de6e741
2021-01-07 15:03:18 +08:00
Zuul e4f2cf4a5f Merge "Bump crypto requirement to accomodate security standards" 2020-12-10 20:52:42 +00:00
Fabien Boucher 7d552ad346 use python3 as context for build-python-release
Change-Id: Iebda27b20aaa94539766feeeb0236675c2ebdac6
2020-09-09 09:57:38 +02:00
Fabien Boucher 79e1c3009e Bump crypto requirement to accomodate security standards
Depends-on: https://review.opendev.org/742165

On Fedora rawhide the gear package no longer build.
https://koschei.fedoraproject.org/package/python-gear?

This patch ensures that the ssl engine does not complains about:
- ssl.SSLError: [SSL: EE_KEY_TOO_SMALL] ee key too small (_ssl.c:2951)
- ssl.SSLError: [SSL: CA_MD_TOO_WEAK] ca md too weak (_ssl.c:2951)

To reproduce the issue:
podman run -it --root fedora:rawhide
dnf install git libffi-devel python-devel tox gcc
git clone https://opendev.org/opendev/gear.git && cd gear
tox -epy39
tox -epy38

Change-Id: I57cd9c4750f27b7b76e92a0eef03e7de70c13dd5
2020-07-21 12:16:31 +00:00
Tobias Henkel 9261c8f4a7
Move handleDisconnect into BaseClientServer
It's called from there using self.handleDisconnect so define it there
as well.

Change-Id: I90fef55a79168e082e69f81a717c08badd4163a9
2020-03-24 17:43:46 +01:00
Tobias Henkel 9933f06821
Revert "Add BSD/Darwin support."
This floods the zuul test logs with poll messages which indicates that
something changed to a busy loop.

This reverts commit 103ad3e8ed.

Change-Id: Id3347136507e7e65ccde937f1c2fd303aa3dfbbe
2020-02-17 22:36:01 +01:00
Zuul 4fbb6d1a57 Merge "Ignore keepalive on unsupported platforms" 2020-02-03 17:33:59 +00:00
Zuul 4243fc031b Merge "Add BSD/Darwin support." 2020-02-03 17:29:55 +00:00
Sorin Sbarnea 3125fa8755 packaging: updated project urls
Make package listing on PYPA more informative for the users.

Change-Id: Ifd20ae3ee006c8018d04782f3328799769ce4f30
2020-01-21 19:50:18 +00:00
Tobias Henkel c9f23749c6
Ignore keepalive on unsupported platforms
Gear currently supports keepalive only on linux platforms. On mac the
socket must be configured differently. For now just ignore the
keepalive flag in this case and emit a warning.

Change-Id: I276967b720742fa64e5bc6eb769c75590141275c
2020-01-16 09:11:51 +01:00
Tobias Henkel 103ad3e8ed Add BSD/Darwin support.
Switch between Epoll and Poll depending on the OS capabilities.

Change-Id: Iaf1324d0c9d43c76e3f228f1a176e453a82a71a4
Co-Authored-By: Jan Kubovy <jan.kubovy@bmw.de>
Co-Authored-By: Tobias Henkel <tobias.henkel@bmw.de>
2019-10-15 10:45:22 +02:00
Zuul 5eae2bf566 Merge "Add in-repo zuul config" 2019-10-14 22:27:32 +00:00
James E. Blair 1ed76ef85e Add in-repo zuul config
We're moving this from the openstack to the opendev tenant.  We
should move the jobs in-repo at the same time.

Depends-On: https://review.opendev.org/688451
Change-Id: I23b4a0d78d515557bfb676f206dcc0a91ecb8502
2019-10-14 14:55:13 -07:00
Mohammed Naser a4449758bf Fix documentation builds
This patch does a few things to fix the documentation builds for
this project

- Move requirements to doc/requirements.txt for building docs to
  avoid installing extra dependencies.
- Bump sphinx version to a newer release which is compatible with
  sphinxcontrib-programoutput
- Remove default theme option to use the latest Sphinx theme that
  is shipped directly from upstream.
- Bumped basepython for documentation jobs to Python 3.

These are all squashed because the job is currently broken.

Change-Id: Ib998923a5daaa5e9d3ddc748b76b6304e5c39b22
2019-10-14 12:47:52 -04:00
Zuul 483ab492a8 Merge "Add support for server name indication" 2019-06-17 16:17:30 +00:00
Zuul 8d3ea4bcff Merge "add missing str to bytes conversion for Python3" 2019-05-06 15:58:05 +00:00
Tobias Henkel 58b2f277b7
Add support for server name indication
According to the python docs [1] it is recommended to use
SSLContext.wrap_socket to create an ssl connection since Python 3.2
and 2.7.9. This enables us to also leverage server name indication
(SNI).

One use case where SNI is beneficial is an easy and standard way to
route traffic into an Openshift cluster. The most common way to get
traffic into an Openshift cluster is using a routes. The routes in an
openshift cluster work with either HTTP, HTTPS with SNI or TLS with
SNI [2]. TLS with SNI in this case also works with non-http
connections like gearman is using.

[1] https://docs.python.org/3/library/ssl.html#socket-creation
[2] https://docs.okd.io/3.11/dev_guide/expose_service/expose_internal_ip_router.html#overview

Change-Id: I19c1edc4a14a303d2a91894e0065c8d31f89ce24
2019-05-04 12:11:41 +02:00
OpenDev Sysadmins 88b2c09878 OpenDev Migration Patch
This commit was bulk generated and pushed by the OpenDev sysadmins
as a part of the Git hosting and code review systems migration
detailed in these mailing list posts:

http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.html
http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004920.html

Attempts have been made to correct repository namespaces and
hostnames based on simple pattern matching, but it's possible some
were updated incorrectly or missed entirely. Please reach out to us
via the contact information listed at https://opendev.org/ with any
questions you may have.
2019-04-19 19:26:05 +00:00
Zuul 6a7417702d Merge "Prefer ipv6 for listen addrs if available" 2019-01-28 16:21:02 +00:00
Zuul 7a6ef8ac60 Merge "Add support for keepalive to client" 2018-12-11 20:11:16 +00:00
qingszhao 61f6f97c4b Change openstack-dev to openstack-discuss
Mailinglists have been updated. Openstack-discuss replaces openstack-dev.

Change-Id: Ia1f74031b69da2d50f0404fac4698823526ecd83
2018-12-04 13:51:00 +00:00
Clark Boylan ca733f3e07 Prefer ipv6 for listen addrs if available
If the listen address allows for ipv4 or ipv6 values we want to prefer
ipv6 if the host is configured with working ivp6. We add the ai_flag
AF_ADDRCONFIG to filter out ipv6 (and ipv4) if the host isn't configure
for this AF_INET version. Then we sort based on the family to prefer
ipv6 over ipv4.

The reason for this is clients will prefer ipv6 before falling back to
ipv4 when attempting to connect to a hostname. If the server isn't
listening on ipv6 this makes new connections happen slowly.

Change-Id: I9f7a235b04068856c6cceeb2c230f3b56945572e
2018-10-19 14:33:37 -07:00
Tobias Henkel 71dbac070c
Add support for keepalive to client
A gearman client only waiting for jobs will wait indefinitely if the
gearman server vanishes (e.g. due to a VM crash). In this case there
is no traffic on the connection and the client blocks forever if there
is nothing in between that forcefully terminates the connection.

Adding tcp keepalive can mitigate that and the connection will be
terminated by the kernel in this situation which then triggers a
reconnect.

Change-Id: I8589cd45450245a25539c051355b38d16ee9f4b9
2018-09-04 13:50:04 +02:00
Benoit Bayszczak f75a3c8509 add missing str to bytes conversion for Python3
Using Python3, Gearman unexpectedly closed the socket with a client when typing the 'workers' command.

gearman-debug.log:
    File "/usr/local/lib/python3.5/dist-packages/gear/__init__.py", line 3250, in handleWorkers
      (fd, ip, client_id.decode('utf8'),
  AttributeError: 'str' object has no attribute 'decode'

Change-Id: I610bd44c76a0e52f8d4e8f24c82c636d4ebef0ae
2018-08-01 11:06:00 +02:00
Paul Belanger c00ca944db
Add --listen-address flag to geard
Add the ability for an operator to control which interface to listen
on. By default we use None to maintain backwards compatibility.

Change-Id: I14c13ff500317d5a7b580e1b2a7f798a8db5de1d
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-04-16 16:27:49 -04:00
Paul Belanger 5ad274a17e
Build universal wheels
It is possible to use python3 for gear, so support both python2 and
python3 wheels.

Also update classifier to indicate which versions of python we
support.

Change-Id: I74384871cabc8d5b22f2d7555201c21f1bf37099
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
2018-04-13 19:37:27 -04:00
James E. Blair eee504755d Automatically send GRAB_JOB after CAN_DO
After registering a function, if a connection is sleeping (ie,
waiting for a NOOP to wake it up), cause it to send another
grab_job in case the newly registered job is something it can
handle.

Change-Id: Ibea13726f4a451ebc67850b17e168bd4accfbc0b
2018-02-02 11:19:14 -08:00
James E. Blair da802f2828 Make workers admin command py3 safe
Change-Id: I05598ceea20169a625bbb47f15288e6eadbc88d2
2018-02-01 12:02:14 -08:00
shangxdy 8a76de29e1 Modify connection timeout process
When connection to gearman server is timeout, it's necessary to
release lock before raising a exception, otherwise the client may
be dead locked.

Change-Id: Idc9c9c4bd439b122dc7855ca05d962c4e6687829
Signed-off-by: shangxdy <shang.xiaodong@zte.com.cn>
2018-01-26 15:31:37 +08:00
James E. Blair 49d4bef593 server: make stats more efficient
Keep track of the three main queue stats we report, so that we don't
have to iterate over all the queued jobs to determine whether they
are running.

Also, drop the workers stat because it's not very useful and not
entirely trivial to calculate.

Change-Id: Id42a05e5626096d1100a9cb9e8166c8ec5103b41
2017-10-30 11:46:08 -07:00
Jenkins c4a5decdc6 Merge "Add a send lock to the base Connection class" 2017-10-09 15:40:40 +00:00
James E. Blair b24b4d2f6e Add a send lock to the base Connection class
The sendRaw method (and therefore sendPacket) was originally
thread safe by virtue of consisting of a single socket.send()
call, but when we added SSL, we added a loop within the method
to handle the increased likelihood that not all data would be
sent in one call.  Of course, the method should have been written
this way to start with.

However, this means that we can end up with partial packets being
sent before a context switch to another thread which may also
want to send a packet.  To handle that case, place the entire
method in a lock.

Note, this doesn't affect server connections as they use a
non-blocking connection which has a send queue, so only one thread
ever actuall transmits.

Change-Id: I3bda6fda5f762d18f28b56a43b7dc28f37dbc427
2017-09-30 09:18:27 -07:00
James E. Blair ca4666f005 Server: support background jobs
Change-Id: Ic15ab05c16f143f1d557d935aecb6d0afe419d59
2017-09-13 18:21:39 -06:00
James E. Blair bf8d96cb77 Fix exception setter
This method had a typo.

Change-Id: I49b745387626f664235998de52d4c58927149b4d
2017-05-18 17:19:46 -07:00
Jenkins d76e998959 Merge "Replace assertEquals with assertEqual" 2017-05-17 16:59:54 +00:00
Jenkins 50616da084 Merge "Provide TextJob and TextWorker for convenience" 2017-05-17 16:58:31 +00:00