Run script generate-gitmodules.py to sync with governace repository to
add new repositories and remove retired ones.
Change-Id: Ia42b717ad242958ba4a9689ac814c956f460712f
* Update gear from branch 'master'
- Revert "Add BSD/Darwin support."
This floods the zuul test logs with poll messages which indicates that
something changed to a busy loop.
This reverts commit 103ad3e8ed78c6895c425115fda45f28441bbfaf.
Change-Id: Id3347136507e7e65ccde937f1c2fd303aa3dfbbe
* Update gear from branch 'master'
- Merge "Ignore keepalive on unsupported platforms"
- Ignore keepalive on unsupported platforms
Gear currently supports keepalive only on linux platforms. On mac the
socket must be configured differently. For now just ignore the
keepalive flag in this case and emit a warning.
Change-Id: I276967b720742fa64e5bc6eb769c75590141275c
* Update gear from branch 'master'
- Merge "Add BSD/Darwin support."
- Add BSD/Darwin support.
Switch between Epoll and Poll depending on the OS capabilities.
Change-Id: Iaf1324d0c9d43c76e3f228f1a176e453a82a71a4
Co-Authored-By: Jan Kubovy <jan.kubovy@bmw.de>
Co-Authored-By: Tobias Henkel <tobias.henkel@bmw.de>
* Update gear from branch 'master'
- packaging: updated project urls
Make package listing on PYPA more informative for the users.
Change-Id: Ifd20ae3ee006c8018d04782f3328799769ce4f30
* Update gear from branch 'master'
- Merge "Add in-repo zuul config"
- Add in-repo zuul config
We're moving this from the openstack to the opendev tenant. We
should move the jobs in-repo at the same time.
Depends-On: https://review.opendev.org/688451
Change-Id: I23b4a0d78d515557bfb676f206dcc0a91ecb8502
* Update gear from branch 'master'
- Fix documentation builds
This patch does a few things to fix the documentation builds for
this project
- Move requirements to doc/requirements.txt for building docs to
avoid installing extra dependencies.
- Bump sphinx version to a newer release which is compatible with
sphinxcontrib-programoutput
- Remove default theme option to use the latest Sphinx theme that
is shipped directly from upstream.
- Bumped basepython for documentation jobs to Python 3.
These are all squashed because the job is currently broken.
Change-Id: Ib998923a5daaa5e9d3ddc748b76b6304e5c39b22
* Update gear from branch 'master'
- Merge "Add support for server name indication"
- Add support for server name indication
According to the python docs [1] it is recommended to use
SSLContext.wrap_socket to create an ssl connection since Python 3.2
and 2.7.9. This enables us to also leverage server name indication
(SNI).
One use case where SNI is beneficial is an easy and standard way to
route traffic into an Openshift cluster. The most common way to get
traffic into an Openshift cluster is using a routes. The routes in an
openshift cluster work with either HTTP, HTTPS with SNI or TLS with
SNI [2]. TLS with SNI in this case also works with non-http
connections like gearman is using.
[1] https://docs.python.org/3/library/ssl.html#socket-creation
[2] https://docs.okd.io/3.11/dev_guide/expose_service/expose_internal_ip_router.html#overview
Change-Id: I19c1edc4a14a303d2a91894e0065c8d31f89ce24
* Update gear from branch 'master'
- Merge "add missing str to bytes conversion for Python3"
- OpenDev Migration Patch
This commit was bulk generated and pushed by the OpenDev sysadmins
as a part of the Git hosting and code review systems migration
detailed in these mailing list posts:
http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003603.htmlhttp://lists.openstack.org/pipermail/openstack-discuss/2019-April/004920.html
Attempts have been made to correct repository namespaces and
hostnames based on simple pattern matching, but it's possible some
were updated incorrectly or missed entirely. Please reach out to us
via the contact information listed at https://opendev.org/ with any
questions you may have.
- add missing str to bytes conversion for Python3
Using Python3, Gearman unexpectedly closed the socket with a client when typing the 'workers' command.
gearman-debug.log:
File "/usr/local/lib/python3.5/dist-packages/gear/__init__.py", line 3250, in handleWorkers
(fd, ip, client_id.decode('utf8'),
AttributeError: 'str' object has no attribute 'decode'
Change-Id: I610bd44c76a0e52f8d4e8f24c82c636d4ebef0ae
* Update gear from branch 'master'
- Merge "Prefer ipv6 for listen addrs if available"
- Merge "Add support for keepalive to client"
- Change openstack-dev to openstack-discuss
Mailinglists have been updated. Openstack-discuss replaces openstack-dev.
Change-Id: Ia1f74031b69da2d50f0404fac4698823526ecd83
- Prefer ipv6 for listen addrs if available
If the listen address allows for ipv4 or ipv6 values we want to prefer
ipv6 if the host is configured with working ivp6. We add the ai_flag
AF_ADDRCONFIG to filter out ipv6 (and ipv4) if the host isn't configure
for this AF_INET version. Then we sort based on the family to prefer
ipv6 over ipv4.
The reason for this is clients will prefer ipv6 before falling back to
ipv4 when attempting to connect to a hostname. If the server isn't
listening on ipv6 this makes new connections happen slowly.
Change-Id: I9f7a235b04068856c6cceeb2c230f3b56945572e
- Add support for keepalive to client
A gearman client only waiting for jobs will wait indefinitely if the
gearman server vanishes (e.g. due to a VM crash). In this case there
is no traffic on the connection and the client blocks forever if there
is nothing in between that forcefully terminates the connection.
Adding tcp keepalive can mitigate that and the connection will be
terminated by the kernel in this situation which then triggers a
reconnect.
Change-Id: I8589cd45450245a25539c051355b38d16ee9f4b9
* Update gear from branch 'master'
- Add --listen-address flag to geard
Add the ability for an operator to control which interface to listen
on. By default we use None to maintain backwards compatibility.
Change-Id: I14c13ff500317d5a7b580e1b2a7f798a8db5de1d
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
* Update gear from branch 'master'
- Build universal wheels
It is possible to use python3 for gear, so support both python2 and
python3 wheels.
Also update classifier to indicate which versions of python we
support.
Change-Id: I74384871cabc8d5b22f2d7555201c21f1bf37099
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
* Update gear from branch 'master'
- Automatically send GRAB_JOB after CAN_DO
After registering a function, if a connection is sleeping (ie,
waiting for a NOOP to wake it up), cause it to send another
grab_job in case the newly registered job is something it can
handle.
Change-Id: Ibea13726f4a451ebc67850b17e168bd4accfbc0b
* Update gear from branch 'master'
- server: make stats more efficient
Keep track of the three main queue stats we report, so that we don't
have to iterate over all the queued jobs to determine whether they
are running.
Also, drop the workers stat because it's not very useful and not
entirely trivial to calculate.
Change-Id: Id42a05e5626096d1100a9cb9e8166c8ec5103b41
* Update gear from branch 'master'
- Merge "Add a send lock to the base Connection class"
- Add a send lock to the base Connection class
The sendRaw method (and therefore sendPacket) was originally
thread safe by virtue of consisting of a single socket.send()
call, but when we added SSL, we added a loop within the method
to handle the increased likelihood that not all data would be
sent in one call. Of course, the method should have been written
this way to start with.
However, this means that we can end up with partial packets being
sent before a context switch to another thread which may also
want to send a packet. To handle that case, place the entire
method in a lock.
Note, this doesn't affect server connections as they use a
non-blocking connection which has a send queue, so only one thread
ever actuall transmits.
Change-Id: I3bda6fda5f762d18f28b56a43b7dc28f37dbc427
Project: openstack-infra/gear ca4666f005735078727c5158c518031c9f09c72b
Server: support background jobs
Change-Id: Ic15ab05c16f143f1d557d935aecb6d0afe419d59
Project: openstack-infra/gear bf8d96cb7732bdd265d0360652c2df269d356ea9
Fix exception setter
This method had a typo.
Change-Id: I49b745387626f664235998de52d4c58927149b4d
Project: openstack-infra/gear d76e99895953bfe3a7775441bd5c18f08ff421d1
Replace assertEquals with assertEqual
The method assertEquals has been deprecated since python 2.7.
http://docs.python.org/2/library/unittest.html#deprecated-aliases
Also in Python 3, a deprecated warning is raised when using assertEquals
therefore we should use assertEqual instead.
Change-Id: Ic99ea3f57bb6fd986ce2626e800e3d310a2e3df7
Closes-Bug: #1218185
Project: openstack-infra/gear 50616da084f9142e4746891e94be76ca6d1d14f9
Provide TextJob and TextWorker for convenience
These classes will provide a smoother path from python2 -> python3 when
using gear, as it will relieve the need for encoding/decoding.
Change-Id: I93bfe33f898294f30a82c0a24a18a081f9752354
Project: openstack-infra/gear 77df412aea40eb7be10efe3da14da13c105afc8d
Make Job.name assume utf-8.
This breaks the API in subtle ways, but only really for people who want
to use Python 3 and/or not utf-8 function names.
Change-Id: If6bfc35d916cfb84d630af59f4fde4ccae5187d4
Project: openstack-infra/gear f79777289a789939dd959aca0e5042cc09c069d9
Uncap pbr dependency
With the release of pbr 2.0.0, uncap so we can use the latest version.
Change-Id: I32536cc8fb2dde06971206fefee9742400b19387
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Project: openstack-infra/gear ae0769d02f98529d1f31b76baeeb87bdf309bd59
Update hacking dependency
Sync with openstack global requirements for hacking. This allows us to
properly use pbr 2.0.0.
Change-Id: Icde44550ce50065b4f1072b65d6de0e2f7cd45c4
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Project: openstack-infra/gear 640e0785da894be975f11ec8f966e3a30faaa134
Ignore E129
This rule is not backed by the pep8 standard and should not be
enforced (in fact, it is contra-indicated by the standard). It
was previously covered by E125 in an earlier version of the pep8
tool, but has since split out.
Change-Id: Ib9d39c44de29a633e02e36b8ddae87d5ec28d874
Project: openstack-infra/gear a74b8d50b8b8c14c97fc528c903fa59a8f555b58
Re-enable flake8
The line in tox.ini "select=H231" did not add H231 to the list of flake8
checks. It made us only run H231.
Remove the H231 line and then fix the flake8 issues. The most
substantial of these was the shadowing of the queue module with local
variables named queue. Made that one import queue as queue_mod because
changing the variable names in the code was yuckier to look at.
Change-Id: I4a26a20889132f7f4525b17392088dda6bd5bbd2
Project: openstack-infra/gear fca6f5e6364ad5515ddaef81ddce3113450f9ca9
Move py27 after py34 and py35
testr database format gets wonky if you do python2 before python3. Yay.
Change-Id: I8c250e8f71ce95e6224c90b9596d5a176e75626d
Project: openstack-infra/gear 9d2821ddfb12870b77f9b8916c4fe66a79e6f84a
Add py35 to tox.ini default envs
Ubuntu 16.04 switches to this for python3, so it's good to include it in
the default list that is attempted when one runs 'tox'.
Change-Id: Idae2118b3179010df8a9559a059d50b1decca8e5
Project: openstack-infra/gear 55ab77bcb63d9197be8d3f50b66baf3e1b0803bf
Remove discover from test-requirements
It's only needed for python < 2.7 which is not supported
Change-Id: I582fe81f7ad07d7be1c73a4799501d5f5c5cebcc
Project: openstack-infra/gear 5dc4124b8d97ac45d79519d111d3907c06705add
Move other-requirements.txt to bindep.txt
The default filename for documenting binary dependencies has been
changed from "other-requirements.txt" to "bindep.txt" with the release
of bindep 2.1.0. While the previous name is still supported, it will
be deprecated.
Move the file around to follow this change.
Note that this change is self-testing, the OpenStack CI infrastructure
will use a "bindep.txt" file to setup nodes for testing.
For more information about bindep, see also:
http://docs.openstack.org/infra/manual/drivers.html#package-requirementshttp://docs.openstack.org/infra/bindep/
Change-Id: I184b552d34b0f2016a38167bd19287943f51be41
Project: openstack-infra/gear b604bc9fb64e5ae83076add7fb6f61b29211b857
Handle SIGINT correctly in Client.waitForServer()
The Python 2.7 threading.Condition.wait() function will block the
process from responding to any signals until the condition triggers.
This was especially bad for commandline applications. Before this
commit, if an application didn't manage to connect to the Gearman server
it would block forever and ignore CTRL+C.
Adding a 'timeout' works around this issue. Behaviour is unchanged
except that the 'Waiting for at least one active connection' debug
message will now be logged once per second.
This problem is fixed upstream in Python 3.2:
<https://bugs.python.org/issue8844>
Change-Id: Ib1043948b1b37a4a6732176314b8a243aad73397
Project: openstack-infra/gear 1f8889595bb506559bfe6157e4d536e11c8239fb
Add test for AdminRequest.isComplete
Change-Id: I16fd6b6debc3171ec9a346be079c32018e5b5f64
Project: openstack-infra/gear 2defbeaa1beb9ff9e0e3e31529f1f761ac4fcae5
Parse admin requests 2880 times faster
Both returning the incomplete data from the isComplete method, and
scanning the entire incomplete string for the terminator each time
are O(n^2).
Also, returning the string is *very* slow, accounting for 99.6% of
the speedup; scanning the string each time accounts for 0.6%.
Both changes together find the terminus of a 481MiB response in
about 1.6 seconds compared to 76 minutes prior to this change.
Change-Id: I3614e8f3ff8ad7c3d9ca6da1b520b89cd9d5d603
Project: openstack-infra/gear 550873b76abded12cd27c054b0dab0511ffd5e36
Geard: Handle connections closed while sending
If a connection is closed while sending a large amount of data,
the send() call may return EAGAIN, even though the socket is in
CLOSE_WAIT. In that case, just continue queuing the data, and
let the main loop handle the disconnect (which it will detect
via the poll call followed by a null read).
Change-Id: Ib15eae81077b58d2f95fb0989ed1139c6d542f49
Project: openstack-infra/gear defb9722b8188981f15555ba04d5598da722e9c7
Support TCP keepalives in geard
Support configuration of TCP keepalives in gear server.
When connections are closed not gracefully they stays opened
for a long time, which leads to file descriptors leak.
Default values for interval, probe and count are such as in
most Linux distributions.
By default is disabled to keep compatibility with older versions.
Change-Id: I335f68a24dda409b1a48c6e4e520ec08dfd38078
Project: openstack-infra/gear d39384513bc23baf55685a77acb6388616801bf4
List system dependencies for running common tests
Add an other-requirements.txt file containing a cross-platform list of
dependencies needed for running included tox-based tests. Also include a
tox environment for convenience calling the bindep[*] utility to list
any missing system requirements.
This change is self-testing.
For other-requirements.txt see also
http://docs.openstack.org/infra/manual/drivers.html#package-requirements
[*] http://docs.openstack.org/infra/bindep/
Change-Id: I165b531c55a6ea27af742fcb3bd995cd4a3dce15
Project: openstack-infra/gear 59d29104cb9c370e49b44f313e568bd43b172e1b
Do not encode the type (b) in the job name
Don't encode string type in the job name in handleRequest(). Without
the explicit .decode() before passing to the string formatter it
would end up looking like b"b'<jobname'\t\n...." which made for
working with gear on the other side near impossible because a
decode of the job data would not result in a consumable string
matching expected values.
Now there is an explicit decode prior to formatting the string
data.
Change-Id: Ib2996b84bce719a2f91c166aaa4278c18f89f88f
Project: openstack-infra/gear f7a82065b81a11c21d5ca59f0079dcfcb633bc07
Bump version requirement for pbr
Otherwise there's a conflict between "gear" and "turbo-hipster".
Also bump hacking version because otherwise it conflicts with the pbr
version.
Change-Id: Ib9ef16cac16184fe6ee2ec5358b5e16480e31e15
Project: openstack-infra/gear 875c8839481c49752562dcecd4dfdc33d9b9b903
Remove py26 from tox.ini
We don't support python2.6 anymore.
Change-Id: I2755a7890236f48c037ac07078c732a2f53a2192
Project: openstack-infra/gear d0520c2754ac84883ca44ec939837af31239ae47
Switch to six for configparser
I noticed fedora doesn't yet ship python3-gear so this is the main
reason to add six support.
Change-Id: I3d8d37870a0ab865dc0a32e123bfd16e91fa5b72
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Project: openstack-infra/gear f3d1c06c37c23f3e35d1a5e68207b7e3e453307a
Add a test for worker termination
This was an attempt to find a code path that could cause
stopWaitingForJobs to raise an exception. So far, that has failed,
but add this test anyway to exercise some of that code.
Change-Id: I39955fc7250af9b93f336c7c3e0315528bd8d4a9
Project: openstack-infra/gear 40f6feefe46271c75c7e8faaec6280bd34705599
Remove argparse from requirements
argparse was external in python 2.6 but not anymore, remove it from
requirements.
This should help with pip 8.0 that gets confused in this situation.
Installation of the external argparse is not needed.
Change-Id: Ib7e74912b36c1b5ccb514e31fac35efeff57378d
Project: openstack-infra/gear 9e98e23c86b5d03d534a7df3ecc51cb2435d5886
Remove shebang as the script is managed by an entry_point
The shebang is not needed in cmd/geard.py. The file
is not intended to be used directly as it is managed
by an entry_point.
Change-Id: I4e2a8715cca81a8626f6bcd86a368a537305bc0b
Signed-off-by: Paul Belanger <pabelanger@redhat.com>
Project: openstack-infra/gear 2f47efa4758308f526f4db48a74467c416d79b13
Add py34 to tox.ini envlist
A minor oversight, easily corrected.
Change-Id: I1418bc42f56cd96ed11f1fc85a1667530d7dd495
Project: openstack-infra/gear 3fb6b1c5a1519a47221ec60c145b7a5bdfe5a3a3
Do not change object size when iterating
In `gear.Server._lostConnection()` the jobs list pulls .values() from
`conn.related_jobs`, in `gear.server._removeJob()` it is possible to
change the conn.related_jobs dictionary which will cause a
`RuntimeError: dictionary changed size during iteration` exception to
be raised in python 3 since .values() is an iterator instead of a
rendered list() like in python 2. This RuntimeError is eliminated
by forcing the python 2 behavior by wrapping the `.values()` call
to be wrapped into an explicit list.
Change-Id: I730473f3200a427cce774df534f1ff4958866dac
Project: openstack-infra/gear c313be6e181bd4916e3728d9889d865f011b51c7
Guard against double shutdown in client
If we have already performed a shutdown we can prevent exploding when
writing to self.wake_write if we short circuit.
Change-Id: If5352a373d5fa61dd1ee661f4a37976b3447dd9d
Project: openstack-infra/gear 5491e93d95e1dcde6de541f3e8a3452c9a0523d1
Do not raise RetryIOError on blocked write
When EAGAIN is returned on a socket write, do not raise RetryIOError.
This exception is used principally to indicate that we have encountered
a blocked read, and a method farther up the call stack may need to
perform some cleanup. However, within the sendQueuedData method,
all necessary cleanup is performed by the finally handler itself.
Therefore, it is safe to ignore a blocked write, and count on either
a subsequent write (which will append data to the queue and retry
sending the oldest data) or the poll edge trigger to indicate that
we should retry the write.
There is nothing in the call stack above this method that has an
exception handler for RetryIOError. Therefore, in the current state,
the raised exception was causing connections to be disconnected
due to the error.
Change-Id: I9211fb6365f8f3b6dd0310430cf97926ce1f5d07
Project: openstack-infra/gear bb681360fcbcbb9a9b53f7e0fc02a752996a3cf3
Fix regression in wakeConnections
In d5e03ac9307c9cc6f79fc22f15e177d3662dc936, an optional argument
to wakeConnections was added so that only connections related to
a job would be awoken. However, the default value, None, meant
that no connection would be awoken. This doesn't make much sense,
and indeed, Zuul, in its unit tests, was relying on all connections
being awoken. Fix this by awakening all connections if the job
argument is None (which never happens within gear itself).
Change-Id: Ib7632ae0353c414cc1b68dbe430cb57292058012