Commit Graph

1731 Commits

Author SHA1 Message Date
Alexey Pervushin 165b897139 Use for join-time-max and sync-time-max metrics Max() measure function (#1146) 2017-07-20 22:07:50 -07:00
Harald 0c78f70452 added gssapi support (Kerberos) for SASL (#1152) 2017-07-20 10:09:26 -07:00
Taras Voinarovskyi c8237fc53b Add note, that `max_in_flight_requests_per_connection>1` may change order or messages (#1149) 2017-07-17 16:39:37 -07:00
Mika Eloranta 6f0395786a producer: fix produce timeout message (#1151) 2017-07-17 16:39:04 -07:00
Dana Powers d0c6b1f95c Protocol updates for 0.11.0.0 (#1127) 2017-07-09 08:04:39 -07:00
Jeff Widman 2f75169504 Use logging's built-in string interpolation 2017-07-07 13:40:31 -07:00
Petr Šebek 43939b29b5 change_subscription called only when necessary (#1132)
When we are using subscription by pattern change subscription is
called every metadata update even when nothing changes. This PR
ensures that change subscription is called only when set of topics
changes.
2017-07-07 12:29:40 -07:00
Jeff Widman 71ce772484 Fix typo 2017-07-01 02:37:52 -07:00
Hannu Valtonen b8da199b54 producer: Set exit timeout to 0 for atexit handler to match __del__ (#1126)
Hit a problem with pytest hitting the atexit handler and waiting
for close() timeout forever at teardown.

This commit makes atexit close() equivalent to __del__ behavior,
namely using timeout of 0 for close() completion. If you need a
longer timeout you should be setting it explicitly.
2017-06-19 17:09:43 -07:00
Dana Powers b1cc966439 KIP-144: Exponential backoff for broker reconnections (#1124) 2017-06-19 10:16:02 -07:00
Dana Powers cceaf4ae09 Backoff on unavailable group coordinator retry (#1125) 2017-06-19 09:18:56 -07:00
Dana Powers bbbac3dc36 Fixup for #1085 -- only check for changed metadata on disconnected nodes 2017-06-18 23:18:41 -07:00
Mike Fischer 2a41fa1fe4 Deal with brokers that disappear, reappear with different IP address (#1085)
When KafkaClient connects to a broker in _maybe_connect,
it inserts into self._conns a BrokerConnection configured
with the current host/port for that node.  The BrokerConnection
remains there forever, though, so if the broker's IP or host
ever changes, KafkaClient has no way to deal with this.

The fix is to compare the latest metadata with the current
node's connection, and if the host/IP has changed, decommission
the old connection and allow a new one to be created.

There's also a common race condition on broker startup where
the initial metadata request sometimes returns an empty list
of brokers, but subsequent requests behave normally.  So, we
must deal with broker being None here.  This change is conservative
in that it doesn't remove the connection from self._conns unless
the new broker metadata contains an entry for that same node
with a new IP/port.
2017-06-18 23:17:31 -07:00
Dana Powers 26a810220a Do not test fetched_size with fetch_max_bytes since 0.11 brokers will return more data than 0.10 2017-06-18 22:43:21 -07:00
Dana Powers c5f1c6901f Increase max_buffer_size for test_large_messages 2017-06-18 22:43:21 -07:00
Dana Powers b4f71229d0 Fix fetch_max_bytes=1 consumer integration test 2017-06-18 22:43:21 -07:00
Dana Powers d127928e44 Use fixture hostname (dont assume localhost) 2017-06-18 22:43:21 -07:00
Dana Powers 6b316c0971 Add 0.11.0.0 server resources (reduced offsets.topic.replication.factor=1) 2017-06-18 22:43:21 -07:00
Linus Wallgren 97261f491d Describe consumer thread-safety 2017-06-16 23:26:14 -07:00
Andrew Kowalik 73d78bc76a Update exception docstring 2017-06-16 23:21:11 -07:00
Andrew Kowalik 83f2d322a9 raise KafkaTimeoutException when flush times out 2017-06-16 23:21:11 -07:00
Dana Powers bb626dbffb Follow-up: support manual py26 testing; dont advertise 3.3 support 2017-06-16 23:05:17 -07:00
Jianbin Wei b1a6d3301f Add kafka 0.10.2.1 into integration testing version (#1096)
* Add kafka 0.10.2.1 into integration testing version
* Disable tests for python 2.6 and kafka 0.8.0 and 0.8.1.1
* Remove references to python 2.6 support
2017-06-16 22:58:58 -07:00
Jianbin Wei 72205c4692 pylint 1.7.0+ supports python 3.6 and merge py36 into common tstenv (#1095) 2017-05-03 14:44:19 -07:00
Dana Powers a2b5ddc375 Improve error message when expiring batches in KafkaProducer (#1077) 2017-05-03 08:08:06 -07:00
Dana Powers 83617b956d Warn dont raise on DNS lookup failures (#1091) 2017-05-03 08:06:59 -07:00
Dana Powers 384420425c Ignore not-context-manager pylint error (#1092) 2017-04-29 11:36:35 -07:00
Dana Powers 120410bcb4 Follow-up to #1068: remove not-implemented comments 2017-04-10 12:38:02 -07:00
Dana Powers a503682ee6 readthedocs.io follow-up to #1074 for docs/ 2017-04-10 10:11:05 -07:00
Jeff Widman 5bcb333b3a Update RTD links with https / .io (#1074) 2017-04-10 10:06:55 -07:00
Dana Powers 04296994de Timeout idle connections via connections_max_idle_ms (#1068) 2017-04-10 10:06:46 -07:00
Dana Powers 7c24135eaf Avoid multiple connection attempts when refreshing metadata (#1067) 2017-04-05 23:13:02 -07:00
Dana Powers bb9642f04c Catch socket.errors when sending / recving bytes on wake socketpair (#1069) 2017-04-05 10:11:21 -07:00
Jeff Widman d40d106c73 Fix poll() hyperlink in KafkaClient
Previously Sphinx was auto-linking to `poll()` in `KafkaConsumer`, so made the link explicit.
2017-03-29 17:35:09 -07:00
Kirill Ignatev acbb510b8d Update README: Prefer python-lz4 over lz4tools (#1057) 2017-03-27 12:41:30 -07:00
Jeff Widman bee0bbe3af Fix typo (#1054) 2017-03-27 12:40:30 -07:00
Dana Powers 998147dae9 Bump version for development 2017-03-14 15:43:16 -07:00
Dana Powers 0e24da3410 Release 1.3.3 2017-03-14 13:36:40 -07:00
Dana Powers 65ba8822b1 Derive all api classes from Request / Response base classes (#1030) 2017-03-14 13:34:37 -07:00
Dana Powers a00f9ead16 Alter test skips: python-lz4 works on python26, but not pypy 2017-03-14 13:30:35 -07:00
Dana Powers fea10d9c16 LZ4 support in kafka 0.8/0.9 does not accept a ContentSize header 2017-03-14 13:30:35 -07:00
Dana Powers fb023fe85d Prefer python-lz4 over lz4f if available 2017-03-14 13:30:35 -07:00
Dana Powers 3b899decb8 Free lz4 decompression context to avoid leak 2017-03-14 13:30:35 -07:00
Dana Powers e775e05d65 Update changelog 2017-03-13 19:04:02 -07:00
Dana Powers 47004bbd02 Avoid re-encoding for message crc check (#1027) 2017-03-13 16:41:38 -07:00
Dana Powers 92a66e3009 Additional docstrings for autocommit close option 2017-03-13 16:39:53 -07:00
Dana Powers 195df5fb98 Optionally skip auto-commit during consumer.close (#1031) 2017-03-13 16:39:23 -07:00
Dana Powers ce57dac0c6 Return copy of consumer subscription set (#1029) 2017-03-13 12:42:47 -07:00
Dana Powers 899f11730d Fix kwarg handing in kafka.protocol.struct.Struct (#1025) 2017-03-09 15:12:27 -08:00
Dana Powers bb709f4c14 Short-circuit group coordinator requests when NodeNotReady (#995) 2017-03-09 14:26:09 -08:00