Kafka driver deployment guide

Depends-On: Idfb9fe3700d882c8285c6dc56b0620951178eba2
Change-Id: If8370c0c83312d675bde837f768ae40ec3603972
This commit is contained in:
Andy Smith 2018-10-15 13:57:10 -04:00
parent 344ec5e8bb
commit 0953fa1759
3 changed files with 247 additions and 1 deletions

View File

@ -7,3 +7,4 @@ Deployment Guide
drivers
AMQP1.0
kafka

245
doc/source/admin/kafka.rst Normal file
View File

@ -0,0 +1,245 @@
-----------------------------
Kafka Driver Deployment Guide
-----------------------------
============
Introduction
============
The Kafka Driver is an experimental messaging transport backend
in *oslo.messaging*. The driver maps the base oslo.messaging
capabilities for notification message exchange onto V2.0 of the
Apache Kafka distributed streaming platform. More detail regarding
the Apache Kafka server is available from the `Apache Kafka website`__.
__ https://kafka.apache.org/
More detail regarding the driver's implementation is available from
the `adding kafka driver specification`_ and the `update kafka driver
specification`_ .
.. _adding kafka driver specification: https://git.openstack.org/cgit/openstack/oslo-specs/tree/specs/liberty/adding-kafka-support.rst
.. _update kafka driver specification: https://git.openstack.org/cgit/openstack/oslo-specs/tree/specs/queens/update-kafka-support.rst
========
Overview
========
The Kafka driver **only** supports use for sending and receiving
*oslo.messaging* notifications. Specifically, the Kafka driver
does not support *oslo.messaging* RPC transfers. Communications between
the driver and Kafka server backend uses a `binary protocol over TCP`_
that defines all APIs as request response message pairs. The Kafka
driver integrates the confluent kafka python client for full
protocol support and utilizes the Producer API to publish notification
messages and the Consumer API for notification listener subscriptions.
The driver is able to work with a single instance of a Kafka server or
a clustered Kafka server deployment.
.. _binary protocol over TCP: https://kafka.apache.org/protocol.html
Hybrid Messaging Deployment
---------------------------
*Oslo.messaging* provides a mechanism to configure separate messaging
backends for RPC and notification communications. This is supported
through the definition of separate RPC and notification
`transport urls`_ in the service configuration. When the Kafka driver
is deployed for *oslo.messaging* notifications, a separate driver and
messaging backend must be deployed for RPC communications. For these
hybrid messaging configurations, either the `rabbit`_ or `amqp`_
drivers can be deployed for *oslo.messaging* RPC.
.. _transport urls: https://docs.openstack.org/oslo.messaging/latest/reference/transport.html
.. _rabbit: https://docs.openstack.org/oslo.messaging/latest/admin/drivers.html#rabbit
.. _amqp: https://docs.openstack.org/oslo.messaging/latest/admin/AMQP1.0.html
Topics and vhost Support
------------------------
The Kafka topic is the feed name to which records are
published. Topics in Kafka are multi-subscriber such that a topic can
have zero, one or many consumers that subscribe to the data written to
it. In *oslo.messaging*, a notification listener subscribes to a topic
in a supplied target that is directly mapped by the driver to the
Kafka topic. The Kafka server architecture does not natively support
vhosts. In order to support the presence of a vhost in the transport
url provided to the driver, the topic created on the Kafka server will
be appended with the virtual host name. This creates a unique topic
per virtual host but **note** there is otherwise no access
control or isolation provided by the Kafka server.
Listener Pools
--------------
The Kafka driver provides support for listener pools. This capability
is realized by mapping the listener pool name to a Kafka server
*consumer group* name. Each record published to a topic will be
delivered to one consumer instance within each subscribing pool
(e.g. *consumer group*). If a listener pool name is not assigned to the
notification listener, a single default *consumer group* will be used by
the Kafka driver and all listeners will be assigned to that
group and the messages will effectively be load balanced across the
competing listener instances.
Synchronous Commit
------------------
A primary functional difference between a Kafka server and a
classic broker queue is that the offset or position of the
message read from the commit log is controlled by the listener
(e.g. consumer). The driver will advance the offset it maintains
linearly as it reads message records from the server. To ensure that
duplicate messages are not generated during downtime or communication
interruption, the driver will *synchronously* commit the consumed
messages prior to the notification listener dispatch. Due to this, the
driver does not support the re-queue operation and the driver can not
replay messages from a Kafka partition.
=============
Prerequisites
=============
In order to run the driver the confluent-kafka python client must be
installed. The Kafka driver integrates a `Python client based on librdkafka`_
for full protocol support and utilizes the Producer API to publish
notification messages and the Consumer API for notification listener
subscriptions.
.. _Python client based on librdkafka: https://github.com/confluentinc/confluent-kafka-python
Source packages for the `confluent-kafka library`_ are available via PyPI.
Since the Kafka driver is an optional extension to *oslo.messaging*
these packages are not installed by default. Use the 'kafka' extras
tag when installing *oslo.messaging* in order to pull in these extra
packages:
::
python -m pip install oslo.messaging[kafka]
.. _confluent-kafka library: https://pypi.org/project/confluent-kafka/
=============
Configuration
=============
Transport URL Enable
--------------------
In *oslo.messaging*, the transport_url parameters define the OpenStack service
backends for RPC and Notify. The url is of the form::
transport://user:pass@host1:port[,hostN:portN]/virtual_host
Where the transport value specifies the rpc or notification backend as
one of ``amqp``, ``rabbit``, ``kafka``, etc. To specify and enable the
Kafka driver for notifications, in the section
``[oslo_messaging_notifications]`` of the service configuration file,
specify the ``transport_url`` parameter::
[oslo_messaging_notifications]
transport_url = kafka://username:password@kafkahostname:9092
Note, that if a ``transport_url`` parameter is not specified in the
``[oslo_messaging_notifications]`` section, the ``[DEFAULT]``
transport_url will be used for both RPC and Notify backends.
Driver Options
--------------
It is recommended that the default configuration options provided by
the Kafka driver be used. The configuration options can be modified
in the ``oslo_messaging_kafka`` section of the service configuration file.
Notification Listener Options
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
kafka_max_fetch_bytes
Initial maximum number of bytes per topic+partition to request
when fetching messages from the broker.
kafka_consumer_timeout
Consumer polling interval timeout.
consumer_group
The default client group identifier.
enable_auto_commit
Indicator to perform asynchronous consumer message commits.
max_poll_records
The maximum number of messages to return per consume/poll operation.
Notifier Options
^^^^^^^^^^^^^^^^
producer_batch_timeout
Delay (ms) to wait for messages in the producer queue to
accumulate before constructing message sets to transmit to broker
producer_batch_size
The maximum number of messages batched into one message set
Security Options
^^^^^^^^^^^^^^^^
security_protocol
The protocol used to communicate with the Kafka brokers.
sasl_mechanisms
SASL mechanism to use for authentication. Current driver support
is for PLAIN only.
ssl_cafile
A file containing the trusted Certificate Authority's digital
certificate (in PEM format). This certificate is used to
authenticate the messaging backend.
================
DevStack Support
================
The plugin for the Kafka *oslo.messaging* driver is supported by
DevStack. As the Kafka driver can only be deployed for notifications,
the plugin supports the deployment of several message bus configurations.
In local.conf ``[localrc]`` section, the `devstack-plugin-kafka`_
plugin repository must be enabled. For example:
::
[[local|localrc]]
enable_plugin kafka https://git.openstack.org/openstack/devstack-plugin-kafka
Set the Kafka and Scala version and location variables if needed for
the configuration
::
KAFKA_VERSION=2.0.0
KAFKA_BASEURL=http://www.apache.org/dist/kafka
SCALA_VERSION=2.12
SCALA_BASEURL=http://www.scala-lang.org/riles/archive
The **RPC_** and **NOTIFY_** variables will define the message bus
configuration that will be used. The hybrid configurations will allow
for the *rabbit* and *amqp* drivers to be used for the RPC transports
while the *kafka* driver will be used for the Notify transport. The
setting of the service variables will select which messaging
intermediary is enabled for the configuration:
+------------+--------------------+--------------------+
| | RPC | NOTIFY |
| +-----------+--------+-----------+--------+
| | SERVICE | PORT | SERVICE | PORT |
+------------+-----------+--------+-----------+--------+
| Config 1 | rabbit | 5672 | kafka | 9092 |
+------------+-----------+--------+-----------+--------+
| Config 1 | amqp | 5672 | kafka | 9092 |
+------------+-----------+--------+-----------+--------+
.. _devstack-plugin-kafka: https://github.com/openstack/devstack-plugin-kafka.git

View File

@ -378,7 +378,7 @@ class KafkaListener(base.PollStyleListener):
class KafkaDriver(base.BaseDriver):
"""Kafka Driver
Note: Current implementation of this driver is experimental.
See :doc:`kafka` for details.
"""
def __init__(self, conf, url, default_exchange=None,