Merge "Introducing a dnsmasq PXE filter driver"

This commit is contained in:
Zuul 2017-11-24 13:28:57 +00:00 committed by Gerrit Code Review
commit 2d3d32ba22
9 changed files with 588 additions and 0 deletions

View File

@ -0,0 +1,123 @@
.. _dnsmasq_pxe_filter:
**dnsmasq** PXE filter
======================
Often an inspection PXE DHCP stack is implemented by the **dnsmasq** service.
This PXE filter implementation relies on directly configuring the **dnsmasq**
DHCP service to provide a caching PXE-traffic filter of node MAC addresses.
How it works
------------
Using a configuration *file per MAC address* allows one to implement a
filtering mechanism based on the ``ignore`` directive::
$ cat /etc/dnsmasq.d/de-ad-be-ef-de-ad
de:ad:be:ef:de:ad,ignore
$
The filename is used to keep track of all MAC addresses in the cache, avoiding
file parsing. The content of the file determines the MAC address access policy.
Thanks to the ``inotify`` facility, **dnsmasq** is notified instantly once a
new file is *created* or an existing file is *modified* in the
DHCP hosts directory. Thus, to white-list a MAC address, one has to
remove the ``ignore`` directive::
$ cat /etc/dnsmasq.d/de-ad-be-ef-de-ad
de:ad:be:ef:de:ad
$
The hosts directory content establishes a *cached* MAC addresses filter that is
kept synchronized with the **ironic** port list.
.. note::
The **dnsmasq** inotify facility implementation doesn't react on a file being
removed or truncated.
Configuration
-------------
To enable the **dnsmasq** PXE filter, update the PXE filter driver name::
[pxe_filter]
driver = dnsmasq
The DHCP hosts directory can be specified to override the default
``/var/lib/ironic-inspector/dhcp-hostsdir``::
[dnsmasq_pxe_filter]
dhcp_hostsdir = /etc/ironic-inspector/dhcp-hostsdir
The filter design relies on the hosts directory being in exclusive
**inspector** control. The hosts directory should be considered a *private
cache* directory of **inspector** that **dnsmasq** polls configuration updates
from, through the ``inotify`` facility. The directory has to be writable by
**inspector** and readable by **dnsmasq**.
One can also override the default start and stop commands to control the
**dnsmasq** service::
[dnsmasq_pxe_filter]
dnsmasq_start_command = dnsmasq --conf-file /etc/ironic-inspector/dnsmasq.conf
dnsmasq_stop_command = kill $(cat /var/run/dnsmasq.pid)
.. note::
It is also possible to set an empty/none string or to use shell expansion in
place of the commands. An empty start command means the **dnsmasq** service
won't be started upon the filter initialization, an empty stop command means
the service won't be stopped upon an (error) exit.
.. note::
These commands are executed through the ``rootwrap`` facility, so overriding
may require a filter file to be created in the ``rootwrap.d`` directory. A
sample configuration for **devstack** use might be:
.. code-block:: console
sudo cat > "$IRONIC_INSPECTOR_CONF_DIR/rootwrap.d/ironic-inspector-dnsmasq-systemctl.filters" <<EOF
[Filters]
# ironic_inspector/pxe_filter/dnsmasq.py
systemctl: CommandFilter, systemctl, root, restart, devstack@ironic-inspector-dnsmasq
systemctl: CommandFilter, systemctl, root, stop, devstack@ironic-inspector-dnsmasq
EOF
Supported dnsmasq versions
--------------------------
This filter driver has been checked by **inspector** CI with **dnsmasq**
versions `>=2.76`. The ``inotify`` facility was introduced_ to **dnsmasq** in
the version `2.73`.
.. _introduced: http://www.thekelleys.org.uk/dnsmasq/CHANGELOG
Caveats
-------
The initial synchronization will put some load on the **dnsmasq** service
starting based on the amount of ports **ironic** keeps. This can take up to a
minute of full CPU load for huge amounts of MACs (tens of thousands).
Subsequent filter synchronizations will only cause the **dnsmasq** to parse
the modified files. Typically those are the bare metal nodes being added or
phased out from the compute service, meaning dozens of file updates per sync
call.
The **inspector** takes over the control of the DHCP hosts directory to
implement its filter cache. Files are generated dynamically so should not be
edited by hand. To minimize the interference between the deployment and
introspection, **inspector** has to start the **dnsmasq** service only after
the initial synchronization. Conversely, the **dnsmasq** service is stopped
upon (unexpected) **inspector** exit.
To avoid accumulating stale DHCP host files over time, the driver cleans up
the DHCP hosts directory during the ``init_filter`` call.
Although the filter driver tries its best to always stop the **dnsmasq**
service, it is recommended that the operator configures the **dnsmasq**
service in such a way that it terminates upon **inspector** (unexpected) exit
to prevent a stale blacklist from being used by the **dnsmasq** service.

View File

@ -8,3 +8,11 @@ How to upgrade Ironic Inspector
:maxdepth: 2
upgrade
Dnsmasq PXE filter driver
-------------------------
.. toctree::
:maxdepth: 2
dnsmasq-pxe-filter

View File

@ -76,6 +76,9 @@ Fill in these minimum configuration values:
(defaults to ``br-ctlplane`` which is a sane default for **tripleo**-based
installations but is unlikely to work for other cases).
* if you wish to use the ``dnsmasq`` PXE/DHCP filter driver rather than the
default ``iptables`` driver, see the :ref:`dnsmasq_pxe_filter` description.
See comments inside `example.conf
<https://github.com/openstack/ironic-inspector/blob/master/example.conf>`_
for other possible configuration options.

View File

@ -340,6 +340,25 @@
#enroll_node_driver = fake
[dnsmasq_pxe_filter]
#
# From ironic_inspector
#
# The MAC address cache directory, exposed to dnsmasq.This directory
# is expected to be in exclusive control of the driver. (string value)
#dhcp_hostsdir = /var/lib/ironic-inspector/dhcp-hostsdir
# A (shell) command line to start the dnsmasq service upon filter
# initialization. Default: don't start. (string value)
#dnsmasq_start_command =
# A (shell) command line to stop the dnsmasq service upon inspector
# (error) exit. Default: don't stop. (string value)
#dnsmasq_stop_command =
[iptables]
#

View File

@ -209,11 +209,26 @@ PXE_FILTER_OPTS = [
'update of the filter.')),
]
DNSMASQ_PXE_FILTER_OPTS = [
cfg.StrOpt('dhcp_hostsdir',
default='/var/lib/ironic-inspector/dhcp-hostsdir',
help=_('The MAC address cache directory, exposed to dnsmasq.'
'This directory is expected to be in exclusive control '
'of the driver.')),
cfg.StrOpt('dnsmasq_start_command', default='',
help=_('A (shell) command line to start the dnsmasq service '
'upon filter initialization. Default: don\'t start.')),
cfg.StrOpt('dnsmasq_stop_command', default='',
help=_('A (shell) command line to stop the dnsmasq service '
'upon inspector (error) exit. Default: don\'t stop.')),
]
cfg.CONF.register_opts(SERVICE_OPTS)
cfg.CONF.register_opts(IPTABLES_OPTS, group='iptables')
cfg.CONF.register_opts(PROCESSING_OPTS, group='processing')
cfg.CONF.register_opts(PXE_FILTER_OPTS, 'pxe_filter')
cfg.CONF.register_opts(DNSMASQ_PXE_FILTER_OPTS, group='dnsmasq_pxe_filter')
def list_opts():
@ -222,6 +237,7 @@ def list_opts():
('iptables', IPTABLES_OPTS),
('processing', PROCESSING_OPTS),
('pxe_filter', PXE_FILTER_OPTS),
('dnsmasq_pxe_filter', DNSMASQ_PXE_FILTER_OPTS),
]

View File

@ -0,0 +1,176 @@
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# NOTE(milan) the filter design relies on the hostdir[1] being in exclusive
# inspector control. The hostdir should be considered a private cache directory
# of inspector that dnsmasq has read access to and polls updates from, through
# the inotify facility.
#
# [1] see the --dhcp-hostsdir option description in
# http://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html
import os
from oslo_concurrency import processutils
from oslo_config import cfg
from oslo_log import log
from oslo_utils import timeutils
from ironic_inspector.common import ironic as ir_utils
from ironic_inspector import node_cache
from ironic_inspector.pxe_filter import base as pxe_filter
CONF = cfg.CONF
LOG = log.getLogger(__name__)
_ROOTWRAP_COMMAND = 'sudo ironic-inspector-rootwrap {rootwrap_config!s}'
_MACBL_LEN = len('ff:ff:ff:ff:ff:ff,ignore\n')
class DnsmasqFilter(pxe_filter.BaseFilter):
"""The dnsmasq PXE filter driver.
A pxe filter driver implementation that controls access to dnsmasq
through amending its configuration.
"""
def reset(self):
"""Stop dnsmasq and upcall reset."""
_execute(CONF.dnsmasq_pxe_filter.dnsmasq_stop_command,
ignore_errors=True)
super(DnsmasqFilter, self).reset()
def _sync(self, ironic):
"""Sync the inspector, ironic and dnsmasq state. Locked.
:raises: IOError, OSError.
:returns: None.
"""
LOG.debug('Syncing the driver')
timestamp_start = timeutils.utcnow()
active_macs = node_cache.active_macs()
ironic_macs = set(port.address for port in
ironic.port.list(limit=0, fields=['address']))
blacklist_macs = _get_blacklist()
# NOTE(milan) whitelist MACs of ports not kept in ironic anymore
# also whitelist active MACs that are still blacklisted in the
# dnsmasq configuration but have just been asked to be introspected
for mac in ((blacklist_macs - ironic_macs) |
(blacklist_macs & active_macs)):
_whitelist_mac(mac)
# blacklist new ports that aren't being inspected
for mac in ironic_macs - (blacklist_macs | active_macs):
_blacklist_mac(mac)
timestamp_end = timeutils.utcnow()
LOG.debug('The dnsmasq PXE filter was synchronized (took %s)',
timestamp_end - timestamp_start)
@pxe_filter.locked_driver_event(pxe_filter.Events.sync)
def sync(self, ironic):
"""Sync dnsmasq configuration with current Ironic&Inspector state.
Polls all ironic ports. Those being inspected, the active ones, are
whitelisted while the rest are blacklisted in the dnsmasq
configuration.
:param ironic: an ironic client instance.
:raises: OSError, IOError.
:returns: None.
"""
self._sync(ironic)
@pxe_filter.locked_driver_event(pxe_filter.Events.initialize)
def init_filter(self):
"""Performs an initial sync with ironic and starts dnsmasq.
The initial _sync() call reduces the chances dnsmasq might lose
some inotify blacklist events by prefetching the blacklist before
the dnsmasq is started.
:raises: OSError, IOError.
:returns: None.
"""
_purge_dhcp_hostsdir()
ironic = ir_utils.get_client()
self._sync(ironic)
_execute(CONF.dnsmasq_pxe_filter.dnsmasq_start_command)
LOG.info('The dnsmasq PXE filter was initialized')
def _purge_dhcp_hostsdir():
"""Remove all the DHCP hosts files.
:raises: FileNotFoundError in case the dhcp_hostsdir is invalid.
IOError in case of non-writable file or a record not being a file.
:returns: None.
"""
dhcp_hostsdir = CONF.dnsmasq_pxe_filter.dhcp_hostsdir
LOG.debug('Purging %s', dhcp_hostsdir)
for mac in os.listdir(dhcp_hostsdir):
path = os.path.join(dhcp_hostsdir, mac)
# NOTE(milan) relying on a failure here aborting the init_filter() call
os.remove(path)
LOG.debug('Removed %s', path)
def _get_blacklist():
"""Get addresses currently blacklisted in dnsmasq.
:raises: FileNotFoundError in case the dhcp_hostsdir is invalid.
:returns: a set of MACs currently blacklisted in dnsmasq.
"""
hostsdir = CONF.dnsmasq_pxe_filter.dhcp_hostsdir
# whitelisted MACs lack the ,ignore directive
return set(address for address in os.listdir(hostsdir)
if os.stat(os.path.join(hostsdir, address)).st_size ==
_MACBL_LEN)
def _blacklist_mac(mac):
"""Creates a dhcp_hostsdir ignore record for the MAC.
:raises: FileNotFoundError in case the dhcp_hostsdir is invalid,
IOError in case the dhcp host MAC file isn't writable.
:returns: None.
"""
path = os.path.join(CONF.dnsmasq_pxe_filter.dhcp_hostsdir, mac)
# NOTE(milan) line-buffering enforced to ensure dnsmasq record update
# through inotify, which reacts on f.close()
with open(path, 'w', 1) as f:
f.write('%s,ignore\n' % mac)
LOG.debug('Blacklisted %s', mac)
def _whitelist_mac(mac):
"""Un-ignores the dhcp_hostsdir record for the MAC.
:raises: FileNotFoundError in case the dhcp_hostsdir is invalid,
IOError in case the dhcp host MAC file isn't writable.
:returns: None.
"""
path = os.path.join(CONF.dnsmasq_pxe_filter.dhcp_hostsdir, mac)
with open(path, 'w', 1) as f:
# remove the ,ignore directive
f.write('%s\n' % mac)
LOG.debug('Whitelisted %s', mac)
def _execute(cmd=None, ignore_errors=False):
# e.g: '/bin/kill $(cat /var/run/dnsmasq.pid)'
if not cmd:
return
helper = _ROOTWRAP_COMMAND.format(rootwrap_config=CONF.rootwrap_config)
processutils.execute(cmd, run_as_root=True, root_helper=helper, shell=True,
check_exit_code=not ignore_errors)

View File

@ -0,0 +1,236 @@
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import datetime
import os
import fixtures
import mock
from oslo_config import cfg
import six
from ironic_inspector.common import ironic as ir_utils
from ironic_inspector import node_cache
from ironic_inspector.pxe_filter import dnsmasq
from ironic_inspector.test import base as test_base
CONF = cfg.CONF
class DnsmasqTestBase(test_base.BaseTest):
def setUp(self):
super(DnsmasqTestBase, self).setUp()
self.driver = dnsmasq.DnsmasqFilter()
class TestDnsmasqDriverAPI(DnsmasqTestBase):
def setUp(self):
super(TestDnsmasqDriverAPI, self).setUp()
self.mock__execute = self.useFixture(
fixtures.MockPatchObject(dnsmasq, '_execute')).mock
self.driver._sync = mock.Mock()
self.driver._tear_down = mock.Mock()
self.mock__purge_dhcp_hostsdir = self.useFixture(
fixtures.MockPatchObject(dnsmasq, '_purge_dhcp_hostsdir')).mock
self.mock_ironic = mock.Mock()
get_client_mock = self.useFixture(
fixtures.MockPatchObject(ir_utils, 'get_client')).mock
get_client_mock.return_value = self.mock_ironic
self.start_command = '/far/boo buzz -V --ack 42'
CONF.set_override('dnsmasq_start_command', self.start_command,
'dnsmasq_pxe_filter')
self.stop_command = '/what/ever'
CONF.set_override('dnsmasq_stop_command', self.stop_command,
'dnsmasq_pxe_filter')
def test_init_filter(self):
self.driver.init_filter()
self.mock__purge_dhcp_hostsdir.assert_called_once_with()
self.driver._sync.assert_called_once_with(self.mock_ironic)
self.mock__execute.assert_called_once_with(self.start_command)
def test_sync(self):
self.driver.init_filter()
# NOTE(milan) init_filter performs an initial sync
self.driver._sync.reset_mock()
self.driver.sync(self.mock_ironic)
self.driver._sync.assert_called_once_with(self.mock_ironic)
def test_tear_down_filter(self):
mock_reset = self.useFixture(
fixtures.MockPatchObject(self.driver, 'reset')).mock
self.driver.init_filter()
self.driver.tear_down_filter()
mock_reset.assert_called_once_with()
def test_reset(self):
self.driver.init_filter()
# NOTE(milan) init_filter calls _base_cmd
self.mock__execute.reset_mock()
self.driver.reset()
self.mock__execute.assert_called_once_with(
self.stop_command, ignore_errors=True)
class TestMACHandlers(test_base.BaseTest):
def setUp(self):
super(TestMACHandlers, self).setUp()
self.mock_listdir = self.useFixture(
fixtures.MockPatchObject(os, 'listdir')).mock
self.mock_stat = self.useFixture(
fixtures.MockPatchObject(os, 'stat')).mock
self.mock_remove = self.useFixture(
fixtures.MockPatchObject(os, 'remove')).mock
self.mac = 'ff:ff:ff:ff:ff:ff'
self.dhcp_hostsdir = '/far'
CONF.set_override('dhcp_hostsdir', self.dhcp_hostsdir,
'dnsmasq_pxe_filter')
self.mock_join = self.useFixture(
fixtures.MockPatchObject(os.path, 'join')).mock
self.mock_join.return_value = "%s/%s" % (self.dhcp_hostsdir, self.mac)
def test__whitelist_mac(self):
with mock.patch.object(six.moves.builtins, 'open',
new=mock.mock_open()) as mock_open:
dnsmasq._whitelist_mac(self.mac)
mock_fd = mock_open.return_value
self.mock_join.assert_called_once_with(self.dhcp_hostsdir, self.mac)
mock_open.assert_called_once_with(self.mock_join.return_value, 'w', 1)
mock_fd.write.assert_called_once_with('%s\n' % self.mac)
def test__blacklist_mac(self):
with mock.patch.object(six.moves.builtins, 'open',
new=mock.mock_open()) as mock_open:
dnsmasq._blacklist_mac(self.mac)
mock_fd = mock_open.return_value
self.mock_join.assert_called_once_with(self.dhcp_hostsdir, self.mac)
mock_open.assert_called_once_with(self.mock_join.return_value, 'w', 1)
mock_fd.write.assert_called_once_with('%s,ignore\n' % self.mac)
def test__get_blacklist(self):
self.mock_listdir.return_value = [self.mac]
self.mock_stat.return_value.st_size = len('%s,ignore\n' % self.mac)
ret = dnsmasq._get_blacklist()
self.assertEqual({self.mac}, ret)
self.mock_listdir.assert_called_once_with(self.dhcp_hostsdir)
self.mock_join.assert_called_once_with(self.dhcp_hostsdir, self.mac)
self.mock_stat.assert_called_once_with(self.mock_join.return_value)
def test__get_no_blacklist(self):
self.mock_listdir.return_value = [self.mac]
self.mock_stat.return_value.st_size = len('%s\n' % self.mac)
ret = dnsmasq._get_blacklist()
self.assertEqual(set(), ret)
self.mock_listdir.assert_called_once_with(self.dhcp_hostsdir)
self.mock_join.assert_called_once_with(self.dhcp_hostsdir, self.mac)
self.mock_stat.assert_called_once_with(self.mock_join.return_value)
def test__purge_dhcp_hostsdir(self):
self.mock_listdir.return_value = [self.mac]
dnsmasq._purge_dhcp_hostsdir()
self.mock_listdir.assert_called_once_with(self.dhcp_hostsdir)
self.mock_join.assert_called_once_with(self.dhcp_hostsdir, self.mac)
self.mock_remove.assert_called_once_with('%s/%s' % (self.dhcp_hostsdir,
self.mac))
class TestSync(DnsmasqTestBase):
def setUp(self):
super(TestSync, self).setUp()
self.mock__get_blacklist = self.useFixture(
fixtures.MockPatchObject(dnsmasq, '_get_blacklist')).mock
self.mock__whitelist_mac = self.useFixture(
fixtures.MockPatchObject(dnsmasq, '_whitelist_mac')).mock
self.mock__blacklist_mac = self.useFixture(
fixtures.MockPatchObject(dnsmasq, '_blacklist_mac')).mock
self.mock_ironic = mock.Mock()
self.mock_utcnow = self.useFixture(
fixtures.MockPatchObject(dnsmasq.timeutils, 'utcnow')).mock
self.timestamp_start = datetime.datetime.utcnow()
self.timestamp_end = (self.timestamp_start +
datetime.timedelta(seconds=42))
self.mock_utcnow.side_effect = [self.timestamp_start,
self.timestamp_end]
self.mock_log = self.useFixture(
fixtures.MockPatchObject(dnsmasq, 'LOG')).mock
get_client_mock = self.useFixture(
fixtures.MockPatchObject(ir_utils, 'get_client')).mock
get_client_mock.return_value = self.mock_ironic
self.mock_active_macs = self.useFixture(
fixtures.MockPatchObject(node_cache, 'active_macs')).mock
self.ironic_macs = {'new_mac', 'active_mac'}
self.active_macs = {'active_mac'}
self.blacklist_macs = {'gone_mac', 'active_mac'}
self.mock__get_blacklist.return_value = self.blacklist_macs
self.mock_ironic.port.list.return_value = [
mock.Mock(address=address) for address in self.ironic_macs]
self.mock_active_macs.return_value = self.active_macs
def test__sync(self):
self.driver._sync(self.mock_ironic)
self.mock__whitelist_mac.assert_has_calls([mock.call('active_mac'),
mock.call('gone_mac')],
any_order=True)
self.mock__blacklist_mac.assert_has_calls([mock.call('new_mac')],
any_order=True)
self.mock_ironic.port.list.assert_called_once_with(limit=0,
fields=['address'])
self.mock_active_macs.assert_called_once_with()
self.mock__get_blacklist.assert_called_once_with()
self.mock_log.debug.assert_has_calls([
mock.call('Syncing the driver'),
mock.call('The dnsmasq PXE filter was synchronized (took %s)',
self.timestamp_end - self.timestamp_start)
])
class Test_Execute(test_base.BaseTest):
def setUp(self):
super(Test_Execute, self).setUp()
self.mock_execute = self.useFixture(
fixtures.MockPatchObject(dnsmasq.processutils, 'execute')
).mock
CONF.set_override('rootwrap_config', '/path/to/rootwrap.conf')
self.rootwrap_cmd = dnsmasq._ROOTWRAP_COMMAND.format(
rootwrap_config=CONF.rootwrap_config)
self.useFixture(fixtures.MonkeyPatch(
'ironic_inspector.pxe_filter.dnsmasq._ROOTWRAP_COMMAND',
self.rootwrap_cmd))
self.command = 'foobar baz'
def test__execute(self):
dnsmasq._execute(self.command)
self.mock_execute.assert_called_once_with(
self.command, run_as_root=True, shell=True,
check_exit_code=True, root_helper=self.rootwrap_cmd)
def test__execute_ignoring_errors(self):
dnsmasq._execute(self.command, ignore_errors=True)
self.mock_execute.assert_called_once_with(
self.command, run_as_root=True, shell=True,
check_exit_code=False, root_helper=self.rootwrap_cmd)
def test__execute_empty(self):
dnsmasq._execute()
self.mock_execute.assert_not_called()

View File

@ -0,0 +1,6 @@
---
features:
- |
Introduces the **dnsmasq** PXE filter driver. This driver takes advantage of
the ``inotify`` facility to reconfigure the **dnsmasq** service in real time
to implement a caching black-/white-list of port MAC addresses.

View File

@ -58,6 +58,7 @@ ironic_inspector.rules.actions =
set-capability = ironic_inspector.plugins.rules:SetCapabilityAction
extend-attribute = ironic_inspector.plugins.rules:ExtendAttributeAction
ironic_inspector.pxe_filter =
dnsmasq = ironic_inspector.pxe_filter.dnsmasq:DnsmasqFilter
iptables = ironic_inspector.pxe_filter.iptables:IptablesFilter
noop = ironic_inspector.pxe_filter.base:NoopFilter
oslo.config.opts =