add X.509 certificate check plugin
Currently we don't have any capability to monitor the internal TLS/SSL certificates. i.e. SSL certificates used by MySQL for replication, RabbitMQ for distribution, etc. The cert_check plugin is not adequate for this purpose becaue it can only check on certficates over HTTPS endpoints. Furthermore, checking on these internal certificates over the network is cumbersome because the agent plugin would have to speak specific protocols. This patch adds a cert_file_check plugin to detect the certificate expiry (in days from now) for the given X.509 certificate file in PEM format. Similar to cert_check plugin, this plugin will a metric 'cert_file.cert_expire_days' which contains the number of days from now the given certificate will be expired. If the certificate has already expired, this will be a negative number. Change-Id: Id95cc7115823f972e234417223ab5906b57447cc Story: 2006753
This commit is contained in:
Guang Yee
parent
4b70995282
commit
e1d73c4b5d
|
|
@ -30,6 +30,7 @@
|
|||
- [Check_MK_Local](#check_mk_local)
|
||||
- [Ceph](#ceph)
|
||||
- [Certificate Expiration (HTTPS)](#certificate-expiration-https)
|
||||
- [Certificate Expiration (PEM File)](#certificate-expiration-file)
|
||||
- [Congestion](#congestion)
|
||||
- [Couch](#couch)
|
||||
- [Couchbase](#couchbase)
|
||||
|
|
@ -146,6 +147,7 @@ The following plugins are delivered via setup as part of the standard plugin che
|
|||
| cacti | | |
|
||||
| cAdvisor_host | | |
|
||||
| cert_check | | |
|
||||
| cert_file_check | | |
|
||||
| check_mk_local | | |
|
||||
| couch | | |
|
||||
| couchbase | | |
|
||||
|
|
@ -326,6 +328,7 @@ These are the detection plugins included with the Monasca Agent. See [Customiza
|
|||
| ceilometer | ServicePlugin |
|
||||
| ceph | Plugin |
|
||||
| cert_check | ArgsPlugin |
|
||||
| cert_file_check | ArgsPlugin |
|
||||
| check_mk_local | Plugin |
|
||||
| cinder | ServicePlugin |
|
||||
| cloudkitty | ServicePlugin |
|
||||
|
|
@ -920,6 +923,78 @@ These options can be set if desired:
|
|||
* collect_period: Integer time in seconds between outputting the metric. Since the metric is in days, it makes sense to output it at a slower rate. The default is 3600, once per hour
|
||||
* timeout: Float time in seconds before timing out the connect to the url. Increase if needed for very slow servers, but making this too long will increase the time this plugin takes to run if the server for the url is down. The default is 1.0 seconds
|
||||
|
||||
## Certificate Expiration (PEM file)
|
||||
An extension to the Agent provides the ability to determine the expiration date
|
||||
of the PEM formatted X.509 certificate in a file. The metric is days until the
|
||||
certificate expires. If the given certificate has already expired, it will
|
||||
return a negative number. For example, if the certificate has already expired
|
||||
5 days prior to the check, -5 will be returned.
|
||||
|
||||
Notice that the days till expiration value is calculated based on every 24
|
||||
hours block, rounded down. For example, if the certificate will be expiring in
|
||||
23 hours, the days till expiration value will be 0. Here are some more
|
||||
examples.
|
||||
|
||||
| Certificate Expiration Date | Plugin Execution Date | cert_file.cert_expire_days |
|
||||
| --------------------------- | --------------------- | -------------------------- |
|
||||
| Nov 1 23:00:50 2019 GMT | Oct 29 23:01:00 2019 GMT | 2 |
|
||||
| Nov 1 23:00:50 2019 GMT | Oct 30 23:01:00 2019 GMT | 1 |
|
||||
| Nov 1 23:00:50 2019 GMT | Nov 1 10:00:50 2019 GMT | 0 |
|
||||
| Nov 1 23:00:50 2019 GMT | Nov 1 23:01:50 2019 GMT | 0 |
|
||||
| Nov 1 23:00:50 2019 GMT | Nov 2 23:01:50 2019 GMT | -1 |
|
||||
| Nov 1 23:00:50 2019 GMT | Nov 3 23:23:00 2019 GMT | -2 |
|
||||
|
||||
The following dimensions are included with the metric by default:
|
||||
```
|
||||
cert_file: cert_file
|
||||
```
|
||||
|
||||
### Configuration
|
||||
A YAML file (cert_file_check.yaml) contains the list of cert_files to check.
|
||||
|
||||
The configuration of the certificate expiration check is done in YAML, and consists of two keys:
|
||||
|
||||
* init_config
|
||||
* instances
|
||||
|
||||
The init_config section lists the global configuration settings, such as the
|
||||
period (in seconds) at which to output the metric. The default value for
|
||||
collect_period is 1 hour (3600 seconds). To change it to emit metric every
|
||||
24 hours, set it to 86400.
|
||||
|
||||
```yaml
|
||||
init_config:
|
||||
collect_period: 3600
|
||||
```
|
||||
|
||||
The instances section contains the urls to check.
|
||||
|
||||
```yaml
|
||||
instances:
|
||||
- built_by: CertificateFileCheck
|
||||
cert_file: /etc/myservice/myservice.pem
|
||||
- built_by: CertificateFileCheck
|
||||
cert_file: /etc/ssl/myotherservice.pem
|
||||
```
|
||||
|
||||
The certicate expiration checks return the following metrics
|
||||
|
||||
| Metric Name | Dimensions | Semantics |
|
||||
| ----------- | ---------- | --------- |
|
||||
| cert_file.cert_expire_days | cert_file=supplied certificate file being checked | The number of days until the certificate expires
|
||||
|
||||
|
||||
There is a detection plugin that should be used to configure this extension. It is invoked as:
|
||||
|
||||
$ monasca-setup -d CertificateFileCheck -a "cert_files=/etc/myservice/myservice.pem,/etc/myotherservice/myotherservice.pem"
|
||||
|
||||
The cert_files option is a comma separated list of certificate files to check.
|
||||
|
||||
These options can be set if desired:
|
||||
* collect_period: Integer time in seconds between outputting the metric.
|
||||
Since the metric is in days, it makes sense to output it at a slower rate.
|
||||
The default is 3600, once per hour
|
||||
|
||||
## Congestion
|
||||
|
||||
This section describes the congestion check performed by monasca-agent.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,68 @@
|
|||
# Copyright 2019 SUSE LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
from cryptography.hazmat.backends import default_backend
|
||||
from cryptography import x509
|
||||
|
||||
from monasca_agent.collector.checks import AgentCheck
|
||||
|
||||
|
||||
class CertificateFileCheck(AgentCheck):
|
||||
"""Check the given certificate file and output a metric
|
||||
which is the number of days until it expires
|
||||
"""
|
||||
|
||||
def __init__(self, name, init_config, agent_config, instances=None):
|
||||
super(CertificateFileCheck, self).__init__(name, init_config,
|
||||
agent_config, instances)
|
||||
|
||||
def check(self, instance):
|
||||
cert_file = instance.get('cert_file', None)
|
||||
|
||||
if cert_file is None:
|
||||
self.log.warning('Instance have no "cert_file" configured.')
|
||||
return
|
||||
|
||||
dimensions = self._set_dimensions(None, instance)
|
||||
dimensions['cert_file'] = cert_file
|
||||
self.log.info('cert_file = %s' % cert_file)
|
||||
expire_in_days = self.get_expire_in_days(cert_file)
|
||||
if expire_in_days is not None:
|
||||
self.gauge('cert_file.cert_expire_days', expire_in_days,
|
||||
dimensions=dimensions)
|
||||
self.log.debug('%d days till expiration for %s' % (expire_in_days,
|
||||
cert_file))
|
||||
|
||||
def get_expire_in_days(self, cert_file):
|
||||
"""Take the path the the TLS certificate file and returns the number
|
||||
of till the certificate expires. If the certificate has already
|
||||
expired, it will return a negative number. For example,
|
||||
if the certificate has already expired 5 days prior to the check,
|
||||
-5 will be returned.
|
||||
"""
|
||||
try:
|
||||
with open(cert_file, 'r') as cf:
|
||||
pem_data = cf.read().encode('ascii')
|
||||
|
||||
cert = x509.load_pem_x509_certificate(pem_data, default_backend())
|
||||
return (cert.not_valid_after - datetime.utcnow()).days
|
||||
except IOError:
|
||||
self.log.warning(
|
||||
'Unable to read certificate from %s' % (cert_file))
|
||||
except ValueError:
|
||||
self.log.warning(
|
||||
'Unable to load certificate from %s. Invalid content.' % (
|
||||
cert_file))
|
||||
|
|
@ -0,0 +1,73 @@
|
|||
# Copyright 2019 SUSE LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
import logging
|
||||
|
||||
import monasca_setup.agent_config
|
||||
import monasca_setup.detection
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
DEFAULT_COLLECT_PERIOD = 3600
|
||||
|
||||
|
||||
class CertificateFileCheck(monasca_setup.detection.ArgsPlugin):
|
||||
"""Setup a X.509 certificate file check according to the passed in args.
|
||||
|
||||
Outputs one metric: cert_file.cert_expire_days which is the number of
|
||||
days until the certificate expires
|
||||
|
||||
Despite being a detection plugin, this plugin does no detection and
|
||||
will be a NOOP without arguments. Expects one argument, 'cert_files'
|
||||
which is a comma-separated list of PEM-formatted X.509 certificate
|
||||
files.
|
||||
|
||||
Examples:
|
||||
|
||||
monasca-setup -d CertificateFileCheck -a "cert_files=cert1.pem,cert2.pem"
|
||||
|
||||
These arguments are optional:
|
||||
collect_period: Integer time in seconds between outputting the metric.
|
||||
Since the metric is in days, it makes sense to output
|
||||
it at a slower rate. The default is once per hour
|
||||
"""
|
||||
|
||||
def _detect(self):
|
||||
"""Run detection, set self.available True if cert_files are detected
|
||||
"""
|
||||
self.available = self._check_required_args(['cert_files'])
|
||||
|
||||
def build_config(self):
|
||||
"""Build the config as a Plugins object and return.
|
||||
"""
|
||||
config = monasca_setup.agent_config.Plugins()
|
||||
instances = []
|
||||
init_config = {'collect_period': DEFAULT_COLLECT_PERIOD}
|
||||
if 'collect_period' in self.args:
|
||||
collect_period = int(self.args['collect_period'])
|
||||
init_config['collect_period'] = collect_period
|
||||
for cert_file in self.args['cert_files'].split(','):
|
||||
cert_file = cert_file.strip()
|
||||
# Allow comma terminated lists
|
||||
if not cert_file:
|
||||
continue
|
||||
log.info("\tAdding X.509 Certificate expiration check for {}".format(cert_file))
|
||||
instance = self._build_instance([])
|
||||
instance.update({'cert_file': cert_file, 'name': cert_file})
|
||||
instances.append(instance)
|
||||
|
||||
config['cert_file_check'] = {'init_config': init_config,
|
||||
'instances': instances}
|
||||
|
||||
return config
|
||||
|
|
@ -69,6 +69,8 @@ ovs =
|
|||
python-neutronclient>=6.3.0 # Apache-2.0
|
||||
swift_handoffs =
|
||||
swift >= 2.0.0 # Apache-2.0
|
||||
cert_file_check =
|
||||
cryptography>=2.1 # BSD/Apache-2.0
|
||||
|
||||
[global]
|
||||
setup-hooks =
|
||||
|
|
|
|||
|
|
@ -9,3 +9,5 @@ oslotest>=3.2.0 # Apache-2.0
|
|||
prometheus_client
|
||||
stestr>=1.0.0 # Apache-2.0
|
||||
docutils>=0.11 # OSI-Approved Open Source, Public Domain
|
||||
freezegun>=0.3.6 # Apache-2.0
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,144 @@
|
|||
# Copyright 2019 SUSE LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
import datetime
|
||||
import logging
|
||||
import mock
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
from cryptography import x509
|
||||
from cryptography.x509.oid import NameOID
|
||||
from cryptography.hazmat.primitives import hashes
|
||||
from cryptography.hazmat.backends import default_backend
|
||||
from cryptography.hazmat.primitives import serialization
|
||||
from cryptography.hazmat.primitives.asymmetric import rsa
|
||||
import freezegun
|
||||
|
||||
from monasca_agent.collector.checks_d import cert_file_check
|
||||
|
||||
LOG = logging.getLogger('monasca_agent.collector.checks.check.cert_file_check')
|
||||
|
||||
|
||||
def generate_selfsigned_cert(expired_in):
|
||||
key = rsa.generate_private_key(
|
||||
public_exponent=65537,
|
||||
key_size=1024,
|
||||
backend=default_backend()
|
||||
)
|
||||
issuer = subject = x509.Name([
|
||||
x509.NameAttribute(NameOID.COMMON_NAME, u'foo')
|
||||
])
|
||||
now = datetime.datetime.utcnow()
|
||||
cert = (
|
||||
x509.CertificateBuilder()
|
||||
.subject_name(issuer)
|
||||
.issuer_name(subject)
|
||||
.public_key(key.public_key())
|
||||
.serial_number(1)
|
||||
.not_valid_before(now)
|
||||
.not_valid_after(now + datetime.timedelta(days=expired_in))
|
||||
.sign(key, hashes.SHA256(), default_backend())
|
||||
)
|
||||
cert_pem = cert.public_bytes(encoding=serialization.Encoding.PEM)
|
||||
|
||||
return cert_pem.decode('ascii')
|
||||
|
||||
|
||||
class TestCertificateFileCheck(unittest.TestCase):
|
||||
|
||||
def setUp(self):
|
||||
super(TestCertificateFileCheck, self).setUp()
|
||||
self.cert_file_check_obj = cert_file_check.CertificateFileCheck(
|
||||
name='cert_file_check',
|
||||
init_config={},
|
||||
instances=[],
|
||||
agent_config={}
|
||||
)
|
||||
|
||||
def test_cert_file_is_none(self):
|
||||
with mock.patch.object(LOG, 'warning') as mock_log:
|
||||
self.cert_file_check_obj.check({'foo': 'bar'})
|
||||
mock_log.assert_called_with(
|
||||
'Instance have no "cert_file" configured.')
|
||||
|
||||
def test_unable_to_read_file(self):
|
||||
tmp_certdir = tempfile.mkdtemp(prefix='test-cert-file-check-')
|
||||
try:
|
||||
bogus_cert_file = os.path.join(tmp_certdir, 'foo')
|
||||
with mock.patch.object(LOG, 'warning') as mock_log:
|
||||
self.cert_file_check_obj.check({'cert_file': bogus_cert_file})
|
||||
mock_log.assert_called_with(
|
||||
'Unable to read certificate from %s' % (bogus_cert_file))
|
||||
finally:
|
||||
shutil.rmtree(tmp_certdir)
|
||||
|
||||
def test_unable_to_load_file(self):
|
||||
tmp_certdir = tempfile.mkdtemp(prefix='test-cert-file-check-')
|
||||
try:
|
||||
bogus_cert_file = os.path.join(tmp_certdir, 'foo')
|
||||
# create a non-PEM formatted certificate file
|
||||
with open(bogus_cert_file, 'w') as f:
|
||||
f.write('foo')
|
||||
|
||||
with mock.patch.object(LOG, 'warning') as mock_log:
|
||||
self.cert_file_check_obj.check({'cert_file': bogus_cert_file})
|
||||
mock_log.assert_called_with(
|
||||
'Unable to load certificate from %s. Invalid content.' % (
|
||||
bogus_cert_file))
|
||||
finally:
|
||||
shutil.rmtree(tmp_certdir)
|
||||
|
||||
def test_check(self):
|
||||
tmp_certdir = tempfile.mkdtemp(prefix='test-cert-file-check-')
|
||||
try:
|
||||
cert_file = os.path.join(tmp_certdir, 'foo')
|
||||
# create a self-signed cert that expires in 10 days from now
|
||||
with open(cert_file, 'w') as f:
|
||||
f.write(generate_selfsigned_cert(10))
|
||||
|
||||
with mock.patch.object(cert_file_check.CertificateFileCheck,
|
||||
'gauge') as mock_gauge:
|
||||
self.cert_file_check_obj.check({'cert_file': cert_file})
|
||||
args, kwargs = mock_gauge.call_args
|
||||
# make sure the plugin correctly detect the given cert will be
|
||||
# expiring in equal or less than 10 days from now
|
||||
self.assertEqual('cert_file.cert_expire_days', args[0])
|
||||
self.assertLessEqual(args[1], 10)
|
||||
finally:
|
||||
shutil.rmtree(tmp_certdir)
|
||||
|
||||
def test_check_expired_cert(self):
|
||||
tmp_certdir = tempfile.mkdtemp(prefix='test-cert-file-check-')
|
||||
try:
|
||||
cert_file = os.path.join(tmp_certdir, 'foo')
|
||||
# create a self-signed cert that has expired 10 days ago
|
||||
with open(cert_file, 'w') as f:
|
||||
f.write(generate_selfsigned_cert(10))
|
||||
|
||||
now = datetime.datetime.utcnow()
|
||||
back_to_the_future = now + datetime.timedelta(days=20)
|
||||
with freezegun.freeze_time(back_to_the_future):
|
||||
with mock.patch.object(cert_file_check.CertificateFileCheck,
|
||||
'gauge') as mock_gauge:
|
||||
self.cert_file_check_obj.check({'cert_file': cert_file})
|
||||
args, kwargs = mock_gauge.call_args
|
||||
# make sure the plugin correctly detect that the given
|
||||
# cert has already expired
|
||||
self.assertEqual('cert_file.cert_expire_days', args[0])
|
||||
self.assertLessEqual(args[1], -9)
|
||||
finally:
|
||||
shutil.rmtree(tmp_certdir)
|
||||
|
|
@ -0,0 +1,65 @@
|
|||
# Copyright 2019 SUSE LLC
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
import logging
|
||||
import unittest
|
||||
|
||||
from mock import patch
|
||||
|
||||
from monasca_setup.detection.plugins.cert_file_check import CertificateFileCheck
|
||||
|
||||
LOG = logging.getLogger('monasca_setup.detection.plugins.cert_check')
|
||||
|
||||
|
||||
class TestCertFileCheck(unittest.TestCase):
|
||||
|
||||
def setUp(self):
|
||||
unittest.TestCase.setUp(self)
|
||||
with patch.object(CertificateFileCheck, '_detect') as mock_detect:
|
||||
self.cert_obj = CertificateFileCheck('temp_dir')
|
||||
self.assertTrue(mock_detect.called)
|
||||
self.cert_obj.args = {'cert_files': '/etc/myservice/myserver.pem'}
|
||||
|
||||
def test_detect(self):
|
||||
self.cert_obj.available = False
|
||||
with patch.object(self.cert_obj, '_check_required_args',
|
||||
return_value=True) as mock_check_required_args:
|
||||
self.cert_obj._detect()
|
||||
self.assertTrue(self.cert_obj.available)
|
||||
self.assertTrue(mock_check_required_args.called)
|
||||
|
||||
def _build_config(self):
|
||||
with patch.object(self.cert_obj, '_build_instance',
|
||||
return_value={}) as mock_build_instance:
|
||||
result = self.cert_obj.build_config()
|
||||
self.assertTrue(mock_build_instance.called)
|
||||
self.assertEqual(
|
||||
result['cert_file_check']['instances'][0]['cert_file'],
|
||||
'/etc/myservice/myserver.pem')
|
||||
self.assertEqual(result['cert_file_check']['instances'][0]['name'],
|
||||
'/etc/myservice/myserver.pem')
|
||||
return result
|
||||
|
||||
def test_build_config_without_args(self):
|
||||
result = self._build_config()
|
||||
self.assertEqual(
|
||||
result['cert_file_check']['init_config']['collect_period'],
|
||||
3600)
|
||||
|
||||
def test_build_config_with_args(self):
|
||||
self.cert_obj.args.update({'collect_period': 1200})
|
||||
result = self._build_config()
|
||||
self.assertEqual(
|
||||
result['cert_file_check']['init_config']['collect_period'],
|
||||
1200)
|
||||
Loading…
Reference in New Issue