554 lines
26 KiB
ReStructuredText
554 lines
26 KiB
ReStructuredText
..
|
|
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
|
License.
|
|
|
|
http://creativecommons.org/licenses/by/3.0/legalcode
|
|
|
|
===========================
|
|
Nova Certificate Validation
|
|
===========================
|
|
|
|
https://blueprints.launchpad.net/nova/+spec/nova-validate-certificates
|
|
|
|
OpenStack now supports signature verification for signed images. However, it
|
|
does not support strong certificate validation for certificates used to
|
|
generate image signatures. Specifically, nova has no mechanism to identify
|
|
trusted certificates. While nova verifies the signature of a signed image
|
|
using cursive, there is no way to determine if the certificate used to
|
|
generate and verify that signature is a certificate that is trusted by the
|
|
user. This change will introduce an addition to the nova API allowing the
|
|
user to specify a list of trusted certificates when creating or rebuilding
|
|
a server. These trusted certificates will be used to conduct certificate
|
|
validation in concert with signature verification in cursive, providing the
|
|
user confidence in the identity and integrity of the image being booted.
|
|
|
|
|
|
Problem description
|
|
===================
|
|
|
|
Nova is capable of verifying the signature of a signed image by using cursive,
|
|
the OpenStack signature verification library [2]. Signature verification
|
|
ensures that unmodified image data is retrieved from glance. However, the
|
|
validation of the certificate used to generate the signature of the signed
|
|
image is currently limited to a timestamp validity check, ensuring only that
|
|
the certificate is valid for use at the time of signature verification. There
|
|
is no mechanism to ensure that the certificate used is one approved by the end
|
|
user. An attacker with access to glance could replace a user's signed image
|
|
with a modified, malicious image signed with the attacker's certificate,
|
|
stored in the OpenStack deployment's certificate manager. If asked to boot
|
|
this modified image, the compute service would retrieve the image and its
|
|
corresponding certificate, verify the image signature, and proceed to boot a
|
|
virtual machine using the malicious image data. Providing support for
|
|
certificate validation in cursive helps nova detect this attack scenario and
|
|
take steps to alert the user of the potential compromise.
|
|
|
|
Note that this threat model considers glance to be untrusted and does not
|
|
include threats to the integrity, availability, or confidentiality of nova. It
|
|
assumes that: (1) an attacker has access to the certificate manager and is
|
|
able to store certificates for use with image signing, and (2) this attacker
|
|
is unable to access arbitrary certificate public/private key pairs belonging
|
|
to other users. An attacker with such access would be able to impersonate the
|
|
user, replacing signed images and perfectly updating the corresponding image
|
|
signatures and metadata as needed to conceal the attack.
|
|
|
|
Use Cases
|
|
---------
|
|
|
|
Nova users want to ensure that they are booting images they trust by
|
|
controlling the set of certificates used to sign their images.
|
|
|
|
With this change, a nova user can specify the identities of trusted
|
|
certificates when creating/rebuilding a server from a signed image if
|
|
signature verification and certificate validation are enabled. One of these
|
|
trusted certificates is expected to be the signing certificate of the
|
|
certificate used to generate the image signature.
|
|
|
|
With certificate validation enabled, image signature verification will only
|
|
succeed if the image signing certificate was generated by a trusted
|
|
certificate. This allows users to place themselves in-the-loop of the
|
|
signature verification process, requiring valid certificate information for
|
|
boot to succeed.
|
|
|
|
Note that these trusted certificates are stored in a certificate manager
|
|
independent of the compute service. For this work, a certificate manager is
|
|
any service backend supported by castellan that provides management
|
|
operations for certificates objects. Certificate management is often a
|
|
subset of the functionality provided by generic key managers, which are
|
|
capable of managing different types of cryptographic secrets (e.g.,
|
|
encryption keys, passwords). As of the Ocata release, barbican (the OpenStack
|
|
key management service) is the only OpenStack service that satisfies the
|
|
requirements for a certificate manager. In the future, any OpenStack or
|
|
third-party service that is supported by castellan and provides certificate
|
|
management could be used instead of barbican.
|
|
|
|
Proposed change
|
|
===============
|
|
|
|
Supporting certificate validation requires several changes. The initial change
|
|
adds a pair of new configuration options. The first configuration option,
|
|
enable_certificate_validation, will enable the use of the cursive certificate
|
|
validation routine when conducting image signature verification (see the
|
|
fifth change below). This option will only be used if verify_glance_signatures
|
|
is set to True and will default to False, allowing signature verification to
|
|
work without certificate validation until a compute deployment has fully
|
|
upgraded. Certificate validation will only be performed if this option is set
|
|
to True.
|
|
|
|
The second configuration option, default_trusted_certificate_ids, will contain
|
|
a list of certificate IDs that are designated as trusted by the compute
|
|
deployment. This list of trusted certificate IDs will only be used if
|
|
certificate validation is enabled and if no trusted certificate IDs were
|
|
provided by the user (see the second and third changes below). The list should
|
|
be defined by an administrator as it will be the default set of trusted
|
|
certificate IDs for the compute deployment. The default value of this option
|
|
will be an empty list, requiring a user-provided set of trusted certificate
|
|
IDs if left empty. If the user does provide a list of trusted certificate IDs,
|
|
the list of default trusted certificate IDs will be ignored.
|
|
|
|
The second change adds a parameter, trusted_certificates, to the server create
|
|
command of the nova API. The value of the parameter is an array containing the
|
|
string IDs of the trusted certificates used to validate the signed image's
|
|
signing certificate. These IDs are assigned by the certificate manager upon
|
|
upload of the trusted certificates. Multiple IDs are allowed here to provide
|
|
flexibility for the user. It may not be feasible for the user to know which
|
|
specific certificate corresponds to their image. Allowing the user to define a
|
|
set of trusted certificates removes the need for an image/certificate mapping,
|
|
simplifying the user experience. When provided, nova will pass these values to
|
|
the certificate validation routine in cursive, overriding the default list of
|
|
trusted certificate IDs (see the second configuration option above). Cursive
|
|
will use them to fetch the trusted certificates using castellan. This
|
|
parameter is optional and is ignored by nova when booting a non-signed image
|
|
or when certificate validation is not enabled. If provided, the value of the
|
|
parameter is saved and will persist for the lifetime of the server data.
|
|
|
|
The third change adds a parameter, trusted_certificates, to the server
|
|
rebuild command of the nova API. This parameter is identical to the parameter
|
|
described in the first change above. This parameter is optional and is
|
|
ignored by nova when rebuilding a server with a non-signed image or when
|
|
certificate validation is not enabled. If provided, the value of the parameter
|
|
is saved and will persist for the lifetime of the server data. If the
|
|
parameter is not provided when rebuilding a server with a signed image, the
|
|
prior set of trusted certificate IDs associated with the server will be
|
|
preserved.
|
|
|
|
The fourth change adds a certificate verification context to conduct
|
|
certificate validation. This work will live alongside the signature
|
|
verification routine in the new cursive certificate_utils module. The
|
|
verification context stores an arbitrary set of certificates and uses them to
|
|
validate individual certificates submitted for certificate validation. The
|
|
context attempts to build a certificate chain for submitted certificates,
|
|
cryptographically verifying that each parent certificate in the chain has
|
|
signed its child certificate. pyca/cryptography is used to conduct all
|
|
certificate and cryptographic operations here. The context also checks various
|
|
constraints to determine if a certificate is valid, including valid timestamp
|
|
checks. If the context cannot build a valid certificate chain for a submitted
|
|
certificate, or if any of the certificate constraint checks fail, the
|
|
submitted certificate fails validation.
|
|
|
|
The fifth change integrates the certificate verification routine into the
|
|
signature verification workflow. When the certificate verification routine
|
|
fetches the image's signing certificate, it builds the verification context
|
|
using the trusted certificates provided by the user (see the first and second
|
|
changes above). It then passes the signing certificate through the context
|
|
for certificate validation. If validation succeeds, signature verification
|
|
can proceed as normal. If validation fails, signature verification fails as
|
|
well, the server is placed in an ERROR state and a fault is returned to the
|
|
user.
|
|
|
|
The sixth change updates the nova data model to support certificate
|
|
validation during server operations, like server evacuation and cold or live
|
|
migrations. More generally, this applies to any operation that may be done by
|
|
an admin against the server without all the information the end user
|
|
originally supplied to the nova boot command. To support these cases, the
|
|
trusted certificate IDs used for certificate validation must be stored with
|
|
the instance data, since the user cannot be expected to provide them. The
|
|
InstanceExtra functionality in nova already supports keypairs associated with
|
|
instances. This change will update the InstanceExtra schema to support a
|
|
trusted_certificate_ids column, which will contain the list of trusted
|
|
certificate IDs. The underlying storage will leverage oslo versionedobjects,
|
|
requiring a new trusted_certificate_id module in nova/objects.
|
|
|
|
The seventh change updates the novaclient/openstackclient to support the
|
|
trusted_certificates parameter for the server create/rebuild commands. This
|
|
includes support for a new environment variable, OS_TRUSTED_CERTIFICATE_IDS,
|
|
that can be used to define a comma-delimited list of trusted certificate IDs.
|
|
If the trusted_certificates parameter is not used, the client will pull the
|
|
value of the environment variable and use it instead. This value will be
|
|
converted into a list before being passed on.
|
|
|
|
If the user does not provide a value for the trusted_certificates parameter,
|
|
either explicitly or through the OS_TRUSTED_CERTIFICATE_IDS environment
|
|
variable, nova will pull the list of trusted certificate IDs from the
|
|
default_trusted_certificate_ids configuration option. If this option is left
|
|
as an empty list, there is no way for nova to obtain a trusted certificate for
|
|
certificate validation. In this case there would be no way to determine if
|
|
the image's signing certificate is trusted so signature verification would
|
|
fail, in turn failing server creation.
|
|
|
|
Alternatives
|
|
------------
|
|
|
|
An alternative approach to certificate validation here would be to support
|
|
certificate trust stores, collections of trusted certificates associated with
|
|
individual users or projects. When creating a new server, the user would
|
|
specify their trust store as a source of trusted certificates, replacing the
|
|
list of certificates provided in the trusted_certificates parameter. There
|
|
are many ways to support trust stores, including: a filesystem directory
|
|
trust store containing trusted certificate files stored locally on the
|
|
compute host, a metadata/managed resource approach supported under services
|
|
like nova or keystone, and a container-based secret storage approach
|
|
supported by services like barbican. While useful in defining collections of
|
|
trusted certificates, a trust store approach would need to scale for large
|
|
cloud deployments which may be difficult from a management and maintenance
|
|
perspective. Trust stores also introduce a new construct that must be
|
|
trusted by the user, especially if the user is not directly responsible for
|
|
maintaining their trust store. These restrictions may not be feasible for
|
|
some cloud deployments.
|
|
|
|
An alternative to the user providing trusted certificates, or storing trusted
|
|
certificates in a trust store, would be to dynamically fetch certificates
|
|
using information stored in the Private Internet Extension of the signed
|
|
certificate being validated. This approach allows deployers and users to use
|
|
signing certificates without needing to pre-fetch all of the root and
|
|
intermediate certificates required to complete the certificate validation
|
|
process. However, this approach requires the compute service have persistent
|
|
network access to all possible certificate repositories where root and
|
|
intermediate certificates may be stored. In many cases, this will include
|
|
network access to the public Internet which may not be feasible for a generic
|
|
deployment.
|
|
|
|
An enhanced certificate validation routine would include certificate
|
|
revocation, supporting commonly used approaches like certificate revocation
|
|
lists (CRLs) and/or the Online Certificate Status Protocol (OCSP). Supporting
|
|
certificate revocation would allow the compute service to dynamically
|
|
determine when certificates become invalid in real time due to compromise,
|
|
further improving the security of booting signed images. However, supporting
|
|
certificate revocation involves dynamically fetching and trusting network
|
|
resources, often under the control and authority of third-parties. This may
|
|
not be feasible for some deployments. It is possible that certificate
|
|
revocation could be integrated outside of the compute service, for example
|
|
within the certificate manager or through another third-party service. This
|
|
would grant nova the benefits of timely revocation without complicating the
|
|
signature verification and certificate validation features in nova itself.
|
|
|
|
It should be noted here that support for certificate revocation is intended
|
|
to be added in future work for this feature.
|
|
|
|
Data model impact
|
|
-----------------
|
|
|
|
The InstanceExtra database model will be updated to include a new text
|
|
column, trusted_certificate_ids, which will contain the list of trusted
|
|
certificate IDs provided with server create/rebuild requests. As stated above,
|
|
if the IDs are not included with the server request, they will be pulled
|
|
from the default_trusted_certificate_ids configuration option. Like the
|
|
existing fields in InstanceExtra, this addition will leverage oslo
|
|
versionedobjects for storing the list, requiring the addition of a
|
|
nova/objects/trusted_certificate_id module defining the necessary objects.
|
|
|
|
REST API impact
|
|
---------------
|
|
|
|
The following are example requests to (1) create a new server from a signed
|
|
image and (2) rebuild a server from a signed image, including the new
|
|
trusted_certificates parameter. This update will be done under a new API
|
|
microversion.
|
|
|
|
.. code-block:: javascript
|
|
|
|
{
|
|
"server": {
|
|
"name": "example-name",
|
|
"imageRef": "70a599e0-31e7-49b7-b260-868f441e862b",
|
|
"flavorRef": "http://openstack.example.com/flavors/1",
|
|
"trusted_certificates": [
|
|
"00000000-0000-0000-0000-000000000000",
|
|
"11111111-1111-1111-1111-111111111111",
|
|
"22222222-2222-2222-2222-222222222222"
|
|
],
|
|
"metadata": {
|
|
"My Server Name": "Example Signed Server"
|
|
}
|
|
}
|
|
}
|
|
|
|
.. code-block:: javascript
|
|
|
|
{
|
|
"rebuild": {
|
|
"name": "example-name",
|
|
"imageRef": "70a599e0-31e7-49b7-b260-868f441e862b",
|
|
"trusted_certificates": [
|
|
"00000000-0000-0000-0000-000000000000",
|
|
"11111111-1111-1111-1111-111111111111",
|
|
"22222222-2222-2222-2222-222222222222"
|
|
],
|
|
"metadata": {
|
|
"My Server Name": "Example Signed Server"
|
|
}
|
|
}
|
|
}
|
|
|
|
Note that while in these examples the values in trusted_certificates are
|
|
UUIDs they are not guaranteed to be so. Certificate managers use different
|
|
ID allocation schemes; while some use strict UUIDs, others use simple
|
|
incrementing integers or raw hex strings. For this feature, the type of
|
|
trusted_certificates will be an array containing zero or more JSON string
|
|
values.
|
|
|
|
The following is a JSON schema description of the trusted_certificates
|
|
parameter:
|
|
|
|
.. code-block:: javascript
|
|
|
|
{
|
|
"type": "array",
|
|
"minItems": 0,
|
|
"maxItems": 50,
|
|
"uniqueItems": true,
|
|
"items": {
|
|
"type": "string"
|
|
}
|
|
}
|
|
|
|
Note the upper and lower bounds for the number of certificate IDs included
|
|
in the trusted_certificates parameter. If an API call is made for a signed
|
|
image and exceeds the maximum number of allowed certificate IDs, then the
|
|
API call will fail.
|
|
|
|
Security impact
|
|
---------------
|
|
|
|
With the added verification step provided by this feature when enabled, the
|
|
security of the signed image verification feature is improved.
|
|
|
|
Notifications impact
|
|
--------------------
|
|
|
|
None
|
|
|
|
Other end user impact
|
|
---------------------
|
|
|
|
This change imposes additional restrictions on the certificates that can be
|
|
used to sign images, and may cause migration challenges if used with images
|
|
signed before the feature is enabled.
|
|
|
|
Migration will require users to upload their trusted certificates to the
|
|
certificate manager if they intend to specify them with the create or rebuild
|
|
request. All image signing certificates must already be in the certificate
|
|
manager to support signature verification.
|
|
|
|
With support being added for the OS_TRUSTED_CERTIFICATE_IDS environment
|
|
variable, users are encouraged to set the variable with the list of trusted
|
|
certificate IDs through their openrc file, alongside their authentication
|
|
credentials. The value of the OS_TRUSTED_CERTIFICATE_IDS environment variable
|
|
is a comma-delimited string of trusted certificate IDs, which will be
|
|
converted into a list of certificate IDs for the trusted_certificates
|
|
parameter.
|
|
|
|
An example openrc file is shown below, using the same trusted certificate IDs
|
|
as those used in the API example (see REST API Impact above):
|
|
|
|
.. code-block:: javascript
|
|
|
|
export OS_USERNAME=username
|
|
export OS_PASSWORD=password
|
|
export OS_TENANT_NAME=projectName
|
|
export OS_AUTH_URL=https://identityHost:portNumber/v2.0
|
|
export OS_TRUSTED_CERTIFICATE_IDS=00000000-0000-0000-0000-000000000000,111111
|
|
11-1111-1111-1111-111111111111,22222222-2222-2222-2222-222222222222
|
|
|
|
Note that in this example, the second certificate ID is split to satisfy line
|
|
wrap formatting for this spec. No explicit linebreaks should be used in the
|
|
actual openrc file.
|
|
|
|
Performance Impact
|
|
------------------
|
|
|
|
Nova will load the user's trusted certificates via cursive every time
|
|
signature verification is performed. Depending upon the size and number of
|
|
certificates, and the frequency of signature verification, this could
|
|
introduce a performance burden on the compute service. To alleviate this, see
|
|
Alternatives above regarding a persistent certificate trust store and
|
|
dynamically loading certificates from remote storage.
|
|
|
|
Other deployer impact
|
|
---------------------
|
|
|
|
The inclusion of two new configuration options, enable_certificate_validation
|
|
and default_trusted_certificate_ids, will smooth the transition for
|
|
deployments looking to enable this feature. If these options are enabled, all
|
|
prior usage of the server create/rebuild API when booting signed images will
|
|
now fail if trusted certificates cannot be located.
|
|
|
|
Developer impact
|
|
----------------
|
|
|
|
None
|
|
|
|
|
|
Implementation
|
|
==============
|
|
|
|
Assignee(s)
|
|
-----------
|
|
|
|
Primary assignee:
|
|
Peter Hamilton
|
|
|
|
Work Items
|
|
----------
|
|
|
|
* Add two new configuration options, enable_certificate_validation and
|
|
default_trusted_certificate_ids. The first will enable the use of
|
|
certificate validation if signature verification is enabled. The second will
|
|
provide a default list of trusted certificate IDs that can be used if no
|
|
trusted certificate IDs are provided with the server request.
|
|
* Update cursive to support certificate validation. This includes the addition
|
|
of the certificate verification context class and the verify_certificate
|
|
routine which loads certificates from the certificate manager and uses the
|
|
certificate verification context to conduct certificate validation.
|
|
* Update the existing signature verification workflow in nova to incorporate
|
|
certificate validation, using the verify_certificate routine in cursive to
|
|
validate the signing certificate.
|
|
* Update the InstanceExtra database model to include a new text column,
|
|
trusted_certificate_ids. Database migrations will be included to add/remove
|
|
this column when updating/downgrading the database schema.
|
|
* Add a new nova API parameter, trusted_certificates, to the server create
|
|
and rebuild commands. The value of this parameter will need to be passed
|
|
through to the signature verification step when downloading the image from
|
|
glance.
|
|
* Update novaclient to support the trusted_certificates parameter.
|
|
* Update novaclient to pull the value of the OS_TRUSTED_CERTIFICATE_IDS
|
|
environment variable when the trusted_certificates parameter is not provided
|
|
by the user.
|
|
* Update openstackclient to support the trusted_certificates parameter.
|
|
* Update openstackclient to pull the value of the OS_TRUSTED_CERTIFICATE_IDS
|
|
environment variable when the trusted_certificates parameter is not provided
|
|
by the user.
|
|
|
|
|
|
Dependencies
|
|
============
|
|
|
|
This work is dependent on the creation and deployment of a
|
|
gate-tempest-dsvm-security-ubuntu-xenial job which runs tempest with signed
|
|
images and barbican as the certificate manager. For more information on this
|
|
work, see the corresponding tempest blueprint [6].
|
|
|
|
|
|
Testing
|
|
=======
|
|
|
|
Unit tests will be included to test the functionality implemented in nova,
|
|
novaclient, and openstackclient. Tempest tests will also be implemented to
|
|
test the end-to-end feature across glance and nova.
|
|
|
|
|
|
Documentation Impact
|
|
====================
|
|
|
|
Documentation on the trusted_certificates API parameter and the two new
|
|
configuration options will need to be added, as will instructions defining
|
|
the OS_TRUSTED_CERTIFICATE_IDS environment variable and its usage.
|
|
|
|
|
|
References
|
|
==========
|
|
[1] "Nova Signature Verification." Online: http://specs.openstack.org/openstack/nova-specs/specs/mitaka/implemented/image-verification.html
|
|
|
|
[2] "Cursive." Online: https://launchpad.net/cursive
|
|
|
|
[3] "Cleanup of signature_utils code." Online: https://blueprints.launchpad.net/nova/+spec/signature-code-cleanup
|
|
|
|
[4] "Use cursive for signature verification." Online: https://review.openstack.org/#/c/351232/
|
|
|
|
[5] "Extend Extras Functionality." Online: https://review.openstack.org/#/c/343939/
|
|
|
|
[6] "Create experimental gate job to test Nova's image signature verification." Online: https://blueprints.launchpad.net/tempest/+spec/image-signing-experimental-gate
|
|
|
|
[7] "Options for using trusted certificates in Nova image signature verification." Online: http://lists.openstack.org/pipermail/openstack-dev/2016-October/105454.html
|
|
|
|
[8] "pyca/cryptography." Online: https://github.com/pyca/cryptography
|
|
|
|
|
|
History
|
|
=======
|
|
|
|
This specification has received extensive review from the OpenStack community
|
|
given that it involves security features in nova. The following is a brief
|
|
timeline of this proposal's history, with major changes documented below
|
|
during each development cycle.
|
|
|
|
.. list-table:: Revisions
|
|
:header-rows: 1
|
|
|
|
* - Release Name
|
|
- Description
|
|
* - Newton
|
|
- Rough draft published
|
|
* - Ocata
|
|
- Introduced for official review
|
|
* - Pike
|
|
- Re-proposed for official review
|
|
|
|
Newton
|
|
------
|
|
|
|
The initial version of this spec was released towards the end of the Newton
|
|
development cycle in preparation for Ocata, focusing on a certificate trust
|
|
store implementation rooted on the compute host filesystem and managed by the
|
|
cloud administrator. Versions 2, 3, and 4 involved minor formatting and
|
|
grammatical updates.
|
|
|
|
Version 5 received feedback from the nova core team, focusing specifically
|
|
on (1) the need for tighter integration between trusted certificate
|
|
management and tenant users, and (2) the potential scalability issues with
|
|
distributed certificate file management across large clouds. Further feedback
|
|
was also solicited from the community through a post to the openstack-dev
|
|
mailing list [7].
|
|
|
|
Version 6 updated the proposed approach, preserving the filesystem-based
|
|
certificate trust store while adding an update to the nova API. This API
|
|
change allowed users to specify the trusted certificate ID when creating new
|
|
instances.
|
|
|
|
Ocata
|
|
-----
|
|
|
|
Version 7 incorporated feedback received from the Ocata Design Summit,
|
|
officially removing the filesystem-based certificate trust store approach
|
|
and focusing solely on updating the nova API to allow the user to submit
|
|
a set of trusted certificate IDs when creating new instances.
|
|
|
|
Version 8 addressed further feedback from the nova core team, including:
|
|
|
|
* highlighting a dependency on barbican as the only supported castellan
|
|
backend
|
|
* updating the nova API changes to include the rebuild operation
|
|
* updating the nova data model to support storing the set of trusted
|
|
certificate IDs with the instance data in instance_extras, thereby
|
|
supporting automatic operations like instance evacuation and cold/live
|
|
migrations
|
|
|
|
Pike
|
|
----
|
|
|
|
Version 9 duplicated Version 8 as a clean slate for the Pike review process.
|
|
Version 10 addressed minor whitespace and spec formatting errors.
|
|
|
|
Version 11 added the new History section from the Pike spec template and
|
|
incorporated feedback received on Version 9, clarifying API details and
|
|
reordering the Security, Other end user, and Other deployer impact sections.
|
|
|
|
Version 12 addressed further reviewer feedback, clarifying nova handling of
|
|
the API changes and resolving discrepancies in spec details.
|
|
|
|
Version 13 updated the spec to reflect the integration of cursive into nova,
|
|
moving the certificate validation code to cursive.
|
|
|
|
Version 14 added support for two new configuration options to smooth the
|
|
transition to use for certificate validation, in addition to clarifying the
|
|
use of oslo versionedobjects for the modification to InstanceExtra.
|