The PEGLEG_PASSPHRASE and PEGLEG_SALT environment variables are used to
generate the fernet key for encryption and decryption of site documents,
and the global passphrase and salt are used to generate the fernet key
for global documents.
This change caches the resulting fernet keys to avoid recalculating them
for each document that needs to be encrypted or decrypted, resulting in
a small time savings.
Change-Id: I7b7e77a4740e7abb54efce2fcb3cca6d84a9d7d4
Some operators may have externally managed certificates that do not
conform to the originally expected certificate pattern of
---BEGIN CERTIFICATE-----
foo
-----END CERTIFICATE-----
and may instead include additional information on the subject/issuer. In
these cases the current regex will fail to load certs that can be parsed
with the existing cfssl command. Addressing this by tightening up the
regex prior to trying to obtain certificate information.
Change-Id: Ief9993632718caa46b52761b49a97621f134ca53
Signed-off-by: Alexander Hughes <Alexander.Hughes@pm.me>
Patch PyYAML (via the pylibyaml library) to automatically enable the
LibYAML parser and emitter, which are faster than the Python versions.
https://pypi.org/project/pylibyaml/
Change-Id: Ic48d2234ca3107404d9f883ca6038a12ca06a408
When serializing a block literal, be explicit that we want to treat it
as a string, instead of relying on implicit conversion.
Change-Id: I01a06c49f6112ef3e4313030ada0a7ea6adb5fb4
Updated basedeployment_data to have version from site-definition.yaml
and implemented exception if no global revision existed.
Change-Id: I14caf8244a7d36201e9cc4ebc90e310d594743ec
This change adds the site_type parameter in deployment_data
The value will be taken from respective site-definition.yaml
Change-Id: I8e65b39c73c94caf3ed4cc517520b9577160b20d
Currently the Render command expects either no save option or a file
path. No save option defaults to stdout, a file path writes to the
specified file.
If a directory is specified no error is thrown until rendering is
complete, by performing save location validation early Pegleg will now
fail early.
Change-Id: If75655e240c0ecbda00ea591e948e71010b4521d
Signed-off-by: Alexander Hughes <Alexander.Hughes@pm.me>
This patchset aims to address least privileged concerns, namely that
Pegleg's current behavior is to require decryption of all site
documents prior to rendering. Failure to do so leads to a duplicate
document error.
Operators of Pegleg may not have a valid reason to access secrets
that are not being modified during their current workflow, their
work may be limited to non-secrets but need to test their changes by
rendering the site manifests.
To enable this, the get_rendered_documents function has been updated
such that if a document is encrypted, the secret value will be
converted to a string to pass schema validation, and then used for
rendering. This will allow operators of Pegleg to render documents
without decrypting secrets. Instead the encrypted string value
of the secret will be used.
Change-Id: I8656b5496e2225e6eb59727c4f79326a1406147c
When pegleg wraps documents, it uses the original document name as the
name of the managed document. This often results in duplicate documents
(i.e. identical in name and schema). For example, it is expected to have
identically named deckhand document pairs: Certificate & CertificateKey;
CertificateAuthority & CertificateAuthorityKey; PublicKey & PrivateKey.
However, this could also occur for unrelated document types that happen
to have the same name, and generally defeats the principle that each
document is identified by a schema top-level key and the metadata.name.
This change uses a combination of the original document schema and name
to build the name of the pegleg/PeglegManagedDocument/v1.
Change-Id: Iab186ae7e9d24a30cb413be89a17fad960e10bea
Deprecating cli options before removing them is a standard courtesy that
allows project consumers to handle such changes according to their
needs.
Option is removed here: https://review.opendev.org/#/c/701274
Change-Id: Iba4a4574426315a76294269d4287fd3ea18aa99e
The default of pre-command decrypts (--decrypt) being enabled is causing
some non-intuitive behavior in what files are being overwritten by the
decrypt action. This change will disable pre-command decrypt by default
(--no-decrypt) until pre-command decrypt is updated.
Change-Id: Iadad0f211fbc549f0f998266666e8ad784e6b8b1
This patch adds support for:
- Checking expiration of CAs in manifests
- Multiple certs per data field of a YAML document
Change-Id: I9dae69acb4252d4de4469eb6733b533ef479f7b4
Adds a path option to the encrypt command to specify what directory of
file to encrypt. If path is not specified, all applicable files in the
repositories will be encrypted (this is the current behavior).
Change-Id: Idd5e063a54cf157a8ec761de85cbd67edd05364c
Adds an option on all site commands to decrypt site files before
executing the command's actions. This option will be enabled by default.
If the command clones repository contents to a temporary directory, only
that temporary data will be decrypted.
Change-Id: Ic10c7196592c6d0e1c69a85b265259357ac28169
Debugging pegleg can currently be difficult and the Click CLI does not
easily allow debuggers like pdb or PyCharm to use breakpoints. By moving
all CLI command calls into singular functions, we can easily create an
"if __name__ == '__main__'" entry point to call these functions and
investigate any bugs that may arise.
We also gain the ability to reuse more portions of our code by
refactoring these methods.
Change-Id: Ia9739931273eb6458f82dbb7e702a505ae397ae3
Adds an option to specify a passphrase catalog to override catalogs
discovered in the site repository. This allows the generation of a
specified subset of passphrases instead of the entire site's catalog.
Change-Id: I797107234292eea8ca788b7a94ed5e2c90566bf5
Adds alternate message when no certificates are expiring
Exit code will now be 1 if there are certificates expiring and 0 if no
certificates are expiring
Change-Id: I94a7a5af0c5469b83001b5439f18691140de6245
Adds a table column for the certificate name. Each row of data will now
be printed with the file name, certificate name, and certificate
expiration date.
Change-Id: I088ba2e794f33cd858f36275d00dd431862f1c25
In [0] the secrets generate-pki command was moved to secrets
generate certificates. While release notes were added, this change
impacts automation set up for users of Pegleg. This change adds
back the generate-pki command but marks it as deprecated.
[0] https://review.opendev.org/#/c/694810/
Change-Id: I6a3841e5f5313511ec2afd8340bcae5857cd81fa
Allows users to specify a save location for newly generated certificates
instead of always writing them to the site repository. The functionality
is identical to generate passphrase's save_location option.
Change-Id: I8625fba75160c441dbf3f345af99eb0733b2c37d
Barbican is being enabled, as such the metadata field should not be
modified by Pegleg. If it says encrypted, then Barbican will
encrypt.
If it says cleartext, Barbican won't. All pegleg needs to do is
decrypt the document prior to bundling it which exists already
without this change.
This reverts commit 2d88f48989.
Change-Id: I8900f910f9816508a8ec5c23932252bb9d1fde09
This patch updates layer for wrapped documents to preserve original
layer. Previously all encrypted documents had site layer.
Update encrypt/decrypt logic when determining global keys.
Update units tests.
Change-Id: I447aeaea08a4514655fcabfc7077b6d4b282e27f
Pegleg's current verbose option simply sets the logging level to either
DEBUG or ERROR. This change allows users to enter a specific logging
level anywhere between DEBUG and CRITICAL. The default logging level is
set to 40=ERROR.
The original verbose option will be kept in order to preserve backwards
compatibility on existing scripts.
Change-Id: I2cb81c55ab070380c4336ab8d75a9bf1c18b95fc
A bug was found where the deployment_data document was not being
included in the upload to shipyard. Upon investigation it was also
noticed that deployment_data wasn't being rendered.
This patch updates the render and upload commands to include the
deployment data document.
Change-Id: I916132e80ac13546468f171a75517a9159e48ea6
This change disables and skips input prompts for generate passphrases.
Using the -i option will now only enable prompts for passphrases that
are set to prompt=True.
Change-Id: Ia932305891259d9d1430e1d184dbf39892d4a5d3
Barbican expects storagePolicy:cleartext when accepting documents
from Pegleg. Once the document is decrypted updated storagePolicy to
cleartext.
During genesis bundle creation, the encrypted documents are
decrypted, then the whole bundle is encrypted. Once the bundle gets
deployed the contained documents should all have the correct storage
policy of cleartext.
As a result, two unit tests are updated to no longer compare the
storagePolicy, as it is expected everything going to Barbican is
cleartext, but in order for Pegleg to know to encrypt the original
document is storagePolicy:encrypted
Change-Id: I5167ce6b3030d143d1ff0b789883529a6557eeca
It is possible for Pegleg to generate a commit on top of a repo if
the repo is dirty (aka, has uncommited/untracked files). This
effectively makes the repo appear "clean", and also changes the head
of the repo. This can potentially interfere with the deployment_data
generation that analyzes the cleanliness of the repo as well as the
commit at the head of the repo.
This patch set updates the deployment_data generation logic, to be
able to detect Pegleg-generated commits at the head of a repo, and
instead go off of the Pegleg-generated commit's parent commit when
generating the data. It also ensures the repo in the data is always
marked dirty if a Pegleg-generated commit is seen, because the
Pegleg-generated commit would not exist unless the repo was dirty.
Change-Id: I863b3f2f661f11c36ba939ee3023f78733021b96
This patch adds functionality Pegleg currently lacks: the ability to
regenerate expired certificates.
This patch adds:
1. CLI toggle --regenerate-all to generate_pki. Default is False,
which means if no certificates are present, generate what is in
the pki catalogue. If new certs have been added to the catalogue
generate just those. If the --regenerate-all flag is True, then
Pegleg will ignore any existing certs and regenerate (or generate
for the first time) all certificates defined in the PKI catalogue.
2. Documentation updates for CLI change.
3. Updates to pki_utility to accomodate the new flag.
4. Updates pki_generator methods to use rendered documents to
accommodate documents that have to be layered.
5. Updates pki_generator unit tests to include a layering definition
which is now required to run the commands.
Change-Id: I2d8086770e9226e44598ef40eca790981279f626
Promenade supports generating a bundle with or without encryption
Pegleg currently does not, it requires encryption.
This patch changes Pegleg behavior to look for an encryption method
and key before generating a bundle.
1. If both are present - generate with encryption
2. If neither are present - generate without encryption
3. If one or the other is present - raise an error
Change-Id: Id55e310ced3d650f64d3f6f626faa10dbe3ab052
This change adds logic to prevent users from auto generating passphrases
when "prompt=True" and "regenerable=False". Users must manually enter a
passphrase under these conditions or a message will be echoed and users
will be asked to enter a passphrase again.
Prevent auto generation under specified conditions
Moves all passphrase validation into static methods
Restructures validation workflow
Change-Id: If858510f9a84df2257e7f232363a57368005bf56
One of the well-known issues of Python is that dictionaries do not
maintain order in their keys once created. This causes YAML data dumps
to output in a seemingly random order or alphabetically. As these output
files are often kept in their own repositories, they must go through
review or comparison in VCS. If the order of keys is switching for these
files every time Pegleg is ran, it makes it difficult for a user to
compare newly generated files with the old.
To fix this issue, we can change all dictionaries used to template
YAML files into OrderedDict objects. The OrderedDict objects will
maintain order through their dumping to YAML.
Change-Id: I0c1ee3f3f37ed8598d2ba81528d5c61447cbd0d0
I recently received a request to add additional features to Pegleg's
generate passphrases command. The desire was to support multiple
types of secrets:
1. passphrases (24+ characters, including characters from upper,
lower, number, symbol).
2. base64 encoded passphrases.
3. UUID4.
As well as adding an additional flag to prevent Pegleg from
regenerating specific passphrases that are sensitive to rotation.
Finally, responding to an enhancement request interactive
passphrase generation can now be specified via the command line for
all passphrases, or by specifying 'prompt': True for specific
passphrases in passphrase-catalog.yaml
These objectives were completed by:
1. Updating passphrase_catalog.py to support a type field. If a
type is not specified, default to existing passphrase generation.
If an invalid value is specified, raise an exception.
2. Updating passphrase_catalog.py to support a regenerable field. If
the regenerable field is not specified, default to True. If an
invalid value is specified, raise an exception. When regenerable
is determined, secrets of 'uuid' type always use regenerable=False
as they should be one time values created at time of deployment
but not rotated.
3. Updating passphrase_catalog.py to support a prompt field. If the
prompt field is not specified, default to False. If an invalid
value is specified, raise an exception.
4. Adding appropriate exceptions.
5. Updating passphrase_generator.py to handle the new type checks,
UUID will use UUID4, base64 uses the existing logic of generating
a random passphrase and base64 encoding it, and existing logic
remains for generating a random passphrase.
6. Updating passphrase_generator.py to handle the regenerable field.
It checks if a file is present at the expected save path, and if
regenerable=False. If both are true, the passphrase is skipped so
the passphrase is not overwritten.
7. Updating unit tests to validate the new type checks.
NOTE: # nosec is used in passphrase_generator.py on the
'if passphrase_type == <special type>' statements. These are not a
security concern, but do cause Bandit error B105. See documentation
for B105 in [0]
Local testing of the generate passphrase command with the following
passphrase types:
passphrase_b64 : base64
passphrase_uuid : uuid
passphrase_specified : passphrase (specified)
passphrase_defaulted : passphrase (defaulted)
Resulted in the following data for each:
passphrase_b64.yaml:data: !!binary |
UDI1SGFFZHFlbWhITjBrdGJHZGFWRkp6UlZWdFdVNUQ=
passphrase_uuid.yaml:data: 5ce7c6bc-00d2-4b2c-9222-54891f075656
passphrase_specified.yaml:data: cYTenMYXFHUKn6ppYjx#+Hdx
passphrase_defaulted.yaml:data: 13ryjaM?I@sP#3&YQXuQEik4
[0] https://bandit.readthedocs.io/en/latest/plugins/b105_hardcoded_password_string.html
Change-Id: I389316c5194ffa06f3df5114f7ac5f4f2887b319