Commit Graph

170 Commits

Author SHA1 Message Date
Sergiy Markin 5470976d4a Airflow stable 2.8.1
Change-Id: I5e78501487e0e1cd3ec1be657f96b80ce951637c
2024-02-23 19:57:08 +00:00
Phil Sphicas c7b903a6ea Cache fernet keys when encrypting or decrypting
The PEGLEG_PASSPHRASE and PEGLEG_SALT environment variables are used to
generate the fernet key for encryption and decryption of site documents,
and the global passphrase and salt are used to generate the fernet key
for global documents.

This change caches the resulting fernet keys to avoid recalculating them
for each document that needs to be encrypted or decrypted, resulting in
a small time savings.

Change-Id: I7b7e77a4740e7abb54efce2fcb3cca6d84a9d7d4
2021-01-19 15:26:11 +00:00
Alexander Hughes 452734fb72 Making certificate check more robust
Some operators may have externally managed certificates that do not
conform to the originally expected certificate pattern of

---BEGIN CERTIFICATE-----
foo
-----END CERTIFICATE-----

and may instead include additional information on the subject/issuer. In
these cases the current regex will fail to load certs that can be parsed
with the existing cfssl command. Addressing this by tightening up the
regex prior to trying to obtain certificate information.

Change-Id: Ief9993632718caa46b52761b49a97621f134ca53
Signed-off-by: Alexander Hughes <Alexander.Hughes@pm.me>
2020-12-29 16:16:57 -05:00
Phil Sphicas 0524e01779 Accelerate YAML operations with LibYAML
Patch PyYAML (via the pylibyaml library) to automatically enable the
LibYAML parser and emitter, which are faster than the Python versions.

https://pypi.org/project/pylibyaml/

Change-Id: Ic48d2234ca3107404d9f883ca6038a12ca06a408
2020-09-25 01:01:20 +00:00
Phil Sphicas 88281bff56 Fix bug with block_literal_representer
When serializing a block literal, be explicit that we want to treat it
as a string, instead of relying on implicit conversion.

Change-Id: I01a06c49f6112ef3e4313030ada0a7ea6adb5fb4
2020-09-22 04:53:31 +00:00
francisy 7997ace493 Ensure Tag Name is a string in deployment data
Tag into a string instead of an array

Change-Id: I5cec243fe957a573cf70eea53bfb861a7b1d5888
2020-07-06 17:58:27 -04:00
francisy b5a58659d8 Add global revision to deployment status
Updated basedeployment_data to have version from site-definition.yaml
and implemented exception if no global revision existed.

Change-Id: I14caf8244a7d36201e9cc4ebc90e310d594743ec
2020-06-30 18:30:53 +00:00
Apurva Gokani 22e6df0870 adding site_type to deployment-data schema
This change adds the site_type parameter in deployment_data
The value will be taken from respective site-definition.yaml

Change-Id: I8e65b39c73c94caf3ed4cc517520b9577160b20d
2020-06-26 12:35:22 -05:00
Alexander Hughes cb3046ee53 Fail early on invalid save locations
Currently the Render command expects either no save option or a file
path. No save option defaults to stdout, a file path writes to the
specified file.

If a directory is specified no error is thrown until rendering is
complete, by performing save location validation early Pegleg will now
fail early.

Change-Id: If75655e240c0ecbda00ea591e948e71010b4521d
Signed-off-by: Alexander Hughes <Alexander.Hughes@pm.me>
2020-05-04 12:35:53 -04:00
Alexander Hughes e4ff07c793 Enable rendering without decrypting
This patchset aims to address least privileged concerns, namely that
Pegleg's current behavior is to require decryption of all site
documents prior to rendering. Failure to do so leads to a duplicate
document error.

Operators of Pegleg may not have a valid reason to access secrets
that are not being modified during their current workflow, their
work may be limited to non-secrets but need to test their changes by
rendering the site manifests.

To enable this, the get_rendered_documents function has been updated
such that if a document is encrypted, the secret value will be
converted to a string to pass schema validation, and then used for
rendering. This will allow operators of Pegleg to render documents
without decrypting secrets. Instead the encrypted string value
of the secret will be used.

Change-Id: I8656b5496e2225e6eb59727c4f79326a1406147c
2020-05-01 20:12:59 +00:00
Phil Sphicas 417975b596 Uniquely name managed documents
When pegleg wraps documents, it uses the original document name as the
name of the managed document. This often results in duplicate documents
(i.e. identical in name and schema). For example, it is expected to have
identically named deckhand document pairs: Certificate & CertificateKey;
CertificateAuthority & CertificateAuthorityKey; PublicKey & PrivateKey.

However, this could also occur for unrelated document types that happen
to have the same name, and generally defeats the principle that each
document is identified by a schema top-level key and the metadata.name.

This change uses a combination of the original document schema and name
to build the name of the pegleg/PeglegManagedDocument/v1.

Change-Id: Iab186ae7e9d24a30cb413be89a17fad960e10bea
2020-04-20 18:45:55 +00:00
Andrii Ostapenko 572ce3f6eb Return -o (--output*) option back as deprecated.
Deprecating cli options before removing them is a standard courtesy that
allows project consumers to handle such changes according to their
needs.

Option is removed here: https://review.opendev.org/#/c/701274

Change-Id: Iba4a4574426315a76294269d4287fd3ea18aa99e
2020-01-28 19:53:53 -06:00
Ian H. Pittwood fe592312aa Change wrap command to add YAML section delimiter
Change-Id: Ic2f57e14c1c71d49bbd92a241dec9ed3163f6195
2020-01-21 13:24:12 -06:00
Ryan Schroder 4cbc708c1d Standardize save location
Changes to make --save-location standard for all commands

Change-Id: I86a2da01ad1cc1c4d708131b5161182e16b5bb5e
2020-01-15 16:42:20 -06:00
Zuul 3e1859010b Merge "Check cert expiry for multiple types" 2020-01-15 19:16:57 +00:00
Zuul 9d152ae73b Merge "Default site commands to --no-decrypt" 2020-01-15 16:42:07 +00:00
Ian H. Pittwood 1ef5024eb0 Accept multiple path options for decrypt
Change-Id: Ic6c337334f1a21e1f3e303f4ccfb56caeddf4500
2020-01-14 11:42:28 -06:00
Ian H. Pittwood 69603045a5 Default site commands to --no-decrypt
The default of pre-command decrypts (--decrypt) being enabled is causing
some non-intuitive behavior in what files are being overwritten by the
decrypt action. This change will disable pre-command decrypt by default
(--no-decrypt) until pre-command decrypt is updated.

Change-Id: Iadad0f211fbc549f0f998266666e8ad784e6b8b1
2020-01-10 12:53:04 -06:00
Alexander Hughes c6e34b47ca Check cert expiry for multiple types
This patch adds support for:
- Checking expiration of CAs in manifests
- Multiple certs per data field of a YAML document

Change-Id: I9dae69acb4252d4de4469eb6733b533ef479f7b4
2020-01-08 19:05:52 +00:00
Ian H Pittwood ff9c95f423 Add path option to encrypt command
Adds a path option to the encrypt command to specify what directory of
file to encrypt. If path is not specified, all applicable files in the
repositories will be encrypted (this is the current behavior).

Change-Id: Idd5e063a54cf157a8ec761de85cbd67edd05364c
2020-01-07 15:40:05 -06:00
Ian H Pittwood 3a6e3d7cce Add pre-command decrypt option
Adds an option on all site commands to decrypt site files before
executing the command's actions. This option will be enabled by default.
If the command clones repository contents to a temporary directory, only
that temporary data will be decrypted.

Change-Id: Ic10c7196592c6d0e1c69a85b265259357ac28169
2019-12-19 14:52:27 -06:00
Ian H. Pittwood fff70ad861 Refactors pegleg CLI to use single commands
Debugging pegleg can currently be difficult and the Click CLI does not
easily allow debuggers like pdb or PyCharm to use breakpoints. By moving
all CLI command calls into singular functions, we can easily create an
"if __name__ == '__main__'" entry point to call these functions and
investigate any bugs that may arise.

We also gain the ability to reuse more portions of our code by
refactoring these methods.

Change-Id: Ia9739931273eb6458f82dbb7e702a505ae397ae3
2019-12-17 19:32:28 +00:00
Zuul ae5db00f83 Merge "Improve check-pki-certs output" 2019-12-11 15:25:35 +00:00
Ian H. Pittwood 9163ef08ca Add passphrase catalog override option
Adds an option to specify a passphrase catalog to override catalogs
discovered in the site repository. This allows the generation of a
specified subset of passphrases instead of the entire site's catalog.

Change-Id: I797107234292eea8ca788b7a94ed5e2c90566bf5
2019-12-10 20:40:31 +00:00
Ian H. Pittwood a97c9cdba2 Improve check-pki-certs output
Adds alternate message when no certificates are expiring

Exit code will now be 1 if there are certificates expiring and 0 if no
certificates are expiring

Change-Id: I94a7a5af0c5469b83001b5439f18691140de6245
2019-12-10 13:40:07 -06:00
Ian H. Pittwood 87d24d530a Add cert name to check-pki-certs command
Adds a table column for the certificate name. Each row of data will now
be printed with the file name, certificate name, and certificate
expiration date.

Change-Id: I088ba2e794f33cd858f36275d00dd431862f1c25
2019-12-10 15:20:27 +00:00
Raven Daugherty 158768278e Added YAML file linting
Change-Id: I17208c96610ee13772c44b7d765d3014f0023134
2019-12-09 16:43:45 +00:00
Alexander Hughes 1a1c31b5a7 Add generate-pki command as deprecated
In [0] the secrets generate-pki command was moved to secrets
generate certificates. While release notes were added, this change
impacts automation set up for users of Pegleg. This change adds
back the generate-pki command but marks it as deprecated.

[0] https://review.opendev.org/#/c/694810/

Change-Id: I6a3841e5f5313511ec2afd8340bcae5857cd81fa
2019-12-06 20:22:54 +00:00
Ian H Pittwood 2e0e9eab8c Add save location option to generate certs
Allows users to specify a save location for newly generated certificates
instead of always writing them to the site repository. The functionality
is identical to generate passphrase's save_location option.

Change-Id: I8625fba75160c441dbf3f345af99eb0733b2c37d
2019-12-03 22:14:44 +00:00
Ian H. Pittwood 7306c3af9b Move generate-pki under generate sub-command
Change-Id: I83b078d27eb2c07c3be7eb6543de85ce32fb2a6e
2019-11-20 19:16:12 +00:00
Alexander Hughes 85fdcd497a Revert "Update storage policy on decrypt"
Barbican is being enabled, as such the metadata field should not be
modified by Pegleg.  If it says encrypted, then Barbican will
encrypt.
If it says cleartext, Barbican won't.  All pegleg needs to do is
decrypt the document prior to bundling it which exists already
without this change.

This reverts commit 2d88f48989.

Change-Id: I8900f910f9816508a8ec5c23932252bb9d1fde09
2019-10-14 12:08:53 +00:00
Alexander Hughes f56d20a2da bugfix for global encrypt/decrypt
This patch updates layer for wrapped documents to preserve original
layer.  Previously all encrypted documents had site layer.

Update encrypt/decrypt logic when determining global keys.

Update units tests.

Change-Id: I447aeaea08a4514655fcabfc7077b6d4b282e27f
2019-10-02 19:13:33 +00:00
Ian H. Pittwood 5ef28bf804 Change verbose option to granular verbosity
Pegleg's current verbose option simply sets the logging level to either
DEBUG or ERROR. This change allows users to enter a specific logging
level anywhere between DEBUG and CRITICAL. The default logging level is
set to 40=ERROR.

The original verbose option will be kept in order to preserve backwards
compatibility on existing scripts.

Change-Id: I2cb81c55ab070380c4336ab8d75a9bf1c18b95fc
2019-09-25 08:32:41 -05:00
Alexander Hughes 14f8600e37 Add profiles to passphrase catalog
Change-Id: Id6e7cddd123e31f0df963167ddf3fa8f33e9060c
2019-09-24 16:19:36 +00:00
Zuul 76b14da0ab Merge "Add deployment_data to rendered docs" 2019-09-18 14:19:10 +00:00
Zuul a1735882b1 Merge "Handle Pegleg-generated commits in deployment data" 2019-09-17 12:51:26 +00:00
Alexander Hughes bc6554241b Add deployment_data to rendered docs
A bug was found where the deployment_data document was not being
included in the upload to shipyard. Upon investigation it was also
noticed that deployment_data wasn't being rendered.

This patch updates the render and upload commands to include the
deployment data document.

Change-Id: I916132e80ac13546468f171a75517a9159e48ea6
2019-09-17 12:39:33 +00:00
Ian H. Pittwood e2dad75a99 Default to non-interactive passphrase generation
This change disables and skips input prompts for generate passphrases.
Using the -i option will now only enable prompts for passphrases that
are set to prompt=True.

Change-Id: Ia932305891259d9d1430e1d184dbf39892d4a5d3
2019-09-16 13:13:04 -05:00
Alexander Hughes 2d88f48989 Update storage policy on decrypt
Barbican expects storagePolicy:cleartext when accepting documents
from Pegleg. Once the document is decrypted updated storagePolicy to
cleartext.

During genesis bundle creation, the encrypted documents are
decrypted, then the whole bundle is encrypted.  Once the bundle gets
deployed the contained documents should all have the correct storage
policy of cleartext.

As a result, two unit tests are updated to no longer compare the
storagePolicy, as it is expected everything going to Barbican is
cleartext, but in order for Pegleg to know to encrypt the original
document is storagePolicy:encrypted

Change-Id: I5167ce6b3030d143d1ff0b789883529a6557eeca
2019-09-12 13:44:29 -05:00
Carter, Matt (mc981n) 55d0961410 Handle Pegleg-generated commits in deployment data
It is possible for Pegleg to generate a commit on top of a repo if
the repo is dirty (aka, has uncommited/untracked files). This
effectively makes the repo appear "clean", and also changes the head
of the repo. This can potentially interfere with the deployment_data
generation that analyzes the cleanliness of the repo as well as the
commit at the head of the repo.

This patch set updates the deployment_data generation logic, to be
able to detect Pegleg-generated commits at the head of a repo, and
instead go off of the Pegleg-generated commit's parent commit when
generating the data. It also ensures the repo in the data is always
marked dirty if a Pegleg-generated commit is seen, because the
Pegleg-generated commit would not exist unless the repo was dirty.

Change-Id: I863b3f2f661f11c36ba939ee3023f78733021b96
2019-09-12 13:38:54 -05:00
Alexander Noskov 4f6df0b9fd Fix small typo
Change-Id: Ie478b742e6752d9ae6f9d3882ba01ef613524a31
2019-08-15 14:11:37 -05:00
Zuul 567a134e01 Merge "Support regenerating PKI" 2019-08-06 18:45:24 +00:00
Alexander Hughes 7018d5941c Support regenerating PKI
This patch adds functionality Pegleg currently lacks: the ability to
regenerate expired certificates.

This patch adds:
1. CLI toggle --regenerate-all to generate_pki.  Default is False,
   which means if no certificates are present, generate what is in
   the pki catalogue. If new certs have been added to the catalogue
   generate just those.  If the --regenerate-all flag is True, then
   Pegleg will ignore any existing certs and regenerate (or generate
   for the first time) all certificates defined in the PKI catalogue.
2. Documentation updates for CLI change.
3. Updates to pki_utility to accomodate the new flag.
4. Updates pki_generator methods to use rendered documents to
   accommodate documents that have to be layered.
5. Updates pki_generator unit tests to include a layering definition
   which is now required to run the commands.

Change-Id: I2d8086770e9226e44598ef40eca790981279f626
2019-08-06 17:15:02 +00:00
Zuul ca7bd712a6 Merge "Fix: Allow Pegleg to generate unencrypted bundle" 2019-08-06 14:49:05 +00:00
Svetlana Shturm 4d8336d639 Add support domain-scoped token for CLI
Change-Id: Iac9556a20ccf2c36537836b0693f5384c7c1d181
2019-08-06 13:45:34 +00:00
Alexander Hughes 55349d078e Fix: Allow Pegleg to generate unencrypted bundle
Promenade supports generating a bundle with or without encryption
Pegleg currently does not, it requires encryption.

This patch changes Pegleg behavior to look for an encryption method
and key before generating a bundle.
1. If both are present - generate with encryption
2. If neither are present - generate without encryption
3. If one or the other is present - raise an error

Change-Id: Id55e310ced3d650f64d3f6f626faa10dbe3ab052
2019-08-06 13:17:55 +00:00
Zuul ccd05998b2 Merge "Disable some passphrase auto generation" 2019-08-05 15:01:44 +00:00
Ian H Pittwood 2966c92396 Disable some passphrase auto generation
This change adds logic to prevent users from auto generating passphrases
when "prompt=True" and "regenerable=False". Users must manually enter a
passphrase under these conditions or a message will be echoed and users
will be asked to enter a passphrase again.

Prevent auto generation under specified conditions

Moves all passphrase validation into static methods

Restructures validation workflow

Change-Id: If858510f9a84df2257e7f232363a57368005bf56
2019-08-02 18:49:15 +00:00
Ian H Pittwood eb6c2574bc Set a fixed order in which data is dumped to YAML files
One of the well-known issues of Python is that dictionaries do not
maintain order in their keys once created. This causes YAML data dumps
to output in a seemingly random order or alphabetically. As these output
files are often kept in their own repositories, they must go through
review or comparison in VCS. If the order of keys is switching for these
files every time Pegleg is ran, it makes it difficult for a user to
compare newly generated files with the old.

To fix this issue, we can change all dictionaries used to template
YAML files into OrderedDict objects. The OrderedDict objects will
maintain order through their dumping to YAML.

Change-Id: I0c1ee3f3f37ed8598d2ba81528d5c61447cbd0d0
2019-08-02 18:33:26 +00:00
HUGHES, ALEXANDER (ah8742) 174e356214 Update Passphrase Catalog to support types
I recently received a request to add additional features to Pegleg's
generate passphrases command.  The desire was to support multiple
types of secrets:
1. passphrases (24+ characters, including characters from upper,
   lower, number, symbol).
2. base64 encoded passphrases.
3. UUID4.

As well as adding an additional flag to prevent Pegleg from
regenerating specific passphrases that are sensitive to rotation.

Finally, responding to an enhancement request interactive
passphrase generation can now be specified via the command line for
all passphrases, or by specifying 'prompt': True for specific
passphrases in passphrase-catalog.yaml

These objectives were completed by:
1. Updating passphrase_catalog.py to support a type field. If a
   type is not specified, default to existing passphrase generation.
   If an invalid value is specified, raise an exception.
2. Updating passphrase_catalog.py to support a regenerable field. If
   the regenerable field is not specified, default to True. If an
   invalid value is specified, raise an exception. When regenerable
   is determined, secrets of 'uuid' type always use regenerable=False
   as they should be one time values created at time of deployment
   but not rotated.
3. Updating passphrase_catalog.py to support a prompt field. If the
   prompt field is not specified, default to False. If an invalid
   value is specified, raise an exception.
4. Adding appropriate exceptions.
5. Updating passphrase_generator.py to handle the new type checks,
   UUID will use UUID4, base64 uses the existing logic of generating
   a random passphrase and base64 encoding it, and existing logic
   remains for generating a random passphrase.
6. Updating passphrase_generator.py to handle the regenerable field.
   It checks if a file is present at the expected save path, and if
   regenerable=False. If both are true, the passphrase is skipped so
   the passphrase is not overwritten.
7. Updating unit tests to validate the new type checks.

NOTE: # nosec is used in passphrase_generator.py on the
'if passphrase_type == <special type>' statements. These are not a
security concern, but do cause Bandit error B105.  See documentation
for B105 in [0]

Local testing of the generate passphrase command with the following
passphrase types:
passphrase_b64 : base64
passphrase_uuid : uuid
passphrase_specified : passphrase (specified)
passphrase_defaulted : passphrase (defaulted)

Resulted in the following data for each:
passphrase_b64.yaml:data: !!binary |
  UDI1SGFFZHFlbWhITjBrdGJHZGFWRkp6UlZWdFdVNUQ=
passphrase_uuid.yaml:data: 5ce7c6bc-00d2-4b2c-9222-54891f075656
passphrase_specified.yaml:data: cYTenMYXFHUKn6ppYjx#+Hdx
passphrase_defaulted.yaml:data: 13ryjaM?I@sP#3&YQXuQEik4

[0] https://bandit.readthedocs.io/en/latest/plugins/b105_hardcoded_password_string.html

Change-Id: I389316c5194ffa06f3df5114f7ac5f4f2887b319
2019-08-02 07:30:51 -05:00