Commit Graph

118 Commits

Author SHA1 Message Date
SPEARS, DUSTIN (ds443n) 6f266ef8a4 Bump promenade changes
upgrade etcd to v3.5.11

https: //review.opendev.org/c/airship/promenade/+/907761
Change-Id: I2bc304d7d19306bcb3d5f5a49ab7cd06052ea082
2024-02-09 14:44:03 -05:00
Ruslan Aliev 9fde367ddb Bump promenade dependency
* add full support of operator-based bootstrap

Signed-off-by: Ruslan Aliev <raliev@mirantis.com>
Change-Id: I092b90a58525fbc78293ca5e7ea29c4341ca62c5
2024-01-16 10:35:49 -06:00
Sergiy Markin c052d40277 Pegleg focal upgrade
This PS delivers focal version of Pegleg image and has the following updates:
- removed release-notes-jobs-python3 gate job because of incompartibility with Sphinx from current requirements
- added focal gate node and switched gates to use it
- added bindep.txt file into project root
- added bindep role into gate jobs
- added ubuntu_focal dockerfile for building focal pegleg image
- switched tox profiles to py38
- uplifted references to shipyard_client, promenade and deckhand projects
- resolved required dependencies conflicts by weakening constraints in Pipfile
- updated tox profile update-requirements for generate requirements.txt and test-requirements.txt
- generated new Pipfile.lock, requirements.txt and test-requirements.txt from Pipfile
- switched tox profiles to use requirements.txt and test-requirements.txt instead of pipenv because of upstream zuul nodes Pypi mirrorring issue
- updated reference to seaworthy site certificates in treasuremap repo
- fixed unit tests issues caused by pytest/mock updates and new openssl version
- fixed focal docker image publishing issue
- added multiprocessing into coverage tests running process
- made unit and coveraget tests more verbosive

Change-Id: I5c4c519dc725cfb8c7b4e14756347c9336028aff
2023-05-02 15:05:45 +00:00
Phil Sphicas 1a7e2da9d7 Update treasuremap ref for cert tests
Unit tests are failing for check_pki_certs against airsloop, since those
certs are expiring in less than 60 days.

This change updates the reference to a commit with regenerated certs.

Change-Id: Ib1ea40a37eef14c2df44b8446fdbcfcd0e345a95
2022-03-22 07:49:00 -07:00
Phil Sphicas 2a0519e397 Fix failing check_pki_certs test
Unit tests are failing (as they will annually) for check_pki_certs,
which is run against the airsloop site in the treasuremap repo. Similar
to the change last year [0], this updates the treasuremap ref to the tip
of the v1.9 branch, which includes new certs for airsloop [1], seaworthy
[2], and seaworthy-virt [3] sites.

Change-Id: Ibfcdde11a35ed2207e78c2d2f372ab2d2e1f2e63
0: https://review.opendev.org/c/airship/pegleg/+/729269
1: https://review.opendev.org/c/airship/treasuremap/+/789482
2: https://review.opendev.org/c/airship/treasuremap/+/789483
3: https://review.opendev.org/c/airship/treasuremap/+/789484
2021-05-04 16:26:33 +00:00
Phil Sphicas 0524e01779 Accelerate YAML operations with LibYAML
Patch PyYAML (via the pylibyaml library) to automatically enable the
LibYAML parser and emitter, which are faster than the Python versions.

https://pypi.org/project/pylibyaml/

Change-Id: Ic48d2234ca3107404d9f883ca6038a12ca06a408
2020-09-25 01:01:20 +00:00
francisy b5a58659d8 Add global revision to deployment status
Updated basedeployment_data to have version from site-definition.yaml
and implemented exception if no global revision existed.

Change-Id: I14caf8244a7d36201e9cc4ebc90e310d594743ec
2020-06-30 18:30:53 +00:00
Apurva Gokani 22e6df0870 adding site_type to deployment-data schema
This change adds the site_type parameter in deployment_data
The value will be taken from respective site-definition.yaml

Change-Id: I8e65b39c73c94caf3ed4cc517520b9577160b20d
2020-06-26 12:35:22 -05:00
Alexander Hughes 6dbaa4e6e0 Update Treasuremap ref to use valid certs
Unit tests are failing for check_pki_certs against airsloop. The ref
used is at a previous point in time where the certs are expiring.
This patch uses the most recent (as of this patch date) ref of
Treasuremap to get valid certificates for these tests.

Change-Id: I1b276f0eab24148b341676dadfb6e031bfd7c802
Signed-off-by: Alexander Hughes <Alexander.Hughes@pm.me>
2020-05-21 12:33:47 -04:00
Alexander Hughes e4ff07c793 Enable rendering without decrypting
This patchset aims to address least privileged concerns, namely that
Pegleg's current behavior is to require decryption of all site
documents prior to rendering. Failure to do so leads to a duplicate
document error.

Operators of Pegleg may not have a valid reason to access secrets
that are not being modified during their current workflow, their
work may be limited to non-secrets but need to test their changes by
rendering the site manifests.

To enable this, the get_rendered_documents function has been updated
such that if a document is encrypted, the secret value will be
converted to a string to pass schema validation, and then used for
rendering. This will allow operators of Pegleg to render documents
without decrypting secrets. Instead the encrypted string value
of the secret will be used.

Change-Id: I8656b5496e2225e6eb59727c4f79326a1406147c
2020-05-01 20:12:59 +00:00
Phil Sphicas 417975b596 Uniquely name managed documents
When pegleg wraps documents, it uses the original document name as the
name of the managed document. This often results in duplicate documents
(i.e. identical in name and schema). For example, it is expected to have
identically named deckhand document pairs: Certificate & CertificateKey;
CertificateAuthority & CertificateAuthorityKey; PublicKey & PrivateKey.

However, this could also occur for unrelated document types that happen
to have the same name, and generally defeats the principle that each
document is identified by a schema top-level key and the metadata.name.

This change uses a combination of the original document schema and name
to build the name of the pegleg/PeglegManagedDocument/v1.

Change-Id: Iab186ae7e9d24a30cb413be89a17fad960e10bea
2020-04-20 18:45:55 +00:00
Zuul 521219740a Merge "Removed Unused Variable in Unit Test" 2020-01-27 20:43:38 +00:00
Aaron Sheffield 2c1a95f105 Removed Unused Variable in Unit Test
- Removed an optional parameter on with since the variable was not used.

Change-Id: I9794ae07e8904e71dee2cc55cc6ba3c9b71fc06f
2020-01-16 10:30:00 -06:00
Ryan Schroder 4cbc708c1d Standardize save location
Changes to make --save-location standard for all commands

Change-Id: I86a2da01ad1cc1c4d708131b5161182e16b5bb5e
2020-01-15 16:42:20 -06:00
Ian H. Pittwood 1ef5024eb0 Accept multiple path options for decrypt
Change-Id: Ic6c337334f1a21e1f3e303f4ccfb56caeddf4500
2020-01-14 11:42:28 -06:00
Ian H Pittwood ff9c95f423 Add path option to encrypt command
Adds a path option to the encrypt command to specify what directory of
file to encrypt. If path is not specified, all applicable files in the
repositories will be encrypted (this is the current behavior).

Change-Id: Idd5e063a54cf157a8ec761de85cbd67edd05364c
2020-01-07 15:40:05 -06:00
Ian H Pittwood 3a6e3d7cce Add pre-command decrypt option
Adds an option on all site commands to decrypt site files before
executing the command's actions. This option will be enabled by default.
If the command clones repository contents to a temporary directory, only
that temporary data will be decrypted.

Change-Id: Ic10c7196592c6d0e1c69a85b265259357ac28169
2019-12-19 14:52:27 -06:00
Ian H. Pittwood fff70ad861 Refactors pegleg CLI to use single commands
Debugging pegleg can currently be difficult and the Click CLI does not
easily allow debuggers like pdb or PyCharm to use breakpoints. By moving
all CLI command calls into singular functions, we can easily create an
"if __name__ == '__main__'" entry point to call these functions and
investigate any bugs that may arise.

We also gain the ability to reuse more portions of our code by
refactoring these methods.

Change-Id: Ia9739931273eb6458f82dbb7e702a505ae397ae3
2019-12-17 19:32:28 +00:00
Zuul ae5db00f83 Merge "Improve check-pki-certs output" 2019-12-11 15:25:35 +00:00
Ian H. Pittwood 9163ef08ca Add passphrase catalog override option
Adds an option to specify a passphrase catalog to override catalogs
discovered in the site repository. This allows the generation of a
specified subset of passphrases instead of the entire site's catalog.

Change-Id: I797107234292eea8ca788b7a94ed5e2c90566bf5
2019-12-10 20:40:31 +00:00
Ian H. Pittwood a97c9cdba2 Improve check-pki-certs output
Adds alternate message when no certificates are expiring

Exit code will now be 1 if there are certificates expiring and 0 if no
certificates are expiring

Change-Id: I94a7a5af0c5469b83001b5439f18691140de6245
2019-12-10 13:40:07 -06:00
Ian H. Pittwood 7306c3af9b Move generate-pki under generate sub-command
Change-Id: I83b078d27eb2c07c3be7eb6543de85ce32fb2a6e
2019-11-20 19:16:12 +00:00
Alexander Hughes 85fdcd497a Revert "Update storage policy on decrypt"
Barbican is being enabled, as such the metadata field should not be
modified by Pegleg.  If it says encrypted, then Barbican will
encrypt.
If it says cleartext, Barbican won't.  All pegleg needs to do is
decrypt the document prior to bundling it which exists already
without this change.

This reverts commit 2d88f48989.

Change-Id: I8900f910f9816508a8ec5c23932252bb9d1fde09
2019-10-14 12:08:53 +00:00
Alexander Hughes f56d20a2da bugfix for global encrypt/decrypt
This patch updates layer for wrapped documents to preserve original
layer.  Previously all encrypted documents had site layer.

Update encrypt/decrypt logic when determining global keys.

Update units tests.

Change-Id: I447aeaea08a4514655fcabfc7077b6d4b282e27f
2019-10-02 19:13:33 +00:00
Alexander Hughes 14f8600e37 Add profiles to passphrase catalog
Change-Id: Id6e7cddd123e31f0df963167ddf3fa8f33e9060c
2019-09-24 16:19:36 +00:00
Alexander Hughes bc6554241b Add deployment_data to rendered docs
A bug was found where the deployment_data document was not being
included in the upload to shipyard. Upon investigation it was also
noticed that deployment_data wasn't being rendered.

This patch updates the render and upload commands to include the
deployment data document.

Change-Id: I916132e80ac13546468f171a75517a9159e48ea6
2019-09-17 12:39:33 +00:00
Alexander Hughes 2d88f48989 Update storage policy on decrypt
Barbican expects storagePolicy:cleartext when accepting documents
from Pegleg. Once the document is decrypted updated storagePolicy to
cleartext.

During genesis bundle creation, the encrypted documents are
decrypted, then the whole bundle is encrypted.  Once the bundle gets
deployed the contained documents should all have the correct storage
policy of cleartext.

As a result, two unit tests are updated to no longer compare the
storagePolicy, as it is expected everything going to Barbican is
cleartext, but in order for Pegleg to know to encrypt the original
document is storagePolicy:encrypted

Change-Id: I5167ce6b3030d143d1ff0b789883529a6557eeca
2019-09-12 13:44:29 -05:00
Ian H. Pittwood f91ed2ec68 Update treasuremap revision for tests
Pegleg has been using a year old revision of treasuremap for internal
testing using seaworthy. The age of this revision has become a problem
as the certificates in seaworthy expired on August 20th. This change
updates pegleg to the latest version of treasuremap and makes any
necessary path updates for the new version to work with Pegleg tests.

Change-Id: Icea70b8d9bf24c8f85360719945b5899cab6b396
2019-08-22 17:51:59 +00:00
Alexander Hughes 7018d5941c Support regenerating PKI
This patch adds functionality Pegleg currently lacks: the ability to
regenerate expired certificates.

This patch adds:
1. CLI toggle --regenerate-all to generate_pki.  Default is False,
   which means if no certificates are present, generate what is in
   the pki catalogue. If new certs have been added to the catalogue
   generate just those.  If the --regenerate-all flag is True, then
   Pegleg will ignore any existing certs and regenerate (or generate
   for the first time) all certificates defined in the PKI catalogue.
2. Documentation updates for CLI change.
3. Updates to pki_utility to accomodate the new flag.
4. Updates pki_generator methods to use rendered documents to
   accommodate documents that have to be layered.
5. Updates pki_generator unit tests to include a layering definition
   which is now required to run the commands.

Change-Id: I2d8086770e9226e44598ef40eca790981279f626
2019-08-06 17:15:02 +00:00
Ian H Pittwood eb6c2574bc Set a fixed order in which data is dumped to YAML files
One of the well-known issues of Python is that dictionaries do not
maintain order in their keys once created. This causes YAML data dumps
to output in a seemingly random order or alphabetically. As these output
files are often kept in their own repositories, they must go through
review or comparison in VCS. If the order of keys is switching for these
files every time Pegleg is ran, it makes it difficult for a user to
compare newly generated files with the old.

To fix this issue, we can change all dictionaries used to template
YAML files into OrderedDict objects. The OrderedDict objects will
maintain order through their dumping to YAML.

Change-Id: I0c1ee3f3f37ed8598d2ba81528d5c61447cbd0d0
2019-08-02 18:33:26 +00:00
HUGHES, ALEXANDER (ah8742) 174e356214 Update Passphrase Catalog to support types
I recently received a request to add additional features to Pegleg's
generate passphrases command.  The desire was to support multiple
types of secrets:
1. passphrases (24+ characters, including characters from upper,
   lower, number, symbol).
2. base64 encoded passphrases.
3. UUID4.

As well as adding an additional flag to prevent Pegleg from
regenerating specific passphrases that are sensitive to rotation.

Finally, responding to an enhancement request interactive
passphrase generation can now be specified via the command line for
all passphrases, or by specifying 'prompt': True for specific
passphrases in passphrase-catalog.yaml

These objectives were completed by:
1. Updating passphrase_catalog.py to support a type field. If a
   type is not specified, default to existing passphrase generation.
   If an invalid value is specified, raise an exception.
2. Updating passphrase_catalog.py to support a regenerable field. If
   the regenerable field is not specified, default to True. If an
   invalid value is specified, raise an exception. When regenerable
   is determined, secrets of 'uuid' type always use regenerable=False
   as they should be one time values created at time of deployment
   but not rotated.
3. Updating passphrase_catalog.py to support a prompt field. If the
   prompt field is not specified, default to False. If an invalid
   value is specified, raise an exception.
4. Adding appropriate exceptions.
5. Updating passphrase_generator.py to handle the new type checks,
   UUID will use UUID4, base64 uses the existing logic of generating
   a random passphrase and base64 encoding it, and existing logic
   remains for generating a random passphrase.
6. Updating passphrase_generator.py to handle the regenerable field.
   It checks if a file is present at the expected save path, and if
   regenerable=False. If both are true, the passphrase is skipped so
   the passphrase is not overwritten.
7. Updating unit tests to validate the new type checks.

NOTE: # nosec is used in passphrase_generator.py on the
'if passphrase_type == <special type>' statements. These are not a
security concern, but do cause Bandit error B105.  See documentation
for B105 in [0]

Local testing of the generate passphrase command with the following
passphrase types:
passphrase_b64 : base64
passphrase_uuid : uuid
passphrase_specified : passphrase (specified)
passphrase_defaulted : passphrase (defaulted)

Resulted in the following data for each:
passphrase_b64.yaml:data: !!binary |
  UDI1SGFFZHFlbWhITjBrdGJHZGFWRkp6UlZWdFdVNUQ=
passphrase_uuid.yaml:data: 5ce7c6bc-00d2-4b2c-9222-54891f075656
passphrase_specified.yaml:data: cYTenMYXFHUKn6ppYjx#+Hdx
passphrase_defaulted.yaml:data: 13ryjaM?I@sP#3&YQXuQEik4

[0] https://bandit.readthedocs.io/en/latest/plugins/b105_hardcoded_password_string.html

Change-Id: I389316c5194ffa06f3df5114f7ac5f4f2887b319
2019-08-02 07:30:51 -05:00
Ian H. Pittwood 4480ab5574 Restructure usage of test fixtures
Pytest includes a fixture that can be used to generate temporary
directories. Previously Pegleg had implemented a hombrewed version of a
temporary directory fixture. This change removes the homebrewed version
and replaces it with the tmpdir fixture.

Implement tmpdir fixture in tests

Upgrade all testing packages to use the latest features

Removes unused imports and organizes import lists

Removes mock package requirement and uses unittest.mock, included in
python >3.3

Implements a slightly cleaner method to get proxy info

Change-Id: If66e1cfba858d5fb8948529deb8fb2d32345f630
2019-07-29 11:37:36 -05:00
Alexander Hughes 1c8d92ef6b Standardize Pegleg code with YAPF
This patch addresses inconsistent code style and enforces it with a
gate for future submissions.

Separate work will be done in the future to address several of the
PEP8 ignores for docstrings, and attempt to bring the tests directory
to PEP8 compliance.

This patch:
1. Updates .style.yapf to set the knobs desired for YAPF.
2. Updates tox.ini to allow one of the knobs to work.
3. Removes unused code from several __init__.py files.
4. Updates the YAPF version in test-requirements.txt to latest (this
   is needed for several knobs to work).
5. Stylistic changes to the python codebase in Pegleg.
6. Updates to tox.ini to run YAPF during PEP8 check.

Change-Id: Ieaa0fdef2b601d01c875d64b840986e54df73abf
2019-07-25 17:28:18 +00:00
HUGHES, ALEXANDER (ah8742) d888b3e138 Add support for globally encrypted secrets
This patchset adds support for globally encrypted secrets.
Documents with a "site" layer will be encrypted/decrypted with the
standard PEGLEG_PASSPHRASE and PEGLEG_SALT environment variables.

If any secrets exist for the site with a schema of "global_passphrase"
or "global_salt" their values will be captured and used to decrypt
any secrets that do not belong to "site" layer.  If the global keys
do not exist, Pegleg will default to using site keys.

Expected usage:
1. Set site passphrase/salt environment variables
2. Select a global passphrase and salt
3. Use Pegleg's "wrap" command to wrap and encrypt the global keys
4. Encrypt or wrap documents with "global" layer
5. Provide Pegleg path to decrypt

In the case of (4) and (5) Pegleg will determine the correct keys
to use automatically

Change-Id: I5de6d63573619b346fe011628ae21e053e0711f6
2019-07-02 13:54:04 -05:00
HUGHES, ALEXANDER (ah8742) a8620cfd8d Implement default umask for 640 file permissions
Some secrets are being created with undesirable permissions. Upon
inspection it was noticed that in general Pegleg is creating files,
then changing permissions after the fact. This leads to a small
window where the permissions on a file are overly permissive.

This patchset:
1. Sets default umask of 0o027 (640 permissions for files)
2. Explicitly adds the open flag ('r', 'w' etc.) to all open() calls.
3. Replaces sys.stdout.write calls with click.echo() calls to be more
   in line with the rest of the project.
4. Re-orders methods that write so that data is always first, and the
   path is always second.
5. Updates unit tests.
6. Adds unit tests for testing directory and file permissions.
7. Minor style changes.

Change-Id: I0c154aa311ea371940fd24b0aabf58fffaf1d231
2019-06-29 17:56:55 +00:00
Hughes, Alexander (ah8742) ea99c79844 Move credentials logic into config.py
Currently there isn't a uniform or easily expandable way to manage
how Pegleg gets credentials or enforces any complexity on them. This
patchset attempts to address this by:

1. Moving all logic for credentials into config.py
2. Using PeglegSecretManagement as the source of interfacing with
   config.py as this code is the entry point for any encryption or
   decryption work
3. Remove unnecessary code related to this change
4. Update unit tests

In future patchsets the goal is to use these changes to add in a global
passphrase and salt variable into config.py so that encrypt/decrypt type
commands can be executed one time against a site and intelligently
handle retrieval of global credentials for use with global secrets, site
credentials in the form of environment variables will remain used for
site secrets and will not be overridden by any global operations.

Change-Id: I0b6acd3ef5eab6b1f8931f46544bc53443f5c2c0
2019-06-20 11:31:18 +00:00
Hughes, Alexander (ah8742) 489e641702 Remove duplicate flags as options
Multiple occurences of -p or -f in the same command is vague.
Removing duplicate shorthand flags in favor of the explicit long form
flags instead for several commands.

Change-Id: Ic26360e517ea8f7ad2e0e5354d34a61fd622e0f1
2019-06-10 13:30:41 +00:00
Alexander Hughes e6abbf5b07 Update tests dir to use yaml.safe_load()
Unit tests are warning that yaml.load() without a loader are deprecated.
Switch these calls to yaml.safe_load() to resolve warnings.

Change-Id: Ia8e080fc5317eefe432eee984608df190546530c
2019-06-03 16:21:34 +00:00
Alexander Hughes b18c4c0093 Update AIAB repo names
The Airship-In-A-Bottle repo was previously airship/in-a-bottle but has
been updated to airship/airship-in-a-bottle. Updating these references
to use the new name. This resolves both a consistency issue and failing
unit tests.

Change-Id: Ifa36e352a2718fb9998bf26f9f743ca2a7e8f8d4
2019-06-03 15:59:22 +00:00
Zuul 6eb014e350 Merge "Fix multiple I/O issues in cert generation" 2019-05-29 14:12:02 +00:00
Ian H. Pittwood d6ee04f9a3 Specify collection for upload command
Currently, using the upload command in Pegleg will upload all discovered
collections to Shipyard by repo. Uploading multiple of these repos can
result in 409 errors during uplift scenarios. This change compiles all
documents into a single collection document that can then be uploaded to
Shipyard.

Requires a collection name to be specified that will be used as the
'collection_id' for uploading to Shipyard.

Buffer mode is set by default to 'replace' instead of 'auto'.

Change-Id: I546b03fd82873296fff10aba355a50e4b11352d0
2019-05-28 16:00:21 -05:00
Zuul 849d8dd6f8 Merge "Make -u required in CLI when required by repo" 2019-05-28 19:56:46 +00:00
Lev Morgan d6ead96119 Fix multiple I/O issues in cert generation
This patch handles the case where CA certs or authorities are loaded as
byte strings. It also disables parsing YAML documents with python/object
types directly into (non-dict) Python objects (which is PyYaml's
default behavior), as it creates issues with the PeglegManagedDocument
module.
The patch also fixes a bug where attempting to re-encrypt an already
encrypted file would result in a serialized python object being written
rather than the expected output YAML.

Change-Id: I4b84ee8f9922ae042411e70242ffda4622647e86
2019-05-28 14:36:07 -05:00
Ian H. Pittwood 2fa6a1a7bd Allow decryption of directories
This change allows users to specify a directory or file to be decrypted.

Allows directory decryption.

Adds flag to overwrite encrypted file with decrypted data.

Intelligently recognizes paths vs files in CLI input and outputs data 
accordingly.

Change-Id: I0d5e77f0eb1adb42165aa9b214aa90a0db0a3131
2019-05-28 15:08:14 +00:00
Zuul 5b8bfd9211 Merge "Fix to DeploymentData generation" 2019-05-26 17:01:37 +00:00
Lev Morgan aefa569cad Fix to DeploymentData generation
Reorganized DeploymentData to match spec, added Deckhand validation to
unit tests for Pegleg collect.

Change-Id: I9d8d5819bfd55b960ae7dd3c93f5a03b4ea364f1
2019-05-23 16:26:27 -05:00
Zuul e4496a5530 Merge "Update decrypt command" 2019-05-23 16:20:59 +00:00
Zuul 416adce7ab Merge "Support b64 encoding of passphrase catalog" 2019-05-21 20:03:37 +00:00
Zuul 4593523dd2 Merge "Added DeploymentData document generation" 2019-05-15 18:58:15 +00:00
Lev Morgan b8733ea7ec Make -u required in CLI when required by repo
This patch detects when a repository URL requires username substitution
and raises an exception when no username was specified.

Change-Id: Ia60982ecddd957cff8709118b3eb8a905258dd06
2019-05-15 18:38:38 +00:00