This patch set simply removes an errant file left over from
directory restructure in https://review.openstack.org/#/c/604157/
Somehow this was missed are crept in via a rebase issue. Either
way there are no dependencies on this so just clean it up.
Change-Id: Ia936419f99ef2b9143d14876190d9b674d3d5a05
This patch set standardizes the Pegleg directory
structure because of the following reasons:
1) src/bin/pegleg is not necessary and only makes
building (e.g. documentation building) and running
of tox targets unnecessarily difficult.
2) src/bin/pegleg is a Java-like standard that
bears no relevance to Python.
Change-Id: I37d39d3d6186b92f8fbfe234221c9e44da48cf10
This patch set removes the impacted CLI command for
the following reasons:
1) it currently isn't being used by anyone or any automation
2) it's not clear how to use it
3) making it work with git diff or git show in order to determine
changed files automatically is sleaker design but currently
there is no use case for this command
Associated engine code, CLI documentation and helper functionality
only associated with this command are dropped.
Change-Id: Ic09951e988b82a386d2a1a7b9ce9d77f78bd00d7
This patch set rolls back previously introduced behavior in
https://review.openstack.org/#/c/584482/ which forbids users
from basically performing any Pegleg command that references
a dirty local repository. This is annoying, forcing users to
create temporary commits before executing a Pegleg command.
Fortunately with https://review.openstack.org/#/c/577886/
Pegleg copies over all repositories to temporary folders,
within which dirty repos can have their changes temporarily
committed, allowing different references to then be safely
checked out, without ever modifying any local repositories.
Change-Id: I2142ae434f8ad57d0ab81cb104e21d952dc23148
https://review.openstack.org/#/c/582652/ added support for cloning
from repositories using various reference types and allowing checkouts
from already-cloned repositories.
This patch set expands on that behavior to enhance the CLI to allow
for using revisioned repositories rather than relying on pseudo-revision
subfolders. That is, previously Pegleg achieved revision behavior
through subdirectories. With this patch set, Pegleg will treat every
manifests directory it encounters as a proper Git repository, such
that revision support for clones/checkouts is now supported.
Thus, this patch set is backwards incompatible and breaks the previous
behavior of treating every directory as a flat directory from which
revision sub-folders were referenced. Going forward, only repositories
will be supported.
* Remove directory revision support
* Add support for repository definitions within the site-definition:
repositories:
global:
revision: 0.9
url: ssh://REPO_USERNAME@gerrit:29418/aic-clcp-manifests.git
secrets:
revision: master
url: ssh://REPO_USERNAME@gerrit:29418/aic-clcp-security-manifests.git
where REPO_USERNAME would be replaced with the value from -u and use
the public SSH key supplied with -k.
* Remove primary and aux repository logic and replace with site
repository and extra repositories
* Allow overriding definitions in the above site-definition on the
command line: -e global=/mydir@revision
* Move linting into a site action even though more work to be
done to restrict it to a site
Co-Authored-By: Felipe Monteiro <felipe.monteiro@att.com>
Change-Id: Iaa928ec2f777ed6f899d3b1790f5f9de613da9bb
This change adds the global zuul pep8 tox job, which runs both
bandit and pep8 using tox. This also removes the two other airship
specific lint-pep8 and bandit zuul jobs since they are both covered
by the default openstack global one.
Also cleaned up the tox.ini by moving the requirements into the
test-requirements.txt file.
Change-Id: Iab37a8090515936732e390b1f7c6d281e014e31c
Autouse fixtures in pytest are a powerful way of executing a
fixture for each test. We should leverage one for automatically
cleaning up temporarily cloned Git repos cloned during unit tests.
Change-Id: I21288cf34b90cd8b280cd78275aab12a42de06a7
This patch set adds logic to examine the currently checked out
reference in a repository if ref=None is passed to git.git_handler.
This makes it convenient to simply reuse the revision that is
currently checked out without having to re-supply it. Note that
ref is still a required parameter so providing None will have
to be done explicitly.
Defaults for ref='master' have been changed to ref=None.
Turns out the GitPython library has logic to use the
repo's Git config's fetch refspec by default [0],
which means that by default 'master' will be checked out
for many repositories.
Also adds a ``is_repository`` function to ``git`` module to
more robustly check whether a directory is a repo.
Unit tests and doc strings are updated as well.
[0] a8591a094a/git/remote.py (L779)
Change-Id: Ib1736eb20a51996cf03fa7f6187f078b39595267
Autouse fixtures in pytest are a powerful way of executing a
fixture for each test. Since Pegleg uses a global in-memory
object for storing config info, this dictionary should be
cleaned up after each test to ensure test idempotency.
The best way (and cleanest) is to use an autouse fixture inside
a conftest.py which ensures that it is used after each and every
unit test [0].
[0] https://docs.pytest.org/en/latest/fixture.html#conftest-py-sharing-fixture-functions
Change-Id: I0c647d3d7547d1bdcd7c289a4c11159376f9f58d
This addresses 2 concerns:
1) When -s or --save-location is omitted for pegleg
site collect command, the following error is raised:
TypeError: '_io.TextIOWrapper' object is not subscriptable
That is because the default kwarg for the associated option
is sys.stdout [0] which returns an object not a path, causing
the Click library itself to blow up. Note that this is dropped
in master [1], which isn't correct, as there is a desire to
use stdout as the default location for collection output.
Thus, new logic is added to output the collected document data
to stdout.
2) When a directory doesn't exist and -s is provided (as an example
pegleg site -p /path/to/primary/dir collect <site_name> -s <non_existent_dir>)
it is better to just create it via os.makedirs() than to error out
with an exception [2].
[0] d9692126ed (diff-bd81bbb896486546c9e59d018a83d05d)
[1] f25b5b6593/src/bin/pegleg/pegleg/cli.py (L72)
[2] 843d1a5010/src/bin/pegleg/pegleg/engine/site.py (L36)
Change-Id: Ie1da73efa437cbdfbe6d2b7ab616a1467eb57358
This raises an exception on trying to checkout a ref from a
dirty repo in the git_handler module. The parent patch
https://review.openstack.org/#/c/582652/ currently forcibly
cleans the repo but this is undesirable as it may have local
user changes that need to be resolved first.
The safest path is for Pegleg to immediately raise an exception
on any tracked/untracked files that are detected using the
GitPython API.
Unit tests are added for both untracked/tracked file cases.
Change-Id: I2241bc981dca1999508c3c7e95948fe47a5ddebf
This patch set removes substitution_sources kwarg from the
Deckhand layering call as it is deprecated [0] because
all concrete documents will simply be used by default as the
substitution source documents.
[0] 1583b78902/deckhand/engine/layering.py (L480)
Change-Id: Ib0ecc9fffdcc539f7fc804824e50981219353292
* Add support for URLs and directories including git clone support
* Add support for http://, https://, and ssh:// git cloning
* Add support for cloning behind proxy
* Add support for checking out references of cloned repos
* Add support for checking out references of local repos
* Add support for Pegleg Git exceptions
This patch set also adds support for including Pegleg
source code in documentation and adds exceptions
documentation.
Change-Id: I417a62c815f97a70f3abc432cc342707e8ce1f54
1. Currently Pegleg render does not show error output when deckhand produce
error exception.
2. Added unit test case for this PS
Change-Id: If1699545dfc539dc09cea449bd2a48c888294c44
Pegleg should have the ability to perform validations as part of
"collect" by default and bypass validation using optional parameter.
This PS adds automatic validation (enabled by default) to Pegleg's
collect command or else skips it if --no-validate is provided.
The "collect" command also takes the same optional arguments of
-x/-w for controlling the lint options which are excluded or warned
about.
Change-Id: Iee9f27f1133d780d507f7a13cce42076768a0c1c
This patchset updates pegleg.engine.deckhand to also log the
errors associated with an exception object following a rendering
failure to aid with troubleshooting.
Change-Id: I9ca46e4b102bd39fb7c582bb1720702892df7f9e
Pegleg is tossing an error that's difficult to decipher when an output
directory is not specified during "collect". Currently "save-location"
param is optional and can be missed during command execution. This PS
handles missing or invalid output directory exception.
Change-Id: I6f41e4951be29d8ecdce73f6090f0bcc9e7ee2d9
In one of the case, when pegleg is installed as python package, pegleg
is having issue in importing dependency packages. This ps is used for
finding dependency packages automatically.
Change-Id: Iaf3fb0f344e1902af41bcbf0e05e2a43a32836a8
This patchset renames the tox.ini jobs:
* lint => pep8
to comply with OpenStack standards [0]. This is needed so that [1]
works seamlessly (it will enable py35 and cover jobs for Pegleg).
[0] e.g. 04469a5181/tox.ini (L119)
[1] I174d5df008f6e000da1a3878afe75919312ea7aa
Change-Id: Ic3a5f2391e129cae32f60c0fb17197169df1b4c6
This patchset changes pegleg's lint function `_verify_deckhand_render`
to render each batch of documents by site, rather than attempting
to render all documents for every site in one go, which results
in false positives.
For example, given 3 sites -- site1, site2, and site3 -- Deckhand
will now render all documents from global, all documents in type,
and all documents in site1, then repeat the same for site2, then
repeat the same for site3, and finally aggregate the error results
from each Deckhand render output, which is returned back to the user.
Change-Id: If853ba0331470d7ec66dfc6116e626251a9f496d
This is to add a top-level lint/fmt job to Pegleg so that tox -e lint
is executed in CICD. lint is included under envlist to accomplish this.
fmt is excluded from envlist as it doesn't need to be executed in CICD
but is included nonetheless for convenience.
Also fixes some typos.
Change-Id: I76165704b32653c98e506ddde71c8240d5f28492
This is to include the 'deployment' subdirectory in the FULL_STRUCTURE
data structure which defines the tree for the expected Pegleg
directory. It is included in Treasuremap [0] and aic-global-manifests
so should be included under FULL_STRUCTURE.
[0] https://github.com/att-comdev/treasuremap/tree/master/deployment_files/site/atl-lab1
Change-Id: I6e91f36977dcaffe8600892a3d59885c9ec0e9fd
Add a unit test to make sure Pegleg only considers YAML files
when linting
Fixes a typo in filename for test_selectable_linting
Changed site_yaml file stucture to more resemble real site
directory structure
Change-Id: Ib1520f9509d279d88b4408858878022fb05c7707
This is a trivial fix that adds the Apache copyright to all
Python files as this is an open source project.
Change-Id: I712b34809dce99fc52385f236f5c10d5c805937a
* Fix inconsistencies of lint errors WRT string vs tuple
* Adds error codes to all lint errors
* Update Deckhand version
Change-Id: Ib672be7f1d8805e5e6afbb8f942693995aa45591
This is to add a tox target for running unit tests in
src/bin/pegleg and to add an alias to the Makefile in root
directory.
Change-Id: Ia138c57e8d732f6dbf19f00e452b0eac1a1aa9cb
- site_definition.yaml was collected twice.
- Update Makefile with format target and ignore
pep8 rule that conflicts w/ YAPF formatting
Change-Id: I2bc6d40e9e9085558315224df7d9127a8945ee1c
Adds option -x to exclude certain linting policies
and -w to warn of failure for certain linting policies
if failures are expected.
Updates gitignore to exclude files created when running
tests.
Adds requirements for testing.
Adds unit test for new cli options and test
site-definition.yaml
Documentation for cli options can be found here [0].
[0]-https://review.gerrithub.io/#/c/403216/
Change-Id: I6e905c1ba7a23d0b2fdbf9552bec8a6620ff9731
- A single primary repo must be specified which
holds the site_definition.yaml file.
- Zero or more auxiliary repos can be specified which
have additional documents used in the site definition.
- Collected documents are written to a file named after their
source repo. Collection must always be given a output directory
rather than a single file now.
Change-Id: Iceda4da18c4df45d917d88a49144e39e3f1743ed
Felipe Monteiro