New tag -r added so that a rules yaml file can be input.
Rules file outlines rules for data manipulation in the engine.
Preexisting rules left in as the default.
Change-Id: Ide8af31b018b4f888486ae6d48ffb441bf9634a7
Upgrades yapf to newest version, 0.28.0, and runs the formatter to
update all existing code.
Adds entry to .gitignore to ignore pyenv installations.
Change-Id: I11512a8a522cc530165461cc8f52f7ff010dd092
This change addresses some of the TODOs made in the Spyglass engine.
There will be additional follow-up patchsets that will address issues
with the rules engine and intermediary validation.
Change-Id: Iba70a51d291659bf827e46fc9070a898303082d1
From recently merged document updates in [0] there is a desire to
standardize the Airship project python codebase. This is the effort
to do so for the Spyglass project.
[0] https://review.opendev.org/#/c/671291/
Co-Authored-By: Ian Pittwood <pittwoodian@gmail.com>
Change-Id: I2b0f2491d83675d7742b168ecb9e1dfb9a11e719
This change makes some mild restructuring and naming changes to the
data-extractor plugin base. Data extraction will now be separated into
two main workflows: loading raw data and source specific parsing into
model objects. Naming of methods was changed to help more accurately
reflect their function in these workflows.
Change-Id: Ia3bc892994ff96cce5c1672fadf35ef2d1c4164b
Increases test coverage of the Spyglass parser engine from 0% to 82%,
bringing overall test coverage to 86.4%. This change moves minimum
coverage to 84%.
Change-Id: I2de496b8d7f4c4252be22c713605fae6fd565b66
Plugins currently use a couple configuration methods to set up field
data for the class. This seems superfluous. This change moves these
configuration steps into the init method so plugin classes can ingest
data as kwargs.
Here is an example of how the change will be implemented in plugins:
https://review.opendev.org/#/c/670171/
Change-Id: Ib26636f1eb4146902ee801af5bcce53d137be2ad
This change moves all the shared CLI options between Spyglass and its
plugins back into the main Spyglass CLI file. The plugins will then
reference the main CLI file for these options instead of redefining
them in a future change.
Change-Id: I9a6fe7a7d84fed71c372beea9cf7d74a2f6430b1
Spyglass has three different files that define exceptions. This change
merges all of those definitions into a single file for simplicity.
Before this change can be merged, spyglass-plugin-xls needs to move the
exceptions needed for its functions.
Related Change: https://review.opendev.org/#/c/667243/
Depends-On: I81c5ca2d9083aece3641bc8b5405dfd44baec810
Change-Id: Ibf34771653247850903e529beb17dbd60ba35fbf
Spyglass uses additional configuration files to add in any information
not included by a data source. This configuration file should be usable
independent of the data source used. This change adds the
site_config.yaml example file back into the examples folder after it was
initially removed in the project separation of spyglass-plugin-xls.
Change-Id: I9be76c89076327b8ff928aa3da6c91cca0397cc2
This is a follow-up change to [0] which further implements data objects
to be used in the generation of documents in Jinja2. The following
additions and changes are made:
- Adds helper functions to data objects to filter hosts and networks for
ease of use in Jinja2 templates
- Adds SiteDocumentData factory function to convert intermediary yaml
dictionaries into a SiteDocumentData object with all associated objects
- Updates Jinja2 templates to use data objects
- Cleans up overly complex looping in Jinja2
- Adds tests for new code in models.py
[0] https://review.opendev.org/#/c/662092/
Change-Id: I66ebfeaf5d6ca76b6dee5a2285a74bad8b06b720
This change implements data object models from [0] in data extraction
and parsing. The change results in minor modifications to the outputted
intermediary, which can be seen between these two example intermeidary
files [1].
This fully implements the data objects from models.py in data extraction
and parsing. A follow-up change will implement use of the data objects
in Jinja2. Temporarily, all objects will be converted to dictionaries
for generating documents from templates.
[0] https://review.opendev.org/#/c/658917/
[1] https://www.diffchecker.com/NnjjJrb2
Change-Id: Ifd867787aab541be5dabecf9f6026faa2ec7049e
Adds unit tests for all data objects created in [0]. The changes in [0]
were merged in short succession with [1], causing the test coverage gate
to fail with all the newly introduced lines of code. This change adds
tests for all of the newly added code and increases test coverage
requirement to 60% (currently at 65.93%).
[0] https://review.opendev.org/#/c/658917/
[1] https://review.opendev.org/#/c/663729/
Change-Id: I96931e3e415af80ca5ab9202c2bda0344a9901f0
This change switches Spyglass from storing data in large multi-level
dictionaries to using several data objects. The intent of this change is
to make management of Spyglass data easier for developers to understand.
Implementation of the data objects will be handled in a separate change
to keep code review more manageable.
Change-Id: I101ad1ccbd95822965b8da8b6a644522eb2908e7
This change verifies that manifests generated by Jinja2 do not contain
undefined data. If Spyglass attempts to generate a manifest file that
references undefined data, all previously created manifests will be
deleted and an error will be thrown. Users may bypass this function by
using the "--force" CLI option which will change all undefined data
errors to warnings instead.
Adds undefined data validation to Jinja2 manifest generation.
Adds "--force" option to bypass undefined data validation.
Adds tests for site_processor.py and enables tox testing/coverage.
Change-Id: Iff000eb173995156fbc6b44e621c59ba4dffae35
This change removes plugins from Spyglass and places them in separate
repositories. Formation, a proprietary plugin, will be removed by this
change and Tugboat will become its own OpenDev maintained repo,
spyglass-plugin-xls. By creating more streamlined plugin management,
end users should be able to more easily create their own plugins for
different data sources.
Related change https://review.opendev.org/#/c/659116/
Depends-On: Ib2f75878b1a29e835cb8e2323aebe9d431c479e7
Change-Id: Ie0eb2e5aefe6bb764e1aa608e53371adaabb9a17
The initial addition of yapf into Spyglass caused a few alignment
issues that were temporarily fixed by disabling yapf. This change adds
a knob to the yapf configuration that causes long function statements
to always break before the first statement. This results in more
consistent, visually pleasing code.
Change-Id: I18f9a7677c61524fed12e71a2ecf1003a6ee0ad9
Spyglass will soon have its plugins separated from its main repo. Many
other OpenStack repositories that utilize plugins have implemented
Python Build Reasonableness to manage their entry points for plugins.
As Spyglass will be managing plugins, it makes sense to stay uniform
with other projects and implement PBR.
Implements PBR using setup.py and setup.cfg files.
Changes README to correct file extension.
Freezes PIP requirements file.
Moves YAPF configuration into setup.cfg file.
Change-Id: I48ac5aeb3e544d3da7ce25bd9b827d8fd9c4e108
The original CLI is based on click, but includes all options stacked on
one main method. This is in contrast to other Airship CLIs such as
Pegleg which utilizes click's nesting features to organize its CLI into
multiple groups and commands that each have their own options.
This change separates the Spyglass CLI into three different commands:
generate intermediary, generate manifests, and generate manifests from
intermediary.
Adds a 'verbose' flag on Spyglass.
Defaults plugin to 'tugboat'.
Adds validation for options that apply specifically to tugboat or
formation.
Related docs change: https://review.opendev.org/#/c/650137/
Change-Id: I92e5f040d5205c3ab36ec1d46ecd57bc97849cef
Some of the default behaviors of yapf do not match the styling
preferences of Airship and OpenDev, such as line breaks between
dictionary key/value pairs. This change sets the style template for yapf
(PEP-8) and sets a few knobs to help remedy problematic behavior.
Adds .style.yapf configuration file.
Updates styling of existing files using new configuration.
Change-Id: Ifea58ef2d7be93f47dd8e4f42cac6aea514b83bd
From Bandit's docs:
Binding to all network interfaces can potentially open up a service to
traffic on unintended interfaces, that may not be properly documented or
secured. This plugin test looks for a string pattern “0.0.0.0” that may
indicate a hardcoded binding to all network interfaces.
This change fixes B104 errors by removing an unused method and ignoring
a default line used for manifest generation.
Change-Id: Id9cf501829be0166dff933e013c7ac47088bb763
The tox configuration needs to be updated in order for it to be helpful
for Zuul. This change makes some minor modifications to the tox config
to improve its usability and uniformity with other Open Stack configs.
Makes bare minimum changes for pep8 tests compliance.
Adds gate scripts for white space linting.
In the future, additional changes will need to be made to add tests to
Spyglass. These changes will need to enable the 'py36' and 'cover' tox
env and set coverage levels. Additional linting should also be
performed to improve Spyglass code readability and organization.
Change-Id: I73f946619786e661d02c69bd8e197453f049e0c7
Spyglass was originally built with a web-based YAML editor.
Unfortunately, the presence of the editor has caused some problems. It
can cause the Docker container to freeze before creating the
intermediary YAML file. The execution of the flask app is also the
root cause for the Bandit B104 errors and B605 error.
Since the target audience for Spyglass is developers, it can be
assumed that they will have access to an editor with support for YAML
files. Having a web-based version of the YAML editor is unnecessary and
will just result in more code to maintain in the future.
Removes the editor script from utils.
Removes the yaml-editor entry point from the package.
Removes references to the yaml-editor from the CLI and engine.
Resolves all known Bandit errors.
In the future, a pause in execution could be provided to allow users to
make quick edits. Log messages could also notify users when a
placeholder value is inserted in the intermediary so they can fix it.
Change-Id: Ibc37e61f93b33904ea839e12fe5a8d586985e0b1
From Bandit's docs:
It was discovered that some projects used assert to enforce interface
constraints. However, assert is removed with compiling to optimised byte
coden (python -o producing *.pyo files). This caused various protections
to be removed. The use of assert is also considered as general bad
practice in OpenStack codebases.
Change-Id: Ie2b5e9cc3e1afcf9e9bd0f8675947754fa1e4b7d
From Bandit's read the docs:
Jinja2 is a Python HTML templating system. It is typically used to build web
applications, though appears in other places well, notably the Ansible
automation system. When configuring the Jinja2 environment, the option to use
autoescaping on input can be specified. When autoescaping is enabled, Jinja2
will filter input strings to escape any HTML content submitted via template
variables. Without escaping HTML input the application becomes vulnerable to
Cross Site Scripting (XSS) attacks.
Change-Id: I95481c1e863ee144611f85d63274b4514d99a926