Commit Graph

49 Commits

Author SHA1 Message Date
Ryan Schroder 386d7c0e57 Allow rules to be input by user
New tag -r added so that a rules yaml file can be input.
Rules file outlines rules for data manipulation in the engine.
Preexisting rules left in as the default.

Change-Id: Ide8af31b018b4f888486ae6d48ffb441bf9634a7
2020-01-16 16:48:48 +00:00
Ryan Schroder 9d983e4874 [FIX] Bug fix for template path ValueError
Add cleaning step to remove ending slash in template path entered from CLI

Change-Id: I1ec8f48918b54b4733658ed7001b6e6462bef7a8
2019-12-11 14:44:51 +00:00
Ian H Pittwood 0bed580daa Implement intermediary file validation
Implements JSON schema validation for intermediary YAML files

Adds tests for intermediary validation

Change-Id: Iaa385d265b027426f8e5f2376462ffb4c0d1d3fa
2019-08-29 08:39:24 -05:00
Zuul 5d756ece9a Merge "Address TODO notes in engine" 2019-08-15 12:52:50 +00:00
Ian H Pittwood dae192efc6 Upgrade yapf to 0.28
Upgrades yapf to newest version, 0.28.0, and runs the formatter to
update all existing code.

Adds entry to .gitignore to ignore pyenv installations.

Change-Id: I11512a8a522cc530165461cc8f52f7ff010dd092
2019-08-14 19:15:04 +00:00
Ian H Pittwood 81092e6d7c Address TODO notes in engine
This change addresses some of the TODOs made in the Spyglass engine.
There will be additional follow-up patchsets that will address issues
with the rules engine and intermediary validation.

Change-Id: Iba70a51d291659bf827e46fc9070a898303082d1
2019-08-14 19:14:19 +00:00
Alexander Hughes 9689dae61f Standardize Spyglass code with YAPF
From recently merged document updates in [0] there is a desire to
standardize the Airship project python codebase.  This is the effort
to do so for the Spyglass project.

[0] https://review.opendev.org/#/c/671291/

Co-Authored-By: Ian Pittwood <pittwoodian@gmail.com>
Change-Id: I2b0f2491d83675d7742b168ecb9e1dfb9a11e719
2019-07-25 17:01:33 +00:00
Zuul 1b5d64fe32 Merge "Add tests for parsing engine" 2019-07-22 14:39:58 +00:00
Ian H Pittwood fd3d935941 Restructure plugin base
This change makes some mild restructuring and naming changes to the
data-extractor plugin base. Data extraction will now be separated into
two main workflows: loading raw data and source specific parsing into
model objects. Naming of methods was changed to help more accurately
reflect their function in these workflows.

Change-Id: Ia3bc892994ff96cce5c1672fadf35ef2d1c4164b
2019-07-18 12:26:46 -05:00
Ian H Pittwood b7c2bc7ccd Add tests for parsing engine
Increases test coverage of the Spyglass parser engine from 0% to 82%,
bringing overall test coverage to 86.4%. This change moves minimum
coverage to 84%.

Change-Id: I2de496b8d7f4c4252be22c713605fae6fd565b66
2019-07-18 14:27:45 +00:00
Zuul 880f6d455a Merge "Use init to configure plugins" 2019-07-17 15:02:14 +00:00
Ian H. Pittwood b3f1e59148 Use init to configure plugins
Plugins currently use a couple configuration methods to set up field
data for the class. This seems superfluous. This change moves these
configuration steps into the init method so plugin classes can ingest
data as kwargs.

Here is an example of how the change will be implemented in plugins:
https://review.opendev.org/#/c/670171/

Change-Id: Ib26636f1eb4146902ee801af5bcce53d137be2ad
2019-07-15 21:08:57 +00:00
Ian H. Pittwood e58167af68 Centralizes shared CLI options
This change moves all the shared CLI options between Spyglass and its
plugins back into the main Spyglass CLI file. The plugins will then
reference the main CLI file for these options instead of redefining
them in a future change.

Change-Id: I9a6fe7a7d84fed71c372beea9cf7d74a2f6430b1
2019-07-10 15:43:46 +00:00
Ian H. Pittwood 43b6d78d00 Combines all exceptions into a single file
Spyglass has three different files that define exceptions. This change
merges all of those definitions into a single file for simplicity.
Before this change can be merged, spyglass-plugin-xls needs to move the
exceptions needed for its functions.

Related Change: https://review.opendev.org/#/c/667243/

Depends-On: I81c5ca2d9083aece3641bc8b5405dfd44baec810
Change-Id: Ibf34771653247850903e529beb17dbd60ba35fbf
2019-07-09 12:29:01 -05:00
Zuul ca9756de09 Merge "Use data objects for document generation" 2019-07-01 13:51:37 +00:00
Ian H Pittwood 746d7ca2ec Add site_config.yaml back into examples
Spyglass uses additional configuration files to add in any information
not included by a data source. This configuration file should be usable
independent of the data source used. This change adds the
site_config.yaml example file back into the examples folder after it was
initially removed in the project separation of spyglass-plugin-xls.

Change-Id: I9be76c89076327b8ff928aa3da6c91cca0397cc2
2019-06-26 14:58:54 -05:00
Ian H. Pittwood efe24d8a5f Use data objects for document generation
This is a follow-up change to [0] which further implements data objects
to be used in the generation of documents in Jinja2. The following
additions and changes are made:

- Adds helper functions to data objects to filter hosts and networks for
ease of use in Jinja2 templates
- Adds SiteDocumentData factory function to convert intermediary yaml
dictionaries into a SiteDocumentData object with all associated objects
- Updates Jinja2 templates to use data objects
- Cleans up overly complex looping in Jinja2
- Adds tests for new code in models.py

[0] https://review.opendev.org/#/c/662092/

Change-Id: I66ebfeaf5d6ca76b6dee5a2285a74bad8b06b720
2019-06-26 12:03:40 -05:00
Ian H. Pittwood 4747222641 Implements data object models
This change implements data object models from [0] in data extraction
and parsing. The change results in minor modifications to the outputted
intermediary, which can be seen between these two example intermeidary
files [1].

This fully implements the data objects from models.py in data extraction
and parsing. A follow-up change will implement use of the data objects
in Jinja2. Temporarily, all objects will be converted to dictionaries
for generating documents from templates.

[0] https://review.opendev.org/#/c/658917/
[1] https://www.diffchecker.com/NnjjJrb2

Change-Id: Ifd867787aab541be5dabecf9f6026faa2ec7049e
2019-06-21 15:00:46 -05:00
Ian H. Pittwood 313058b8fb Adds tests for Spyglass data objects
Adds unit tests for all data objects created in [0]. The changes in [0]
were merged in short succession with [1], causing the test coverage gate
to fail with all the newly introduced lines of code. This change adds
tests for all of the newly added code and increases test coverage
requirement to 60% (currently at 65.93%).

[0] https://review.opendev.org/#/c/658917/
[1] https://review.opendev.org/#/c/663729/

Change-Id: I96931e3e415af80ca5ab9202c2bda0344a9901f0
2019-06-18 15:15:28 -05:00
Zuul 6b8d6f2aae Merge "Data objects for Spyglass" 2019-06-17 17:47:58 +00:00
Ian Pittwood 6ee44d4974 Adds unit tests for Spyglass CLI
This change implements unit testing for code in the Spyglass CLI.

Change-Id: I4d57bb4e7ee1a2fed8d10cab5eb10636ec599a17
2019-06-17 16:56:18 +00:00
Ian H. Pittwood ccd3912dc7 Data objects for Spyglass
This change switches Spyglass from storing data in large multi-level
dictionaries to using several data objects. The intent of this change is
to make management of Spyglass data easier for developers to understand.

Implementation of the data objects will be handled in a separate change
to keep code review more manageable.

Change-Id: I101ad1ccbd95822965b8da8b6a644522eb2908e7
2019-06-14 13:23:47 -05:00
Ian H. Pittwood 0d6eca47a1 Manifest undefined data validation
This change verifies that manifests generated by Jinja2 do not contain
undefined data. If Spyglass attempts to generate a manifest file that
references undefined data, all previously created manifests will be
deleted and an error will be thrown. Users may bypass this function by
using the "--force" CLI option which will change all undefined data
errors to warnings instead.

Adds undefined data validation to Jinja2 manifest generation.

Adds "--force" option to bypass undefined data validation.

Adds tests for site_processor.py and enables tox testing/coverage.

Change-Id: Iff000eb173995156fbc6b44e621c59ba4dffae35
2019-06-11 15:31:53 +00:00
Ian H. Pittwood 60da55cd18 Schema validation
Adds JSON schema validation to Spyglass.

Change-Id: Ib29bbf9fa02cd6623c75db37a4c8d6f510b52831
2019-06-06 12:56:43 +00:00
Ian Pittwood a002e4203d Separate plugins from Spyglass
This change removes plugins from Spyglass and places them in separate
repositories. Formation, a proprietary plugin, will be removed by this
change and Tugboat will become its own OpenDev maintained repo,
spyglass-plugin-xls. By creating more streamlined plugin management,
end users should be able to more easily create their own plugins for
different data sources.

Related change https://review.opendev.org/#/c/659116/

Depends-On: Ib2f75878b1a29e835cb8e2323aebe9d431c479e7

Change-Id: Ie0eb2e5aefe6bb764e1aa608e53371adaabb9a17
2019-05-29 10:30:14 -05:00
Ian Pittwood 3396b9fa0e Removes remaining yapf: disable statements
The initial addition of yapf into Spyglass caused a few alignment
issues that were temporarily fixed by disabling yapf. This change adds
a knob to the yapf configuration that causes long function statements
to always break before the first statement. This results in more
consistent, visually pleasing code.

Change-Id: I18f9a7677c61524fed12e71a2ecf1003a6ee0ad9
2019-05-14 12:30:29 -05:00
Ian Pittwood d44dfd7526 Implements PBR for packaging
Spyglass will soon have its plugins separated from its main repo. Many
other OpenStack repositories that utilize plugins have implemented
Python Build Reasonableness to manage their entry points for plugins.
As Spyglass will be managing plugins, it makes sense to stay uniform
with other projects and implement PBR.

Implements PBR using setup.py and setup.cfg files.

Changes README to correct file extension.

Freezes PIP requirements file.

Moves YAPF configuration into setup.cfg file.

Change-Id: I48ac5aeb3e544d3da7ce25bd9b827d8fd9c4e108
2019-05-10 18:49:54 +00:00
Ian Pittwood c46e15385c Refactors CLI to match style of Airship CLIs
The original CLI is based on click, but includes all options stacked on
one main method. This is in contrast to other Airship CLIs such as
Pegleg which utilizes click's nesting features to organize its CLI into
multiple groups and commands that each have their own options.

This change separates the Spyglass CLI into three different commands:
generate intermediary, generate manifests, and generate manifests from
intermediary.

Adds a 'verbose' flag on Spyglass.

Defaults plugin to 'tugboat'.

Adds validation for options that apply specifically to tugboat or
formation.

Related docs change: https://review.opendev.org/#/c/650137/

Change-Id: I92e5f040d5205c3ab36ec1d46ecd57bc97849cef
2019-05-06 16:04:02 -05:00
Ian Pittwood 69f0975dec Replace git.openstack.org URLs with opendev.org URLs
Change-Id: I1a933b8794c97244b95b6782033b52cbca74b0b6
2019-05-03 13:48:27 -05:00
Ian H. Pittwood 7cf016e05f Add yapf configuration file
Some of the default behaviors of yapf do not match the styling
preferences of Airship and OpenDev, such as line breaks between
dictionary key/value pairs. This change sets the style template for yapf
(PEP-8) and sets a few knobs to help remedy problematic behavior.

Adds .style.yapf configuration file.

Updates styling of existing files using new configuration.

Change-Id: Ifea58ef2d7be93f47dd8e4f42cac6aea514b83bd
2019-04-29 15:37:25 -05:00
Ian Pittwood 1509ac5cd9 Resolves Bandit 104 errors
From Bandit's docs:
Binding to all network interfaces can potentially open up a service to
traffic on unintended interfaces, that may not be properly documented or
secured. This plugin test looks for a string pattern “0.0.0.0” that may
indicate a hardcoded binding to all network interfaces.

This change fixes B104 errors by removing an unused method and ignoring
a default line used for manifest generation.

Change-Id: Id9cf501829be0166dff933e013c7ac47088bb763
2019-04-25 12:55:26 +00:00
Alexander Hughes 3bf68e464a PEP-8 code styling and linting
As a newer project, Spyglass is still a work in progress in terms of its
styling. This change is meant to help improve readability and compliance
of Spyglass source code.

To match other Airship projects, this change implements YAPF into tox so
linting is automatically ran. This will hopefully keep formatting grey
areas styled consistently.

PEP-8: https://www.python.org/dev/peps/pep-0008/

OpenStack Style Guidelines:
https://docs.openstack.org/hacking/latest/user/hacking.html

PEP Index: https://www.python.org/dev/peps/

Depends on https://review.openstack.org/#/c/648764.

Change-Id: I45b19cc8a7932fd7823dcb69f64a0a1bf19fc434
2019-04-22 18:46:48 +00:00
Alexander Hughes 03a058bfcf Various fixes to Spyglass to resolve tox errors
The tox configuration needs to be updated in order for it to be helpful
for Zuul. This change makes some minor modifications to the tox config
to improve its usability and uniformity with other Open Stack configs.

Makes bare minimum changes for pep8 tests compliance.

Adds gate scripts for white space linting.

In the future, additional changes will need to be made to add tests to
Spyglass. These changes will need to enable the 'py36' and 'cover' tox
env and set coverage levels. Additional linting should also be
performed to improve Spyglass code readability and organization.

Change-Id: I73f946619786e661d02c69bd8e197453f049e0c7
2019-04-16 12:22:53 -05:00
Ian Pittwood 8dd891aced Remove flask YAML web editor from Spyglass
Spyglass was originally built with a web-based YAML editor.
Unfortunately, the presence of the editor has caused some problems. It
can cause the Docker container to freeze before creating the
intermediary YAML file. The execution of the flask app is also the
root cause for the Bandit B104 errors and B605 error.

Since the target audience for Spyglass is developers, it can be
assumed that they will have access to an editor with support for YAML
files. Having a web-based version of the YAML editor is unnecessary and
will just result in more code to maintain in the future.

Removes the editor script from utils.

Removes the yaml-editor entry point from the package.

Removes references to the yaml-editor from the CLI and engine.

Resolves all known Bandit errors.

In the future, a pause in execution could be provided to allow users to
make quick edits. Log messages could also notify users when a
placeholder value is inserted in the intermediary so they can fix it.

Change-Id: Ibc37e61f93b33904ea839e12fe5a8d586985e0b1
2019-04-15 18:11:51 +00:00
Zuul 6149c4a1a5 Merge "Resolves Bandit 101 errors from misusing asserts" 2019-04-10 17:39:04 +00:00
Ian Pittwood 8a39a6fd17 Resolves Bandit 101 errors from misusing asserts
From Bandit's docs:
It was discovered that some projects used assert to enforce interface
constraints. However, assert is removed with compiling to optimised byte
coden (python -o producing *.pyo files). This caused various protections
to be removed. The use of assert is also considered as general bad
practice in OpenStack codebases.

Change-Id: Ie2b5e9cc3e1afcf9e9bd0f8675947754fa1e4b7d
2019-04-09 19:47:13 +00:00
Zuul 0cf4fb0523 Merge "Set autoescape=True to resolve Bandit B701 error" 2019-04-08 20:48:31 +00:00
Tin Lam 89dfec7b4c Add gate
This patch set puts in a rudimentary gate.

Change-Id: I3a2466bd7be5352b46273b385d215913eb8079ba
Signed-off-by: Tin Lam <tin@irrational.io>
2019-04-04 19:25:58 -05:00
Alexander Hughes fab8844461 Set autoescape=True to resolve Bandit B701 error
From Bandit's read the docs:
Jinja2 is a Python HTML templating system. It is typically used to build web
applications, though appears in other places well, notably the Ansible
automation system. When configuring the Jinja2 environment, the option to use
autoescaping on input can be specified. When autoescaping is enabled, Jinja2
will filter input strings to escape any HTML content submitted via template
variables. Without escaping HTML input the application becomes vulnerable to
Cross Site Scripting (XSS) attacks.

Change-Id: I95481c1e863ee144611f85d63274b4514d99a926
2019-04-01 12:42:05 -05:00
lijunjie dcf0735b08 Fix the misspelling of "configuration"
Change-Id: I7becd34e9b29ed73d79742f8a8bd8110f436a3e5
2018-12-27 15:26:41 +08:00
lijunjie e15045a6d3 Fix the misspelling of "configuration"
Change-Id: Ia5aa6e331cc6f29a94ff9b8898dd78273a41f85a
2018-12-27 15:25:11 +08:00
Hemanth Nakkina 303fce156b Fix: remove references to previous repo
Remove references to att-comdev/spyglass repo

Change-Id: Ie07a5ae6062e61c2705a056713a9fb50ad2361fe
2018-12-04 18:51:57 +05:30
Pawan Singh Pal c82debd18a Update build settings and documentation 2018-11-29 23:29:17 +05:30
Pawan Singh Pal 6c8533c9a1 Add yaml-editor documentation. 2018-11-29 21:01:27 +05:30
Pawan Singh Pal 12e25206f3 Add yaml-editor to utils for editing yaml-files on the go
Add setup changes to install yaml-editor with spyglass
2018-11-29 17:25:10 +05:30
hemanthnakkina ecaef2b549
Merge pull request #3 from purnendu15/master
Base Code for Tugboat Plugin and Addition of config files, templates
2018-11-29 10:21:00 +05:30
Purnendu Ghosh 4a8e2720e1 Base Code for Tugboat Plugin and Addition of config files, templates 2018-11-29 02:20:31 +05:30
Gurpreet Singh 76114e0b6a Add tugboat plugin required files
- Add excel parser file for tugboat plugin
- Update the jsonschema
- Update the custom exceptions
2018-11-29 01:25:46 +05:30
Hemanth Nakkina 296705a0a5 Base code for spyglass
- Spyglass skelton with engine, site processor
- Spyglass data extractor with formation plugin
- Docker files and scripts to run spyglass
2018-11-28 15:19:43 +05:30