openstack
/
congress-specs
Code Issues Proposed changes

Retire the Congress Specs project 

Recently the TC has worked on determining the criteria for when an
OpenStack project should be retired.  When there was not a PTL nominee
for the Congress project, that triggered the TC to review the project
health per [1], and the TC has determined [2] that development work on
the project has ceased.  This decision was announced in the
openstack-discuss mailing list in April 2020 [3].

This commit retires the repository per the process for governance
removal in the Victoria cycle as specified in the Mandatory Repository
Retirement resolution [4] and detailed in the infra manual [5].

Should interest in developing Congress as part of OpenStack revive,
please revert this commit to have the project rejoin the list of active
projects.

The community wishes to express our thanks and appreciation to all of
those who have contributed to the Congress project over the years.

[1] https://governance.openstack.org/tc/reference/dropping-projects.html
[2] http://eavesdrop.openstack.org/irclogs/%23openstack-tc/latest.log.html#t2020-04-20T15:36:59
[3] http://lists.openstack.org/pipermail/openstack-discuss/2020-April/014292.html
[4] https://governance.openstack.org/tc/resolutions/20190711-mandatory-repository-retirement.html
[5] https://docs.opendev.org/opendev/infra-manual/latest/drivers.html#retiring-a-project

Change-Id: Ide2a1f572f046e2d9f1bf9468556dbb39c736c0f
This commit is contained in:
Nate Johnston 2020-04-21 17:37:25 -04:00 committed by Andreas Jaeger
parent 9bf73cc80c
commit 1f031e89c2
72 changed files with 11 additions and 13527 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

16
.gitignore vendored
View File

@ -1,16 +0,0 @@
# build
.venv
*.egg*
*.swp
*.swo
*.pyc
.stestr/
# Editors
*~
.*.swp
.bak
/.project
# Testenvironment
.tox/

3
.stestr.conf
View File

@ -1,3 +0,0 @@
[DEFAULT]
test_path=./tests
top_dir=./

9
.zuul.yaml
View File

@ -1,9 +0,0 @@
- project:
templates:
- openstack-specs-jobs
check:
jobs:
- openstack-tox-py37
gate:
jobs:
- openstack-tox-py37

201
LICENSE
View File

@ -1,201 +0,0 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "{}"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright {yyyy} {name of copyright owner}
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

62
README.rst
View File

@ -1,54 +1,14 @@
========================
Team and repository tags
========================
=========================
Welcome to Congress Specs
=========================
.. image:: http://governance.openstack.org/badges/congress-specs.svg
:target: http://governance.openstack.org/reference/tags/index.html
This project is no longer maintained.
.. Change things from this point on
The contents of this repository are still available in the Git
source code management system. To see the contents of this
repository before it reached its end of life, please check out the
previous commit with "git checkout HEAD^1".
=================================
OpenStack Congress Specifications
=================================
This git repository is used to hold approved design specifications for additions
to the Congress project. Reviews of the specs are done in gerrit, using a similar
workflow to how we review and merge changes to the code itself.
The layout of this repository is::
specs/<release>/
You can find an example spec in `doc/source/specs/template.rst`.
Specifications are proposed for a given release by adding them to the
`specs/<release>` directory and posting it for review. The implementation
status of a blueprint for a given release can be found by looking at the
blueprint in launchpad. Not all approved blueprints will get fully implemented.
Specifications have to be re-proposed for every release. The review may be
quick, but even if something was previously approved, it should be re-reviewed
to make sure it still makes sense as written.
Prior to the Juno development cycle, this repository was not used for spec
reviews. Reviews prior to Juno were completed entirely through Launchpad
blueprints::
http://blueprints.launchpad.net/congress
Please note, Launchpad blueprints are still used for tracking the
current status of blueprints. For more information, see::
https://wiki.openstack.org/wiki/Blueprints
For more information about working with gerrit, see::
http://docs.openstack.org/infra/manual/developers.html#development-workflow
To validate that the specification is syntactically correct (i.e. get more
confidence in the Jenkins result), please execute the following command::
$ tox
After running ``tox``, the documentation will be available for viewing in HTML
format in the ``doc/build/`` directory.
For any further questions, please email
openstack-discuss@lists.openstack.org or join #openstack-dev on
Freenode.

273
doc/source/conf.py
View File

@ -1,273 +0,0 @@
# -*- coding: utf-8 -*-
#
# Tempest documentation build configuration file, created by
# sphinx-quickstart on Tue May 21 17:43:32 2013.
#
# This file is execfile()d with the current directory set to its containing dir.
#
# Note that not all possible configuration values are present in this
# autogenerated file.
#
# All configuration values have a default; values that are commented out
# serve to show the default.
import sys
import os
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#sys.path.insert(0, os.path.abspath('.'))
# -- General configuration -----------------------------------------------------
# If your documentation needs a minimal Sphinx version, state it here.
#needs_sphinx = '1.0'
# Add any Sphinx extension module names here, as strings. They can be extensions
# coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = ['sphinx.ext.autodoc',
'sphinx.ext.todo',
'sphinx.ext.viewcode',
'openstackdocstheme'
]
todo_include_todos = True
# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']
# The suffix of source filenames.
source_suffix = '.rst'
# The encoding of source files.
#source_encoding = 'utf-8-sig'
# The master toctree document.
master_doc = 'index'
# General information about the project.
project = u'Congress Specs'
copyright = u'2014, Congress Contributors'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
#language = None
# There are two options for replacing |today|: either, you set today to some
# non-false value, then it is used:
#today = ''
# Else, today_fmt is used as the format for a strftime call.
#today_fmt = '%B %d, %Y'
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
exclude_patterns = ['_build']
# The reST default role (used for this markup: `text`) to use for all documents.
#default_role = None
# If true, '()' will be appended to :func: etc. cross-reference text.
#add_function_parentheses = True
# If true, the current module name will be prepended to all description
# unit titles (such as .. function::).
add_module_names = False
# If true, sectionauthor and moduleauthor directives will be shown in the
# output. They are ignored by default.
show_authors = False
# The name of the Pygments (syntax highlighting) style to use.
pygments_style = 'sphinx'
# A list of ignored prefixes for module index sorting.
modindex_common_prefix = ['congress-specs.']
# -- Options for man page output ----------------------------------------------
man_pages = []
# -- Options for HTML output ---------------------------------------------------
# The theme to use for HTML and HTML Help pages. See the documentation for
# a list of builtin themes.
html_theme = 'openstackdocs'
# Theme options are theme-specific and customize the look and feel of a theme
# further. For a list of options available for each theme, see the
# documentation.
#html_theme_options = {}
# Add any paths that contain custom themes here, relative to this directory.
#html_theme_path = []
# The name for this set of Sphinx documents. If None, it defaults to
# "<project> v<release> documentation".
#html_title = None
# A shorter title for the navigation bar. Default is the same as html_title.
#html_short_title = None
# The name of an image file (relative to this directory) to place at the top
# of the sidebar.
#html_logo = None
# The name of an image file (within the static path) to use as favicon of the
# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32
# pixels large.
#html_favicon = None
# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
# using the given strftime format.
git_cmd = "git log --pretty=format:'%ad, commit %h' --date=local -n1"
html_last_updated_fmt = os.popen(git_cmd).read()
# If true, SmartyPants will be used to convert quotes and dashes to
# typographically correct entities.
#html_use_smartypants = True
# Custom sidebar templates, maps document names to template names.
#html_sidebars = {}
# Additional templates that should be rendered to pages, maps page names to
# template names.
#html_additional_pages = {}
# If false, no module index is generated.
html_domain_indices = False
# If false, no index is generated.
html_use_index = False
# If true, the index is split into individual pages for each letter.
#html_split_index = False
# If true, links to the reST sources are added to the pages.
#html_show_sourcelink = True
# If true, "Created using Sphinx" is shown in the HTML footer. Default is True.
#html_show_sphinx = True
# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True.
#html_show_copyright = True
# If true, an OpenSearch description file will be output, and all pages will
# contain a <link> tag referring to it. The value of this option must be the
# base URL from which the finished HTML is served.
#html_use_opensearch = ''
# This is the file name suffix for HTML files (e.g. ".xhtml").
#html_file_suffix = None
# Output file base name for HTML help builder.
htmlhelp_basename = 'Congress-Specsdoc'
# -- Options for LaTeX output --------------------------------------------------
latex_elements = {
# The paper size ('letterpaper' or 'a4paper').
#'papersize': 'letterpaper',
# The font size ('10pt', '11pt' or '12pt').
#'pointsize': '10pt',
# Additional stuff for the LaTeX preamble.
#'preamble': '',
}
# Grouping the document tree into LaTeX files. List of tuples
# (source start file, target name, title, author, documentclass [howto/manual]).
latex_documents = [
('index', 'Congress-specs.tex', u'Congress Specs',
u'Congress Contributors', 'manual'),
]
# The name of an image file (relative to this directory) to place at the top of
# the title page.
#latex_logo = None
# For "manual" documents, if this is true, then toplevel headings are parts,
# not chapters.
#latex_use_parts = False
# If true, show page references after internal links.
#latex_show_pagerefs = False
# If true, show URL addresses after external links.
#latex_show_urls = False
# Documents to append as an appendix to all manuals.
#latex_appendices = []
# If false, no module index is generated.
#latex_domain_indices = True
# -- Options for Texinfo output ------------------------------------------------
# Grouping the document tree into Texinfo files. List of tuples
# (source start file, target name, title, author,
# dir menu entry, description, category)
texinfo_documents = [
('index', 'Congress-specs', u'Congress Design Specs',
u'Congress Contributors', 'congress-specs', 'Design specifications for the Congress project.',
'Miscellaneous'),
]
# Documents to append as an appendix to all manuals.
#texinfo_appendices = []
# If false, no module index is generated.
#texinfo_domain_indices = True
# How to display URL addresses: 'footnote', 'no', or 'inline'.
#texinfo_show_urls = 'footnote'
# -- Options for Epub output ---------------------------------------------------
# Bibliographic Dublin Core info.
epub_title = u'Congress Specs'
epub_author = u'Congress Contributors'
epub_publisher = u'Congress Contributors'
epub_copyright = u'2014, Congress Contributors'
# The language of the text. It defaults to the language option
# or en if the language is not set.
#epub_language = ''
# The scheme of the identifier. Typical schemes are ISBN or URL.
#epub_scheme = ''
# The unique identifier of the text. This can be a ISBN number
# or the project homepage.
#epub_identifier = ''
# A unique identification for the text.
#epub_uid = ''
# A tuple containing the cover image and cover page html template filenames.
#epub_cover = ()
# HTML files that should be inserted before the pages created by sphinx.
# The format is a list of tuples containing the path and title.
#epub_pre_files = []
# HTML files shat should be inserted after the pages created by sphinx.
# The format is a list of tuples containing the path and title.
#epub_post_files = []
# A list of files that should not be packed into the epub file.
#epub_exclude_files = []
# The depth of the table of contents in toc.ncx.
#epub_tocdepth = 3
# Allow duplicate toc entries.
#epub_tocdup = True
# -- Options for openstackdocstheme ---------------------------------------
repository_name = 'openstack/congress-specs'
bug_project = 'congress-specs'
bug_tag = ''

100
doc/source/index.rst
View File

@ -1,100 +0,0 @@
.. congress-specs documentation master file
===============================
Congress Project Specifications
===============================
Train approved specs:
..
.. toctree::
:glob:
:maxdepth: 1
specs/train/*
Stein approved specs:
.. toctree::
:glob:
:maxdepth: 1
specs/stein/*
Rocky approved specs:
.. toctree::
:glob:
:maxdepth: 1
specs/rocky/*
Queens approved specs:
.. toctree::
:glob:
:maxdepth: 1
specs/queens/*
Pike approved specs:
.. toctree::
:glob:
:maxdepth: 1
specs/pike/*
Newton approved specs:
.. toctree::
:glob:
:maxdepth: 1
specs/newton/*
Mitaka approved specs:
.. toctree::
:glob:
:maxdepth: 1
specs/mitaka/*
Liberty approved specs:
.. toctree::
:glob:
:maxdepth: 1
specs/liberty/*
Kilo approved specs:
.. toctree::
:glob:
:maxdepth: 1
specs/kilo/*
Juno approved specs:
.. toctree::
:glob:
:maxdepth: 1
specs/juno/*
Spec Template:
.. toctree::
:glob:
:maxdepth: 1
specs/*
==================
Indices and tables
==================
* :ref:`search`

1
doc/source/specs
View File

@ -1 +0,0 @@
../../specs

5
requirements.txt
View File

@ -1,5 +0,0 @@
openstackdocstheme>=1.18.1 # Apache-2.0
pbr>=0.6,!=0.7,<1.0
sphinx!=1.6.6,!=1.6.7,>=1.6.2;python_version>='3.4' # BSD
stestr>=2.0.0
testtools>=0.9.34

23
setup.cfg
View File

@ -1,23 +0,0 @@
[metadata]
name = congress-specs
summary = OpenStack Congress Project Development Specs
description-file =
README.rst
author = OpenStack
author-email = openstack-discuss@lists.openstack.org
home-page = http://specs.openstack.org/openstack/congress-specs/
classifier =
Intended Audience :: Developers
License :: OSI Approved :: Apache Software License
Operating System :: POSIX :: Linux
[build_sphinx]
all_files = 1
build-dir = doc/build
source-dir = doc/source
[pbr]
warnerrors = True
[wheel]
universal = 1

22
setup.py
View File

@ -1,22 +0,0 @@
#!/usr/bin/env python
# Copyright (c) 2013 Hewlett-Packard Development Company, L.P.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# THIS FILE IS MANAGED BY THE GLOBAL REQUIREMENTS REPO - DO NOT EDIT
import setuptools
setuptools.setup(
setup_requires=['pbr'],
pbr=True)

144
specs/juno/Plexxi-Driver.rst
View File

@ -1,144 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Plexxi Driver
==========================================
https://blueprints.launchpad.net/congress/+spec/plexxi-driver
This blueprint is to layout the design of a driver that integrates PlexxiCore
into congress.
Problem description
===================
A detailed description of the problem:
* A PlexxiCore Database can contain a vast wealth of information about a
network that can be an asset to a Congress system in a network where
Plexxi is present, and without this driver Congress does not have access
to this data.
Proposed change
===============
Integrate the Plexxi Driver driver.
Alternatives
------------
None
Policy
------------
This example compares data stored in the Plexxi tables with data stored in
Nova tables and looks for Virtual Machines with names that are identical in
both data sets
RepeatedName(vname,pvuuid)
:- plexxi:vms(pvuuid,vname,phuuid,pvip,pvmaccount,pvaffin),
nova:servers(nvuuid,vname,a,nstatus,b,c,d,num)
Policy Actions
--------------
This driver provides monitoring of the Plexxi Core state as well
and currently demonstrations reactive actions based on the data provided.
Data Sources
------------
Primary: PlexxiCore
Secondary: Nova/Congress - For demo policy
Data model impact
-----------------
None
REST API impact
---------------
Users of Congress who have PlexxiCore integrated in their network will now be
able to access data stored in their PlexxiCore database through the congress
API.
Security impact
---------------
Network data stored in the PlexxiCore Database will now be accessible through
Congress
Notifications impact
--------------------
None
Other end user impact
---------------------
None
Performance impact
------------------
None
Other deployer impact
---------------------
None
Developer impact
----------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
Conner-Ferguson
Work items
----------
None
Dependencies
============
Plexxi Class package
Plexxi-Core used within the network
Testing
=======
Documentation impact
====================
Documentation for the Driver is currently located in the bitbucket
that found that can be found in the references section
References
==========
Working Demo -
https://www.youtube.com/watch?v=ZAEydTlIW64
Current Code Location -
https://bitbucket.org/ConnerFerguson/plexxi-driver/overview
Devloper Email -
Conner.Ferguson1@marist.edu

124
specs/juno/cinder-datasource-driver.rst
View File

@ -1,124 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Cinder Data Source Driver
==========================================
https://blueprints.launchpad.net/congress/+spec/cinder-data-source-driver
This blueprint is to add a data source driver for cinder.
Problem description
===================
A datasource driver is required to expose block storage parameters to the
Congress policy framework so that we can write policies involving block storage
entities. Cinder is the OpenStack component that provides block storage
service. The cinder-datasource-driver interacts with cinderclient to provide
block storage specific states to congress for policy monitoring.
Proposed change
===============
Add data source driver that integrates congress with cinder.
Alternatives
------------
N/A
Policy
------
This will use the congress language. cinder:volumes(X) etc.
Policy Actions
--------------
Just monitoring right now.
Data Sources
------------
Cinder
Data model impact
-----------------
TBD
REST API impact
---------------
N/A
Security impact
---------------
N/A
Notifications impact
--------------------
N/A
Other end user impact
---------------------
N/A
Performance impact
------------------
N/A
Other deployer impact
---------------------
N/A
Developer impact
----------------
N/A
Implementation
==============
Assignee(s)
-----------
Samta Rangare
srangare@mvista.com
Work items
----------
- Implement cinder driver with essential tables
- Implement test code to test the driver
- Implement tempest code for real-time tests
Dependencies
============
python-cinderclient
Testing
=======
- Will need to add unit test code
- Will add tempest tests.
Documentation impact
====================
N/A
References
==========
N/A

153
specs/juno/datalog-aggregates.rst
View File

@ -1,153 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Support for using Aggregates in Datalog
==========================================
https://blueprints.launchpad.net/congress/+spec/datalog-aggregates
Aggregates in datalog are required for writing policies involving multiple
records in a single ATOM.
Problem description
===================
* Consider a policy statement "The maximum number of ports on the Subnet X
belonging to tenant Y should not exceed N"
To realize this policy, we need to use aggregations like COUNT, MAX, SUM etc.
which span across multiple records of the table.
* The current Datalog support in Congress does not allow writing policies with
such aggregates.
* Also there is no way to simulate aggregations using existing Datalog support.
* When a policy engine subscribes to an aggregate on a data source table,
the aggregates must be updated every time there is a change in the
corresponding table.
Proposed change
===============
The aggregate support for Datalog allows writing policy atoms of the following
form:
Example policies involving aggregates:
1. Restriction on number of ports on a subnet
error(subnet) :- neutron:networks.subnets(subnet, port),
neutron:ports(port, ip), count(num_port, port), not gt(num_port, 10)
2. Restriction on average CPU Load on a host
error(host) :- ceilometer:meters(host, cpu_util), nova:servers(host, vm_host),
average(avg_cpu, vm_host, cpu_util), not gt(avg_cpu, 80)
average(x, y, z) :- sum(z, cpu_util), count(y, vm_host), div(x, z, y)
(Note: These are tentative policy statements and may be subject to change as
the design/implementation progresses)
* The aggregates like "SUM", "COUNT" must be defined as ANTLR Terms and
Congress must be able to parse and compile these terms.
The initial implementation will use SUM and COUNT as prototype aggregations.
Other similar aggregates can be considered as enhancements to be taken up
at a later date.
* The runtime execution should be able to update tables corresponding to the
aggregate terms with computed values
Alternatives
------------
N/A
Data model impact
-----------------
No data model changes are expected.
REST API impact
---------------
This change does not impact REST APIs as the changes involved will only extend
the policy syntax which is fed as input to the policy APIs
Security impact
---------------
None.
Notifications impact
--------------------
None
Other end user impact
---------------------
End user will be able to write policies involving aggregates.
Performance impact
------------------
Evaluation of policies involving aggregates may involve additional processing
and thereby impacting performance.
Other deployer impact
---------------------
None.
Developer impact
----------------
None.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
Madhu Mohan (mmohan@mvista.com)
Work items
----------
* Add new terms in Congress.g to allow ANTLR to recognize aggregate keywords
* Extend parser/compiler to accept aggregate keywords as part of the rule
syntax and validate
* Extend runtime to compute aggregate values and update the corresponding
tables
* Add sufficient tests to validate support for aggregates
Dependencies
============
None.
Testing
=======
* Add test cases involving aggregates
* Add Tempest tests
* Create sample policies to test some real scenarios
Documentation impact
====================
* Docs for external consumption would need to be updated with explanation on
writing aggregate-based policies
References
==========
1. http://logic.stanford.edu/reports/LG-2012-01.pdf

236
specs/kilo/action-execution-interface.rst
View File

@ -1,236 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Add action execution interface
==========================================
https://blueprints.launchpad.net/congress/+spec/action-execution-interface
Add analogy to datasource driver for services that can execute actions.
Problem description
===================
Datasource drivers pull information from cloud services, allowing Congress
to read the current state of the cloud (e.g. the list of all servers).
Some cloud services also have the capacity to change the state of the cloud
(e.g. create a new server). Today there is no way for the policy engine
to ask for changes to be made in the state of the cloud. Congress needs
an interface for cloud services that allows it to execute actions that
change the state of the cloud.
Proposed change
===============
This change will create a new (mix-in) class called ExecutionDriver that has
one main function: execute(name, positional-args, named-args). That interface
takes the name of the action, a list of its positional arguments, and a
dictionary of its named arguments and executes that action on the cloud
service.
This change will also overload the functionality of
policy/runtime.py:Runtime.execute()
in
policy/dseruntime.py:DseRuntime.execute()
so that a call to 'execute' by the policy engine will send a DSE message to
the appropriate service to execute the ExecutionDriver:execute() function.
This will be a unicast message from the policy engine to the appropriate
service on the DSE. That service will be included in the name of the action,
e.g. 'nova:disconnectNetwork' is an action that is sent to the 'nova' service.
As a starting point, this change will include an implementation of the
ExecutionDriver mixin within datasources/nova_driver.py
(or another service that the primary assignee chooses).
Alternatives
------------
An alternative is to make the ExecutionDriver class not a mix-in but
a standalone class, thereby having two separate files for each
cloud service, e.g. nova_datasource_driver.py and nova_executor_driver.py.
Treating the ability to execute a service as an interface is beneficial
because in reality services typically have some functionality that allows
them to read the state of the cloud and other functionality that allows
them to change the state of the cloud. Treating them as different files
makes them seem to be separate services.
Another alternative choice in the design is to use publish/subscribe to
communicate action execution between the policy engine and the instance
of ExecutionDriver. Publish/subscribe has the drawback that it could
result in multiple services subscribing to the same action (perhaps by
accident) and a single 'execute' command producing several API calls
modifying the state of the cloud. The unicast choice eliminates this
as a potential problem (though admittedly eliminates the possibility
of a use case where there are legitimately 2 different services that
need to be informed of the action).
Policy
------
None
Policy Actions
--------------
This is a step toward reactive enforcement.
Data Sources
------------
Eventually all data sources will want to implement the ExecutionDriver
interface.
Data model impact
-----------------
None.
REST API impact
---------------
We will want to expose the policy/dse_runtime.py:DseRuntime.execute()
command via the API.
Question: Should we rename the "data-sources" in the API to "cloud-services"?
If so, then we'd have...
Execute an action on a specified service. ::
POST v1/cloud-services/nova?action=execute -d {'name': 'disconnectNetwork',
'args': ['vm123', 'net456'],
'options': {}}
Execute an action as if the given policy dictated it should be executed. ::
POST v1/policies/alice?action=execute -d {'name': 'nova:disconnectNetwork',
'args': ['vm123', 'net456'],
'options': {}}
These API calls will have an empty response (HTTP code 204 perhaps) or an
appropriate error response if there was an error raised *before* the API call
was actually made. These API calls will return before the call we are
executing returns.
Security impact
---------------
This change gives anyone with the ability to write policy the ability to
execute API calls using the same rights Congress has been granted. Since
we typically run Congress with administrative rights, we need to secure
Congress authentication properly. But that is already accomplished because
we use the standard Keystone authentication system.
Notifications impact
--------------------
None
Other end user impact
---------------------
python-congressclient would need a new endpoint if we expose the execute()
functionality via the API.
Performance impact
------------------
Because the DatasourceDriver and ExecutionDriver will most often be implemented
as part of the same DSE instance, a poorly written ExecutionDriver could
impair the ability of that driver to properly read information via the
DatasourceDriver interface. The two pieces of functionality
run in a single thread, i.e. either one runs or the other runs. This is
actually beneficial because we would not want the driver to be pulling new
data at the same time it executes a change in that data.
Other deployer impact
---------------------
None
Developer impact
----------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
<launchpad-id or None>
Other contributors:
<launchpad-id or None>
Work items
----------
- Rename folder congress/datasources to congress/cloudservices and change
etc/congress/datasources.conf.sample appropriately. There may be other
places that reference 'datasources' explicitly.
- Add congress/cloudservices/execution_driver.py to include the mixin class
ExecutionDriver
- Implement policy/dseruntime.py:DseRuntime.execute() to send a unicast
message to the appropriate service or raise an error if that service does
not exist on the bus.
- Implement the ExecutionDriver for an existing service, e.g. Nova.
Dependencies
============
None
Testing
=======
- Unit tests that ensure a call to DseRuntime.execute() invokes the appropriate
ExecutionDriver.execute().
- Tempest tests that invoke execute() and ensure the proper change actually
happens.
Documentation impact
====================
Need to add description of actions and action format to docs, along with
documentation for execute().
References
==========
None

149
specs/kilo/add-extract-fn-to-all-translators.rst
View File

@ -1,149 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Add Extract-fn To All Translators
==========================================
https://blueprints.launchpad.net/congress/+spec/add-extract-fn-to-all-translators
This proposal would expand the general functionality of the translate_obj()
method by adding a general extract-fn method to translators for HDICT, VDICT,
and LIST. This gives datasource developers versatile tool to use when
translating data into congress congress drivers.
Problem description
===================
Today, if a datasource returns an object that contains a json string instead
of pure python objects, lists, and dicts, then the translator has no way to
read data out of the json object.
Proposed change
===============
By including the extract-fn method as a translator, the translate_obj() method
could then preprocess the data object with extract-fn before running the
existing translation code. This way,the programmer can set the extract-fn to
a json extractor to convert the json into a python object.
Alternatives
------------
Currently if a datasource returns data in a non-standard format a driver
developer can create unique methods inside their drivers to handle this data.
This proposed change potentially eliminates the need for developers to
individually generate code to use in their solutions by providing a tool to
assist with translations in base driver class.
Policy
------
None
Policy actions
--------------
None
Data sources
------------
None
Data model impact
-----------------
None
REST API impact
---------------
None
Security impact
---------------
None
Notifications impact
--------------------
None
Other end user impact
---------------------
None
Performance impact
------------------
None
Other deployer impact
---------------------
None
Developer impact
----------------
This change provides developers more versatility when creating datasource
drivers.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
Alex Yip
Other contributors:
Conner Ferguson
Work items
----------
Add extract-fn as a parameter to HDICT, VDICT, and LIST translators.
Dependencies
============
None
Testing
=======
To test the results of this change, data will need to be passed through the
extract-fn translator then this data will be checked to ensure that it is
being displayed in the expected manor.
Documentation impact
====================
Minor additions to the datasource driver based class documentation to inform
new developers about what the extract-fn method does as a translator.
References
==========
None

152
specs/kilo/add-in-list-type-for-hdict.rst
View File

@ -1,152 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Add in list support for hdict
==========================================
https://blueprints.launchpad.net/congress/+spec/add-in-list-support-hdict
This blueprint is to add support for a new translator value called in-list
which is useful in translating a hdict that contains another hdict which
is a list.
Problem description
===================
Several datasources return data in the format of a dict(list(dict()) (for
example neutron ports for fixed_ips)::
{
"ports": [
{
"status": "DOWN",
"id": "11ce1474-e395-4bda-b48c-820f0d542acd",
"fixed_ips": [
{
"subnet_id": "e56370ba-d255-486c-9907-ad4c6aed0241",
"ip_address": "10.2.0.2"
}
]
}
]
}
congress currently does not have an easy way to deal with data in this format
unless an extra table is created. This blueprint is to provide an
easy generic way to do so without creating the extra table.
NOTE: I only plan to implement this at first for hdict because we believe we
might be able to remove the vdict type as it might not be needed. If it
turns out that we need vdict I'll implement this for vdict as well.
Proposed change
===============
To implement this solution I propose we add a new in-list attribute to the
child translator. If present, the datasource driver class will just simply
interate this list and treat each list element as an hdict.
Alternatives
------------
We could force a datasource driver to manually do this themselves though it
would be better to automatically handle this for a user.
Policy
------
N/A
Policy Actions
--------------
N/A
Data Sources
------------
N/A
Data model impact
-----------------
New element 'in-list' can be specified in the translator.
REST API impact
---------------
N/A
Security impact
---------------
N/A
Notifications impact
--------------------
N/A
Other end user impact
---------------------
N/A
Performance impact
------------------
N/A
Other deployer impact
---------------------
N/A
Developer impact
----------------
N/A
Implementation
==============
Assignee(s)
-----------
Primary assignee:
arosen
Work items
----------
Implement this feature.
Dependencies
============
* This needs to be done before we can refactor the neutron driver to remove
the subtables that it creates.
Testing
=======
Unit tests.
Documentation impact
====================
Will document.
References
==========
N/A

188
specs/kilo/basic-high-availability.rst
View File

@ -1,188 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
===============================
Support basic high availability
===============================
Include the URL of your launchpad blueprint:
https://blueprints.launchpad.net/congress/+spec/basic-high-availability
Congress needs to support high availability (HA) for API requests so that
a client can continue to make successful API requests to Congress even if a
congress server becomes unavailable. This proposal describes a basic HA
solution that replicates the entire Congress server as-is. Each replica runs
the policy engine, contains all the table data, and runs the datasource
drivers.
Problem description
===================
Today, Congress runs as a single standalone server. That single server is
responsible for handling all API queries and is a single point of failure. If
the server fails, it will cause downtime for clients that integrate Congress.
Proposed change
===============
This spec proposes to
* Replicate the entire Congress server
* Use an off the shelf load balancer to distribute requests to the replicas
and avoid failed replicas
* Write API calls modify the database
* Each replica periodically checks the database for changes to policy or
datasources
Alternatives
------------
A more advanced design separates the policy engine from the datasources, and
replicates the policy engine N times, but uses a master-standby configuration
for the datasource driver. This way only the master datasource driver talks
to datasources thus reducing the load on datasources. The datasource
communicates incoming data changes to the replica over a message bus. This
change would require more code changes to separate the engine from the
datasources and to change how the messaging bus works.
Yet another proposal is to funnel all datasource updates to a central machine
which would precompute materialized views of all tables. This has the
advantage of giving the replicated Congress API consistency, but it relies on
a single machine to compute the materialized views, and relies on
materializing the views which stores all intermediate table content in memory,
which can consume an unmanagable amount of memory with many intermediate
tables. This alternate proposal would also require a significant amount of
code changes.
Policy
------
None
Policy actions
--------------
None
Data sources
------------
We need to ensure that each data service (such as Nova or Neutron) can accept
and handle requests from more than one datasource driver instance at the same
time since each replica will be fetching data from each data service. In
other words, if there are N replicas, then each data service must respond with
all the data separate N times, and the data service must be able to cope with
that higher request load.
Data model impact
-----------------
None
REST API impact
---------------
None
Security impact
---------------
None
Notifications impact
--------------------
None
Other end user impact
---------------------
Two API calls may return different data if a different replica serves each API
call because both the data from the datasources and the policy rules may be
out of sync between two replicas. The rate at which each replica checks the
database for updates can limit the problem for policy rules, but data skew
will still affect the replicas.
Performance impact
------------------
This change should improve throughput for the Congress server since there can
be multiple replicas instead of a single server. However, there may be an
impact on datasources since each replica will be requesting data from the
datasource. The period database requests to check for updates should have a
minimal impact on performance.
Other deployer impact
---------------------
None
Developer impact
----------------
All shared state must be stored in the database, and periodically checked at
all replicas.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
ayip@vmware.com
Work items
----------
* Add period check for database updates and add a test
* Add a test that starts two replicas and queries both
Dependencies
============
None
Testing
=======
Start two replicas, using the same database. Write a policy change on one
replica and check that the policy change occurs on the second replica.
Start two replicas, and kill one. Make sure the second replica can still
serve requests. Start first replica again and make sure it can still serve
requests.
Documentation impact
====================
We should add a description of how to configure Congress in HA mode, with a
load balancer and a shared database.
References
==========
None

256
specs/kilo/datalog-column-names.rst
View File

@ -1,256 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Example Spec - The title of your blueprint
==========================================
Include the URL of your launchpad blueprint:
https://blueprints.launchpad.net/congress/+spec/datalog-column-names
Writing policy can be challenging when tables have many columns.
This spec aims to give end users the ability to write policy by
referencing column names directly in Datalog rules.
Problem description
===================
The main problem is that wide tables (those with many columns) are hard to use.
(a) it is hard to remember what all the columns are,
(b) it is easy to mistakenly use the same variable in two different tables in
the body of the rule,
i.e. to create an accidental join,
(c) changes to the datasource drivers can require tedious/error-prone
modifications to policy.
Proposed change
===============
Today people write policy statements as follows:
error(net) :-
neutron:networks(x0,x1,net,x3,x4,x5,x6,x7,x8,x9,x10),
...
This change enables an alternative syntax where columns are referenced by name
instead
of position.
error(net) :-
neutron:networks(id=net),
...
To implement this change, we will expand the grammar to accept column-name
references and modify the AST constructor to translate the rules from
column-name references back into position references.
Alternatives
------------
There were 2 alternatives described in a ML discussion.
2) Be disciplined about writing narrow tables and write
tutorials/recommendations demonstrating how.
Instead of a table like...
neutron:ports(port_id, addr_pairs, security_groups, extra_dhcp_opts,
binding_cap, status, name, admin_state_up, network_id, tenant_id, binding_vif,
device_owner, mac_address, fixed_ips, router_id, binding_host)
we would have many tables...
neutron:ports(port_id)
neutron:ports.addr_pairs(port_id, addr_pairs)
neutron:ports.security_groups(port_id, security_groups)
neutron:ports.extra_dhcp_opts(port_id, extra_dhcp_opts)
neutron:ports.name(port_id, name)
...
People writing policy would write rules such as ...
p(x) :- neutron:ports.name(port, name), ...
[Here, the period e.g. in ports.name is not an operator--just a convenient
way to spell the tablename.]
To do this, Congress would need to know which columns in a table are sufficient
to uniquely identify a row, which in most cases is just the ID.
Pros:
(i) this requires only changes in the datasource drivers; everything else
remains the same
(ii) still leveraging database technology under the hood
(iii) policy is robust to changes in fields of original data
Cons:
(i) datasource driver can force policy writer to use wide tables
(ii) this data model is much different than the original data models
(iii) we need primary-key information about tables
3) Enhance the Congress policy language to handle objects natively.
Instead of writing a rule like the following ...
p(port_id, name, group) :-
neutron:ports(port_id, addr_pairs, security_groups, extra_dhcp_opts,
binding_cap, status, name, admin_state_up, network_id, tenant_id,
binding_vif, device_owner, mac_address, fixed_ips, router_id, binding_host),
neutron:ports.security_groups(security_group, group)
we would write a rule such as
p(port_id, name) :-
neutron:ports(port),
port.name(name),
port.id(port_id),
port.security_groups(group)
The big difference here is that the period (.) is an operator in the language,
just as in C++/Java.
Pros:
(i) The data model we use in Congress is almost exactly the same as the data
model we use in Neutron/Nova.
(ii) Policy is robust to changes in the Neutron/Nova data model as long as
those changes only ADD fields.
(iii) Programmers may be slightly more comfortable with this language.
Cons:
(i) The obvious implementation (changing the engine to implement the (.)
operator directly is quite a change from traditional database technology.
At this point, that seems risky.
(ii) It is unclear how to implement this via a preprocessor (thereby
leveraging database technology). The key problem I see is that we would need
to translate port.name(...) into something like option (2) above. The
difficulty is that TABLE could sometimes be a port, sometimes be a network,
sometimes be a subnet, etc.
(iii) Requires some extra syntactic restrictions to ensure we don't lose
decidability.
(iv) Because the Congress and Nova/Neutron models are the same, changes to
the Nova/Neutron model can require rewriting policy.
Policy
------
error(net) :-
neutron:networks(id=net)
Policy Actions
--------------
None
Data Sources
------------
None
Data model impact
-----------------
None
REST API impact
---------------
String-arguments to API calls representing policy can use either
position or column syntax.
Security impact
---------------
None
Notifications impact
--------------------
None
Other end user impact
---------------------
Other UIs can expose this enhanced syntax, but since currently all UIs
just pass policy statements as strings, they will require no actual changes
to leverage the enhancement.
Performance impact
------------------
Should have no performance impact, with the possible exception that
eventually we will want to reverse the preprocessing step for tracing
so that we present users with a more intuitive trace.
Other deployer impact
---------------------
None
Developer impact
----------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
thinrich
Work items
----------
* Modify grammar
* Make datasource schemas available to runtime
* Add preprocessor to rule AST constructor to convert column references into
positional references.
Dependencies
============
The following change makes datasource schema available to policy engine
Change-Id: I7cfbd82c721509634c0acfd51e66031af2ed7f2d
Testing
=======
No tempest tests are necessary. Unit tests only.
Documentation impact
====================
Should include modifications to docs to simplify the examples that use
tables with many columns. Will also require modifying the section where
we introduce Datalog.
References
==========
Mailing list discussion:
* http://lists.openstack.org/pipermail/openstack-dev/2014-August/041862.html

340
specs/kilo/datalog-ng.rst
View File

@ -1,340 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========
Datalog-ng
==========
Include the URL of your launchpad blueprint:
https://blueprints.launchpad.net/congress/+spec/datalog-ng
Congress needs an easy-to-use declarative language for expressing policy. This
design specification describes Datalog-ng, a version of standard Datalog that
has some extra features that are required for data querying and manipulation in
general and declarative networking in particular.
Problem description
===================
Congress needs an expressive policy language that is both easy-to-use as well
as efficient for information processing and declarative networking. Datalog is
a declarative language that provides a higher-level abstraction for querying
graphs and relational structures. It defines an efficient, recursive, query
execution and incremental data update mechanisms based on the relational model.
This design specification specifies a formal EBNF grammar for Datalog-ng, an
extension of Datalog that 1) enables tables to be referenced more easily in
queries, 2) defines syntax for adding and removing facts from the policy
database, 3) defines syntax for rules, queries, and constraints.
Proposed change
===============
This specification proposes a generalization of Datalog, called Datalog-ng,
which is suitable for declarative networking. The grammar will be defined in
Extended Backus-Naur Form (EBNF) [2]_. An EBNF specification is
platform-neutral, and formally defines the syntax of a language. Note that the
full expressive power of EBNF will NOT be used; this enables access to a wider
variety of tools that may not be able to understand all of the EBNF standard.
Hence, a wider variety of tools can be used to implement a parser, an
interpreter, and/or a compiler. See [3]_ for more information on the benefits
of formally defining a grammar.
Change #1: Table Referencing
[1]_ says: "Conceptually, Datalog describes policy in terms of a collection
of tables." Tables are a simple way of conveying information, and lend
themselves to querying, editing, and reporting. Policy rules can be thought of
as how input from one or more tables can be transformed into output in one or
more tables. Tables are full-fledged objects, so this enables us to not only
reuse tables (i.e., the actual data), but also reuse the policies that created
the data. However, tables that have a large number of columns are hard to use,
since there is no simple way to reference their name, or even know the number
of columns they have. In addition, there is no way for a policy to know about
changes in the table, so it would likely break as well.
Change #2: Fact Semantics
Datalog (and Datalog-ng) work by adding facts into, and removing facts from,
the database. However, there is no standard way for a policy author to
introduce semantics associated with these operations. For example, if a fact is
removed by one policy rule, it could adversely affect other policy rules that
were going to use that rule.
Change #3: Query and Rule Semantics
Formal semantics need to be added to standardize how queries and rules are
expressed and evaluated. Currently, there is no query semantics, but this is
very easy to add.
Change #4: Constraints
Currently, there is no way to express constraints in Datalog. For example, if
table A has four columns, and if one of the columns must not be null, then
there is no way to check that an insert of three values into a row will fail or
not. Similarly, if there are semantic constraints (e.g., row #3 must use one of
a set of enumerated values), or data type constraints, incorrect entries will
either be very hard or impossible to check, and will fail when applied. This
may be delayed, due to its difficulty and potential to introduce problems.
Change #5: Safety
Currently, there is no way to guarantee rule safety. For example, the following
queries are NOT safe: ::
foo(X, Y, Z) :- rel1(X, y) & X < Z
foo(X, Y, Z) :- rel1(X, Y) & NOT rel2(X, Y, Z)
foo(X, Y) :- rel1(X)
Change #6: Grammatical Improvements
Datalog is powerful, but somewhat hard to use. A set of "syntactical sugar"
will be defined to make Datalog easier to use, especially for the native Python
developer.
Alternatives
------------
N/A
Policy
------
Traditional policy rules take the form of statements that have either two or
three clauses [5]_ ; they are called ECA (event-condition-action) and CA
(condition-action): ::
ON <event-clause> IF <condition-clause> THEN <action-clause>
or
IF <condition-clause> THEN <action-clause>
In both ECA and CA, each clause can be a Boolean combination of atoms. However,
there are other types of policy rules:
• Goal policies
• Utility functions
• Promises
[6]_ covers the first two, and [7]_ is the latest of Mark's publications on
promise theory. All three of the above are different in form and function than
ECA and CA policy rules. Datalog-ng can model the intent of most of these forms
of policy rules, which is what is needed in Congress – the ability to
declaratively specify intent.
Policy Actions
--------------
Uses of Congress Policy Rules
Possible candidates include:
* Monitoring
* Reporting (including filtering for selected values, so the user is not
inundated with data)
* Configuring a device reactively (e.g., a threshold was violated)
* Configuring a device proactively (e.g., trending analysis predicts that a
threshold will be violated in the future)
The first three are straightforward; the latter may be pushed beyond the Kilo
release.
Policy Rule Implementation Alternatives
The advantages of Datalog are that it is a declarative subset of first-order
logic. Declarative languages express the logic in a task without specifying the
flow of control to perform the task. First-order logic is a formal system of
logic in which each statement consists of a subject and a predicate. A
predicate can only refer to a single subject. Sentences are combined and
manipulated using the same rules as those used in Boolean algebra. Two
quantifiers exist: "for all" and "for some" (higher-order logics have
additional quantifiers, such as "for every property of an object").
Datalog is thus more powerful than simple propositional logic, but not as
powerful as first-order logic. However, it provides a combination of power and
simplicity that is hard to match.
Data Sources
------------
This section will describe where the data is coming from. It will define
exemplar projects that produce these data.
Data Sinks
This section will describe who is consuming the data. It will define
exemplar
projects that consume these data.
Future Extensions
Since Datalog-ng is based on a subset of first-order logic, it also provides
formal reasoning and analysis. This is most likely the subject for the next
version of this specification, but it should be kept in mind so that we do not
limit this feature in any way.
Data model impact
-----------------
N/A
REST API impact
---------------
N/A
Security impact
---------------
Policy can contain the proverbial "keys to the kingdom". So, if someone
hacks their way into the system and can start issuing policies, game over.
Therefore, some type of access control should be used with policy-based
systems.
Notifications impact
--------------------
N/A
Other end user impact
---------------------
Datalog-ng is intended for Developers and Administrators, not End Users.
Performance impact
------------------
N/A
Other deployer impact
---------------------
For this to work in a secure manner, I really think we need to operate in a
secure environment, such as role-based access control (RBAC).
Developer impact
----------------
Implementation will provide policy writers with a richer set of options
Implementation
==============
Assignee(s)
-----------
Primary assignee:
straz
Other contributors:
thinrichs, sarob
Work items
----------
The following is a short description of how the above changes will be addressed
in this specification.
Change #1: Table Referencing
Table referencing will be supported by introducing namespaces into the policy
grammar, as well as supporting syntax, to allow tables and table elements to be
referred to by name and/or relative position.
Change #2: Fact Semantics
New syntax will be added to differentiate between adding and removing a fact.
This will enable optional rules to identify these operations and perform
additional tasks, if required.
Change #3: Query and Rule Semantics
This change involves adding dedicated syntax for differentiating between rules
and queries.
Change #4: Constraints
There are many types of constraints that can be enforced. The first set of
examples comes from relational theory. For the following definitions, assume
that a table represents an entity, such as a router or a network. Note that
these types of constraints are critical for being able to safely reference
different tables from different namespaces.
* Entity integrity: a row has a unique identifier, called a primary key. The
primary key is unique and not null. This enables each row in the entity to
be identified.
* Referential integrity: sometimes, tables reference other tables. A foreign
key is a set of attributes in one table that uniquely identifies a row of
another table a primary key from one table that appears in another table.
Referential integrity defines the dependency of one table on another table.
* Value constraints: data has constraint(s) on the value(s) that it can take
on. For example, a physical chassis can only be mounted in a single rack.
* Domain constraints: entity attributes in a given domain are restricted in
one or more ways. For example, the sum of two values from two columns in the
same row is less than or equal to another value. This generally include data
type, data value, and the defaulting of values.
The second set of examples comes from applications of security. In this view, a
constraint is an assertion that needs to be satisfied. The typical example is
that a policy should be able to specify the resources that a user (or
application) can access, as well as the set of operations that this user can
perform on that set of resources. For example, specifying permissions for all
sub-directories and files under a given directory in intractable in Datalog,
because the set of resources could be infinite, and Datalog does not have
function symbols.
This will be defined using constraint domains, and is an optional part of the
Datalog-ng language. It may be pushed out beyond Kilo if it becomes too
difficult.
Change #5: Safety
A rule is safe if all of the variables in the head of the rule also occur in a
positive, non-arithmetic literal in the body of the rule. This guarantees
termination of the rule. The grammar should include safety checks to protect
developers from themselves. :-)
Change #6: Grammatical Improvements
A set of grammatical improvements will be defined to simplify the use of
Datalog-ng, and especially to make its syntax friendlier to Python developers.
Examples include more recognizable comments (e.g., familiar "//" or
"/*..*/" instead of the native Datalog '%'), the ability to use single
and/or double quotes, and English equivalents to some commands
(e.g., '!' or 'NOT' or 'not').
Dependencies
============
This spec may be broken up into multiple blueprints for implementation. The
list of spec and/or blueprints will be listed at the top of this spec.
Testing
=======
N/A
Documentation impact
====================
N/A
References
==========
The following are references for this specification.
.. [1] Congress Design, http://goo.gl/YFd2Fr
.. [2] ISO/IEC, "Information technology - Syntactic metalanguage - Extended
BNF", 14977, 12/15/1996
.. [3] J. Strassner, "A Gentle Introduction to EBNF", TBD
.. [4] Congress Policy Workshop, TBD
.. [5] J. Strassner, "Policy Based Network Management", Morgan Kaufman
Publishing, 978-1558608597, 9/2003
.. [6] J. Strassner, J. Kephart, "Autonomic Systems and Networks – Theory and
Practice", NOMS 2006 Tutorial
.. [7] M. Burgess, J. Bergstra, "Promise Theory - An Introduction", xtAxis
Press, 2014

168
specs/kilo/dse-control-bus.rst
View File

@ -1,168 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
======================
Add Control Bus to DSE
======================
https://blueprints.launchpad.net/congress/+spec/dse-control-bus
Nodes participating on the data services engine (DSE) bus need a means of
discovering each other and their exposed data types. Currently a periodic
broadcast mechanism is used for this purpose. While effective, these
broadcasts are inefficient, and complicate debugging.
Instead of using the exiting broadcast-based discovery mechanism, nodes on
the DSE should publish events on a well-known "collections registry,"
that every node subscribes to.
Problem description
===================
Distribution of DSE endpoint information is currently performed using an
inefficient broadcast mechanism. Instead, this information should be
delivered only when changes occur, or when specifically requested.
Proposed change
===============
* All instances of deepSix will subscribe to a well-known table,
which we will refer to as the 'collection registry'.
* When the data indexes exposed by a deepSix instance changes (including
instance startup), the instance will publish an update to the collection
registry.
* Each deepsix instance will maintain a list of it's data_indexes that it has
a desired subscription for. The instance will do a lookup in it's local
copy of the collection registry to find associated endpoints to send
subscription requests to. As changes occur to the collection registry,
each instance will re-evaluate it's own subscriptions against these
updates.
* There is a collection registry per instance of the DSE. In a hierarchical
DSE, there could be multiple registries.
* The periodic broadcast of data_index information can be removed after
these changes.
Alternatives
------------
We can continue to use the existing mechanism as an alternative, but
difficulties with inefficiencies and complex debugging will grow substantially
as the number of nodes on the bus increases.
Policy
------
N/A
Policy Actions
--------------
N/A
Data Sources
------------
N/A
Data model impact
-----------------
N/A
REST API impact
---------------
N/A
Security impact
---------------
N/A
Notifications impact
--------------------
N/A
Other end user impact
---------------------
N/A
Performance impact
------------------
At runtime, this change will reduce the time it takes to distribute
information about new nodes in the system. It will also reduce the overall
"chatter" on the bus by eliminating unnecessary broadcasts. We anticipate
the reduction in chatter will greatly simplify debugging the message bus.
Other deployer impact
---------------------
N/A
Developer impact
----------------
N/A
Implementation
==============
Assignee(s)
-----------
Primary assignee:
None
Other contributors:
None
Work items
----------
TBD
Dependencies
============
N/A
Testing
=======
The existing testing framework is sufficient to test to change.
Documentation impact
====================
N/A
References
==========
N/A

148
specs/kilo/dse-dataservices-status.rst
View File

@ -1,148 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
=============================================
Get status from the cage about dataservices
=============================================
https://blueprints.launchpad.net/congress/+spec/dse-dataservices-status
Get metrics about the data services and have it available
through the api. This information will include things so that the policy engine
will know the status of the data services and when a data service is not able
to do its job correctly.
Problem description
===================
Currently each of the datasources implemented as nodes in the DSE have
2 fields for status: the time the datasource last refreshed its data and
the last error that was generated when it tried to refresh its data.
But there are additional metrics that would be
nice to have as part of the status of a datasource/DSE node, e.g. outstanding
messages, message rate, subscribers, subscriptions.
Proposed change
===============
This change will add a status() method to each DSE node that returns
a dictionary of useful information about that node. The node will
also compute those metrics and will be configurable so that metrics
that are expensive to compute can be turned off.
Alternatives
------------
The status() information could be implemented on top of DSE instead of
directly within DSE. But doing so would break the abstraction that
DSE provides: a message bus that takes care of message queues, message
delivery, queueing times, etc.
Policy
------
N/A
Policy Actions
--------------
N/A
Data Sources
------------
N/A
Data model impact
-----------------
N/A
REST API impact
---------------
None since the API already exposes a 'status' query for each datasource.
But the implementation of that API will change to incorporate the additional
status information provided by DSE.
Security impact
---------------
None
Notifications impact
--------------------
None
Other end user impact
---------------------
None
Performance impact
------------------
It will be possible to turn off high-cost status computation; thus, people
can decide whether each status item is worth the cost of computing it.
Other deployer impact
---------------------
None
Developer impact
----------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
<launchpad-id or None>
Other contributors:
<launchpad-id or None>
Work items
----------
- Add implementations for metrics. The decision as to whether the metric
is continually computed or computed on demand will be made on a
metric-by-metric basis.
- Add status() method to DeepSix class and return a dictionary of metrics.
Dependencies
============
None
Testing
=======
Unit tests will suffice: simulate activity in the DSE and check that the
status metrics are as expected. Ideally would also simulate metric computation
under load.
Documentation impact
====================
None
References
==========
None

165
specs/kilo/explicit-reactive-enforcement.rst
View File

@ -1,165 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Explicit reactive enforcement
==========================================
https://blueprints.launchpad.net/congress/+spec/explicit-reactive-enforcement
Enable policy writers to describe conditions under which Congress should
execute specific actions, e.g. when a network is connected to a VM that
contains a virus, disconnect it from the internet.
Problem description
===================
Currently Congress takes no direct actions to help a cloud obey
the policy that governs it. Today Congress can identify violations of
policy but it cannot correct them.
Proposed change
===============
This change enables policy writers to include policy snippets such as
the one below to encode instructions for how Congress ought to
react to changes in the cloud. It enables the policy writer to dictate
that Congress must execute certain actions under certain conditions.
// disconnect a VM from the network whenever it is infected by a virus::
execute[disconnectNetwork(vm, network=net)] :-
antivirus:infected(vm),
nova:network(vm, net)
Alternatives
------------
This same kind of functionality could be written in a traditional programming
language such as Python, instead of encoded as policy. But the benefit
of doing it as policy is that we can build analysis tools that help
policy authors understand if they have created infinite loops and other
undesirable featues.
Policy
------
See example above.
Policy Actions
--------------
The purpose of this blueprint is to implement a form of reactive enforcement.
Data Sources
------------
None
Data model impact
-----------------
None
REST API impact
---------------
None. This functionality is leveraged by writing policy statements,
which the existing API already handles properly.
Security impact
---------------
One potential security impact is that Congress typically runs as an
administrator and thus all the actions that Congress executes will be
as an administrator as well. Thus for the first time a security breach
into Congress means a security breach into all of the services that
Congress is permitted to execute actions on. Though in OpenStack,
a breach of security for Congress means a breach of Keystone,
which would still give administrative rights to the other OpenStack
services anyway.
Notifications impact
--------------------
None
Other end user impact
---------------------
None
Performance impact
------------------
None for this spec; other, dependent specs have performance impacts.
Other deployer impact
---------------------
None.
Developer impact
----------------
None.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
<launchpad-id or None>
Other contributors:
<launchpad-id or None>
Work items
----------
- Modify trigger framework to operate on modal operators directly, e.g. so
that we can register a trigger whenever execute[x] changes.
- Set up a trigger on execute[x]. When that trigger fires, it invokes
the action on the appropriate ExecutionDriver.
Dependencies
============
* modal-operators-for-policy: enable execute[] syntax
* policy-triggers: initial version of trigger framework
* action-execution-interface: add executor driver concept to Congress
Testing
=======
This change will require unit tests for the enhanced trigger framework.
It should include tempest tests to test end-to-end functionality.
Documentation impact
====================
This will require a new documentation section for reactive enforcement.
References
==========
None

151
specs/kilo/fact-datastructure.rst
View File

@ -1,151 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
=====================================
Create a Fact class for storing facts
=====================================
Include the URL of your launchpad blueprint:
https://blueprints.launchpad.net/congress/+spec/fact-datastructure
Today, the congress runtime stores facts as Rules. This is inefficient from a
memory perspective since each Rule contains a Literal, which in turn contains
a Term for each column. Each of these is a python object, and each of them
also contains a few extra fields like head, body, location, negated,
etc. Using Rules is also inefficient with CPU since congress needs to
construct all these objects. This blueprint proposes to use a Fact data
structure to store each fact. A Fact is a subclass of a native tuple, plus one
field for table name. This is much more efficient memorywise and CPUwise than
using a Rule because there are no extra objects like Literal and
Term. Preliminary tests show a 10x reduction in CPU for initializing tables
plus a 3x reduction in memory use.
Problem description
===================
A detailed description of the problem:
* Today, congress stores each fact as a Rule object
* A Rule object contains many objects and fields
* Many objects and fields means that creating and storing a fact uses lots of
CPU and memory resources.
* High CPU and memory use makes congress unable to scale to larger datasets.
Proposed change
===============
We propose to create a new class called Fact to store each fact. A Fact is a
native tuple plus one string for table name. Using a Fact will eliminate all
the subfields and subobjects in Rule.
Alternatives
------------
None
Policy
------
None
Policy Actions
--------------
None
Data Sources
------------
None
Data model impact
-----------------
None
REST API impact
---------------
None
Security impact
---------------
None
Notifications impact
--------------------
None
Other end user impact
---------------------
None
Performance impact
------------------
Preliminary testing shows a 10x reduction in CPU use and a 3x reduction in
memory use in initialize_table() for 7M facts where the payload is 700MB.
Other deployer impact
---------------------
None
Developer impact
----------------
A Theory object will internally contain Rules and Facts. The caller an insert
a Fact into a RuleSet. However, whenever someone fetches the rules from a
Theory the RuleSet converts Facts to Rules before returning them.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
ayip
Work items
----------
* Implement FactSet
* Use FactSet inside of RuleSet
* Change initialize_tables() to avoid instantiating a list of all facts coming
from DSE
Dependencies
============
None
Testing
=======
Add a unit test for FactSet
Documentation impact
====================
None
References
==========
None

116
specs/kilo/glance-datasource-driver.rst
View File

@ -1,116 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Glance Data Source Driver
==========================================
https://blueprints.launchpad.net/congress/+spec/glance-data-source-driver
This blueprint is to add a data source driver for glance.
Problem description
===================
N/A
Proposed change
===============
Add data source driver that integrates congress with glance.
Alternatives
------------
N/A
Policy
------
This will use the congress language. glance:image(X) etc.
Policy Actions
--------------
Just monitoring right now.
Data Sources
------------
Glance
Data model impact
-----------------
TBD
REST API impact
---------------
N/A
Security impact
---------------
N/A
Notifications impact
--------------------
N/A
Other end user impact
---------------------
N/A
Performance impact
------------------
N/A
Other deployer impact
---------------------
N/A
Developer impact
----------------
N/A
Implementation
==============
Assignee(s)
-----------
Primary assignee:
arosen
Work items
----------
Dependencies
============
python-glanceclient
Testing
=======
Will add tempest tests.
Documentation impact
====================
N/A
References
==========
N/A

158
specs/kilo/heat-datasource-driver.rst
View File

@ -1,158 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
===============================
Create a Heat Datasource Driver
===============================
Include the URL of your launchpad blueprint:
https://blueprints.launchpad.net/congress/+spec/heat-datasource-driver
This Heat driver will allow Congress to interact with the Openstack Heat API
for orchestration. The first version will provide data from Heat's read API
calls until Congress does has infrastructure to handle writing to drivers.
Subsequent versions may be able to send requests and write to the Heat API.
Problem description
===================
Today, there is no Congress driver for Heat, either for reading or writing.
This driver will give Congress eyes into Heat so that a policy writer can
inspect Heat state such as details about each stack, stack snapshots,
resources, software configurations, software deployments, and perhaps
templates themselves.
Proposed change
===============
This driver will be similar to other existing drivers like neutron_driver.py
and nova_driver.py. The Heat driver will read data from the following Heat
API calls and convert the responses to Congress tables:
* list stacks
* show stack detail
* list snapshots
* list resources
* show resource data
* show resource metadata
* list resource types
* show configuration details
* list stack events
* show event details
* show software_configs
* show software_deployments
Alternatives
------------
The alternative is to have no driver for Heat, which is not a good option for
those admins that use Heat in their cloud.
Policy
------
N/A
Policy actions
--------------
N/A
Data sources
------------
Openstack Heat
Data model impact
-----------------
N/A
REST API impact
---------------
N/A
Security impact
---------------
The Heat driver will need to authenticate to the Heat API just like all the
other datasource drivers.
Notifications impact
--------------------
N/A
Other end user impact
---------------------
N/A
Performance impact
------------------
It is possible that the API calls will be expensive. We will need to measure
the impact of the API calls on Heat and Congress performance.
Other deployer impact
---------------------
N/A
Developer impact
----------------
N/A
Implementation
==============
Assignee(s)
-----------
Primary assignee:
tengqim
Work items
----------
* Write unit test
* Write tempest test
* Write API client code
* Write translators for tables
Dependencies
============
N/A
Testing
=======
This work must include a unit test and a tempest test. The driver translator
infrastructure makes most of the translation code robust, but the driver is
still dependent on the Heat API, so the tempest test is particularly important
as an integration test.
Documentation impact
====================
N/A
References
==========
Blueprint:
https://blueprints.launchpad.net/congress/+spec/heat-datasource-driver

169
specs/kilo/horizon-create-policies.rst
View File

@ -1,169 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Policy creation in Horizon
==========================================
https://blueprints.launchpad.net/congress/+spec/horizon-create-policies
Congress policies can be created from the command line interface (CLI). Provide
a way to create policies from the Horizon dashboard that helps users who have
little or no experience with writing Datalog.
Problem description
===================
Users unfamiliar with Datalog may find it challenging to write rules for a
Congress policy.
Proposed change
===============
Add a way to create Congress policies in the Policies panel in Horizon that
does not require users to have knowledge of Datalog to write policy rules.
Something like the draggable blocks in Node-RED or BipIO could be moved around
and connected together by the user to construct a policy rule. Or have a
template that presents the Datalog pieces of a rule in a natural language
format.
Alternatives
------------
One alternative is to enhance the CLI to assist users in writing policy rules.
However, a graphical interface can have interactive mechanisms to help users
navigate the structure of a policy rule that a CLI cannot provide. For example,
rule components could be more easily represented with graphics in a web
interface vs. a CLI-based wizard, which is limited to text representations.
Policy
------
None.
Policy Actions
--------------
None.
Data Sources
------------
None.
Data model impact
-----------------
None.
REST API impact
---------------
None.
Security impact
---------------
Caution must be exercised when using data entered by a user. An example of a
type of security issue is SQL injections, if the user data is going to be
inserted into a database. User data should be sanitized to mitigate such risks.
Notifications impact
--------------------
None.
Other end user impact
---------------------
The end user will be able to go to the Policies panel in Horizon and use some
knobs to write policy rules and create a policy in Congress.
Performance impact
------------------
None.
Other deployer impact
---------------------
None.
Developer impact
----------------
None.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
jwy
Other contributors:
<launchpad-id or None>
Work items
----------
- Knobs and form to enter policy details and rules.
- Logic (e.g., autocompletion) to fill in policy table names, data source table
names, etc., that can be used in a policy rule.
- Backend to pass information on to Congress for policy creation.
- Form to enter Datalog directly for a rule, for users who are more comfortable
with the language.
Keep in mind that we may want to use the same UI for updating policies later.
Dependencies
============
None.
Testing
=======
Additional Tempest tests are not needed because the Congress code is not being
modified here.
Documentation impact
====================
None.
References
==========
Datalog Policy Language:
https://github.com/stackforge/congress/blob/master/doc/source/policy.rst#2-datalog-policy-language
Node-RED:
http://nodered.org
BipIO:
https://bip.io

145
specs/kilo/horizon-datasource-status.rst
View File

@ -1,145 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
=====================================================
Congress OpenStack Horizon : Data Source Status Table
=====================================================
https://blueprints.launchpad.net/congress/+spec/horizon-datasource-status-table
This blue print describes integration of datasources status with Horizon
dashboard. Admin will be able to see detailed status of each datasource.
Problem description
===================
A detailed description of the problem:
* In the existing horizon implementation congress data source status is not
shown
* This poses problem to admins who want to see the status of data sources
Proposed change
===============
* Proposal is to add data source status table to the landing page
contrib/horizon/datasources/templates/datasources/index.html
* Add appropriate table to the file /contrib/horizon/datasources/tables.py
Alternatives
------------
none
Screens
-------
none
Policy actions
--------------
none
Data sources
------------
none
Data model impact
-----------------
none
REST API impact
---------------
no updates required
Security impact
---------------
Authentication of python-congressclient through Keystone token
Notifications impact
--------------------
none
Other end user impact
---------------------
* User will be able to view Data Source Status
Performance impact
------------------
none
Other deployer impact
---------------------
none
Developer impact
----------------
none
Implementation
==============
Assignee(s)
-----------
Primary assignee:
<jwy>
<rajdeepd>
Other contributors:
<None>
Work items
----------
* Modify base Panel for Policies and Data sources tables
* Add congress.py to add calls for getting status information
* Create a Data Source status table
Dependencies
============
* Horizon
* python-congressclient
Testing
=======
Unit testing using mocks.
Documentation impact
====================
none
References
==========
https://github.com/stackforge/python-congressclient

191
specs/kilo/horizon-integration.rst
View File

@ -1,191 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
================================================
Congress OpenStack Horizon Dashboard Integration
================================================
https://blueprints.launchpad.net/congress/+spec/horizon-integration
This blue print describes integration of Congress with Horizon dashboard.
Admin should be able to create/update or view policies and rules. He should
also be able to see information fetched from the various data sources.
* ..
*
Problem description
===================
A detailed description of the problem:
* In the existing implementation congress data elements like policies, rules
and data sources can be accessed only through command line or python client.
* This poses problem to admins who want to use UI based dashboard to manage
policies and its associated elements
Proposed change
===============
* Proposal is to integrate Congress read/ write and update use cases into
Horizon
* A new panel named `policies` will be added to the `admin` dashboard.
+-----------+ +------------+
| ++------+ | |
| || | REST API Call| |
| || +------------------> |
| OpenStack || congress | Congress |
| Horizon || python-client | API |
| || | | Service |
| || | | |
| || | | |
| || | | |
| || | | |
| ++------+ | |
+-----------+ +------------+
* This Panel will have a Tab Group
* First Tab will be Policies Tab
* Second Tab will be DataSources Tab
Policies Tab will cover following use cases
* List of policies
* List of Rules in a Policy
* List of tables in a Policy
* Create a Rule
* Update a Rule
* Delete a Rule
DataSource Tab will show
* List of DataSources
* Tables returned by DataSources
Alternatives
------------
Implement a dashboard which is independent of Horizon, in case there is a need
to integrate congress in a non openstack scenario.
Screens
-------
none
Policy Actions
--------------
none
Data Sources
------------
none
Data model impact
-----------------
none
REST API impact
---------------
To be determined.
We might need some additional data to be exposed by python-congressclient
Security impact
---------------
Authentication of python-congressclient through Keystone token
Notifications impact
--------------------
none
Other end user impact
---------------------
* User will be able to view, configure and update Policies, Rules.
* User will be able to data exposed by the DataSources
Performance impact
------------------
none
Other deployer impact
---------------------
Integration with Devstack.
Developer impact
----------------
none
Implementation
==============
Assignee(s)
-----------
Primary assignee:
<jwy>
<rajdeepd>
Other contributors:
<None>
Work items
----------
* Add base Panel for Policies
* TabGroup for Policies and DataSources
* Implement Policy Tab
* Implement DataSource Tab
Dependencies
============
* Horizon
* python-congressclient
Testing
=======
Unit testing using mocks.
Documentation impact
====================
Document the screenflow.
References
==========
https://github.com/stackforge/python-congressclient

230
specs/kilo/horizon-policy-abstraction.rst
View File

@ -1,230 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
===========================================
Abstraction policy definitions in Horizon
===========================================
https://blueprints.launchpad.net/congress/+spec/horizon-policy-abstraction
Congress aims to provide an extensible open-source framework for governance
and regulatory compliance across any cloud services. Users can define policies
in congress by pure Datalog. But Datalog, working with data-tables, is a very
difficult and complex description for policies. Users may have some problems
to understand and deploy policies by Datalog. So this specification provides
an abstraction for policies, and shows it in an abstraction form in Horizon,
which will facilitate users to express their policies.
Problem description
====================
Datalog is not intuitive to use, even difficult for users to express their
policies who are not familiar with it. And it may cause some misinterpretation
when translating real intent to Datalog because of the complex logic.
Proposed change
===============
Congress makes the whole cloud compliant by defining violation state and action
for violation.
Policies in Congress can be expressed by BNF as below.
::
Congress Policy ::= violation condition, "do" action for violation
So, policy abstraction is to abstract violation state and corresponding action
to make the policy more intuitive and easy to use.
By analyzing typical scenarios, violation mainly can be divided into two parts.
One is the constraint of objects' attributes, and another is the constraint
of relationship between several objects' attributes.
All the objects and constraints are not just a simple set of data source
tables, but they can be divided into some categories according to their
functions and relations. So users just need to choose objects they care about
without worrying about which tables they are in.
The violation condition can be expressed by BNF as below.
::
violation condition ::=object attribute constraint (value | object attribute)
object attribute::=object "." attribute
For any violation state, congress will take some actions, such as monitoring,
proactive and reactive. Though congress has just realized monitor violation,
changing cloud state to make the cloud compliant is also an important function
for congress. So, policy abstraction will provide some optional reactive
actions for different objects to resolve violations.
The action for violation state can be expressed by BNF as below.
::
action ::= ("monitor"| "proactive"| "reactive") data
So policies in Congress can be abstracted into "name", "objects",
"violation condition", "action" and "data".
Among these, element "name" defines a marker of a policy, which is used to
be a unique identification for a policy.
Element "objects" defines all objects which are concerned by this policy.
They are not just simple display of data source tables, but a organized set
which contains the relationship between different objects, such as, "servers",
"networks", "hosts", "subnets", etc.
For example, table "statistics" will not appear as a object, but it will be
a attribute of other objects, such as, "servers", "networks".
Another example is users could choose "servers" and "networks" without caring
about what put them together ("ports", actually).
Element "violation condition" defines the state of objects' attributes
which can produce violation, and the constraint will include comparison,
arithmetic and some predefined relationship/functions, such as, "same_group".
Element "action" defines the action needs to take for this policy,
and actions will include "proactive", "monitoring" and some specific actions,
such as, "remove", "delete". All these actions depend on the ability of
underlying components.
Element "data" defines the information gotten or needed when executing the
action, for example, when monitoring a servers violation, users can define
"data" as servers' name to be a return parameters.
That is, Congress UI will provide many elements of policies as drop-down lists,
and the combination of these elements will form various policies. All these
elements are from the summary of typical scenarios and the ability of
underlying components.
Users need to choose which one can match their needs.
Then UI will translate the information in UI into Datalog.
Alternatives
------------
N/A
Policy
-------
There is one example to express typical policy by abstraction form in Horizon.
Example: every network connected to a VM must either be public or
owned by someone in the same group as the VM.
For this example, users care about "servers" and "networks", so users will
choose these two objects from a drop-down list.
After users decide the objects, corresponding optional violation state will
be decided, which will include these two objects' attributes and some
predefined relationship, so users can choose "not same_group" and choose who
are not in the same group. All the choices will be show as drop-down lists,
too. And users need to choose the action and data for this violation.
For example, users choose "monitoring", attributes of servers and networks
will appear in "data".
In this policy, users can create a policy as below.
+----------+----------+-----------------------------------------------+------------+--------------+
| name | objects | violation state | action | data |
+----------+----------+-----------------------------------------------+------------+--------------+
| policy_1 | servers |not equal(networks.share, public) | monitoring | servers.name |
| | networks |not same_group(servers.tenant, networks.tenant)| | |
+----------+----------+-----------------------------------------------+------------+--------------+
Policy Actions
--------------
The action can be monitoring, proactive or some execute actions
which can make the cloud compliant.
Data Source
-----------
N/A
Data model impact
-----------------
N/A
REST API impact
---------------
N/A
Security impact
---------------
All parameters inputted by users need satisfy predefined standard, for example,
if values inputted in "violation condition" in reasonable range
(e.g. 0-100% for CPU utilization).
Notifications impact
--------------------
N/A
Other end user impact
---------------------
End users can be able to write policies in Horizon and use some drop-down lists
and some simple inputs to create a policy. Then Horizon will translate the
information in UI into Datalog, which will be processed in Congress.
Performance impact
------------------
N/A
Other deployer impact
---------------------
N/A
Developer impact
----------------
N/A
Implementation
==============
Assignee(s)
-----------
Primary assignee:
Yali Zhang
Other contributors:
Jim Xu; Yinben Xia
Work items
-----------
- Abstraction form to write policies rules and actions for policies.
- Build mapping relationship between abstraction form and Datalog,
so users can write a policy in UI other than Datalog.
- Pass information from Horizon to Congress to finish the policy creation.
Dependencies
============
N/A
Testing
=======
Need to be tested with a variety of scenarios.
Documentation impact
====================
Add instructions for policy abstraction in UI.
References
==========
N/A

158
specs/kilo/ironic-datasource-driver.rst
View File

@ -1,158 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==================================
Create an ironic Datasource Driver
==================================
Include the URL of your launchpad blueprint:
https://blueprints.launchpad.net/congress/+spec/ironic-datasource-driver
This ironic driver will allow Congress to interact with the Openstack ironic
API for orchestration. The first version will provide data from ironic's read
API calls until Congress does have infrastructure to handle writing to drivers.
Subsequent versions may be able to send requests and write to the ironic API.
Problem description
===================
Today, there is no Congress driver for ironic, either for reading or writing.
This driver will give Congress eyes into ironic so that a policy writer can
inspect Bare Metal state such as details about enabled drivers, each chassis,
each node, each port, supported boot devices of a node.
Proposed change
===============
This driver will be similar to other existing drivers like neutron_driver.py
and nova_driver.py. The ironic driver will read data from the following ironic
API calls and convert the responses to Congress tables:
* list enabled drivers
* show driver detail
* list driver properties
* list chassis
* show chassis detail
* list nodes contained in a chassis
* list registered nodes
* show node detail
* list supported boot devices for a node
* list ports assocaited with a node
* list all registered ports
* show port detail
Alternatives
------------
The alternative is to have no driver for ironic, which is not a good option for
those admins that use ironic in their cloud to manage bare metal servers.
Policy
------
N/A
Policy actions
--------------
N/A
Data sources
------------
Openstack ironic
Data model impact
-----------------
N/A
REST API impact
---------------
N/A
Security impact
---------------
The ironic driver will need to authenticate to the ironic API just like all the
other datasource drivers.
Notifications impact
--------------------
N/A
Other end user impact
---------------------
N/A
Performance impact
------------------
It is possible that the API calls will be expensive. We will need to measure
the impact of the API calls on ironic and Congress performance.
Other deployer impact
---------------------
N/A
Developer impact
----------------
N/A
Implementation
==============
Assignee(s)
-----------
Primary assignee:
zhenzan-zhou
Work items
----------
* Write unit test
* Write tempest test
* Write API client code
* Write translators for tables
Dependencies
============
N/A
Testing
=======
This work must include a unit test and a tempest test. The driver translator
infrastructure makes most of the translation code robust, but the driver is
still dependent on the ironic API, so the tempest test is particularly
important as an integration test.
Documentation impact
====================
N/A
References
==========
Blueprint:
https://blueprints.launchpad.net/congress/+spec/ironic-datasource-driver

198
specs/kilo/modal-operators-for-policy.rst
View File

@ -1,198 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Add modal operators to policy language
==========================================
https://blueprints.launchpad.net/congress/+spec/modal-operators-for-policy
To express access control policies and reactionary enforcement policies, we
need modal operators like 'execute' and 'permit' added to the language. Modal
operators are identifiers that apply to literals or rules in the language and
hence cannot be encoded (naturally) without special syntactic support.
Problem description
===================
Example policy fragments that are enabled by this change:
1) Reactionary policies: describing what actions/API-calls Congress should
execute and when.
execute[nova:disconnectNetwork(vm, net)] :-
nova:virtual_machine(vm),
nova:network(vm, net),
bad_network(vm, net)
2) Access control policies: describing when other components are permitted
to execute certain actions/API calls.
permit[nova:disconnectNetwork(vm, net)] :-
nova:owner(vm, owner),
role(owner, 'admin')
3) Action descriptions: describing the effects of an action
delete[nova:network(vm, net)] :-
execute[nova:disconnectNetwork(vm, net)]
Proposed change
===============
This change requires modifying the grammar to so that the constructs above
are permitted. To do this, we would want to make the definition for 'rule'
into something like the following.
rule ::= modal_list COLONMINUS modal_list
modal_list ::= modal (COMMA literal)*
modal ::= literal | ID LBRACKET literal RBRACKET
We also need to change compile.py:Literal class to have a field 'modal' that
is either None or is an ID (string).
We also need to change the unifier so that it checks the 'modal' field when
doing unification.
Finally we need to introduce functionality so that we can ask for all
x such that execute[x] is true in the current state. One option is to
construct a special function in the Runtime; another option is to expand the
definition of a modal so that it can take a variable, such as shown below,
and modify the unifier and runtime.py:TopDownTheory.top_down_eval routines
appropriately.
modal ::= literal | ID LBRACKET literal RBRACKET | ID LBRACKET ID RBRACKET
Alternatives
------------
Today the way we write action descriptions is by using + and - as suffixes
on table-names to denote insert and delete respectively. This
does not extend well to other modals like 'execute' and 'permit'.
The prototype code already in place assumes it can identify what is to be
'execute'd and what is to be 'permit'ed based on the policy the rules
reside in. Having separate policies for implementing this functionality
is a poor solution because multiple policies ought to be able to represent
multiple policy authors' contributions to policy--a single policy ought to
be able to include reactive policy, access control policy, error policy,
and action-descriptions.
In addition, the functionality we currently use for dealing with
action-descriptions while implementing simulation uses the 'consequences'
construct to compute all of the literals inserted and deleted upon execution
of a given action. The 'consequences' functionality is computationally
expensive in that it computes the contents of all tables in the policy. Once
a single policy contains more than simply the action-descriptions, we cannot
afford to use the 'consequences' functionality.
Policy
------
See use cases above.
Policy Actions
--------------
N/A
Data Sources
------------
N/A
Data model impact
-----------------
None since rules are stored as strings in the DB.
REST API impact
---------------
None since policy snippets are passed as strings.
Security impact
---------------
None
Notifications impact
--------------------
None
Other end user impact
---------------------
None
Performance impact
------------------
None
Other deployer impact
---------------------
None
Developer impact
----------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
<launchpad-id or None>
Other contributors:
<launchpad-id or None>
Work items
----------
- Modify grammar as described above
- Change parser to read in new grammar
- Change runtime to properly unify with modals
- Add modal-level queries, e.g. find all x such that execute[x]
Dependencies
============
N/A
Testing
=======
Unit tests are sufficient. Ensure that the new syntactic constructs
can be used anywhere and that the runtime produces the proper results
when the new syntactic constructs are in place.
Documentation impact
====================
End-user documentation is not necessary for this change. But documentation
will be necessary for the functionality that uses this change, e.g.
the simulation() docs will change to describe using insert[]/delete[] instead
of +/- suffixes.
References
==========
None

289
specs/kilo/multiple-policies.rst
View File

@ -1,289 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Example Spec - The title of your blueprint
==========================================
https://blueprints.launchpad.net/congress/+spec/multiple-policies
Currently, there are only a couple of different policies people can use.
We want to enable people to utilize multiple policies for the purpose of
modularity and encapsulation.
Problem description
===================
We want users to be able to create and delete entire policies.
We want users to be able to choose which type of policy they want
to create.
We want users to write policies that reference tables defined in
other policies as naturally as they reference tables defined within
datasources.
Proposed change
===============
Users may create/delete any number of policies and give them names.
Each of these policies look like the classification policy we have
today.
Users can insert/delete rules into each such policy
just as they do with the classification policy today.
Users may write the same kind of rules they always have, except
they can reference tables defined in any other policy in
the body of rules.
The rules are prohibited from prefixing the tablename in the head
of a rule with a policy-name. The rules are prohibited from
recursing across theories.
1) Example
policy1:
p(x) :- policy2:q(x)
policy2:
q(x) :- nova:servers(x)
2) Non-example: do not include policy name in head of rule
policy:p(x) :- q(x)
3) Non-example: do not recurse across policies
policy1:
p(x) :- policy2:q(x)
policy2:
q(x) :- policy1:p(x)
Alternatives
------------
We're already doing this (conceptually) with the prefixes (such as
nova: and neutron:) on datasource tablenames. This proposal simply
generalizes that idea.
Another approach is to put everything into a single theory and explicitly
prefix all tables with the module in which they are defined.
Implementationally, that is basically what we're doing, but leaving the
policies in separate datastructures so that they can leverage different theory
types.
Many other module systems are available, but this is basically Python's
module system. Everything is visible to everything as long as you
have a reference to it.
This proposal does not include hierarchical policies (policies defined
within other policies). While that model is sensible for a file-based
programming language, it is less clearly sensible for a restful API
programming language since there is no inherent structure. This
proposal still allows people to use hierarchical policy *names*, e.g.
marketing:manager:alice, but there is no semantics to that hierarchy
built into the language.
Policy
------
Example with policies policy1 and policy2. We assume we write the
following rule in policy0.
error(vm) :-
nova:virtual_machine(vm),
policy1:vm_property(vm),
neutron:port(vm, src_ip),
policy2:ip_bad(src_ip).
Policy Actions
--------------
None
Data Sources
------------
None
Data model impact
-----------------
None. Just a change in the policy engine's implementation.
REST API impact
---------------
Each API method which is either added or changed should have the following
create_policy
* Description: create a new policy of the given name, abbreviation, and
type
* Method: POST
* Normal http response codes: success
* HTTP errors:
** conflict: policy already exists
** bad request: type does not exist
* URL: /v1/policy/
* Parameters: name, abbreviation (for traces), type (Nonrecursive,
Materialized)
Example:
curl -X PUT http://localhost:1789/v1/policy -d
'{"id": "test_policy",
"description": "a great policy",
"abbreviation": "test",
"type": "nonrecursive"}'
Example output:
{"id": "test_policy",
"description": "a great policy",
"abbreviation": "test",
"type": "nonrecursive",
"owner": "alice"}'
delete_policy: standard deletion operation
Example:
curl -X DELETE http://localhost:1789/v1/policy/test_policy
Security impact
---------------
None
Notifications impact
--------------------
None
Other end user impact
---------------------
Users will interact with other policies when writing rules.
Performance impact
------------------
None
Other deployer impact
---------------------
None
Developer impact
----------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
thinrichs@vmware.com
Work items
----------
- Add create_policy/delete_policy to congress/policy/policy.py:Runtime.
The arguments to create_policy should include name/abbr/type, where
type is either NonrecursiveRuleTheory or MaterializedViewTheory.
- Change compile.atom to a separate 'module' field and 'table' field to
avoid repeatedly parsing the tablename. The 'module' field is None if
there is none.
- Modify top_down_eval so that at every point in the search, it
jumps to the policy in which the table is defined (or stays in the
current policy if 'module' is None).
- Ensure no infinite loops across theories. We need to check
that the graph obtained by rules that cross policy boundaries
is non-recursive; we can ignore rules that do not cross policy
boundaries.
- Can leave 'includes' functionality for internal implementation.
Should not need to use it for the change above.
- Expose this functionality through the API and CLI
Dependencies
============
None
Testing
=======
Unit tests, both positive and negative.
Positive
---------
1) Test 1
policy1:
p(x) :- policy2:q(x)
policy2:
q(1)
q(2)
Query: policy1:p(x) yields {p(1), p(2)}
2) Test 2
policy1:
p(x) :- policy2:q(x)
r(1)
r(2)
policy2
q(x) :- policy1:r(x)
Query: policy1:p(x) yields {p(1), p(2)}
3) Test 3 (namespace separation)
policy1:
p(x) :- policy2:q(x)
q(1)
q(2)
policy2
q(3)
q(4)
Query: policy1:p(x) yields {p(3), p(4)}
Negative
----------
1) Test 1
policy1:
p(x) :- policy2:q(x)
policy2:
q(x) :- policy1:p(x)
Should throw error.
Documentation impact
====================
Need to add docs that describe new capabilities.
References
==========
None

133
specs/kilo/murano-congress-integration.rst
View File

@ -1,133 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
murano-congress-integration
==========================================
https://blueprints.launchpad.net/congress/+spec/murano-congress-integration
This blueprint is to convert the integration with murano and congress.
Problem description
===================
A detailed description of the problem:
* Make murano and congress interface with each other.
Proposed change
===============
This blueprint covers 2 different things that we need to do.
- The first is write a datasource driver for congress that interacts with
murano.
- The second is to integrate murano with congress. To do this we might want
to provide some kind of congressmiddeware if there are any details we can
abstract there but we will figure this out later after that datasource
driver is written.
Alternatives
------------
N/A
Policy
------
N/A
Policy Actions
--------------
N/A
Data Sources
------------
The Murano Project introduces an application catalog to OpenStack,
enabling application developers and cloud administrators to publish
various cloud-ready applications in a browsable categorized catalog.
Cloud users -- including inexperienced ones -- can then use the catalog
to compose reliable application environments with the push of a button [1]
Data model impact
-----------------
N/A
REST API impact
---------------
N/A
Security impact
---------------
N/A
Notifications impact
--------------------
N/A
Other end user impact
---------------------
N/A
Performance impact
------------------
N/A
Other deployer impact
---------------------
N/A
Developer impact
----------------
N/A
Implementation
==============
Assignee(s)
-----------
Primary assignee:
Murano Team
Work items
----------
Data source driver
murano integration with congress api
Dependencies
============
Data source driver for murano should be done first.
Testing
=======
Tempest and unit tests will be added.
Documentation impact
====================
N/A
References
==========
[1] - https://wiki.openstack.org/wiki/Murano

191
specs/kilo/murano-datasource-driver.rst
View File

@ -1,191 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Murano Data Source Driver
==========================================
https://blueprints.launchpad.net/congress/+spec/murano-driver
This blueprint is to add a data source driver for Murano
Problem description
===================
We need a data source to get the Murano environments and applications related
data so that we can define policies during and after application service
fulfillment
Proposed change
===============
Add data source driver that integrates Congress with Murano with following
data tables.
* Data Tables
* *murano:objects(object_id, owner_id, type)*
This table holds every MuranoPL object instance in an environment.
* *object_id* - uuid of the object as used in Murano
* *owner_id* - uuid of the owner object as used in Murano
* *type* - string with full type identification as used in Murano
(e.g., io.murano.Environment,...)
* *murano:parent-types(id, parent_type)*
This table holds parent types of *obj_id* object.
* *id* - uuid of the object as used in Murano
* *parent_type* - string with full type identification of the parent
object
Note that Murano supports multiple inheritance, so there can be several
parent types for one object
* *murano:properties(id, name, value)*
This table stores object's properties. For multiple cardinality
properties, there can be number of records here.
* *id* - uuid of the object as used in Murano
* *name* - property name
* *value* - property value
MuranoPL properties referencing class type (i.e., another object)
are stored in *murano:relationship*. Properties with structure has
not direct mapping here - it has to be stored here component
by component.
* *murano:relationships(source_id, target_id, name)*
This table stores relationship between objects (i.e., MuranoPL property
to *class*).
* *source_id* - uuid of the source object
* *target_id* - uuid of the target object
* *name* - name of the relationship between source and target object
For multiple cardinality relationships there is several records in the
table.
* *murano:states(id, state)*
This table stores *EnvironmentStatus* of Murano environment
* *id* - uuid of the environment
* *state* - the state status of the environment
The state status will be one of 'READY', 'PENDING', 'DEPLOYING',
'DEPLOY_FAILURE', 'DELETING', 'DELETE_FAILURE'.
Alternatives
------------
N/A
Policy
------
We will have pre and post deployment policies for Murano applications based
on the data tables defined in the driver.
Pre deployment policies will cover policies during application service
fulfillment and will leverage the Congress simulate feature.
Post deployment policies will cover monitoring policies on fulfilled service
instances.
Policy actions
--------------
Actions will be added in the second phase.
Data sources
------------
Murano
Data model impact
-----------------
N/A
REST API impact
---------------
N/A
Security impact
---------------
N/A
Notifications impact
--------------------
N/A
Other end user impact
---------------------
N/A
Performance impact
------------------
N/A
Other deployer impact
---------------------
N/A
Developer impact
----------------
N/A
Implementation
==============
Assignee(s)
-----------
kishan-thomas
kishan.thomas@hp.com
Work items
----------
- Implement driver and data tables
- Implement unit tests
- Integration tests with Murano
- Add example policies
Dependencies
============
Murano API
Testing
=======
- Unit tests
- Integration tests
- Tempest tests
Documentation impact
====================
Add new documents in standard documentation location
References
==========
https://etherpad.openstack.org/p/policy-congress-murano-spec

132
specs/kilo/neutron-datasource-driver-refactor.rst
View File

@ -1,132 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
neutron-datasource-driver-refactor
==========================================
https://blueprints.launchpad.net/congress/+spec/neutron-datasource-driver-refactor
This blueprint is to cover the refactoring work to improve how the neutron
datasource driver structures its data. In addition, to add security-group-rule
support.
Problem description
===================
* Refractor neutron driver table schema.
* Add security group/security-group-rule support
Proposed change
===============
* Leverage parent-key attribute to remove subtables.
* Restructure table schema to be more natural.
* Implement security-group/security-group-rule support.
Alternatives
------------
N/A
Policy
------
N/A
Policy actions
--------------
N/A
Data sources
------------
Neutron
Data model impact
-----------------
The schema will change to remove the subtables. This will simplify how one
writes policy though it will break existing policies that have been written
against the neutron driver.
REST API impact
---------------
N/A
Security impact
---------------
N/A
Notifications impact
--------------------
N/A
Other end user impact
---------------------
N/A
Performance impact
------------------
N/A
Other deployer impact
---------------------
The datasources.conf file will have to be updated to include the new
neutron driver.
Developer impact
----------------
N/A
Implementation
==============
Assignee(s)
-----------
Who is leading the writing of the code? Or is this a blueprint where you're
throwing it out there to see who picks it up?
If more than one person is working on the implementation, please designate the
primary author and contact.
Primary assignee:
arosen
Work items
----------
Implement
Dependencies
============
All other patches this requires have already been merged into congress.
Testing
=======
Tempest and unittests will be present.
Documentation impact
====================
Will update docs
References
==========
N/A

134
specs/kilo/parent-key-improvements.rst
View File

@ -1,134 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Parent key improvements
==========================================
https://blueprints.launchpad.net/congress/+spec/parent-key-improvements
There are two current shortcomings of the parent_key implementation in
congress.
* It's unable to be accessed from a subtable. This is needed in the neutron
refactoring working to handle how neutron structures the response for
routers.
* It's column name is always named parent_key. It would be helpful if we
could have the column in the schema actually reflect it's name.
For example, router_id per say.
Problem description
===================
* Parent key cannot be accessed from subtables which is needed.
* No way to rename parent key column name.
Proposed change
===============
Change the datasource framework code so this can be done.
To handle specifiy the column name on a parent_key on should add:
'parent-col-name': <NAME> to the translator. If this is not present
then the old behavior will still occur of having the column name called
parent_key.
Alternatives
------------
N/A
Policy
------
N/A
Policy actions
--------------
N/A
Data sources
------------
N/A
Data model impact
-----------------
N/A
REST API impact
---------------
N/A
Security impact
---------------
N/A
Notifications impact
--------------------
N/A
Other end user impact
---------------------
N/A
Performance impact
------------------
N/A
Other deployer impact
---------------------
N/A
Developer impact
----------------
N/A
Implementation
==============
Assignee(s)
-----------
Primary assignee:
arosen
Work items
----------
Implement
Dependencies
============
* This needs to be done before neutronv2 refactoring can be done.
Testing
=======
Unit tests will be included.
Documentation impact
====================
N/A
References
==========
N/A

129
specs/kilo/persistent-storage-for-api.rst
View File

@ -1,129 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
persistent-storage-for-api
==========================================
https://blueprints.launchpad.net/congress/+spec/persistent-storage-for-api
Problem description
===================
* Currently we store all the api configuration data in memory so when the
server is restarted all of ones data is lost. This blueprint is to
implement a persistent layer to save this info.
* Needs to handle schema changes and data migration
Proposed change
===============
Add Persistance layer
Alternatives
------------
Use a flat file which might be easier for the first implementation but is more
error prone than using a db and doesn't let us scale out horizontally easily.
Policy
------
N/A
Policy actions
--------------
N/A
Data sources
------------
N/A
Data model impact
-----------------
N/A
REST API impact
---------------
N/A
Security impact
---------------
There could be some private data here as we are storing peoples policies here.
Though access to the db will require one to authenticate so hopefully there
is no security flaws here that allow uninteneded access.
Notifications impact
--------------------
N/A
Other end user impact
---------------------
* Users will now have to deploy a db for congress to leverage.
Performance impact
------------------
* Hopefully none
Other deployer impact
---------------------
N/A
Developer impact
----------------
N/A
Implementation
==============
Assignee(s)
-----------
Primary assignee:
arosen
Work items
----------
Implementation
Dependencies
============
* We'll leverage alembic to handle the migrations and reuse a lot of the code
that neutron already has to do this.
Testing
=======
Unit tests will be provided
Documentation impact
====================
N/A
References
==========
http://alembic.readthedocs.org/en/latest/
https://github.com/openstack/neutron/blob/master/neutron/db/migration/README

210
specs/kilo/policy-engine-trigger.rst
View File

@ -1,210 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Policy Engine Triggers
==========================================
Blueprint: https://blueprints.launchpad.net/congress/+spec/policy-engine-trigger
Currently there is no mechanism for treating changes to
policy-defined tables as events (aka triggers). Treating changes to
policy-defined tables as events is useful because
it enables us to, for example,
(i) publish those changes on the message bus
(ii) set up reactive enforcement, either written in code or
in policy,
(iii) kicking off translations to other policy
engines.
Problem description
===================
This spec aims to provide a programmatic interface for triggers.
A programmer will register an event handler to run whenever
a change to a table occurs. The framework will then
periodically (e.g. whenever inserts/deletes occur)
compute updates to tables and run the registered event handlers.
Proposed change
===============
We will create a trigger-registry interface that enables a programmer
to dictate a function to run each time the contents of a given
table changes.
Programmer's perspective
------------------------
The functions that a programmer will register have a signature like
the one that follows.
def respond_to_trigger(oldtable, newtable, delta)
The programmer will register this handler on a particular theory
and a particular table. In this example, we register the
function above so it runs each time the table "p" changes in
the policy/theory "th".
# instance of congress/policy/runtime.py:Theory
th = engine.policy("alice_policy")
# register 'respond_to_trigger' for table 'p' on that policy
th.register_trigger("p", respond_to_trigger)
From the programmer's point of view, each time the contents of
table "p" changes in the theory named "alice_policy", the function
respond_to_trigger is run and given as arguments the original
table, the new table, and the difference in the two.
Implementation
---------------
The function 'register_trigger' will be implemented in the
congress/policy/runtime.py:Theory class. We will also have
an 'unregister_trigger' function call. The Theory class will
be augmented to contain a hash table 'self.triggers' that
maps each table name to the set of functions that have
been registered for that table. 'register_trigger' and
'unregister_trigger' change the contents of that hash table.
Then changes will be made to NonrecursiveRuleTheory,
MaterializedViewTheory, and Database to implement the trigger:
each time update() is called, the theory will
(i) compute the contents of each table that has a trigger registered,
(ii) apply the usual update() logic,
(iii) compute the contents of each table with a registered trigger (again),
(iv) compute the deltas on each of the tables with triggers
(v) for each table t with non-empty delta, invoke all of the triggers
registered for t giving the arguments (i), (iii), (iv)
Obviously triggers can be expensive because they require 2 queries
each time data is changed. Ideally, we would utilize policy analysis
to only query those tables when the data being update might possibly
change the contents of those tables.
For example, suppose we registered a trigger for table p, and we
had 2 rules:
p(x) :- q(x)
r(x) :- t(x)
Updates to table t could never change the contents of p, but updates
to table q COULD change the contents of p. Hence, the trigger
implementation would run queries on p only when the incoming update
changes q.
Alternatives
------------
Programmers could write their own trigger-logic for each feature
that needs this functionality. The benefit would be that the
programmer can customize the functionality to fit exactly her
purpose. The drawback would be that the user would need to
understand the internals of all the Theory subclasses.
Policy
------
None
Policy Actions
--------------
None
Data Sources
------------
None
Data model impact
-----------------
None
REST API impact
---------------
None
Security impact
---------------
None
Notifications impact
--------------------
None
Other end user impact
---------------------
None
Performance impact
------------------
This added functionality may or may not come with a performance cost.
NonrecursiveRuleTheory will have a performance cost because currently
we do not evaluate a table on insert/delete, but after the change
we will need to do more query evaluation.
MaterializedViewTheory will incur no performance cost because it already
evaluates all tables at every insert/delete.
Other deployer impact
---------------------
None
Developer impact
----------------
None outside of Congress. Developers can ignore the new interface
if they want.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
Tim Hinrichs
Work items
----------
- Add event-handler registry to theory base class
- Add implementation of event-handler to all theory subclasses
Dependencies
============
None
Testing
=======
Unit tests: register event handler, insert/delete policy data, check if
event handler actually executed
Documentation impact
====================
None
References
==========
None

180
specs/kilo/publish-policy-results-to-bus.rst
View File

@ -1,180 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Publish policy results to DSE message bus
==========================================
https://blueprints.launchpad.net/congress/+spec/publish-policy-results-to-bus
Implement subscriptions for the policy engine. That is, every time an entity
on the DSE subscribes to a policy engine table, the policy engine should
publish the results of that table on the message bus. The implementation
should utilize the trigger mechanism, which is the subject of another spec.
Problem description
===================
Currently, datasource drivers obey the publish/subscribe paradigm for tables.
The policy engine subscribes to the tables it needs from the datasources,
and the datasources publish information as appropriate.
But there are several use cases when other entities on the DSE message bus
(not necessarily "datasources" per se) would like to subscribe to tables
defined within policy.
- Building proof of concepts where an external service is informed of
policy violations and reacts accordingly, implemented for the sake of
convenience as another entity on the DSE message bus.
- Interoperable policy engines that publish their monitoring results
on the bus for other policy engines to consume.
Proposed change
===============
Every time the policy engine gets a subscription request for a specific
table, e.g. 'alice_policy:error', the policy engine registers a trigger
for that trigger. When the trigger fires, it publishes the contents of
that table on the message bus.
Every time the policy engine gets an unsubscribe request for a specific
table, the policy engine removes the trigger for that table.
The subscribe/unsubscribe functionality should be implemented within
policy/dse_policy.py:DseRuntime.
Because it is likely that the tables could be large, it makes sense to
publish deltas for those tables on the bus, just as the datasources do.
The functionality that does this for datasource drivers can be found
in datasource_driver.py:DatasourceDriver.prepush_processor.
To implement the delta publishing, we should look into creating a subclass
of dse/deepsix.py:DeepSix that includes the prepush_processor functionality and
have both DatasourceDriver and DseRuntime inherit from it instead of DeepSix.
Alternatives
------------
The policy engine could publish the entire table to the bus. The downside
is that large tables with frequent small changes would cause a large amount
of unneeded bus traffic. The upside would be that the subscriber might
be simpler to write if it receives the entire table. If that turns out to
be the case, we could always build convenience functions that compute
the entire table from the deltas.
Policy
------
N/A
Policy Actions
--------------
N/A
Data Sources
------------
N/A
Data model impact
-----------------
N/A
REST API impact
---------------
N/A
Security impact
---------------
N/A
Notifications impact
--------------------
N/A
Other end user impact
---------------------
N/A
Performance impact
------------------
Performance will be impacted, but little moreso than because of the triggers.
The triggers will need to compute the delta; that delta will then simply
be published on the message bus. Publishing is fast (especially compared
to computing the contents of tables and then their deltas).
Other deployer impact
---------------------
N/A
Developer impact
----------------
N/A
Implementation
==============
Assignee(s)
-----------
Primary assignee:
<launchpad-id or None>
Other contributors:
<launchpad-id or None>
Work items
----------
- Create subclass of DeepSix that includes delta publication functionality
and have DatasourceDriver and DseRuntime inherit from that subclass
instead of DeepSix
- Alter DseRuntime so that every subscribe message sets up the appropriate
trigger.
- Alter DseRuntime so that every unsubscribe message removes the appropriate
trigger.
Dependencies
============
* Requires triggers: policy-engine-triggers
Testing
=======
Non-tempest tests that subscribe to policy engine tables, cause
changes to those tables, and verify that the appropriate deltas
are sent on the bus are adequate.
Documentation impact
====================
None required--we're just making a policy engine implement the same interface
as datasource drivers.
References
==========
N/A

166
specs/kilo/refactor-datasource-driver-framework.rst
View File

@ -1,166 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
refactor-datasource-driver-framework
==========================================
https://blueprints.launchpad.net/congress/+spec/refactor-datasource-driver-framework
Currently the congress datasource driver is still pretty complex. This
blueprint aims to simplify it.
Problem description
===================
A detailed description of the problem:
* current there is a lot of logic in the convert_obj() method which is
used to load the data into the data source driver. The majority of this
logic is to validate that the table schema is valid. We should only need
to do this one though on load of the data source driver.
* currently if you list the tables of a data source driver they aren't always
populated in via __init__ self.state[table_name] = set(). I want to add a
method to the DatasourceDriver base class that load_translator() which
will allow one to pass in translators via the __init__ of a
datasource driver. This will validate the schema there.
Proposed change
===============
Add load_translator() to datasource driver base class as descripted as able.
In addition to other refactoring that comes up as this work is being done.
Alternatives
------------
N/A
Policy
------
N/A
Using the Congress datalog syntax, write out an example policy using
https://wiki.openstack.org/wiki/Congress#Policy_Language
Example:
error(vm) :-
nova:virtual_machine(vm),
ids:ip_packet(src_ip, dst_ip),
neutron:port(vm, src_ip), //finds out the port that has the VM's IP
ids:ip_blacklist(dst_ip).
Policy Actions
--------------
N/A
Describe the policy activities in terms of monitoring, reactive, proactive,
and other ways to explain how the policy will implement it's desired state.
Data Sources
------------
N/A
Describe which projects and/or services the data is coming from
Data model impact
-----------------
N/A
Changes which require modifications to the data model often have a wider impact
on the system. The community often has strong opinions on how the data model
should be evolved, from both a functional and performance perspective. It is
therefore important to capture and gain agreement as early as possible on any
proposed changes to the data model.
Questions which need to be addressed by this section include:
* What new data objects and/or database schema changes is this going to
require?
* What database migrations will accompany this change.
* How will the initial set of new data objects be generated, for example if you
need to take into account existing instances, or modify other existing data
describe how that will work.
REST API impact
---------------
N/A
Security impact
---------------
N/A
Notifications impact
--------------------
N/A
Other end user impact
---------------------
N/A
Performance impact
------------------
N/A
Other deployer impact
---------------------
N/A
Developer impact
----------------
N/A
Implementation
==============
Assignee(s)
-----------
Primary assignee:
arosen
Work items
----------
Push code to do this.
Dependencies
============
N/A
Testing
=======
Unit tests and tempest tests will be present to confirm this works as desired.
Documentation impact
====================
Will update docs for how this now works.
References
==========
N/A

163
specs/kilo/refactor-drivers.rst
View File

@ -1,163 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
===============================================================================
Pull common functionality into driver superclass, including data transformation
===============================================================================
Include the URL of your launchpad blueprint:
https://blueprints.launchpad.net/congress/+spec/refactor-drivers
Currently, each datasource driver contains code to convert data from the API
call response to the Congress data tables. This change will make it easier
for a developer to add each incremental data source driver.
Problem description
===================
Today, it takes more code than necessary to write a data source driver.
The code is also more difficult to write than necessary. Some of these API
responses (like in neutron list networks) contain nested data, for example in
the list of networks, each network can contain a sub-list of subnets. The
driver populates a separate table for subnets and the creates a key to link
between the network table and the subnet table.
Proposed change
===============
To generalize the driver, this change will allow a driver class to specify how
to extract data from the API response, and into which table/field to put the
response data.
To handle the sublist conversion, the driver class will specify if a field is
a sublist, and then also specify a key, so that the main table can link to the
subtable. This sublist relationship will be recursive, so that a sublist can
also contain another sublist.
Alternatives
------------
None
Policy
------
None
Policy Actions
--------------
None
Data Sources
------------
None
Data model impact
-----------------
None
REST API impact
---------------
None
Security impact
---------------
None
Notifications impact
--------------------
None
Other end user impact
---------------------
None
Performance impact
------------------
None
Other deployer impact
---------------------
This change will have an effect as soon as it is merged, but Congress should
behave exactly as it did without the change.
Developer impact
----------------
The new refactoring will change how programmers write datasource drivers. The
new way should be easier, require less code, and be less bug prone. To take
advantage of the new refactoring, we'll need to rewrite the existing drivers,
but this need not happen at the same time as writing the new datasource driver
superclass.
Implementation
==============
Assignee(s)
-----------
ayip
Work items
----------
1) Write new superclass.
2) Rewrite Nova driver.
3) Rewrite Neutron driver.
4) Rewrite Keystone driver.
Dependencies
============
None
Testing
=======
Unit tests for superclass, and modify existing test cases for individual
drivers.
Documentation impact
====================
This change will include new documentation for how to use the new datasource
driver superclass.
References
==========
None

148
specs/kilo/support-python3.rst
View File

@ -1,148 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
====================================
Add Support for Python 3 to Congress
====================================
https://blueprints.launchpad.net/congress/+spec/support-python3
This specification describes how to gradually add Python 3 support to
Congress.
Problem description
===================
Currently Python 3 tests are failing. In an effort to support both Python 2
and Python 3 concurrently a number of changes to Congress are needed.
Proposed change
===============
This specification will be used to track a number of successive, minor changes
to Congress with the goal of fully supporting Python 3. Each commit will
address one change to Congress to ensure compatibility with Python 3 while
continuing support for Python 2.
Alternatives
------------
None.
Policy
------
None.
Policy actions
--------------
None.
Data sources
------------
None.
Data model impact
-----------------
None.
REST API impact
---------------
None.
Security impact
---------------
None.
Notifications impact
--------------------
None.
Other end user impact
---------------------
None.
Performance impact
------------------
None.
Other deployer impact
---------------------
None.
Developer impact
----------------
Once fully implemented commits must pass tox tests against py34 and should be
rejected if tests fail.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
jzabala
Other contributors:
None
Work items
----------
- Determine what sections of code need to be modified to ensure Python 3
compatibility. This can be done by generating list of fixers which run
when '2to3' is used to transform code to Python 3.
- Run code through 2to3 for each fixer (see previous bullet), also ensuring
that changes to the code do not break compatibility with Python 2.
- Confirm that changes are gradually improving tox -e py34 test outcomes.
Dependencies
============
tbd
Testing
=======
Implementation of this specification should result in tox tests against py34
succeeding. Current unit tests (and the code being tested) may have to be
modified during the course of the implementation of this specification.
Documentation impact
====================
None.
References
==========
None.

125
specs/kilo/swift-datasource-driver.rst
View File

@ -1,125 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Swift Data Source Driver
==========================================
https://blueprints.launchpad.net/congress/+spec/swift-data-source-driver
This blueprint is to add a data source driver for swift.
Problem description
===================
A datasource driver is required to expose list of containers and
list of objects in each container to the Congress policy framework
so that we can write policies involving object storage entities.
Swift is the OpenStack component that provides object storage service.
The swift-datasource-driver interacts with swift service to provide
object storage specific states to congress for policy monitoring.
Proposed change
===============
Add data source driver that integrates congress with swift.
Alternatives
------------
N/A
Policy
------
This will use the congress language. swift:containers(X) etc.
Policy actions
--------------
Just monitoring right now.
Data sources
------------
Swift
Data model impact
-----------------
TBD
REST API impact
---------------
N/A
Security impact
---------------
N/A
Notifications impact
--------------------
N/A
Other end user impact
---------------------
N/A
Performance impact
------------------
N/A
Other deployer impact
---------------------
N/A
Developer impact
----------------
N/A
Implementation
==============
Assignee(s)
-----------
Srinivasa Rao Ragolu
sragolu@mvista.com
Work items
----------
- Implement swift driver with essential tables
- Implement test code to test the driver
- Implement tempest code for real-time tests
Dependencies
============
python-swiftclient
Testing
=======
- Will need to add unit test code
- Will add tempest tests.
Documentation impact
====================
N/A
References
==========
N/A

146
specs/kilo/vCenter_Driver.rst
View File

@ -1,146 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
vCenter Driver
==========================================
https://blueprints.launchpad.net/congress/+spec/vcenter-driver
This blueprint is to add a data source driver for vCenter, giving congress
access to new information from an external datasource.
Problem description
===================
N/A
Proposed change
===============
Add a data source driver that integrates congress with vCenter by connecting
to vCenter using oslo.vmware.
Alternatives
------------
N/A
Policy
------
This will use the congress language. vCenter:hosts(X) etc.
Example - Creating a whitelist of all MAC addresses of hosts found in vCenter
WhiteList(vnic_macs,pnic_macs) :-
vCenter:hosts(host:vnic_mac_id,host:pnic_mac_id),
vCenter:host.pnic_macs(host:pnic_mac_id,pnic_macs),
vCenter:host.vnic_macs(host:vnic_mac_id,vnic_macs)
Policy Actions
--------------
Monitoring Hosts and Virtual Machines
Data Sources
------------
vCenter
Data model impact
-----------------
N/A
REST API impact
---------------
N/A
Security impact
---------------
This driver will require vCenter credentials to be input into congresses
configuration, and will provide congress data by using those credentials. It
will be important for those implementing this driver to be aware of what data
is visible from congresses API.
Notifications impact
--------------------
N/A
Other end user impact
---------------------
N/A
Performance impact
------------------
Implementing this driver will add another data source for congress to parse
data from, and since this driver pulls data from a non-openstack source this
will generate additional traffic on the network.
Other deployer impact
---------------------
To use this driver a deployer will need to configure this driver in
datasource.config.
Developer impact
----------------
N/A
Implementation
==============
Assignee(s)
-----------
Primary assignee:
Conner Ferguson
Work items
----------
N/A
Dependencies
============
N/A
Testing
=======
TBD
Documentation impact
====================
Documentation can already be found at
https://bitbucket.org/ConnerFerguson/vcenter-driver
References
==========
https://bitbucket.org/ConnerFerguson/vcenter-driver - Current code hosting

165
specs/liberty/add-action-listing.rst
View File

@ -1,165 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Add Action Listing to api
==========================================
Just as the API allows us to query the available tables that a
datasource provides, it should be possible to query the available
actions that a datasource can execute.
Problem description
===================
Currently, datasources have the ability to execute actions, but there is no api
call to ask what the available actions are.
Proposed change
===============
Add an API call that returns the list of available actions.
This requires adding a method to the
congress/datasources/datasource_driver.py:ExecutionDriver
class that returns the list of actions that the datasource will execute.
It would be nice if the ExecutionDriver also enforced that it only
executed the actions in that list.
One might consider trying to auto-generate the list of actions based
on the methods available in the python-client being used (for those
datasources using a python-client).
Alternatives
------------
The alternative is to provide no API and ask people to consult the
documentation to find the list of available actions.
This is more
attractive than it sounds because it gives us the freedom to
execute any action that the python-client for the underlying
datasource exposes. If someone adds a new method for the python-client
then a policy writer immediately gets access to that method, without
requiring any changes in Congress.
The downside to this approach is that end-users need to understand
which python client they are using within each datasource driver and
then look up the docs for that python client. So in terms of usability,
it's pretty terrible.
Policy
------
N/A
Policy actions
--------------
N/A
Data sources
------------
N/A
Data model impact
-----------------
There should be no data-model impact, since the added information is
implemented as part of the datasource driver, not stored in the database.
REST API impact
---------------
GET /v1/data-sources/<datasource-id>/actions
Should error if the datasource does not implement the proper method.
Result should be an array of dicts, where each dict describes an action:
[{'name': <action-name>,
'args': <list-of-parameter-names>}]
The assignee is free to extend the fields of an action to include additional
meta-information, such as a description of what the action does, or a
description of what each of the parameters are.
Security impact
---------------
N/A
Notifications impact
--------------------
N/A
Other end user impact
---------------------
Need to add similar functionality to the python-congressclient.
Performance impact
------------------
N/A
Other deployer impact
---------------------
N/A
Developer impact
----------------
N/A
Implementation
==============
Assignee(s)
-----------
Primary assignee:
<launchpad-id or None>
Other contributors:
<launchpad-id or None>
Work items
----------
0. Modify API design doc (linked from the wiki)
1. Add method to ExecutionDriver in congress/datasources/datasource_driver.py
2. Add call handler in congress/harness.py
3. Add route call in congress/api/router.py
4. Add API implementation to congress/api/action_model.py
5. Update python-congressclient
Dependencies
============
N/A
Testing
=======
- Unit tests for API to congress/tests/test_congress.py
- Unit tests for ExecutionDriver
Documentation impact
====================
Add API call
References
==========
N/A

189
specs/liberty/api-validation.rst
View File

@ -1,189 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==============
API Validation
==============
https://blueprints.launchpad.net/congress/+spec/api-validation
Users (and programs) sometimes provide invalid inputs to APIs. The API should
be resilient to this, and deny bad requests.
Problem description
===================
Any request that does not conform to the form expected by the API is deemed
invalid. Invalid requests should be rejected by the API, along with
information to aid in correcting the problem.
The following features are desirable in an API validation solution:
* Validation should enforce a declarative model that is visible external to
the validation implementation.
* The declarative model should follow standard formatting and conventions
that are understood in the API caller's context.
* For cross-platform APIs, the model should be consumable without knowledge
of a particular programming language.
* The model should provide sufficient information for generation of
resource documentation in associated API reference material.
* Validation should be performed by a common framework which facilitates
binding the declarative models with the API implementation.
* The API implementation may assume trusted inputs by relying on the
framework to offload validation.
* Validation may be selectively enabled on API outputs to facilitate
development and testing.
Proposed change
===============
We will introduce JSON schemas to model the expected inputs and outputs of
each API call.
The API resource manager will be updated to support binding of schemas to
each handler.
When dispatching API calls, the framework will validate the call body using
the 'jsonschema' python validator.
If an optional flag is provided, the dispatcher will validate the API handler
response body before passing the response up the stack.
Alternatives
------------
Other components, such as neutron, have created custom validation utilities.
These utilities are often not as rich as JSON schema, and do not address
many of the desirable features described above.
The nova project is currently retrofitting their API to utilize JSON Schema
validation: https://blueprints.launchpad.net/nova/+spec/v3-api-schema
Data model impact
-----------------
None.
REST API impact
---------------
The current API provides very minimal validation. This change will add
validation to all existing API calls.
Security impact
---------------
Adding validation to the API should increase security of the system by
protecting the API implementation from unexpected inputs.
Notifications impact
--------------------
None.
Other end user impact
---------------------
None.
Performance impact
------------------
Validation of inputs will have a negative performance impact on latency and
throughput of API calls. JSON schema does not necessarily impose larger
performance impacts than other validation solutions, despite its increased
power and flexibility.
Other deployer impact
---------------------
None.
Developer impact
----------------
After introduction of API validation, new API calls will need to introduce an
associated schema.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
pballand
Other contributors:
None
Work items
----------
* Create JSON schemas for each API resource (This is already done in the API
design, and simply needs to be translated to the source tree.)
* Add support to the API resource manager to bind schemas with resource
endpoints.
* Add jsonschema as a runtime dependency.
* Use jsonschema to validate requests before dispatching to API implementation.
* Expose flag to support output validation. Optionally use jsonschema to
validate body of API result before returning to the wsgi server.
Dependencies
============
None.
Testing
=======
The schema validator is assumed to be thoroughly tested. We will test that
validation is being performed by issuing requests that should and should not
validate against the schema.
Documentation impact
====================
This change itself does not impact documentation. The addition of schemas
for each API call should be included in the documentation.
References
==========
JSON Schema definition:
http://json-schema.org/
Python jsonschema validator:
https://python-jsonschema.readthedocs.org/en/latest/
Nova spec for updating validation to use JSON Schema:
https://blueprints.launchpad.net/nova/+spec/v3-api-schema

230
specs/liberty/api-version.rst
View File

@ -1,230 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
========================
Support version list API
========================
https://blueprints.launchpad.net/congress/+spec/api-version
The Congress API should be able to list current supported API versions.
Problem description
===================
Users want to talk with Congress API, but no API support to query the supported
versions, so users have no idea about which versions can be supported in the
current Congress deployment.
Proposed change
===============
Export the version list API, and show the current API details, include: id,
status, update time, links, like other OpenStack project: nova, neutron and
etc. Although Congress only support one version API(v1) currently, but the
version list API make sense for future API evolution.
Alternatives
------------
None
Policy
------
None
Policy actions
--------------
None
Data sources
------------
None
Data model impact
-----------------
None
REST API impact
---------------
* Specification for the method
* List the current supported API versions.
* Method type: GET
* Normal http response code(s): 200
* Expected error http response code(s): None
* ``/``
* Parameters which can be passed via the url: None
* JSON schema definition for the body data if allowed: None
* JSON schema definition for the response data if any:
::
{
"type": "object",
"properties": {
"versions": {
"type": "array",
"items": {
"type": "object",
"properties": {
"status": {
"type": "string"
},
"updated": {
"type": "string"
},
"id": {
"type": "string"
},
"links": {
"type": "array",
"items": {
"type": "object",
"properties": {
"href": {
"type": "string"
},
"rel": {
"type": "string"
}
},
"additionalProperties": false,
"required": ["href", "rel"]
}
}
},
"additionalProperties": false,
"required": ["status", "updated", "id", "links"]
}
}
},
"additionalProperties": false,
"required": ["versions"]
}
* Example use case:
::
GET /
{
"versions": [{
"status": "CURRENT",
"updated": "2015-07-03T11:33:21Z",
"id": "v1",
"links": [{
"href": "http://10.250.10.29:1789/v1/",
"rel": "self"
}]
}]
}
* There should not be any impacts to policy.json files for this change.
Security impact
---------------
None
Notifications impact
--------------------
None
Other end user impact
---------------------
The related works in python-congressclient will also be added.
After this modification, user could get the API version details, like this:
::
openstack congress version list
Performance impact
------------------
None
Other deployer impact
---------------------
We modify api-paste.ini to add some stuff, so if the operator prepare to
upgrade from old release, he need to add the new config items to old
api-paste.ini file or override the old using new one.
Developer impact
----------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
Rui Chen <chenrui.momo@gmail.com>
Work items
----------
* Adds Version class to assemble API versions response.
* modify api-paste.ini to route the request to the new logic.
* Make python-congressclient supporting this API.
Dependencies
============
None
Testing
=======
Some unit tests should been added to cover the new API.
Documentation impact
====================
The related content should be added in Congress API document.
References
==========
None

182
specs/liberty/compromised-vm.rst
View File

@ -1,182 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Compromised VM
==========================================
https://blueprints.launchpad.net/congress/+spec/compromised-vm
Security policy use case
Problem description
===================
IDS service notices malicious traffic originating from an insider VM trying to
send packets to hosts inside and outside of the tenant perimeter. As this is
detected, some reactive response would need to be taken, such as isolating the
offending VM from the rest of the network. This policy would facilitate one of
the reactive responses to be invoked when a compromise is reported by an IDS
service.
Proposed change
===============
An IDS will need to monitor traffic in and out of virtual machines. The IDS
will maintain the blacklist of known bad destinations. The patterns will need
to maintained in the IDS. While the IDS is monitoring for traffic patterns and
blacklist, it will need to notify Congress of the event through the Congress
IDS plugin. Through the Congress Neutron plugin, Neutron API will need to be
updated blocked IP. Congress-client and Horizon to alert the operator of
changes.
Alternatives
------------
(sarob)
Instead of a blacklist maintained by the IDS, use an real time check of a
public blacklist.
(thinrichs)
Would it be simpler to flag an error anytime a VM was identified by IDS?
error(vm) :- ids:ip_blacklist(ip), neutron:port(vm, ip)
(thinrichs)
We should try to work out how the reactive bit would work. The way we'll do
that for the beta is to write another policy that dictates which action to take
under certain conditions. Imagine the policy above but instead of error(vm), we
write something like neutron:block_ip(secgrp, ip) :- ids:ip_blacklist(ip),
neutron:port(vm, ip), neutron:security_group(vm, secgrp)
I'm sure not all the details there are right. But ideally the thing in the head
of the rule would be a Neutron API call. In the worst case it can just the name
of a script that we write.
Policy
----------------
error(vm) :-
nova:virtual_machine(vm),
ids:ip_packet(src_ip, dst_ip),
neutron:port(vm, src_ip), //finds out the port that has the VM's IP
ids:ip_blacklist(dst_ip).
Policy actions
-----------------
* Monitoring: report/log the incident including the VM's IP address, external
IP, etc.
* Reactive: Invoke the nova API to add the VM to IDS security group restricting
access to make changes. Invoke neutron to block all traffic to/from the
VM's IP address. Alternatives are to restart the VM on a nova IDS
schedule filter (limiting traffic chaos while maintaining the ability to
access the VM) and/or a no route network or removing the VM network
interface(s).
Data sources
-----------------
* IDS (intrusion detection service VM): IP address of the offending VM
* neutron: network details, IP details
* nova: VM details; instance ID, interface(s) status, instance state, security
group
Data model impact
------------------
none
REST API impact
---------------
Needs explanation
Security impact
---------------
* DoS
Notifications impact
--------------------
* IDS API
* congress-client
* horizon
Other end user impact
---------------------
Unknown at this time
Performance impact
------------------
Unknown at this time
Other deployer impact
---------------------
Unknown at this time
Developer impact
----------------
Unknown at this time
Implementation
==============
Assignee(s)
-----------
Unknown at this time
Work items
----------
* Congress IDS plugin
* Compromised VM policy
* Congress client notification
* Horizon notification
Dependencies
============
Neutron really needs to provide port allow/deny primitive. Today it blocks
everything and we can only poke pin holes what's allowed out -- this makes it
complex to implement policy
(thinrichs)
Need to understand more about these limitations.
(sarob)
Testing
=======
TBD
Documentation impact
====================
TBD
References
==========
https://docs.google.com/document/d/1ExDmT06vDZjzOPePYBqojMRfXodvsk0R8nRkX-zrkSw/edit#
https://wiki.openstack.org/wiki/Congress

208
specs/liberty/congress-gbp-translation.rst
View File

@ -1,208 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==================================================================
Translation of Congress Table to Neutron Group-based Policy Tables
==================================================================
https://blueprints.launchpad.net/congress/+spec/congress-gbp-translation
This specification describes how to integrate Congress Policy with Neutron
Group-based Policy (GBP).
Problem description
===================
Congress provides a mechanism to allow OpenStack clients to define policy
to be applied across all OpenStack components including networking.
Neutron Group-based policy provides a high level abstraction for defining
network connectivity between groups of endpoints.
It is desirable to integrate Congress and GBP so that Congress can
monitor and enforce GBP policies. A Congress Reachability table
can be used to specify connectivity policy between endpoints. This table
can be translated into a set of tables that represent GBP entities, such as
Policy Target Groups, Policy Rules, Policy Classifier, etc.
Proposed change
===============
GBP can be represented by these tables:
* Endpoints (endpoint_id)
* Endpoint_group (endpoint_id, endpoint_group_id)
* Classifiers (classifier_id, port, protocol, direction)
* Classifier_group (classifier_id, classifer_group_id)
* Actions (action_id, action_type, action_value)
* Action_group (action_id, action_group_id)
* Policy_rule (policy_rule_id, classifier_group_id, action_group_id)
* Contracts (contract_id, policy_rule_id)
* PolicyInstance (endpoint_group_id, relation, contract_id)
A Congress Reachability policy table may be defined to form
a policy between two groups of endpoints:
Reachable (id, group1, group2, src_port, dst_port)
The goal is to translate from the input Reachable policy table to
the output GBP tables using the Congress policy language using
functions such as:
PolicyInstance(group_id1, relation, contract_id) :-
reachable(contract_id, group_id1, group_id2, x, y),
producer_relation(relation)
Alternatives
------------
None.
Policy
------
An example of such a policy written using Congress datalog syntax
is shown below for two groups, tier 1 and tier 2, to communicate
bidirectionally on port 80.
Operator Input Data (from operator or cloud management system)
Tier Membership (tier_id, vm_id)
(1, 100)
(1, 101)
(2, 102)
Policy Input Data
Tiers(tier_id)
(1)
(2)
Reachability policy table
Reachable (id, src, dst, src_port, dst_port)
(10, tier1, tier2, \*, 80)
(11, tier2, tier1, \*, 80)
Policy Actions
--------------
These tables allow Congress to monitor and enforce GBP policies.
Data Sources
------------
Neutron Group-based Policy. Details of GBP can be found here:
https://wiki.openstack.org/wiki/GroupBasedPolicy/StackForge/repos
Data model impact
-----------------
None
REST API impact
---------------
None.
Security impact
---------------
None.
Notifications impact
--------------------
None.
Other end user impact
---------------------
None.
Performance impact
------------------
None.
Other deployer impact
---------------------
None.
Developer impact
----------------
None.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
louis.fourie, alex.yip, cathy.zhang
Other contributors:
Work items
----------
* Define translation functions.
* Implement the new constant tables and function tables to perform the
translation.
Dependencies
============
* This is dependent on the implementation of a GBP data-source driver for
Congress.
Testing
=======
Some sample input tables will be created and the translation verified by
checking the contents of the output trigger tables.
Documentation impact
====================
All translation details will be documented.
References
==========
* Juno Mid-cycle Policy Summit
https://etherpad.openstack.org/p/juno-midcycle-policy-summit

164
specs/liberty/datalog-aggregates.rst
View File

@ -1,164 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Datalog aggregates
==========================================
Add aggregates sum, count, min, max, avg to the policy language.
Problem description
===================
Many policies require counting the number of VMs
with a certain property, computing the average, min, max, or sum
of certain values. Without aggregates, these policies cannot be
expressed in Datalog.
Proposed change
===============
This change adds aggregates to the Datalog policy language.
Alternatives
------------
The alternative is to leave aggregates out of the language. The
drawback is that some common policies would be inexpressible, but
the benefit is that the algorithms for reasoning about policy
would be simpler. By prohibiting aggregates, we exclude the possibility
that the user wants to use aggregates at the cost of their inclusion.
Policy
------
Example: define a table that counts the number of VMs in the web tier
of each application.
app_web_size(id, count(vm)) :-
appservice:app(id),
appservice:webtier(id, vm)
Another option for syntax puts the aggregates into the body of the rule.
This option will likely require new syntactic restrictions on rules
to function properly.
app_web_size(id, cnt) :-
appservice:app(id),
appservice:webtier(id, vm),
count(vm, cnt)
Suggestions for other syntaxes?
Policy actions
--------------
N/A
Data sources
------------
N/A
Data model impact
-----------------
The syntax for rules will be extended, but the database schema for storing
rules will not, since they are stored as strings.
REST API impact
---------------
N/A
Security impact
---------------
N/A
Notifications impact
--------------------
N/A
Other end user impact
---------------------
N/A
Performance impact
------------------
There will be some performance impact in using aggregates since
they cost more to compute. These aggregates will also make reasoning
about the policy itself (i.e. when data is unknown) more difficult.
Other deployer impact
---------------------
N/A
Developer impact
----------------
N/A
Implementation
==============
Assignee(s)
-----------
Primary assignee:
<launchpad-id or None>
Other contributors:
<launchpad-id or None>
Work items
----------
0. Choose aggregates to implement, e.g. count, min, max, sum, avg
1. Modify congress/datalog/congress.g to accept the new statements (possibly)
2. Modify congress/datalog/compiler.g to include new datastructures for
representing aggregates internally
3. Modify congress/datalog/topdown.py:TopDown.select() to handle aggregates.
This will require adding in a step after computing all solutions
that applies the aggregate function.
Dependencies
============
N/A
Testing
=======
- Unit tests will be added to congress/tests/datalog/test_nonrecur.py
- A unit test or two ensuring the API properly accepts the new syntax
will be added to congress/tests/test_congress.py
Documentation impact
====================
Need to add explanation of new syntax constructs to docs.
References
==========
1. Overview of a Datalog language with aggregates:
http://dexter.stanford.edu/main/dexlog.html
2. Recent Stanford paper on aggregates
http://stanford.edu/~abhijeet/papers/abhijeetSARA13.pdf

150
specs/liberty/dist-remove-d6cage-from-api.rst
View File

@ -1,150 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Removing d6cage dependency in api process
==========================================
https://blueprints.launchpad.net/congress/+spec/dist-remove-d6cage-from-api
remove d6cage dependency
Problem description
===================
API server currently relies on d6cage to create new API instances. Congress
will remove d6cage in the next distributed architecture. It needs to remove
dependency on d6cage of API instance creation.
Proposed change
===============
1. Introduce a new dict object to create instances of API models. The dict
has references to each API instances d6cage has as services[name]['object']
now. Then router.py:APIRouterV1 uses the references to register API
with the instance.
2. Change base class of some instances, which aren't need to be a deepsix
class. This BP also changes these kind of class to python normal object.
SchemaModel and DatasourceModel aren't need to be deepSix's subclass both
in the current architecture and in the distributed one. others aren't need
to be deepSix's subclass in the new distributed one.
How to change to distributed version
It replace calling harness.py:create() in service.py:congress_app_factory()
with a new API instance creating method which has instantiation steps for
API model in harness.py:create(). And it's only to make the dict with new
instances.
Alternatives
------------
The alternative is to spread codes in harness.py:create() into
router.py:APIRouterV1.__init__(). This change is easier to track
in current one process solution.
How to change to distributed version
It need to replace all d6cage.createservice() in APIRouterV1.__init__()
with new API instance creating method. There are a lot of place we have
to change for it. So this way is an alternative for this BP.
Policy
------
N/A
Policy actions
--------------
N/A
Data sources
------------
N/A
Data model impact
-----------------
N/A
REST API impact
---------------
N/A
Security impact
---------------
N/A
Notifications impact
--------------------
N/A
Other end user impact
---------------------
N/A
Performance impact
------------------
N/A
Other deployer impact
---------------------
N/A
Developer impact
----------------
N/A
Implementation
==============
Assignee(s)
-----------
Primary assignee:
muroi-masahito
Other contributors:
<launchpad-id or None>
Work items
----------
0. create a new dict to have references to API instances
1. pass it as an argument for router.py:APIRouterV1()
2. use it to register API and the instances
3. remove API instances which don't need to inherit deepsix from d6cage
4. change direct access with variable to indirect access with method
Dependencies
============
N/A
Testing
=======
- change Unit tests for non-deepsix object if needed
Documentation impact
====================
N/A
References
==========
N/A

256
specs/liberty/horizon-policy-abstraction.rst
View File

@ -1,256 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
=========================================
Abstraction policy definitions in Horizon
=========================================
https://blueprints.launchpad.net/congress/+spec/horizon-policy-abstraction
Congress aims to provide an extensible open-source framework for governance
and regulatory compliance across any cloud services. Users can define policies
in congress by pure Datalog. But Datalog, working with data-tables, is a very
difficult and complex description for policies. Users may have some problems
to understand and deploy policies with Datalog. So this specification provides
an abstraction for policies, and shows it in an abstraction form in Horizon,
which will facilitate users to express their policies.
Problem description
===================
Datalog is not intuitive to use, even difficult for users to express their
policies who are not familiar with it. And it may cause some misinterpretation
when translating real intent to Datalog because of the complex logic.
Proposed change
===============
Congress makes the whole cloud compliant by defining violation state and action
for violation.
Policies in Congress can be expressed by BNF as below.
::
Congress Policy ::= violation-condition, "do" action for violation
So, policy abstraction is to abstract violation state and corresponding action
to make the policy more intuitive and easy to use.
By analyzing typical scenarios, violation mainly can be divided into two parts.
One is the constraint of objects' attributes, and another is the constraint
of relationship between several objects' attributes.
All the objects and constraints are not just a simple set of data source
tables, but they can be divided into some categories according to their
functions and relations. So users just need to choose objects they care about
without worrying about which tables they are in.
The violation-condition can be expressed by BNF as below.
::
violation-condition ::=object attribute constraint (value | object-attribute)
object-attribute::=object "." attribute
For any violation state, congress will take some actions, such as monitoring,
proactive and reactive. Of course, there may be more than one action defined to
a violation. Though monitoring violation is the fundamental function of
congress, changing cloud state to make the cloud compliant is also an important
function. So, policy abstraction will provide some optional reactive actions
for different objects to resolve violations.
The action for violation state can be expressed by BNF as below.
::
action ::= ("monitoring"| "proactive"| "reactive action") data
So policies in Congress can be abstracted into "name", "objects",
"violation-condition", "action" and "data".
Among these, element "name" defines a marker of a policy, which is used to
be a unique identification for a policy.
Element "objects" defines all objects which are concerned by this policy.
They are not just simple display of data source tables, but a organized set
which contains the relationship between different tables and objects,
such as, "servers", "networks", "hosts", "subnets", etc.
For example, table "statistics" will not appear as a object, but it will be
a attribute of other objects, such as, "servers", "networks".
Another example is users could choose "servers" and "networks" without caring
about what put them together ("ports", actually).
Element "violation-condition" defines the state of objects' attributes
which can produce violation, and the constraint will include comparison,
arithmetic and some predefined relationship/functions, such as, "same_group".
Element "action" defines the action needs to take for this policy,
and actions will include "proactive", "monitoring" and some specific actions,
such as, "create", "pause". All these actions depend on the ability of
underlying components.
Element "data" defines the information gotten or needed when executing the
action, for example, when monitoring a servers violation, users can define
"data" as servers' name to be a return parameters.
That is, Congress UI will provide many elements of policies as drop-down lists,
and the combination of these elements will form various policies. All these
elements are from the summary of typical scenarios and the ability of
underlying components.
Users need to choose which one can match their needs.
If there are some extensions of underlying component, attributes of any
component, or packaging functions, users could add objects, attributes and
define functions in advanced UI which show all detailed tables and attributes
in the underlying components. All extensions defined in advanced UI will
be available in abstraction UI,so users could customize their own policies
by combining these two UIs.
When user submits what he choose, UI will translate the information in UI
into Datalog.
Alternatives
------------
N/A
Policy
-------
There is one example to express typical policy by abstraction form in Horizon.
Example: every network connected to a VM must either be public or
owned by someone in the same group as the VM.
For this example, users care about "servers" and "networks", so users will
choose these two objects from a drop-down list.
After users decide the objects,users could make use of these attributes to
define violation state. In this example, violation-condition is that servers
tenant's group is not same with networks tenant's group. So users could choose
these two attributes and set their relation is "not equal".
All the choices will be show as drop-down lists, too.
And users need to choose the action and data to define which actions should be
applied to this violation. For example, users choose "monitoring", attributes
of servers and networks will appear in "data".
In this policy, users can create a policy as below.
+----------+----------+------------------------------------------------------+------------+--------------+
| name | objects | violation-condition | action | data |
+----------+----------+------------------------------------------------------+------------+--------------+
| policy_1 | servers |not equal(networks.share, public) | monitoring | servers.name |
| | networks |not equal(servers.tenant.group, networks.tenant.group)| | |
+----------+----------+------------------------------------------------------+------------+--------------+
If user have defined a packaging function for same_group, it will be added into
violation-condition, so user could choose this function and set which two
attributes are the parameters of this function.
If use take use of this way, above policy will be showed as below.
+----------+----------+-----------------------------------------------+------------+--------------+
| name | objects | violation-condition | action | data |
+----------+----------+-----------------------------------------------+------------+--------------+
| policy_1 | servers |not equal(networks.share, public) | monitoring | servers.name |
| | networks |not same_group(servers.tenant, networks.tenant)| | |
+----------+----------+-----------------------------------------------+------------+--------------+
Policy Actions
--------------
The action can be monitoring, proactive or some execute actions
which can make the cloud compliant.
Data Source
-----------
N/A
Data model impact
-----------------
N/A
REST API impact
---------------
N/A
Security impact
---------------
All parameters inputted by users need satisfy predefined standard, for example,
if values inputted in "violation-condition" in reasonable range
(e.g. 0-100% for CPU utilization).
Notifications impact
--------------------
N/A
Other end user impact
---------------------
End users can be able to write policies in Horizon and use some drop-down lists
and some simple inputs to create a policy. Then Horizon will translate the
information in UI into Datalog, which will be processed in Congress.
Performance impact
------------------
N/A
Other deployer impact
---------------------
N/A
Developer impact
----------------
N/A
Implementation
===============
Assignee(s)
-----------
Primary assignee:
Yali Zhang
Other contributors:
Jim Xu; Yinben Xia
Work items
-----------
- Abstraction form to write policies rules and actions for policies.
- Build mapping relationship between abstraction form and Datalog,
so users can write a policy in UI other than Datalog.
- Pass information from Horizon to Congress to finish the policy creation.
Dependencies
============
N/A
Testing
=======
Need to be tested with a variety of scenarios.
Documentation impact
====================
Add instructions for policy abstraction in UI.
References
==========
N/A

334
specs/liberty/policy-engine-api.rst
View File

@ -1,334 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Add policy engine class to API
==========================================
This blueprint makes it possible to access policy-engines (other than
the domain-agnostic policy engine) via the API.
Problem description
===================
Currently there is no way to access policy engines other than the current
domain-agnostic policy engine through the API. Now that we are adding
other policy engines, we need to give the user a way to interact with them.
In addition to being useful for domain-specific policy engines, enabling
multiple policy engines should make it easier to handle upgrades for the
domain-agnostic policy engine: bring up the new one alongside the old one,
then swap the old one with the new one atomically.
Proposed change
===============
Here we discuss two proposals. We list tradeoffs in the next section.
Proposal 1: Type-based
--------------------------
Here we simply add a top-level 'policy-engines' API endpoint, giving us ...
/v1/policy-engines/<engine-id>/...
/v1/data-sources/<datasource-id>/...
/v1/system/...
For example:
/data-sources/nova/...
/data-sources/neutron/...
/policy-engines/agnostic/...
Data-sources would support:
* schema: available tables
* actions: available actions
* status: status
Policy-engines would support:
* schema: available tables (e.g. classification:connected_to_internet)
* actions: available actions (e.g. scripts built into a policy-engine for
carrying out some task)
* status: status
* policies: available policies
That is, the only difference between a policy-engine and a data-source is that
the datasource doesn't support 'policies'. If we were to think of 'policies'
as simply 'modules' then we could imagine a datasource-driver exposing
hierarchical tables, just like a policy-engine does. And in that case,
the data-source would have something analogous to the policy-engine 'policies'.
Proposal 2: Service-based
--------------------------
Here we do away with the types policy-engine and data-sources and give clients
the ability to access both policy-engines and datasources from the same
endpoint. This is possible because you cannot give a policy-engine and
a datasource the same name (a restriction in place to ensure references to
other services in policy are unambiguous).
/v1/services/<service-id>/...
/v1/system/...
For example:
/v1/services/nova/...
/v1/services/neutron/...
/v1/services/agnostic/...
All services would support:
* schema: available tables (e.g. classification:connected_to_internet)
* actions: available actions (e.g. scripts built into a policy-engine for
carrying out some task)
* status: status
* policies: available policies
In short, this proposal treats everything as if it is a policy-engine. The
datasource policy engines prohibit users from making changes directly (and
even if they did, they would only accept a very restricted form of policy
statements: ground facts).
This approach can be backwards compatible as well. We can still support
/v1/datasources/<datasource-id>/...
which gets routed to
/v1/services/<datasource-id>.
And we can support
/v1/policies
/v1/actions
/v1/tables
/v1/status
as syntactic sugar for
/v1/services/<default-service>/policies
/v1/services/<default-service>/actions
/v1/services/<default-service>/tables
/v1/services/<default-service>/status
This would enable the user to choose a single touchpoint for managing policy
in the datacenter, while at the same time enabling them direct access to all
the services in the datacenter. (The only worry here is that )
To set the <default-service>, we'd want a parameter the user can set
dynamically (to help with upgrade).
Right now we would set that to /services/agnostic.
And maybe we can have arbitrary aliases for services as well, so that we can
upgrade any service without changing policy.
One worry with providing the /v1/policies, etc. endpoints is that it may
seem to mask Congress's overall status, policies, actions, and tables. That is,
people might expect those endpoints to aggregate all the potential policies,
actions, tables, and statuses. But if such functionality ever becomes
necessary, we can attach those endpoints to /v1/services, giving us the
following end points.
/services/policies
/services/actions
/services/tables
/services/status
Here is an example of the entry points if we had Nova, GBP, and our policy
engine. The name of the service is whatever name DSE expects.
/policies
/actions
/tables
/services/policies
/services/actions
/services/tables
/services/nova/policies -> empty
/services/nova/actions -> createVM, deleteVM, migrateVM, etc.
/services/nova/tables -> servers, hosts, etc.
/services/gbp/policies
/services/gbp/actions
/services/gbp/tables
/services/engine/policies
/services/engine/actions
/services/engine/tables
/system/drivers/
/system/engine-drivers/nova-uber-driver
/system/datasource-drivers/nova-uber-driver
/system/action-drivers/nova-uber-driver
/users
/stats
One benefit to enabling people to modify domain-specific policy engines
through the Congress API is that we provide a single policy language for
managing all the policy engines running in the datacenter. For delegation,
we already need adapters that translate Datalog into the native language of
each policy engine, so here we expose that functionality directly to the user
as well.
Alternatives
------------
N/A
Tradeoffs
------------
Pros for the type-based approach:
* Easy for users to understand
* Simple extension of the current API
Cons for the type-based approach:
* Awkward that data-sources and policy-engines implement almost exactly the
same interface and have separate namespaces, but are represented as
distinct classes in the API.
* Enables us to build datasources with significantly different programmatic
interface than policy-engines. If at the API-layer the two classes of
objects were almost indistinguishable, it would lead to better abstraction
and interfaces in the underlying implementation.
Pros for the the service-based approach:
* All services running on the DSE are accessed identically from the API. This
is a more natural reflection of the reality of the nature of those services.
Cons for the service-based approach:
* Bigger change
* May be more difficult for users to understand initially.
* Eventually the policy-engine class will include functionality that the
datasource class does not. Executing that functionality on a datasource
will cause a 404, and we cannot predict which will occur based on just the
URL.
* Doing something like listing all the datasources will require an API like
/v1/services?action=list&type=datasource instead of the more obvious
/v1/data-sources/.
Overall, the types (datasource vs. policyengine) will be present in both
proposals, but they will be emphasized much less in the service-based approach.
The service-based approach is closer to Python in that the system isn't able
to look at the code you've written (the URL) and check if the method you asked
for exists. The type-based approach is closer to C/Java in that the system IS
able to tell you if the method exists by just looking at the code (URL).
Typically policy systems are quite dynamic in nature (you can change the
policy/code at runtime), and hence are closer to dynamic programming languages
like Python than to static languages like C/Java. We therefore typically
bias our decisions toward dynamism, which in this case would mean the
service-based approach.
Policy
------
N/A
Policy actions
--------------
N/A
Data sources
------------
N/A
Data model impact
-----------------
N/A
REST API impact
---------------
See above. No changes to API results--just the paths for invoking them.
Security impact
---------------
N/A
Notifications impact
--------------------
N/A
Other end user impact
---------------------
N/A
Performance impact
------------------
N/A
Other deployer impact
---------------------
See above.
Developer impact
----------------
N/A
Implementation
==============
Assignee(s)
-----------
Primary assignee:
<launchpad-id or None>
Other contributors:
<launchpad-id or None>
Work items
----------
Once we decide on the approach, we will figure out the necessary work items.
But here's a rough cut.
- type-based approach: add routes, create congress/api/engine_model.py, modify
congress/api/\*_model to enable tables/actions/policies/etc. for engines.
- service-based approach: add routes, create congress/api/service_model.py,
(including an API to list different types of objects), modify the
congress/api/\*_model to eliminate distinction between datasources and
policies
Dependencies
============
Assumes that we have added an API call for 'actions', though this work could
be done without that: add-action-listing.
Testing
=======
- Change unit tests in congress/tests/test_congress.py
- Change congress_pythonclient (which will handle tempest tests)
Documentation impact
====================
Many URLs may change.
References
==========
N/A

174
specs/liberty/table-publishing-middleware.rst
View File

@ -1,174 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Create table publishing middleware
==========================================
https://blueprints.launchpad.net/congress/+spec/table-publishing-middleware
To lower the barrier to entry for other projects to publish data to Congress
on the (extended) DSE, create middleware that automates as much of the process
as possible.
Problem description
===================
Currently Congress periodically polls the services it is managing to get their
data. It would be preferable if services sent data to Congress only when that
data changes. For example, instead of Congress pulling the list of Nova's
servers every minute, Nova would send its servers to Congress whenever a
new server is created, an existing server is deleted, or an existing server
is updated.
oslo.messaging already makes it possible for projects to publish data
to other projects, but that mechanism is underutilized today.
Proposed change
===============
Solving this problem means making it even easier for other projects to send
their data to Congress than oslo.messaging does. This change will include
middleware, perhaps made available via oslo, that publishes all changes to the
underlying database tables on the bus. Moreover, it will
send not the entire table but rather the delta on the table that occurred.
It will be tightly integrated into existing oslo.db so that existing projects
need make no code changes; they need only include and configure the code.
There will need to be a synchronization mechanism: Congress pulls all the data
once and then needs deltas published on the bus from that point on. The deltas
sent on the bus will include timestamps that allow Congress to synchronize
the initial pull of data with the deltas.
Alternatives
------------
Another option is to have a separate effort, say oslo.publish_tables, that
is not so tightly integrated with oslo.db. The upside to this approach is that
projects not using oslo.db (even those outside of OpenStack) could leverage
the code. The downside to this approach is that projects already using oslo.db
would need to write and maintain a bunch of special-purpose code for publishing
their data--an approach that has seen little success til now.
Perhaps we will find that we can implement both approaches with little extra
effort, but the primary goal of this spec is the tight integration with
oslo.db.
Policy
------
None
Policy Actions
--------------
None
Data Sources
------------
None
Data model impact
-----------------
None
REST API impact
---------------
None
Security impact
---------------
None
Notifications impact
--------------------
The goal of this spec is to improve the ability of other projects to
utilize the notifications mechanism provided by oslo.messaging.
Other end user impact
---------------------
None
Performance impact
------------------
There will be minimal performance impact on the projects that utilize this
code because only the deltas that hit the database will be published on
the bus. Moreover, because each project can configure which tables
are published on the bus, the project owner can tune any
possible performance impact.
Other deployer impact
---------------------
When configuring the middleware, we propose one key configuration option:
which tables should be published on the bus.
Developer impact
----------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
<launchpad-id or None>
Other contributors:
<launchpad-id or None>
Work items
----------
- Write basic middleware functionality
- Integrate it into oslo.db
- Install it into an existing service, say Nova.
- Standalone documentation
Dependencies
============
enable-delta-driven-datasources: enable the datasources/DatasourceDriver
class to connect to datasources that send deltas.
Testing
=======
- Unit testing the basic functionality
- Tempest tests
o Verify that changes to tables configured to be published actually
get published
o Verify that changes to tables configured NOT to be published do not
get published
Documentation impact
====================
Need documentation in oslo for the basic middleware.
Need documentation update to oslo.db
References
==========
None

152
specs/mitaka/add-az-to-nova-driver.rst
View File

@ -1,152 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
===============================================================
Add availability zones to nova datasource driver
===============================================================
https://blueprints.launchpad.net/congress/+spec/add-az-to-nova-driver
This specification presents to extend the current nova datasource driver for
supporting availability zone and its fields available from nova.
Problem description
===================
The current nova provides avaiability zones for server and host. A host is a
part of availability zone and server can be booted on the availability zone.
However, the current nova datasource driver does not support it.
This availability zone are useful for creating any policy related to server
location.
For example, we can specify a policy for checking status of any VMs located in
the specific availability zone. In addition, we can specify an action policy
for migrating some VMs from availability zone A to availability zone B.
Recently, availability zone was added to servers using
OS-EXT-AZ:availability_zone [1].
However, it is hard to make full hierarchy among availability zone, hosts, and
servers and lack of flexiblity and extensibility. If we add the availability
zone to nova data source driver, it will be more flexible and extensible to use
more fields available from the availability zone such as zone state.
Proposed change
===============
Basically, we can extend the curret Nova datasource driver
(datasources/nova_driver.py) by adding the following a translator and
two fields.
* availabilty_zones_translator
* zone_name
* zone_state
Alternatives
------------
N/A
Policy
------
Using the Congress datalog syntax, write out an example policy using
https://wiki.openstack.org/wiki/Congress#Policy_Language
Example:
host_availability_zone(vm,zone_name) :-
nova:hosts(host_name,zone=zone_name),
nova:availability_zones(zone_name=zone_name)
Policy actions
--------------
Data sources
------------
nova
Data model impact
-----------------
N/A
REST API impact
---------------
N/A
Security impact
---------------
N/A
Notifications impact
--------------------
N/A
Other end user impact
---------------------
N/A
Performance impact
------------------
N/A
Other deployer impact
---------------------
N/A
Developer impact
----------------
N/A
Implementation
==============
Assignee(s)
-----------
Primary assignee:
Joon Kang <joon-myung-kang>
Work items
----------
* Implement availability_zone_translator in nova datasource driver
* Perform unit testing
* Perform functional testing for the new field
Dependencies
============
N/A
Testing
=======
* Test host and availability zones using the policy
Documentation impact
====================
TBD
References
==========
[1] /congress/commit/9f9e26f850d12c73721b5e6b45ac00997c8c24c2
[2] http://docs.openstack.org/openstack-ops/content/scaling.html

199
specs/mitaka/push-type-datasource-driver.rst
View File

@ -1,199 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Push Type DataSource Driver
==========================================
https://blueprints.launchpad.net/congress/+spec/push-type-datasource-driver
Problem description
===================
Currently DataSource Driver retrieves all cloud service's info by polling
each datasource. The purpose of the polling is just to monitor cloud
service periodically. Thus Congress can't notice a change in cloud service
just after the change occurs. It means a policy violation isn't detected
until Congress polls the change. Additionally, Congress calls the service's
API to get the data for a table, so datasource table's schema depends on API
response contents.
In the area of monitoring, tons of well-developed monitoring system have
already existed in the world. The systems have an ability to notify a change
in cloud services just after a change occurs. And then these monitoring systems
can detect many kind of changes in cloud services that Congress can't do.
Congress should retrieve the notification as a DataSource since it helps
Congress to calculate a policy violation faster than now and also enables
us to define varied policies and rules.
Proposed change
===============
Following changes are proposed in the BP.
1. Create PollingDataSourceDriver and PushedDataSourceDriver class. Current
DataSourceDriver has 2 features. One is converting data to datasource table
and sending it to Policy Engine. Another is polling data from datasource.
We should separate the polling feature into another class that is subclass
of DataSourceDriver named PollingDataSourceDriver. And we create
PushedDataSourceDriver, which is subclass of DataSourceDriver and receives
data from a monitoring system.
2. Create HttpReceiverDriver subclass or OsloMsgReceiverDriver subclass
Both Drivers are a subclass of PushedDataSourceDriver and receive
datasource info by the protocol the class name has. DataSourceDriver has
to keep datasource status in itself, so the classes will support 2 types
of a data receiving method. One is receiving all status data each push
time. Another is receiving only delta data that represents which row is
deleted or added in the status like REST API.
First of all, the BP chooses and implements one of them based on an usecase,
which needs to use PushedDatasourceDriver.
There are some restrictions as a first implementation.
* Table definitions
Ideally speaking, PushedDataSourceDriver should allow admin to change a
datasource name, table name, schema and so on by Congress API.
PushedDataSourceDriver defines the protocol name as a datasource name in
first implementation. Then the admin can define a translator in another
python file. In the current architecture, the path is set by 'config' key
in a call for create datasource API. In the distributed architecture, the
path is set by a config file for each datasource driver.
Alternatives
------------
The alternative is that Congress allows admin to add and delete a row by API
calls. The API is a REST API, so the benefit is that both create and delete
methods are already developed in itself. However, the way has following
bad points. First is all request needs an authentication to push data. It
means all datasource have to be related to keystone authority. Second is
it's not easy to divide a network segmentation based on each API's purpose.
If Congress enables API to retrieve datasource, the datasource receiver is
exposed to user.
Policy
------
Example: define a table that shows service name related to host's failures
detected by monitoring system.
Assumption: monitoring system pushes data by http. col_1 represents name of
a host where the monitoring system detects
some failure. col_2 represents a failure type.
host_failures(host):-
http:table(col_1=host, col_2="nic_error")
host_failures(host):-
http:table(col_1=host, col_2="disk_error")
failure_effected_service(service, host) :-
nova:service(host_name=host, service=service),
host_failures(host=host)
Policy actions
--------------
N/A
Data sources
------------
Add new datasource model and table.
Data model impact
-----------------
N/A
REST API impact
---------------
Creating new datasource name, table name and table schema by API is a future
work. It means this feature is out of scope for this BP.
Security impact
---------------
It enables others to push datasource row into the datasource driver. It would
cause undesired policy violation if malicious users send data on purpose.
Notifications impact
--------------------
N/A
Other end user impact
---------------------
N/A
Performance impact
------------------
PushedDataSourceDriver notifies a status change when it receives a change from
a monitoring system. If the system frequently sends a notification, it would
make Congress busy with calculating policy table.
Other deployer impact
---------------------
* Add a config for column number
* Disable the PushedDataSourceDriver in default because of a reason described
in Security Impact section
Developer impact
----------------
N/A
Implementation
==============
Assignee(s)
-----------
Primary assignee:
muroi-masahito
Other contributors:
None
Work items
----------
0. Create PollingDataSourceDriver subclass and pull out methods related to
polling from DataSourceDriver class into the new Driver
1. Create PushedDataSourceDriver subclass
2. Create HttpReceiverDriver or OsloMsgReceiverDriver
Dependencies
============
N/A
Testing
=======
* Add unit tests related to new class
* Add tempest scenario tests which use pushed data in rules if possible
Documentation impact
====================
DataSource list will be updated if the list is described in docs.
References
==========
Tokyo Summit discussion: https://etherpad.openstack.org/p/congress-mitaka-external

208
specs/mitaka/unknown-schemas.rst
View File

@ -1,208 +0,0 @@
..
==========================================
Policy Support for Unknown Table Schemas
==========================================
The new distributed architecture requires the policy engine to
handle the case when the schema for some datasource drivers are
unknown. Today the policy engine assumes the schemas for all
datasource drivers are known at load-time. This spec outlines
a mechanism for supporting unknown schemas.
Problem description
===================
For the new distributed architecture, the policy engine will not know
the schema for all the datasources at the time rules are loaded from the
database. This is currently problematic because column-references are
compiled away at the time rules are loaded from the database, and that
compilation procedure requires the schema. Thus in the new architecture,
the policy engine will crash when it tries to load policy rules that
contain column references.
Proposed change
===============
The fix to this problem is to enable the policy engine to load rules
that include column references without compiling away those column references.
That is, the main reason to compile away column references is that the
internal datastructures for representing rules cannot represent those
column references natively, and hence, the column references must be
removed at load-time. The first task is to extend the internal
datastructures in compile.py so they can represent named-columns.
The second reason column references get compiled away is that they cannot
be evaluated (even semantically) without the schema. The second task
is to extend the run-time capabilities of the policy engine so that
rules can be disabled without being deleted. A disabled rule will
be completely hidden from the evaluation engine when answering queries
yet will still be visible but marked as "disabled" when users view
the rules.
Every time a new rule is inserted, if its schema is unknown, that
rule must be disabled. Moreover, every rule using a table dependent
on the table in the head of that rule must be disabled. Similarly
for deletion except that deletion can cause other rules to be enabled.
Every time the schema changes, all rules impacted by that schema
change should be checked for consistency, and disabled rules
must be enabled once all schemas are known. Once a rule
is enabled, the column references are compiled away.
If a 2nd schema arrives (unequal to the first), the policy engine
must check for consistency and recompile any rules whose schema
may have changed.
For example, if the following rule is inserted before the schemas
for nova-servers and neutron-networks is known, the rule will
be disabled since it has column references.
p(x, z) :- nova:servers(id=x, network=y), neutron:networks(id=y, status=z)
Then when the nova schema becomes known this rule is validated
against that schema but is not enabled because the neutron schema
is unknown.
Finally when the neutron schema becomes known, the column references
are compiled away and the rule is officially enabled.
Alternatives
------------
Instead of disabling rules, another option is to modify the
evaluation engine to do a best-effort query evaluation. The evaluation
algorithms themselves would know about column-references, and would
attempt to operate even if the schema was unknown.
The downside to this alternative is that the rules are actually
semantically ambiguous, and hence the result of evaluation has
unknown semantic value.
Policy
------
N/A
Policy actions
--------------
N/A
Data sources
------------
N/A
Data model impact
-----------------
No database modifications are required.
REST API impact
---------------
The Rule object will have an additional boolean field representing whether
or not the rule is disabled.
Security impact
---------------
N/A
Notifications impact
--------------------
N/A
Other end user impact
---------------------
N/A
Performance impact
------------------
Rule inserts could now be slower since if the rule inserted gets disabled,
that could cause many other rules to be disabled.
Rule deletions likewise could cause many policy rules to be enabled.
Schema updates are expensive because the policy engine must do consistency
checks on all rules that are relevant, and potentially re-compile rules.
Other deployer impact
---------------------
N/A
Developer impact
----------------
N/A
Implementation
==============
Assignee(s)
-----------
Primary assignee:
thinrichs
Other contributors:
<launchpad-id or None>
Work items
----------
1. Modify compile.py datastructures to natively represent column
references. Include a 'disabled' flag.
2. Modify query evaluation engine to ignore disabled rules
3. Modify triggers to ignore disabled tables
4. Enable/disable rules on insert/delete/set-schema
- Write dependency analysis routine to compute the rules/tables
that are disabled once a given table is disabled.
Likely to need a datastructure that tracks disabled tables.
- Modify update routine to do schema check and enable/disable rules
as appropriate using the dependency analysis.
- Modify set-schema to appropriately enable/disable rules
May want to add field to rules that say which tables the compilation
was dependent on.
Dependencies
============
N/A
Testing
=======
Unit test coverage should be mostly adequate.
Only real need for tempest tests would be testing the startup of Congress,
but that is not supported with tempest.
Documentation impact
====================
N/A
References
==========
The need for this spec was discussed at the Liberty Midcycle Sprint.
https://etherpad.openstack.org/p/congress-liberty-sprint

746
specs/newton/high-availability-design.rst
View File

@ -1,746 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
=====================================================
Support high availability, high throughput deployment
=====================================================
Include the URL of your launchpad blueprint:
https://blueprints.launchpad.net/congress/+spec/high-availability-design
Some applications require Congress to be highly available (HA). Some
applications require a Congress policy engine to handle a high volume of
queries (high throughput - HT). This proposal describes how we can support
several deployment schemes that address several HA and HT requirements.
Problem description
===================
This spec aims to address three main problems:
1. Congress is not currently able to provide high query throughput because
all queries are handled by a single, single-threaded policy engine instance.
2. Congress is not currently able to failover quickly when its policy engine
becomes unavailable.
3. If the policy engine and a push datasource driver both crash, Congress is
not currently able to restore the latest data state upon restart or
failover.
Proposed change
===============
Implement the required code changes and create deployment guides for the
following reference deployments.
Warm standby for all Congress components in single process
----------------------------------------------------------
- Downtime: ~1 minute (start a new Congress instance and ingest data from
scratch)
- Reliability: action executions may be lost during downtime.
- Performance considerations: uniprocessing query throughput
- Code changes: minimal
Active-active PE replication, DSDs warm-standby
----------------------------------------------------
Run N instances of Congress policy engine in active-active configuration. One
datasource driver per physical datasource published data on oslo-messaging to
all policy engines.
::
+-------------------------------------+ +--------------+
| Load Balancer (eg. HAProxy) | <----+ Push client |
+----+-------------+-------------+----+ +--------------+
| | |
PE | PE | PE | all+DSDs node
+---------+ +---------+ +---------+ +-----------------+
| +-----+ | | +-----+ | | +-----+ | | +-----+ +-----+ |
| | API | | | | API | | | | API | | | | DSD | | DSD | |
| +-----+ | | +-----+ | | +-----+ | | +-----+ +-----+ |
| +-----+ | | +-----+ | | +-----+ | | +-----+ +-----+ |
| | PE | | | | PE | | | | PE | | | | DSD | | DSD | |
| +-----+ | | +-----+ | | +-----+ | | +-----+ +-----+ |
+---------+ +---------+ +---------+ +--------+--------+
| | | |
| | | |
+--+----------+-------------+--------+--------+
| |
| |
+-------+----+ +------------------------+-----------------+
| Oslo Msg | | DBs (policy, config, push data, exec log)|
+------------+ +------------------------------------------+
- Downtime: < 1s for queries, ~2s for reactive enforcement
- Deployment considerations:
- Cluster manager (eg. Pacemaker + Corosync) can be used to manage warm
standby
- Does not require global leader election
- Performance considerations:
- Multi-process, multi-node query throughput
- No redundant data-pulling load on datasources
- DSDs node separate from PE, allowing high load DSDs to operate more
smoothly and avoid affecting PE performance.
- PE nodes are symmetric in configuration, making it easy to load balance
evenly.
- Code changes:
- New synchronizer and harness to support two different node types:
API+PE node and all-DSDs node
Details
###########################################################################
- Datasource drivers (DSDs):
- One datasource driver per physical datasource.
- All DSDs run in a single DSE node (process)
- Push DSDs: optionally persist data in push data DB, so a new snapshot can
be obtained whenever needed.
- Policy engine (PE):
- Replicate policy engine in active-active configuration.
- Policy synchronized across PE instances via Policy DB
- Every instance subscribes to the same data on oslo-messaging.
- Reactive enforcement:
All PE instances initiate reactive policy actions, but each DSD locally
selects a "leader" to "listen to". The DSD ignores execution requests
initiated by all other PE instances.
- Every PE instance computes the required reactive enforcement actions and
initiate the corresponding execution requests over oslo-messaging.
- Each DSD locally picks PE instance as leader (say the first instance the
DSD hears from in the asymmetric node deployment, or the PE instance on
the same node as the DSD in a symmetric node deployment) and executes
only requests from that PE.
- If heartbeat contact is lost with the leader, the DSD selects a new
leader.
- Each PE instance is unaware of whether it is a "leader"
- API:
- Each node has an active API service
- Each API service routes requests for PE to its associated intranode PE
- Requests for any other service(eg. get data source status) are routed to
DSE2, which will be fielded by some active instance of the service on some
node
- Details:
- in API models, replace every invoke_rpc with a conditional:
- if the target is policy engine, target same-node PE
eg.::
self.invoke_rpc(
caller, 'get_row_data', args, server=self.node.pe.rpc_server)
- otherwise, invoke_rpc stays as is, routing to DSE2
eg.::
self.invoke_rpc(caller, 'get_row_data', args)
- Load balancer:
- Layer 7 load balancer (e.g. HAProxy) distributes incoming API calls among
the nodes (each running API service).
- load balancer optionally configured to use sticky session to pin each API
caller to a particular node. This configuration avoids the experience of
going back in time.
- External components (load balancer, DBs, and oslo messaging bus) can be made
highly available using standard solutions (e.g. clustered LB, Galera MySQL
cluster, HA rabbitMQ)
Dealing with missed actions during failover
###########################################################################
When a leader fails (global or local), it takes time for the failure to be
detected and a new leader anointed. During the failover, reactive enforcement
actions expected to be triggered would be missed. Four proposed approaches
are discussed below.
- Tolerate missed actions during failover: for same applications, it may be
acceptable to miss actions during failover.
- Re-execute recent actions after failover
- Each PE instance remembers its recent action requests (including the
requests a follower PE computed but did not send)
- On failover, the DSD requests the recent action requests from the new
leader and executes them (within a certain recency window)
- Duplicate execution expected on failover.
- Re-execute recent unmatched actions after failover (possible future work)
- We can reduce the number of duplicate executions on failover by attempting
to match a new leader's recent action requests with the already executed
requests, and only additionally executing those unmatched.
- DSD logs all recent successfully executed action requests in DB
- Request matching can be done by a combination of the following information:
- the action requested
- the timestamp of the request
- the sequence number of the data update that triggered the action
- the full derivation tree that triggers the action
- Matching is imperfect, but still helpful
- Log and replay data updates (possible future work)
- Log every data update from every data source, and let a new leader replay
the updates where the previous leader left off to generate the needed
action requests.
- The logging can be directly supported by transport or by additional DB
- kafka naturally supports this model
- hard to do directly with oslo-messaging + RabbitMQ
- Leaderless de-duplication (possible future work)
- If a very good matching method is implemented for re-execution for recent
unmatched actions after failover, it is possible to go one stop further
and simply operate in this mode full time.
- Each incoming action request is matched against all recently executed
action requests.
- Discard if matched.
- Execute if unmatched.
- Eliminates the need for selecting leader (global or local) and improves
failover speed
We propose to focus first on supporting the first two options
(deployers' choice). The more complex options may be implemented and supported
in future work.
Alternatives
------------
We first discuss the main decision points before detailing several alternative
deployments.
For active-active replication of the PE, here are the main decision points:
A. node configurations
- Options:
1. single node-type (every node has API+PE+DSDs).
2. two node-types (API+PE nodes, all-DSDs node). [proposed target]
3. many node-types (API+PE nodes, all DSDs in separate nodes).
- Discussions: The many node-types configuration is most flexible and has
the best support for high-load DSDs, but it also requires the most work to
dev and to deploy.
We propose to target the two node-types configuration because it gives
reasonable support for high-load DSDs while keeping both the development
and the deployment complexities low.
B. global vs local leader for action execution
- Options:
1. global leader: Pacemaker anoints a global leader among PE instances;
only the leader sends action-execution requests.
2. local leader: every PE instance sends action-execution requests, but
each receiving DSD locally picks a "leader" to listen to.
[proposed target]
- Discussions: Because there is a single active DSD for a given data source,
it is a natural spot to locally choose a "leader" among the PE instances
sending reactive enforcement action execution requests.
We propose to target the local leader style because it avoids the
development and deployment complexities associated with global leader
election.
Furthermore, because all PE instances perform reactive enforcement and send
action execution requests, the redundancy opens up the possibility for
zero disruption to reactive enforcement when a PE intance fails.
C. DSD redundancy
- Options:
1. warm standby: only one set of DSDs running at a given time; backup
instances ready to launch.
2. hot standby: multiple instances running, but only one set is active.
3. active-active: multiple instances active.
- Discussions:
- For pull DSDs, we propose to target warm standby seems most appropriate
because warm startup time is low (seconds) relative to frequency of data
pulls.
- For push DSDs, warm standby is generally sufficient except for use cases
that demand sub-second latency even during a failover. Those use cases
would require active-active replication of the push DSDs. But even with
active-active replication of push DSDs, other unsolved issues in
action-execution prevent us from delivering sub-second end-to-end latency
(push data to triggered action executed) during failover (see leaderless
de-duplication approach for sub-second action execution failover).
Since we cannot yet realize the benefit of active-active replication of
push DSDs, we propose to target a warm-standby deployment, leaving
active-active replication as potential future work.
Active-active PE replication, DSDs hot-standby, all components in one node
###########################################################################
Run N instances of single-process Congress.
One instance is selected as leader by Pacemaker. Only the leader has active
datasource drivers (which pull data, accept push data, and accept RPC calls
from the API service), but all instances subscribes to and processes data on
oslo-messaging. Queries are load balanced among instances.
::
+-----------------------------------------------------------------------+
| Load Balancer (eg. HAProxy) |
+----+------------------------+------------------------+----------------+
| | |
| leader | follower | follower
+---------------------+ +---------------------+ +---------------------+
| +-----+ +---------+ | | +-----+ +---------+ | | +-----+ +---------+ |
| | API | |DSD (on) | | | | API | |DSD (off)| | | | API | |DSD (off)| |
| +-----+ +---------+ | | +-----+ +---------+ | | +-----+ +---------+ |
| +-----+ +---------+ | | +-----+ +---------+ | | +-----+ +---------+ |
| | PE | |DSD (on) | | | | PE | |DSD (off)| | | | PE | |DSD (off)| |
| +-----+ +---------+ | | +-----+ +---------+ | | +-----+ +---------+ |
+---------------------+ +---------------------+ +---------------------+
| | |
| | |
+---------+--------------+-------------------+----+
| |
| |
+--------------+-----------+ +--------------------+---------------------+
| Oslo Msg | | DBs (policy, config, push data, exec log)|
+--------------------------+ +------------------------------------------+
- Downtime: < 1s for queries, ~2s for reactive policy
- Deployment considerations:
- Requires cluster manager (Pacemaker) and cluster messaging (Corosync)
- Relatively simple Pacemaker deployment because every node is identical
- Requires leader election (handled by Pacemaker+Corosync)
- Easy to start new DSD (make API call, all instances sync via DB)
- Performance considerations:
- [Pro] Multi-process query throughput
- [Pro] No redundant data-pulling load on datasources
- [Con] If some data source drivers have high demand (say telemetry data),
performance may suffer when deployed in the same python process as other
Congress components.
- [Con] Because the leader has the added load of active DSDs, PE performance
may be reduced, making it harder to evenly load balance across instances.
- Code changes:
- Add API call to designate a node as leader or follower
- Custom resource agent that allows Pacemaker to start, stop, promote, and
demote Congress instances
Details
++++++++++++++
- Pull datasource drivers (pull DSD):
- One active datasource driver per physical datasource.
- Only leader node has active DSDs (active polling loop and active
RPC server)
- On node failure, new leader node activates DSDs.
- Push datasource drivers (push DSD):
- One active datasource driver per physical datasource.
- Only leader node has active DSDs (active RPC server)
- On node failure, new leader node activates DSDs.
- Persist data in push data DB, so a new snapshot can be obtained.
- Policy engine (PE):
- Replicate policy engine in active-active configuration.
- Policy synchronized across PE instances via Policy DB
- Every instance subscribes to the same data on oslo-messaging.
- Reactive enforcement: See later section "Reactive enforcement architecture
for active-active deployments"
- API:
- Add new API calls for designating the receiving node as leader or follower.
The call must complete all tasks before returning (eg. start/stop RPC)
- Each node has an active API service
- Each API service routes requests for PE to its associated intranode PE,
bypassing DSE2.
- Requests for any other service(eg. get data source status) are routed to
DSE2, which will be fielded by some active instance of the service on some
node
- Details:
- in API models, replace every invoke_rpc with a conditional:
- if the target is policy engine, target same-node PE
eg.::
self.invoke_rpc(
caller, 'get_row_data', args,
server=self.node.pe.rpc_server)
- otherwise, invoke_rpc stays as is, routing to DSE2
eg.::
self.invoke_rpc(caller, 'get_row_data', args)
- Load balancer:
- Layer 7 load balancer (e.g. HAProxy) distributes incoming API calls among
the nodes (each running API service).
- load balancer optionally configured to use sticky session to pin each API
caller to a particular node. This configuration avoids the experience of
going back in time.
- Each DseNode is monitored and managed by a cluster manager (eg. Pacemaker)
- External components (load balancer, DBs, and oslo messaging bus) can be made
highly available using standard solutions (e.g. clustered LB, Galera MySQL
cluster, HA rabbitMQ)
- Global leader election with Pacemaker:
- A resource agent contains the scripts that tells a Congress instance it is
a leader or follower.
- Pacemaker decides which Congress instance to promote to leader (master).
- Pacemaker promotes (demotes) the appropriate Congress instance to leader
(follower) via the resource agent.
- Fencing:
- If the leader node stops responding, and a new node is promoted to
leader, it is possible that the unresponsive node is still doing work
(eg. listening on oslo-messaging, issuing action requests).
- It is generally not a catastrophe if for a time there is more than one
Congress node doing the work of a leader. (Potential effects may include:
duplicate action requests and redundant data source pulls)
- Pacemaker can be configured with strict fencing and STONITH for
deployments that require it. (deployers' choice)
http://clusterlabs.org/doc/en-US/Pacemaker/1.1/html-single/Pacemaker_Explained/#_what_is_stonith
- In case of network partitions:
- Pacemaker can be configured to stop each node that is not part of a
cluster reaching quorum, or to allow each partition to continue
operating. (deployers' choice)
http://clusterlabs.org/doc/en-US/Pacemaker/1.0/html/Pacemaker_Explained/s-cluster-options.html
Active-active PE replication, DSDs warm-standby, each DSD in its own node
###########################################################################
Run N instances of Congress policy engine in active-active configuration. One
datasource driver per physical datasource published data on oslo-messaging to
all policy engines.
::
+-------------------------------------+
| Load Balancer (eg. HAProxy) |
+----+-------------+-------------+----+
| | |
| | |
+---------+ +---------+ +---------+
| +-----+ | | +-----+ | | +-----+ |
| | API | | | | API | | | | API | |
| +-----+ | | +-----+ | | +-----+ |
| +-----+ | | +-----+ | | +-----+ |
| | PE | | | | PE | | | | PE | |
| +-----+ | | +-----+ | | +-----+ |
+---------+ +---------+ +---------+
| | |
| | |
+------------------------------------------+-----------------+
| | | | |
| | | | |
+----+-------------+-------------+---+ +-------+--------+ +-----+-----+
| Oslo Msg | | Push Data DB | | DBs |
+----+-------------+-------------+---+ ++---------------+ +-----------+
| | | | (DBs may be combined)
+----------+ +----------+ +----------+
| +------+ | | +------+ | | +------+ |
| | Poll | | | | Poll | | | | Push | |
| | Drv | | | | Drv | | | | Drv | |
| | DS 1 | | | | DS 2 | | | | DS 3 | |
| +------+ | | +------+ | | +------+ |
+----------+ +----------+ +----------+
| | |
+---+--+ +---+--+ +---+--+
| | | | | |
| DS 1 | | DS 2 | | DS 3 |
| | | | | |
+------+ +------+ +------+
- Downtime: < 1s for queries, ~2s for reactive policy
- Deployment considerations:
- Requires cluster manager (Pacemaker) and cluster messaging (Corosync)
- More complex Pacemaker deployment because there are many different
kinds of nodes
- Does not require global leader election (but that's not a big saving if
we're running Pacemaker+Corosync anyway)
- Performance considerations:
- [Pro] Multi-process query throughput
- [Pro] No redundant data-pulling load on datasources
- [Pro] Each DSD can run in its own node, allowing high load DSDs to operate
more smoothly and avoid affecting PE performance.
- [Pro] PE nodes are symmetric in configuration, making it easy to load
balance evenly.
- Code changes:
- New synchronizer and harness and DB schema to support per node
configuration
Hot standby for all Congress components in single process
###########################################################################
Run N instances of single-process Congress, as proposed in:
https://blueprints.launchpad.net/congress/+spec/basic-high-availability
A floating IP points to the primary instance which handles all queries and
requests, failing over when primary instance is down.
All instances ingest and process data to stay up to date.
::
+---------------+
| Floating IP | - - - - - - + - - - - - - - - - - - -+
+----+----------+ | |
|
| | |
+---------------------+ +---------------------+ +---------------------+
| +-----+ +---------+ | | +-----+ +---------+ | | +-----+ +---------+ |
| | API | | DSD | | | | API | | DSD | | | | API | | DSD | |
| +-----+ +---------+ | | +-----+ +---------+ | | +-----+ +---------+ |
| +-----+ +---------+ | | +-----+ +---------+ | | +-----+ +---------+ |
| | PE | | DSD | | | | PE | | DSD | | | | PE | | DSD | |
| +-----+ +---------+ | | +-----+ +---------+ | | +-----+ +---------+ |
| +-----------------+ | | +-----------------+ | | +-----------------+ |
| | Oslo Msg | | | | Oslo Msg | | | | Oslo Msg | |
| +-----------------+ | | +-----------------+ | | +-----------------+ |
+---------------------+ +---------------------+ +---------------------+
| | |
| | |
+----------+------------------------+----------------------+------------+
| Databases |
+-----------------------------------------------------------------------+
- Downtime: < 1s for queries, ~2s for reactive policy
- Feature limitations:
- Limited support for action execution: each action execution is triggered
N times)
- Limited support for push drivers: push updates only primary instance
(optional DB-sync to non-primary instances)
- Deployment considerations:
- Very easy to deploy. No need for cluster manager. Just start N independent
instances of Congress (in-process messaging) and setup floating IP.
- Performance considerations:
- Performance characteristics similar to single-instance Congress
- [Con] uniprocessing query throughput (optional load balancer can be added
to balance queries between instances)
- [Con] Extra load on data sources from replicated data source drivers
pulling same data N times
- Code changes:
- (optional) DB-sync of pushed data to non-primary instances
Policy
------
Not applicable
Policy actions
--------------
Not applicable
Data sources
------------
Not applicable
Data model impact
-----------------
No impact
REST API impact
---------------
No impact
Security impact
---------------
No major security impact identified compared to a non-HA distributed
deployment.
Notifications impact
--------------------
No impact
Other end user impact
---------------------
Proposed changes generally transparent to end user. Some exceptions:
- Different PE instances may be out-of-sync in their data and policies
(eventual consistency). The issue is generally made transparent to the end
user by making each user sticky to a particular PE instance. But if
a PE instance goes down, the end user reaches a different instance and may
experience out-of-sync artifacts.
Performance impact
------------------
- In single node deployment, there is generally no performance impact.
- Increased latency due to network communication required by multi-node
deployment
- Increased reactive enforcement latency if action executions are persistently
logged to facilitate smoother failover
- PE replication can achieve greater query throughput
Other deployer impact
---------------------
- New config settings:
- set DSE node type to one of the following:
- PE+API node
- a DSDs node
- all-in-one node (backward compatible default)
- set reactive enforcement failover strategy:
- do not attempt to recover missed actions (backward compatible default)
- after failover, repeat recent action requests
- after failover, repeat recent action requests not matched to logged
executions
- Proposed changes have no impact on existing single-node deployments.
100% backward compatibility expected.
- Changes only have effect if deployer chooses to set up a multi-node
deployment with the appropriate configs.
Developer impact
----------------
No major impact identified.
Implementation
==============
Assignee(s)
-----------
Work to be assigned and tracked via launchpad.
Work items
----------
Items with order dependency:
1. API routing.
Change API routing to support active-active PE replication, routing PE-bound
RPCs to the PE instance on the same node as the receiving API server.
Changes expected to be confined to congress/api/*
2. Reactive enforcement.
Change datasource_driver.py:ExecutionDriver class to handle action execution
requests from replicated PE-nodes (locally choose leader to follow)
3. (low priority) Missed actions mitigation.
- Implement changes to mitigate missed actions during DSD failover
- Implement changes to mitigate missed actions during PE failover
Items without dependency:
- Push data persistence.
Change datasource_driver.py:PushedDataSourceDriver class to support
persistence of pushed data. Corresponding DB changes also needed.
- (potential) Leaderless de-duplication of action execution requests.
- HA guide.
Create HA guide sketching the properties and trade-offs of each different
deployment types.
- Deployment guide.
Create deployment guide for active-active PE replication
Dependencies
============
- Requires Congress to support distributed deployment (for example with policy
engine and datasource drivers on separate DseNodes.) Distributed deployment
has been addressed by several implemented blueprints. The following patch is
expected to be the final piece required.
https://review.openstack.org/#/c/307693/
- This spec does not introduce any new code of library dependencies.
- In line with OpenStack recommendation
(http://docs.openstack.org/ha-guide/controller-ha-pacemaker.html), some
reference deployments use open source software outside of OpenStack:
- HAProxy: http://www.haproxy.org
- Pacemaker: http://clusterlabs.org/wiki/Pacemaker
- Corosync: http://corosync.github.io/corosync/
Testing
=======
We propose to add tempest tests for the following scenarios:
- Up to (N - 1) PE-nodes killed
- Previously killed PE-nodes rejoin.
- Kill and restart DSDs-node, possibly at the same time PE-nodes are killed.
Split brain scenarios can be manually tested.
Documentation impact
====================
Deployment guide to be added for each supported reference deployment. No impact
on existing documentation.
References
==========
- IRC discussion on major design decisions (#topic HA design):
http://eavesdrop.openstack.org/meetings/congressteammeeting/2016/congressteammeeting.2016-06-09-00.04.log.txt
- Notes from summit session:
https://etherpad.openstack.org/p/newton-congress-availability
- OpenStack HA guide:
http://docs.openstack.org/ha-guide/controller-ha-pacemaker.html
- HAProxy documentation:
http://www.haproxy.org/#docs
- Pacemaker documentation:
- directory: http://clusterlabs.org/doc/
- Cluster from scratch:
http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html/Clusters_from_Scratch/index.html
- Configuration explained:
http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html/Pacemaker_Explained/index.html
- OCF resource agents:
http://www.linux-ha.org/wiki/OCF_Resource_Agents

249
specs/newton/selectability-for-translator.rst
View File

@ -1,249 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
============================
Selectability for Translator
============================
https://blueprints.launchpad.net/congress/+spec/selectable-translator
Problem description
===================
DataSource Driver creates all translator and publishes its schema when
it's initialized. If the driver is one of polling driver, it polls data
of all translator from related services, even though any policy rules don't
use the schema. The behavior causes performance issues when an unnecessary
translator takes a long time to fetch data from related service.
Admin can choose which DataSource Driver they use or don't use by configuring
congress.conf and creating DataSource by API. On the other hands, there is
no way for admin to choose which translator in the created DataSource publishes
their schema. So when the update frequency of an unnecessary translator is
high, the issue described above also happens.
Proposed change
===============
This BP proposes a new feature that admin can set *lazy* to polling
data from services until other service requires schema of the datasource.
The issue described above doesn't happen in the usual case. So datasources
poll data for all table in default. If admin don't want a datasource to
poll unnecessary data, they disableds polling flag of the datasource and then
the datasource doesn't poll data if not needed.
Admin defines a list of lazy tables in datasource config parameter when
he/she instantiates a new datasouce. The datasource doesn't poll data related
to tables listed in the config value while any policy rule doesn't subscribe
the table. If datasource notices/recieves the subscription it starts to poll
data.
There are 2 possible corner cases.
* lazy parent translator and non-lazy child translator
This issue happens when parent translator is defined as a lazy but child
translator is not defined. Each table is mapped to one translator and
some translators have a hierarchy for extracting one pulled data to some
tables. When admin specifies a table mapped to a child translator as a lazy
but parent translator is not lazy, details of the issue are that the
datasource should pull data or not.
In this proposal, the datasource tries to pull data. Having the hierarchy
is depending on an implementation of the client the datasource uses. If
the client has separeted methods for pulling both data the datasource could
use each method.
* list table row API for a lazy table
This issue happens when user calls list table row API for a lazy table before
any service subscribes it. The table has no data about the table and the
datasource can't return row data. Details of the issue are that the
datasource should pull data and return a response of row or not when user
calls the API.
In this proposal, the API returns 400 Bad Request error with a message about
the table doesn't start to poll since the table is defined as lazy. If the
API returns a response with row data user can't recognize the table is a lazy
and the row data is up-to-date or not.
Alternatives
------------
Specifying a table that doesn't poll is one of alternatives. This allows
admin to handle each table to poll or not. Additionally, the datasource
doesn't pull data in not polling status even though policy rules using the
table are defined. But it also requires admin to manage all table manually.
It's hard to operate.
Enabling admin to config polling time delta for each translator is one of
answers to resolve a polling performance issue. The good point is admin
can write a policy rule related to the translator anytime since PolicyEngine
has its schema. The down side, however, is policy calculation could be delayed
since the delta is longer than in usual case.
Admin can set the enabled flag by config file for each translator too.
Using config file is an easy way to disable it and has low impacts for codes.
However, he/she have to restart DataSource everytime they want to change the
flag.
Policy
------
N/A
Policy actions
--------------
N/A
Data sources
------------
Datasource Driver have a functionality to start/stop pull data from service.
Datasource starts to pull ALL data in default just after initialized. If lazy
config is specified to some tables, the datasource waits to start to pull data
related to the tables until other service subscribes the table. And then when
there is no subscriber for the lazy table the datasource stops to pull data.
There is no change from PolicyEngine perspectives.
Data model impact
-----------------
Datasources have a list of lazy table in its config values. Currently, all
config values are stored in DB, so the list is also stored there.
REST API impact
---------------
This proposal expands parameters of existing APIs.
* Define a list of lazy tables
* API Schema
POST /v1/data-sources
* Request Body
.. code-block:: javascript
{
"config": {
...
"lazy_table": "[table1-a, table-b]",
...
},
...
}
* Response Body
.. code-block:: javascript
{
"config": {
...
"lazy_table": "[table1-a, table-b]",
...
},
...
}
List show a datasource details
* API Schema
GET /v1/data-sources/<datasoure-id>
* Response Body
.. code-block:: javascript
{
"config": {
...
"lazy_table": "[table1-a, table-b]",
...
},
...
}
Security impact
---------------
N/A
Notifications impact
--------------------
N/A
Other end user impact
---------------------
N/A
Performance impact
------------------
As described in Problem description section, the feature is to improve
PollingDataSource Driver. It reduces polling time and amount of sent data
from DataSource to PolicyEngine.
Other deployer impact
---------------------
N/A
Developer impact
----------------
Developers who write another PollingDataSource Driver
Implementation
==============
Assignee(s)
-----------
Primary assignee:
muroi-masahito
Other contributors:
muroi-masahito
Work items
----------
0. Disable polling data
When no subscrier exists for lazy tables, PollingDataSourceDriver
stops pulling data. And the driver starts pulling data after some
other service start to subscribe the table.
1. "lazy_table" config
Subclass of PollingDataSourceDriver will have "lazy_table" key in
its config parameter. This key is optional config.
Dependencies
============
N/A
Testing
=======
* Add unit tests related to the functionality
Documentation impact
====================
API reference will be updated based on the API changes.
References
==========
N/A

354
specs/newton/template.rst
View File

@ -1,354 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Example Spec - The title of your blueprint
==========================================
Include the URL of your launchpad blueprint:
https://blueprints.launchpad.net/congress/+spec/example
Introduction paragraph -- why are we doing anything? A single paragraph of
prose that operators can understand.
Some notes about using this template:
* Your spec should be in ReSTructured text, like this template.
* Please wrap text at 79 columns.
* The filename in the git repository should match the launchpad URL, for
example a URL of: https://blueprints.launchpad.net/congress/+spec/awesome-thing
should be named awesome-thing.rst
* Please do not delete any of the sections in this template. If you have
nothing to say for a whole section, just write: None
* For help with syntax, see http://sphinx-doc.org/rest.html
* To test out your formatting, build the docs using tox, or see:
http://rst.ninjs.org
* If you would like to provide a diagram with your spec, ascii diagrams are
required. http://asciiflow.com/ is a very nice tool to assist with making
ascii diagrams. The reason for this is that the tool used to review specs is
based purely on plain text. Plain text will allow review to proceed without
having to look at additional files which can not be viewed in gerrit. It
will also allow inline feedback on the diagram itself.
Problem description
===================
A detailed description of the problem:
* For a new feature this might be use cases. Ensure you are clear about the
actors in each use case: End User vs Deployer
* For a major reworking of something existing it would describe the
problems in that feature that are being addressed.
Proposed change
===============
Here is where you cover the change you propose to make in detail. How do you
propose to solve this problem?
If this is one part of a larger effort make it clear where this piece ends. In
other words, what's the scope of this effort?
Alternatives
------------
What other ways could we do this thing? Why aren't we using those? This doesn't
have to be a full literature review, but it should demonstrate that thought has
been put into why the proposed solution is an appropriate one.
Policy
------
Using the Congress datalog syntax, write out an example policy using
https://wiki.openstack.org/wiki/Congress#Policy_Language
Example:
error(vm) :-
nova:virtual_machine(vm),
ids:ip_packet(src_ip, dst_ip),
neutron:port(vm, src_ip), //finds out the port that has the VM's IP
ids:ip_blacklist(dst_ip).
Policy actions
--------------
Describe the policy activities in terms of monitoring, reactive, proactive,
and other ways to explain how the policy will implement it's desired state.
Data sources
------------
Describe which projects and/or services the data is coming from
Data model impact
-----------------
Changes which require modifications to the data model often have a wider impact
on the system. The community often has strong opinions on how the data model
should be evolved, from both a functional and performance perspective. It is
therefore important to capture and gain agreement as early as possible on any
proposed changes to the data model.
Questions which need to be addressed by this section include:
* What new data objects and/or database schema changes is this going to
require?
* What database migrations will accompany this change.
* How will the initial set of new data objects be generated, for example if you
need to take into account existing instances, or modify other existing data
describe how that will work.
REST API impact
---------------
Each API method which is either added or changed should have the following
* Specification for the method
* A description of what the method does suitable for use in
user documentation
* Method type (POST/PUT/GET/DELETE)
* Normal http response code(s)
* Expected error http response code(s)
* A description for each possible error code should be included
describing semantic errors which can cause it such as
inconsistent parameters supplied to the method, or when an
instance is not in an appropriate state for the request to
succeed. Errors caused by syntactic problems covered by the JSON
schema definition do not need to be included.
* URL for the resource
* Parameters which can be passed via the url
* JSON schema definition for the body data if allowed
* JSON schema definition for the response data if any
* Example use case including typical API samples for both data supplied
by the caller and the response
* Discuss any policy changes, and discuss what things a deployer needs to
think about when defining their policy.
Example JSON schema definitions can be found in the Nova tree
https://opendev.org/openstack/nova/src/branch/master/nova/api/openstack/compute/schemas
Note that the schema should be defined as restrictively as
possible. Parameters which are required should be marked as such and
only under exceptional circumstances should additional parameters
which are not defined in the schema be permitted (eg
additionaProperties should be False).
Reuse of existing predefined parameter types such as regexps for
passwords and user defined names is highly encouraged.
Security impact
---------------
Describe any potential security impact on the system. Some of the items to
consider include:
* Does this change touch sensitive data such as tokens, keys, or user data?
* Does this change alter the API in a way that may impact security, such as
a new way to access sensitive information or a new way to login?
* Does this change involve cryptography or hashing?
* Does this change require the use of sudo or any elevated privileges?
* Does this change involve using or parsing user-provided data? This could
be directly at the API level or indirectly such as changes to a cache layer.
* Can this change enable a resource exhaustion attack, such as allowing a
single API interaction to consume significant server resources? Some examples
of this include launching subprocesses for each connection, or entity
expansion attacks in XML.
For more detailed guidance, please see the OpenStack Security Guidelines as
a reference (https://wiki.openstack.org/wiki/Security/Guidelines). These
guidelines are a work in progress and are designed to help you identify
security best practices. For further information, feel free to reach out
to the OpenStack Security Group at openstack-security@lists.openstack.org.
Notifications impact
--------------------
Please specify any changes to notifications. Be that an extra notification,
changes to an existing notification, or removing a notification.
Other end user impact
---------------------
Aside from the API, are there other ways a user will interact with this
feature?
* Does this change have an impact on python-congressclient? What does the user
interface there look like?
Performance impact
------------------
Describe any potential performance impact on the system, for example
how often will new code be called, and is there a major change to the calling
pattern of existing code.
Examples of things to consider here include:
* A periodic task might look like a small addition but if it calls conductor or
another service the load is multiplied by the number of nodes in the system.
* Scheduler filters get called once per host for every instance being created,
so any latency they introduce is linear with the size of the system.
* A small change in a utility function or a commonly used decorator can have a
large impacts on performance.
* Calls which result in a database queries (whether direct or via conductor)
can have a profound impact on performance when called in critical sections of
the code.
* Will the change include any locking, and if so what considerations are there
on holding the lock?
Other deployer impact
---------------------
Discuss things that will affect how you deploy and configure OpenStack
that have not already been mentioned, such as:
* What config options are being added? Should they be more generic than
proposed (for example a flag that other hypervisor drivers might want to
implement as well)? Are the default values ones which will work well in
real deployments?
* Is this a change that takes immediate effect after its merged, or is it
something that has to be explicitly enabled?
* If this change is a new binary, how would it be deployed?
* Please state anything that those doing continuous deployment, or those
upgrading from the previous release, need to be aware of. Also describe
any plans to deprecate configuration values or features. For example, if we
change the directory name that instances are stored in, how do we handle
instance directories created before the change landed? Do we move them? Do
we have a special case in the code? Do we assume that the operator will
recreate all the instances in their cloud?
Developer impact
----------------
Discuss things that will affect other developers working on OpenStack,
such as:
* If the blueprint proposes a change to the driver API, discussion of how
other hypervisors would implement the feature is required.
Implementation
==============
Assignee(s)
-----------
Who is leading the writing of the code? Or is this a blueprint where you're
throwing it out there to see who picks it up?
If more than one person is working on the implementation, please designate the
primary author and contact.
Primary assignee:
<launchpad-id or None>
Other contributors:
<launchpad-id or None>
Work items
----------
Work items or tasks -- break the feature up into the things that need to be
done to implement it. Those parts might end up being done by different people,
but we're mostly trying to understand the timeline for implementation.
Dependencies
============
* Include specific references to specs and/or blueprints in congress, or in
other projects, that this one either depends on or is related to.
* If this requires functionality of another project that is not currently used
by congress (such as the glance v2 API when we previously only required v1),
document that fact.
* Does this feature require any new library dependencies or code otherwise not
included in OpenStack? Or does it depend on a specific version of library?
Testing
=======
Please discuss how the change will be tested. We especially want to know what
tempest tests will be added. It is assumed that unit test coverage will be
added so that doesn't need to be mentioned explicitly, but discussion of why
you think unit tests are sufficient and we don't need to add more tempest
tests would need to be included.
Is this untestable in gate given current limitations (specific hardware /
software configurations available)? If so, are there mitigation plans (3rd
party testing, gate enhancements, etc).
Documentation impact
====================
What is the impact on the docs team of this change? Some changes might require
donating resources to the docs team to have the documentation updated. Don't
repeat details discussed above, but please reference them here.
References
==========
Please add any useful references here. You are not required to have any
reference. Moreover, this specification should still make sense when your
references are unavailable. Examples of what you could include are:
* Links to mailing list or IRC discussions
* Links to notes from a summit session
* Links to relevant research, if appropriate
* Related specifications as appropriate (e.g. if it's an EC2 thing, link the
EC2 docs)
* Anything else you feel it is worthwhile to refer to

628
specs/pike/policy-library.rst
View File

@ -1,628 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Policy Library
==========================================
Include the URL of your launchpad blueprint:
https://blueprints.launchpad.net/congress/+spec/policy-library
When Congress is first deployed, it does not perform any monitoring or
enforcement because there are no policies specified. It takes significant
investment in time and energy to learn the policy language and write the
desired policies before Congress provides any value.
Problem description
===================
To help deployers and administrators quickly get started and see value,
we propose to include with Congress a library of useful policies that a
deployer or administrator can easily enable, disable, and customize.
Proposed change
===============
Overview
------------
The proposed changes come in two main pieces, the new policy library and the
changes to existing Congress server functionalities to support the use of
policies from the policy library.
The policy library
^^^^^^^^^^^^^^^^^^^^^
Introduce a new concept of the policy library in Congress server,
where policies are retained and manipulated but are not used by the policy
engine (for monitoring or enforcement).
Include with Congress server a collection of useful policies,
each in a YAML file. These policies automatically populate the policies
library.
To make it easy for us to populate the library and for admins to modify/create
their own library, the API/CLI/GUI should support full CRUD into the library.
* Congress server can have new config option that points to a directory of
policies to load into the library.
* Once the library is initialized from the files, restart of Congress server
do not re-initialize the policy library from the files
(empty library is assumed to be un-initialized). A new interface
allows an administrator to trigger re-initialization from files.
.. note:: The policy library is persisted in the database and has a
separate existence in Congress server apart from the files in the policy
library directory. See also discussion of the alternative model
_`Policy library directory files as truth`.
* Future feature: reloads changed files automatically into library
* Will not update policies imported from library into policy engine
* Open question:
Should changes to the policy library through the API sync back to the
files in the policy library directory?
* Care must be taken to consider concurrent changes and potential conflicts
from multiple congress server instances.
.. warning:: When Congress server policy engine service is replicated,
different server instances must have the same
policy library files. If not, the library will be populated based on the
an arbitrary combination of the policy files on the different server
instances.
* Installation
* Standalone: installer scripts copy the default library directory from the
Congress git repo into /etc/congress/.
* Devstack: same as standalone
Support for using policies from the library
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
We propose to add all the functionalities required to support the following
workflows for activating a policy from the policy library (i.e., creating an
(active) policy in the policy engine by importing a (inactive) policy from the
policy library).
* Activating a policy from library using CLI/API.
#. Administrator inspects policies in library to find one they want to use.
#. Administrator downloads the relevant policy, in YAML or JSON.
#. Administrator modifies YAML/JSON encoding and uses it as the payload to
create a new policy.
* Activating a policy from library using GUI.
#. Administrator peruses policies in library
#. Administrator clicks on an interesting policy, which brings up a textbox
(or more sophisticated policy editor) to customize.
#. Administrator clicks on a 'Create' button that creates that policy.
Below are the specific functionalities needed to support these workflows.
* A backward compatible change to the policies POST API to allow for a 'rules'
key in the JSON/YAML payload body.
* A new optional parameter to the ``policy create`` CLI command that allows
the administrator to import and activate a policy from a YAML file.
* A new optional parameter to the ``policy create`` CLI command that allows
the administrator to import and activate a policy directly from the policy
library without saving to file.
.. note:: The policy engine rule insert code would require some careful
reworking to ensure policy creation with rules is transactional. That is,
if insertion fails at the third rule (say because of cross-policy
recursion), the first and second rule insertions are undone without having
caused any effect (triggered actions or altered query answers).
Three main things are needed.
* Write to database only after the insertion of all the policy rules have
succeeded in the policy engine.
* Trigger evaluation must be delayed until after all the rules are inserted
successfully.
* The policy engine must be prevented from answering queries while in the
middle of processing a batch of rule insertions. This can be done either
with locks or by making sure the greenthread executing the batch rule
insertion does not yield.
* A new page in the Horizon GUI where an administrator can perform the
following tasks.
- browse and inspect policies in the policy library
- active (i.e., import to policy engine) a policy from the policy library
- customize a policy in the policy library prior to import
Alternatives
------------
This section discusses several alternative designs and implementation to
various aspects of this proposal.
Policy library strictly external to Congress
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* No concept of library in API/data model. The policy library is simply a
directory of files the administrator can import into the policy engine.
* Least development effort.
* Not as usable. Administrator cannot use the existing Congress interface
(CLI/GUI/API) to browse, inspect, customize, and activate policies in the
policy library.
* This version is realized by a subset of the changes proposed in the spec.
This subset should be prioritized in the development to realize the minimal
viable functionality described here.
Policy library directory files as truth
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* Instead of database, use the policy files in the policy library directory as
source of truth.
* Complex in error-prone in a setup with multiple policy engines.
* Different model from all the other persisted data in Congress server,
adding more complexity for developers.
Library rules as disabled policy engine rules in the same database table
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
* Instead of storing the rules in the library policy in a separate logical
repository as well as a separate database table from the active rules in the
rule engine, store all the rules as rules in the rule engine, with an
additional flag indicating which ones are active.
* Complicates the existing data model for rules in the policy engine.
* What is the benefit?
Policy
------
Not applicable.
Policy actions
--------------
Not applicable
Data sources
------------
Not applicable.
Data model impact
-----------------
No impact on existing tables.
One new database tables are needed:
* A ``library_policies`` table to store the policies in the policies library.
Below are the columns.
* ``name =
sa.Column(sa.String(255), nullable=False, unique=True, primary_key=True)``
* ``abbreviation = sa.Column(sa.String(5), nullable=False)``
* ``description = sa.Column(sa.Text(), nullable=False)``
* ``kind = sa.Column(sa.Text(), nullable=False)``
* ``rules = sa.Column(sa.Text(), nullable=False)``
.. note:: Because the rules in the library are accessed at the granularity of
one entire policy, we can improve simplicity and performance by storing the
rules as a text blob instead of as individual rows.
.. warning:: the TEXT type is limited to 2^16 - 1 bytes in length in MySQL.
Very large policies beyond this size are not supported. If this limit turns
out to be too restrictive, MEDIUMTEXT or LONGTEXT type can be forced by
specifying a longer length in the SQL Alchemy column declaration, at the
cost of losing SQLite support (primarily used for unit testing).
Empty tables will be created using sqlalchemy migration scripts. Tables will be
populated by Congress server from policy files in the policy libraries folder.
REST API impact
---------------
Policy library methods
^^^^^^^^^^^^^^^^^^^^^^^^^^^
* Re-initialize
* Re-initialize policy library from the files in the policy library directory
as set in configuration.
.. warning:: This operation destroys the current content of the policy
library.
* Method type: PUT
* Normal http response code(s): 200
* Expected error http response code(s): 401
* ``/v1/library``
* Parameters which can be passed via the url: None
* JSON schema definition for the body data if allowed: Not allowed
* JSON schema definition for the response data if any: None
* List policies
* List metadata for all policies in the library (id, name, desc, number of
rules)
* Method type: GET
* Normal http response code(s): 200
* Expected error http response code(s): None
* ``/v1/library``
* Parameters which can be passed via the url: None
* JSON schema definition for the body data if allowed: Not allowed
* JSON schema definition for the response data if any:
::
- !Type:
title: collection of PolicyProperties
type: array
items:
type: object
properties:
name:
title: Policy unique name
type: string
required: true
description:
title: Policy description
type: string
required: true
kind:
title: Policy kind
type: string
required: true
abbreviation:
title: Policy name abbreviation
type: string
required: false
* Show policy
* Show specified library policy (in specified format)
* Method type: GET
* Normal http response code(s): 200
* Expected error http response code(s): 401, 404
* ``/v1/library/<policy-name>``
* Parameters which can be passed via the url: None
* JSON schema definition for the body data if allowed: Not allowed
* JSON schema definition for the response data if any:
`Policy full schema`_.
* Create policy
* Create a new policy in library
* Method type: POST
* Normal http response code(s): 200
* Expected error http response code(s): 400, 401, 409
* ``/v1/library``
* Parameters which can be passed via the url: None
* JSON schema definition for the body data if allowed:
`Policy full schema`_.
* JSON schema definition for the response data if any:
`Policy metadata schema without UUID`_.
* Update policy
* Update a policy in library
* Method type: PUT
* Normal http response code(s): 200
* Expected error http response code(s): 400, 401, 404, 409
* ``/v1/library/<policy-name>``
* Parameters which can be passed via the url:
* format: {json, yaml} (default is json)
* JSON schema definition for the body data if allowed:
`Policy full schema`_.
* JSON schema definition for the response data if any:
`Policy metadata schema without UUID`_.
* Delete policy
* Delete policy from library
* Method type: DELETE
* Normal http response code(s): 200
* Expected error http response code(s): 401, 404
* ``/v1/library/<policy-name>``
* Parameters which can be passed via the url: None
* JSON schema definition for the body data if allowed: Not allowed
* JSON schema definition for the response data if any: None
Policy engine methods
^^^^^^^^^^^^^^^^^^^^^^^^
* Create policy
* Create a policy in policy engine (either from input data or from library)
* Method type: POST
* Normal http response code(s): 200
* Expected error http response code(s): 400, 401, 404, 409
* ``/v1/policies``
* Parameters which can be passed via the url:
* library_policy=<name of policy in library>. This creates the policy
directly from the library policy, without needing body data. A request
where both this parameter and a body data are specified should result in
error 400.
* JSON schema definition for the body data if allowed: `Policy full schema`_.
* JSON schema definition for the response data if any:
`Policy metadata schema with UUID`_.
Common JSON Schemas
^^^^^^^^^^^^^^^^^^^^^^^^^
Policy metadata schema without UUID
++++++++++++++++++++++++++++++++++++++++++++
::
- !Type:
id: PolicyProperties
title: Policy Properties
type: object
properties:
name:
title: Policy unique name
type: string
required: true
description:
title: Policy description
type: string
required: true
kind:
title: Policy kind
type: string
required: true
abbreviation:
title: Policy name abbreviation
type: string
required: false
.. note:: UUIDs are not used for policies in the policy library. The reason
is gets confusing if the UUID is changed when the policy is activated,
but also problematic if the UUID is retained in the activated policy in
policy engine, leading to two different entities sharing the same UUID.
Policy metadata schema with UUID
++++++++++++++++++++++++++++++++++++++++++++
::
- !Type:
id: PolicyProperties
title: Policy Properties
type: object
properties:
id:
title: UUID for the policy
type: string
required: true
name:
title: Policy unique name
type: string
required: true
description:
title: Policy description
type: string
required: false
kind:
title: Policy kind
type: string
required: false
abbreviation:
title: Policy name abbreviation
type: string
required: false
Policy full schema
++++++++++++++++++++++++++++++++++++++++++
.. note:: It is a design decision that rules in a policy library do not
have IDs and cannot be referred to individually. Policies in the library
are created, updated, or deleted at the level of the whole policy.
::
- !Type:
id: PolicyProperties
title: Policy Properties
type: object
properties:
name:
title: Policy unique name
type: string
required: true
description:
title: Policy description
type: string
required: true
kind:
title: Policy kind
type: string
required: true
abbreviation:
title: Policy name abbreviation
type: string
required: false
rules:
title: collection of rules
type: array
required: true
items:
type: object
properties:
id: PolicyRule
title: Policy rule
type: object
properties:
rule:
title: Rule definition following policy grammar
type: string
required: true
name:
title: User-friendly name
type: string
required: false
comment:
title: User-friendly comment
type: string
required: false
Security impact
---------------
In general, no major security impact is expected. However, below are some
security considerations.
* If policy library changes are sync'ed back to the policy library directory on
the server, then an attacker who gains administrative privilege to Congress
can also create and modify files in the policy library directory of the
Congress servers.
* The ability to submit multiple rules in a policy creation API can increase
the potential for resource exhaustion attacks.
Notifications impact
--------------------
No impact
Other end user impact
---------------------
No impact to existing work flows.
Performance impact
------------------
Minimal performance impact expected. Care must be taken to limit the
performance impact of the feature to automatically detect changes in
the policy library folder and update the policy library accordingly.
Other deployer impact
---------------------
There is minimal negative impact on deployers.
* No existing database table schemas are changed.
* New database tables are used, which would be added by alembic migration
scripts.
Developer impact
----------------
Minimal developer impact.
Implementation
==============
Assignee(s)
-----------
Assignment to be done on launchpad.
Work items
----------
To be organized on launchpad.
Dependencies
============
No new dependencies.
Testing
=======
In addition to unit tests, straight-forward additions to the tests in
congress_tempest_tests/tests/scenarios/test_congress_basic_ops.py would
suffice.
Documentation impact
====================
Minimal impact.
References
==========
No references.

365
specs/queens/configuration-files-validation.rst
View File

@ -1,365 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
========================================
OpenStack Configuration Files Validation
========================================
https://blueprints.launchpad.net/congress/+spec/configuration-files-validation
Congress could be used by cloud operators to formalize the constraints between
options defined in configuration files with the help of business rules written
in Datalog and verify the compliance of their deployments automatically.
It is intended to be complementary to config management systems
that should only create valid configuration files
and to integration tests such as RefStack.
Problem description
===================
Each OpenStack service on each node in parameterized by a set of configuration
files describing the value of the various configuration options. In this
proposal we only consider configuration files managed by the oslo-config
library. Configuration options are not independent:
* Each option value is constrained but the constraints may go beyond the
expressive power of oslo-config type system.
* For a given service on a given node, constraints may exist between different
options.
* On a given node, constraints and incompatibilities may exist between various
options belonging to different services. For example, when a functionality in
a given service depends on the availability of another functionality in
another service.
* Between nodes, other constraints exist either for a given service or between
services.
Constraints may be defined by different actors:
* Service developers know the constraints on the options they define and
usually document them informally in the source code in the description field
associated to the option.
* Cloud integrators will discover additional constraints when they develop
deployment code. Most of the time those options are only implicitly defined
in the source code of the scripts.
* Administrators may want to enforce additional constraints reflecting the
operational environment constraints.
* For companies having multiple instances of OpenStack, rules could be
used to ensure compliance of the instances with their global engineering
rules.
* Although OpenStack provides abstractions that hide implementation details
from end-users, some back-ends or some back-end combinations may impose
constraints on what is available to the end-user. When administrators
discover some back-end limitations, it may be useful to add congress rules
reflecting those limitations before a more permanent solution is found.
Datalog could be used as a formal language to describe those constraints and
Congress could be used to enforce those constraints if the configuration files
were available for processing. Although Congress is usually used for more
dynamic data coming from running instances, nothing precludes the use of more
static data sources.
Proposed change
===============
The extension adds a new agent deployed on OpenStack nodes that collect the
configuration files as defined in its own configuration. The agent communicates
with a datasource driver included in the congress analyzer through the
message bus. The agent is configured with its own configuration file describing
the files it transmit (an example configuration file is given in the 'Other
deployer impact' section).
The agent is responsible for conveying information on option values but also on
their meta-data: layout in groups, type, flags such as deprecation, secret
status, etc. Meta-data are described in templates which are in fact
a collection of namespaces. Namespace contain the actual definition of
meta-data. To avoid versionning problems, Meta-data must be obtained directly
from the service.
To limit the amount of traffic between the driver and agents, template files
are hashed. Agents first reply to queries with hashes. The driver will request
the content only if the hash is unknown.
The same process is used for namespaces.
The only processing performed by agents on the option files is the removal of
the values of secret options.
The datasource driver is responsible for populating the extensional database
with information provided by the agents:
* option definitions are extracted from templates and namespaces and translated
in new Datalog tables.
* option values are parsed by the oslo-config library and stored with an
identification of the origin (file and node).
The Datalog model defines several tables:
* hosts
* files
* templates
* namespaces
* option values
The definition of an option is split between three tables:
* The option table defines the mapping between the option id and the fields
that build up the key: name, namespace and group of the option.
* The option_info table defines the type and the flags of the option. It uses
the id defined in the option table to uniquely identify the option.
* Depending on the type of the option, attributes specific to a given type are
defined in a specialized table. The option id is used again as a key to refer
to the option table.
Regarding the uniqueness of configuration meta-data in the extensional
database, the driver must ensure that the ids are deterministic. An option
identified by the same name, same group name and same namespace name should
always be given the same unique id. The use of the MD5 hash function (there is
no cryptographic requirement) guarantees uniqueness and determinism.
Alternatives
------------
Installers usually generate correct set of configuration files for a
more or less extensible set of options but provide little help for live
maintenance of those files. They may also restrict the configuration
space beyond what is necessary.
The proposed change must be considered as a work in progress. It does not fully
address the problem of the location and the management of constraints. Most
constraints are known by the service developers and should be maintained in the
source code of services in a dedicated meta-data field of
the option definition.
The use of an external agent to push a service configuration is not the only
solution. The oslo-config library could be modified to push the configuration
read by the service to the datasource driver. This could be done through
the use of a hook in oslo-config. It would require additional configuration of
the services to identify the endpoint.
Policy
------
We would like to aggregate policy violations in a few tables partitioning
violations by their degree (error, warn, info).
Example:
X: neutron-server host
Y: ovs-agt host
Z: lb-agt host
::
warn('Multinode OVS and linuxbridge use incompatible UDP ports',
'vxlan_conflicting_ovs_lb_udp_ports') :-
vxlan_conflicting_ovs_lb_udp_ports()
vxlan_conflicting_ovs_lb_udp_ports(Y, Z) :-
value(X, 'neutron.conf', 'DEFAULT', 'core_plugin', 'ml2'),
value(X, 'ml2_conf.ini', 'ml2', 'type_drivers', 'vxlan'),
value(X, 'ml2_conf.ini', 'ml2', 'mechanism_drivers', 'openvswitch'),
value(X, 'ml2_conf.ini', 'ml2', 'mechanism_drivers', 'linuxbridge'),
value(Y, 'openvswitch_agent.ini', 'agent', 'tunnel_types', 'vxlan'),
not value(Y, 'openvswitch_agent.ini', 'agent', 'vxlan_udp_port', 8472),
value(Z, 'linuxbridge_agent.ini', 'vxlan', 'enable_vxlan', 'True')
We'd like the 'value' table to be intelligible to most potential contributor,
and also meaningful enough to spare them from predictable joins.
Currently the 'value' table is derived from the extensional tables, which
describe configuration files and their meta-data. Although it is defined
intentionally, it could be useful to consider it as an extensional predicate.
Here is how we derived it:
::
value(hostname, file, group, name, value) :-
config:option(id=option_id, group=group, name=name),
config:file(id=file_id, host_id=host_id, name=file),
config:host(id=host_id, name=hostname),
config:value(option_id=option_id, file_id=file_id, val=value)
Policy actions
--------------
None.
Data sources
------------
The data sources are the different configuration files of OpenStack projects
using the Oslo.config library for their configuration.
Data model impact
-----------------
None
REST API impact
---------------
None
Security impact
---------------
Configuration files contain sensitive credentials. Those credentials MUST NOT
be transmitted to the Congress engine. The agent has access to types and must
filter out credentials. Values of any option marked as secret will not be
available within the engine.
Notifications impact
--------------------
None
Other end user impact
---------------------
None other than the usual management of the datasource and policy.
Eventually, we would like to feed the engine with rules that are coming from
and maintained in the services source code.
Performance impact
------------------
Performance impact should be limited because of the static nature of the values
provided by this datasource.
The main impact is the traffic on the message bus used to exchange
configuration files between agents and Congress server. The server may
rate-limit this traffic. If performance is still a concern, another solution is
to limit the use of the bus to announcement and use a REST endpoint on the
server to record new configuration files.
We give the driver control over the way data is retrieved. We want to prevent
the duplicated sending of files and templates, and to prevent overloading the
driver. When the driver is activated, it periodically notifies agents, over the
communication bus requesting their data description. An agent send description
of the files it has been set to provide. The description contains hashes of
namespaces, templates and configs. The driver then requests the resources,
which hashes have not been recognized.
We use the RPC server of the datasource associated DseNode.
Other deployer impact
---------------------
We add a dedicated group and options to configure an agent and the
configuration files to manage.
*validator.host*
A string option serving as a node id, in a way that is meaningful to
administrators.
*validator.version*
A string option introducing the notion of version and what it would be on this
node. It could be used to discriminate handling of different version of a
config-file coexisting in a cloud instance during a migration.
This information may be provided differently in the future to be defined
at the services level.
*validator.services*
An dict option describing the OpenStack services activated on this node. The
values are also dictionaries. Keys are paths to config-files,
values are paths to the associated templates. For instance::
congress: { /etc/congress/congress.conf:
/opt/stack/congress/etc/congress-config-generator.conf }
Example config :
::
[DEFAULT]
transport_url = rabbit://..@control:5672/
[validator]
services = nova : { /etc/nova.conf:
/opt/stack/nova/etc/nova/nova-config-generator.conf:
version = ocata
host = node_A
Developer impact
----------------
Discuss things that will affect other developers working on OpenStack,
such as:
* If the blueprint proposes a change to the driver API, discussion of how
other hypervisors would implement the feature is required.
Implementation
==============
Assignee(s)
-----------
Primary assignees:
* Valentin Matton
* Pierre Crégut
Work items
----------
* Agent to collect the config files
* Datasource interacting with the agents
* Integration to devstack
Dependencies
============
We will depend on oslo-config-generator config-files, which can be used to
describe OS services config-files.
https://github.com/openstack/oslo-specs/blob/master/specs/juno/oslo-config-generator.rst
We rely extensively on the oslo-config lib.
Testing
=======
We propose to use tempest tests in a setting with 2 nodes. One of which hosts
the congress-server and the second an agent. Communications will be tested :
sending of meta-data and files.
Documentation impact
====================
This feature introduces an agent component that requires separate
configuration. It also defines new datasources.
References
==========

314
specs/rocky/alternative-engine-z3.rst
View File

@ -1,314 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
===================================
Z3 as an alternative Datalog engine
===================================
https://blueprints.launchpad.net/congress/+spec/alternative-engine-z3
Z3 is an open source Datalog engine developed by Microsoft.
We propose an extension of Congress that lets the user define
theories (ie. set of rules producing new tables from existing ones
that can be later used by other theories) that will be treated
by the Z3 engine instead of the existing one.
Problem description
===================
A cloud operator may want to check if connectivity exists between two machines
or if they are effectively isolated. This means understanding how traffic is
routed (ie. what are the path the traffic may take) and how security groups and
firewall rules are applied along those path.
Here are the main characteristics needed by the Datalog engine for
solving such problems:
* Recursion: routing packets means building paths (link paths, networks)
and paths are naturally described recursively.
* Efficient representation of IP addresses and IP address ranges as commonly
found in switching devices and firewalls.
* Efficient handling of large volumes of data.
Proposed change
===============
We propose to integrate Z3, Microsoft open-source automatic prover, as an
alternate Datalog engine for Congress.
Z3 theories will be expressed in the same syntax as standard theories.
But the Datalog language supported by these theories will be tailored to what
Z3 supports in Datalog mode:
* different and reduced set of built-ins as explained in the Policy section,
* no modals.
Architecture
------------
Overview
^^^^^^^^
A new kind of theory will be introduced alongside the classical
NonRecursiveTheory. Z3 theories will consume data from
other theories and use the Z3 engine to apply rules.
There will be one single Z3 solver instance per analyzer that will be
abstracted in Python as a Z3 context object. Z3 context will host all tables
of Z3 theories in Z3 format.
Z3 theories should be callable from other theories in particular from
regular non-recursive theories.
Z3 theories can use data from other theories. The external tables used by
Z3 theories will be completely imported in the Z3 context.
Compilation of Z3 theories will be delayed until queries are done on those
theories. We will regularly update the representation of external tables in
the Z3 engine to keep them synchronized. Z3 does not support undoing relations
or rules, so a new context must be created each time a change occurs.
Type-checking
^^^^^^^^^^^^^
Z3 requires typed data. Type-checking will be completely implicit and we will
not add type-constraints in the language.
The type-checker will preserve the types from external tables. It will raise
an error if a variable should have two different types according to two
different tables.
The type-checker will try to give the most precise type to constants that
appear in Datalog programs to make them compatible with external tables.
For example the value ``Ingress`` is usually typed as a string (``Str`` type)
but in the context of a Neutron security group, it will be given the type
``Direction``
To be able to have two representations for a value, we will introduce
explicit converter built-ins: ``builtin:cast(x, 'Direction', y, 'Str')``
will convert a variable ``x`` of type ``Direction`` in a variable ``y``
of the standard string type. It will be implemented as a table where each
row is a pair of the representations in the two different types of a same
object. Those tables will be filled when constants in programs are compiled
and external tables are imported.
The same trick will be used to have a bit field representation of Ip addresses.
Finally the type-checker should support some kind of generic polymorphism for
built-ins. For example equalities and comparison should work as long as the
compared objects are of the same type, addition of two items belonging to a
given subtype of integers should give back an item in the same subtype. We do
not plan to support user-defined polymorphic tables. For example the following
rule is ill defined: ::
add3(x,y,z,t) :- builtin:plus(x,y,u), builtin:plus(u,z,t)
It may be desirable to have type-constraints in the language to constrain
such variables.
Internal representations
^^^^^^^^^^^^^^^^^^^^^^^^
Internally Z3 will use bit-vectors. Integers will be translated in 32 bits
bit-vectors. Strings will be bijectively associated with an integer and
represented by its associated bit-vector.
When possible we will use finer types (enumerations) that require smaller
bit-vector.
Alternatives
------------
We could try to rephrase the theories to fit the current engine. For example we
could unroll recursive calls up to a reasonable depth (packets have a TTL).
Unfortunately this solution does not scale well. Unrolling recursive calls
change in fact the overall complexity of the program. Experiments with octant
have shown that some problems that require several minutes of computation with
the current engine can be solved in tenth of seconds with octant.
We could try to incorporate Z3 algorithmic in Congress engine: this would be
time consuming and hazardous. Z3 uses very efficient C++ data structures to
represent tables and more that ten years have been devoted to its development.
We could use another external engine: there may be cases where another Datalog
engine could be a better candidate. The new Congress architecture with explicit
data-type has been designed to let several engines share a same pool of data.
The strong points of Z3 are its overall efficiency and its data-structures
specially crafted for network related problems.
There may be cases where we may wish to have several Z3 contexts with different
settings. Contexts define the variant of the Z3 engine in use and different
theories may benefit from different settings.
Policy
------
Here are the main characteristics of Z3 policies:
* Adds support for recursive rules.
* Support of built-in predicates for equalities / comparison
* Support for basic bit arithmetic (to be used with IP address).
* New built-ins for converting opaque IpAddress to bit fields and IpNetwork
to a pair of bit-fields (mask and address).
* Built-ins to convert a value that exists in different types from one
representation to the other.
The last two kinds of built-ins will in fact be implemented by tables computed
when data-sources and programs are synchronized.
Here is an example of policy that verifies if two networks are connected by
a chain of routers and if VMs are attached to two networks that belong to two
sets of networks that should be kept isolated: ::
connect1(X) :- neutronv2:networks(id=X, name="N1")
connect2(X) :- neutronv2:networks(id=X, name="N2")
linked(X,Y) :-
neutronv2:ports(network_id=X, device_id=Z), neutronv2:routers(id=Z),
neutronv2:ports(network_id=Y, device_id=Z)
path(X,Y) :- linked(X,Y)
path(X,Y) :- path(X,Z), linked(Z,Y)
interco_error(X,Y) :- connect1(X), connect2(Y), path(X,Y)
network1(X) :- connect1(Y), path(Y, X)
network2(X) :- connect2(Y), path(Y, X)
double_attach(X) :-
nova:server(id=X),
neutronv2:ports(network_id=X, device_id=Y), network1(Y),
neutronv2:ports(network_id=X, device_id=Z), network2(Z)
Policy actions
--------------
None. Z3 theories will not support actions. Support may be added later.
Data sources
------------
Independent from the data-source used.
Data model impact
-----------------
None
REST API impact
---------------
We may want to move to more atomic definitions of theories. A theory is
a complete program and should be considered as such. Adding and removing
rules on the fly makes compilation and type-checking unnecessarily complex.
Security impact
---------------
None
Notifications impact
--------------------
None
Other end user impact
---------------------
For python-congressclient, when a new theory is created, there is a new
kind alternative: z3.
Performance impact
------------------
Well behaved Z3 theories will be much faster than equivalent non-recursive
theories. But programs that require too much computing power may be harder to
abort on the Z3 engine.
Other deployer impact
---------------------
Z3 is not a pypi package. There is an unofficial z3solver package but it has
some limitations (not an up-to-date version and a version emitting a lot of
spurious messages). For some Linux distributions, there is no pre-compiled
version of Z3 available. Z3 must be compiled from source on those systems.
Due to all these constraints, we have chosen NOT to add z3 as a requirement.
z3theory will be available IF z3 has been deployed on the server hosting
congress.
Developer impact
----------------
None
Implementation
==============
Assignee(s)
-----------
Primary assignee:
pcregut
Other contributors:
Work items
----------
For Rocky we do not plan to add built-ins. The basic Datalog language is
sufficient for interesting examples.
During Stein iteration, built-ins and the treatment of polymorphism
in the type-checker will be added.
Dependencies
============
This feature depends on Microsoft theorem prover Z3.
Testing
=======
Unit tests will be done with a mock of z3 library.
For integration tests, new settings will be added to devstack so that Z3
can be either installed from source (rather slow) or directly deployed from
a pre-compiled release (only available for Ubuntu 14.04, 16.04 and Debian
8.10). As the python package is deployed globally, tempest may skip Z3 tests
if it cannot import the python package.
Documentation impact
====================
Documentation should at least mention:
* How to enable Z3,
* Z3 theories and the kind of builtins supported,
* Explain the benefit of recusive theories.
References
==========
[1] https://github.com/Z3Prover/z3
[2] https://github.com/Orange-OpenSource/octant
[3] https://specs.openstack.org/openstack/congress-specs/specs/rocky/explicit-data-types

678
specs/rocky/explicit-data-types.rst
View File

@ -1,678 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
===================
Explicit Data Types
===================
https://blueprints.launchpad.net/congress/+spec/explicit-data-types
In preparation to support more policy engines in addition to the standard
Congress policy engine (e.g., ones based on SQL, influxDB, or Z3), we need
explicit typing of table columns in the data sources and the
data service engine. This spec proposes to support explicit typing, including
custom types, in an extensible, evolutionary, and backwards-compatible manner.
Problem description
===================
Because every rule engine has limits in expressiveness and performance, there
is no one-size-fits-all rule engine that works well for all use cases. For
this reason, Congress was architected from the beginning to support multiple
policy engines in addition to the standard ``agnostic`` engine.
One problem is that Congress deals with untyped table columns, which
conflicts with many candidate bases for policy engines (e.g., SQL, influxDB,
Z3).
To have an open framework in which multiple policy engines and mulitple
data sources work well together, we need an explicit type system that is
extensible (to fit the particular needs for different data sources and
different policy engines), loosely-coupled (between different data sources and
policy engines), and backwards-compatible with the standard ``agnostic`` engine
and existing data source drivers.
As an illustration of the need for extensibility and loose-coupling, consider
that a SMT-based Z3 prover needs types that are as narrow as possible
(e.g., enum{'ingress', 'egress'}), but another engine does not understand
such narrow types. A data source driver then must specify its types
in a way that works well for both engines.
Proposed change
===============
Overview
--------
The basic idea is that we let each data source driver describe its data as
precisely and narrowly as practical using as combination of base types
and custom types.
Then each policy engine is responsible for using that information to represent
the data using its internal representation. A policy engine would likely not
understand every custom type used by a data source driver (especially because
we expect new ones to be introduced over time). To allow a policy engine to
handle custom data types it does not "understand", we require that every
custom data type inherit from another data type, all of which ultimately
inherit from one of the base types all policy engines must handle. That way,
when a policy engine encounters a value of a custom type
it does not understand, the policy engine can fall back to handling the value
according to an ancestor type the engine does understand.
What is a type?
---------------
To define what a type is in this spec, we first establish four related
concepts:
#. *source representation*: the representation used in the data received from
an external source. Each external data source has its own source
representation defined outside of Congress. In an IP address example, one
source could use IPv4 dotted decimal string ``"1.0.0.1"`` while another
source could use IPv6 (short) hexadecimal string ``"::ffff:100:1"``.
#. *internal representation*: the representation used by a Congress service
(e.g., a policy engine). Each Congress service can use a different internal
representation. For example, the IP address may be represented as a string
in one engine, as an integer another engine, and as a bit-vector in yet
another engine.
#. *exchange representation*: the representation used as the data is published
and received among Congress services. In the IP address example, this could
be the integer ``0x01000001``.
Data is received from an external data source in the form of the
*source representation*. The data source driver converts from the source
representation to the *exchange representation* before publishing the data to
Congress services such as the policy engine. A Congress service such as a
policy engine will process the data and use some form of
*internal representation* for storing the data for future use.
The primary purpose of a Congress data type is to be an exchange format
specification used between the various Congress data sources and services each
with their own representations. The developers of each Congress service
can use the Congress data type of a piece of data or a table column to
know what exchange format to expect and what is being represented. For this
purpose, a Congress data type can be thought of as a tag associated with each
piece of data or table column.
As a secondary convenience, each Congress data type also provides helpful
methods for processing a value of that type. At a minimun, the Congress
data type would provide a method for validating a value as valid for that
type.
Finally, the entire collection of Congress data types cannot be anticipated
upfront. So it is impractical for each Congress data consumer to be programmed
to recognize and handle every Congress data type. We need a more structured
and extensible collection of "tags". We organize the Congress data types
in an is-a hierarchy. First there are a number of base types which all
Congress data consumers must recognize and handle. All additional non-base
types inherit from a base type or another non-base type in a well-founded
hierarchy. Then, each data consumer can handle all data types simply by
treating each piece of data according to its base type. Optionally, a data
consumer can add special handling of certain non-base types.
As an example of inheritance, the IP address type could inherit from the
general string type so that policy engines which do not specifically
handle the IP address type will handle the data as a general string.
Putting all three of the above pieces together, we define a Congress data type
as a subclass of the following ``CongressDataType`` abstract base class.
.. code-block:: python
@six.add_metaclass(abc.ABCMeta)
class CongressDataType(object):
@classmethod
@abc.abstractmethod
def validate(cls, value):
'''Validate a value as valid for this type.
:Raises ValueError: if the value is not valid for this type
'''
raise NotImplementedError
@classmethod
def _get_parent(cls):
congress_parents = [parent for parent in cls.__bases__
if issubclass(parent, CongressDataType)]
if len(congress_parents) == 1:
return congress_parents[0]
elif len(congress_parents) == 0:
raise cls.CongressDataTypeNoParent(
'No parent type found for {0}'.format(cls))
else:
raise cls.CongressDataTypeHierarchyError(
'More than one parent type found for {0}: {1}'
.format(cls, congress_parents))
.. important::
These classes are not expected to be instantiated as wrapper objects, but
they merely act as organizing units of these class methods that serve to
describe and manipulate the values as well as encode the data type hierarchy.
The reason why classes are used is two fold.
1. The Python inheritance hierarchy is a convenient way to declare and access
the relationship between the Congress data types. Each non-root concrete
subclass (A) of CongressDataType inherits from exactly one other concrete
subclass (B) of CongressDataType to encode that (A) is a subtype of (B)
in the data type hierarchy. A concrete subclass of CongressDataType may
inherit from multiple parent Python classes, but among the parents there
must not be more than one subclass of CongressDataType.
2. Python inheritance makes it convenient for a Congress data type to reuse
the code in a parent Congress data type.
Further details
---------------
Specification of data types
```````````````````````````
Data source drivers may optionally specify explicit data types in its
value translator definition. The data service base class attaches data type
information (table schema) in each table publish.
For flexibility and backwards-compatibility, an explicit type is not required
for every column.
The standard ``agnostic`` engine handles untyped table columns.
Other policy engines may or may not handle untyped tables columns.
When a policy engine that requires typing receives untyped data, the policy
engine may either reject the data or impose typing.
Below is a sample translator definition showing how the explicit data types
could be specified for the ``flavors`` table in the Nova data source driver.
.. code-block:: python
flavors_translator = {
'translation-type': 'HDICT',
'table-name': FLAVORS,
'selector-type': 'DOT_SELECTOR',
'field-translators':
({'fieldname': 'id', 'desc': 'ID of the flavor',
'translator': {'translation-type': 'VALUE',
'data-type': CongressStr}},
{'fieldname': 'name', 'desc': 'Name of the flavor',
'translator': {'translation-type': 'VALUE',
'data-type': CongressStr}},
{'fieldname': 'vcpus', 'desc': 'Number of vcpus',
'translator': {'translation-type': 'VALUE',
'data-type': CongressInt}},
{'fieldname': 'ram', 'desc': 'Memory size in MB',
'translator': {'translation-type': 'VALUE',
'data-type': CongressInt}},
{'fieldname': 'disk', 'desc': 'Disk size in GB',
'translator': {'translation-type': 'VALUE',
'data-type': CongressInt}},
{'fieldname': 'ephemeral', 'desc': 'Ephemeral space size in GB',
'translator': {'translation-type': 'VALUE',
'data-type': CongressInt}},
{'fieldname': 'rxtx_factor', 'desc': 'RX/TX factor',
'translator': {'translation-type': 'VALUE',
'data-type': CongressFloat}})
}
Union of types
``````````````
When a policy engine derives a new table out from existing tables, it create
a table column which contains data from more than one type. In these cases,
the new column shall be typed as the least common ancestor type of all the
constituent data types. To ensure that a least common ancestor type always
exists, we establish the Congress string type as the root parent of all the
other types.
Types hierarchy
```````````````
This spec proposes a flexible and extensible type framework. The precise
collection of types would be decided in implementation patches and also
evolved over time. As point of reference, we provide below a sample hierarchy
of types below. (Items enclosed in are abstract categories of types.)
- string
- [bounded length sting]
- string of length up to N
- [string enumerations]
- boolean
- network direction (ingress and egress)
- [fixed length string]
- string of length N
- UUID
- decimal (fixed precision rational number represented as a string)
- integer
- [integer enumerations]
- integer between A and B
- short integer
- floating point
- IP address
Data types sample code
``````````````````````
Sample code for a few selected data types are included here for reference.
.. code-block:: python
class CongressStr(CongressDataType):
'''Base type representing a string.
Sample implementation only. Final implementation needs to deal with
nullability, unicode, and other issues.'''
@classmethod
def validate(cls, value):
if not isinstance(value, six.string_types):
raise ValueError
class CongressBool(CongressStr):
@classmethod
def validate(cls, value):
if not isinstance(value, bool):
raise ValueError
class CongressIPAddress(CongressStr):
@classmethod
def validate(cls, value):
try:
ipaddress.IPv4Address(value)
except ipaddress.AddressValueError:
try:
ipv6 = ipaddress.IPv6Address(value)
if ipv6.ipv4_mapped:
# as a design decision in standardized format
# addresses in ipv4 range should be in ipv4 syntax
raise ValueError
except ipaddress.AddressValueError:
raise ValueError
Abstract categories of data types
`````````````````````````````````
Sometimes, a collection of types all follow a certain pattern. In these
cases, we can provide a framework for creating and handling these types in
a more generic way.
For example, many types are characterized by a fixed, finite domain of strings.
In such cases, we can provide a type factory for creating types in this space.
The following is a sample implementation for a type factory function for
a fixed, finite domain of strings.
In addition, we can also provide a mix-in abstract base class that data
consumers can use to handle the data in a more generic way. In this case,
the abstract base class ``CongressTypeFiniteDomain`` stipulates that
each type inheriting from this class must have a class variable DOMAIN
which is a frozenset of the set of values allowed in the type.
A data consumer may use this information to handle the value in a generic
way without recognizing the specific concrete type used.
Sample code included below for reference.
.. code-block:: python
@six.add_metaclass(abc.ABCMeta)
class CongressTypeFiniteDomain(object):
'''Abstract base class for a Congress type of bounded domain.
Each type inheriting from this class must have a class variable DOMAIN
which is a frozenset of the set of values allowed in the type.
'''
pass
def create_congress_str_enum_type(class_name, enum_items):
'''Return a sub-type of CongressStr for representing a string from
a fixed, finite domain.'''
for item in enum_items:
if not isinstance(item, six.string_types):
raise ValueError
class NewType(CongressStr, CongressTypeFiniteDomain):
DOMAIN = frozenset(enum_items)
@classmethod
def validate(cls, value):
if not value in cls.DOMAIN:
raise ValueError
NewType.__name__ = class_name
return NewType
As an example, a particular data source driver dealing with firewall rules may
use the factory function to create a custom type as follows.
.. code-block:: python
NetworkDirection = create_congress_str_enum_type(
'NetworkDirection', ('ingress', 'egress'))
Converting to ancestor types
````````````````````````````
Generally, each descendent type value is directly interpretable as a value
of an ancestor type. Ideally, every policy engine would recognize and support
The only exception is that values of the non-string types
inheriting from CongressStr need to be converted to string to be interprable
as a value of CongressStr type.
The CongressDataType abstract base class can include additional helper methods
to make the interpretation easy. Below is an expanded
CongressDataType definition including the additional helper methods.
.. code-block:: python
@six.add_metaclass(abc.ABCMeta)
class CongressDataType(object):
@classmethod
@abc.abstractmethod
def validate(cls, value):
'''Validate a value as valid for this type.
:Raises ValueError: if the value is not valid for this type
'''
raise NotImplementedError
@classmethod
def least_ancestor(cls, target_types):
'''Find this type's least ancestor among target_types
This method helps a data consumer find the least common ancestor of
this type among the types the data consumer supports.
:param supported_types: iterable collection of types
:returns: the subclass of CongressDataType
which is the least ancestor
'''
target_types = frozenset(target_types)
current_class = cls
try:
while current_class not in target_types:
current_class = current_class._get_parent()
return current_class
except cls.CongressDataTypeNoParent:
return None
@classmethod
def convert_to_ancestor(cls, value, ancestor_type):
'''Convert this type's exchange value
to ancestor_type's exchange value
Generally there is no actual conversion because descendant type value
is directly interpretable as ancestor type value. The only exception
is the conversion from non-string descendents to string. This
conversion is needed by Agnostic engine does not support boolean.
.. warning:: undefined behavior if ancestor_type is not
an ancestor of this type.
'''
if ancestor_type == CongressStr:
return json.dumps(value)
else:
return value
@classmethod
def _get_parent(cls):
congress_parents = [parent for parent in cls.__bases__
if issubclass(parent, CongressDataType)]
if len(congress_parents) == 1:
return congress_parents[0]
elif len(congress_parents) == 0:
raise cls.CongressDataTypeNoParent(
'No parent type found for {0}'.format(cls))
else:
raise cls.CongressDataTypeHierarchyError(
'More than one parent type found for {0}: {1}'
.format(cls, congress_parents))
class CongressDataTypeNoParent(TypeError):
pass
class CongressDataTypeHierarchyError(TypeError):
pass
Re-imposing types
`````````````````
(Potential future work included here for reference.)
Policy engines may publish data for consumption by other policy engines.
When a policy engine (e.g., Z3) with narrower types publish to another policy
engine (e.g., Congress agnostic) with
unfixed types or broader types, the handling is natural. When the reverse
situation happens, the receiving policy engine may be unable to handle the
data.
As future work, the publishing policy engine can optionally infer and
re-impose narrower types where possible.
Also as future work, policy engines that require narrower types can employ
an additional import construct that imposes narrower types on the tables they
subscribe to from other policy engines.
Alternatives
------------
In this section we discuss a few ways alternative proposals may deviate from
this spec.
#. Require that all policy engines support untyped data.
This alternative would rule out many top candidates for policy engines
(e.g., SQL-based, influxDB-based, Z3-based).
#. Use a fixed set of types all data sources and all policy engines use.
A fixed set of types should be a first candidate because it is the simplest
approach. Unfortunately, it is impossible to anticipate all the types
needed. Whenever a new policy engine is added, distinctions that may not
have mattered before becomes critical. For example, without an SMT-based
engine such as Z3, string enum types are not so important to distinguish
from general string types. So when a Z3 engine is introduced, we then need
to 1) create new types, 2) update data source drivers to specify using the
new types, and 3) update the existing policy engines to handle the new
types.
The decoupled and extensible approach proposed in this spec provides for a
much smoother way to evolve the types. Each data source driver specifies
types as narrowly as practical, without regard to what the policy engines
want. Each policy engine can then make use of the type information as much
or as little as needed to map data to its own internal representations.
So when new types, new data source drivers, or new policy engines are
introduced, none of the existing code need to change.
#. Instead of specifying types in the data publisher, let types be imposed
exclusively in the data consumer.
This alternative is helpful when a fixed data type cannot be obtained from
the data publisher (see point 5 in the section above). When a data
source/publisher does have information on a fixed data type (inferred from
fixed data types defined in code), it is better to avoid the unnecessary
operational burden of the operator having to define data types at the point
of consumption.
#. Instead of requiring that each value of a child type be directly
interpretable as a value of a parent type, allowing a conversion between
child and parent can give more flexibility to choose the exchange
representation that best fits a particular type, independently of
the type's parent type. For example, an IP address can be represented
as a single integer value instead of as a string.
The increase flexibility could be useful, but so far has not proven to be
necessary.
Policy
------
Not applicable.
Policy actions
--------------
Not applicable.
Data sources
------------
Each data source driver may optionally specify data types in its value
translator. Data source drivers can be updated with explicit types as needed
by new use cases and new policy engines.
See section `Specification of data types`_ for additional details.
Data model impact
-----------------
No impact on Congress's data model as defined in congress/db/
REST API impact
---------------
No impact on REST API.
Security impact
---------------
There is little new security impact. Because new/custom types include
new/custom data handling methods, it does theoretically increase the attack
surface. Care needs to be taken to make sure the data handling methods are safe
against malformed or possibility malicious input. There are well-known best
practices to minimize such risks.
Notifications impact
--------------------
No impact.
Other end user impact
---------------------
No immediate impact on end users. This spec can indirectly benefits end
users through the additional policy engines made possible.
All existing behaviors and workflows are preserved.
Performance impact
------------------
The data and memory impact is expected to be minimal because the data is stored
and transmitted in primitive form without the overhead of custom objects.
The other main performance impact is the treatment of a piece of data as an
ancestor type.
Identifying the least ancestor type by tracing up the type hierarchy. Done
on a per-column basis, this step has negligible performance cost because the
type hierarchy is shallow and because each type has at most one ancestor. The
tracing up has worst case running time linear in the depth of the type
hierarchy.
(See `Converting to ancestor types`_ for sample
implementation.)
Other deployer impact
---------------------
No direct impact on deployers. This spec can indirectly benefits end
users through the additional policy engines made possible.
All existing behaviors and workflows are preserved.
Developer impact
----------------
No significant changes are forced on a developer. Use of type information by
a policy engine is optional. Specification of type information in a data source
driver is also optional.
Implementation
==============
Assignee(s)
-----------
Individual tasks will be picked up and tracked on launchpad. Primary contact is
ekcs <ekcs.openstack@gmail.com>.
Work items
----------
- Minimal implementation.
- Abstract base class for type
- Primitive types
- Data Service Engine (DSE) changes to store and convey (optional) type
information on columns.
- Further implementation. These further implementations are optional and can be
done in an incremental fashion as needed by use cases.
- Adding type information to data source drivers.
- Standard policy engine (Agnostic) changes to use type information.
- Extended types (e.g., IP address)
- Categories of types (e.g., fixed enumeration of strings)
- Framework for loading custom types defined in data source drivers
- Inference and re-imposition of types when a policy engine publishes tables
- Subscriber-end imposition of additional type information
Dependencies
============
No notable external dependencies.
Testing
=======
The incorporation of additional type information in the data service engine
will primarily tested through unit tests in the style of the existing tests in
congress/tests/dse2/
When a data source driver is updated to include additional type information,
additional checks on the type information can be naturally added to the
existing tempest tests that compares an independently obtained service state
to the state obtained by the data source driver and translated into tables.
Documentation impact
====================
No direct impact.
References
==========
A relevant type system for a Z3-based datalog solver:
- https://github.com/Orange-OpenSource/octant/blob/master/doc/source/user/index.rst
- https://github.com/Orange-OpenSource/octant/blob/master/octant/datalog_primitives.py

0
specs/stein/.gitkeep
View File

511
specs/stein/json-data-model.rst
View File

@ -1,511 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
============================================
Experimental component for a JSON data model
============================================
https://blueprints.launchpad.net/congress/+spec/json-data-model
The translation from JSON source data to tabular Congress data places the
limited resources of Congress developers in the way of the Congress users
having the flexibility to quickly try/use any data source and data field as
the need arises in their policy use cases. This spec is a proposal to
experiment with removing the translation step, thus retaining the source JSON
data model, as a promising way to overcome the limitation.
Problem description
===================
In cloud computing, most of the source data is JSON. In the Congress model,
the developers carefully understand and curate the source data to develop the
data source drivers that make the data available to end users in table form.
The translation into table model has the advantage of presenting a simpler
and more consistent data model to the user writing policy on the data.
However, with so many sources of data out there and new data being added to
existing sources through frequent microversions (e.g., Nova), it is impossible
for Congress developers to anticipate and keep up with all the data sources or
even all the data fields in a given data source the end users desire.
It often happens that as the user writes policy, the user finds out that
additional data fields or sources are needed for the policy. In that case,
the user needs to contact upstream developers, wait for them to add the data
fields or sources, then wait for the new release, then upgrade to the latest
release. All this before the end user can even test out the desired policy.
In order to better support those operators who have rapidly emerging and
evolving use cases, we need to experiment with an additional model: remove
translation and developer curation from the loop, allowing users to integrate
new data sources without code change.
Proposed change
===============
Create an experimental component of Congress (tentatively congress-json)
to allow writing policy over json data stored in PostgreSQL json data model.
Congress-CloudState would ingest the source data mostly as-is, in JSON format,
into a PostgreSQL database using the JSONB column type in PostgreSQL. The
`PostgreSQL JSON syntax
<https://hackernoon.com/how-to-query-jsonb-beginner-sheet-cheat-4da3aa5082a3>`_
(see the policy_ section for examples) would allow for rules (SQL views) to
operate directly over the JSON data.
Because no translation is needed, new data sources can be configured without
code change. For example, the following sample YAML config file configures
Congress-CloudState to poll the nova endpoint. It tells Congress-CloudState to
poll the path servers/detail, extract the results using the jsonpath
expression ``$.servers[:]``, and finally populate the nova.servers table with
the results.
::
name: nova
poll: 60
authentication:
type: keystone_password
user: congress_access
password: secret
api_endpoint: https://stack.org/compute/v2.26
tables:
servers:
api_path: servers/detail
api_verb: get
jsonpath: $.servers[:]
Architecture
------------
The architecture is fairly simple. Congress-CloudState has two major
components.
- The data ingestor polls the cloud services via API and populates the
database with the data. The ingestor also accepts webhook notifications.
- The executor reads from the database to determine what actions to execute,
then calls the appropriate APIs in the cloud services to execute the
actions.
The users mostly interact directly with the database to add/remove policy
rules and query policy results.
The following diagram shows the high level components and flow of data.
::
+----------------------------------------------------------+
| Cloud Services |
+--------------------------------------------------^-------+
| |
| |
+------v--------+ +---------------+ +----------------+
| Congress-JSON | | | | Congress-JSON |
| Data Ingestor +----> PostgreSQL +----> Executor |
| | | | | |
+---------------+ +----------^----+ +----------------+
| |
| |
+----v----------+
| |
| Users |
| |
+---------------+
Benefits
--------
- Very easy to add new data source because no translation is required. This
`blog post
<https://pndablog.com/2017/04/26/schema-on-write-vs-schema-on-read/>`_
has more discussion of the pros and cons of the approach (schema-on-read).
- No code change and release needed to consume a new field in a new API
microversion. The operator simply changes the desired microversion in
Congress-CloudState configuration.
This `patch <https://review.openstack.org/#/c/611516/>`_ is a recent
example of how much time and work it takes to consume a new field previously
not included by Congress, not to mention the delay for users waiting for a
new release to consume a new field.
- Much is available in the way of editors, debuggers, tutorials, and online
help for PostgreSQL language, making it easy for new users to write
policies.
- PostgreSQL has `JSON indexing support
<https://blog.2ndquadrant.com/nosql-postgresql-9-4-jsonb/>`_
which allows for a wide class of performant queries.
Alternatives
------------
- Directly make changes to Congress agnostic policy engine to support JSON data
- Here is a rough estimate of the work required to implement similar
functionality in Congress itself.
- Design a new version of Datalog language to support JSON. This step is
hard. As of now, there is no concensus in the Datalog community on how
best to do this.
- Major changes to congress/datalog/* to process the new language.
- Major changes to the policy engine to support the new language.
- Changes to congress/dse2/* to accommodate JSON data.
- Instead of attempting all the work outlined above, we propose to create
a very simple new component which leverages PostgreSQL allow policy over
JSON data.
If the approach proves successful, a decision could be made in the future
to allow congress-json to be used as a tool for extracting from JSON data
the table data needed by other parts of Congress (e.g., agnostic, z3).
- Homegrow a policy engine supporting JSON data rather than use one
off-the-shelf
- It would take a tremendous amount of unnecessary development and
maintenance work.
- Choose a datastore other than PostgreSQL
- The following are the major requirements for the datastore
- JSON data support
- Widely-adopted declarative language
- Performant (off-key) join
- Support for rules (views)
- Transactional write
- Strong open source community with appropriate licensing
- Highly-available deployment option
- Fine-grained permissions
- Most of the well-known NoSQL solutions do not provide performant off-key
joins
- MySQL/MariaDB has support for JSON data but limited indexing support
(typically requiring the indexed value be declared as a virtual column)
- Among all the options evaluated, PostgreSQL best satisfies the
requirements, with the following notable features:
- Flexible indexing for JSONB data type supporting performant off-key
joins
- Very strong open source community with permissive license
- A wealth of (first party and third party) tools and online knowledge for
learning, query writing, query debugging, and performance-tuning
- A flexible and expressive permissions system
- We propose to start with PostgreSQL, while leaving the option open to add
support for other datastores down the road through a plugin architecture.
Policy
------
Here we give several examples to demonstrate how policy and rules would be
created. Each interaction is shown in both classic Congress interaction as
well as Congress-CloudState interaction for reader’s understanding.
Sample data
~~~~~~~~~~~
The nova.servers table is the collection of all the JSON documents
representing servers, each document in a row with a single column d containing
the document. Each server is represented by a JSON document as supplied by
Nova’s list servers (detailed) API. Here is a simplified version of a sample
JSON document representing a Nova server (the UUIDs have been replaced with
more readable strings for ease of illustration):
::
{
"id":"server-134",
"name":"server 134",
"status":"ACTIVE",
"tags":[
"production",
"critical"
],
"hostId":"host-05",
"host_status":"ACTIVE",
"metadata":{
"HA_Enabled":false
},
"tenant_id":"tenant-52",
"user_id":"user-830",
"flavor": {
"disk": 1,
"ephemeral": 0,
"extra_specs": {
"hw:cpu_policy": "dedicated",
"hw:mem_page_size": "2048"
},
"original_name": "m1.tiny.specs",
"ram": 512,
"swap": 0,
"vcpus": 1
}
}
Example 1
~~~~~~~~~
- Create policy (schema).
- Congress syntax:
::
congress policy create vm_error
- Congress-CloudState equivalent:
::
CREATE SCHEMA vm_host_down;
- Create policy rule (view) identifying those servers whose host is down.
- Congress syntax:
::
congress policy rule create vm_host_down '
error(server_id) :-
nova:servers(id=server_id, host_id=host_id),
nova:hypervisors(id=host_id, state="DOWN")'
- Congress-CloudState equivalent:
::
CREATE VIEW vm_host_down.error AS
SELECT d->>'id' AS server_id
FROM nova.servers
WHERE d->>'host_status' = 'DOWN';
- Note on syntax: the :code:`->>` operator accesses the content of a JSON
object field and returns the result as text.
- Query the results in the policy table.
- Congress syntax:
::
congress policy row list vm_host_down error
- Congress-CloudState equivalent:
::
SELECT * FROM vm_host_down.error;
- Create policy rule (view) identifying 'critical'-tagged servers whose host
is down.
- Congress syntax:
::
congress policy rule create vm_host_down '
critical(server_id) :-
nova:servers(id=server_id, host_id=host_id),
nova:hypervisors(id=host_id, state="DOWN"),
nova:tags(server_id=server_id, tag="critical")'
- Congress-CloudState equivalent:
::
CREATE VIEW vm_host_down.critical AS
SELECT d->>'id' AS server_id
FROM nova.servers
WHERE d->>'host_status' = 'DOWN'
AND d->'tags' ? 'critical';
- Note on syntax:
- The :code:`->` operator accesses the content of a JSON object field
and returns the result as JSON. In this example, :code:`d->'tags'`
returns the array of tags associated with each server.
- The :code:`?` operator checks that a string is a top-level key/element
in a JSON structure. In this example, the
:code:`d->'tags' ? 'critical'` condition checks that the string
'critical' is in the array of tags retrieved by :code:`d->'tags'`.
Example 2
~~~~~~~~~
- Create policy (schema).
- Congress syntax:
::
congress policy create production_stable
- Congress-CloudState equivalent:
::
CREATE SCHEMA production_stable;
- Create policy rule (view) identifying production servers using an unstable
image.
- Congress syntax:
::
congress policy rule create production_stable '
error(id) :-
nova:servers(id=id, image=image_id),
nova:tags(id=id, tag="production"),
glance:tags(image_id=image_id, tag="unstable")'
- Congress-CloudState equivalent:
::
CREATE VIEW production_stable.error AS
SELECT server.d->>'id' AS server_id,
image.d->>'id' AS image_id
FROM nova.servers server
JOIN glance.images image
ON server.d->'image'->'id' = image.d->'id'
WHERE (server.d->'tags' ? 'production')
AND (image.d->'tags' ? 'unstable');
- Note on syntax: the join-condition
:code:`server.d->'image'->'id' = image.d->'id'` matches the glance image
with the server whose image ID matches the glance image ID.
Example 3
~~~~~~~~~
This example illustrates that accessing deeply nested data can be relatively
straightforward.
The following view (rule) identifies all the servers using a flavor with cpu
policy 'dedicated'. Despite the information being buried several layers deep,
it is relatively straightforward to access simply by following the structure.
::
CREATE VIEW dedicated_servers AS
SELECT *
FROM nova.servers
WHERE d -> 'flavor' -> 'extra_specs' ->> 'hw:cpu_policy' = 'dedicated';
Policy actions
--------------
Policy actions would work similarly to agnostic engine.
The policy writer would define a policy rule (view) specifying in each row
what action to call on which service using what parameter values.
As in agnostic engine, the new component would query the 'execute' table and
then call the appropriate client/API methods according to the rows in the
'execute' table of each policy.
Data sources
------------
A new type of data source is introduced which is configured through YAML files
and stores data in the original JSON format without translation.
Data model impact
-----------------
No impact on the data model.
REST API impact
---------------
Minimal REST API impact. The webhook model for accepting webhooks is naturally
extended to the JSON data sources when configured.
Security impact
---------------
No security impact when the experimental component is not explicitly.
When enabled, there is an additional data access which would be protected by
PostgreSQL permissions set up by Congress.
Notifications impact
--------------------
No impact.
Other end user impact
---------------------
End users will have the option of using the well-known
PostgreSQL syntax to write policy directly on the source data format,
having direct access to all the source data fields.
Performance impact
------------------
Preliminary performance testing on 100,000 records show adequate performance
on both data ingestion and query evaluation. If a query performance issues
does arise, there is a wealth of tools and knowledge for tuning query
performance in PostgreSQL.
Other deployer impact
---------------------
Additional config/commandline options to control the deployment of the new
experimental component.
Developer impact
----------------
congress-json is expected to be a very simple component. As a result,
congress-json is expected to require very minimal maintenance from
developers.
Implementation
==============
Assignee(s)
-----------
Primary assignee: ekcs
Others welcome to pick up tasks.
Work items
----------
- Database initializer
- JSON-postgres data source driver
- Integration testing
Dependencies
============
The only major new dependency is PostgreSQL, which is already used in
OpenStack deployments. Version 9.4 or higher required (Ubuntu xenial comes with
9.5). It affects only those enabling the experimental new component.
Testing
=======
Simple tempest scenarios for the new component. They may piggyback on the
existing congress-tempest-postgres jobs.
Documentation impact
====================
Standard documentation expected for the new component.
References
==========
* `Schema-on-read pros and cons
<https://pndablog.com/2017/04/26/schema-on-write-vs-schema-on-read/>`_
* `PostgreSQL JSON operators and functions
<https://www.postgresql.org/docs/9.4/functions-json.html>`_
* `PostgreSQL JSON syntax cheat sheet
<https://hackernoon.com/how-to-query-jsonb-beginner-sheet-cheat-4da3aa5082a3>`_
* `JSON indexing support
<https://blog.2ndquadrant.com/nosql-postgresql-9-4-jsonb/>`_

354
specs/template.rst
View File

@ -1,354 +0,0 @@
..
This work is licensed under a Creative Commons Attribution 3.0 Unported
License.
http://creativecommons.org/licenses/by/3.0/legalcode
==========================================
Example Spec - The title of your blueprint
==========================================
Include the URL of your launchpad blueprint:
https://blueprints.launchpad.net/congress/+spec/example
Introduction paragraph -- why are we doing anything? A single paragraph of
prose that operators can understand.
Some notes about using this template:
* Your spec should be in ReSTructured text, like this template.
* Please wrap text at 79 columns.
* The filename in the git repository should match the launchpad URL, for
example a URL of: https://blueprints.launchpad.net/congress/+spec/awesome-thing
should be named awesome-thing.rst
* Please do not delete any of the sections in this template. If you have
nothing to say for a whole section, just write: None
* For help with syntax, see http://sphinx-doc.org/rest.html
* To test out your formatting, build the docs using tox, or see:
http://rst.ninjs.org
* If you would like to provide a diagram with your spec, ascii diagrams are
required. http://asciiflow.com/ is a very nice tool to assist with making
ascii diagrams. The reason for this is that the tool used to review specs is
based purely on plain text. Plain text will allow review to proceed without
having to look at additional files which can not be viewed in gerrit. It
will also allow inline feedback on the diagram itself.
Problem description
===================
A detailed description of the problem:
* For a new feature this might be use cases. Ensure you are clear about the
actors in each use case: End User vs Deployer
* For a major reworking of something existing it would describe the
problems in that feature that are being addressed.
Proposed change
===============
Here is where you cover the change you propose to make in detail. How do you
propose to solve this problem?
If this is one part of a larger effort make it clear where this piece ends. In
other words, what's the scope of this effort?
Alternatives
------------
What other ways could we do this thing? Why aren't we using those? This doesn't
have to be a full literature review, but it should demonstrate that thought has
been put into why the proposed solution is an appropriate one.
Policy
------
Using the Congress datalog syntax, write out an example policy using
https://wiki.openstack.org/wiki/Congress#Policy_Language
Example:
error(vm) :-
nova:virtual_machine(vm),
ids:ip_packet(src_ip, dst_ip),
neutron:port(vm, src_ip), //finds out the port that has the VM's IP
ids:ip_blacklist(dst_ip).
Policy actions
--------------
Describe the policy activities in terms of monitoring, reactive, proactive,
and other ways to explain how the policy will implement it's desired state.
Data sources
------------
Describe which projects and/or services the data is coming from
Data model impact
-----------------
Changes which require modifications to the data model often have a wider impact
on the system. The community often has strong opinions on how the data model
should be evolved, from both a functional and performance perspective. It is
therefore important to capture and gain agreement as early as possible on any
proposed changes to the data model.
Questions which need to be addressed by this section include:
* What new data objects and/or database schema changes is this going to
require?
* What database migrations will accompany this change.
* How will the initial set of new data objects be generated, for example if you
need to take into account existing instances, or modify other existing data
describe how that will work.
REST API impact
---------------
Each API method which is either added or changed should have the following
* Specification for the method
* A description of what the method does suitable for use in
user documentation
* Method type (POST/PUT/GET/DELETE)
* Normal http response code(s)
* Expected error http response code(s)
* A description for each possible error code should be included
describing semantic errors which can cause it such as
inconsistent parameters supplied to the method, or when an
instance is not in an appropriate state for the request to
succeed. Errors caused by syntactic problems covered by the JSON
schema definition do not need to be included.
* URL for the resource
* Parameters which can be passed via the url
* JSON schema definition for the body data if allowed
* JSON schema definition for the response data if any
* Example use case including typical API samples for both data supplied
by the caller and the response
* Discuss any policy changes, and discuss what things a deployer needs to
think about when defining their policy.
Example JSON schema definitions can be found in the Nova tree
https://opendev.org/openstack/nova/src/branch/master/nova/api/openstack/compute/schemas
Note that the schema should be defined as restrictively as
possible. Parameters which are required should be marked as such and
only under exceptional circumstances should additional parameters
which are not defined in the schema be permitted (eg
additionaProperties should be False).
Reuse of existing predefined parameter types such as regexps for
passwords and user defined names is highly encouraged.
Security impact
---------------
Describe any potential security impact on the system. Some of the items to
consider include:
* Does this change touch sensitive data such as tokens, keys, or user data?
* Does this change alter the API in a way that may impact security, such as
a new way to access sensitive information or a new way to login?
* Does this change involve cryptography or hashing?
* Does this change require the use of sudo or any elevated privileges?
* Does this change involve using or parsing user-provided data? This could
be directly at the API level or indirectly such as changes to a cache layer.
* Can this change enable a resource exhaustion attack, such as allowing a
single API interaction to consume significant server resources? Some examples
of this include launching subprocesses for each connection, or entity
expansion attacks in XML.
For more detailed guidance, please see the OpenStack Security Guidelines as
a reference (https://wiki.openstack.org/wiki/Security/Guidelines). These
guidelines are a work in progress and are designed to help you identify
security best practices. For further information, feel free to reach out
to the OpenStack Security Group at openstack-security@lists.openstack.org.
Notifications impact
--------------------
Please specify any changes to notifications. Be that an extra notification,
changes to an existing notification, or removing a notification.
Other end user impact
---------------------
Aside from the API, are there other ways a user will interact with this
feature?
* Does this change have an impact on python-congressclient? What does the user
interface there look like?
Performance impact
------------------
Describe any potential performance impact on the system, for example
how often will new code be called, and is there a major change to the calling
pattern of existing code.
Examples of things to consider here include:
* A periodic task might look like a small addition but if it calls conductor or
another service the load is multiplied by the number of nodes in the system.
* Scheduler filters get called once per host for every instance being created,
so any latency they introduce is linear with the size of the system.
* A small change in a utility function or a commonly used decorator can have a
large impacts on performance.
* Calls which result in a database queries (whether direct or via conductor)
can have a profound impact on performance when called in critical sections of
the code.
* Will the change include any locking, and if so what considerations are there
on holding the lock?
Other deployer impact
---------------------
Discuss things that will affect how you deploy and configure OpenStack
that have not already been mentioned, such as:
* What config options are being added? Should they be more generic than
proposed (for example a flag that other hypervisor drivers might want to
implement as well)? Are the default values ones which will work well in
real deployments?
* Is this a change that takes immediate effect after its merged, or is it
something that has to be explicitly enabled?
* If this change is a new binary, how would it be deployed?
* Please state anything that those doing continuous deployment, or those
upgrading from the previous release, need to be aware of. Also describe
any plans to deprecate configuration values or features. For example, if we
change the directory name that instances are stored in, how do we handle
instance directories created before the change landed? Do we move them? Do
we have a special case in the code? Do we assume that the operator will
recreate all the instances in their cloud?
Developer impact
----------------
Discuss things that will affect other developers working on OpenStack,
such as:
* If the blueprint proposes a change to the driver API, discussion of how
other hypervisors would implement the feature is required.
Implementation
==============
Assignee(s)
-----------
Who is leading the writing of the code? Or is this a blueprint where you're
throwing it out there to see who picks it up?
If more than one person is working on the implementation, please designate the
primary author and contact.
Primary assignee:
<launchpad-id or None>
Other contributors:
<launchpad-id or None>
Work items
----------
Work items or tasks -- break the feature up into the things that need to be
done to implement it. Those parts might end up being done by different people,
but we're mostly trying to understand the timeline for implementation.
Dependencies
============
* Include specific references to specs and/or blueprints in congress, or in
other projects, that this one either depends on or is related to.
* If this requires functionality of another project that is not currently used
by congress (such as the glance v2 API when we previously only required v1),
document that fact.
* Does this feature require any new library dependencies or code otherwise not
included in OpenStack? Or does it depend on a specific version of library?
Testing
=======
Please discuss how the change will be tested. We especially want to know what
tempest tests will be added. It is assumed that unit test coverage will be
added so that doesn't need to be mentioned explicitly, but discussion of why
you think unit tests are sufficient and we don't need to add more tempest
tests would need to be included.
Is this untestable in gate given current limitations (specific hardware /
software configurations available)? If so, are there mitigation plans (3rd
party testing, gate enhancements, etc).
Documentation impact
====================
What is the impact on the docs team of this change? Some changes might require
donating resources to the docs team to have the documentation updated. Don't
repeat details discussed above, but please reference them here.
References
==========
Please add any useful references here. You are not required to have any
reference. Moreover, this specification should still make sense when your
references are unavailable. Examples of what you could include are:
* Links to mailing list or IRC discussions
* Links to notes from a summit session
* Links to relevant research, if appropriate
* Related specifications as appropriate (e.g. if it's an EC2 thing, link the
EC2 docs)
* Anything else you feel it is worthwhile to refer to

0
specs/train/.gitkeep
View File

0
tests/__init__.py
View File

107
tests/test_titles.py
View File

@ -1,107 +0,0 @@
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
import glob
import re
import docutils.core
import testtools
class TestTitles(testtools.TestCase):
def _get_title(self, section_tree):
section = {
'subtitles': [],
}
for node in section_tree:
if node.tagname == 'title':
section['name'] = node.rawsource
elif node.tagname == 'section':
subsection = self._get_title(node)
section['subtitles'].append(subsection['name'])
return section
def _get_titles(self, spec):
titles = {}
for node in spec:
if node.tagname == 'section':
section = self._get_title(node)
titles[section['name']] = section['subtitles']
return titles
def _check_titles(self, fname, titles):
expected_titles = ('Problem description', 'Proposed change',
'Implementation', 'Dependencies',
'Testing', 'Documentation impact',
'References')
self.assertEqual(
sorted(expected_titles),
sorted(titles.keys()),
"Expected titles not found in document %s" % fname)
proposed = 'Proposed change'
msg = 'Subtitle missing from %s' % fname
self.assertIn('Alternatives', titles[proposed], message=msg)
self.assertIn('Data model impact', titles[proposed], message=msg)
self.assertIn('REST API impact', titles[proposed], message=msg)
self.assertIn('Security impact', titles[proposed], message=msg)
self.assertIn('Notifications impact', titles[proposed], message=msg)
self.assertIn('Other end user impact', titles[proposed], message=msg)
self.assertIn('Performance impact', titles[proposed], message=msg)
self.assertIn('Other deployer impact', titles[proposed], message=msg)
self.assertIn('Developer impact', titles[proposed], message=msg)
impl = 'Implementation'
self.assertIn('Assignee(s)', titles[impl], message=msg)
self.assertIn('Work items', titles[impl], message=msg)
def _check_lines_wrapping(self, tpl, raw):
for i, line in enumerate(raw.split("\n")):
if "http://" in line or "https://" in line:
continue
# skip long table
elif "-+-" in line or " | " in line:
continue
self.assertTrue(
len(line) < 80,
msg="%s:%d: Line limited to a maximum of 79 characters." %
(tpl, i+1))
def _check_no_cr(self, tpl, raw):
matches = re.findall('\r', raw)
self.assertEqual(
len(matches), 0,
"Found %s literal carriage returns in file %s" %
(len(matches), tpl))
def _check_trailing_spaces(self, tpl, raw):
for i, line in enumerate(raw.split("\n")):
trailing_spaces = re.findall(" +$", line)
self.assertEqual(len(trailing_spaces),0,
"Found trailing spaces on line %s of %s" % (i+1, tpl))
def test_template(self):
files = ['specs/template.rst'] + glob.glob('specs/*/*')
for filename in files:
self.assertTrue(filename.endswith(".rst"),
"spec's file must uses 'rst' extension.")
with open(filename) as f:
data = f.read()
spec = docutils.core.publish_doctree(data, source_path=filename)
titles = self._get_titles(spec)
self._check_titles(filename, titles)
self._check_lines_wrapping(filename, data)
self._check_no_cr(filename, data)
self._check_trailing_spaces(filename, data)

21
tox.ini
View File

@ -1,21 +0,0 @@
[tox]
# Hold back to 1.4, since that's what's in Fedora 20 repos
# and we don't need anything newer for nova-specs tests
minversion = 3.1.1
envlist = docs,pep8
skipsdist = True
ignore_basepython_conflict = True
[testenv]
basepython = python3
usedevelop = True
setenv = VIRTUAL_ENV={envdir}
install_command = pip install -U {opts} {packages}
deps = -r{toxinidir}/requirements.txt
commands = stestr run --slowest {posargs}
[testenv:venv]
commands = {posargs}
[testenv:docs]
commands = python setup.py build_sphinx