diff --git a/doc/source/index.rst b/doc/source/index.rst index 96e8eb5..224e4d6 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -4,6 +4,14 @@ Congress Project Specifications =============================== +Pike approved specs: + +.. toctree:: + :glob: + :maxdepth: 1 + + specs/pike/* + Newton approved specs: .. toctree:: diff --git a/specs/pike/policy-library.rst b/specs/pike/policy-library.rst new file mode 100644 index 0000000..01b376f --- /dev/null +++ b/specs/pike/policy-library.rst @@ -0,0 +1,645 @@ +.. + This work is licensed under a Creative Commons Attribution 3.0 Unported + License. + + http://creativecommons.org/licenses/by/3.0/legalcode + +========================================== +Policy Library +========================================== + +Include the URL of your launchpad blueprint: + +https://blueprints.launchpad.net/congress/+spec/policy-library + +When Congress is first deployed, it does not perform any monitoring or +enforcement because there are no policies specified. It takes significant +investment in time and energy to learn the policy language and write the +desired policies before Congress provides any value. + + +Problem description +=================== + +To help deployers and administrators quickly get started and see value, +we propose to include with Congress a library of useful policies that a +deployer or administrator can easily enable, disable, and customize. + + +Proposed change +=============== + + +Overview +------------ + +The proposed changes come in two main pieces, the new policy library and the +changes to existing Congress server functionalities to support the use of +policies from the policy library. + +The policy library +^^^^^^^^^^^^^^^^^^^^^ + +Introduce a new concept of the policy library in Congress server, +where policies are retained and manipulated but are not used by the policy +engine (for monitoring or enforcement). +Include with Congress server a collection of useful policies, +each in a YAML file. These policies automatically populate the policies +library. + +To make it easy for us to populate the library and for admins to modify/create +their own library, the API/CLI/GUI should support full CRUD into the library. + +* Congress server can have new config option that points to a directory of + policies to load into the library. + + * Once the library is initialized from the files, restart of Congress server + do not re-initialize the policy library from the files + (empty library is assumed to be un-initialized). A new interface + allows an administrator to trigger re-initialization from files. + + .. note:: The policy library is persisted in the database and has a + separate existence in Congress server apart from the files in the policy + library directory. See also discussion of the alternative model + _`Policy library directory files as truth`. + + * Future feature: reloads changed files automatically into library + + * Will not update policies imported from library into policy engine + * Open question: + Should changes to the policy library through the API sync back to the + files in the policy library directory? + * Care must be taken to consider concurrent changes and potential conflicts + from multiple congress server instances. + +.. warning:: When Congress server policy engine service is replicated, + different server instances must have the same + policy library files. If not, the library will be populated based on the + an arbitrary combination of the policy files on the different server + instances. + +* Installation + + * Standalone: installer scripts copy the default library directory from the + Congress git repo into /etc/congress/. + * Devstack: same as standalone + + +Support for using policies from the library +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +We propose to add all the functionalities required to support the following +workflows for activating a policy from the policy library (i.e., creating an +(active) policy in the policy engine by importing a (inactive) policy from the +policy library). + +* Activating a policy from library using CLI/API. + + #. Administrator inspects policies in library to find one they want to use. + #. Administrator downloads the relevant policy, in YAML or JSON. + #. Administrator modifies YAML/JSON encoding and uses it as the payload to + create a new policy. + +* Activating a policy from library using GUI. + + #. Administrator peruses policies in library + #. Administrator clicks on an interesting policy, which brings up a textbox + (or more sophisticated policy editor) to customize. + #. Administrator clicks on a ‘Create’ button that creates that policy. + +Below are the specific functionalities needed to support these workflows. + +* A backward compatible change to the policies POST API to allow for a 'rules' + key in the JSON/YAML payload body. + +* A new optional parameter to the ``policy create`` CLI command that allows + the administrator to import and activate a policy from a YAML file. + +* A new optional parameter to the ``policy create`` CLI command that allows + the administrator to import and activate a policy directly from the policy + library without saving to file. + + .. note:: The policy engine rule insert code would require some careful + reworking to ensure policy creation with rules is transactional. That is, + if insertion fails at the third rule (say because of cross-policy + recursion), the first and second rule insertions are undone without having + caused any effect (triggered actions or altered query answers). + + Three main things are needed. + + * Write to database only after the insertion of all the policy rules have + succeeded in the policy engine. + + * Trigger evaluation must be delayed until after all the rules are inserted + successfully. + + * The policy engine must be prevented from answering queries while in the + middle of processing a batch of rule insertions. This can be done either + with locks or by making sure the greenthread executing the batch rule + insertion does not yield. + +* A new page in the Horizon GUI where an administrator can perform the + following tasks. + + - browse and inspect policies in the policy library + - active (i.e., import to policy engine) a policy from the policy library + - customize a policy in the policy library prior to import + + +Alternatives +------------ + +This section discusses several alternative designs and implementation to +various aspects of this proposal. + +Policy library strictly external to Congress +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +* No concept of library in API/data model. The policy library is simply a + directory of files the administrator can import into the policy engine. + + * Least development effort. + + * Not as usable. Administrator cannot use the existing Congress interface + (CLI/GUI/API) to browse, inspect, customize, and activate policies in the + policy library. + + * This version is realized by a subset of the changes proposed in the spec. + This subset should be prioritized in the development to realize the minimal + viable functionality described here. + +Policy library directory files as truth +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +* Instead of database, use the policy files in the policy library directory as + source of truth. + + * Complex in error-prone in a setup with multiple policy engines. + + * Different model from all the other persisted data in Congress server, + adding more complexity for developers. + +Library rules as disabled policy engine rules in the same database table +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +* Instead of storing the rules in the library policy in a separate logical + repository as well as a separate database table from the active rules in the + rule engine, store all the rules as rules in the rule engine, with an + additional flag indicating which ones are active. + + * Complicates the existing data model for rules in the policy engine. + + * What is the benefit? + + +Policy +------ + +Not applicable. + +Policy actions +-------------- + +Not applicable + + +Data sources +------------ + +Not applicable. + + +Data model impact +----------------- + +No impact on existing tables. + +One new database tables are needed: + +* A ``library_policies`` table to store the policies in the policies library. + Below are the columns. + + * ``name = + sa.Column(sa.String(255), nullable=False, unique=True, primary_key=True)`` + * ``abbreviation = sa.Column(sa.String(5), nullable=False)`` + * ``description = sa.Column(sa.Text(), nullable=False)`` + * ``owner = sa.Column(sa.Text(), nullable=False)`` + * ``kind = sa.Column(sa.Text(), nullable=False)`` + * ``rules = sa.Column(sa.Text(), nullable=False)`` + + .. note:: Because the rules in the library are accessed at the granularity of + one entire policy, we can improve simplicity and performance by storing the + rules as a text blob instead of as individual rows. + + .. warning:: the TEXT type is limited to 2^16 - 1 bytes in length in MySQL. + Very large policies beyond this size are not supported. If this limit turns + out to be too restrictive, MEDIUMTEXT or LONGTEXT type can be forced by + specifying a longer length in the SQL Alchemy column declaration, at the + cost of losing SQLite support (primarily used for unit testing). + +Empty tables will be created using sqlalchemy migration scripts. Tables will be +populated by Congress server from policy files in the policy libraries folder. + +REST API impact +--------------- + +Policy library methods +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +* Re-initialize + + * Re-initialize policy library from the files in the policy library directory + as set in configuration. + + .. warning:: This operation destroys the current content of the policy + library. + + * Method type: PUT + + * Normal http response code(s): 200 + + * Expected error http response code(s): 401 + + * ``/v1/library`` + + * Parameters which can be passed via the url: None + + * JSON schema definition for the body data if allowed: Not allowed + + * JSON schema definition for the response data if any: None + +* List policies + + * List metadata for all policies in the library (id, name, desc, number of + rules) + + * Method type: GET + + * Normal http response code(s): 200 + + * Expected error http response code(s): None + + * ``/v1/library`` + + * Parameters which can be passed via the url: None + + * JSON schema definition for the body data if allowed: Not allowed + + * JSON schema definition for the response data if any: + + :: + + - !Type: + title: collection of PolicyProperties + type: array + items: + type: object + properties: + name: + title: Policy unique name + type: string + required: true + owner_id: + title: Policy owner + type: string + required: true + description: + title: Policy description + type: string + required: true + kind: + title: Policy kind + type: string + required: true + abbreviation: + title: Policy name abbreviation + type: string + required: false + + +* Show policy + + * Show specified library policy (in specified format) + + * Method type: GET + + * Normal http response code(s): 200 + + * Expected error http response code(s): 401, 404 + + * ``/v1/library/`` + + * Parameters which can be passed via the url: None + + * JSON schema definition for the body data if allowed: Not allowed + + * JSON schema definition for the response data if any: + `Policy full schema`_. + +* Create policy + + * Create a new policy in library + + * Method type: POST + + * Normal http response code(s): 200 + + * Expected error http response code(s): 400, 401, 409 + + * ``/v1/library`` + + * Parameters which can be passed via the url: None + + * JSON schema definition for the body data if allowed: + `Policy full schema`_. + + * JSON schema definition for the response data if any: + `Policy metadata schema without UUID`_. + +* Update policy + + * Update a policy in library + + * Method type: PUT + + * Normal http response code(s): 200 + + * Expected error http response code(s): 400, 401, 404, 409 + + * ``/v1/library/`` + + * Parameters which can be passed via the url: + + * format: {json, yaml} (default is json) + + * JSON schema definition for the body data if allowed: + `Policy full schema`_. + + * JSON schema definition for the response data if any: + `Policy metadata schema without UUID`_. + +* Delete policy + + * Delete policy from library + + * Method type: DELETE + + * Normal http response code(s): 200 + + * Expected error http response code(s): 401, 404 + + * ``/v1/library/`` + + * Parameters which can be passed via the url: None + + * JSON schema definition for the body data if allowed: Not allowed + + * JSON schema definition for the response data if any: None + + +Policy engine methods +^^^^^^^^^^^^^^^^^^^^^^^^ + +* Create policy + + * Create a policy in policy engine (either from input data or from library) + + * Method type: POST + + * Normal http response code(s): 200 + + * Expected error http response code(s): 400, 401, 404, 409 + + * ``/v1/policies`` + + * Parameters which can be passed via the url: + + * library_policy=. This creates the policy + directly from the library policy, without needing body data. A request + where both this parameter and a body data are specified should result in + error 400. + + * JSON schema definition for the body data if allowed: `Policy full schema`_. + + * JSON schema definition for the response data if any: + `Policy metadata schema with UUID`_. + + +Common JSON Schemas +^^^^^^^^^^^^^^^^^^^^^^^^^ + + +Policy metadata schema without UUID +++++++++++++++++++++++++++++++++++++++++++++ + +:: + + - !Type: + id: PolicyProperties + title: Policy Properties + type: object + properties: + name: + title: Policy unique name + type: string + required: true + owner_id: + title: Policy owner + type: string + required: true + description: + title: Policy description + type: string + required: true + kind: + title: Policy kind + type: string + required: true + abbreviation: + title: Policy name abbreviation + type: string + required: false + +.. note:: UUIDs are not used for policies in the policy library. The reason + is gets confusing if the UUID is changed when the policy is activated, + but also problematic if the UUID is retained in the activated policy in + policy engine, leading to two different entities sharing the same UUID. + + +Policy metadata schema with UUID +++++++++++++++++++++++++++++++++++++++++++++ + +:: + + - !Type: + id: PolicyProperties + title: Policy Properties + type: object + properties: + id: + title: UUID for the policy + type: string + required: true + name: + title: Policy unique name + type: string + required: true + owner_id: + title: Policy owner + type: string + required: true + description: + title: Policy description + type: string + required: true + kind: + title: Policy kind + type: string + required: true + abbreviation: + title: Policy name abbreviation + type: string + required: false + + +Policy full schema +++++++++++++++++++++++++++++++++++++++++++ + +.. note:: It is a design decision that rules in a policy library do not + have IDs and cannot be referred to individually. Policies in the library + are created, updated, or deleted at the level of the whole policy. + +:: + + - !Type: + id: PolicyProperties + title: Policy Properties + type: object + properties: + name: + title: Policy unique name + type: string + required: true + owner_id: + title: Policy owner + type: string + required: true + description: + title: Policy description + type: string + required: true + kind: + title: Policy kind + type: string + required: true + abbreviation: + title: Policy name abbreviation + type: string + required: false + rules: + title: collection of rules + type: array + required: false + items: + type: object + properties: + id: PolicyRule + title: Policy rule + type: object + properties: + rule: + title: Rule definition following policy grammar + type: string + required: true + name: + title: User-friendly name + type: string + required: false + comment: + title: User-friendly comment + type: string + required: false + + +Security impact +--------------- + +In general, no major security impact is expected. However, below are some +security considerations. + +* If policy library changes are sync'ed back to the policy library directory on + the server, then an attacker who gains administrative privilege to Congress + can also create and modify files in the policy library directory of the + Congress servers. + +* The ability to submit multiple rules in a policy creation API can increase + the potential for resource exhaustion attacks. + +Notifications impact +-------------------- + +No impact + +Other end user impact +--------------------- + +No impact to existing work flows. + +Performance impact +------------------ + +Minimal performance impact expected. Care must be taken to limit the +performance impact of the feature to automatically detect changes in +the policy library folder and update the policy library accordingly. + +Other deployer impact +--------------------- + +There is minimal negative impact on deployers. + +* No existing database table schemas are changed. + +* New database tables are used, which would be added by alembic migration + scripts. + +Developer impact +---------------- + +Minimal developer impact. + + +Implementation +============== + +Assignee(s) +----------- + +Assignment to be done on launchpad. + +Work items +---------- + +To be organized on launchpad. + + +Dependencies +============ + +No new dependencies. + + +Testing +======= + +In addition to unit tests, straight-forward additions to the tests in +congress_tempest_tests/tests/scenarios/test_congress_basic_ops.py would +suffice. + + +Documentation impact +==================== + +Minimal impact. + + +References +========== + +No references.