Reorganize guidelines into guidelines directory and create
current_guideline that is softlink to the latest approved guideline.
The same with add-ons guidelines - they are moved to guidelines
directory within add-ons one and soft links are created pointing
to the latest guidelines for each add-on.
Also cleaned up some tooling that hardwired where guidelines lived.
Change-Id: I5ad4b91b1afb44a0a6987b339f7efba14f395302
Reorganize guidelines into previous_guideline directory
and current_guideline that is softlink to the latest approved guideline.
Cleaned up some tooling that hardwired where guidelines lived.
Change-Id: Ia6be9ca6326718488ee5668df3806da5f76dc456
Clouds are hard. English, doubly so. We have a longstanding
spelling error in the schema: "superceded" should be spelled
"superseded". This patch corrects it and updates all the documents
the typo was propagated to over time.
Change-Id: I72f24bf026919f4ccce0ce0c42e1c33c4042aca4
This commit changes the 2015.07 Guideline from "approved"
to "superseded" and changes the 2016.08 Guideline from "draft"
to "approved" status. This patch should not be merged until
the Board of Directors has approved the 2016.08 Guideline. Voting
is planned for the August 23 Board meeting.
Note: this change also regenerates documentation for all other
superceded Guidelines.
Change-Id: I630978bdff9c166a4a3340f8c26bcdcbf7013cbf
Certain Compute tests [1] require the fixed_network_name
configuration to be set or for a provider network to be
defined. Setting either of these triggers a code path
that uses the os-networks extension to look up the
network's id. If this extension is not enabled, then
any test that creates a server will fail. The credential
provider needs additional logic to handle the case
where this extension is not enabled.
[1] https://github.com/openstack/tempest/blob/master/tempest/api/compute/servers/test_list_server_filters.py#L270
Change-Id: I34b8cc50f35d7207d12c73f980966066eddbcf81
Related-Bug: #1592173
Related-Bug: #1600349
Between 2015.05 and 2016.01 the compute-image flags were not
carried forward. This patch reconciles the lost flags and fixes
other errors in the existing guidelines, as well as next.
Partial-Bug: 1585773
Partial-Bug: 1599563
Change-Id: Ie405d3f24e0754a99f274aedcbd6b56adabaa5b6
Following the July 2016 board meeting, add minor update to guidelines
to move them to version 1.5 of the DefCore schema and add json
schema validation to gate job.
Change-Id: Iac667f8a87b50a15f4dfe13c1ef69efa61c10059
Fix errors in 2015.07.json, 2016.01.json, and next.json
guidelines. Fixes include removal of trailing commas,
changing empty lists to empty sets for consistency across
designated sections, and removal of duplicate test entries
from capabilities.
Co-Authored-By: Catherine Diep <cdiep@us.ibm.com>
Change-Id: Ib1a046b732f3807cabfd0c448a2cbf698a5cf44c
Flag tests that have been removed from the Tempest test
suite[1]. Capabilities are covered by other tests. Tests
to be permanently removed from future guidelines.
compute-auth-* capabilities removed because of empty
test sets.
[1] https://review.openstack.org/#/c/271467
Change-Id: Ib6b8ca03f39c7590163a9db710e5ca8aa373d659
Guiding core priciple number 7 from the DefCore core
definition is [1]:
"7. Tests can be remotely or self-administered"
Tests that require multiple tenants and users
create a barrier to remote adminstration and
independent verification of test results. Requiring
users to enroll for multiple tenant-isolated accounts
on a public cloud creates a resource cost that
may be insurmountable for end users, especially
when comparing multiple public clouds.
In addition, tenant isolation is less a capability
for writing interoperable applications, and more
a security concern for cloud implementers. While
important, the goal of interopearbility is
orthgonal to the security goals of tenant isolation.
The flagged tests in this patch require multiple
tenant-isloated accounts, and are marked to
to be fixed where possible, or removed where not
possible.
This list was determined by successfully running
the defcore suite against a test cloud[2], then
re-running the tests configured with only one
account against the same test cloud[3].
At the January QA meetup, this test list was
refined by refactoring Server Negative tests
into two classes, one that requires multi-tenant
access, one one that does not[4]. This significantly
reduced the number of flagged tests.[5]
[1] https://github.com/openstack/defcore/blob/master/doc/source/process/CoreDefinition.rst
[2] https://refstack.openstack.org/#/results/0b8ac51d-3288-42d8-b276-4a54046e41bd
[3] https://refstack.openstack.org/#/results/01f29358-8870-4e15-8f7a-f6a082d44e29
[4] https://review.openstack.org/#/c/283819/
[5] https://etherpad.openstack.org/p/multi-tenant-defcore-tests
Change-Id: I5ec801ae48e8220abdaecebe43e77465684e8b5b
"test_rebuild_reboot_deleted_server" was renamed to
"test_rebuild_deleted_server" in patch [1] which was merged on
December 5, 2015.
[1] https://review.openstack.org/#/c/239710/
Change-Id: If64855268c167ae39661b6396d82658872309b92
At the DefCore workshop at the OpenStack Summit in Tokyo, we
discussed the need to migrate our older specs to a newer version
of the schema. Doing so would make it easier to deal with tests
that have been renamed, and would mean fewer schemas for RefStack
to have to parse.
This patch migrates 2015.07 from schema 1.3 to 1.4. Older Guidelines
are currently in schemas 1.2 or older and are therefore a bit more
complex to migrate--so those will be dealt with in a separate patch.
Change-Id: I2090166eefeb9157636ecb34a78b061422662c60
Chagne password is not widely supported across hypervisors
(in fact, it only currently works with Xen [1]) and therefore
fails to meet DefCore criteria. This test has also been flagged
in previous Guidelines [2] and we've discussed removing it entirely
since it's extremely unlikely that all other backends will ever
implement this capability.
This patch carries forward the flag from previous Guidelines and
removes the test completely from .next.json so we don't have to
re-flag it again in subsequent Guidelines.
[1]
http://docs.openstack.org/developer/nova/support-matrix.html#operation_set_admin_password
[2] https://review.openstack.org/#/c/196153/3
Change-Id: I13ed481324cf1b0b62c7c02886bdf02906550cbf
In the 2015.07 spec we moved to schema 1.3 [1] which changes the structure
of flags in the JSON documents. Currently a few tests with flags
have a key of "flag" where schema 1.3 dictates they have a key of
"flagged". This patch corrects those sections.
[1]
http://git.openstack.org/cgit/openstack/defcore/tree/doc/source/schema/1.3.rst#n75
Change-Id: Ic4fbd81662f8ff63abccca09b3f4a0e778cd5234
This change was larger than expected because the
capabilities listed in the components did not match the
actual capabilities.
Change-Id: I085d92ef11c399b487e0a827cde00e83df14fbe4
ArkadyKanevsky