Reorganize guidelines into guidelines directory and create
current_guideline that is softlink to the latest approved guideline.
The same with add-ons guidelines - they are moved to guidelines
directory within add-ons one and soft links are created pointing
to the latest guidelines for each add-on.
Also cleaned up some tooling that hardwired where guidelines lived.
Change-Id: I5ad4b91b1afb44a0a6987b339f7efba14f395302
Reorganize guidelines into previous_guideline directory
and current_guideline that is softlink to the latest approved guideline.
Cleaned up some tooling that hardwired where guidelines lived.
Change-Id: Ia6be9ca6326718488ee5668df3806da5f76dc456
Clouds are hard. English, doubly so. We have a longstanding
spelling error in the schema: "superceded" should be spelled
"superseded". This patch corrects it and updates all the documents
the typo was propagated to over time.
Change-Id: I72f24bf026919f4ccce0ce0c42e1c33c4042aca4
This patch should be approved only if and when the OpenStack
Foundation Board of Directors votes to approve the proposed 2017.01
Guideline. It does the following:
1.) Marks 2017.01 as "approved."
2.) Marks 2016.01 as "superceded."
3.) Adjusts the "replaces" target of next.json and it's target
approval date to reflect work begining on the next Guideline.
Note that the Board of Directors meeting list for 2017[1]
hasn't yet been set, so the date listed here is approximate.
[1]
https://wiki.openstack.org/wiki/Governance/Foundation#OpenStack_Board_of_Director_Meetings
Change-Id: I566e52fdc82af57a5bf24bfd3db5d10a26887127
Commit I8d9296f958e17d20a39a05800ac1808347d952bd, removed
this test as it is being merged to CRUD tests.
This patch flag the update metadata tests and also remove
from next guidlines.
Change-Id: Id9a53c13eb501a131d4bab48414dd8deecc17d90
Depends-On: I8d9296f958e17d20a39a05800ac1808347d952bd
Commit I8d9296f958e17d20a39a05800ac1808347d952bd , changed the
name of the tests on Tempest side. So this patch addes aliases.
Change-Id: I4707b4956c2b92dc76cf4720016e737edd21b8b3
Depends-On: I8d9296f958e17d20a39a05800ac1808347d952bd
Commit I8d9296f958e17d20a39a05800ac1808347d952bd , changed the
name of the tests on Tempest side. So this patch addes aliases.
Change-Id: I4d4432bc1d2fd8b1c68cd6d6183bc456ce3b4ff4
Depends-On: I8d9296f958e17d20a39a05800ac1808347d952bd
Since Ic19fd7bd4f0afd6a551133d7db96737487b52663 , the name has been
renamed on Tempest side. So this patch addes aliases.
Change-Id: Ie929b481d8989433922ab960d11590aba31e8374
Depends-On: Ic19fd7bd4f0afd6a551133d7db96737487b52663
Since I7c99f61dc54ef7c437c7e4574eba935d76ef9318, QA team has removed
test_get_invalid_volume_id from Tempest because the test is duplicated
with test_volume_get_nonexistent_volume_id.
Then this patch changes the compute-volume-get test.
Change-Id: I54f9924c1c007ea2d34b7ef502df8edf8032549e
Since Ieab05d793039cea1d4bb14e91ea0f4cbea3df02b , the name has been
renamed on Tempest side.
So this patch addes aliases.
Change-Id: Ibd8b3f2494046b194d77e9d563ebecef3a883b8f
Tempest patch [1] will move these tests to a new file. This will
change the fully qualified test names.
[1] https://review.openstack.org/#/c/332066/
Change-Id: I3262f7414629d460b3660191e7f0c04198e22dd2
In 2016.01 we introduced a new advisory test that involves creating
a Neutron port with a fixed IP address and then updating it to add
a second IP address to the same port. However, the nunber of fixed
IP addresses that can be assigned to a port is dependent on the
plugin that Neutron is using--and not all plugins support more than
one IP address per port (the maximum number of IP's per port is
configurable in neutron.conf and clouds whose drivers do not support
multiple fixed addresses per port simply set it to 1). Further,
some clouds limit the number of IP's per port for security reasons
to prevent malicious activity (e.g. spoofguard).
This patch adds an informational flag to the 2016.01 guideline
(where the test is advisory) and removes it from next.json.
Change-Id: Ie7d84d5d608bfc86501a5f7e097373c4565e4eb0
Certain Compute tests [1] require the fixed_network_name
configuration to be set or for a provider network to be
defined. Setting either of these triggers a code path
that uses the os-networks extension to look up the
network's id. If this extension is not enabled, then
any test that creates a server will fail. The credential
provider needs additional logic to handle the case
where this extension is not enabled.
[1] https://github.com/openstack/tempest/blob/master/tempest/api/compute/servers/test_list_server_filters.py#L270
Change-Id: I34b8cc50f35d7207d12c73f980966066eddbcf81
Related-Bug: #1592173
Related-Bug: #1600349
In 2016.01 we added a new advisory capability:
volumes-v2-snapshot-create-delete. One of the tests for that Capability
involves creating a volume, attaching it to an instance, and then
creating a snapshot of the volume while it's in-use. This is
potentially problematic for two reasons: first, it requires the use of
the force flag, which wasn't introduced until Liberty[1]. Kilo is still
covered until 2016.01 and 2016.08. Second, snapshotting an in-use
volume is an operation which may produce corrupted data (which isn't
always well understood [2]) and therefore seems like a thing that some
clouds may not wish to offer to their clients.
Since the force flag wasn't available in all the releases covered by
the 2016.01 and 2016.08 Guidelines, we should probably wait to include
this test until a later release. Waiting would also give us some time
to see if it's something clouds do indeed expose to their users...my
guess is that it is, but I could see why some clouds might not.
[1]
http://docs.openstack.org/admin-guide/blockstorage_volume_backups.html
[2]
https://support.rackspace.com/white-paper/allowing-snapshots-without-detaching-the-volumes/
Change-Id: I9dab755bde2f169d52503d47fcfca404b8777f77
Between 2015.05 and 2016.01 the compute-image flags were not
carried forward. This patch reconciles the lost flags and fixes
other errors in the existing guidelines, as well as next.
Partial-Bug: 1585773
Partial-Bug: 1599563
Change-Id: Ie405d3f24e0754a99f274aedcbd6b56adabaa5b6
Following the July 2016 board meeting, add minor update to guidelines
to move them to version 1.5 of the DefCore schema and add json
schema validation to gate job.
Change-Id: Iac667f8a87b50a15f4dfe13c1ef69efa61c10059
Fix errors in 2015.07.json, 2016.01.json, and next.json
guidelines. Fixes include removal of trailing commas,
changing empty lists to empty sets for consistency across
designated sections, and removal of duplicate test entries
from capabilities.
Co-Authored-By: Catherine Diep <cdiep@us.ibm.com>
Change-Id: Ib1a046b732f3807cabfd0c448a2cbf698a5cf44c
We recently added a field in Schema 1.4 to record a
"cutoff_score".[1] The field reflects the minimum score that a
Capability needs to achieve for a given Guideline in order to
warrant inclusion. The current draft Guideline, however, hasn't yet
been updated to include a cutoff score.
This patch adds a cutoff_score field to next.json and sets it to
a preliminary vale of 74 (which is indicative of the score we've
used in the past as a cutoff, but which can be changed by the
DefCore Committee any time between now and when the Guideline goes to
the Board for approval before it becomes binding). The patch also
updates the tabulate_scores.py script to read the cutoff_score from the
JSON document and adds an asterisk (*) to lines in the scoring file whose
totals meet or exceed the cutoff as a visual indicator of which
Capabilities have scored highly enough to warrant inclusion in the
Guideline. Verbage in the header of the scoring.txt file has also been
updated to reflect the meaning of the indicator.
The patch also adds the cutoff_score field to the 2016.01 document,
where it was inadvertently left off since the field was only added
to the schema in close proximity to the document being finalized for
Board approval.
[1] https://review.openstack.org/#/c/224868/
Change-Id: I73d08baae13f373c955ff94640fac43109125886
Flag tests that have been removed from the Tempest test
suite[1]. Capabilities are covered by other tests. Tests
to be permanently removed from future guidelines.
compute-auth-* capabilities removed because of empty
test sets.
[1] https://review.openstack.org/#/c/271467
Change-Id: Ib6b8ca03f39c7590163a9db710e5ca8aa373d659
Currently, each DefCore Guideline covers the most recent three
releases of the OpenStack Software at the time of the Guideline's
approval (which is generally 3-4 months after the release of a
new version of the software). This timing means that there's a
3-4 month period during which products built on the most recent
OpenStack release may not be able to apply for an OpenStack
Powered(TM) license agreement because the version of OpenStack on
which they are built isn't covered by a Board-approved Guideline.
Whereas the OpenStack community and Foundation have an expressed
interest in enabling users to adopt new software quickly, this
patch closes the 3-4 month window by amending the 2016.01
Guideline (and future Guidelines) to allow it to cover the most
recent version of the software starting on the day that the
software is released.
Note that because this change entails a modification to a Board-
approved Guideline document, this change should not merge until
after the Board of Directors has approved the change. The
matter will be discussed during the 2016-04-24 Board meeting.
Change-Id: I6db11762e48ff3297eb9f90c93fb68717b0f3d43
Guiding core priciple number 7 from the DefCore core
definition is [1]:
"7. Tests can be remotely or self-administered"
Tests that require multiple tenants and users
create a barrier to remote adminstration and
independent verification of test results. Requiring
users to enroll for multiple tenant-isolated accounts
on a public cloud creates a resource cost that
may be insurmountable for end users, especially
when comparing multiple public clouds.
In addition, tenant isolation is less a capability
for writing interoperable applications, and more
a security concern for cloud implementers. While
important, the goal of interopearbility is
orthgonal to the security goals of tenant isolation.
The flagged tests in this patch require multiple
tenant-isloated accounts, and are marked to
to be fixed where possible, or removed where not
possible.
This list was determined by successfully running
the defcore suite against a test cloud[2], then
re-running the tests configured with only one
account against the same test cloud[3].
At the January QA meetup, this test list was
refined by refactoring Server Negative tests
into two classes, one that requires multi-tenant
access, one one that does not[4]. This significantly
reduced the number of flagged tests.[5]
[1] https://github.com/openstack/defcore/blob/master/doc/source/process/CoreDefinition.rst
[2] https://refstack.openstack.org/#/results/0b8ac51d-3288-42d8-b276-4a54046e41bd
[3] https://refstack.openstack.org/#/results/01f29358-8870-4e15-8f7a-f6a082d44e29
[4] https://review.openstack.org/#/c/283819/
[5] https://etherpad.openstack.org/p/multi-tenant-defcore-tests
Change-Id: I5ec801ae48e8220abdaecebe43e77465684e8b5b
Add aliases to the "test_nework" tests to handle test path name
update as described in [1].
Add alias to the "test_create_router_setting_tenant_id" test due
to test name changed [2].
Add test python file name ("test_api_discovery") to the path of
Keystone API discorvery tests.
[1] https://review.openstack.org/#/c/153621/
[2] https://review.openstack.org/#/c/214987/
Change-Id: I482f0b1ad6b07549aba8f2ba13e4c1a32c70c72b
Remove test case
tempest.api.image.v2.test_images.ListImagesTest.test_index_no_params
from the advisory test list because this test case has been a required
test case since 2014.07.
Change-Id: I1c70f3ca6aa762340a2eeaa9dafa25ed85087e5d
The "identity-v2-tokens-create" capability was deprecated. As such,
its "required-since" field should be set to empty.
Change-Id: I84890d2e5011e4e4cf3142c9d590043ca2c891d2
This test has been flagged for some time due to a longstanding
bug [1] in Tempest (open since 2012). In discussion with QA, it appears
that the bug isn't going to be fixed and the test is likely to be
removed. Since the tests has been flagged for several cycles now
and the test itself is likely to be dropped, we should remove it
from the 2016.01 Guideline and future Guidelines.
[1] https://bugs.launchpad.net/tempest/+bug/1014647
Change-Id: Ia1adc9e6b3f81c65922b5167d99347d47bc95ac7
Since schema version 1.3 [1], a key of "flagged" should be used
for flagged tests.
[1] https://review.openstack.org/#/c/215312/
Change-Id: Ifeac35407a589ec413748edf26bf24c73fa5e39f
It's premature to include an Images v2 API import capability in DefCore
(even in an advisory capacity) given that image import functionality has
yet to be developed in Glance.
The change proposed by this patch is in accord with the TC resolution
"Compute Requirements for Images and Kernels" [0], which states that "An
OpenStack Powered Compute Cloud MUST allow end user Image Uploads".
(End user image upload is typically referred to in the industry as
"image import" in order to distinguish it from from the simple upload
facility used by trusted sources, for example, an OpenStack service like
Nova, or Glance administrators supplying public images for use in a
cloud). I interpret the TC resolution as a forward-looking statement
saying that the capability must be exposed as soon as it's available,
not that the capability must be made available immediately using the
(possibly improper) means currently at hand. In other words, if we want
to impose such a requirement on OpenStack deployers, then it's incumbent
upon OpenStack to provide a means by which this can responsibly be done
in an interoperable way. That means is scheduled to be delivered in
Glance in the Mitaka cycle.
The Glance image import spec [1] contains discussion and links to even
more discussion concerning this issue. There's also lively discussion
of the issue on the gerrit review for the spec. [2]
The images-v2-import capability should be resinstated as an advisory
capability after the Mitaka release. (The set of tests defining the
capability, however, will have to be modified.)
[0] https://governance.openstack.org/resolutions/20151211-bring-your-own-kernel.html
[1] http://specs.openstack.org/openstack/glance-specs/specs/mitaka/approved/image-import/image-import-refactor.html
[2] https://review.openstack.org/#/c/232371/
Change-Id: I98714d97a80a14b62b936ea51ccbde50117401b5
"test_rebuild_reboot_deleted_server" was renamed to
"test_rebuild_deleted_server" in patch [1] which was merged on
December 5, 2015.
[1] https://review.openstack.org/#/c/239710/
Change-Id: If64855268c167ae39661b6396d82658872309b92
At the DefCore workshop at the OpenStack Summit in Tokyo, we
discussed the need to migrate our next.json and in-review 2016.01
Guidelines to the new schema 1.4. The new schema removes a duplicate
source of truth around the status of each capability, provides a
target date, and an alias field for tests that have been renamed.
This patch migrates the two as-yet unapproved JSON docs to the
new schema. We've also discussed migrating older specs, which will
be done in a subsequent patch.
Change-Id: I7f0a370d4fd3e2650d70a29accdd17f4894e5a8f
Per discussion at Defcore working group @ Tokyo summit on Oct 28th
2016. Since various platforms do not yet support floating IPs, they
will not be an advisory guideline for 2016-01. Floating IPs will be
retained in json.next for future consideration.
Change-Id: I98beb54e8f7007b4b2d6c8c9b3cd9da87ef5a24a
Create a new Guideline for Board and community review
from the current .next guideline.
This patch will be merged at the Board meeting.
NOTE: it is REVIEW status, not approved. Approval
is scheduled for 2016.01.
Change-Id: I7f0152a80ae4a4619717b9ac89eb691a018847b7
ArkadyKanevsky