22 KiB
Scenario tests design
https://blueprints.launchpad.net/manila/+spec/scenario-tests
Manila has good functional test coverage of its features, but narrow coverage of scenarios. And with addition of new features this coverage becomes even less.
Problem description
Drivers often lack features or implement them incorrectly and it's very difficult for humans to notice this in code review. Tests ensure that drivers which lack features must explicitly skip tests to get a passing result, and the non-skipped tests ensure that the drivers implement features correctly. Documentation for how features should work is often sparse and driver authors don't always know what is the correct behavior when implementing a driver feature. Tests allow driver authors to code the feature to pass the test, which simplifies the driver author's job.
Use Cases
Consider the case when new manila deployment becomes ready. And one wants to verify that main user use cases work in general. Having automatic scenario tests one could test his deployment much faster than manually, as it is now. Also, it could be used in CI systems to test share drivers continuously.
Proposed change
It is proposed to get agreement on scenario tests design (this spec) and implement them in manila plugin for tempest. After that these tests could be used in CI systems as well as on customer deployments. This spec covers only existing features in manila as of last available (Newton) release. All newly added features should be covered separately.
Prerequisites for scenarios:
- Depending on share driver mode, it can be required to create share-network with or without security-services.
- Depending on protocol, its versions and access type, "mount" operations should be defined explicitly. Scenario tests assume it as predefined and known. Hence, scenarios do not include difference between access type (IP, User, Cert). Due to this, scenario tests should be data driven by "access_type", "access_proto", "access_level" and "mount command with all expected options".
- Bold texts depend on specific implementations and can be unsupported by share backends.
- Text in brackets depend on share driver configuration and may be optional.
- All user machines are separate VMs from share-nodes. They should have network connectivity with share host. User VMs should be built with image that has shared file systems clients.
- Share host should have open ports for SSH protocol and protocols of shared file systems.
Here is list of scenarios for implementation:
- 1) Share access and file operations
- 2) Share access with multiple guests
- 3) Relationships between source shares and child shares
- 4) Create/extend share and write data
- 5) Create/shrink share and write data
- 6) Create/manage share and write data
- 7) Create/manage share and snapshot and write data
- 8) Replicate ‘writable’ share and write data
- 9) Replicate and promote ‘readable’ share and write data
- 10) Replicate and promote ‘dr’ share and write data
- 11) Get a snapshot of a replicated share
- 12) Migrate share and write data
_1) Share access and file operations
Driver mode: any
Involved APIs:
- [create share network]
- [create share type]
- create share
- allow share access
- deny share access
- delete share
| Step | Action | Result |
|---|---|---|
| 1 | Create user VM (UVM) | ok, created |
| 2 | Create share (S) | ok, created |
| 3 | SSH to UVM | ok, connected |
| 4 | Try mount S to UVM | fail, access denied |
| 5 | Provide RO access to S | ok, provided |
| 6 | Try mount S to UVM | ok, mounted |
| 7 | Try create files on S | fail, access denied |
| 8 | Unmount S from UVM | ok, unmounted |
| 9 | Remove RO access from S | ok, removed |
| 10 | Try mount S to UVM | fail, access denied |
| 11 | Provide RW access to S | ok, provided |
| 12 | Try mount S to UVM | ok, mounted |
| 13 | Try write files to S | ok, written |
| 14 | Try read files from S | ok, read |
| 15 | Try delete files on S | ok, deleted |
| 16 | Unmount S from UVM | ok, unmounted |
| 17 | Delete S | ok, deleted |
| 18 | Try mount S | fail, not found |
| 19 | Delete UVM | ok, deleted |
_2) Share access with multiple guests
Driver mode: any
Involved APIs:
- [create share network]
- [create share type]
- create share
- allow share access
- delete share
| Step | Action | Result |
|---|---|---|
| 1 | Create UVM1 | ok, created |
| 2 | Create UVM2 | ok, created |
| 3 | Create share S | ok, created |
| 4 | Add RW access to UVM1 | ok, added |
| 5 | SSH to UVM1 | ok, connected |
| 6 | Try mount S from UVM1 | ok, mounted |
| 7 | SSH to UVM2 | ok, connected |
| 8 | Try mount S from UVM2 | fail, access denied |
| 9 | Add RW access for UVM2 | ok, added |
| 10 | Try mount S from UVM2 | ok, two VMs have it mounted at once. |
| 11 | Create test file in mounted share from UVM1 | ok, created. Available from UVM2 |
| 12 | Write data to test file from UVM2 | ok, written. Available from UVM1 too |
| 13 | Unmount S on UVM1 | ok, unmounted |
| 14 | Unmount S on UVM2 | ok, unmounted |
| 15 | Delete UVM1 | ok, deleted |
| 16 | Delete UVM2 | ok, deleted |
| 17 | Delete S | ok, deleted |
_3) Relationships between source shares and child shares
Driver mode: any
Involved APIs:
- [create share network]
- [create share type]
- create share
- allow share access
- create share snapshot
- delete share snapshot
- delete share
| Step | Action | Result |
|---|---|---|
| 1 | Create UVM | ok, created |
| 2 | Create share S1 | ok, created |
| 3 | Provide RW access to S1 | ok, provided |
| 4 | SSH to UVM | ok, connected |
| 5 | Try mount S1 to UVM | ok, mounted |
| 6 | Create "file1" | ok, created |
| 7 | Create snapshot SS1 from S1 | ok, created |
| 8 | Create "file2" in share S1 | ok, created. We expect that snapshot will not contain any data created after snapshot creation. |
| 9 | Create share S2 from SS1 | ok, created |
| 10 | Try mount S2 | fail, access denied. We test that child share did not get access rules from parent share. |
| 11 | Provide RW access to S2 | ok, provided |
| 12 | Try mount S2 | ok, mounted |
| 13 | List files on S2 | only "file1" exists |
| 14 | Create file3 on S2 | ok, file created |
| 15 | List files on S1 | two files exist - "file1" and "file2" |
| 16 | List files on S2 | two files exist - "file1" and "file3" |
| 17 | Unmount S1 and S2 | ok, unmounted |
| 18 | Delete S2, then SS1, then S1, then UVM | ok, all deleted |
_4) Create/extend share and write data
Driver mode: any
Involved APIs:
- [create share network]
- [create share type]
- create share
- allow share access
- extend share
- delete share
| Step | Action | Result |
|---|---|---|
| 1 | Create UVM | ok, created |
| 2 | Create share S1 of 1Gb size | ok, created |
| 3 | Provide RW access to S1 | ok, provided |
| 4 | SSH to UVM | ok, connected |
| 5 | Try mount S1 to UVM | ok, mounted |
| 6 | Create "file1" | ok, created |
| 7 | Fill file1 with data as possible | size of a file does not exceed share size quota |
| 8 | Extend share S1 to 2Gb | ok, extended |
| 9 | Write additional data to file1 | data written, size of a file does not exceed new share size quota and it is more than old one |
| 10 | Unmount S1 | ok, unmounted |
| 11 | Delete share S1 | ok, deleted |
| 12 | Delete UVM | ok, deleted |
_5) Create/shrink share and write data
Driver mode: any
Involved APIs:
- [create share network]
- [create share type]
- create share
- allow share access
- shrink share
- delete share
| Step | Action | Result |
|---|---|---|
| 1 | Create UVM | ok, created |
| 2 | Create share S1 of 2Gb size | ok, created |
| 3 | Provide RW access to S1 | ok, provided |
| 4 | SSH to UVM | ok, connected |
| 5 | Try mount S1 to UVM | ok, mounted |
| 6 | Write some data for 2 Gb | ok, created |
| 7 | Fill file1 with data as possible | size of a file does not exceed share size quota |
| 8 | Try shrink share S1 to 1Gb | fail, possible data loss exception |
| 9 | Delete data for amount of 1 Gb | data deleted |
| 10 | Shrink share S1 to 1Gb | ok, shrinked |
| 11 | Try write data more than new size of 1 Gb | fail, cannot write |
| 12 | Unmount S1 | ok, unmounted |
| 13 | Delete share S1 | ok, deleted |
| 14 | Delete UVM | ok, deleted |
_6) Create/manage share and write data
Driver mode: DHSS=False only
Involved APIs:
- [create share type]
- create share
- allow share access
- manage share
- unmanage share
- delete share
| Step | Action | Result |
|---|---|---|
| 1 | Create UVM | ok, created |
| 2 | Create share S1 of 1Gb size | ok, created |
| 3 | Provide RW access to S1 | ok, provided |
| 4 | SSH to UVM | ok, connected |
| 5 | Try mount S1 to UVM | ok, mounted |
| 6 | Write some data | ok, written |
| 7 | Unmount S1 | ok, unmounted |
| 8 | Unmanage share | ok, unmanaged |
| 9 | Try get share S1 | fail, 404 code in response |
| 10 | Manage share S1 | ok, managed. |
| 11 | Provide RW access to S1 again | ok, provided. We make sure that even if rule has existed on backend, we do not fail if explicitly try add it again after ‘manage’ operation. |
| 12 | Try mount S1 to UVM | ok, mounted. Previously created data still there. |
| 13 | Unmount S1 | ok, unmounted |
| 14 | Delete share | ok, deleted |
| 15 | Try manage share again | fail, resource not found |
| 16 | Delete UVM | ok, deleted |
_7) Create/manage share and snapshot and write data
Driver mode: DHSS=False only
Involved APIs:
- [create share type]
- create share
- allow share access
- create snapshot
- manage share
- unmanage share
- manage snapshot
- unmanage snapshot
- delete snapshot
- delete share
| Step | Action | Result |
|---|---|---|
| 1 | Create UVM | ok, created |
| 2 | Create share S1 of 1Gb size | ok, created |
| 3 | Provide RW access to S1 | ok, provided |
| 4 | SSH to UVM | ok, connected |
| 5 | Try mount S1 to UVM | ok, mounted |
| 6 | Write some data | ok, written |
| 7 | Create snapshot SS1 | ok, created |
| 8 | Unmanage snapshot SS1 | ok, unmanaged |
| 9 | Unmanage share S1 | ok, unmanaged |
| 10 | Try get share S1 | fail, 404 code in response |
| 11 | Manage share S1 | ok, managed. |
| 12 | Provide RW access to S1 again | ok, provided. We make sure that even if rule has existed on backend, we do not fail if explicitly try add it again after ‘manage’ operation. |
| 13 | Try mount S1 to UVM | ok, mounted. Previously created data still there. |
| 14 | Manage snapshot SS1 | ok, managed |
| 15 | Delete snapshot SS1 | ok, deleted |
| 16 | Unmount S1 | ok, unmounted |
| 17 | Delete share S1 | ok, deleted |
| 18 | Try manage share S1 again as S2, this should fail asynchronously since the resource is gone | S2 has a status set to 'error' |
| 19 | Delete S2 | ok, deleted |
| 20 | Delete UVM | ok, deleted |
_8) Replicate ‘writable’ share and write data
Driver mode: DHSS=false only
Involved APIs:
- [create share type]
- create share
- allow share access
- create replica
- delete replica
- delete share
| Step | Action | Result |
|---|---|---|
| 1 | Create UVM1 | ok, created |
| 2 | Create share S1-R1 of 1Gb size | ok, created |
| 3 | Provide RW access to S1-R1 | ok, provided |
| 4 | SSH to UVM1 | ok, connected |
| 5 | Try mount S1-R1 to UVM1 | ok, mounted |
| 6 | Create file1 | ok, created |
| 7 | Create share replica S1-R2 | ok, created |
| 8 | Create UVM2 | ok, created |
| 9 | SSH to UVM2 | ok, connected |
| 10 | Try mount S1-R2 to UVM2 | fail, access denied |
| 11 | Try mount S1-R2 to UVM1 | ok,mounted. Same files exist. |
| 12 | Provide RW access to S1-R2 | ok, provided |
| 13 | Try mount S1-R2 to UVM2 | ok, mounted |
| 14 | Create file2 in S1-R2 | ok, created. S1-R1 has both files too. |
| 15 | Create file3 in S1-R1 | ok, created. Both replicas have three created files. |
| 16 | Unmount both replicas | ok, unmounted |
| 17 | Delete original replica S1-R1 | ok, deleted. second and the only replica now still exists and has all files that were created. |
| 18 | Delete share S1 | ok, deleted |
_9) Replicate and promote ‘readable’ share and write data
Driver mode: DHSS=false only
Involved APIs:
- [create share type]
- create share
- allow share access
- create replica
- promote replica
- delete replica
- delete share
| Step | Action | Result |
|---|---|---|
| 1 | Create UVM1 | ok, created |
| 2 | Create share S1-R1 of 1Gb size | ok, created |
| 3 | Provide RW access to S1-R1 | ok, provided |
| 4 | SSH to UVM1 | ok, connected |
| 5 | Try mount S1-R1 to UVM1 | ok, mounted |
| 6 | Create file1 | ok, created |
| 7 | Create share replica S1-R2 | ok, created |
| 8 | Create UVM2 | ok, created |
| 9 | SSH to UVM2 | ok, connected |
| 10 | Try mount S1-R2 to UVM2 | fail, access denied |
| 11 | Try mount S1-R2 to UVM1 | ok, mounted. Same files exist. |
| 12 | Provide RW access to S1-R2 | ok, provided |
| 13 | Try mount S1-R2 to UVM2 | ok, mounted |
| 14 | Try create some file in S1-R2 | fail, filesystem is RO only. |
| 15 | Create file2 in S1-R1 | ok, created. Both replicas have two created files. |
| 16 | Promote S1-R2 to active | ok, promoted. S1-R1 became RO. |
| 17 | Create file3 in S1-R2 | ok, created. S1-R1 has all files too. |
| 18 | Try create some file in S1-R1 | fail, filesystem is RO |
| 19 | Unmount both replicas | ok, unmounted |
| 20 | Delete original (now RO) replica S1-R1 | ok, deleted. Second and the only replica (active) now still exists and has all files that were created. |
| 21 | Delete share S1 | ok, deleted |
_10) Replicate and promote ‘dr’ share and write data
Driver mode: DHSS=false only
Involved APIs:
- [create share type]
- create share
- allow share access
- create replica
- promote replica
- delete replica
- delete share
| Step | Action | Result |
|---|---|---|
| 1 | Create UVM | ok, created |
| 2 | Create share S1-R1 | ok, created |
| 3 | Provide RW access to S1-R1 | ok, provided |
| 4 | SSH to UVM | ok, connected |
| 5 | Try mount S1-R1 to UVM | ok, mounted |
| 6 | Create file1 | ok, created |
| 7 | Create share replica S1-R2 | ok, created |
| 8 | Unmount S1-R1 | ok, unmounted |
| 9 | Promote S1-R2 | ok, promoted. S1-R1 became ‘dr’-only |
| 10 | Try mount S1-R2 to UVM | ok, mounted. ‘file1’ exists |
| 11 | Create file2 | ok, created |
| 12 | Unmount S1-R2 | ok, unmounted |
| 13 | Promote S1-R1 | ok, promoted. S1-R2 became ‘dr’-only. |
| 14 | Try mount S1-R1 to UVM | ok, mounted. Files ‘file1’ and ‘file2’ exist. |
| 15 | Unmount S1-R1 | ok, unmounted |
| 16 | Delete S1-R2 (current secondary) | ok, deleted. |
| 17 | Delete share | ok, deleted |
_11) Get a snapshot of a replicated share
Driver mode: DHSS=false only
Involved APIs:
- [create share type]
- create share
- allow share access
- create snapshot
- create share from snapshot
- create replica
- promote replica
- delete replica
- delete snapshot
- delete share
| Step | Action | Result |
|---|---|---|
| 1 | Create UVM | ok, created |
| 2 | Create share S1-R1 | ok, created |
| 3 | Provide RW access to S1-R1 | ok, provided |
| 4 | SSH to UVM | ok, connected |
| 5 | Try mount S1-R1 to UVM | ok, mounted |
| 6 | Create ‘file1’ | ok, created |
| 7 | Create snapshot SS1 | ok, created |
| 8 | Create replica S1-R2 | ok, created |
| 9 | Create ‘file2’ | ok, created |
| 10 | Create snapshot SS2 | ok, created |
| 11 | Unmount S1-R1 | ok, unmounted |
| 12 | Promote S1-R2 (For non-’writable’ replication types) | ok, promoted |
| 13 | Try mount S1-R2 to UVM | ok, mounted |
| 15 | Delete S1-R1 | ok, deleted |
| 16 | Create share S2 from SS2 | ok, created |
| 17 | Provide RW access to S2 | ok, provided |
| 18 | SSH to UVM | ok, connected |
| 19 | Try mount S2 to UVM | ok, mounted. All created files exist |
| 20 | Unmount S2 | ok, unmounted |
| 21 | Delete S2, SS2, SS1, S1 | ok, deleted |
_12) Migrate share and write data
Driver mode: any
Involved APIs:
- [create share network]
- [create share type]
- create share
- allow share access
- migration-start share
- migration-complete share
- delete share
| Step | Action | Result |
|---|---|---|
| 1 | Create UVM | ok, created |
| 2 | Create share S1 of 1Gb size | ok, created |
| 3 | Provide RW access to S1 | ok, provided |
| 4 | SSH to UVM | ok, connected |
| 5 | Try mount S1 to UVM | ok, mounted |
| 6 | Create file1 | ok, created |
| 7 | Unmount share S1 | ok, unmounted |
| 8 | Do "migration-start" | ok, finished. 1 phase is completed. |
| 9 | Do "migration-complete" | ok, share instance only one - new one. it has previously created file1. |
| 10 | Try mount S1 to UVM | ok, mounted. Created file1 exists |
| 11 | Unmount share S1 | ok, unmounted |
| 12 | Delete share S1 | ok, deleted |
| 13 | Delete UVM | ok, deleted |
Alternatives
Alternative is what we have now. It is requirement to test each share driver manually and dependency on presence of detailed docs for each feature share drivers implement.
Data model impact
None
REST API impact
None
Driver impact
None
Security impact
None
Notifications impact
None
Other end user impact
End users will be able to run scenario tests against their manila deployment to test workability of various features.
Performance Impact
None
Other deployer impact
None
Developer impact
None
Implementation
Assignee(s)
Original assignee:
- vponomaryov
Other contributors:
- We're inviting more contributors to continue to improve scenario tests. Adding new scenario test cases to manila-tempest-plugin does not require adding the test case description to this spec, but it is encouraged if you like feedback for your new test cases.
Work Items
- Implement designed scenario tests in manila plugin for tempest.
Dependencies
None
Testing
It is expected that all first-party drivers as well as third-party drivers will be covered in CI systems with designed here scenario tests. Due to big amount of optional features that are covered by scenario tests, only appropriate scenario tests for specific back-end should run in CI systems. Scenarios that include only required features (1-4) are a must for running in CI systems for each share driver.
Documentation Impact
Doc describing usage of manila plugin for tempest [1] should be extended with configuration and usage details of scenario tests.