Remove bundled intree freezer tempest plugin
* https://review.openstack.org/#/c/526664/ moves the intree tempest plugin to freezer-tempest-plugin repo. * Excluding freezer/tests/freezer_tempest_plugin/tests/api/test_version.py as it is dependent on freezer project and moving under integration tests. Depends-On: I6967f915758728827e8ddcd1a45a7023904b694e Change-Id: I4625d55a768f1ad0762fc2d8554998825f0d6716
This commit is contained in:
Chandan Kumar
parent
3bbebaa416
commit
01da61b237
|
|
@ -21,6 +21,7 @@
|
|||
- openstack/freezer-api
|
||||
- openstack/freezer-web-ui
|
||||
- openstack/python-freezerclient
|
||||
- openstack/freezer-tempest-plugin
|
||||
|
||||
- job:
|
||||
name: freezer-centos-7
|
||||
|
|
@ -35,6 +36,7 @@
|
|||
- openstack/freezer-api
|
||||
- openstack/freezer-web-ui
|
||||
- openstack/python-freezerclient
|
||||
- openstack/freezer-tempest-plugin
|
||||
|
||||
- job:
|
||||
name: freezer-opensuse-423
|
||||
|
|
@ -49,3 +51,4 @@
|
|||
- openstack/freezer-api
|
||||
- openstack/freezer-web-ui
|
||||
- openstack/python-freezerclient
|
||||
- openstack/freezer-tempest-plugin
|
||||
|
|
|
|||
|
|
@ -1,259 +0,0 @@
|
|||
Freezer Tempest Tests
|
||||
=====================
|
||||
|
||||
Integration tests in Freezer are implemented using tempest. This document describes different approaches to run these tests.
|
||||
|
||||
Where to start?
|
||||
|
||||
* If you just want to run the tests as quickly as possible, start with `Run tests inside a devstack VM`_.
|
||||
* If you want to run tests on your local machine (with services running in devstack), start with `Run tests outside a devstack VM`_.
|
||||
|
||||
Alternatively there is a slightly different version that uses nose as a testrunner: `Run tests outside a devstack VM (alternative instructions using nose)`_.
|
||||
|
||||
* If you want to run tests on your local machine in PyCharm, start with `Run tests in PyCharm`_.
|
||||
|
||||
* If you want to run the tests on Mac OS X, start with `Mac OS X Instructions`_ and continue with one of the options above.
|
||||
|
||||
Setting up a devstack VM
|
||||
------------------------
|
||||
|
||||
Install devstack with swift and the freezer [1]_ as well as the freezer-api [2]_ plugins by adding the following lines to you `local.conf`:
|
||||
|
||||
::
|
||||
|
||||
enable_plugin freezer https://git.openstack.org/openstack/freezer master
|
||||
enable_plugin freezer-api https://git.openstack.org/openstack/freezer-api master
|
||||
enable_service s-proxy s-object s-container s-account
|
||||
|
||||
.. [1] https://github.com/openstack/freezer/blob/master/devstack/README.rst
|
||||
.. [2] https://github.com/openstack/freezer-api/blob/master/devstack/README.rst
|
||||
|
||||
Run tests inside a devstack VM
|
||||
-------------------------------
|
||||
|
||||
#. Create a devstack VM as described in `Setting up a devstack VM`_
|
||||
|
||||
#. Inside your devstack VM, navigate to `/opt/stack/tempest`.
|
||||
|
||||
#. Run `ostestr -r freezer`
|
||||
|
||||
Debugging tests inside a devstack VM
|
||||
------------------------------------
|
||||
|
||||
Often a devstack VM is used via SSH without graphical interface. Python has multiple command line debuggers. The out-of-the-box pdb works fine but I recommend pudb [3]_ which looks a bit like the old Turbo-Pascal/C IDE. The following steps are necessary to get it running:
|
||||
|
||||
#. Follow the steps in `Run tests inside a devstack VM`_.
|
||||
|
||||
#. Log into the devstack VM
|
||||
|
||||
#. Install pudb:
|
||||
|
||||
::
|
||||
|
||||
pip install pudb
|
||||
|
||||
#. Open the test file were you want to set the first breakpoint (more breakpoints can be set interactively later) and add the following line
|
||||
|
||||
::
|
||||
|
||||
import pudb;pu.db
|
||||
|
||||
#. Navigate to `/opt/stack/tempest`.
|
||||
|
||||
#. `ostestr` runs tests in parallel which causes issues with debuggers. To work around that you need to run the relevant test directly. E.g.:
|
||||
|
||||
::
|
||||
|
||||
python -m unittest freezer.tests.freezer_tempest_plugin.tests.scenario.test_backups.TestFreezerScenario
|
||||
|
||||
#. It should drop you into the debugger!
|
||||
|
||||
.. [3] https://pypi.python.org/pypi/pudb
|
||||
|
||||
Run tests outside a devstack VM
|
||||
-------------------------------
|
||||
|
||||
This section describes how to run the tests outside of a devstack VM (e.g. in PyCharm) while using services (keystone, swift, ...) inside a VM.
|
||||
|
||||
#. Create a devstack VM as described in `Setting up a devstack VM`_.
|
||||
|
||||
#. Create and activate a virtual environment for Tempest:
|
||||
::
|
||||
|
||||
virtualenv --no-site-packages tempest-venv
|
||||
. tempest-venv/bin/activate
|
||||
|
||||
#. Clone and install the Tempest project into the virtual environment:
|
||||
::
|
||||
|
||||
git clone https://github.com/openstack/tempest
|
||||
pip install tempest/
|
||||
|
||||
#. Clone and install the Freezer project into the virtual environment:
|
||||
::
|
||||
|
||||
git clone https://github.com/openstack/freezer
|
||||
pip install -e freezer/
|
||||
|
||||
#. Clone and install the Freezer API project into the virtual environment:
|
||||
::
|
||||
|
||||
git clone https://github.com/openstack/freezer-api
|
||||
pip install -e freezer-api/
|
||||
|
||||
#. Initialise a Tempest working directory:
|
||||
::
|
||||
|
||||
mkdir tempest-working
|
||||
cd tempest-working
|
||||
tempest init .
|
||||
|
||||
#. Configure `tempest-working/etc/tempest.conf`. The easiest way to do this is to just copy the config from `/opt/stack/tempest/etc/tempest.conf` inside the devstack VM.
|
||||
|
||||
#. Run the freezer test inside the tempest working directory:
|
||||
::
|
||||
|
||||
cd tempest-working
|
||||
ostestr -r freezer
|
||||
|
||||
Run tests outside a devstack VM (alternative instructions using nose)
|
||||
---------------------------------------------------------------------
|
||||
|
||||
#. Need to make sure that there is a Devstack or other environment for running Keystone and Swift.
|
||||
|
||||
#. Clone the Tempest Repo::
|
||||
|
||||
run 'git clone https://github.com/openstack/tempest.git'
|
||||
|
||||
#. Create a virtual environment for Tempest. In these instructions, the Tempest virtual environment is ``~/virtualenvs/tempest-freezer``.
|
||||
|
||||
#. Activate the Tempest virtual environment::
|
||||
|
||||
run 'source ~/virtualenvs/tempest-freezer/bin/activate'
|
||||
|
||||
#. Make sure you have latest pip installed::
|
||||
|
||||
run 'pip install --upgrade pip'
|
||||
|
||||
#. Install Tempest requirements.txt and test-requirements.txt in the Tempest virtual environment::
|
||||
|
||||
run 'pip install -r requirements.txt -r test-requirements.txt'
|
||||
|
||||
#. Install Tempest project into the virtual environment in develop mode::
|
||||
|
||||
run ‘python setup.py develop’
|
||||
|
||||
#. Create logging.conf in Tempest Repo home dir/etc
|
||||
|
||||
Make a copy of logging.conf.sample as logging.conf
|
||||
|
||||
In logging configuration
|
||||
|
||||
You will see this error on Mac OS X
|
||||
|
||||
socket.error: [Errno 2] No such file or directory
|
||||
|
||||
To fix this, edit logging.conf
|
||||
|
||||
Change ‘/dev/log/ to '/var/run/syslog’ in logging.conf
|
||||
|
||||
see: https://github.com/baremetal/python-backoff/issues/1 for details
|
||||
|
||||
#. Create tempest.conf in Tempest Repo home dir/etc::
|
||||
|
||||
run 'oslo-config-generator --config-file etc/config-generator.tempest.conf --output-file etc/tempest.conf'
|
||||
|
||||
Add the following sections to tempest.conf and modify uri and uri_v3 to point to the host where Keystone is running::
|
||||
|
||||
[identity]
|
||||
|
||||
username = freezer
|
||||
password = secretservice
|
||||
tenant_name = service
|
||||
domain_name = default
|
||||
admin_username = admin
|
||||
admin_password = secretadmin
|
||||
admin_domain_name = default
|
||||
admin_tenant_name = admin
|
||||
alt_username = admin
|
||||
alt_password = secretadmin
|
||||
alt_tenant_name = admin
|
||||
use_ssl = False
|
||||
auth_version = v3
|
||||
uri = http://10.10.10.6:5000/v2.0/
|
||||
uri_v3 = http://10.10.10.6:35357/v3/
|
||||
|
||||
[auth]
|
||||
|
||||
allow_tenant_isolation = true
|
||||
tempest_roles = admin
|
||||
|
||||
|
||||
#. Clone freezer Repo::
|
||||
|
||||
run 'git clone https://github.com/openstack/freezer.git'
|
||||
|
||||
#. Set the virtual environment to the Tempest virtual environment::
|
||||
|
||||
run 'source ~/virtualenvs/tempest-freezer/bin/activate'
|
||||
|
||||
#. pip install freezer requirements.txt and test-requirements.txt in Tempest virtual environment::
|
||||
|
||||
run 'pip install -r requirements.txt -r test-requirements.txt'
|
||||
|
||||
#. Install nose in the Temptest virtual environment::
|
||||
|
||||
run 'pip install nose'
|
||||
|
||||
#. Install freezer project into the Tempest virtual environment in develop mode::
|
||||
|
||||
run ‘python setup.py develop’
|
||||
|
||||
#. Set project interpreter (pycharm) to Tempest virtual environment.
|
||||
|
||||
#. Create test config (pycharm) using the Tempest virtual environment as python interpreter::
|
||||
|
||||
Set the environment variable OS_AUTH_URL to the URI where Keystone is running. For example, OS_AUTH_URL=http://10.10.10.6:5000/v2.0.
|
||||
Set the Working Directory to the Tempest home dir. This will allow Tempest to find the etc/tempest.conf file.
|
||||
|
||||
#. Run the tests in the api directory in the freezer_tempest_plugin directory.
|
||||
|
||||
Mac OS X Instructions
|
||||
---------------------
|
||||
|
||||
For Mac OS X users you will need to install gnu-tar in ``/usr/local/bin`` and make sure that ``/usr/local/bin`` is in the PATH environment variable before any other directories where a different version of tar can be found. Gnu-tar can be installed as ``gtar`` or ``tar``, either name works.
|
||||
|
||||
Also, currently for Mac OS X users, the latest version of gnu-tar (1.29) will not allow ``--unlink-first`` and ``--overwrite`` options to be used together. Also, gnu-tar will complain about the ``--unlink-first`` argument. To get around these limitations, you will need to modify ``tar_builders.py`` and remove the ``--unlink-first`` option from the ``UNIX_TEMPLATE`` variable.
|
||||
|
||||
Run tests in PyCharm
|
||||
--------------------
|
||||
|
||||
#. Set up the test environment as described in `Run tests outside a devstack VM`_.
|
||||
|
||||
#. Start PyCharm and open a new project pointing to the cloned freezer directory.
|
||||
|
||||
#. Click `File > Settings > Project: freezer > Project Interpreter`.
|
||||
|
||||
#. Click the gear-wheel icon next to `Project Interpreter` and choose `Add Local`.
|
||||
|
||||
#. Navigate to your virtual environment and select the Python interpreter under `bin/python` and confirm with `OK`
|
||||
|
||||
#. In the left pane, navigate to one of the test scripts in `freezer/tests/freezer_tempest_plugin/tests/[api or scenario]/*.py`.
|
||||
|
||||
#. Right-click the file and choose `Run 'Unittests in [..]'`
|
||||
|
||||
#. This test run will most likely fail because it is started from the wrong directory. To fix this, open the dropdown box next to the run button in the top-right corner. Choose `Edit Configurations ..`
|
||||
|
||||
#. Point `Working directory:` to your tempest working directory.
|
||||
|
||||
#. Run the test again, this time it should work!
|
||||
|
||||
|
||||
|
||||
Troubleshooting
|
||||
---------------
|
||||
|
||||
If tests fail these are good places to check:
|
||||
|
||||
* freezer-api log: `/var/log/apache2/freezer-api.log`
|
||||
* freezer-agent log: `$HOME/.freezer/freezer.log`
|
||||
|
|
@ -1,21 +0,0 @@
|
|||
# Copyright 2015
|
||||
# All Rights Reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
from oslo_config import cfg
|
||||
|
||||
service_option = cfg.BoolOpt('freezer',
|
||||
default=True,
|
||||
help="Whether or not freezer is expected to be "
|
||||
"available")
|
||||
|
|
@ -1,36 +0,0 @@
|
|||
# Copyright 2015
|
||||
# All Rights Reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
import os
|
||||
|
||||
from tempest.test_discover import plugins
|
||||
|
||||
from freezer.tests.freezer_tempest_plugin import config as freezer_config
|
||||
|
||||
|
||||
class FreezerTempestPlugin(plugins.TempestPlugin):
|
||||
def load_tests(self):
|
||||
base_path = os.path.split(os.path.dirname(
|
||||
os.path.abspath(__file__)))[0]
|
||||
test_dir = "freezer_tempest_plugin/tests"
|
||||
full_test_dir = os.path.join(base_path, test_dir)
|
||||
return full_test_dir, base_path
|
||||
|
||||
def register_opts(self, conf):
|
||||
conf.register_opt(freezer_config.service_option,
|
||||
group='service_available')
|
||||
|
||||
def get_opt_lists(self):
|
||||
return [('service_available', [freezer_config.service_option])]
|
||||
|
|
@ -1,269 +0,0 @@
|
|||
# (C) Copyright 2016 Hewlett Packard Enterprise Development Company LP
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
from datetime import datetime
|
||||
from datetime import timedelta
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
|
||||
from time import mktime
|
||||
|
||||
from tempest import test
|
||||
|
||||
from freezer.tests.integration.common import Temp_Tree
|
||||
|
||||
|
||||
def resolve_paths(metadata):
|
||||
"""Find all paths associated with a particular backup
|
||||
|
||||
freezer-agent stores all backups in the timestamped sub-directory of the
|
||||
first backup created with a particular (name, hostname) pair, so it isn't
|
||||
possible to guess the true location of backup data. This function searches
|
||||
the backup container to find both the true parent directory and a list of
|
||||
all files associated with the given metadata.
|
||||
|
||||
:param metadata: the metadata associated with the backup to resolve
|
||||
:return: a tuple containing the parent directory and a list of associated
|
||||
files, or (None, None)
|
||||
"""
|
||||
base_name = '{}_{}'.format(metadata['hostname'], metadata['backup_name'])
|
||||
expected_name = '{}_{}'.format(base_name, metadata['time_stamp'])
|
||||
|
||||
backup_base_path = os.path.join(metadata['container'], base_name)
|
||||
for timestamp in os.listdir(backup_base_path):
|
||||
timestamp_abs = os.path.join(backup_base_path, timestamp)
|
||||
|
||||
matching = filter(lambda p: expected_name in p,
|
||||
os.listdir(timestamp_abs))
|
||||
if matching:
|
||||
return timestamp_abs, matching
|
||||
|
||||
return None, None
|
||||
|
||||
|
||||
def mutate_timestamp(metadata, days_old):
|
||||
"""Alter all timestamps of an existing backup
|
||||
|
||||
Since there's no proper way to assign a timestamp to a backup, this method
|
||||
takes an existing backup and modifies all associated timestamps to make it
|
||||
otherwise indistinguishable from a backup actually created in the past.
|
||||
|
||||
:param metadata: the metadata associated with the backup to mutate
|
||||
:param days_old: the age (i.e. days before now) that should be set
|
||||
"""
|
||||
date = datetime.now() - timedelta(days=days_old)
|
||||
old_time_stamp = metadata['time_stamp']
|
||||
new_time_stamp = int(mktime(date.timetuple()))
|
||||
|
||||
parent_dir, files = resolve_paths(metadata)
|
||||
if os.path.basename(parent_dir) == str(old_time_stamp):
|
||||
# rename the parent dir, but only if it was created for this
|
||||
# backup (the dir may contain other backups with different
|
||||
# timestamps that we shouldn't touch)
|
||||
new_path = os.path.join(os.path.dirname(parent_dir),
|
||||
str(new_time_stamp))
|
||||
os.rename(parent_dir, new_path)
|
||||
parent_dir = new_path
|
||||
|
||||
# rename each file associated with the backup, since each filename
|
||||
# contains the timestamp as well
|
||||
for old_file in files:
|
||||
new_file = old_file.replace(str(old_time_stamp), str(new_time_stamp))
|
||||
os.rename(os.path.join(parent_dir, old_file),
|
||||
os.path.join(parent_dir, new_file))
|
||||
|
||||
# update the metadata before saving to keep things consistent
|
||||
metadata['time_stamp'] = new_time_stamp
|
||||
|
||||
|
||||
def load_metadata(path):
|
||||
"""Given a metadata path, return a dict containing parsed values.
|
||||
|
||||
:param path: the path to load
|
||||
:return: a metadata dict
|
||||
"""
|
||||
with open(path, 'r') as f:
|
||||
return json.load(f)
|
||||
|
||||
|
||||
def save_metadata(metadata, path):
|
||||
"""Write the given metadata object to the provided path.
|
||||
|
||||
:param metadata: the metadata dict to write
|
||||
:param path: the path at which to write the metadata
|
||||
"""
|
||||
with open(path, 'w') as f:
|
||||
json.dump(metadata, f)
|
||||
|
||||
|
||||
class BaseFreezerTest(test.BaseTestCase):
|
||||
credentials = ['primary']
|
||||
|
||||
def __init__(self, *args, **kwargs):
|
||||
|
||||
super(BaseFreezerTest, self).__init__(*args, **kwargs)
|
||||
|
||||
# noinspection PyAttributeOutsideInit
|
||||
def setUp(self):
|
||||
super(BaseFreezerTest, self).setUp()
|
||||
|
||||
self.storage = Temp_Tree()
|
||||
self.source_trees = []
|
||||
self.backup_count = 0
|
||||
self.backup_name = 'backup_test'
|
||||
|
||||
self.get_environ()
|
||||
|
||||
def tearDown(self):
|
||||
|
||||
super(BaseFreezerTest, self).tearDown()
|
||||
|
||||
for tree in self.source_trees:
|
||||
tree.cleanup()
|
||||
|
||||
self.storage.cleanup()
|
||||
|
||||
@classmethod
|
||||
def get_auth_url(cls):
|
||||
return cls.os_primary.auth_provider.auth_client.auth_url[:-len(
|
||||
'/auth/tokens')]
|
||||
|
||||
@classmethod
|
||||
def setup_clients(cls):
|
||||
super(BaseFreezerTest, cls).setup_clients()
|
||||
cls.get_environ()
|
||||
|
||||
@classmethod
|
||||
def get_environ(cls):
|
||||
os.environ['OS_PASSWORD'] = cls.os_primary.credentials.password
|
||||
os.environ['OS_USERNAME'] = cls.os_primary.credentials.username
|
||||
os.environ['OS_PROJECT_NAME'] = cls.os_primary.credentials.tenant_name
|
||||
os.environ['OS_TENANT_NAME'] = cls.os_primary.credentials.tenant_name
|
||||
os.environ['OS_PROJECT_DOMAIN_NAME'] = \
|
||||
cls.os_primary.credentials.project_domain_name
|
||||
os.environ['OS_USER_DOMAIN_NAME'] = \
|
||||
cls.os_primary.credentials.user_domain_name
|
||||
|
||||
# Allow developers to set OS_AUTH_URL when developing so that
|
||||
# Keystone may be on a host other than localhost.
|
||||
if 'OS_AUTH_URL' not in os.environ:
|
||||
os.environ['OS_AUTH_URL'] = cls.get_auth_url()
|
||||
|
||||
# Mac OS X uses gtar located in /usr/local/bin
|
||||
os.environ['PATH'] = '/usr/local/bin:' + os.environ['PATH']
|
||||
|
||||
return os.environ
|
||||
|
||||
def run_subprocess(self, sub_process_args, fail_message):
|
||||
|
||||
proc = subprocess.Popen(sub_process_args,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
env=self.environ, shell=False)
|
||||
|
||||
out, err = proc.communicate()
|
||||
|
||||
self.assertEqual(0, proc.returncode,
|
||||
fail_message + " Output: {0}. "
|
||||
"Error: {1}".format(out, err))
|
||||
|
||||
self.assertEqual('', err,
|
||||
fail_message + " Output: {0}. "
|
||||
"Error: {1}".format(out, err))
|
||||
|
||||
def create_local_backup(self, hostname=None, compression=None,
|
||||
consistency_check=None, incremental=True,
|
||||
always_level=None, restart_always_level=None,
|
||||
max_level=None):
|
||||
"""Creates a new backup with the given parameters.
|
||||
|
||||
The backup will immediately be created using a randomly-generated
|
||||
source tree on the local filesystem, and will be stored in a random
|
||||
temporary directory using the 'local' storage mode. All generated data
|
||||
files will be automatically removed during `tearDown()`, though
|
||||
implementations are responsible for cleaning up any additional copies
|
||||
or restores created via other methods.
|
||||
|
||||
:param hostname: if set, set `--hostname` to the given value
|
||||
:param compression: if set, set `--compression` to the given value
|
||||
:param consistency_check: if True, set `--consistency_check`
|
||||
:param incremental: if False, set `--no-incremental`
|
||||
:param always_level: sets `--always-level` to the given value
|
||||
:param restart_always_level: sets `--restart-always-level`
|
||||
:param max_level: sets `--max-level` to the given value
|
||||
:return: the path to the stored backup metadata
|
||||
"""
|
||||
metadata_path = os.path.join(
|
||||
self.storage.path,
|
||||
'metadata-{}.json'.format(self.backup_count))
|
||||
self.backup_count += 1
|
||||
|
||||
tree = Temp_Tree()
|
||||
tree.add_random_data()
|
||||
self.source_trees.append(tree)
|
||||
|
||||
backup_args = [
|
||||
'freezer-agent',
|
||||
'--path-to-backup', tree.path,
|
||||
'--container', self.storage.path,
|
||||
'--backup-name', self.backup_name,
|
||||
'--storage', 'local',
|
||||
'--metadata-out', metadata_path,
|
||||
]
|
||||
|
||||
if hostname:
|
||||
backup_args += ['--hostname', hostname]
|
||||
|
||||
if compression:
|
||||
backup_args += ['--compression', compression]
|
||||
|
||||
if consistency_check:
|
||||
backup_args += ['--consistency-check']
|
||||
|
||||
if incremental:
|
||||
if always_level is not None:
|
||||
backup_args += ['--always-level', str(always_level)]
|
||||
|
||||
if max_level is not None:
|
||||
backup_args += ['--max-level', str(max_level)]
|
||||
|
||||
if restart_always_level:
|
||||
backup_args += ['--restart-always-level',
|
||||
str(restart_always_level)]
|
||||
else:
|
||||
backup_args += ['--no-incremental', 'NO_INCREMENTAL']
|
||||
|
||||
self.run_subprocess(backup_args, 'Test backup to local storage.')
|
||||
|
||||
return metadata_path
|
||||
|
||||
def create_mutated_backup(self, days_old=30, **kwargs):
|
||||
"""Create a local backup with a mutated timestamp
|
||||
|
||||
This creates a new backup using `create_local_backup()`, modifies it
|
||||
using `mutate_timestamp()`, and then returns the resulting (loaded)
|
||||
metadata dict.
|
||||
|
||||
:param days_old: the age of the backup to create
|
||||
:param kwargs: arguments to pass to `create_local_backup()`
|
||||
:return: the loaded metadata
|
||||
"""
|
||||
metadata_path = self.create_local_backup(**kwargs)
|
||||
|
||||
metadata = load_metadata(metadata_path)
|
||||
mutate_timestamp(metadata, days_old)
|
||||
save_metadata(metadata, metadata_path)
|
||||
|
||||
return metadata
|
||||
|
|
@ -1,188 +0,0 @@
|
|||
# (C) Copyright 2016 Hewlett Packard Enterprise Development Company LP
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
|
||||
from tempest.lib import decorators
|
||||
|
||||
from freezer.tests.freezer_tempest_plugin.tests.api import base
|
||||
from freezer.tests.integration import common
|
||||
|
||||
|
||||
class TestFreezerCompressGzip(base.BaseFreezerTest):
|
||||
def __init__(self, *args, **kwargs):
|
||||
super(TestFreezerCompressGzip, self).__init__(*args, **kwargs)
|
||||
|
||||
# noinspection PyAttributeOutsideInit
|
||||
def setUp(self):
|
||||
super(TestFreezerCompressGzip, self).setUp()
|
||||
|
||||
# create a source tree to backup with a few empty files
|
||||
# (files must be empty to avoid encoding errors with pure random data)
|
||||
self.source_tree = common.Temp_Tree()
|
||||
self.source_tree.add_random_data(size=0)
|
||||
|
||||
self.storage_tree = common.Temp_Tree()
|
||||
self.dest_tree = common.Temp_Tree()
|
||||
|
||||
self.environ = super(TestFreezerCompressGzip, self).get_environ()
|
||||
|
||||
def tearDown(self):
|
||||
super(TestFreezerCompressGzip, self).tearDown()
|
||||
|
||||
self.source_tree.cleanup()
|
||||
self.dest_tree.cleanup()
|
||||
self.storage_tree.cleanup()
|
||||
|
||||
def _backup(self, name, method):
|
||||
# perform a normal backup, with gzip specified
|
||||
backup_args = ['freezer-agent',
|
||||
'--path-to-backup',
|
||||
self.source_tree.path,
|
||||
'--container',
|
||||
self.storage_tree.path,
|
||||
'--backup-name',
|
||||
name,
|
||||
'--storage',
|
||||
'local',
|
||||
'--compress',
|
||||
method,
|
||||
'--metadata-out',
|
||||
os.path.join(self.storage_tree.path, 'metadata.json')]
|
||||
|
||||
self.run_subprocess(backup_args, 'Test gzip backup to local storage.')
|
||||
|
||||
def _restore(self, name, method):
|
||||
restore_args = ['freezer-agent',
|
||||
'--action',
|
||||
'restore',
|
||||
'--restore-abs-path',
|
||||
self.dest_tree.path,
|
||||
'--container',
|
||||
self.storage_tree.path,
|
||||
'--backup-name',
|
||||
name,
|
||||
'--storage',
|
||||
'local',
|
||||
'--compress',
|
||||
method]
|
||||
|
||||
self.run_subprocess(restore_args, 'Test restore from local storage.')
|
||||
|
||||
def _metadata(self):
|
||||
path = os.path.join(self.storage_tree.path, 'metadata.json')
|
||||
with open(path, 'r') as f:
|
||||
return json.load(f)
|
||||
|
||||
def _file_get_mimetype(self, metadata):
|
||||
"""Given some file metadata, find its mimetype using the file command
|
||||
|
||||
:param metadata: the parsed json file metadata
|
||||
:return: the mimetype
|
||||
"""
|
||||
"""
|
||||
Data is stored like data/tar/localhost_False/1469786264/0_1469786264 so
|
||||
we need build the same directory structure.
|
||||
data: the directory that holds the backup data
|
||||
tar: the engine used to create backup
|
||||
localhost: the hostname of the machine where the backup was taken
|
||||
False: it should be backup name or False is backup is not provided
|
||||
1469786264: timestamp
|
||||
0_1469786264: level zero timestamp
|
||||
"""
|
||||
data_file_path = 'data{0}{1}{0}{2}_{3}{0}{4}{0}{5}_{4}{0}data'.format(
|
||||
os.path.sep,
|
||||
"tar", # currently we support only tar
|
||||
metadata['hostname'],
|
||||
metadata['backup_name'],
|
||||
metadata['time_stamp'],
|
||||
metadata['curr_backup_level']
|
||||
)
|
||||
data_file_path = os.path.join(self.storage_tree.path,
|
||||
data_file_path)
|
||||
self.assertEqual(True, os.path.exists(data_file_path))
|
||||
|
||||
# run 'file' in brief mode to only output the values we want
|
||||
proc = subprocess.Popen(['file', '-b', '--mime-type', data_file_path],
|
||||
stdout=subprocess.PIPE)
|
||||
out, err = proc.communicate()
|
||||
self.assertEqual(0, proc.returncode)
|
||||
|
||||
return out.strip()
|
||||
|
||||
@decorators.attr(type="gate")
|
||||
def test_freezer_backup_compress_gzip(self):
|
||||
backup_name = 'freezer-test-backup-gzip-0'
|
||||
|
||||
self._backup(backup_name, 'gzip')
|
||||
self._restore(backup_name, 'gzip')
|
||||
|
||||
# metadata should show the correct algorithm
|
||||
metadata = self._metadata()
|
||||
self.assertIn('compression', metadata)
|
||||
self.assertEqual('gzip', metadata['compression'])
|
||||
|
||||
# file utility should detect the correct mimetype
|
||||
gizp_mimetypes = ['application/gzip', 'application/x-gzip']
|
||||
mimetype = self._file_get_mimetype(metadata)
|
||||
self.assertIn(mimetype, gizp_mimetypes)
|
||||
|
||||
# actual contents should be the same
|
||||
diff_args = ['diff', '-r', '-q',
|
||||
self.source_tree.path,
|
||||
self.dest_tree.path]
|
||||
self.run_subprocess(diff_args, 'Verify restored copy is identical to '
|
||||
'original.')
|
||||
|
||||
@decorators.attr(type="gate")
|
||||
def test_freezer_backup_compress_bzip2(self):
|
||||
backup_name = 'freezer-test-backup-bzip2-0'
|
||||
|
||||
self._backup(backup_name, 'bzip2')
|
||||
self._restore(backup_name, 'bzip2')
|
||||
|
||||
metadata = self._metadata()
|
||||
self.assertIn('compression', metadata)
|
||||
self.assertEqual('bzip2', metadata['compression'])
|
||||
|
||||
mimetype = self._file_get_mimetype(metadata)
|
||||
self.assertEqual('application/x-bzip2', mimetype)
|
||||
|
||||
diff_args = ['diff', '-r', '-q',
|
||||
self.source_tree.path,
|
||||
self.dest_tree.path]
|
||||
self.run_subprocess(diff_args, 'Verify restored copy is identical to '
|
||||
'original.')
|
||||
|
||||
@decorators.attr(type="gate")
|
||||
def test_freezer_backup_compress_xz(self):
|
||||
backup_name = 'freezer-test-backup-xz-0'
|
||||
|
||||
self._backup(backup_name, 'xz')
|
||||
self._restore(backup_name, 'xz')
|
||||
|
||||
metadata = self._metadata()
|
||||
self.assertIn('compression', metadata)
|
||||
self.assertEqual('xz', metadata['compression'])
|
||||
|
||||
mimetype = self._file_get_mimetype(metadata)
|
||||
self.assertEqual('application/x-xz', mimetype)
|
||||
|
||||
diff_args = ['diff', '-r', '-q',
|
||||
self.source_tree.path,
|
||||
self.dest_tree.path]
|
||||
self.run_subprocess(diff_args, 'Verify restored copy is identical to '
|
||||
'original.')
|
||||
|
|
@ -1,110 +0,0 @@
|
|||
# (C) Copyright 2016 Hewlett Packard Enterprise Development Company LP
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
import os
|
||||
import shutil
|
||||
|
||||
from oslo_utils import uuidutils
|
||||
from tempest.lib import decorators
|
||||
|
||||
from freezer.tests.freezer_tempest_plugin.tests.api import base
|
||||
|
||||
|
||||
class TestFreezerFSBackup(base.BaseFreezerTest):
|
||||
def __init__(self, *args, **kwargs):
|
||||
super(TestFreezerFSBackup, self).__init__(*args, **kwargs)
|
||||
|
||||
def setUp(self):
|
||||
super(TestFreezerFSBackup, self).setUp()
|
||||
|
||||
test_id = uuidutils.generate_uuid(dashed=False)
|
||||
|
||||
self.backup_source_dir = (
|
||||
"/tmp/freezer-test-backup-source/" + test_id
|
||||
)
|
||||
|
||||
self.backup_source_sub_dir = self.backup_source_dir + "/subdir"
|
||||
|
||||
self.restore_target_dir = (
|
||||
"/tmp/freezer-test-backup-restore/" + test_id
|
||||
)
|
||||
|
||||
self.backup_local_storage_dir = (
|
||||
"/tmp/freezer-test-backup-local-storage/" + test_id
|
||||
)
|
||||
|
||||
self.freezer_backup_name = 'freezer-test-backup-fs-0'
|
||||
|
||||
shutil.rmtree(self.backup_source_dir, True)
|
||||
os.makedirs(self.backup_source_dir)
|
||||
open(self.backup_source_dir + "/a", 'w').close()
|
||||
open(self.backup_source_dir + "/b", 'w').close()
|
||||
open(self.backup_source_dir + "/c", 'w').close()
|
||||
|
||||
os.makedirs(self.backup_source_sub_dir)
|
||||
open(self.backup_source_sub_dir + "/x", 'w').close()
|
||||
open(self.backup_source_sub_dir + "/y", 'w').close()
|
||||
open(self.backup_source_sub_dir + "/z", 'w').close()
|
||||
|
||||
shutil.rmtree(self.restore_target_dir, True)
|
||||
os.makedirs(self.restore_target_dir)
|
||||
|
||||
shutil.rmtree(self.backup_local_storage_dir, True)
|
||||
os.makedirs(self.backup_local_storage_dir)
|
||||
|
||||
self.environ = super(TestFreezerFSBackup, self).get_environ()
|
||||
|
||||
def tearDown(self):
|
||||
super(TestFreezerFSBackup, self).tearDown()
|
||||
shutil.rmtree(self.backup_source_dir, True)
|
||||
shutil.rmtree(self.restore_target_dir, True)
|
||||
shutil.rmtree(self.backup_local_storage_dir)
|
||||
|
||||
@decorators.attr(type="gate")
|
||||
def test_freezer_fs_backup(self):
|
||||
backup_args = ['freezer-agent',
|
||||
'--path-to-backup',
|
||||
self.backup_source_dir,
|
||||
'--container',
|
||||
self.backup_local_storage_dir,
|
||||
'--backup-name',
|
||||
self.freezer_backup_name,
|
||||
'--storage',
|
||||
'local']
|
||||
|
||||
self.run_subprocess(backup_args, "Test backup to local storage.")
|
||||
|
||||
restore_args = ['freezer-agent',
|
||||
'--action',
|
||||
'restore',
|
||||
'--restore-abs-path',
|
||||
self.restore_target_dir,
|
||||
'--container',
|
||||
self.backup_local_storage_dir,
|
||||
'--backup-name',
|
||||
self.freezer_backup_name,
|
||||
'--storage',
|
||||
'local']
|
||||
|
||||
self.run_subprocess(restore_args, "Test restore from local storage.")
|
||||
|
||||
diff_args = ['diff',
|
||||
'-r',
|
||||
'-q',
|
||||
self.backup_source_dir,
|
||||
self.restore_target_dir]
|
||||
|
||||
self.run_subprocess(diff_args,
|
||||
"Test backup restore from local storage "
|
||||
"diff.")
|
||||
|
|
@ -1,94 +0,0 @@
|
|||
# (C) Copyright 2016 Hewlett Packard Enterprise Development Company LP
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
import subprocess
|
||||
|
||||
from tempest.lib import decorators
|
||||
|
||||
from freezer.tests.freezer_tempest_plugin.tests.api import base
|
||||
from freezer.tests.integration import common
|
||||
|
||||
|
||||
class TestFreezerMetadataChecksum(base.BaseFreezerTest):
|
||||
def __init__(self, *args, **kwargs):
|
||||
super(TestFreezerMetadataChecksum, self).__init__(*args, **kwargs)
|
||||
|
||||
# noinspection PyAttributeOutsideInit
|
||||
def setUp(self):
|
||||
super(TestFreezerMetadataChecksum, self).setUp()
|
||||
|
||||
self.environ = super(TestFreezerMetadataChecksum, self).get_environ()
|
||||
self.dest_tree = common.Temp_Tree()
|
||||
self.backup_name = 'backup_checksum_test'
|
||||
|
||||
def tearDown(self):
|
||||
super(TestFreezerMetadataChecksum, self).tearDown()
|
||||
|
||||
self.dest_tree.cleanup()
|
||||
|
||||
@decorators.attr(type="gate")
|
||||
def test_freezer_fs_backup_valid_checksum(self):
|
||||
# perform a normal backup, but enable consistency checks and save the
|
||||
# metadata to disk
|
||||
metadata_path = self.create_local_backup(consistency_check=True)
|
||||
|
||||
metadata = base.load_metadata(metadata_path)
|
||||
|
||||
# load the stored metadata to retrieve the computed checksum
|
||||
self.assertIn('consistency_checksum', metadata,
|
||||
'Checksum must exist in stored metadata.')
|
||||
|
||||
checksum = metadata['consistency_checksum']
|
||||
restore_args = ['freezer-agent',
|
||||
'--action', 'restore',
|
||||
'--restore-abs-path', self.dest_tree.path,
|
||||
'--container', metadata['container'],
|
||||
'--backup-name', self.backup_name,
|
||||
'--storage', 'local',
|
||||
'--consistency-checksum', checksum]
|
||||
|
||||
self.run_subprocess(restore_args,
|
||||
'Test restore from local storage with '
|
||||
'computed checksum.')
|
||||
|
||||
@decorators.attr(type="gate")
|
||||
def test_freezer_fs_backup_bad_checksum(self):
|
||||
# as above, but we'll ignore the computed checksum
|
||||
metadata_path = self.create_local_backup(consistency_check=True)
|
||||
metadata = base.load_metadata(metadata_path)
|
||||
|
||||
# make a failing sha256 checksum (assuming no added path string)
|
||||
bad_checksum = '0' * 64
|
||||
|
||||
# attempt to restore using the bad checksum
|
||||
restore_args = ['freezer-agent',
|
||||
'--action', 'restore',
|
||||
'--restore-abs-path', self.dest_tree.path,
|
||||
'--container', metadata['container'],
|
||||
'--backup-name', self.backup_name,
|
||||
'--storage', 'local',
|
||||
'--consistency-checksum', bad_checksum]
|
||||
|
||||
process = subprocess.Popen(restore_args,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
env=self.environ, shell=False)
|
||||
out, err = process.communicate()
|
||||
|
||||
# make sure the subprocess exist with an error due to checksum mismatch
|
||||
message = '{0} Output: {1} Error: {2}'.format(
|
||||
'Restore process should fail with checksum error.',
|
||||
out, err)
|
||||
self.assertEqual(1, process.returncode, message)
|
||||
self.assertEqual('', out, message)
|
||||
self.assertNotEqual('', err, message)
|
||||
|
|
@ -1,100 +0,0 @@
|
|||
# (C) Copyright 2016 Hewlett Packard Enterprise Development Company LP
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
import os
|
||||
import shutil
|
||||
|
||||
from oslo_utils import uuidutils
|
||||
from tempest.lib import decorators
|
||||
|
||||
from freezer.tests.freezer_tempest_plugin.tests.api import base
|
||||
|
||||
|
||||
class TestFreezerSwiftBackup(base.BaseFreezerTest):
|
||||
def __init__(self, *args, **kwargs):
|
||||
super(TestFreezerSwiftBackup, self).__init__(*args, **kwargs)
|
||||
|
||||
def setUp(self):
|
||||
super(TestFreezerSwiftBackup, self).setUp()
|
||||
|
||||
test_id = uuidutils.generate_uuid(dashed=False)
|
||||
|
||||
self.backup_source_dir = (
|
||||
"/tmp/freezer-test-backup-source/" + test_id
|
||||
)
|
||||
self.backup_source_sub_dir = self.backup_source_dir + "/subdir"
|
||||
|
||||
self.restore_target_dir = (
|
||||
"/tmp/freezer-test-backup-restore/" + test_id
|
||||
)
|
||||
|
||||
self.freezer_container_name = 'freezer-test-container-0'
|
||||
self.freezer_backup_name = 'freezer-test-backup-swift-0'
|
||||
|
||||
shutil.rmtree(self.backup_source_dir, True)
|
||||
os.makedirs(self.backup_source_dir)
|
||||
open(self.backup_source_dir + "/a", 'w').close()
|
||||
open(self.backup_source_dir + "/b", 'w').close()
|
||||
open(self.backup_source_dir + "/c", 'w').close()
|
||||
|
||||
os.makedirs(self.backup_source_sub_dir)
|
||||
open(self.backup_source_sub_dir + "/x", 'w').close()
|
||||
open(self.backup_source_sub_dir + "/y", 'w').close()
|
||||
open(self.backup_source_sub_dir + "/z", 'w').close()
|
||||
|
||||
shutil.rmtree(self.restore_target_dir, True)
|
||||
os.makedirs(self.restore_target_dir)
|
||||
|
||||
self.environ = super(TestFreezerSwiftBackup, self).get_environ()
|
||||
|
||||
def tearDown(self):
|
||||
super(TestFreezerSwiftBackup, self).tearDown()
|
||||
|
||||
shutil.rmtree(self.backup_source_dir, True)
|
||||
shutil.rmtree(self.restore_target_dir, True)
|
||||
|
||||
@decorators.attr(type="gate")
|
||||
def test_freezer_swift_backup(self):
|
||||
backup_args = ['freezer-agent',
|
||||
'--path-to-backup',
|
||||
self.backup_source_dir,
|
||||
'--container',
|
||||
self.freezer_container_name,
|
||||
'--backup-name',
|
||||
self.freezer_backup_name]
|
||||
|
||||
self.run_subprocess(backup_args, "Test backup to swift.")
|
||||
|
||||
restore_args = ['freezer-agent',
|
||||
'--action',
|
||||
'restore',
|
||||
'--restore-abs-path',
|
||||
self.restore_target_dir,
|
||||
'--container',
|
||||
self.freezer_container_name,
|
||||
'--backup-name',
|
||||
self.freezer_backup_name,
|
||||
'--storage',
|
||||
'swift']
|
||||
|
||||
self.run_subprocess(restore_args, "Test restore from swift.")
|
||||
|
||||
diff_args = ['diff',
|
||||
'-r',
|
||||
'-q',
|
||||
self.backup_source_dir,
|
||||
self.restore_target_dir]
|
||||
|
||||
self.run_subprocess(diff_args,
|
||||
"Test backup to swift and restore diff.")
|
||||
|
|
@ -1,25 +0,0 @@
|
|||
# (C) Copyright 2016 Hewlett Packard Enterprise Development Company LP
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
from tempest.lib import decorators
|
||||
|
||||
from freezer.tests.freezer_tempest_plugin.tests.api import base
|
||||
|
||||
|
||||
class TestFreezerTestsRunning(base.BaseFreezerTest):
|
||||
|
||||
@decorators.attr(type="gate")
|
||||
def test_tests_running(self):
|
||||
# See if tempest plugin tests run.
|
||||
self.assertEqual(1, 1, 'Tests are running')
|
||||
|
|
@ -1,305 +0,0 @@
|
|||
# (C) Copyright 2016 Hewlett Packard Enterprise Development Company LP
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License"); you may
|
||||
# not use this file except in compliance with the License. You may obtain
|
||||
# a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
||||
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
||||
# License for the specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
import time
|
||||
|
||||
from tempest.lib.cli import base as cli_base
|
||||
from tempest.lib.cli import output_parser
|
||||
|
||||
from freezer.tests.freezer_tempest_plugin.tests.api import base
|
||||
|
||||
JOB_TABLE_RESULT_COLUMN = 3
|
||||
|
||||
|
||||
class BaseFreezerCliTest(base.BaseFreezerTest):
|
||||
"""Base test case class for all Freezer API tests."""
|
||||
|
||||
credentials = ['primary']
|
||||
|
||||
@classmethod
|
||||
def setup_clients(cls):
|
||||
super(BaseFreezerCliTest, cls).setup_clients()
|
||||
|
||||
cls.cli = CLIClientWithFreezer(
|
||||
username=cls.os_primary.credentials.username,
|
||||
# fails if the password contains an unescaped $ sign
|
||||
password=cls.os_primary.credentials.password.replace('$', '$$'),
|
||||
tenant_name=cls.os_primary.credentials.tenant_name,
|
||||
uri=cls.get_auth_url(),
|
||||
cli_dir='/usr/local/bin' # devstack default
|
||||
)
|
||||
cls.cli.cli_dir = ''
|
||||
|
||||
def delete_job(self, job_id):
|
||||
self.cli.freezer_client(action='job-delete', params=job_id)
|
||||
|
||||
def create_job(self, job_json):
|
||||
|
||||
with tempfile.NamedTemporaryFile(delete=False) as job_file:
|
||||
job_file.write(json.dumps(job_json))
|
||||
job_file.flush()
|
||||
|
||||
output = self.cli.freezer_client(
|
||||
action='job-create',
|
||||
params='--file {} --client {}'.format(job_file.name,
|
||||
job_json['client_id']))
|
||||
job_id = output.split()[1]
|
||||
expected = 'Job {} created'.format(job_id)
|
||||
self.assertEqual(expected, output.strip())
|
||||
|
||||
self.addCleanup(self.delete_job, job_id)
|
||||
|
||||
return job_id
|
||||
|
||||
def find_job_in_job_list(self, job_id):
|
||||
job_list = output_parser.table(
|
||||
self.cli.freezer_client(action='job-list', params='-C test_node'))
|
||||
|
||||
for row in job_list['values']:
|
||||
if row[0].strip() == job_id.strip():
|
||||
return row
|
||||
|
||||
self.fail('Could not find job: {}'.format(job_id))
|
||||
|
||||
def wait_for_job_status(self, job_id, timeout=720):
|
||||
start = time.time()
|
||||
|
||||
while True:
|
||||
row = self.find_job_in_job_list(job_id)
|
||||
|
||||
if row[JOB_TABLE_RESULT_COLUMN]:
|
||||
return
|
||||
elif time.time() - start > timeout:
|
||||
self.fail("Status of job '{}' is '{}'."
|
||||
.format(job_id, row[JOB_TABLE_RESULT_COLUMN]))
|
||||
else:
|
||||
time.sleep(1)
|
||||
|
||||
def assertJobColumnEqual(self, job_id, column, expected):
|
||||
row = self.find_job_in_job_list(job_id)
|
||||
self.assertEqual(expected, row[column])
|
||||
|
||||
|
||||
class CLIClientWithFreezer(cli_base.CLIClient):
|
||||
def freezer_scheduler(self, action, flags='', params='', fail_ok=False,
|
||||
endpoint_type='publicURL', merge_stderr=False):
|
||||
"""Executes freezer-scheduler command for the given action.
|
||||
|
||||
:param action: the cli command to run using freezer-scheduler
|
||||
:type action: string
|
||||
:param flags: any optional cli flags to use
|
||||
:type flags: string
|
||||
:param params: any optional positional args to use :type params: string
|
||||
:param fail_ok: if True an exception is not raised when the
|
||||
cli return code is non-zero
|
||||
:type fail_ok: boolean
|
||||
:param endpoint_type: the type of endpoint for the service
|
||||
:type endpoint_type: string
|
||||
:param merge_stderr: if True the stderr buffer is merged into stdout
|
||||
:type merge_stderr: boolean
|
||||
"""
|
||||
|
||||
flags += ' --os-endpoint-type %s' % endpoint_type
|
||||
flags += ' --os-cacert /etc/ssl/certs/ca-certificates.crt'
|
||||
flags += ' --os-project-domain-name Default'
|
||||
flags += ' --os-user-domain-name Default'
|
||||
|
||||
return self.cmd_with_auth(
|
||||
'freezer-scheduler', action, flags, params, fail_ok, merge_stderr)
|
||||
|
||||
def freezer_client(self, action, flags='', params='', fail_ok=False,
|
||||
endpoint_type='publicURL', merge_stderr=True):
|
||||
flags += ' --os-endpoint-type %s' % endpoint_type
|
||||
flags += ' --os-cacert /etc/ssl/certs/ca-certificates.crt'
|
||||
flags += ' --os-project-domain-name Default'
|
||||
flags += ' --os-user-domain-name Default'
|
||||
return self.cmd_with_auth(
|
||||
'freezer', action, flags, params, fail_ok, merge_stderr)
|
||||
|
||||
|
||||
# This class is just copied from the freezer repo. Depending on where the
|
||||
# scenario tests end up we may need to refactore this.
|
||||
class Temp_Tree(object):
|
||||
def __init__(self, suffix='', dir=None, create=True):
|
||||
self.create = create
|
||||
if create:
|
||||
self.path = tempfile.mkdtemp(dir=dir, prefix='__freezer_',
|
||||
suffix=suffix)
|
||||
else:
|
||||
self.path = dir
|
||||
self.files = []
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def cleanup(self):
|
||||
if self.create and self.path:
|
||||
shutil.rmtree(self.path)
|
||||
|
||||
def __exit__(self, exc_type, exc_val, exc_tb):
|
||||
self.cleanup()
|
||||
|
||||
def add_random_data(self, ndir=5, nfile=5, size=1024):
|
||||
"""
|
||||
add some files containing randoma data
|
||||
|
||||
:param ndir: number of dirs to create
|
||||
:param nfile: number of files to create in each dir
|
||||
:param size: size of files
|
||||
:return: None
|
||||
"""
|
||||
for x in range(ndir):
|
||||
subdir_path = tempfile.mkdtemp(dir=self.path)
|
||||
for y in range(nfile):
|
||||
abs_pathname = self.create_file_with_random_data(
|
||||
dir_path=subdir_path, size=size)
|
||||
rel_path_name = abs_pathname[len(self.path) + 1:]
|
||||
self.files.append(rel_path_name)
|
||||
|
||||
def create_file_with_random_data(self, dir_path, size=1024):
|
||||
handle, abs_pathname = tempfile.mkstemp(dir=dir_path)
|
||||
with open(abs_pathname, 'wb') as fd:
|
||||
fd.write(os.urandom(size))
|
||||
return abs_pathname
|
||||
|
||||
def get_file_hash(self, rel_filepath):
|
||||
filepath = os.path.join(self.path, rel_filepath)
|
||||
if os.path.isfile(filepath):
|
||||
return self._filehash(filepath)
|
||||
else:
|
||||
return ''
|
||||
|
||||
def _filehash(self, filepath):
|
||||
"""
|
||||
Get GIT style sha1 hash for a file
|
||||
|
||||
:param filepath: path of file to hash
|
||||
:return: hash of the file
|
||||
"""
|
||||
filesize_bytes = os.path.getsize(filepath)
|
||||
hash_obj = hashlib.sha1()
|
||||
hash_obj.update(("blob %u\0" % filesize_bytes).encode('utf-8'))
|
||||
with open(filepath, 'rb') as handle:
|
||||
hash_obj.update(handle.read())
|
||||
return hash_obj.hexdigest()
|
||||
|
||||
def get_file_list(self):
|
||||
"""
|
||||
walks the dir tree and creates a list of relative pathnames
|
||||
:return: list of relative file paths
|
||||
"""
|
||||
self.files = []
|
||||
for root, dirs, files in os.walk(self.path):
|
||||
rel_base = root[len(self.path) + 1:]
|
||||
self.files.extend([os.path.join(rel_base, x) for x in files])
|
||||
return self.files
|
||||
|
||||
def is_equal(self, other_tree):
|
||||
"""
|
||||
Checks whether two dir tree contain the same files
|
||||
It checks the number of files and the hash of each file.
|
||||
|
||||
NOTE: tox puts .coverage files in the temp folder (?)
|
||||
|
||||
:param other_tree: dir tree to compare with
|
||||
:return: true if the dir trees contain the same files
|
||||
"""
|
||||
lh_files = [x for x in sorted(self.get_file_list())
|
||||
if not x.startswith('.coverage')]
|
||||
rh_files = [x for x in sorted(other_tree.get_file_list())
|
||||
if not x.startswith('.coverage')]
|
||||
if lh_files != rh_files:
|
||||
return False
|
||||
for fname in lh_files:
|
||||
if os.path.isfile(fname):
|
||||
if self.get_file_hash(fname) != \
|
||||
other_tree.get_file_hash(fname):
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
class TestFreezerScenario(BaseFreezerCliTest):
|
||||
def setUp(self):
|
||||
super(TestFreezerScenario, self).setUp()
|
||||
self.source_tree = Temp_Tree()
|
||||
self.source_tree.add_random_data()
|
||||
self.dest_tree = Temp_Tree()
|
||||
|
||||
self.cli.freezer_scheduler(action='start',
|
||||
flags='-c test_node '
|
||||
'-f /tmp/freezer_tempest_job_dir/')
|
||||
|
||||
def tearDown(self):
|
||||
super(TestFreezerScenario, self).tearDown()
|
||||
self.source_tree.cleanup()
|
||||
self.dest_tree.cleanup()
|
||||
|
||||
self.cli.freezer_scheduler(action='stop',
|
||||
flags='-c test_node '
|
||||
'-f /tmp/freezer_tempest_job_dir/')
|
||||
|
||||
def test_simple_backup(self):
|
||||
backup_job = {
|
||||
"client_id": "test_node",
|
||||
"job_actions": [
|
||||
{
|
||||
"freezer_action": {
|
||||
"action": "backup",
|
||||
"mode": "fs",
|
||||
"storage": "local",
|
||||
"backup_name": "backup1",
|
||||
"path_to_backup": self.source_tree.path,
|
||||
"container": "/tmp/freezer_test/",
|
||||
},
|
||||
"max_retries": 3,
|
||||
"max_retries_interval": 60
|
||||
}
|
||||
],
|
||||
"description": "a test backup"
|
||||
}
|
||||
restore_job = {
|
||||
"client_id": "test_node",
|
||||
"job_actions": [
|
||||
{
|
||||
"freezer_action": {
|
||||
"action": "restore",
|
||||
"storage": "local",
|
||||
"restore_abs_path": self.dest_tree.path,
|
||||
"backup_name": "backup1",
|
||||
"container": "/tmp/freezer_test/",
|
||||
},
|
||||
"max_retries": 3,
|
||||
"max_retries_interval": 60
|
||||
}
|
||||
],
|
||||
"description": "a test restore"
|
||||
}
|
||||
|
||||
backup_job_id = self.create_job(backup_job)
|
||||
self.cli.freezer_client(action='job-start', params=backup_job_id)
|
||||
self.wait_for_job_status(backup_job_id)
|
||||
self.assertJobColumnEqual(backup_job_id, JOB_TABLE_RESULT_COLUMN,
|
||||
'success')
|
||||
|
||||
restore_job_id = self.create_job(restore_job)
|
||||
self.wait_for_job_status(restore_job_id)
|
||||
self.assertJobColumnEqual(restore_job_id, JOB_TABLE_RESULT_COLUMN,
|
||||
'success')
|
||||
|
||||
self.assertTrue(self.source_tree.is_equal(self.dest_tree))
|
||||
|
|
@ -17,10 +17,10 @@ import subprocess
|
|||
from tempest.lib import decorators
|
||||
|
||||
from freezer import __version__ as freezer_version
|
||||
from freezer.tests.freezer_tempest_plugin.tests.api import base
|
||||
from freezer.tests.integration import common
|
||||
|
||||
|
||||
class TestFreezerVersion(base.BaseFreezerTest):
|
||||
class TestFreezerVersion(common.TestFS):
|
||||
|
||||
@decorators.attr(type="gate")
|
||||
def test_version(self):
|
||||
|
|
@ -35,6 +35,7 @@
|
|||
enable_plugin freezer-api https://git.openstack.org/openstack/freezer-api
|
||||
# enable freezer-web-ui and python-freezerclient
|
||||
enable_plugin freezer-web-ui https://git.openstack.org/openstack/freezer-web-ui
|
||||
TEMPEST_PLUGINS='/opt/stack/new/freezer-tempest-plugin'
|
||||
|
||||
EOF
|
||||
executable: /bin/bash
|
||||
|
|
@ -50,6 +51,7 @@
|
|||
export PROJECTS="openstack/freezer-web-ui $PROJECTS"
|
||||
export PROJECTS="openstack/freezer $PROJECTS"
|
||||
export PROJECTS="openstack/python-freezerclient $PROJECTS"
|
||||
export PROJECTS="openstack/neutron-tempest-plugin $PROJECTS"
|
||||
# tempest config
|
||||
export DEVSTACK_GATE_TEMPEST=1
|
||||
export DEVSTACK_GATE_TEMPEST_ALL_PLUGINS=1
|
||||
|
|
|
|||
|
|
@ -35,6 +35,7 @@
|
|||
enable_plugin freezer-api https://git.openstack.org/openstack/freezer-api
|
||||
# enable freezer-web-ui and python-freezerclient
|
||||
enable_plugin freezer-web-ui https://git.openstack.org/openstack/freezer-web-ui
|
||||
TEMPEST_PLUGINS='/opt/stack/new/freezer-tempest-plugin'
|
||||
|
||||
EOF
|
||||
executable: /bin/bash
|
||||
|
|
@ -50,6 +51,7 @@
|
|||
export PROJECTS="openstack/freezer-web-ui $PROJECTS"
|
||||
export PROJECTS="openstack/freezer $PROJECTS"
|
||||
export PROJECTS="openstack/python-freezerclient $PROJECTS"
|
||||
export PROJECTS="openstack/freezer-tempest-plugin $PROJECTS"
|
||||
# tempest config
|
||||
export DEVSTACK_GATE_TEMPEST=1
|
||||
export DEVSTACK_GATE_TEMPEST_ALL_PLUGINS=1
|
||||
|
|
|
|||
|
|
@ -35,6 +35,7 @@
|
|||
enable_plugin freezer-api https://git.openstack.org/openstack/freezer-api
|
||||
# enable freezer-web-ui and python-freezerclient
|
||||
enable_plugin freezer-web-ui https://git.openstack.org/openstack/freezer-web-ui
|
||||
TEMPEST_PLUGINS='/opt/stack/new/freezer-tempest-plugin'
|
||||
|
||||
EOF
|
||||
executable: /bin/bash
|
||||
|
|
@ -50,6 +51,7 @@
|
|||
export PROJECTS="openstack/freezer-web-ui $PROJECTS"
|
||||
export PROJECTS="openstack/freezer $PROJECTS"
|
||||
export PROJECTS="openstack/python-freezerclient $PROJECTS"
|
||||
export PROJECTS="openstack/freezer-tempest-plugin $PROJECTS"
|
||||
# tempest config
|
||||
export DEVSTACK_GATE_TEMPEST=1
|
||||
export DEVSTACK_GATE_TEMPEST_ALL_PLUGINS=1
|
||||
|
|
|
|||
|
|
@ -63,8 +63,6 @@ oslo.config.opts =
|
|||
console_scripts =
|
||||
freezer-scheduler = freezer.scheduler.freezer_scheduler:main
|
||||
freezer-agent = freezer.main:main
|
||||
tempest.test_plugins =
|
||||
freezer_tempest_tests = freezer.tests.freezer_tempest_plugin.plugin:FreezerTempestPlugin
|
||||
|
||||
|
||||
[pbr]
|
||||
|
|
@ -72,4 +70,4 @@ tempest.test_plugins =
|
|||
autodoc_index_modules = True
|
||||
api_doc_dir = reference/api
|
||||
autodoc_exclude_modules =
|
||||
freezer.scheduler.win*
|
||||
freezer.scheduler.win*
|
||||
|
|
|
|||
Loading…
Reference in New Issue