openstack
/
blazar
Code Issues Proposed changes

Support for DB migrations using Alembic 

Added support for DB migration using Alembic.
Modified devstack/lib/climate to sync the database during the
configuration of Climate.

Added the actual state of the sqlalchemy models in a migration script.
This will be the first version, and will be used by devstack and the
migration tests.

Added a new console_script to the setup.cfg: climate-db-manage. This is
the CLI is used as a wrapper of the alembic functionality.

Added alembic>=0.4.1 as dependecy.

Added some README with documentation about the DB migrations.

Change-Id: I390ccfac1e436db0b04339e60f9f6795b22b8f7e
Implements: blueprint schema-data-migration-with-alembic
This commit is contained in:
Pablo Andres Fuente 2014-02-17 18:02:46 -03:00
parent abf12fb2ed
commit ce13bf6c5d
13 changed files with 1206 additions and 12 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

75
climate/db/migration/README Normal file
View File

@ -0,0 +1,75 @@
# Copyright 2012 New Dream Network, LLC (DreamHost)
# Copyright 2014 Intel Corporation
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
Climate project uses Alembic to handle database migrations. A migration occurs
by executing a script that details the changes needed to upgrade/downgrade
the database. The migration scripts are ordered so that multiple scripts
can run sequentially to update the database.
You can then upgrade to the latest database version via:
$ climate-db-manage --config-file /path/to/climate.conf upgrade head
To check the current database version:
$ climate-db-manage --config-file /path/to/climate.conf current
To create a script to run the migration offline:
$ climate-db-manage --config-file /path/to/climate.conf upgrade head --sql
To run the offline migration between specific migration versions:
$ climate-db-manage --config-file /path/to/climate.conf upgrade \
<start version>:<end version> --sql
Upgrade the database incrementally:
$ climate-db-manage --config-file /path/to/climate.conf \
upgrade --delta <# of revs>
Downgrade the database by a certain number of revisions:
$ climate-db-manage --config-file /path/to/climate.conf downgrade \
--delta <# of revs>
DEVELOPERS:
A database migration script is required when you submit a change to Climate
that alters the database model definition. The migration script is a special
python file that includes code to update/downgrade the database to match the
changes in the model definition. Alembic will execute these scripts in order to
provide a linear migration path between revision. The climate-db-manage command
can be used to generate migration template for you to complete. The operations
in the template are those supported by the Alembic migration library.
After you modified the Climate models acordingly, you can create the revision.
$ climate-db-manage --config-file /path/to/climate.conf revision \
-m "description of revision" \
--autogenerate
This generates a prepopulated template with the changes needed to match the
database state with the models. You should inspect the autogenerated template
to ensure that the proper models have been altered.
In rare circumstances, you may want to start with an empty migration template
and manually author the changes necessary for an upgrade/downgrade. You can
create a blank file via:
$ climate-db-manage --config-file /path/to/climate.conf revision \
-m "description of revision"
The migration timeline should remain linear so that there is a clear path when
upgrading/downgrading. To verify that the timeline does branch, you can run
this command:
$ climate-db-manage --config-file /path/to/climate.conf check_migration
If the migration path does branch, you can find the branch point via:
$ climate-db-manage --config-file /path/to/climate.conf history

53
climate/db/migration/alembic.ini Normal file
View File

@ -0,0 +1,53 @@
# A generic, single database configuration.
[alembic]
# path to migration scripts
script_location = %(here)s/alembic_migrations
# template used to generate migration files
# file_template = %%(rev)s_%%(slug)s
# set to 'true' to run the environment during
# the 'revision' command, regardless of autogenerate
# revision_environment = false
# default to an empty string because the Climate migration cli will
# extract the correct value and set it programatically before alembic is fully
# invoked.
sqlalchemy.url =
# Logging configuration
[loggers]
keys = root,sqlalchemy,alembic
[handlers]
keys = console
[formatters]
keys = generic
[logger_root]
level = WARN
handlers = console
qualname =
[logger_sqlalchemy]
level = WARN
handlers =
qualname = sqlalchemy.engine
[logger_alembic]
level = INFO
handlers =
qualname = alembic
[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic
[formatter_generic]
format = %(levelname)-5.5s [%(name)s] %(message)s
datefmt = %H:%M:%S

84
climate/db/migration/alembic_migrations/env.py Normal file
View File

@ -0,0 +1,84 @@
# Copyright 2014 Intel Corporation
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
from alembic import context
from sqlalchemy import create_engine, pool
from logging import config as log_config
from climate.db.sqlalchemy import model_base
from climate.db.sqlalchemy import models # noqa
# this is the Alembic Config object, which provides
# access to the values within the .ini file in use.
config = context.config
# Interpret the config file for Python logging.
# This line sets up loggers basically.
log_config.fileConfig(config.config_file_name)
# add your model's MetaData object here
# for 'autogenerate' support
# from myapp import mymodel
# target_metadata = mymodel.Base.metadata
target_metadata = model_base.ClimateBase.metadata
# other values from the config, defined by the needs of env.py,
# can be acquired:
# my_important_option = config.get_main_option("my_important_option")
# ... etc.
def run_migrations_offline(config):
"""Run migrations in 'offline' mode.
This configures the context with just a URL
and not an Engine, though an Engine is acceptable
here as well. By skipping the Engine creation
we don't even need a DBAPI to be available.
Calls to context.execute() here emit the given string to the
script output.
"""
url = config.database.connection
context.configure(url=url)
with context.begin_transaction():
context.run_migrations()
def run_migrations_online(config):
"""Run migrations in 'online' mode.
In this scenario we need to create an Engine
and associate a connection with the context.
"""
engine = create_engine(config.database.connection,
poolclass=pool.NullPool)
connection = engine.connect()
context.configure(connection=connection,
target_metadata=target_metadata)
try:
with context.begin_transaction():
context.run_migrations()
finally:
connection.close()
if context.is_offline_mode():
run_migrations_offline(config.climate_config)
else:
run_migrations_online(config.climate_config)

37
climate/db/migration/alembic_migrations/script.py.mako Normal file
View File

@ -0,0 +1,37 @@
# Copyright ${create_date.year} OpenStack Foundation.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""${message}
Revision ID: ${up_revision}
Revises: ${down_revision}
Create Date: ${create_date}
"""
# revision identifiers, used by Alembic.
revision = ${repr(up_revision)}
down_revision = ${repr(down_revision)}
from alembic import op
import sqlalchemy as sa
${imports if imports else ""}
def upgrade():
${upgrades if upgrades else "pass"}
def downgrade():
${downgrades if downgrades else "pass"}

145
climate/db/migration/alembic_migrations/versions/0_1_initial.py Normal file
View File

@ -0,0 +1,145 @@
# Copyright 2014 OpenStack Foundation.
# Copyright 2014 Intel Corporation
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""Icehouse Initial
Revision ID: 0_1
Revises: None
Create Date: 2014-02-19 17:23:47.705197
"""
# revision identifiers, used by Alembic.
revision = '0_1'
down_revision = None
from alembic import op
import sqlalchemy as sa
from sqlalchemy.dialects.mysql import MEDIUMTEXT
from climate.openstack.common import uuidutils
def _generate_unicode_uuid():
return unicode(uuidutils.generate_uuid())
def MediumText():
return sa.Text().with_variant(MEDIUMTEXT(), 'mysql')
def _id_column():
return sa.Column('id', sa.String(36), primary_key=True,
default=_generate_unicode_uuid)
def upgrade():
op.create_table(
'computehosts',
sa.Column('created_at', sa.DateTime(), nullable=True),
sa.Column('updated_at', sa.DateTime(), nullable=True),
_id_column(),
sa.Column('vcpus', sa.Integer(), nullable=False),
sa.Column('cpu_info', MediumText(), nullable=False),
sa.Column('hypervisor_type', MediumText(), nullable=False),
sa.Column('hypervisor_version', sa.Integer(), nullable=False),
sa.Column('hypervisor_hostname', sa.String(length=255), nullable=True),
sa.Column('memory_mb', sa.Integer(), nullable=False),
sa.Column('local_gb', sa.Integer(), nullable=False),
sa.Column('status', sa.String(length=13)),
sa.PrimaryKeyConstraint('id'))
op.create_table(
'leases',
sa.Column('created_at', sa.DateTime(), nullable=True),
sa.Column('updated_at', sa.DateTime(), nullable=True),
_id_column(),
sa.Column('name', sa.String(length=80), nullable=False),
sa.Column('user_id', sa.String(length=255), nullable=True),
sa.Column('tenant_id', sa.String(length=255), nullable=True),
sa.Column('start_date', sa.DateTime(), nullable=False),
sa.Column('end_date', sa.DateTime(), nullable=False),
sa.Column('trust_id', sa.String(length=36)),
sa.PrimaryKeyConstraint('id'),
sa.UniqueConstraint('name'))
op.create_table(
'reservations',
sa.Column('created_at', sa.DateTime(), nullable=True),
sa.Column('updated_at', sa.DateTime(), nullable=True),
_id_column(),
sa.Column('lease_id', sa.String(length=36), nullable=False),
sa.Column('resource_id', sa.String(length=36)),
sa.Column('resource_type', sa.String(length=66)),
sa.Column('status', sa.String(length=13)),
sa.ForeignKeyConstraint(['lease_id'], ['leases.id'], ),
sa.PrimaryKeyConstraint('id'))
op.create_table(
'computehost_extra_capabilities',
sa.Column('created_at', sa.DateTime(), nullable=True),
sa.Column('updated_at', sa.DateTime(), nullable=True),
_id_column(),
sa.Column('computehost_id', sa.String(length=36), nullable=True),
sa.Column('capability_name', sa.String(length=64), nullable=False),
sa.Column('capability_value', MediumText(), nullable=False),
sa.ForeignKeyConstraint(['computehost_id'], ['computehosts.id'], ),
sa.PrimaryKeyConstraint('id'))
op.create_table(
'events',
sa.Column('created_at', sa.DateTime(), nullable=True),
sa.Column('updated_at', sa.DateTime(), nullable=True),
_id_column(),
sa.Column('lease_id', sa.String(length=36), nullable=True),
sa.Column('event_type', sa.String(length=66)),
sa.Column('time', sa.DateTime()),
sa.Column('status', sa.String(length=13)),
sa.ForeignKeyConstraint(['lease_id'], ['leases.id'], ),
sa.PrimaryKeyConstraint('id'))
op.create_table(
'computehost_allocations',
sa.Column('created_at', sa.DateTime(), nullable=True),
sa.Column('updated_at', sa.DateTime(), nullable=True),
_id_column(),
sa.Column('compute_host_id', sa.String(length=36), nullable=True),
sa.Column('reservation_id', sa.String(length=36), nullable=True),
sa.ForeignKeyConstraint(['compute_host_id'], ['computehosts.id'], ),
sa.ForeignKeyConstraint(['reservation_id'], ['reservations.id'], ),
sa.PrimaryKeyConstraint('id'))
op.create_table(
'computehost_reservations',
sa.Column('created_at', sa.DateTime(), nullable=True),
sa.Column('updated_at', sa.DateTime(), nullable=True),
_id_column(),
sa.Column('reservation_id', sa.String(length=36), nullable=True),
sa.Column('resource_properties', MediumText()),
sa.Column('count_range', sa.String(length=36)),
sa.Column('hypervisor_properties', MediumText()),
sa.Column('status', sa.String(length=13)),
sa.ForeignKeyConstraint(['reservation_id'], ['reservations.id'], ),
sa.PrimaryKeyConstraint('id'))
def downgrade():
op.drop_table('computehost_extra_capabilities')
op.drop_table('computehost_allocations')
op.drop_table('computehost_reservations')
op.drop_table('computehosts')
op.drop_table('reservations')
op.drop_table('events')
op.drop_table('leases')

3
climate/db/migration/alembic_migrations/versions/README Normal file
View File

@ -0,0 +1,3 @@
This directory contains the migration scripts for the Climate project. Please
see the README in climate/db/migration on how to use and generate new
migrations.

91
climate/db/migration/cli.py
View File

@ -13,32 +13,92 @@
# See the License for the specific language governing permissions and
# limitations under the License.
"""CLI tool to manage the Climate DB. Inspired by Neutron's same tool."""
import gettext
import os
from alembic import command as alembic_command
from alembic import config as alembic_config
from alembic import util as alembic_util
from oslo.config import cfg
from climate.openstack.common.db import options as db_options
gettext.install('climate', unicode=1)
from climate.db import api as db_api
from climate.openstack.common.gettextutils import _
CONF = cfg.CONF
def map_status(status):
return 'Success' if status else 'Fail'
def do_alembic_command(config, cmd, *args, **kwargs):
try:
getattr(alembic_command, cmd)(config, *args, **kwargs)
except alembic_util.CommandError as e:
alembic_util.err(str(e))
def db_sync():
drop_status = db_api.drop_db()
print(_("Dropping database: %s") % map_status(drop_status))
start_status = db_api.setup_db()
print(_("Creating database: %s") % map_status(start_status))
def do_check_migration(config, cmd):
do_alembic_command(config, 'branches')
def do_upgrade_downgrade(config, cmd):
if not CONF.command.revision and not CONF.command.delta:
raise SystemExit(_('You must provide a revision or relative delta'))
revision = CONF.command.revision
if CONF.command.delta:
sign = '+' if CONF.command.name == 'upgrade' else '-'
revision = sign + str(CONF.command.delta)
else:
revision = CONF.command.revision
do_alembic_command(config, cmd, revision, sql=CONF.command.sql)
def do_stamp(config, cmd):
do_alembic_command(config, cmd,
CONF.command.revision,
sql=CONF.command.sql)
def do_revision(config, cmd):
do_alembic_command(config, cmd,
message=CONF.command.message,
autogenerate=CONF.command.autogenerate,
sql=CONF.command.sql)
def add_command_parsers(subparsers):
parser = subparsers.add_parser('db-sync')
parser.set_defaults(func=db_sync)
for name in ['current', 'history', 'branches']:
parser = subparsers.add_parser(name)
parser.set_defaults(func=do_alembic_command)
parser = subparsers.add_parser('check_migration')
parser.set_defaults(func=do_check_migration)
for name in ['upgrade', 'downgrade']:
parser = subparsers.add_parser(name)
parser.add_argument('--delta', type=int)
parser.add_argument('--sql', action='store_true')
parser.add_argument('revision', nargs='?')
parser.set_defaults(func=do_upgrade_downgrade)
parser = subparsers.add_parser('stamp')
parser.add_argument('--sql', action='store_true')
parser.add_argument('revision')
parser.set_defaults(func=do_stamp)
parser = subparsers.add_parser('revision')
parser.add_argument('-m', '--message')
parser.add_argument('--autogenerate', action='store_true')
parser.add_argument('--sql', action='store_true')
parser.set_defaults(func=do_revision)
command_opt = cfg.SubCommandOpt('command',
@ -51,5 +111,14 @@ CONF.register_cli_opt(command_opt)
def main():
config = alembic_config.Config(
os.path.join(os.path.dirname(__file__), 'alembic.ini')
)
config.climate_config = CONF
CONF()
CONF.command.func()
if not db_options.CONF.database.connection:
raise SystemExit(
_("Provide a configuration file with DB connection information"))
CONF.command.func(config, CONF.command.name)

583
climate/tests/db/migration/__init__.py Normal file
View File

@ -0,0 +1,583 @@
# Copyright 2010-2011 OpenStack Foundation
# Copyright 2012-2013 IBM Corp.
# All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
#
#
# Ripped off from Nova's test_migrations.py
# The only difference between Nova and this code is usage of alembic instead
# of sqlalchemy migrations.
#
# There is an ongoing work to extact similar code to oslo incubator. Once it is
# extracted we'll be able to remove this file and use oslo.
import ConfigParser
import io
import os
import subprocess
import urlparse
from alembic import command
from alembic import config as alembic_config
from alembic import migration
from oslo.config import cfg
import sqlalchemy
import sqlalchemy.exc
import climate.db.migration
from climate import tests
from climate.openstack.common import lockutils
from climate.openstack.common import log as logging
LOG = logging.getLogger(__name__)
CONF = cfg.CONF
synchronized = lockutils.synchronized_with_prefix('climate-')
def _get_connect_string(backend, user, passwd, database):
"""Try to get a connection with a very specific set of values, if we get
these then we'll run the tests, otherwise they are skipped
"""
if backend == "postgres":
backend = "postgresql+psycopg2"
elif backend == "mysql":
backend = "mysql+mysqldb"
else:
raise Exception("Unrecognized backend: '%s'" % backend)
return ("%s://%s:%s@localhost/%s" % (backend, user, passwd, database))
def _is_backend_avail(backend, user, passwd, database):
try:
connect_uri = _get_connect_string(backend, user, passwd, database)
engine = sqlalchemy.create_engine(connect_uri)
connection = engine.connect()
except Exception:
# intentionally catch all to handle exceptions even if we don't
# have any backend code loaded.
return False
else:
connection.close()
engine.dispose()
return True
def _have_mysql(user, passwd, database):
present = os.environ.get('CLIMATE_MYSQL_PRESENT')
if present is None:
return _is_backend_avail('mysql', user, passwd, database)
return present.lower() in ('', 'true')
def _have_postgresql(user, passwd, database):
present = os.environ.get('CLIMATE_TEST_POSTGRESQL_PRESENT')
if present is None:
return _is_backend_avail('postgres', user, passwd, database)
return present.lower() in ('', 'true')
def get_mysql_connection_info(conn_pieces):
database = conn_pieces.path.strip('/')
loc_pieces = conn_pieces.netloc.split('@')
host = loc_pieces[1]
auth_pieces = loc_pieces[0].split(':')
user = auth_pieces[0]
password = ""
if len(auth_pieces) > 1:
if auth_pieces[1].strip():
password = "-p\"%s\"" % auth_pieces[1]
return (user, password, database, host)
def get_pgsql_connection_info(conn_pieces):
database = conn_pieces.path.strip('/')
loc_pieces = conn_pieces.netloc.split('@')
host = loc_pieces[1]
auth_pieces = loc_pieces[0].split(':')
user = auth_pieces[0]
password = ""
if len(auth_pieces) > 1:
password = auth_pieces[1].strip()
return (user, password, database, host)
class CommonTestsMixIn(object):
"""BaseMigrationTestCase is effectively an abstract class,
meant to be derived from and not directly tested against; that's why these
`test_` methods need to be on a Mixin, so that they won't be picked up
as valid tests for BaseMigrationTestCase.
"""
def test_walk_versions(self):
for key, engine in self.engines.items():
# We start each walk with a completely blank slate.
self._reset_database(key)
self._walk_versions(engine, self.snake_walk, self.downgrade)
def test_mysql_opportunistically(self):
self._test_mysql_opportunistically()
def test_mysql_connect_fail(self):
"""Test that we can trigger a mysql connection failure and we fail
gracefully to ensure we don't break people without mysql
"""
if _is_backend_avail('mysql', "openstack_cifail", self.PASSWD,
self.DATABASE):
self.fail("Shouldn't have connected")
def test_postgresql_opportunistically(self):
self._test_postgresql_opportunistically()
def test_postgresql_connect_fail(self):
"""Test that we can trigger a postgres connection failure and we fail
gracefully to ensure we don't break people without postgres
"""
if _is_backend_avail('postgres', "openstack_cifail", self.PASSWD,
self.DATABASE):
self.fail("Shouldn't have connected")
class BaseMigrationTestCase(tests.TestCase):
"""Base class for testing migrations and migration utils. This sets up
and configures the databases to run tests against.
"""
# NOTE(jhesketh): It is expected that tests clean up after themselves.
# This is necessary for concurrency to allow multiple tests to work on
# one database.
# The full migration walk tests however do call the old _reset_databases()
# to throw away whatever was there so they need to operate on their own
# database that we know isn't accessed concurrently.
# Hence, BaseWalkMigrationTestCase overwrites the engine list.
USER = None
PASSWD = None
DATABASE = None
TIMEOUT_SCALING_FACTOR = 2
def __init__(self, *args, **kwargs):
super(BaseMigrationTestCase, self).__init__(*args, **kwargs)
self.DEFAULT_CONFIG_FILE = os.path.join(
os.path.dirname(__file__),
'test_migrations.conf')
# Test machines can set the CLIMATE_TEST_MIGRATIONS_CONF variable
# to override the location of the config file for migration testing
self.CONFIG_FILE_PATH = os.environ.get(
'CLIMATE_TEST_MIGRATIONS_CONF',
self.DEFAULT_CONFIG_FILE)
self.ALEMBIC_CONFIG = alembic_config.Config(
os.path.join(os.path.dirname(climate.db.migration.__file__),
'alembic.ini')
)
self.ALEMBIC_CONFIG.climate_config = CONF
self.snake_walk = False
self.downgrade = False
self.test_databases = {}
self.migration = None
self.migration_api = None
def setUp(self):
super(BaseMigrationTestCase, self).setUp()
self._load_config()
def _load_config(self):
# Load test databases from the config file. Only do this
# once. No need to re-run this on each test...
LOG.debug('config_path is %s' % self.CONFIG_FILE_PATH)
if os.path.exists(self.CONFIG_FILE_PATH):
cp = ConfigParser.RawConfigParser()
try:
cp.read(self.CONFIG_FILE_PATH)
config = cp.options('unit_tests')
for key in config:
self.test_databases[key] = cp.get('unit_tests', key)
self.snake_walk = cp.getboolean('walk_style', 'snake_walk')
self.downgrade = cp.getboolean('walk_style', 'downgrade')
except ConfigParser.ParsingError as e:
self.fail("Failed to read test_migrations.conf config "
"file. Got error: %s" % e)
else:
self.fail("Failed to find test_migrations.conf config "
"file.")
self.engines = {}
for key, value in self.test_databases.items():
self.engines[key] = sqlalchemy.create_engine(value)
# NOTE(jhesketh): We only need to make sure the databases are created
# not necessarily clean of tables.
self._create_databases()
def execute_cmd(self, cmd=None):
process = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
output = process.communicate()[0]
LOG.debug(output)
self.assertEqual(0, process.returncode,
"Failed to run: %s\n%s" % (cmd, output))
@synchronized('pgadmin', external=True, lock_path='/tmp')
def _reset_pg(self, conn_pieces):
(user, password, database, host) = \
get_pgsql_connection_info(conn_pieces)
os.environ['PGPASSWORD'] = password
os.environ['PGUSER'] = user
# note(boris-42): We must create and drop database, we can't
# drop database which we have connected to, so for such
# operations there is a special database template1.
sqlcmd = ("psql -w -U %(user)s -h %(host)s -c"
" '%(sql)s' -d template1")
sqldict = {'user': user, 'host': host}
sqldict['sql'] = ("drop database if exists %s;") % database
droptable = sqlcmd % sqldict
self.execute_cmd(droptable)
sqldict['sql'] = ("create database %s;") % database
createtable = sqlcmd % sqldict
self.execute_cmd(createtable)
os.unsetenv('PGPASSWORD')
os.unsetenv('PGUSER')
@synchronized('mysql', external=True, lock_path='/tmp')
def _reset_mysql(self, conn_pieces):
# We can execute the MySQL client to destroy and re-create
# the MYSQL database, which is easier and less error-prone
# than using SQLAlchemy to do this via MetaData...trust me.
(user, password, database, host) = \
get_mysql_connection_info(conn_pieces)
sql = ("drop database if exists %(database)s; "
"create database %(database)s;" % {'database': database})
cmd = ("mysql -u \"%(user)s\" %(password)s -h %(host)s -e \"%(sql)s\""
% {'user': user, 'password': password,
'host': host, 'sql': sql})
self.execute_cmd(cmd)
@synchronized('sqlite', external=True, lock_path='/tmp')
def _reset_sqlite(self, conn_pieces):
# We can just delete the SQLite database, which is
# the easiest and cleanest solution
db_path = conn_pieces.path.strip('/')
if os.path.exists(db_path):
os.unlink(db_path)
# No need to recreate the SQLite DB. SQLite will
# create it for us if it's not there...
def _create_databases(self):
"""Create all configured databases as needed."""
for key, engine in self.engines.items():
self._create_database(key)
def _create_database(self, key):
"""Create database if it doesn't exist."""
conn_string = self.test_databases[key]
conn_pieces = urlparse.urlparse(conn_string)
if conn_string.startswith('mysql'):
(user, password, database, host) = \
get_mysql_connection_info(conn_pieces)
sql = "create database if not exists %s;" % database
cmd = ("mysql -u \"%(user)s\" %(password)s -h %(host)s "
"-e \"%(sql)s\"" % {'user': user, 'password': password,
'host': host, 'sql': sql})
self.execute_cmd(cmd)
elif conn_string.startswith('postgresql'):
(user, password, database, host) = \
get_pgsql_connection_info(conn_pieces)
os.environ['PGPASSWORD'] = password
os.environ['PGUSER'] = user
sqlcmd = ("psql -w -U %(user)s -h %(host)s -c"
" '%(sql)s' -d template1 -A -t")
sql = ("select count(*) from pg_database WHERE datname = '%s'") \
% database
check_database = sqlcmd % {'user': user, 'host': host, 'sql': sql}
process = subprocess.Popen(check_database, shell=True,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
output = process.communicate()[0]
if output == '1':
sql = ("create database %s;") % database
create_database = sqlcmd % {'user': user,
'host': host, 'sql': sql}
self.execute_cmd(create_database)
os.unsetenv('PGPASSWORD')
os.unsetenv('PGUSER')
def _reset_databases(self):
"""Reset all configured databases."""
for key, engine in self.engines.items():
self._reset_database(key)
def _reset_database(self, key):
"""Reset specific database."""
engine = self.engines[key]
conn_string = self.test_databases[key]
conn_pieces = urlparse.urlparse(conn_string)
engine.dispose()
if conn_string.startswith('sqlite'):
self._reset_sqlite(conn_pieces)
elif conn_string.startswith('mysql'):
self._reset_mysql(conn_pieces)
elif conn_string.startswith('postgresql'):
self._reset_pg(conn_pieces)
class BaseWalkMigrationTestCase(BaseMigrationTestCase):
"""BaseWalkMigrationTestCase loads in an alternative set of databases for
testing against. This is necessary as the default databases can run tests
concurrently without interfering with itself. It is expected that
databases listed under [migration_dbs] in the configuration are only being
accessed by one test at a time. Currently only test_walk_versions accesses
the databases (and is the only method that calls _reset_database() which
is clearly problematic for concurrency).
"""
def _load_config(self):
# Load test databases from the config file. Only do this
# once. No need to re-run this on each test...
LOG.debug('config_path is %s' % self.CONFIG_FILE_PATH)
if os.path.exists(self.CONFIG_FILE_PATH):
cp = ConfigParser.RawConfigParser()
try:
cp.read(self.CONFIG_FILE_PATH)
config = cp.options('migration_dbs')
for key in config:
self.test_databases[key] = cp.get('migration_dbs', key)
self.snake_walk = cp.getboolean('walk_style', 'snake_walk')
self.downgrade = cp.getboolean('walk_style', 'downgrade')
except ConfigParser.ParsingError as e:
self.fail("Failed to read test_migrations.conf config "
"file. Got error: %s" % e)
else:
self.fail("Failed to find test_migrations.conf config "
"file.")
self.engines = {}
for key, value in self.test_databases.items():
self.engines[key] = sqlalchemy.create_engine(value)
self._create_databases()
def _configure(self, engine):
"""For each type of repository we should do some of configure steps.
For migrate_repo we should set under version control our database.
For alembic we should configure database settings. For this goal we
should use oslo.config and openstack.commom.db.sqlalchemy.session with
database functionality (reset default settings).
"""
CONF.set_override('connection', str(engine.url), group='database')
def _test_mysql_opportunistically(self):
# Test that table creation on mysql only builds InnoDB tables
if not _have_mysql(self.USER, self.PASSWD, self.DATABASE):
self.skipTest("mysql not available")
# add this to the global lists to make reset work with it, it's removed
# automatically in tearDown so no need to clean it up here.
connect_string = _get_connect_string(
"mysql", self.USER, self.PASSWD, self.DATABASE)
(user, password, database, host) = \
get_mysql_connection_info(urlparse.urlparse(connect_string))
engine = sqlalchemy.create_engine(connect_string)
self.engines[database] = engine
self.test_databases[database] = connect_string
# build a fully populated mysql database with all the tables
self._reset_database(database)
self._walk_versions(engine, self.snake_walk, self.downgrade)
connection = engine.connect()
# sanity check
total = connection.execute("SELECT count(*) "
"from information_schema.TABLES "
"where TABLE_SCHEMA='%(database)s'" %
{'database': database})
self.assertTrue(total.scalar() > 0, "No tables found. Wrong schema?")
connection.close()
del(self.engines[database])
del(self.test_databases[database])
def _test_postgresql_opportunistically(self):
# Test postgresql database migration walk
if not _have_postgresql(self.USER, self.PASSWD, self.DATABASE):
self.skipTest("postgresql not available")
# add this to the global lists to make reset work with it, it's removed
# automatically in tearDown so no need to clean it up here.
connect_string = _get_connect_string(
"postgres", self.USER, self.PASSWD, self.DATABASE)
engine = sqlalchemy.create_engine(connect_string)
(user, password, database, host) = \
get_mysql_connection_info(urlparse.urlparse(connect_string))
self.engines[database] = engine
self.test_databases[database] = connect_string
# build a fully populated postgresql database with all the tables
self._reset_database(database)
self._walk_versions(engine, self.snake_walk, self.downgrade)
del(self.engines[database])
del(self.test_databases[database])
def _alembic_command(self, alembic_command, engine, *args, **kwargs):
"""Most of alembic command return data into output.
We should redefine this setting for getting info.
"""
self.ALEMBIC_CONFIG.stdout = buf = io.StringIO()
CONF.set_override('connection', str(engine.url), group='database')
getattr(command, alembic_command)(*args, **kwargs)
res = buf.getvalue().strip()
LOG.debug('Alembic command `%s` returns: %s' % (alembic_command, res))
return res
def _get_alembic_versions(self, engine):
"""For support of full testing of migrations
we should have an opportunity to run command step by step for each
version in repo. This method returns list of alembic_versions by
historical order.
"""
full_history = self._alembic_command('history',
engine, self.ALEMBIC_CONFIG)
# The piece of output data with version can looked as:
# 'Rev: 17738166b91 (head)' or 'Rev: 43b1a023dfaa'
alembic_history = [r.split(' ')[1] for r in full_history.split("\n")
if r.startswith("Rev")]
alembic_history.reverse()
return alembic_history
def _up_and_down_versions(self, engine):
"""Since alembic version has a random algorithm of generation
(SA-migrate has an ordered autoincrement naming) we should store
a tuple of versions (version for upgrade and version for downgrade)
for successful testing of migrations in up>down>up mode.
"""
versions = self._get_alembic_versions(engine)
return zip(versions, ['-1'] + versions)
def _walk_versions(self, engine=None, snake_walk=False,
downgrade=True):
# Determine latest version script from the repo, then
# upgrade from 1 through to the latest, with no data
# in the databases. This just checks that the schema itself
# upgrades successfully.
self._configure(engine)
up_and_down_versions = self._up_and_down_versions(engine)
for ver_up, ver_down in up_and_down_versions:
# upgrade -> downgrade -> upgrade
self._migrate_up(engine, ver_up, with_data=True)
if snake_walk:
downgraded = self._migrate_down(engine,
ver_down,
with_data=True,
next_version=ver_up)
if downgraded:
self._migrate_up(engine, ver_up)
if downgrade:
# Now walk it back down to 0 from the latest, testing
# the downgrade paths.
up_and_down_versions.reverse()
for ver_up, ver_down in up_and_down_versions:
# downgrade -> upgrade -> downgrade
downgraded = self._migrate_down(engine,
ver_down, next_version=ver_up)
if snake_walk and downgraded:
self._migrate_up(engine, ver_up)
self._migrate_down(engine, ver_down, next_version=ver_up)
def _get_version_from_db(self, engine):
"""For each type of migrate repo latest version from db
will be returned.
"""
conn = engine.connect()
try:
context = migration.MigrationContext.configure(conn)
version = context.get_current_revision() or '-1'
finally:
conn.close()
return version
def _migrate(self, engine, version, cmd):
"""Base method for manipulation with migrate repo.
It will upgrade or downgrade the actual database.
"""
self._alembic_command(cmd, engine, self.ALEMBIC_CONFIG, version)
def _migrate_down(self, engine, version, with_data=False,
next_version=None):
try:
self._migrate(engine, version, 'downgrade')
except NotImplementedError:
# NOTE(sirp): some migrations, namely release-level
# migrations, don't support a downgrade.
return False
self.assertEqual(version, self._get_version_from_db(engine))
# NOTE(sirp): `version` is what we're downgrading to (i.e. the 'target'
# version). So if we have any downgrade checks, they need to be run for
# the previous (higher numbered) migration.
if with_data:
post_downgrade = getattr(
self, "_post_downgrade_%s" % next_version, None)
if post_downgrade:
post_downgrade(engine)
return True
def _migrate_up(self, engine, version, with_data=False):
"""migrate up to a new version of the db.
We allow for data insertion and post checks at every
migration version with special _pre_upgrade_### and
_check_### functions in the main test.
"""
# NOTE(sdague): try block is here because it's impossible to debug
# where a failed data migration happens otherwise
check_version = version
try:
if with_data:
data = None
pre_upgrade = getattr(
self, "_pre_upgrade_%s" % check_version, None)
if pre_upgrade:
data = pre_upgrade(engine)
self._migrate(engine, version, 'upgrade')
self.assertEqual(version, self._get_version_from_db(engine))
if with_data:
check = getattr(self, "_check_%s" % check_version, None)
if check:
check(engine, data)
except Exception:
LOG.error("Failed to migrate to version %s on engine %s" %
(version, engine))
raise

24
climate/tests/db/migration/test_migrations.conf Normal file
View File

@ -0,0 +1,24 @@
[unit_tests]
# Set up any number of databases to test concurrently.
# The "name" used in the test is the config variable key.
# A few tests rely on one sqlite database with 'sqlite' as the key.
sqlitefile=sqlite:///test_migrations_utils.db
#mysql=mysql+mysqldb://user:pass@localhost/test_migrations_utils
#postgresql=postgresql+psycopg2://user:pass@localhost/test_migrations_utils
[migration_dbs]
# Migration DB details are listed separately as they can't be connected to
# concurrently. These databases can't be the same as above
# Note, sqlite:// is in-memory and unique each time it is spawned.
# However file sqlite's are not unique.
sqlitefile=sqlite:///test_migrations.db
#mysql=mysql+mysqldb://user:pass@localhost/test_migrations
#postgresql=postgresql+psycopg2://user:pass@localhost/test_migrations
[walk_style]
snake_walk=yes
downgrade=yes

114
climate/tests/db/migration/test_migrations.py Normal file
View File

@ -0,0 +1,114 @@
# Copyright 2014 OpenStack Foundation
# Copyright 2014 Mirantis Inc
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may
# not use this file except in compliance with the License. You may obtain
# a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
# WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
# License for the specific language governing permissions and limitations
# under the License.
"""
Tests for database migrations. This test case reads the configuration
file test_migrations.conf for database connection settings
to use in the tests. For each connection found in the config file,
the test case runs a series of test cases to ensure that migrations work
properly.
There are also "opportunistic" tests for both mysql and postgresql in here,
which allows testing against mysql and pg in a properly configured unit
test environment.
For the opportunistic testing you need to set up a db named 'openstack_citest'
with user 'openstack_citest' and password 'openstack_citest' on localhost.
The test will then use that db and u/p combo to run the tests.
For postgres on Ubuntu this can be done with the following commands:
sudo -u postgres psql
postgres=# create user openstack_citest with createdb login password
'openstack_citest';
postgres=# create database openstack_citest with owner openstack_citest;
"""
import sqlalchemy
from oslo.config import cfg
from climate.tests.db import migration
CONF = cfg.CONF
class TestMigrations(migration.BaseWalkMigrationTestCase,
migration.CommonTestsMixIn):
"""Test alembic migrations."""
# This variables are used by BaseWalkMigrationTestCase in order to perform
# the opportunistic testing
USER = "openstack_citest"
PASSWD = "openstack_citest"
DATABASE = "openstack_citest"
def get_table(self, engine, name):
"""Returns an sqlalchemy table dynamically from db.
Needed because the models don't work for us in migrations
as models will be far out of sync with the current data.
"""
metadata = sqlalchemy.MetaData()
metadata.bind = engine
return sqlalchemy.Table(name, metadata, autoload=True)
def assertTableExists(self, engine, table):
metadata = sqlalchemy.MetaData()
metadata.reflect(bind=engine)
self.assertIn(table, metadata.tables)
def assertColumnExists(self, engine, table, column):
t = self.get_table(engine, table)
self.assertIn(column, t.c)
def assertColumnsExists(self, engine, table, columns):
for column in columns:
self.assertColumnExists(engine, table, column)
def assertColumnCount(self, engine, table, columns):
t = self.get_table(engine, table)
self.assertEqual(len(t.columns), len(columns))
def assertColumnNotExists(self, engine, table, column):
t = self.get_table(engine, table)
self.assertNotIn(column, t.c)
def assertIndexExists(self, engine, table, index):
t = self.get_table(engine, table)
index_names = [idx.name for idx in t.indexes]
self.assertIn(index, index_names)
def assertIndexMembers(self, engine, table, index, members):
self.assertIndexExists(engine, table, index)
t = self.get_table(engine, table)
index_columns = None
for idx in t.indexes:
if idx.name == index:
index_columns = idx.columns.keys()
break
self.assertEqual(sorted(members), sorted(index_columns))
def _check_0_1(self, engine, data):
self.assertTableExists(engine, 'computehosts')
self.assertTableExists(engine, 'leases')
self.assertTableExists(engine, 'reservations')
self.assertTableExists(engine, 'computehost_extra_capabilities')
self.assertTableExists(engine, 'events')
self.assertTableExists(engine, 'computehost_allocations')
self.assertTableExists(engine, 'computehost_reservations')

7
contrib/devstack/lib/climate
View File

@ -96,7 +96,6 @@ function configure_climate() {
iniset $CLIMATE_CONF_FILE manager plugins basic.vm.plugin,physical.host.plugin
recreate_database climate utf8
iniset $CLIMATE_CONF_FILE database connection `database_connection_url climate`
iniset $CLIMATE_CONF_FILE DEFAULT use_syslog $SYSLOG
@ -115,6 +114,12 @@ function configure_climate() {
iniadd $NOVA_CONF DEFAULT osapi_compute_extension "climatenova.api.extensions.default_reservation.Default_reservation"
iniadd $NOVA_CONF DEFAULT osapi_compute_extension "climatenova.api.extensions.reservation.Reservation"
# Database
recreate_database climate utf8
# Run Climate db migrations
$CLIMATE_BIN_DIR/climate-db-manage --config-file $CLIMATE_CONF_FILE upgrade head
}
# create_climate_aggregate_freepool() - Create a Nova aggregate to use as freepool (for host reservation)

1
requirements.txt
View File

@ -1,5 +1,6 @@
pbr>=0.6,<1.0
alembic>=0.4.1
Babel>=1.3
eventlet>=0.13.0
Flask>=0.10,<1.0

1
setup.cfg
View File

@ -27,6 +27,7 @@ packages =
[entry_points]
console_scripts =
climate-db-manage=climate.db.migration.cli:main
climate-api=climate.cmd.api:main
climate-rpc-zmq-receiver=climate.cmd.rpc_zmq_receiver:main
climate-manager=climate.cmd.manager:main