Add spec on database migrations
This commit adds a spec on converting Monasca from SQL scripts to database migrations. Change-Id: I398b8e1b5ab9b3529b0e671f197fdec5cf7b8e90 Story: 2001654 Task: 6681
This commit is contained in:
parent
1a572938b7
commit
a2035b5070
|
@ -23,4 +23,4 @@ Rocky approved (but not implemented) specs:
|
||||||
:glob:
|
:glob:
|
||||||
:maxdepth: 1
|
:maxdepth: 1
|
||||||
|
|
||||||
.. approved/*
|
approved/*
|
||||||
|
|
|
@ -0,0 +1,204 @@
|
||||||
|
..
|
||||||
|
This work is licensed under a Creative Commons Attribution 3.0 Unported
|
||||||
|
License.
|
||||||
|
|
||||||
|
http://creativecommons.org/licenses/by/3.0/legalcode
|
||||||
|
|
||||||
|
=============================
|
||||||
|
Introduce Database Migrations
|
||||||
|
=============================
|
||||||
|
|
||||||
|
https://storyboard.openstack.org/#!/story/2001654
|
||||||
|
|
||||||
|
Most OpenStack services use `SQLAlchemy Migrate
|
||||||
|
<https://sqlalchemy-migrate.readthedocs.io/en/latest/>`_ or `Alembic
|
||||||
|
<http://www.alembic.io/>`_ to provide migration scripts that (a) update the
|
||||||
|
service's database schema if it changes and (b) migrate data as required.
|
||||||
|
Currently, Monasca does not follow this pattern. Instead, it provides simple
|
||||||
|
SQL scripts for PostgreSQL or MySQL databases that are only useful for
|
||||||
|
initializing a database schema. For any subsequent changes required by updated
|
||||||
|
code operators are on their own, i.e. they need to apply these manually to keep
|
||||||
|
Monasca working. This spec proposes introducing (a) database migrations and (b)
|
||||||
|
a separate tool that analyses an existing database's schema state and adds the
|
||||||
|
necessary metadata to update it through migrations in the future.
|
||||||
|
|
||||||
|
|
||||||
|
Problem description
|
||||||
|
===================
|
||||||
|
|
||||||
|
In a nutshell, the database schema used for holding various alarm,
|
||||||
|
notification, and metric metadata is not updateable. This is due to the source
|
||||||
|
of truth for the schema being a simple SQL script that is only useful for
|
||||||
|
schema initialization on a blank database.
|
||||||
|
|
||||||
|
The group most affected by this is operators, who are faced with several
|
||||||
|
unattractive choices when a code change requires a database schema change:
|
||||||
|
|
||||||
|
#. They can drop their whole Monasca database and recreate it from scratch with
|
||||||
|
the updated SQL script. That way they will lose all of their alarm
|
||||||
|
definitions along with notification settings, their complete alarm history,
|
||||||
|
and some metrics metadata.
|
||||||
|
|
||||||
|
#. They can manually update the database and migrate data (in the case of
|
||||||
|
columns/tables getting renamed). This is error prone and carries a risk of
|
||||||
|
data loss.
|
||||||
|
|
||||||
|
#. They can decide to live without that code change. If the change breaks on an
|
||||||
|
outdated database they may even have to create and maintain code patches for
|
||||||
|
that.
|
||||||
|
|
||||||
|
Besides operators, the lack of database migration infrastructure is a
|
||||||
|
significant obstacle to Monasca developers as well: it turns any data model
|
||||||
|
change into a breaking change due to the schema update that requires. Thus the
|
||||||
|
current state of affairs prevents new features that require database
|
||||||
|
changes or renders them extremely unpopular at the very least.
|
||||||
|
|
||||||
|
Use Cases
|
||||||
|
---------
|
||||||
|
|
||||||
|
#. Operators will be able to update Monasca without risking breakage due to
|
||||||
|
database changes.
|
||||||
|
|
||||||
|
#. Developers can modify the data model without breaking existing
|
||||||
|
installations.
|
||||||
|
|
||||||
|
Proposed change
|
||||||
|
===============
|
||||||
|
|
||||||
|
The proposal aims to improve the current state of affairs by adding Alembic
|
||||||
|
based command line tooling for the following operations:
|
||||||
|
|
||||||
|
#. A legacy database migration command line tool. This command line tool will
|
||||||
|
detect which revision of the database schema SQL script was used based on
|
||||||
|
the tables and columns currently in the database. It will then initialize
|
||||||
|
the database's migration meta data to allow for regular database migration
|
||||||
|
from that point forward.
|
||||||
|
|
||||||
|
#. A database migration command line tool with multiple base revisions to
|
||||||
|
cover the following scenarios:
|
||||||
|
|
||||||
|
#. An uninitialized database
|
||||||
|
#. A schema state resulting from any of the existing SQL script
|
||||||
|
revisions.
|
||||||
|
|
||||||
|
Both (1) and (2) may be implemented as part of the same tool.
|
||||||
|
|
||||||
|
A database schema based on third party SQL scripts (which may be found in
|
||||||
|
various configuration management tools used for deploying Monasca) is not
|
||||||
|
supported.
|
||||||
|
|
||||||
|
Alternatives
|
||||||
|
------------
|
||||||
|
|
||||||
|
This change is long overdue and best practice throughout OpenStack. That being
|
||||||
|
said, it would be possible to considerably reduce its scope by omitting the
|
||||||
|
heuristics for detecting an existing legacy database schema. That way operators
|
||||||
|
would be forced to drop the database and recreate it using migrations. If
|
||||||
|
possible we should not impose this on operators.
|
||||||
|
|
||||||
|
It would also be possible to use plain SQLAlchemy Migrate rather than Alembic
|
||||||
|
to implement migrations, but OpenStack appears to have standardized on Alembic
|
||||||
|
by now (see https://wiki.openstack.org/wiki/OpenStack_and_SQLAlchemy for a
|
||||||
|
discussion of why that happened).
|
||||||
|
|
||||||
|
Data model impact
|
||||||
|
-----------------
|
||||||
|
|
||||||
|
There will be no impact on the data model itself. There will only be changes to
|
||||||
|
how the data model is synchronized to the database's run time schema.
|
||||||
|
|
||||||
|
REST API impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
There is no REST API impact.
|
||||||
|
|
||||||
|
Security impact
|
||||||
|
---------------
|
||||||
|
|
||||||
|
The only slightly security related aspect of this change is providing the
|
||||||
|
database migration command line tools with access credentials for the data
|
||||||
|
base. The accepted best practice for this is running it as root on the machine
|
||||||
|
where the API service (in our case `monasca-api`) is also running and retrieve
|
||||||
|
these credentials from the API service's configuration.
|
||||||
|
|
||||||
|
Other end user impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
N/A
|
||||||
|
|
||||||
|
Performance Impact
|
||||||
|
------------------
|
||||||
|
|
||||||
|
N/A
|
||||||
|
|
||||||
|
Other deployer impact
|
||||||
|
---------------------
|
||||||
|
|
||||||
|
Configuration management that uses the legacy SQL scripts must be changed to
|
||||||
|
use the new migration tools. Among other things, the Monasca devstack plugin
|
||||||
|
must be modified to use them. Users updating to a Monasca version with this
|
||||||
|
feature may need to run a special migration to get versioning metadata added to
|
||||||
|
their database schema.
|
||||||
|
|
||||||
|
Developer impact
|
||||||
|
----------------
|
||||||
|
|
||||||
|
Any developer making data model changes after this feature is implemented will
|
||||||
|
have to write a migration for updating the database schema accordingly.
|
||||||
|
|
||||||
|
Implementation
|
||||||
|
==============
|
||||||
|
|
||||||
|
Assignee(s)
|
||||||
|
-----------
|
||||||
|
|
||||||
|
Primary assignee:
|
||||||
|
<jgr-launchpad>
|
||||||
|
|
||||||
|
Tasks
|
||||||
|
-----
|
||||||
|
|
||||||
|
* Add command line tool(s) for regular migration (i.e. migration of a versioned
|
||||||
|
database) and transition of an unversioned database to a versioned one.
|
||||||
|
|
||||||
|
* Create migration chains from:
|
||||||
|
|
||||||
|
* Empty database
|
||||||
|
|
||||||
|
* All revisions of the schema file in the repository.
|
||||||
|
|
||||||
|
Dependencies
|
||||||
|
============
|
||||||
|
|
||||||
|
N/A
|
||||||
|
|
||||||
|
Testing
|
||||||
|
=======
|
||||||
|
|
||||||
|
Any meaningful testing can only take place in a full Monasca deployment with an
|
||||||
|
actual database to test against. In particular, testing attention should focus
|
||||||
|
on testing this with all revisions of the legacy SQL script in
|
||||||
|
`devstack/files/schema/mon_mysql.sql`. This testing can take place at an early
|
||||||
|
stage during Devstack setup as follows:
|
||||||
|
|
||||||
|
#. Each SQL script revision is checked out from the git repository, applied to
|
||||||
|
the database and converted to a "migratable" database by running the
|
||||||
|
conversion operation on the database. After each iteration, the database is
|
||||||
|
dropped to prepare for the next revision.
|
||||||
|
#. As a final step, the initial migration for a blank database is performed and
|
||||||
|
Devstack proceeds normally.
|
||||||
|
|
||||||
|
Since PostgreSQL is deprecated in OpenStack it will be sufficient to implement
|
||||||
|
testing with the MySQL flavoured SQL script.
|
||||||
|
|
||||||
|
Documentation Impact
|
||||||
|
====================
|
||||||
|
|
||||||
|
This feature needs to be documented in operator/deployer documentation to
|
||||||
|
ensure operators use migrations rather than legacy SQL scripts.
|
||||||
|
|
||||||
|
References
|
||||||
|
==========
|
||||||
|
|
||||||
|
* Etherpad from Rocky PTG where this was discussed:
|
||||||
|
https://etherpad.openstack.org/p/monasca-ptg-rocky
|
Loading…
Reference in New Issue