From d500b3f883e94a3b82f313bffe6dbeb08d7ee1e4 Mon Sep 17 00:00:00 2001 From: Abhishek Kekane Date: Tue, 20 Feb 2018 14:47:43 +0000 Subject: [PATCH] Revise database rolling upgrade documentation - mark zero-downtime-db-upgrade as EXPERIMENTAL for queens - clarify the relation between the E-M-C strategy and zero-downtime db upgrades - add note that for MySQL, using the glance-manage expand or glance-manage contract command requires that the glance is granted SUPER privileges - add note to contributor docs about checking the trigger flag in expand and contract scripts Co-authored-by: Abhishek Kekane Co-authored-by: Brian Rosmaita Change-Id: I5af4a1428b89ecb05a1be9c420c5f0afc05b9a95 Closes-Bug: #1750555 --- doc/source/admin/rollingupgrades.rst | 23 ++++++ doc/source/admin/zero-downtime-db-upgrade.rst | 81 ++++++++++++++----- .../contributor/database_migrations.rst | 19 ++++- 3 files changed, 101 insertions(+), 22 deletions(-) diff --git a/doc/source/admin/rollingupgrades.rst b/doc/source/admin/rollingupgrades.rst index 74cb364566..1d52738e9e 100644 --- a/doc/source/admin/rollingupgrades.rst +++ b/doc/source/admin/rollingupgrades.rst @@ -19,6 +19,23 @@ Rolling Upgrades .. note:: The Rolling Upgrades feature is EXPERIMENTAL and its use in production systems is currently **not supported**. + This statement remains true for the Queens release of Glance. What + is the holdup, you ask? Before asserting that the feature is fully + supported, the Glance team needs to have automated tests that perform + rolling upgrades in the OpenStack Continuous Integration gates. The + Glance project team has not had sufficient testing and development + resources in recent cycles to prioritize this work. + + The Glance project team is committed to the stability of Glance. As + part of OpenStack, we are committed to `The Four Opens`_. If the + ability to perform rolling upgrades in production systems is + important to you, feel free to participate in the Glance community to + help coordinate and drive such an effort. (We gently remind you that + "participation" includes providing testing and development + resources.) + + .. _`The Four Opens`: https://governance.openstack.org/tc/reference/opens.html + Scope of this document ---------------------- @@ -83,6 +100,12 @@ Following is the process to upgrade Glance with zero downtime: glance-manage db expand + .. warning:: + + For MySQL, using the ``glance-manage db_expand`` command requires that + you either grant your glance user ``SUPER`` privileges, or run + ``set global log_bin_trust_function_creators=1;`` in mysql beforehand. + 5. Then, also on the NEW NODE, perform the data migrations using the command:: glance-manage db migrate diff --git a/doc/source/admin/zero-downtime-db-upgrade.rst b/doc/source/admin/zero-downtime-db-upgrade.rst index 3bae8e85cc..d59de27cc2 100644 --- a/doc/source/admin/zero-downtime-db-upgrade.rst +++ b/doc/source/admin/zero-downtime-db-upgrade.rst @@ -17,11 +17,11 @@ Zero-Downtime Database Upgrades =============================== .. warning:: - This feature is EXPERIMENTAL in the Ocata and Pike releases. + This feature is EXPERIMENTAL in the Ocata, Pike and Queens releases. We encourage operators to try it out, but its use in production environments is currently NOT SUPPORTED. -A zero-downtime database upgrade enables true rolling upgrades of the Glance +A *zero-downtime database upgrade* enables true rolling upgrades of the Glance nodes in your cloud's control plane. At the appropriate point in the upgrade, you can have a mixed deployment of release *n* (for example, Ocata) and release *n-1* (for example, Newton) Glance nodes, take the *n-1* release nodes out of @@ -31,6 +31,13 @@ leaving all Glance nodes in your cloud at release *n*. That's a rough sketch of how a rolling upgrade would work. For full details, see :ref:`rolling-upgrades`. +.. note:: + When we speak of a "database upgrade", we are simply talking about changing + the database schema and its data from the version used in OpenStack release + *n* (say, Pike) to the version used in OpenStack release *n+1* (say, + Queens). We are **not** talking about upgrading the database management + software. + .. note:: Downgrading a database is not supported. See :ref:`downgrades` for more information. @@ -38,18 +45,45 @@ see :ref:`rolling-upgrades`. The Expand-Migrate-Contract Cycle --------------------------------- -For Glance, a zero-downtime database upgrade has three phases: +It's possible to characterize three phases of a database upgrade: -1. **Expand**: in this phase, new columns, tables, indexes, or triggers are - added to the database. +1. **Expand**: in this phase, new columns, tables, indexes, are added to the + database. 2. **Migrate**: in this phase, data is migrated to the new columns or tables. -3. **Contract**: in this phase, the "old" tables or columns (and any database - triggers used during the migration), which are no longer in use, are removed - from the database. +3. **Contract**: in this phase, the "old" tables or columns (which are no + longer in use) are removed from the database. -The above phases are abbreviated as an **E-M-C** database upgrade. +The "legacy" Glance database migrations performed these phases as part of a +single monolithic upgrade script. Currently, the Glance project creates a +separate script for each the three parts of the cycle. We call such an +upgrade an **E-M-C** database migration. + +Zero-Downtime Database Upgrade +------------------------------ + +The E-M-C strategy can be performed offline when Glance is not using the +database. With some adjustments, however, the E-M-C strategy can be applied +online when the database is in use, making true rolling upgrades possible. + +.. note:: + Don't forget that zero-downtime database upgrades are currently considered + experimental and their use in production environments is NOT SUPPORTED. + +A zero-downtime database upgrade takes place as part of a :ref:`rolling upgrade +strategy ` for upgrading your entire Glance installation. In +such a situation, you want to upgrade to release *n* of Glance (say, Queens) +while your release *n-1* API nodes are still running Pike. To make this +possible, in the **Expand** phase, database triggers can be added to the +database to keep the data in "old" and "new" columns synchronized. Likewise, +after all data has been migrated and all Glance nodes have been updated to +release *n* code, these triggers are deleted in the **Contract** phase. + +.. note:: + Unlike the E-M-C scripts, database triggers are particular to each database + technology. That's why the Glance project currently provides experimental + support only for MySQL. New Database Version Identifiers -------------------------------- @@ -59,9 +93,8 @@ database becomes more complicated since it must reflect knowledge of what point in the E-M-C cycle the upgrade has reached. To make this evident, the identifier explicitly contains 'expand' or 'contract' as part of its name. -Thus the ``ocata01`` migration (that is, the migration that's currently used in -the fully supported upgrade path) has two identifiers associated with it for -zero-downtime upgrades: ``ocata_expand01`` and ``ocata_contract01``. +Thus the Ocata cycle migration has two identifiers associated with it: +``ocata_expand01`` and ``ocata_contract01``. During the upgrade process, the database is initially marked with ``ocata_expand01``. Eventually, after completing the full upgrade process, the @@ -100,12 +133,24 @@ version record would go through the following progression: Database Upgrade ---------------- -In order to enable the E-M-C database upgrade cycle, and to enable Glance -rolling upgrades, the ``glance-manage`` tool has been augmented to include the -following operations. +For offline database upgrades, the ``glance-manage`` tool still has the +``glance-manage db sync`` command. This command will execute the expand, +migrate, and contract scripts for you, just as if they were contained in +a single script. + +In order to enable zero-downtime database upgrades, the ``glance-manage`` tool +has been augmented to include the following operations so that you can +explicitly manage the upgrade. + +.. warning:: + + For MySQL, using the ``glance-manage db expand`` or + ``glance-manage db contract`` command requires that + you either grant your glance user ``SUPER`` privileges, or run + ``set global log_bin_trust_function_creators=1;`` in mysql beforehand. Expanding the Database ----------------------- +~~~~~~~~~~~~~~~~~~~~~~ :: glance-manage db expand @@ -116,7 +161,7 @@ any new services are started. Migrating the Data ------------------- +~~~~~~~~~~~~~~~~~~ :: glance-manage db migrate @@ -127,7 +172,7 @@ are started. Contracting the Database ------------------------- +~~~~~~~~~~~~~~~~~~~~~~~~ :: glance-manage db contract diff --git a/doc/source/contributor/database_migrations.rst b/doc/source/contributor/database_migrations.rst index fe68202a3e..da4f19eb8d 100644 --- a/doc/source/contributor/database_migrations.rst +++ b/doc/source/contributor/database_migrations.rst @@ -282,10 +282,21 @@ Data Migrations NOTES ----- -* Starting in Ocata, Glance needs every database migration to include both - monolithic and Expand-Migrate-Contract (E-M-C) style migrations. At some - point in Pike, E-M-C migrations will be made default. At that point, it - would be no longer required to include monolithic migration script. +* In Ocata and Pike, Glance required every database migration to include + both monolithic and Expand-Migrate-Contract (E-M-C) style migrations. In + Queens, E-M-C migrations became the default and a monolithic migration + script is no longer required. + + In Queens, the glance-manage tool was refactored so that the ``glance-manage + db sync`` command runs the expand, migrate, and contract scripts "under + the hood". From the viewpoint of the operator, there is no difference + between having a single monolithic script and having three scripts. + + Since we are using the same scripts for offline and online (zero-downtime) + database upgrades, as a developer you have to pay attention in your scripts + to determine whether you need to add/remove triggers in the expand/contract + scripts. See the changes to the ocata scripts in + https://review.openstack.org/#/c/544792/ for an example of how to do this. * Alembic is a database migration engine written for SQLAlchemy. So, any migration script written for SQLAlchemy Migrate should work with Alembic as