..
  Copyright 2021 Canonical Ltd.

  This work is licensed under a Creative Commons Attribution 3.0
  Unported License.
  http://creativecommons.org/licenses/by/3.0/legalcode

..
  This template should be in ReSTructured text. Please do not delete
  any of the sections in this template.  If you have nothing to say
  for a whole section, just write: "None". For help with syntax, see
  http://sphinx-doc.org/rest.html To test out your formatting, see
  http://www.tele3.cz/jbar/rest/rest.html

=========================
Deferred Service Restarts
=========================

When a charm or package is upgraded services are restarted. This can impact the
data plane causing outages. This spec is aimed at giving the cloud operator the
option of deferring these restarts to a later, more convenient, time.

Problem Description
===================

One example of the issue would be a charm upgrade which includes a minor
template change. Charm upgrades are executed on all units of an application
simultaneously (the cloud operator has no control over this). The charms are
designed to restart services when configuration files are changed so the same
set of services, across all units, will be simultaneously restarted.

Another example is when a package update is triggered by the charm, it may be
useful to defer any service restarts until after a lengthy package update
completes.

Proposed Change
===============

Add charm configuration option to control automatic service restarts.

.. code:: yaml

    enable-auto-restarts:
      type: boolean
      default: True
      description: |
        Allow the charm and packages to restart services automatically when
        required

Add charm action to run service restarts.

.. code:: yaml

    restart-services:
      description: |
        Restarts services this charm manages.
      params:
        deferred-only:
          type: boolean
          default: false
          description: |
            Only restart deferred services.
        services:
          type: string
          default: ""
          description: |
            Optional space seperated list of services to restart. If unset
            then it applies to all services the charm controls.
    show-deferred-restarts:
        description: |
            Show the outstanding restarts


If a charm has `enable-auto-restarts` disabled this will be reflected in the
charms workload status message. The starts that have been requested, but
deferred, by either the charm or a package update can be seen by running
the `show-deferred-restarts` action.

To block restarts, charms will first check whether `enable-auto-restarts`
has been disabled and only perform the restart if permitted. Ideally preventing
services from being stopped or restarted would be done within systemd but this
does not seem to be possible in a clean way. Relying on the charm to police
itself is not ideal as there are many places a restart command can emanate from
and there is always the risk that a future charm update will introduce a
restart call that does not honour `enable-auto-restarts`.

Packages may try and restart a service during an update. To stop this from
happening a custom `/usr/sbin/policy-rc.d` script will be installed to block
the package changes from causing any restarts. Charms will indicate which
services they want to block restarts for by writing a file to
`/etc/policy-rc.d`.  `/usr/sbin/policy-rc.d` will collect all the files from
`/etc/policy-rc.d` to construct a list of services which are not permitted to
be restarted.

The flow for deferring restarts:

#. Operator updates application to set `enable-auto-restarts=False`
#. A `/usr/sbin/policy-rc.d` file is written
#. A charm specific file is placed in `/etc/policy-rc.d` which list which
   services should be blocked from restarting.

The flow for re-enabling restarts:

#. Operator updates application to set `enable-auto-restarts=True`
#. The charm specific file is removed from `/etc/policy-rc.d`
#. Optionally, the operator runs the `restart-services` action against each
   application unit.

The existing `Pause` and `Resume` actions will continue to operate as they
do now irrespective of the value of `enable-auto-restarts`

Alternatives
------------

The problem could be handled if Juju were to implement a feature allowing charm
updates to be performed on a unit by unit basis.

Implementation
==============

Assignee(s)
-----------

Primary assignee:
  gnuoy

Gerrit Topic
------------

Use Gerrit topic "deferred-restarts" for all patches related to this spec.

.. code-block:: bash

    git-review -t deferred-restarts

Work Items
----------

* Write `policy-rc.d` script and add to charm-helpers
* Write functions in charm-helpers for managing the masking of services
  and files in `/etc/policy-rc.d`.
* Each charm will need to add the new action and configuration option and the
  charm will need to gate any service interupting restarts on the value of
  `enable-auto-restarts` in a similar way to the existing `is_unit_paused_set`

Repositories
------------

No new repositories are required.

Documentation
-------------

The new actions will need to be documented in the charm-guide and the upgrade
sections updated.

Security
--------

None

Testing
-------

The regularly scheduled upgrade testing would be a good place to test this
feature.

Dependencies
============

None