From f0fe465f695ff7b4c1a246a07eae877746928e46 Mon Sep 17 00:00:00 2001 From: Alexandra Settle Date: Thu, 8 Jun 2017 15:01:46 +0100 Subject: [PATCH] Adds spec for migrating the documentation suite to project repos Currently the user-facing documentation belongs in the openstack-manuals repo. Due to a severe decline in contributions, the docs team has had to come up with a new way to move forward. This spec proposes the new process for user-facing documentation and how we are going to undertake the changes. Co-Authored By: Doug Hellmann Co-Authored By: Alexandra Settle Change-Id: Ia750cb049c0f53a234ea70ce1f2bbbb7a2aa9454 --- specs/pike/os-manuals-migration.rst | 393 ++++++++++++++++++++++++++++ 1 file changed, 393 insertions(+) create mode 100644 specs/pike/os-manuals-migration.rst diff --git a/specs/pike/os-manuals-migration.rst b/specs/pike/os-manuals-migration.rst new file mode 100644 index 0000000..4e49e67 --- /dev/null +++ b/specs/pike/os-manuals-migration.rst @@ -0,0 +1,393 @@ +=================================== +OpenStack manuals project migration +=================================== + +Problem description +~~~~~~~~~~~~~~~~~~~ + +The documentation team are rapidly losing key contributors and core reviewers. +We are not alone, this is happening across the board. It is making things +harder, but not impossible. +Since our inception in 2010, we’ve been climbing higher and higher trying to +achieve the best documentation we could, and uphold our high standards. +However, we now need to take a step back and realise that the amount of work +we are attempting to maintain is out of reach for the team size that we have. +At the moment we have 13 cores, of whom none are full time contributors or +reviewers. + +Until this point, the documentation team has owned several manuals that +include content related to multiple projects, including an installation +guide, admin guide, configuration guide, networking guide, and security +guide. Because the team no longer has the resources to own that content, +we want to invert the relationship between the doc team and project teams, +so that we become liaisons to help with maintenance instead of asking for +project teams to provide liaisons to help with content. As a part of that +change, we plan to move the existing content out of the central manuals +repository, into repositories owned by the appropriate project teams. +Project teams will then own the content and the documentation team will +assist by managing the build tools, helping with writing guidelines and +style, but not writing the bulk of the text. + +We currently have the infrastructure set up to empower project teams to +manage their own documentation in their own tree, and many do. As part of +this change, the rest of the existing content from the install guide and +admin guide will also move into project-owned repositories. + +Proposed change +~~~~~~~~~~~~~~~ + +Combine all of the documentation builds for a given repository, so +that each repository has a single doc/source directory, a single +sphinx conf.py, and a single job for building and publishing the +content including developer, contributor, and user documentation. This +option would reduce the number of build jobs we have to run, and cut +down on the number of separate sphinx configurations in each +repository. + +Move most of the content from various guides in openstack-manuals into +the same repository as the thing being documented -- the documentation +should live with the code. For example, the installation instructions +for glance move to the glance repository and the reference for using +the glance client command line tool move to the python-glanceclient +repository. + +In order to make it easy to include links from the landing pages on +docs.openstack.org, we need to ensure a minimum level of consistency +in the organization of the docs directory. The sections are based on +the existing high-level groupings for which there are already landing +pages on docs.openstack.org that will need to be updated. Within each +top-level directory, project teams are free to organize their content +however seems most appropriate to them. This is the proposed layout +for phase 1: + +* ``doc/source/`` + + * ``install/`` -- anything having to do with installation of the + project + * ``contributor/`` -- anything related to contributing to the + project or how the team is managed. Applies to some of the current + content under /developer, we are changing the name to emphasize + that not all contributors are developers and sometimes developers + are users but not contributors. Service projects should place + their automatically generated class documentation under this part + of the tree, e.g. in ``contributor/api`` or + ``contributor/internals``. + * ``configuration/`` -- automatically generated configuration + reference information based on oslo.config's sphinx integration + (or manually written for projects not using + oslo.config). Step-by-step guides for doing things like enabling + cells or configuring a specific driver should be placed in the + ``admin/`` section. + * ``cli/`` -- command line tool reference docs, similar to man pages + These may be automatically generated with cliff's sphinx + integration, or manually written when auto-generation is not + possible. Tutorials or other step-by-step guides *using* these + tools should go in either the ``user/`` or ``admin/`` sections, + depending on their audience. Because the documentation for each + project should live in the repository with the code, this + directory may not be present for all service repositories but will + be present for most of the client library repositories. + * ``admin/`` -- any content from the old admin guide or anything + else that discusses how to run or operate the software + * ``user/`` -- end-user content such as concept guides, advice, + tutorials, step-by-step instructions for using the CLI to perform + specific tasks, etc. + * ``reference/`` -- any reference information associated with a + project that is not covered by one of the above categories. + Library projects should place their automatically generated class + documentation here. + +This layout is the *minimum* set. Projects are free and encouraged to +add whatever other docs they need beyond this list, but these items +are listed here explicitly because there are already links to most of +them from landing pages, and landing pages can be created for the +others. + +During a later phase, we will merge the API reference and release notes builds +into the same job, along with the rest of the documentation for a project. +Both of those builds have custom considerations, though, and it is more +important to move the content that is no longer going to be maintained +by the documentation team. + +* ``doc/source/`` + + * ``api/`` -- the REST API reference and Guide content, when it exists + * ``releasenotes/`` -- reno directions (the actual release notes + inputs will stay in /releasenotes/notes, where they are now) + +.. note:: + + Further discussion of the layout of the ``api/`` and + ``releasenotes/`` directories is deferred until we are farther + along with the initial migration work. + +A new documentation build job will be set up to take the output produced from +``tox -e venv -- python setup.py build_sphinx`` (matching the existing +`Consistent Testing Interface" +`_) +and publish it to a new location under ``_ +The results will go to: + +* ``docs.openstack.org/$project-name/latest`` -- build from master +* ``docs.openstack.org/$project-name/$series`` -- build from + stable/$series +* ``docs.openstack.org/$project-name/latest/$lang`` -- build + translated version from master +* ``docs.openstack.org/$project-name/$series/$lang`` -- build + translated version from stable/$series + +Because we plan to reuse the existing `doc/source` directory in each project, +some of the existing content will need to be rearranged as part of importing +the content from openstack-manuals. + +Ultimately, this changes the way we publish results, and redirects will be +required to be setup from all of the existing locations to the new locations, +and move all of the existing documentation under the new structure. We will +retain landing pages for the high level categories such as the install guides, +the configuration reference, and contributor documentation. Those pages will +continue to be maintained by the documentation team, and will deep-link into +the project team documentation. + +For example, we will keep pages like +https://docs.openstack.org/project-install-guide/ocata/ but they will +provide lists of links to URLs like +https://docs.openstack.org/nova/ocata/install, which will be part of +the single doc build for nova. + +What is happening to each guide? +-------------------------------- + +* Installation Guide + + * Most of the content will move from openstack-manuals into each project + tree. + * The chapters not directly related to specific OpenStack projects, + such as the parts related to installing ntp and RabbitMQ, will be + retained in openstack-manuals in a shared guide for setting up + common dependencies so that content does not need to be reproduced + several times. + * The current guide is actually built 3 separate times, to cover 3 + separate deployment platforms. The new build will not support + that, so the migration for the installation guide will involve + breaking the content up into separate pages for each platform (as + needed). Patch https://review.openstack.org/473579 splits the + content up into separate patches, one per OS. + +* Project Installation guides, already in tree + + * We recommend any installation guides already in-tree also move to the new + organization. + +* Administrator Guide + + * This content will move from openstack-manuals into each project tree. No + part will be retained in openstack-manuals. This spec was already + approved: + https://review.openstack.org/#/c/439122/ + +* High Availability Guide + + * This guide will remain in openstack-manuals and be managed by the HA team. + For more information: https://blueprints.launchpad.net/openstack-manuals/+spec/implement-ha-guide-todos + +* Operations Guide + + * This guide will eventually move from openstack-manuals into the wiki. + Nothing will be done with it until a volunteer is found to manage that + move. + +* Security Guide + + * This content will stay in openstack-manuals, and be managed by the + security team. + * A notice is being added to indicate the last time it was updated + and which release is relevant + (https://review.openstack.org/#/c/470059). + +* Architecture Design Guide + + * This content will stay in openstack-manuals, and be deprecated. + * A notice will be added to each page indicating that the guide is up to + date as of $RELEASE after the finalisation of the current set of goals. + For more information on those goals: + https://blueprints.launchpad.net/openstack-manuals/+spec/arch-design-pike + +* Networking Guide + + * This content will move from openstack-manuals to the neutron repository + under docs/source/admin. + +* Configuration Reference + + * A few pages will move from openstack-manuals to the user-facing + documentation in oslo.config. + * The remainder will be removed, and replaced with new pages in the + in-tree documentation built using oslo_config.sphinxext. + * For tracking purposes, please see: + https://blueprints.launchpad.net/openstack-manuals/+spec/automate-config-ref + +* API Documentation + + * No changes. + +* End User Guide + + * This content will be divided between the horizon repository and + python-openstackclient repository. + +* Command-Line Reference + + * This content will move the project-specific client documentation + trees under doc/source/cli. For example, the information about + using the ``glance`` command line tool would move to the + python-glanceclient repository. + +* Virtual Machine Image Reference + * This content will stay in openstack-manuals. + +Migration process +----------------- + +We will need to parallelize the migration work as much as possible if we are +going to complete it by the end of the Pike cycle. We will therefore need +project teams to find volunteers to "pull" the content into their +repositories, instead of having the documentation team "push" it. + +.. note:: + + Use the topic ``doc-migration`` for all patches. + +.. note:: + + Repeat these steps for all server projects, clients, and other + libraries. + +#. Move the existing contributor-focused content to fit the layout + above. Submit that change with ``Depends-On: + Ia750cb049c0f53a234ea70ce1f2bbbb7a2aa9454`` to tie it to this + spec. +#. If your project docs are not already building using + warning-is-error in setup.cfg, turn that on and fix any build + errors. Submit these as patches on top of the first patch. +#. Pull in the content being migrated, following the layout above. + + * Go through the list of manuals in + https://etherpad.openstack.org/p/doc-migration-tracking and take + any actions needed to import content. + * Prepare one patch per manual (so one to import the install guide, + one to import the user guide, etc.). Submit these as patches on + top of any previous patches. + +#. Ensure that there is a top-level index.rst in doc/source that + incorporates all of the documentation for the project by including + all of the subdirectories in a toctree. +#. Update the theme for the in-tree docs to use the openstackdocstheme + instead of oslosphinx. +#. Add auto-generated config reference section(s). +#. If pbr's autodoc feature is being used, update the ``api_doc_dir`` + setting in the ``pbr`` section of ``setup.cfg`` to point to either + ``reference/api`` (for libraries) or ``contributor/api`` (for other + types of projects). +#. Update project-config to have the doc build use the new jobs instead of the + old jobs by replacing 'openstack-server-publish-jobs' with + 'openstack-unified-publish-jobs'. + + Set ``Depends-On`` to the Change-Id from the patch created in + step 1. This ensures that we do not publish the old content to the + new location. + +#. Add links to the reviews for individual TODO items below those + items in the sections dedicated to each manual. That way the docs + team will know when it is safe to start deleting content. +#. If the project has a service-type, add a 302 redirect rule from + ``/$service-type`` to ``/$project-name`` by editing + ``openstack-manuals/www/static/.htaccess``. For example, + ``/compute`` should redirect to ``/nova``. +#. After project-specific install guides are moved into the doc tree, + set up a 302 redirect rule from /project-install-guide to the + new /install location by editing + ``openstack-manuals/www/static/.htaccess``. +#. Update the main landing page(s) starting with docs.o.o/ (see notes above) +#. Remove the original copy of the content from the openstack-manuals repo + + * This patch can be filed early with depends-on for the other related + patches. + +Alternatives +------------ + +#. We could retain the existing trees for developer and API docs, and add a new + one for "user" documentation. The installation guide, configuration guide, + and admin guide would move here for all projects. Neutron's user + documentation would include the current networking guide as well. This + option would add 1 new build to each repository, but would allow us to + easily roll + out the change with less disruption in the way the site is organized and + published, so there would be less work in the short term. +#. We could move the content under separate repositories owned by the project + teams, rather than in-tree with the code. This would allow project teams to + delegate management of the documentation to a separate review + project-sub-team, but would complicate the process of landing code and + documentation updates together so that the docs are always up to date. +#. Do nothing, and watch the world burn. + +We did consider using "service type" instead of "project name" for the +publishing URLs, but not all of the projects that need documentations +are services. We will have user-facing documentation coming from several +Oslo libraries, for example. + +Implementation +~~~~~~~~~~~~~~ + +Assignee(s) +----------- + +Primary assignee: + +* Alexandra Settle (asettle) +* Doug Hellmann (dhellmann) +* Project teams +* Documentation team PTL for Queens +* Documentation team + +Work items +---------- + +The task list is quite long, so rather than repeat it here we give a summary. +There is more detail in the tracking pad mentioned in step 3. + +#. Define new doc build and gate jobs that work like the current job, using + "tox -e venv -- python setup.py build_sphinx`" in a repository, but publish + to the new location of docs.o.o/$project-name/latest (dhellmann) + + * https://review.openstack.org/#/c/471881/ + +#. Define doc build jobs for stable branches that run the same command but + publish to docs.o.o/$project-name/$series (dhellmann) + + * The same job will work for all branches. + +#. In parallel, in each repository, perform the migration steps listed above to + copy the new content into the doc/source directory. Refer to + https://etherpad.openstack.org/p/doc-migration-tracking for details about + which pages go into which project trees. +#. Define new translation jobs based on the ones for the release notes build + but using the main doc build. + +Dependencies +~~~~~~~~~~~~ + +- Project team(s) collaboration +- Infra team assistance +- Reviews from multiple sources + +References +~~~~~~~~~~ + +* https://etherpad.openstack.org/p/doc-planning +* The list of all URLs and where the content will move can be found + in: https://etherpad.openstack.org/p/doc-migration-tracking +* Documentation Publishing future thread: + http://lists.openstack.org/pipermail/openstack-dev/2017-May/117162.html +* Operations Guide Future thread: + http://lists.openstack.org/pipermail/openstack-dev/2017-June/117799.html