=================================== OpenStack manuals project migration =================================== Problem description ~~~~~~~~~~~~~~~~~~~ The documentation team are rapidly losing key contributors and core reviewers. We are not alone, this is happening across the board. It is making things harder, but not impossible. Since our inception in 2010, we’ve been climbing higher and higher trying to achieve the best documentation we could, and uphold our high standards. However, we now need to take a step back and realise that the amount of work we are attempting to maintain is out of reach for the team size that we have. At the moment we have 13 cores, of whom none are full time contributors or reviewers. Until this point, the documentation team has owned several manuals that include content related to multiple projects, including an installation guide, admin guide, configuration guide, networking guide, and security guide. Because the team no longer has the resources to own that content, we want to invert the relationship between the doc team and project teams, so that we become liaisons to help with maintenance instead of asking for project teams to provide liaisons to help with content. As a part of that change, we plan to move the existing content out of the central manuals repository, into repositories owned by the appropriate project teams. Project teams will then own the content and the documentation team will assist by managing the build tools, helping with writing guidelines and style, but not writing the bulk of the text. We currently have the infrastructure set up to empower project teams to manage their own documentation in their own tree, and many do. As part of this change, the rest of the existing content from the install guide and admin guide will also move into project-owned repositories. Proposed change ~~~~~~~~~~~~~~~ Combine all of the documentation builds for a given repository, so that each repository has a single doc/source directory, a single sphinx conf.py, and a single job for building and publishing the content including developer, contributor, and user documentation. This option would reduce the number of build jobs we have to run, and cut down on the number of separate sphinx configurations in each repository. Move most of the content from various guides in openstack-manuals into the same repository as the thing being documented -- the documentation should live with the code. For example, the installation instructions for glance move to the glance repository and the reference for using the glance client command line tool move to the python-glanceclient repository. In order to make it easy to include links from the landing pages on docs.openstack.org, we need to ensure a minimum level of consistency in the organization of the docs directory. The sections are based on the existing high-level groupings for which there are already landing pages on docs.openstack.org that will need to be updated. Within each top-level directory, project teams are free to organize their content however seems most appropriate to them. This is the proposed layout for phase 1: * ``doc/source/`` * ``install/`` -- anything having to do with installation of the project * ``contributor/`` -- anything related to contributing to the project or how the team is managed. Applies to some of the current content under /developer, we are changing the name to emphasize that not all contributors are developers and sometimes developers are users but not contributors. Service projects should place their automatically generated class documentation under this part of the tree, e.g. in ``contributor/api`` or ``contributor/internals``. * ``configuration/`` -- automatically generated configuration reference information based on oslo.config's sphinx integration (or manually written for projects not using oslo.config). Step-by-step guides for doing things like enabling cells or configuring a specific driver should be placed in the ``admin/`` section. * ``cli/`` -- command line tool reference docs, similar to man pages These may be automatically generated with cliff's sphinx integration, or manually written when auto-generation is not possible. Tutorials or other step-by-step guides *using* these tools should go in either the ``user/`` or ``admin/`` sections, depending on their audience. Because the documentation for each project should live in the repository with the code, this directory may not be present for all service repositories but will be present for most of the client library repositories. * ``admin/`` -- any content from the old admin guide or anything else that discusses how to run or operate the software * ``user/`` -- end-user content such as concept guides, advice, tutorials, step-by-step instructions for using the CLI to perform specific tasks, etc. * ``reference/`` -- any reference information associated with a project that is not covered by one of the above categories. Library projects should place their automatically generated class documentation here. This layout is the *minimum* set. Projects are free and encouraged to add whatever other docs they need beyond this list, but these items are listed here explicitly because there are already links to most of them from landing pages, and landing pages can be created for the others. During a later phase, we will merge the API reference and release notes builds into the same job, along with the rest of the documentation for a project. Both of those builds have custom considerations, though, and it is more important to move the content that is no longer going to be maintained by the documentation team. * ``doc/source/`` * ``api/`` -- the REST API reference and Guide content, when it exists * ``releasenotes/`` -- reno directions (the actual release notes inputs will stay in /releasenotes/notes, where they are now) .. note:: Further discussion of the layout of the ``api/`` and ``releasenotes/`` directories is deferred until we are farther along with the initial migration work. A new documentation build job will be set up to take the output produced from ``tox -e venv -- python setup.py build_sphinx`` (matching the existing `Consistent Testing Interface" `_) and publish it to a new location under ``_ The results will go to: * ``docs.openstack.org/$project-name/latest`` -- build from master * ``docs.openstack.org/$project-name/$series`` -- build from stable/$series * ``docs.openstack.org/$project-name/latest/$lang`` -- build translated version from master * ``docs.openstack.org/$project-name/$series/$lang`` -- build translated version from stable/$series Because we plan to reuse the existing `doc/source` directory in each project, some of the existing content will need to be rearranged as part of importing the content from openstack-manuals. Ultimately, this changes the way we publish results, and redirects will be required to be setup from all of the existing locations to the new locations, and move all of the existing documentation under the new structure. We will retain landing pages for the high level categories such as the install guides, the configuration reference, and contributor documentation. Those pages will continue to be maintained by the documentation team, and will deep-link into the project team documentation. For example, we will keep pages like https://docs.openstack.org/project-install-guide/ocata/ but they will provide lists of links to URLs like https://docs.openstack.org/nova/ocata/install, which will be part of the single doc build for nova. What is happening to each guide? -------------------------------- * Installation Guide * Most of the content will move from openstack-manuals into each project tree. * The chapters not directly related to specific OpenStack projects, such as the parts related to installing ntp and RabbitMQ, will be retained in openstack-manuals in a shared guide for setting up common dependencies so that content does not need to be reproduced several times. * The current guide is actually built 3 separate times, to cover 3 separate deployment platforms. The new build will not support that, so the migration for the installation guide will involve breaking the content up into separate pages for each platform (as needed). Patch https://review.openstack.org/473579 splits the content up into separate patches, one per OS. * Project Installation guides, already in tree * We recommend any installation guides already in-tree also move to the new organization. * Administrator Guide * This content will move from openstack-manuals into each project tree. No part will be retained in openstack-manuals. This spec was already approved: https://review.openstack.org/#/c/439122/ * High Availability Guide * This guide will remain in openstack-manuals and be managed by the HA team. For more information: https://blueprints.launchpad.net/openstack-manuals/+spec/implement-ha-guide-todos * Operations Guide * This guide will eventually move from openstack-manuals into the wiki. Nothing will be done with it until a volunteer is found to manage that move. * Security Guide * This content will stay in openstack-manuals, and be managed by the security team. * A notice is being added to indicate the last time it was updated and which release is relevant (https://review.openstack.org/#/c/470059). * Architecture Design Guide * This content will stay in openstack-manuals, and be deprecated. * A notice will be added to each page indicating that the guide is up to date as of $RELEASE after the finalisation of the current set of goals. For more information on those goals: https://blueprints.launchpad.net/openstack-manuals/+spec/arch-design-pike * Networking Guide * This content will move from openstack-manuals to the neutron repository under docs/source/admin. * Configuration Reference * A few pages will move from openstack-manuals to the user-facing documentation in oslo.config. * The remainder will be removed, and replaced with new pages in the in-tree documentation built using oslo_config.sphinxext. * For tracking purposes, please see: https://blueprints.launchpad.net/openstack-manuals/+spec/automate-config-ref * API Documentation * No changes. * End User Guide * This content will be divided between the horizon repository and python-openstackclient repository. * Command-Line Reference * This content will move the project-specific client documentation trees under doc/source/cli. For example, the information about using the ``glance`` command line tool would move to the python-glanceclient repository. * Virtual Machine Image Reference * This content will stay in openstack-manuals. Migration process ----------------- We will need to parallelize the migration work as much as possible if we are going to complete it by the end of the Pike cycle. We will therefore need project teams to find volunteers to "pull" the content into their repositories, instead of having the documentation team "push" it. .. note:: Use the topic ``doc-migration`` for all patches. .. note:: Repeat these steps for all server projects, clients, and other libraries. #. Move the existing contributor-focused content to fit the layout above. Submit that change with ``Depends-On: Ia750cb049c0f53a234ea70ce1f2bbbb7a2aa9454`` to tie it to this spec. #. If your project docs are not already building using warning-is-error in setup.cfg, turn that on and fix any build errors. Submit these as patches on top of the first patch. #. Pull in the content being migrated, following the layout above. * Go through the list of manuals in https://etherpad.openstack.org/p/doc-migration-tracking and take any actions needed to import content. * Prepare one patch per manual (so one to import the install guide, one to import the user guide, etc.). Submit these as patches on top of any previous patches. #. Ensure that there is a top-level index.rst in doc/source that incorporates all of the documentation for the project by including all of the subdirectories in a toctree. #. Update the theme for the in-tree docs to use the openstackdocstheme instead of oslosphinx. #. Add auto-generated config reference section(s). #. If pbr's autodoc feature is being used, update the ``api_doc_dir`` setting in the ``pbr`` section of ``setup.cfg`` to point to either ``reference/api`` (for libraries) or ``contributor/api`` (for other types of projects). #. Update project-config to have the doc build use the new jobs instead of the old jobs by replacing 'openstack-server-publish-jobs' with 'openstack-unified-publish-jobs'. Set ``Depends-On`` to the Change-Id from the patch created in step 1. This ensures that we do not publish the old content to the new location. #. Add links to the reviews for individual TODO items below those items in the sections dedicated to each manual. That way the docs team will know when it is safe to start deleting content. #. If the project has a service-type, add a 302 redirect rule from ``/$service-type`` to ``/$project-name`` by editing ``openstack-manuals/www/static/.htaccess``. For example, ``/compute`` should redirect to ``/nova``. #. After project-specific install guides are moved into the doc tree, set up a 302 redirect rule from /project-install-guide to the new /install location by editing ``openstack-manuals/www/static/.htaccess``. #. Update the main landing page(s) starting with docs.o.o/ (see notes above) #. Remove the original copy of the content from the openstack-manuals repo * This patch can be filed early with depends-on for the other related patches. Alternatives ------------ #. We could retain the existing trees for developer and API docs, and add a new one for "user" documentation. The installation guide, configuration guide, and admin guide would move here for all projects. Neutron's user documentation would include the current networking guide as well. This option would add 1 new build to each repository, but would allow us to easily roll out the change with less disruption in the way the site is organized and published, so there would be less work in the short term. #. We could move the content under separate repositories owned by the project teams, rather than in-tree with the code. This would allow project teams to delegate management of the documentation to a separate review project-sub-team, but would complicate the process of landing code and documentation updates together so that the docs are always up to date. #. Do nothing, and watch the world burn. We did consider using "service type" instead of "project name" for the publishing URLs, but not all of the projects that need documentations are services. We will have user-facing documentation coming from several Oslo libraries, for example. Implementation ~~~~~~~~~~~~~~ Assignee(s) ----------- Primary assignee: * Alexandra Settle (asettle) * Doug Hellmann (dhellmann) * Project teams * Documentation team PTL for Queens * Documentation team Work items ---------- The task list is quite long, so rather than repeat it here we give a summary. There is more detail in the tracking pad mentioned in step 3. #. Define new doc build and gate jobs that work like the current job, using "tox -e venv -- python setup.py build_sphinx`" in a repository, but publish to the new location of docs.o.o/$project-name/latest (dhellmann) * https://review.openstack.org/#/c/471881/ #. Define doc build jobs for stable branches that run the same command but publish to docs.o.o/$project-name/$series (dhellmann) * The same job will work for all branches. #. In parallel, in each repository, perform the migration steps listed above to copy the new content into the doc/source directory. Refer to https://etherpad.openstack.org/p/doc-migration-tracking for details about which pages go into which project trees. #. Define new translation jobs based on the ones for the release notes build but using the main doc build. Dependencies ~~~~~~~~~~~~ - Project team(s) collaboration - Infra team assistance - Reviews from multiple sources References ~~~~~~~~~~ * https://etherpad.openstack.org/p/doc-planning * The list of all URLs and where the content will move can be found in: https://etherpad.openstack.org/p/doc-migration-tracking * Documentation Publishing future thread: http://lists.openstack.org/pipermail/openstack-dev/2017-May/117162.html * Operations Guide Future thread: http://lists.openstack.org/pipermail/openstack-dev/2017-June/117799.html