opendev
/
infra-specs
Code Issues Proposed changes

Spec to retire static.openstack.org 

Change-Id: Ic5557750ee6c52def01c8d362b8d9e7563cc0f8a
This commit is contained in:
Ian Wienand 2019-09-23 17:33:27 +10:00
parent 73f5500b39
commit 1533f03a30
2 changed files with 379 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
doc/source/index.rst
View File

@ -44,6 +44,7 @@ permits.
specs/translation_check_site
specs/wiki_modernization
specs/letsencrypt
specs/retire-static
Help Wanted
===========

378
specs/retire-static.rst Normal file
View File

@ -0,0 +1,378 @@
::
Copyright 2019 Red Hat Inc.
This work is licensed under a Creative Commons Attribution 3.0
Unported License.
http://creativecommons.org/licenses/by/3.0/legalcode
..
This template should be in ReSTructured text. Please do not delete
any of the sections in this template. If you have nothing to say
for a whole section, just write: "None". For help with syntax, see
http://sphinx-doc.org/rest.html To test out your formatting, see
http://www.tele3.cz/jbar/rest/rest.html
===========================
Retire static.openstack.org
===========================
Include the URL of your StoryBoard story:
https://storyboard.openstack.org/#!/story/2006598
Move the services provided by ``static.openstack.org`` into less
centralised approaches more consistent with modern deployment trends.
Problem Description
===================
The ``static.openstack.org`` host is a monolithic server providing
various hosting services via a large amount of volume-attached
storage.
The immediate problem is it currently running Ubuntu Trusty which is
reaching the end of its supported life.
The secondary problems are twofold:
Firstly, we would like to move the various publishing and hosting
operations from centralised volumes on a single server to our AFS
distributed file-system.
Secondly, we would like to make the hosting portion more OpenDev
compatible; this means avoiding working on legacy deployment methods
(i.e. puppet) and integrating with our general idea of a "whitebox"
service that can be used by many different projects.
Thus we propose breaking up the services it offers to utlise more
modern infrastructure alternatives and retiring the host.
Proposed Change
===============
We can break the services down
Log storage
Legacy log storage (~14tb)
Redirects
Apache service redirects a number of legacy URLs to new locations
Static site serving
100gb attached partition holding various static sites (i.e. plain
HTML publishing, no middleware, etc)
Tarball
512gb partition which holds and publishes release tarballs for all
projects.
Alternatives
------------
``apt-get dist-ugprade`` the host to a more recent distribution, fix
any puppet issues and ignore it until next time it needs updating.
Implementation
==============
Assignee(s)
-----------
Primary assignee:
TBD
Gerrit Topic
------------
Use Gerrit topic "static-services" all patches related to this spec.
.. code-block:: bash
git-review -t static-services
Work Items
----------
Log storage
~~~~~~~~~~~
OpenDev CI logs have been moved to various object-storage backends
provided by donors. The existing logs will age out per our existing
old-log cleanup jobs.
Since logs were always ephemeral there should be no issues with old
links. For clarity we will remove (rather than redirect) the
``logs.openstack.org`` DNS entry so there is no confusion that logs
might still live there.
Workitems:
* remove ``logs.openstack.org`` DNS entries after old logs entries
have cleared out
Legacy redirects
~~~~~~~~~~~~~~~~
The following do straight redirects from their config hostnames to
``docs.openstack.org``
* 50-cinder.openstack.org.conf
* 50-devstack.org.conf
* 50-glance.openstack.org.conf
* 50-horizon.openstack.org.conf
* 50-keystone.openstack.org.conf
* 50-nova.openstack.org.conf
* 50-swift.openstack.org.conf
The following have slightly different semantics
* 50-ci.openstack.org.conf
* ``/nodepool``, ``/shade``, ``/zuul``, etc all to docs; see
https://opendev.org/opendev/system-config/src/branch/master/modules/openstack_project/templates/ci.vhost.erb
* 50-qa.openstack.org.conf
* currently redirects to broken link
https://docs.openstack.org/developer/qa
The following redirects to ``openstack.org``
* 50-summit.openstack.org.conf
Clearly there is a need for a generic ability to redirect various URLs
as things change over time.
We will use a single containerised ``haproxy`` instance to handle
redirects for the OpenDev project. Although initially it will simply
be handling 302 redirects, it is imagined that future services can use
it for it's availability or load-balancing services as well. Note
that ``gitea`` services also have their own load-balancer; although it
reuses all the deployment mechanisms, the production service is kept
separately to maintain isolation been probably the most important
service (code) and more informational services.
Proof-of-concept reviews are provided at:
* https://review.opendev.org/677903 : make haproxy role more generic
* https://review.opendev.org/678159 : add a service load balancer
The work items consist of:
* approval of the above reviews
* starting the production host
* iterating the extant DNS records and pointing them to the new
load-balancer
OpenDev infrastructure migration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
We wish to provide new services only using our latest deployment
methods, to avoid introducing even more legacy services and to provide
a basis for the migration process to OpenDev services.
Although ``files02.openstack.org`` has an existing role as a webserver
serving content from the ``/openstack.org`` AFS mount, it is
configured using legacy puppet. Thus a new server will be provisioned
using our Ansible environment, rather than adding more hosts to legacy
configuration.
This server should be a "whitebox" server that is capable of serving a
range of domains that OpenDev would like to serve. However, it's role
will only to be to serve static directories on AFS volumes. After
this process, there will be numerous examples of SSL certificate
generation, vhost configuration, AFS volume setup and publishing jobs
for any other projects to copy and implement.
Initially this server needs to serve https sites for the replacement
services; namely
* governance.openstack.org
* specs.openstack.org
* security.openstack.org
* service-types.openstack.org
* releases.openstack.org
* tarballs.openstack.org
Currently, SSL certificates are manually provisioned and entered into
puppet secret data, where they are deployed to the host. We wish to
use automatically renewing letsencrypt certificates per our other
infrastructure, utilising our DNS based authentication. However,
since ``openstack.org`` remains administered by external teams in
RAX's propietary environment, we will make an exception and setup DNS
validation records manually for these legacy sites until a full
migration of ``openstack.org`` to OpenDev infrastructure is possible.
Other domains will use OpenDev nameservers, which support automated
DNS validation renewals.
We will have the new server provisioned and ready before we begin the
steps of migrating publishing locations. This means we can debug any
setup issues outside production, and effects a zero-downtime cutover
when the sites are ready.
Workitems are as follows:
* Write roles and tests to provision a new ``static01.opendev.org``
server which will be limited to running Apache and serving AFS
directories.
* Create the server
* Create CNAME ``static.opendev.org`` which will be the main service
hostname, to provide for easier server replacement or other updates
in the future.
* Pre-provision https certificates for the above listed services
* Using the RAX web interface for name services and the openstack
infra permissions, setup
``_acme-challenge.<service>.openstack.org`` records as a CNAME to
``acme.opendev.org``.
* Each site should have a separate certificate provisioned. The
configuration would be something like
.. code-block:: yaml
letsencrypt_certs:
governance-openstack-org:
- governance.openstack.org
specs.openstack.org:
- specs.openstack.org
and.so.on.
* Debug any failures; however the theory is (taking one example):
the existing letsencrypt roles should request a certificate for
``governance.openstack.org`` on ``static01.opendev.org`` and
receive the authentication key, which is placed in a TXT record in
``acme.opendev.org``. The certificate creation will will trigger
a lookup of ``_acme-challenge.governance.openstack.org`` which
will be a CNAME to ``acme.opendev.org``, which contains the
correct TXT record. The certificate is issued on
``static01.opendev.org``.
* Preconfigure the vhost configuration for the above sites (using
prior provisioned keys for SSL)
* Confirm correct operation of the sites with dummy content.
Static hosting
~~~~~~~~~~~~~~
A number of jobs publish directly to ``/srv/static`` on the server.
These are then served by Apache as static websites.
In general, we want these jobs to publish to our AFS volumes. By
publishing to AFS we remove the central point of failure of a single
server and it's attached disks (mitigated by multiple AFS servers and
replicas).
The AFS volumes are then served by ``static01.opendev.org`` which has
a dedicated role as an AFS to HTTP bridge.
The sites in question are:
* 50-governance.openstack.org.conf
* https://governance.openstack.org
* main source -> https://opendev.org/openstack/governance-website
* published via https://opendev.org/openstack/project-config/src/branch/master/zuul.d/projects.yaml#L2298
* aliases ``/srv/static/<election|sigs|tc|uc>``
* 50-security.openstack.org.conf
* https://security.openstack.org
* single repo source -> https://opendev.org/openstack/ossa
* deployed by publish-security job -> https://opendev.org/openstack/project-config/src/branch/master/zuul.d/jobs.yaml#L739
* 50-service-types.openstack.org.conf
* https://service-types.openstack.org
* single repo -> https://opendev.org/openstack/service-types-authority
* https://opendev.org/openstack/project-config/src/branch/master/zuul.d/jobs.yaml#L551
* 50-specs.openstack.org.conf
* https://specs.openstack.org
* various spec repos; published by ``openstack-spec-jobs`` to subdirectories
* 50-releases.openstack.org.conf
* https://releases.openstack.org
* generated by -> https://opendev.org/openstack/releases/
* note generates .htaccess with contsraints links, used widely in pip
* publish-tox-jobs-static : https://opendev.org/openstack/project-config/src/branch/master/zuul.d/jobs.yaml#L685
* 50-tarballs.openstack.org.conf
* https://tarballs.openstack.org
* every project's release jobs
The extant AFS layout has volumes for each project. Thus we will
continue this theme and an admin will create one volume for each of
the above static sites; e.g.
* /afs/openstack.org/project/governance.openstack.org (~200mb)
* /afs/openstack.org/project/security.openstack.org (100mb)
* /afs/openstack.org/project/service-types.openstack.org (520k)
* /afs/openstack.org/project/specs.openstack.org (current 706mb)
* /afs/openstack.org/project/releases.openstack.org (current 57mb)
* /afs/openstack.org/project/tarballs.openstack.org (current 134gb)
The work items are as follows
* Create the volumes for each site as described above
* Migrate the extant data to the new volumes. It is impractical to
recreate all the sites as it would require triggering many often
infrequently updated repos.
* Publishing jobs will be updated to use AFS publishing to these new
locations. During transition period, we can publish to both
locations.
* Update the site configuration on ``static01.opendev.org`` to serve
the site from the new location
* We should be able to fully test the new sites at this point with
manual host entries. Ensure:
* https certificates working correctly
* old links remain consistent
* For each site, move to production by updating the CNAME entries in
the ``openstack.org`` domain for the main server to point to
``static.opendev.org`` (note, not the server directly,
i.e. ``static01.opendev.org``, to give us flexibility in managing
the backend service with server replacements or load-balancing in
the future). Per prior testing, this should be transparent.
* Old publishing jobs removed
Repositories
------------
Unlikley to require new repositories
Servers
-------
* a new http server for serving AFS content
* A load-balancer server is suggested to host the haproxy container
DNS Entries
-----------
Quite a few DNS entries will need to be updated as described
Documentation
-------------
Developers should largely not care where the results are published.
Small doc updates for any new services.
A guide to setting up jobs, host configuration, etc. for publishing
static data for other projects may be useful.
Security
--------
N/A
Testing
-------
Since all updates are replacements, we can confirm that the new sites
are operational before putting them into production. Any DNS switches
can be essentially zero impact.
Dependencies
============
N/A at this time