Merge "docs: add Project Testing Interface guide"
This commit is contained in:
commit
619ec91499
|
@ -0,0 +1,12 @@
|
||||||
|
:title: HowTos
|
||||||
|
|
||||||
|
.. _howtos:
|
||||||
|
|
||||||
|
HowTos: Recommended Practices
|
||||||
|
=============================
|
||||||
|
|
||||||
|
.. toctree::
|
||||||
|
:maxdepth: 2
|
||||||
|
|
||||||
|
howtos/pti
|
||||||
|
howtos/cross-project-gating
|
|
@ -0,0 +1,24 @@
|
||||||
|
:title: Cross Project Gating
|
||||||
|
|
||||||
|
.. _cross-project-gating:
|
||||||
|
|
||||||
|
Cross Project Gating
|
||||||
|
====================
|
||||||
|
|
||||||
|
The following sections describe how to test future state of cross projects
|
||||||
|
dependencies.
|
||||||
|
|
||||||
|
Python
|
||||||
|
------
|
||||||
|
|
||||||
|
Python project's requirements are pulled from external source by default, and
|
||||||
|
the zuul-jobs tox playbook implement a special task to ensure the job required
|
||||||
|
projects are installed from the zuul src_root so that the gate effectively test
|
||||||
|
in flight project's state.
|
||||||
|
|
||||||
|
Packaging
|
||||||
|
---------
|
||||||
|
|
||||||
|
Similarly, packaging project, where tests are performed using the final binary
|
||||||
|
artifacts, would need to inject a local repository to ensure the test is using
|
||||||
|
the change as prepared by Zuul.
|
|
@ -0,0 +1,214 @@
|
||||||
|
:title: Project Testing Interface
|
||||||
|
|
||||||
|
.. _pti:
|
||||||
|
|
||||||
|
Project Testing Interface
|
||||||
|
=========================
|
||||||
|
|
||||||
|
The following sections describe an example PTI (Project Testing Interface)
|
||||||
|
implementation. The goal is to setup a consistent interface for driving tests
|
||||||
|
and other necessary tasks to succesfully configure :ref:`project_gating` within
|
||||||
|
your organization.
|
||||||
|
|
||||||
|
|
||||||
|
Projects layout
|
||||||
|
---------------
|
||||||
|
|
||||||
|
A proper PTI needs at least two projects:
|
||||||
|
|
||||||
|
* org-config: a :term:`config-project` curated by administrators, and
|
||||||
|
* org-jobs: a :term:`untrusted-project` to hold common jobs.
|
||||||
|
|
||||||
|
The projects that are being tested or deployed are also
|
||||||
|
:term:`untrusted-project`, and for the purpose of this example we will use a
|
||||||
|
couple of integrated projects named:
|
||||||
|
|
||||||
|
* org-server
|
||||||
|
* org-client
|
||||||
|
|
||||||
|
|
||||||
|
org-config
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
The config project needs careful scrutiny as it defines priviledged Zuul
|
||||||
|
configurations that are shared by all your projects:
|
||||||
|
|
||||||
|
:ref:`pipeline` triggers and requirements let you define when a change is
|
||||||
|
tested and what are the conditions for merging code. Approval from core
|
||||||
|
members or special labels to indicate a change is good to go are pipelines
|
||||||
|
configuration.
|
||||||
|
|
||||||
|
The base job let you define how the test environment is validated before
|
||||||
|
the actual job is executed. The base job also defines how and where the job
|
||||||
|
artifacts are stored.
|
||||||
|
|
||||||
|
More importantly, a config-project may enforce a set of integration jobs to
|
||||||
|
be executed on behalf of the other projects. A regular (untrusted-project) can
|
||||||
|
only manage its own configuration, and as part of a PTI implementation, you
|
||||||
|
want to ensure your projects' changes undergo validation that are defined
|
||||||
|
globally by your organization.
|
||||||
|
|
||||||
|
Because the nature of those configuration settings are priviledged,
|
||||||
|
config-projects changes are only effective when merged.
|
||||||
|
|
||||||
|
|
||||||
|
org-jobs
|
||||||
|
~~~~~~~~
|
||||||
|
|
||||||
|
Jobs definition content are not priviledged Zuul settings and jobs can be
|
||||||
|
defined in a regular :term:`untrusted-project`.
|
||||||
|
As a matter of fact, it is recommended to define jobs outside of the config
|
||||||
|
project so that job updates can be tested before being merged.
|
||||||
|
|
||||||
|
In this example, we are using a dedicated org-jobs project.
|
||||||
|
|
||||||
|
|
||||||
|
Projects content
|
||||||
|
----------------
|
||||||
|
|
||||||
|
In this example PTI, the organization requirements are a consistent code style
|
||||||
|
and an integration test to validate org-client and org-server works according
|
||||||
|
to a reference implementation.
|
||||||
|
|
||||||
|
In the org-jobs project, we define a couple of jobs:
|
||||||
|
|
||||||
|
.. code-block:: yaml
|
||||||
|
|
||||||
|
# org-jobs/zuul.yaml
|
||||||
|
- job:
|
||||||
|
name: org-codestyle
|
||||||
|
parent: run-test-command
|
||||||
|
vars:
|
||||||
|
test_command: $code-style-tool $org-check-argument
|
||||||
|
# e.g.: linters --max-column 120
|
||||||
|
|
||||||
|
- job:
|
||||||
|
name: org-integration-test
|
||||||
|
run: integration-tests.yaml
|
||||||
|
required-projects:
|
||||||
|
- org-server
|
||||||
|
- org-client
|
||||||
|
|
||||||
|
The integration-tests.yaml playbook needs to implement an integration test
|
||||||
|
that checks both the server and client code.
|
||||||
|
|
||||||
|
|
||||||
|
In the org-config project, we define a project template:
|
||||||
|
|
||||||
|
.. code-block:: yaml
|
||||||
|
|
||||||
|
# org-config/zuul.d/pti.yaml
|
||||||
|
- project-template:
|
||||||
|
name: org-pti
|
||||||
|
queue: integrated
|
||||||
|
check:
|
||||||
|
jobs:
|
||||||
|
- org-codestyle
|
||||||
|
- org-integration-test
|
||||||
|
gate:
|
||||||
|
jobs:
|
||||||
|
- org-codestyle
|
||||||
|
- org-integration-test
|
||||||
|
|
||||||
|
|
||||||
|
Finaly, in the org-config project, we setup the PTI template on both projects:
|
||||||
|
|
||||||
|
.. code-block:: yaml
|
||||||
|
|
||||||
|
# org-config/zuul.d/projects.yaml
|
||||||
|
- project:
|
||||||
|
name: org-server
|
||||||
|
templates:
|
||||||
|
- org-pti
|
||||||
|
|
||||||
|
- project:
|
||||||
|
name: org-client
|
||||||
|
templates:
|
||||||
|
- org-pti
|
||||||
|
|
||||||
|
|
||||||
|
Usage
|
||||||
|
-----
|
||||||
|
|
||||||
|
With the above layout, the organization projects use a consistent testing
|
||||||
|
interface.
|
||||||
|
The org-client or org-server does not need extra settings, all new
|
||||||
|
contribution shall pass the codestyle and integration-test as defined by
|
||||||
|
the organization admin.
|
||||||
|
|
||||||
|
|
||||||
|
Project tests
|
||||||
|
~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
Projects may add extra jobs on top of the PTI.
|
||||||
|
For example, the org-client project can add a user interface test:
|
||||||
|
|
||||||
|
.. code-block:: yaml
|
||||||
|
|
||||||
|
# org-client/.zuul.yaml
|
||||||
|
- job:
|
||||||
|
name: org-client-ui-validation
|
||||||
|
|
||||||
|
- project:
|
||||||
|
check:
|
||||||
|
jobs:
|
||||||
|
- org-client-ui-validation
|
||||||
|
gate:
|
||||||
|
jobs:
|
||||||
|
- org-client-ui-validation
|
||||||
|
|
||||||
|
In this example, new org-client change will run the PTI's jobs as well as the
|
||||||
|
org-client-ui-validation job.
|
||||||
|
|
||||||
|
|
||||||
|
Updating PTI test
|
||||||
|
~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
Once the PTI is in place, if a project needs adjustment,
|
||||||
|
it can proceed as follow:
|
||||||
|
|
||||||
|
First a change on org-jobs is proposed to modify a job. For example, update a
|
||||||
|
codestyle check using such commit:
|
||||||
|
|
||||||
|
.. code-block:: text
|
||||||
|
|
||||||
|
# org-jobs/change-url
|
||||||
|
|
||||||
|
Update codestyle to enforce CamelCase.
|
||||||
|
|
||||||
|
Then, without merging this proposal, it can be tested accross the projects using
|
||||||
|
such commit:
|
||||||
|
|
||||||
|
.. code-block:: text
|
||||||
|
|
||||||
|
# org-client/change-url
|
||||||
|
|
||||||
|
Validate new codestyle.
|
||||||
|
Depends-On: org-jobs/change-url
|
||||||
|
|
||||||
|
Lastly the org-jobs may be enriched with:
|
||||||
|
|
||||||
|
.. code-block:: text
|
||||||
|
|
||||||
|
# org-jobs/change-url
|
||||||
|
|
||||||
|
Update codestyle to enforce CamelCase.
|
||||||
|
Needed-By: org-client/change-url
|
||||||
|
|
||||||
|
|
||||||
|
.. note:: Extra care is required when updating PTI jobs as they affects all
|
||||||
|
the projects. Ideally, the org-jobs project would use a org-jobs-check
|
||||||
|
to run PTI jobs change on every projects.
|
||||||
|
|
||||||
|
|
||||||
|
Cross project gating
|
||||||
|
--------------------
|
||||||
|
|
||||||
|
The org-pti template is using the "integrated" queue to ensure projects change
|
||||||
|
are gated by the zuul scheduler. Though, the jobs need extra care to properly
|
||||||
|
test projects as they are prepared by Zuul. For example, the
|
||||||
|
org-integration-test playbook need to ensure the client and server are installed
|
||||||
|
from the zuul src_root.
|
||||||
|
|
||||||
|
This is called sibbling installation, and it is critical piece to ensure cross
|
||||||
|
project gating. Check out the recommended practices :ref:`cross-project-gating`.
|
|
@ -17,3 +17,4 @@ configure it to meet your needs.
|
||||||
jobs
|
jobs
|
||||||
encryption
|
encryption
|
||||||
badges
|
badges
|
||||||
|
howtos
|
||||||
|
|
Loading…
Reference in New Issue