diff --git a/src/zuulv3/gus2019.rst b/src/zuulv3/gus2019.rst index 2cbfa82..b61621a 100644 --- a/src/zuulv3/gus2019.rst +++ b/src/zuulv3/gus2019.rst @@ -65,93 +65,6 @@ What Zuul does * gated commits * testing like deployment -Our Use Case -============ - -OpenStack Is -============ - - * Federated - * Distributed - * Large - * Open - * Not Alone - -Federated -========= - - * Hundreds of involved companies - * No 'main' company - * "Decisions are made by those who show up" - * Union of priorities/use cases - -Impact of being Federated -========================= - - * No company can appoint people to positions in the project - * The project cannot fire anyone - * Variable background of contributors - * Heavy reliance on consensus - -Distributed -=========== - - * There is no office - * Contributor base is global - * Multitude of contributor backgrounds - -Impact of being Distributed -=========================== - - * Tooling must empower all contributors, regardless of background, - skill level or cultural context - * Heavy preference for text-based communication - * Cannot assume US-centric needs or solutions - * "Accept patches from random people on the internet" - -Large numbers of -================ - - * Contributors (\~2k in any given 6 month period) - * Changes - * Code Repositories (2120 as of this morning) - -OpenStack Scale Comparison -========================== - - * 2KJPH (2,000 jobs per hour) - * Build Nodes from 16 Regions of 5 Public and 4 Private OpenStack Clouds - * Rackspace, Internap, OVH, Vexxhost, CityCloud - * Linaro (ARM), Limestone, Packethost - * 10,000 changes merged per month - -OpenStack Scale Comparison -========================== - - * 2KJPH (2,000 jobs per hour) - * Build Nodes from 16 Regions of 5 Public and 4 Private OpenStack Clouds - * Rackspace, Internap, OVH, Vexxhost, CityCloud - * Linaro (ARM), Limestone, Packethost - * 10,000 changes merged per month - - * By comparison, Gerrit has 33k changes all time and 10k in the last 2 years - -Impact of scale -=============== - - * Empower teams to take care of themselves (distributed) - * Efficiency gained from shared solutions (centralized) - * Zuul supports per-repo config, central config, and multiple tenants - * One Zuul install is all you need for an entire Enterprise - -We're Not Alone -=============== - - * Dependencies (libvirt/kvm/xen, mysql/pg, rabbit, - python/javascript, ceph/gluster, ansible/salt/puppet/chef, ovs/odl) - * Adjacencies (kubernetes, ansible, terraform, opnfv, spinnaker) - * Vendors (plugins, products, services, distros) - Developer Process In a Nutshell =============================== @@ -211,6 +124,7 @@ Gerrit Installation * 60G RAM * 16 VCPU * 8x git replicas running Gitea + * 2.13 (sssh, don't tell Luca) Zuul in a nutshell ================== @@ -663,6 +577,8 @@ Cross Source Depends-On: https://gerrit-review.googlesource.com/c/plugins/checks/+/232079 Change-Id: I8e5903f4429c5a1273a6120e0d09c57169e8f938 +.. Insert slides about checks api here? + Lock Step Changes ================= @@ -810,337 +726,6 @@ What about job content? * Ansible is excellent at running one or more tasks in one or more places * The answer to "how do I" is almost always "Ansible" -Playbooks -========= - -* Jobs run playbooks -* Playbooks may be defined centrally or in the repo being tested -* Playbooks can use roles from current or other Zuul repos -* Playbooks are not allowed to execute content on 'localhost' - -Simple Playbook Example -======================= - -* playbooks/tox.yaml - -.. code:: yaml - - - hosts: ubuntu-xenial - tasks: - - name: Run tox - args: - chdir: "{{ zuul_work_dir }}" - shell: "tox -e{{ tox_envlist }}" - -Job Inheritance -=============== - -All Zuuls have a base job - -.. code:: yaml - - - job: - name: base - parent: null - description: | - The base job for Zuul. - timeout: 1800 - nodeset: - nodes: - - name: primary - label: ubuntu-xenial - pre-run: playbooks/base/pre.yaml - post-run: - - playbooks/base/post-ssh.yaml - - playbooks/base/post-logs.yaml - secrets: - - site_logs - -Simple Job Inheritance -====================== - -.. code:: yaml - - - job: - name: tox-py36 - parent: tox - vars: - tox_envlist: py36 - -Inheritance Works Like An Onion -=============================== - - * pre-run playbooks run in order of inheritance - * run playbook of job runs - * post-run playbooks run in reverse order of inheritance - * If pre-run playbooks fail, job is re-tried - * All post-run playbooks run - as far as pre-run playbooks got - -Inheritance Example -=================== - -For tox-py36 job - - * base pre-run playbooks/base/pre.yaml - * tox pre-run playbooks/setup-tox.yaml - * tox run playbooks/tox.yaml - * tox post-run playbooks/fetch-tox-output.yaml - * base post-run playbooks/base/post-ssh.yaml - * base post-run playbooks/base/post-logs.yaml - -Simple Job Variant -================== - -.. code:: yaml - - - job: - name: tox-py27 - branches: stable/mitaka - nodeset: - - name: primary - label: ubuntu-trusty - -Nodesets for Multi-node Jobs -============================ - -.. code:: yaml - - - nodeset: - name: ceph-cluster - nodes: - - name: controller - label: centos-7 - - name: compute1 - label: fedora-28 - - name: compute2 - label: fedora-28 - groups: - - name: ceph-osd - nodes: - - controller - - name: ceph-monitor - nodes: - - controller - - compute1 - - compute2 - -Multi-node Job -============== - -* nodesets are provided to Ansible for jobs in inventory - -.. code:: yaml - - - job: - name: ceph-multinode - nodeset: ceph-cluster - run: playbooks/install-ceph.yaml - -Multi-node Ceph Job Content -=========================== - -.. code:: yaml - - - hosts: all - roles: - - install-ceph - - - hosts: ceph-osd - roles: - - start-ceph-osd - - - hosts: ceph-monitor - roles: - - start-ceph-monitor - - - hosts: all - roles: - - do-something-interesting - -Projects Source Code -==================== - -* Projects are git repositories -* golang git repo naming as been adopted: - -:: - - zuul@ubuntu-xenial:~$ find /home/zuul/src -mindepth 3 -maxdepth 3 -type d - /home/zuul/src/opendev.org/zuul/zuul - /home/zuul/src/github.com/ansible/ansible - /home/zuul/src/gerrit.googlesource.com/gerrit - -Project Config -============== - - * Specify a set of jobs for each pipeline - -.. code:: yaml - - - project: - check: - jobs: - - openstack-tox-py27 - - openstack-tox-py35 - - openstack-tox-docs - gate: - jobs: - - openstack-tox-py27 - - openstack-tox-py35 - - openstack-tox-docs - -Project with Local Variant -========================== - -.. code:: yaml - - - project: - check: - jobs: - - openstack-tox-py27 - - openstack-tox-py35 - - openstack-tox-py36: - voting: false - - openstack-tox-docs - gate: - jobs: - - openstack-tox-py27 - - openstack-tox-py35 - - openstack-tox-docs - -Project with More Local Variants -================================ - -.. code:: yaml - - - project: - check: - jobs: - - openstack-tox-py27 - - openstack-tox-py35 - - openstack-tox-py36: - voting: false - - openstack-tox-docs: - files: '^docs/.*$' - -Project with Many Local Variants -================================ - -.. code:: yaml - - - project: - check: - jobs: - - openstack-tox-py27: - nodeset: - - name: centos-7 - label: centos-7 - - openstack-tox-py27: - branches: stable/newton - nodeset: - - name: ubuntu-trusty - label: ubuntu-trusty - - openstack-tox-py35 - - openstack-tox-py36: - voting: false - - openstack-tox-docs: - files: '^docs/.*$' - -Project With Central and Local Config -===================================== - -.. code:: yaml - - # In opendev.org/openstack/project-config: - - project: - name: openstack/nova - templates: - - openstack-tox-jobs - -.. code:: yaml - - # In opendev.org/openstack/nova/.zuul.yaml: - - project: - check: - - nova-placement-functional-devstack - -Project with Job Dependencies -============================= - -Full DAG of Job Dependencies - -.. code:: yaml - - - project: - release: - jobs: - - build-artifacts - - upload-tarball: - dependencies: build-artifacts - - upload-pypi: - dependencies: build-artifacts - - notify-mirror: - dependencies: - - upload-tarball - - upload-pypi - -Logging -======= - -* Zuul doesn't know about logging - only about URLs to report -* Job Ansible content does *something* with logs -* Job content tells Zuul what URL(s) to report to user -* Allows for some really cool patterns -* (Aiming to integrate this with the Checks Plugin) - -Live Web Preview -================ - -* For static websites, publish built content and report url - -https://review.opendev.org/#/c/635716/ - -https://zuul.opendev.org/t/zuul/build/2ef4ceb232a2467ab56be726f31928ef - -https://470ff2d7584ecc192d95-20fcf28638b4f38abf523a12606af55d.ssl.cf1.rackcdn.com/635716/12/check/zuul-build-dashboard/2ef4ceb/npm/html - -Speculative Container Images -============================ - -* Build container images based on non-published images from other changes -* Test built container images in gate jobs -* Promote exact image from gate after merge -* Soft Dependencies - use artifact from another job only if it was built - -Test Like Production -==================== - -If you use Ansible for deployment, your test and deployment processes -and playbooks are the same - -What if you don't use Ansible? -============================== - -OpenDev Control Plane uses some Puppet (for now) -================================================ - -.. code:: yaml - - - name: Install puppet - shell: ./install_puppet.sh - args: - chdir: "{{ ansible_user_dir }}/src/opendev.org/opendev/system-config" - become: yes - - - name: Copy manifest - copy: - src: manifest.pp - dest: "{{ ansible_user_dir }}/manifest.pp" - - - name: Run puppet - puppet: - manifest: "{{ ansible_user_dir }}/manifest.pp" - become: yes Who Is Running Zuul? ==================== @@ -1155,12 +740,16 @@ Who Is Running Zuul? * GoodMoney (control plane in EKS, adding GKE) * Le Bon Coin * Easystack + * Western Digital * TungstenFabric * Huawei OpenLab * IBM * Red Hat * others ... + Zuul as a Service: https://vexxhost.com/solutions/managed-zuul/ + + Important Links ===============