From b5ceee2004b58df989d3802268265b110a9b068b Mon Sep 17 00:00:00 2001 From: Clark Boylan Date: Fri, 5 Oct 2018 08:52:51 -0700 Subject: [PATCH] Retry failed git pushses on workspace setup We've seen network/ssh failures to push one of the many git repos into the workspace. Followup pushes to other repos succeed. Considering that subsequent pushes do succeed allow for 3 retries to get these workspaces set up. Logs below show this happening. The first push of the repo under test fails (openstack/tripleo-heat-templates) then the push for openstack requirements succeeds. LOOP [mirror-workspace-git-repos : Synchronize src repos to workspace directory] Timeout exception waiting for the logger. Please check connectivity to [158.69.64.232:19885] ubuntu-xenial -> localhost | ssh: connect to host 158.69.64.232 port 22: No route to host ubuntu-xenial -> localhost | fatal: Could not read from remote repository. ubuntu-xenial -> localhost | ubuntu-xenial -> localhost | Please make sure you have the correct access rights ubuntu-xenial -> localhost | and the repository exists. ubuntu-xenial -> localhost | ssh: connect to host 158.69.64.232 port 22: No route to host ubuntu-xenial -> localhost | fatal: Could not read from remote repository. ubuntu-xenial -> localhost | ubuntu-xenial -> localhost | Please make sure you have the correct access rights ubuntu-xenial -> localhost | and the repository exists. ubuntu-xenial -> localhost | To git+ssh://zuul@158.69.64.232//home/zuul/src/git.openstack.org/openstack/requirements ubuntu-xenial -> localhost | 66b2988..740ae1a master -> master ubuntu-xenial -> localhost | * [new branch] stable/newton -> stable/newton ubuntu-xenial -> localhost | * [new branch] stable/ocata -> stable/ocata ubuntu-xenial -> localhost | * [new branch] stable/pike -> stable/pike ubuntu-xenial -> localhost | * [new branch] stable/queens -> stable/queens ubuntu-xenial -> localhost | * [new branch] stable/rocky -> stable/rocky ubuntu-xenial -> localhost | * [new branch] origin/HEAD -> origin/HEAD ubuntu-xenial -> localhost | * [new branch] origin/master -> origin/master ubuntu-xenial -> localhost | * [new branch] origin/stable/newton -> origin/stable/newton ubuntu-xenial -> localhost | * [new branch] origin/stable/ocata -> origin/stable/ocata ubuntu-xenial -> localhost | * [new branch] origin/stable/pike -> origin/stable/pike ubuntu-xenial -> localhost | * [new branch] origin/stable/queens -> origin/stable/queens ubuntu-xenial -> localhost | * [new branch] origin/stable/rocky -> origin/stable/rocky ubuntu-xenial -> localhost | ok: Item: Runtime: 0:00:01.828724 ubuntu-xenial -> localhost | changed: All items complete ubuntu-xenial -> localhost | ssh: connect to host 158.69.64.232 port 22: No route to host ubuntu-xenial -> localhost | fatal: Could not read from remote repository. ubuntu-xenial -> localhost | ubuntu-xenial -> localhost | Please make sure you have the correct access rights ubuntu-xenial -> localhost | and the repository exists. http://logs.openstack.org/56/605356/17/check/openstack-tox-linters/4323abf/ara-report/result/bb14d3fe-338b-4d7a-852f-e66c60d13493/ Related-Bug: 1384373 Change-Id: I6c34d1a6e34762f78b9dd2a2d2a07ceb131a64cc --- roles/mirror-workspace-git-repos/tasks/main.yaml | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/roles/mirror-workspace-git-repos/tasks/main.yaml b/roles/mirror-workspace-git-repos/tasks/main.yaml index dc01fd9a8..2c9f19184 100644 --- a/roles/mirror-workspace-git-repos/tasks/main.yaml +++ b/roles/mirror-workspace-git-repos/tasks/main.yaml @@ -12,6 +12,13 @@ chdir: "{{ zuul.executor.work_root }}/{{ item.value.src_dir }}" with_dict: "{{ zuul.projects }}" delegate_to: localhost + # We occasionally see git pushes in the middle of this loop fail then + # subsequent pushes for other repos succeed. The entire loop ends up + # failing because one of the pushes failed. Mitigate this by retrying + # on failure. + register: git_push + until: git_push is success + retries: 3 # ANSIBLE0006: Skip linting since it triggers on the "git" command, # but push is not supported by ansible git module. tags: