zuul
/
nodepool
Code Issues Proposed changes

Fix potential race in image upload cleanup 

It's possible for an active image upload to complete during the
builder cleanup phase, which would release its lock and change
the state to READY. If the cleanup thread has cached the UPLOADING
state and checks to see if the upload is in progress (just after
the lock has been release), then we could accidentally delete the
upload record because we don't have the correct state. This adds
a second check on the state to make sure it didn't change on us.

Change-Id: I60be43e999cf86d4c3c46e6ea69ecd1bcb69f533
This commit is contained in:
David Shrewsbury 2017-02-16 09:20:39 -05:00
parent cb1f52634a
commit ac1e0bac53
1 changed files with 8 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
nodepool/builder.py
View File

@ -373,6 +373,14 @@ class CleanupWorker(BaseWorker):
if (upload.state == zk.UPLOADING and
not self._inProgressUpload(upload)
):
# Since we cache the uploads above, we need to verify the
# state hasn't changed on us (e.g., it could have gone from
# an in progress upload to a successfully completed upload
# between the getUploads() and the _inProgressUpload() check.
u = self._zk.getImageUpload(image, build_id, provider,
upload.id)
if upload.state != u.state:
continue
self.log.info("Removing failed upload record: %s" % upload)
self._zk.deleteUpload(image, build_id, provider, upload.id)
elif upload.state == zk.DELETING: