Make upload workers processing image layers only once (as the best
effort). This also reworks and simplifies locks management for
individual tasks now managed for the PythonImageUploader class
namespace only.
When fetching source layer, cross-link it for the target
local image, whenever that source is already exists. When pushing a
layer to a target registry, do not repeat transfering the same data,
if already pushed earlier for another image.
The 1st time a layer gets uploaded/fetched for an image, that image and
its known path (local or remote) becomes a reference for future
cross-referencing by other images.
Store such information about already processed layers in global view
shared for all workers to speed-up data transfering jobs they execute.
Having that global view, uploading the 1st image in the tasks list as a
separate (and non-concurrent) job becomes redundant and now will be
executed concurently with other images.
Based on the dynamically picked multi-workers mode, provide the global
view as a graf with its MP/MT state synchronization as the following:
* use globally shared locking info also containing global layers view
for MP-workers. With the shared global view state we can no longer
use local locking objects individual for each task.
* if cannot use multi-process workers, like when executing it via
Mistral by monkey patched eventlet greenthreads, choose threadinglock
and multi-threads-safe standard dictionary in the shared class
namespace to store the global view there
* if it can do MP, pick processlock also containing a safe from data
races Manager().dict() as the global view shared among cooperating OS
processes.
* use that global view in a transparent fashion, provided by a special
classmethod proxying access to the internal state shared for workers.
Ultimately, all that optimizes:
* completion time
* re-fetching of the already processed layers
* local deduplication of layers
* the amount of outbound HTTP requests to registries
* if-layer-exists and other internal logic check executed against the
in-memory cache firstly.
As layers locking and unlocking becomes a popular action, reduce the
noise of the debug messages it produces.
Closes-bug: #1847225
Related-bug: #1844446
Change-Id: Ie5ef4045b7e22c06551e886f9f9b6f22c8d4bd21
Signed-off-by: Bogdan Dobrelya <bdobreli@redhat.com>
46f8129894